The Carlton Dance

AI has long received a lot of public attention for the functionality that it offers in a broad range of data related problems ranging from web applications with big data to smart interfaces to search optimization.  However, AI has also been gaining a lot of traction for its ability to solve problems in the physical domain as well.  Following is a summary of some of the more interesting technologies that I have been following lately

Plasma Shaping for Nuclear Fusion

Plasma is typically considered to be a fourth state of matter, similar to gases in some ways, but containing a large quantity of charged ions and typically occurring at high temperatures.  Control of plasma is particularly challenging and important as it relates to nuclear fusion, since temperatures are far beyond the range which convention materials can withstand.

The most common solution is to maintain isolation between the superheated plasma and the walls of containment vessels using strong magnetic fields.  Maintaining a proper shape of plasma clouds is very important since this affects efficiency, even distribution of temperature and so forth.  In the worst case, losing control of the cloud shape can cause damage to the reactor walls, thus requiring extended shutdown and costly repairs.  Optimized cloud shaping has traditionally been a daunting task due to the complexities of field management, precise requirements and the time needed to change field inputs.  However, researchers for DeepMind and the Swiss Plasma Center recently announced that they had used reinforcement learning to achieve a variety of specified cloud shapes.  This technique will be particularly promising for larger, more complex fusion reactors which will likely be required in order to effectively commercialize this technology.

https://www.sciencealert.com/physics-breakthrough-as-ai-successfully-controls-plasma-in-nuclear-fusion-experiment

Dynamic Systems

Dynamic systems in mechanical engineering have often been considered too complex to model.  Analysis involving such systems typically relies upon methods of estimation which have proven sufficient for many cases, but with the advent of AI there is a growing level of interest about whether these systems could actually be fully modeled and predicted (and what new advances such a capability might unlock).

Recently the National Science Foundation approved grants to investigate such topics, particularly including turbulence in compressible fluids (which includes most gases, such as the Earth’s atmosphere).  Flow of a compressible fluid around an object may generally be thought of as “laminar” or “turbulent”.  It is relatively easy to create turbulent flow, but laminar flow may only be achieved with smooth objects of conducive shape and limited size.  One of the main disadvantages of turbulent flow is that it creates far more drag, hence such research could have enormous implications to fields such as aviation.

https://www.washington.edu/news/2021/07/29/uw-to-lead-new-nsf-institute-for-using-artificial-intelligence-to-understand-dynamic-systems/

Particle and Quantum Physics

AI has also been making strong inroads in the fields of particle and quantum physics, including devising experiments (sometimes accidentally) that are beyond the comprehension of the scientists that these algorithms theoretically serve.  One such case occurred at the University of Vienna in which researchers created a program to simulate results of a quantum physics test, then added the capability to construct tests from combinations of functional elements.  When the algorithm found a useful tests, researchers would save it, which led to increasingly complex tests built on top of other successful tests.  Eventually the algorithm outpaced the researchers and began making discoveries that they had not contemplated.

https://www.scientificamerican.com/article/ai-designs-quantum-physics-experiments-beyond-what-any-human-has-conceived/

Conclusion

While we often think of AI as a phenomenon which occurs only in some virtual data space, it is finding traction in a range of arenas which may unlock new understandings about the physical world around us.  Given the newness of this field and the rapid acceleration of such techniques, I expect to see some fascinating discoveries and transformative developments built upon them.

Read the post...

Over the past few weeks, I have seen quite a bit of discussion about how to land a job at one of the big tech companies, which makes it a bit ironic that I am currently moving on from Amazon after a ten-year stint.  Given the number of questions that I have received about what it is like to work at Amazon, I decided to share a few observations and experiences.

Entering the Amazon

Prior to joining Amazon, I had worked for a plethora of small, early-stage companies, most of which few people have heard of.  This was a great experience because it gave me an opportunity to be involved in every aspect of the business and every part of its lifecycle.  I also found that I had more of a sense of ownership in ventures which I had helped start from the ground up.  However, these smaller, younger companies could also be a bit confining.  At some point the scale of the company itself became a limitation to my growth and I decided that I would like to try working in a larger company.

Before I began applying, I spent about 50 hours reviewing job postings for companies and jobs that interested me.  During this process I made note of any skills, tools and certifications which were commonly requested and devoted a significant amount of time to acquiring all of these over the course of a couple years.  Strength of my resume was enough to get me an interview right away, the main focus on which related to behavioral traits.  Once my soon-to-be new team was satisfied that I possessed the necessary skills to do the job, they were interested in finding out whether I was able to work well in a team, cope with ambiguity and find solutions to problems in the face of adversity.

Life in the Amazon

I saw a classmate’s blog recently which described a fairly thorough and well-organized onboarding process at a new employer.  While I am happy for my classmate, my own experience bore little resemblance to the one described.  My first week at Amazon was spent in Arizona, where I received my badge, laptop and a quick orientation to tools/culture of the company.  After that I spent my Associate Experience Week (AEW) back home in Tennessee.  This is actually a pretty neat experience which gives incoming managers an opportunity to perform all of the jobs most commonly performed by front line associates in our fulfillment centers.  The goal of this experience is to ensure that the participant has a solid understanding of key processes and the challenges that our associates face in performing these processes.

At the end of AEW, I called my manager in Seattle to ask what I was supposed to do next.  He asked me if I could be in Delaware by Monday morning, then gave me an address, a contact name and the door code for a construction trailer.  Over the weekend my wife and I packed up the car and relocated to Delaware for the duration of the project.  I should note that more established teams may have more thorough onboarding processes, but given the rate of growth, most teams have been operating in their current modes for no more than a couple years.  Since our team was focused on rollout of new facilities, our team also tended to be very nomadic and had relatively little opportunity to work together in the same location.  Over the course of my tenure with Amazon, my wife and I relocated to 11 different cities around the US, Canada and Australia.

Based on my observations and experience, most of what we hear in the news about conditions at Amazon fulfillment centers is incorrect or out of context.  However, the reports about work/life balance for technical specialists and managers was fairly consistent with my experience.  I would estimate that I worked more than 50 hours per week about 70 percent of the time… sometimes much more.  I was only able to use about half of my allocated PTO and even on holiday I wound up checking/responding to email and so forth.

Escape from the Amazon

All in all, I am glad that I joined Amazon.  I am glad that I stayed as long as I did, and I am glad to be moving on.  I was able to work on some interesting projects and became acquainted with some really great people.  Compensation was pretty good and experience was invaluable.  Since joining Amazon, I have also been approached by recruiters for other large tech companies including Microsoft, Meta and Oracle.  However, at this point I have plenty of options and the job has been taking more than it is worth.  Joining Amazon was a great career move, but at this point I am excited about a great life move and look forward to working on things that I love.

Read the post...

“You may watch TV, but in Soviet Russia, TV watches you!” – Comedian Yakov Smirnoff

With the recent concerns about a possible conflict in Ukraine, I have been reminded of a Bellingcat report which was issued in 2014.  For those who are not familiar with Bellingcat, it is a group which specializes in open-source investigations, typically based on information which is available in the public domain.  By combining diverse investigative specializations with modern computing methods, they have been able to produce some impressive and often conclusive answers to challenging questions.  Even if you are not researching a specific topic, it may be interesting to read through some of their reports on current events.

https://www.bellingcat.com

The report in 2014 entitled “Origin of Artillery Attacks on Ukrainian Military Positions in Eastern Ukraine between 14 July 2014 and 8 August 2014” was a revelation for me in multiple regards.  For one, this was the first conclusive evidence that I had seen which demonstrated that the Russian military had directly attacked the Ukrainian military in order to prevent assertion of government control over the Donbas region in eastern Ukraine.  For another, this was the watershed which made me realize exactly how transparent the world had become and that the challenge had evolved from “how to get information” to “how to cope with the amount of information that is available, and piece it together into mosaics that make sense.”

Background

In early 2014, President Yanukovych was forced to flee Ukraine after failing to suppress large scale protests in Kyiv and throughout the country.  The protests had erupted largely in response to persistent corruption, but President Yanukovych’s close relationship with Russian leadership, added a geopolitical aspect to an already challenging situation.  Russia responded by invading and annexing Crimea, after which skirmishes also broke out in the Donbas region, near the Russian border.

The Ukrainian government initiated an operation to reestablish control in the region, which was largely successful until it met precision rocket attacks beginning in July 2014.  Given the relatively disorganized and ineffective resistance that separatist fighters had demonstrated prior to this point, many analysts speculated that the attacks had been carried out by Russian forces, which the Kremlin denied.  Ultimately these attacks prevented the Ukrainian government from completing its operations in eastern Ukraine and the Donbas region now remains in a murky condition without effective rule of law or universal international recognition.

The Bellingcat Investigation

To establish attribution for the attack, the Bellingcat team began by examining satellite images of the impact craters from Google Earth.  Even now, seven years later, the locations at which rockets landed can be identified on Google Earth as areas of discoloration in the field at which Ukrainian forces came under attack.  In satellite images taken July 16, 2014 (two days after the attack), the shapes of craters were clear enough to reveal the direction from which projectiles had landed.

(The above image is taken from the Bellingcat report, page 5)

(The above image was taken from Google Earth on February 10, 2022)

Armed with this information, the Bellingcat team began looking for possible launch positions along the line of approach and discovered a likely candidate 14.6km away from the impact zone, near the Russian village of Seleznev.  The area was directly on the projectile path that crater analysis had indicated and within range of all multiple launch rocket systems (MLRSs) commonly used by Russian or Ukrainian armed forces.  Moreover, the images showed tracks of the MLRSs and scorch marks in the July 16 satellite images, neither of which was present in previous satellite images, thus suggesting that rockets had been launched from this location shortly prior.

(The above images was taken from the Bellingcat report, page 7)

Moreover, the team was able to glean some interesting information from the tracks themselves.  For one, they established the direction from which MLRSs had arrived and to which they returned, thus demonstrating that these units had come from within Russia rather than originating in Ukraine as some analysts had postulated.  Through process of elimination, the Bellingcat team was also able to determine that the launch platforms used were BM 21s because all other platforms operated by Russian or Ukrainian forces would have wider tracks and/or multiple steerable axles.

Next, the Bellingcat team turned its attention to several videos showing Russian MLRSs in use, which were posted on YouTube and VKontakte.  Although the specific locations were not specified, each video contained visual queues (such as buildings, towers and distinctive natural features) which allowed the Bellingcat team to triangulate the location from which each video had been filmed, and thus to determine the location of the launch platforms shown.

The Bellingcat team applied such techniques to other rocket attacks against the Ukrainian military between July 14 and August 4 with similar results.  One final touch was that some of the videos captured the angle of inclination at which rockets were being launched.  The BM 21 Grad uses self-propelled projectiles with no ability to regulate their velocity, hence operators must adjust the angle of launch inclination in order to achieve the desired range.  Comparing the inclination shown in video with published firing tables for these systems revealed that the expected range to impact corresponded with the distance to Ukrainian positions which appear to have been targeted.

In Reflection

At the time, I was very surprised that Bellingcat had been able to so conclusively establish that attacks against the Ukrainian military had been carried out by the Russian military. Bellingcat is one of the more prolific open-source investigative groups, but they are by no means alone.  Since the 2014 paper establishing attribution for rocket attacks on the Ukrainian military, I have seen many other well-documented analyses based predominantly on mosaics of publicly available information.

The impact of the events described above were clear for the Ukrainian military.  The attacks killed dozens of service members and injured hundreds more, operations to restore control in Donbas failed and the murky condition of the Donbas region persists 7 years later.  However, there is a broader impact which is considerable and difficult to fully quantify.  Massive amounts of digital information are now available and publicly accessible, thanks to the internet.  The effectiveness of open-source investigations such as this demonstrate the level of visibility that is available to even independent sleuths who have the ability to identify relevant information and make connections which are informative.  Particularly with the proliferation and democratization of AI-based tools, we are all going to have to come to terms with a new reality in which the world is very transparent.

The full Bellingcat report may be found here…

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwieyd2SlPf1AhX5CTQIHfyIChsQFnoECAIQAQ&url=https%3A%2F%2Fwww.bellingcat.com%2Fapp%2Fuploads%2F2016%2F02%2Fbellingcat_-_origin_of_artillery_attacks_02-12-15_final1.pdf&usg=AOvVaw1Iwi4LUMwiHp0-oY-HGkBW

Read the post...

The goal of our project is to identify the best security to be traded using an existing trading engine.  In short, the key deliverable from our project will be code which selects the best of 18 potential leveraged ETFs to trade in over the ensuing month (based primarily on volatility and momentum).  The existing momentum-based trading engine will then initiate trades based on shorter-term variations in momentum indicators.

To this end, we intend to rely upon machine learning methods, potentially including deep learning.  Deep learning is a technique which has gained enormous momentum in adoption during the past five years.  It is based on neural networks, which are patterned after the learning methods of the brain to an extent.  I find this interesting because I remember neural networks being the cause of excitement during the 1990s, after which they fell out of favor for an extended period of time.  In fact, I once proposed such a solution to a mechanical engineering problem when I was completing my first degree, to which a professor whom I respected a great deal responded “what you’re describing is a neural network, and they don’t work in practice.”

As such, it was a bit of a revelation to me when I discovered a year or so ago that deep learning is essentially a modern incarnation of methods which were previously dismissed by a very competent community.  I have held an interest in learning about deep learning for quite some time, but this will be the first time that I have had an opportunity to get any hands-on experience.  Now that I am beginning preparation for an implementation, I have been wondering why neural networks feel out of favor and how there has been such a resurgence with deep learning recently.  Following is quick summary of what I have found.

What is a neural network?

To greatly oversimplify, a neural network is mesh of nodes which are interconnected with weights and thresholds.  When the input of one network meets the specified threshold, it triggers an output to nodes in the subsequent layer.  A neural network must be comprised of at least three layers (an input, and output and additional intermediate layers which perform processing.  The name harkens to the fact that this architecture mimics the processing structure of biological neurons in the brain.  The benefit of a neural network is that it is flexible and can be applied to a wide range of image/pattern recognition patterns.  With repeated use and exposure to more data, the network can be optimized (i.e. “learn”) to perform better.

Why did neural networks fall out of favor?

The general concept and approach of neural networks has not changed so much since the last century.  This begs the question, why did they fall out of favor then and why the resurgence now?  The simple answer is that several prerequisites for widespread adoption have only been met within the past 20 years.

One factor is the adequacy of available resources, particularly process and storage.  Deep learning applications are compute-intensive, hence processors only reached a performance and cost point which would lend itself to widespread application within the past 20 years.  The recent wave of virtualization (including cloud computing) has further accelerated this trend.

A second factor is the extent of data available.  Widespread adoption of internet and the world wide web has driven creation and ingestion of a volume of data which was virtually unimaginable in the 1990s.  Moreover, through its intended use, much of this data is prequalified in such a way that it can be readily used for training and testing models.  Moreover, it has made possible the adoption of various applications which can benefit from machine learning, thus creating both the opportunity and demand for neural networks.

What triggered the recent explosion in deep learning interest?

The key development most responsible for the surge in deep learning’s popularity was the success of AlphaGo.  Go is a game in which two players alternate turns placing colored stones on a gridded game board with the goal of surrounding their opponent’s stones or areas on the board.  Although conceptually straightforward, the game has a vast number of potential permutations and requires layers of strategy which far exceed those of chess.  Although computers have been competitive with the best human players for quite some time, most experts did not believe that a computer would ever be able to achieve a sufficient level of intuition to defeat a human player.

AlphaGo achieved this feat using a deep learning approach, with which it decisively defeated the current European champion in 2015.  This accomplishment was topped the following year when an updated version defeated the original in 100 consecutive matches.  This was particularly noteworthy because the updated version relied upon unsupervised learning.  Collectively these two achievements highlighted the potential of a previously discarded technique, thus triggering a virtual gold rush for adoption and implementation.

It is interesting to note that although the metrics on which decisions were made are recorded and available to researchers, not all of them are fully understandable by humans.  This is also true of other problems (such as protein folding) which have successfully incorporated deep learning.  Thus the implication would seem to be that this technique has enabled computers to tease out certain actionable patterns which are currently beyond our ability to comprehend.

Read the post...

(For anyone who may be wondering, a “Crazy Ivan” is a maneuver for adjusting the direction of a ship.  It became part of the broader pop culture after a reference in the movie Hunt for Red October.)

This week we learned that we would need to make a bit of a course correct in our project plan based on input from our sponsor (hence the title of the post).  We had met several times to discuss the project and we were aware that our work would have to interface with existing code, which we received last weekend.  However, when we reviewed the existing code, we realized that it functioned a bit differently than we had anticipated, which helped to surface some larger gaps in expectation.  As such, we held a quick meeting with our sponsor on Wednesday to confirm our assessment and discuss the path forward.

What is Changing?

Our initial understanding of the project was that we would seek to algorithmically identify a small group of securities which would be suitable for trading based on a momentum strategy.  To this end, we were planning to develop functions which would measure volatility and momentum based on the preceding time period, which would in turn be used for predicting the relative strength of each asset during the upcoming period.  From this data, we understood that existing code would construct a portfolio and initiate Buy/Sell orders to implement the selection of a portfolio to be bought-and-held for a period of months.  However, when we received and reviewed the existing code, we realized that it functioned a bit different than we anticipated, which surfaced some other gaps.

First, the intent of our project code is to select a single volatile security with increasing momentum from a specified universe of candidate securities.  The function of existing code will be to buy and sell this specified security repeatedly in response to shorter-term swings in momentum based on a daily level of resolution.  Thus, our code might be run once monthly to update the selection of the preferred asset for trading.  However, once the asset has been selected, existing code may buy and resell shares in response to momentum fluctuations multiple times before the asset selection is run again (as opposed to our initial understanding that the portfolio would be bought-and-held until the next selection cycle).

Second, our code will also need to incorporate its own hypothetical performance from previous periods based on different asset selections as inputs to the logic of selection for the upcoming period.  These will be additional to the volatility and momentum indicators that we were already planning to use as inputs, but this complicates the problem statement because we will have to run back tests for all assets in the defined universe and all time periods which we intend to consider as inputs for our selection algorithm.

How Will This Affect the Project?

Below is the structural plan for the solution that we planned to implement.

To elaborate upon the Security Scoring Module from the diagram above, following is a high-level overview of the logical process which we had envisioned:

However, based on a review of existing code and the meeting with our sponsor on Wednesday, we will need to make two adjustments to how the application will work.

First, instead of just initiating one-time buy/sell signals to update the portfolio each time we choose a new security for trading, the Trade Signal Generator from Figure 1 above will actively generate multiple transactions based on a given security selection (before the next update in the security selection).  Disruption to our plans will be fairly minimal since this is already implemented in existing code which will be downstream from our area of focus.

The second adjustment will be a bit more disruptive.  Aside from the user-defined configuration, the logic depicted in Figure 2 could be implemented solely on the basis of historical data from a third-party source.  However, based on the call with our sponsor on Wednesday, we now know that we will need to incorporate inputs based on the hypothetical past performance of the algorithm itself, based on various asset selections.

Why Does This Complicate Things?

Hypothetical performance of an algorithm can be measured by “back testing,” which refers to the process using historical data as inputs to simulate how the algorithm would have behaved if it had been implemented in the past.  This approach has limitations (such as susceptibility to overfitting and past data’s limited ability to predict the future), but it is often the best indication of likely performance that one can get without paper trading or actual trading in a live environment.

We always intended to back test algorithms that we were considering for this project using QuantConnect Lean to get an indication of their possible performance.  QuantConnect is designed to efficiently implement back tests and provide relatively detailed data about how the algorithm would have performed under the conditions specified.  However, we did not anticipate incorporating back tests of multiple hypothetical cases as inputs for our selection logic.  Doing so adds a layer of complexity because we are unaware of any way to efficiently trigger a suite of back tests and collect results for each using our code.

What is the Solution?

Given enough time, we could create our own code to simulate the back testing functionality in QuantConnect.  However, since the primary focus of this project is not to reinvent existing software, we agreed with our sponsor that it would be a more valuable use of our time to collect results from a defined suite of back tests manually, save these in a table and then import this data into our algorithm for reference as needed.

What did we Learn?

Frankly, this was mostly a success story.  Although our team did not correctly understand all requirements for the project initially, we did realize there could be some confusion, had requested to review the existing code as a means of identifying any gaps, and proactively discovered the gaps early in the project.  Our first lesson learned should be to always model this behavior of being attentive to possible expectation gaps and driving to clarity as quickly as possible.

One root cause of the miscommunication was an imprecise/inconsistent use of specific terms from the Finance industry.  It merits noting for future reference, that this can be a common source of miscommunication, especially with a group is relatively new and stakeholders have not had an opportunity to align thoroughly on their use of jargon.

Read the post...

Our Week 2 module included a Google video entitled “The Myth of the Genius Programmer,” which really resonated with me and I found myself still thinking back to it this week.  The key underlying tenant of the discussion was that epic achievements in software development tends to result from the collaboration of high-functioning teams rather than the genius of a single individual working in isolation.  The conclusion from this revelation is that being able to improve a team’s performance is actually a more important skill for success than extraordinary individual talent.

By extension, the speakers also discussed that failure and learning are natural parts of the development process, and therefore developers should embrace and plan for both.  Personally, I could not agree more, but I would argue that persuading leadership to provide an environment in which such behavior can flourish is often the bigger challenge.  This can be particularly challenging in the context of a large organization where perverse incentives may persuade managers to act in a manner which is inconsistent with the goal of providing a healthy environment for development.

Failed tests, mistakes and questions which demonstrate a lack of knowledge are all important parts of the development process, and although concerns which prevent such behavior may represent impediments to progress, that does not mean that the concerns are unfounded.  Unfortunately, in their zeal to drive progress, raise issues or even just appear relevant, managers often take actions which perpetuate a team’s discomfort with working in a humble and transparent manner.

The Great Experiment

For the past several years I have worked for one of the large tech companies which is known for stressing results.  After repeatedly encountering teams which were reserved and risk averse, I decided to focus on providing my team the kind of environment in which I would like to work and in which I would feel free to perform my best work (although this required some steps which were not consistent with my own leadership’s initial preferences).

I didn’t employ any magic formula, but mostly it came down to being trustworthy and transparent.  Following are a few principles which I would consider to be key.

Use commitments fairly

Even skilled, knowledgeable professionals may have difficulty projecting how long a task which is new to them will take to complete.  This is particularly true in software development, where most tasks are somewhat unique and problems can be difficult to anticipate.  As a result, well intending developers regularly discover that estimates which they shared in good faith are actually unachievable.

In contrast, program/project managers tend to favor certitude, which creates tension when planning is necessarily based on inexact inputs.  Unfortunately, there is a commonly repeated pattern in which program/project managers insist on team members providing schedule estimates, challenge any estimates they believe to not be aggressive enough, and then cite any miss as a failure, indicative of poor performance.  Honest and transparent communication within a team is important, but can only be achieved if commitments are treated fairly (such as not over-representing the strength of commitments from stakeholders).

Be transparent

In all aspects, it is important for a team leader to exhibit the behavior that (s)he wants to the team to adopt.  This is particularly true with regard to transparency.  In any team, I always make a point to air any problems, issues or mistakes as openly as I possibly can.  I have heard a few opinions that this “lowers the bar” by making stakeholders comfortable with a lower level of achievement, but in my experience, this is only the case if you have hired the wrong people.  In my experience, most people want to succeed together, but the reality is that things do not always go smoothly and it is important to set the tone for a team to share such things openly without fear of judgement.

Ask dumb questions

Much of what I just said about transparency applies to this point also.  Most people don’t enjoy the appearance of ignorance, yet when working on something new, it is completely normal to not have a high level of knowledge about some of the subject matter.  As such I always make a point to ask some fairly elementary questions that I think I already know the answer to.  These are mostly tailored to ensure that everyone on the team hears the answers and has an aligned understanding of the problem at hand.  These types of questions also set a tone which makes it more comfortable for other stakeholders to raise questions they might otherwise find embarrassing, and with fair regularity I receive an unexpected answer which refines or corrects my understanding of the topic.

Be positive and focus on solutions

Most people want to work in a positive, collaborative environment to find solutions to problems.  However, organizational pressures can tempt otherwise well-intended people to become harsh, critical or blamey, particularly when they themselves are uncomfortable or self-conscious about some issue.  One of the best ways to avoid this mentality is to always focus the conversation on solutions.  Discussion of mistakes or failures is only productive to the extent that it provides some learning which helps us understand the problem to be solved.  When people are confident that they will not be attacked for missteps, they tend to surface problems faster so that the entire team can work on them and they waste less time on trying to defend themselves.

Recognize achievement

It seems simple, but many managers fail to recognize the achievements of the team and its members.  This recognition is important, particularly in that it also demonstrates the importance of stakeholder(s) success.  Where possible, I tend to communicate failures as a team failure without calling out specific individuals to an outside audience.

Results of the Great Experiment

Overall, results exceeded my expectation.  Initially team members seemed surprised or even suspicious when I introduced a way of working together which was different from what they had seen previously.  However, after a few weeks, virtually everyone embraced the approach enthusiastically.  Meetings took on a super-positive and supportive tone in which people stopped blaming other stakeholders for issues and focused on solutions.  Everyone on the team felt a strong sense of support from each other, with the result that missteps and problems (even those which might be embarrassing in a more typical environment) came to the forefront almost eagerly.  People were no longer concerned about taking a calculated risk of failure and when a test didn’t work out, the entire team would have a good-natured laugh together then immediately fly into finding a solution.

Although it is difficult to provide a specific metric, I did note that initiatives were completed faster and with fewer misses once the team had embraced the open style of collaboration.  In one case, the team delivered a key strategic win which leadership had assessed had less than a two percent probability of success at the outset.  More importantly to me, people were free to deliver their best work with joy, and our team became one that people tended to gravitate to.

However, if I am totally honest about the results, I also have to note that our leadership decided that although the team was functioning at a very high level, we could get more out of them by taking a tougher stance to hold people accountable.  In the short term this resulted in more aggressive schedule plans, but no improvement to actual delivery.  In the longer term, this resulted in a loss of trust and a return to a mode of work resembled what it had been before I took over the team.  Which brings me back to my original point… it is great to instill an appreciation for transparency and openness in developers, but this will only succeed if we also provide an environment in which people are allowed to work in such a manner and flourish by doing so.

Read the post...

What is Algorithmic Trading?

For millennia, people have invested capital in productive endeavors in the hope of earning a return on their investment.  Over time, standards of financial accounting emerged, which provided insight into the operation and performance of such enterprises.  Eventually markets were established in which investors could buy and sell financial assets based upon their assessments of fundamental value.  These innovations have yielded a mechanism by which entrepreneurs seeking capital may coordinate with willing investors to allocate resources in a very efficient way.

Even a few decades ago these markets were driven by mostly manual analysis of financial statements and other inputs which might reveal the fundamental value of the underlying asset.  However, the abundance of market data which is now available in real time lends itself to a plethora of automated trading strategies.  Recent assessments indicate that more than 60 percent of the total trading volume on US public markets is now driven by algorithmic trading.

The most famous form of algorithmic trading is probably high frequency trading (HFT), in which market participants seek a slight edge on competitors (often measured in milliseconds) by exploiting small, temporal inefficiencies in the market.  In order to beat competitors to these opportunities, the major players implement fairly complex rules which are often executed on platforms located as close to exchanges as possible in the interest of speed.  This has sparked a FinTech arms race of sorts as market participants compete to secure the best locations, fastest equipment and most efficient algorithms.  However, like any arms race, this obsession with fastest execution carries a certain level of risk.

Perhaps the best example of this is the Flash Crash of 2010.  At approximately 2:32pm EDT on May 6, a large trader initiated the sell of a single type of asset valued in excess of $4 billion.  This transaction was large enough to look like a trend to various algorithms, which started trading in a dynamically unstable manner.  During the ensuing 36 minutes, the market logged its second largest intraday swing in history, even though there was no rational motivation for such behavior.  In the end, more than a trillion dollars of capital was wiped out and both the public and regulators became aware of the potential risks of algorithmic trading.

Figure 1 – Dow Jones tracker from May 6, 2010

Momentum Trading

Our project will not enter the foray of HFT, but rather will seek to capitalize on a more stable strategy of momentum investing.  In short, momentum trading recognizes that stocks which have been outperforming the market during the past several months, typically tend to do so over an ensuing period of similar duration.  Similar logic applies to stocks which have been underperforming the market as well.  In truth, the correlation only tends to prevail a little more than 50 percent of the time, but the distribution of performance tends to be skewed towards larger winners than losers, which often leads to momentum strategies out performing the market overall.

Several studies have shown that momentum investing is one of the few strategies which regularly delivers better returns than the market as a whole, although there are multiple opinions and no consensus about why the viability of this method persists. According to the efficient market theory, once a market bias is understood, it should be exploited and thus disappear, but this has not been the case with momentum-based trading. Several theories about the continuing success of momentum methods seem to revolve around persistent behavioral biases of market participants.

Universe Selection

Trying to apply optimization strategy to the market as a whole can be dauting at best and untenable at worst due to the sheer volume of potential investments available.  As such, it is typical to apply some coarse screening criteria to identify a manageable quantity of qualified investments which merit further analysis.  This group of pre-qualified assets is considered to be the investible universe, and deeper analysis will be applied to these assets in order to generate a list of trades.

Our project will focus predominantly on universe selection, and will integrate with existing modules which generate buy and sell signals for based on a broader automated strategy.  We will mostly seek to choose risky (i.e., volatile) assets which have demonstrated favorable momentum over the preceding period.  We will also seek to determine an optimal time window to evaluate, but our initial estimate (based largely on our sponsor’s experience) is that the optimal window to review will be approximately 3 to 6 months.

Asset Selection Criteria

For the first version, we will choose from roughly ten frequently traded ETFs by assessing growth and volatility over the preceding period.  Using this model, we will develop a module and integrate it with existing software which generates actual buy and sell signals for assets from the selected universe.  We will then seek more complex and nuanced logic which will provide better performance.

We will assess our code’s performance based on its ability to choose risky assets with strong returns.  For the initial version with approximately ten assets to choose from, we will check the performance over another period based on each of the ten assets available, and then see how the preferred asset (based on the criteria from our algorithm) performed in comparison to the others.  This will give a coarse, directional indication only, but we will look to tailor a more sophisticated testing regime before implementing more complex (and hopefully effective) strategies.

Read the post...

I was born and raised in Tennessee.  Don’t bother asking which city, because there really wasn’t one, but I still consider Tennessee to be my main home even though I have been away for 15 years.

When I was 14, I took a drafting class because my high school required me to take a vocational course.  To my amazement, I loved it!  It is fair to say that my first efforts met with mixed results, but I had a really awesome teacher who bore with me and encouraged me to stick with it.  As a result, I wound up taking drafting every year in high school and eventually won three state titles in individual competitions.

My enthusiasm for drafting led me to study mechanical engineering, which I also thoroughly enjoyed.  I had every intention of working as a mechanical engineer, but while I was in school a new contraption called the “World Wide Web” made its debut.  (Spoiler alert… it turned out to be a hit!)  During the dotcom boom there was such a shortage of IT talent that companies were hiring anyone with a cross-trainable skill set, so straight out of college I went to work on the road crew for a company that was building out its own national network.

I thought this would be a short-term adventure (much like traveling abroad for a summer or running away to join the circus), after which I would come back to reality and find gainful employment as an engineer.  However, after 4 promotions in 18 months, gainful employment had found me.  My responsibilities were ridiculously above my abilities at that time, but with a lot of effort I was able to grow into the role reasonably well and got to experience the full ride of the dotcom boom firsthand.

Unfortunately, I also got to experience the dotcom bust as well, but the good news is that the bankruptcy court hired me to decommission and recover the entire network that we had just built.  Although less satisfying, this actually proved to be considerably more lucrative than building it had been.  (I also wound up inadvertently setting a personal record by visiting 41 US states in 40 days while we were consolidating assets).

After that, I continued working on one short-term project after another for a few years.  In 2006 I was working on a project in California, when I got a call asking if I could consult on a project in Ukraine.  I agreed, but wound up staying more than 5 years.  During that time, I helped prep a telecom company for sale, after which the new owners asked me to stay on to help launch and run the company.  It was also during my tenure in Ukraine that I began studying Finance seriously.  Collaborating with the C-suite officers on two equity events persuaded me that I needed to learn more in this domain, so I completed the Chartered Financial Analyst (CFA) program, which is generally more common for investment bankers than engineers.

In 2012 I returned to the US (with my wife, whom I had met in Ukraine) and went to work for Amazon, but we still try to go back to Ukraine each year and just bought a dacha (i.e., “country house”) outside of Kyiv.  With Amazon, I spent 5 years setting up fulfillment centers in the US, Canada and Australia, which had very little relevance to my prior experience.  In that role I mostly coordinated construction, conveyors, racking, robots and so forth.  This has been followed by 4 years setting up data centers for AWS, although the hyperscale data centers that support cloud computing have limited similarity to the ones I worked in straight out of college.

When I look back on my career path to date, I am reminded of a well-intending advisor who asked me shortly before graduation what I wanted to be doing in 10 years.  I informed him that if I was doing anything that I could possibly anticipate at the time, I would be sorely disappointed.  I am happy to report that I did not let myself down since I knew nothing about wireless telecom, Ukraine or finance at the time.  As I move towards software projects, I believe I am likely to repeat this feat.  Advanced skills in computing can be a passport to virtually any domain imaginable.  Energy projects, economics, robotics… once again I have no idea what projects I may be working on in 10 years.

But I’m eager to find out.

Read the post...