As I continue my capstone project, Cloud-Based Algorithmic Trading Strategies for Individual Investors, I’ve made significant progress in areas like data acquisition, data transformation, and team collaboration. Each milestone has taught me valuable lessons about software development, investing, and the importance of effective teamwork.
Data Acquisition
One of the key components of our platform is Data Acquisition, and it has been both challenging and rewarding to integrate multiple sources for stock market data. I decided to use Alpaca’s Market Data API for its seamless access to both real-time and historical stock data, and Yahoo Finance as a secondary source to fill in any data gaps and provide additional flexibility for historical analysis. Both of these options are free, in the future, I hope to use more efficient and reliable options.
In developing the live market data collection, I found that Alpaca’s API offers REST and Websocket methods for gathering market data. I had to make the decision to go with the REST option as it better fit our algorithm’s overall strategy and does not have the limit of 30 symbols the Websocket method has.
For gathering historical market data, primarily used in backtesting and displaying stock market data in the frontend, I used Alpaca and YahooFinance. Both have their strengths and weaknesses in certain areas. I had to develop my services so that I stayed under the API call limit but still offered support for as many stock symbols as needed. Additionally, I had to integrate the data of both technologies, although they were initially formatted differently.
The process of working with multiple APIs has taught me the importance of balancing data accuracy, speed, and cost. Each API has its own limitations and quirks, and managing these effectively has been essential to maintain data reliability while staying within API rate limits.
Data Transformation
Once the data is acquired, it flows into the Data Transformation module, where it is processed, cleaned, and prepared for analysis. Here, Python’s pandas library has been my go-to tool for handling large datasets and transforming raw market data into actionable insights.
I’ve gained a deeper understanding of how to:
- Clean and normalize data to ensure consistent formats across different APIs.
- Handle missing or incomplete data, an inevitable challenge when dealing with financial markets.
- Optimize data pipelines to reduce processing time and ensure real-time data is ready for the Trade Signal Generation module.
This experience has significantly improved my ability to work with large datasets in Python and has stressed the importance of clean, well-structured data in algorithmic trading.
Team Collaboration
One of the most challenging aspects of working on this project has been team collaboration. I feel this is my teams biggest struggle currently, and something I hope to continue to improve as we progress through this project.
Looking Ahead
As we move forward, I’m excited to see how our platform evolves. The next steps will involve refining our algorithms, enhancing the user interface, and preparing for live testing. I’m particularly looking forward to integrating our Data Transformation module with the Trade Signal Generation system, which will mark a significant milestone in the project.
