Difficulties of Cloud

It seemed easy enough of a task to transform a desktop application to a serverless architecture on cloud at first. However, there are always issues and challenges with dealing with this and here are the challenges we faced as well as insights into dealing with challenges:

1. How many microservices do we need? It took us a while and still figuring out, how many AWS lambda functions we require to run microservices for our application. There is no clear answer. The trick is however figuring out what is the best and efficient architecture to minimize calls to APIs or between lambda functions in order to minimize latency. This means we have to split up some portions of the code, combine some portions of the code and remove some portions of the code.
2. Using the /tmp folder on lambda function. Like a typical System IO, there is a tmp folder where you can temporarily store or cache data within the function and it will disappear once the function completes its run. This was an extremely important feature for us because we can retain a lot of the filepath used for the desktop application and use it for this serverless cloud architecture.
3. Understanding the requirements of the application and decide what data storage do you need. This sounds simple, but it was not as straight forward as we thought it would be. Initially, we would have thought to use the DynamoDB, which is AWS’s NoSQL data storage to store all the data. However, this was not the case here, as we discovered the original desktop code caches data and streams from a xml or csv file. A lot of this was also temporary data, which needs to be removed over time to ensure there is enough storage space and minimize data cost. It did not make sense to use DynamoDB because of the limitations DynamoDB has with number of rows, especially when you are dealing with large sets of data. You can only transfer data by rows of 200, but you have hundreds and thousands of data entries. At the end, we opted to S3 bucket, which is AWS’s file storage utility and we were able to write code that accesses S3 and do all sorts of CRUD operations with it.
4. Figure out how to chain your lambda functions / microservices together. In most cases, you will need to have some kind of chaining of functions in order to use results of a function as parameter for another function. You would typically think that you can simply call the other function as an API, but that is often not the best way as it will increase latency due to the extra communications needed between the functions. While all your microservices are on the same platform, you should take advantage of the existing utility to minimize latency between functions. It is best if you figure out ahead of time which ways to do it is the best for your application as AWS for example offers multiple options to achieve it such as step functions, asynchronous invocation, etc.

So, that was some quick tips and experience we had on our project. Hope you enjoyed it and learned something from it.