Having bloviated about my capstone project the last two blogs, I’ll use this one to talk about the work I’m doing at my new job at Amazon.
After quite a bit of job hopping, I’ve landed at Amazon on a team responsible for the company’s benefits administration. That offer came just in the nick of time, too, as the contracting agency through which I’d been working for Microsoft was quite suddenly cut loose! I’m glad I started looking for new gigs once I caught the impression that the “contract to hire” arrangement wasn’t nearly as likely to end in a hire as I’d been led to believe.
It’s all worked out for the better, though, as Amazon is proving to be a much better job in every respect. I’m working on a great team and get along great with my manager. And now, after the typical first few weeks of onboarding, mundane tasks, and learning the ropes, I’ve been able to sink my teeth into some “real” code.
I say “real” because what I’m really working on is a build and deployment pipeline for a burgeoning service our team is creating – a new application for viewing and electing all of the benefits for which a user (an Amazon employee) is eligible. It’s not the kind of project I’d have leapt at in prior jobs, but Amazon has a particular technology that makes it a lot more interesting – their Cloud Development Kit, or CDK.
The CDK is basically an infrastructure-as-code construct, and a large one, at that. Virtually everything that can be deployed through the AWS console (the massive GUI application with a subcomponent for every AWS service) or comma d line interface can be programatically deployed through a CDK build. There are a few different ways to write and organize CDK code, but the pattern I’ve most commonly seen at Amazon is to use “stacks” – sets of related resources deployed in an organized fashion via AWS Cloud Formation.
CDK code is most commonly used to deploy resources which are used or invoked at a later time- APIs, database instances and tables, and the like. My current project, on the other hand, adds a bit of a twist to the typical deployment use case. Without getting too into the weeds (or saying anything proprietary), I need to use a CDK stack to not only deploy a DynamoDB database and create a table, but to also to populate that database with entries defined in an adjacent data package in the pipeline. Typically a CDK code change will trigger a pipeline run and, subsequently, an update or recreation of the altered resource. In this case, a change to that source-controlled data package also needs to trigger the pipeline to run, the CDK code to execute, and for the resources which use that data to be invoked. A one-time invocation upon creation isn’t enough.
Fortunately, I’m on the cusp of finishing my implementation. I first tried to see whether any existing AWS services would do what I need out-of-the-box. If that were the case, I’d probably be able to invoke said service(s) from CDK code, pass them a few basic parameters – the resource identifier for my DDB table, for instance – and be done. It turns out my use case isn’t common enough for that to be the case, though – at least not in a CDK context.
Knowing I’d need to implement a custom solution, piecing together other services and constructs, I broke my needs down into three tasks. First, call a certain Gateway API endpoint to validate all of the data pulled in from the data package – the data to be uploaded to our DynamoDB table. Second, query all existing entries in the table and delete the ones that are no longer in the source-controlled config. Third, insert and / or update all of the entries which are in the config.
All of these tasks can be accomplished with Lambda functions – basically code running in a serverless cloud environment. The AWS SDK provides classes for calling Gateway API endpoints and for making DynamoDB calls, and those are the SDKs I put into my Lambda “handlers” – the actual code the Lambdas run. That code can be written in a variety of languages, but since I’m already using Typescript for my CDK, I opted for consistency.
After quite a bit of tinkering, I’ve gotten the Lambdas deployed and working. In isolation, they all perform their respective tasks. They’re also all encapsulated in a construct called a Custom Resource, whose instantiation basically ensures that those Lambdas will be invoked every time the Custom Resource’s dependencies (in my case, the config data) is changed.
Still, I haven’t quite figured out how to get them to run sequentially. The validation needs to happen first, followed by obsolete entry deletion, and finally upsertion of everything else. It looks like the best way to accomplish this is to ditch the current Custom Resources (I’ll still another one for a different purpose) and instead chain the Lambdas together in another AWS construct – the step function. Each lambda invocation is a task, or state, in a state machine which needs to be invoked at deployment time every time the config data is changed.
Custom resources can’t be backed by step functions instead of Lambdas, though, so I’ve had to leverage a solution I’ve see described in a few articles, as well as in my perusal of other employees’ code. I’ll create a new Custom Resource backed by yet another Lambda function whose sole job is to invoke my step function – which will perform the business logic I really need. That step function will conclude with one more additional Lambda function, whose job is to output the result of the step function, which in this case is just a success or failure.
This is the code I’m going to be working on right as I finish this post – wish me luck!
