In my internship, I’m starting a project in which the goal is to keep a record of files that store variables used in a system. Currently, the place where these files are stored does not maintain any type of changelog. User changes to these files are not recorded. My project develops a process to monitor and record file changes by utilizing GitHub for version control, Jenkins Pipeline for automation, and Snap Logic to build the files. Since a lot of this is new for me I thought it would be great to run through these technologies and how they could be used to maintain an automated system to monitor, record, and implement file changes.
Most of us are familiar with using Git and GitHub for version control. Oftentimes remote repositories contain a ‘main’ branch that contains the reviewed and approved code. Source code changes made by users are housed in branches that are merged with the main branch upon approval. Depending on how the repository is set up, a user submitting changes must submit a pull request (PR). Within the pull request various workflow actions, as well as a code review(s), take place. Workflow actions could check for formatting, linting, etc…
Jenkins is a CI/CD automation tool for building, testing, delivering, or deploying software. In Jenkins, a Pipeline can be used to get software or code from a version control system like GitHub to an end-user. A Pipeline is built using a Jenkinsfile which can be written using declarative or scripted syntax. Jenkins declarative syntax is written using Groovy, an object-oriented language for the Java platform. Jenkins can be integrated with GitHub so that a Jenkins Pipeline is triggered when a user merges a PR to a particular branch.
SnapLogic is an Integration Platform as a Service (IPaaS) tool to connect data sources and automate workflows, like ETL (extract, transform, load) flows, through the use of pipelines. SnapLogic can connect and retrieve data from a variety of sources, transform the data, then write files or load data to end-user systems.
With regards to my project, the above-mentioned files will be housed in a GitHub repository so that a record of changes to files can be maintained. When a user wants to make changes to the files, they will submit a PR to merge their changes to main. Once the PR is approved, a Jenkins Pipeline will determine what files are newly added, modified, and/or deleted. These file changes will be mapped to a Groovy object and ultimately output as a JSON string to SnapLogic using the HTTP post method. SnapLogic pipelines will verify the data, then write or delete the files on the end-user.
Just a snapshot of some new technologies I’m working with and how they aid in automation.
