Continuous Integration and Delivery on AWS
About The Lead Group
The Lead Group (TLG) is a premier online marketing company specializing in data and list management, search marketing, and affiliate marketing for the advertising and publishing industries. Their user-friendly, state-of-the-art tracking and reporting systems provide targeted lead generation with real-time distribution capabilities.
TLG’s proprietary search optimization software makes Search Engine Marketing (SEM) a streamlined, less expensive, and more viable option to produce the best quality leads in the industry. Through their proprietary email delivery platform, their creative emails attain unrivaled deliverability rates, resulting in high consumer responses for all major domains. Their affiliate marketing team generates repeat VIP traffic and increases customer retention rates with innovative features such as exit solutions, fail path programs, and landing page optimization.
The primary challenges of TLG are higher AWS cost with each account maintaining a source code (Git) server and multiple AWS Beanstalk environments. The time to market is slow with the absence of continuous integration and deployment mechanism. TLG has a pressing need to support multiple deployments per day across all the websites in an automated fashion. The client websites that are spread across seven AWS accounts. The core challenges include:
- Server hosted in non-VPC classic AWS environment
- Missing unified source code management system
- Custom Webhooks used for deployment to Amazon Elastic Beanstalk that are not manageable, maintainable and consistent
- No Continuous Integration and Continuous Delivery pipeline mechanism which hinders time to market
- Managing and provisioning computing infrastructure on AWS is a manual process, and not an automated, scripted and versioned ‘Infrastructure as Code’ (IaC)
CT introduced DevOps tools and methodologies to implement a continuous integration and deployment solution. This solution enables automated and streamlined process to promote code and deploy changes from developer’s desktop to QA, and render production in a fast, reliable, and consistent manner.
To achieve this, CT proposed the migration of non-VPC to a secure VPC environment across all the AWS accounts. Bitbucket is used as the unified Source Code Management system, with all the Git repositories from different Gitlab accounts migrated to a single team account on Bitbucket.
Jenkins is used as the continuous integration and delivery tool to deploy the changes in test and production environments in an automated, standardized way across all the websites. Ansible is used as the automation tool for configuration management. In order to deliver a cost-effective solution, the Beanstalk environments are removed, and a set of auto-scaling instances or web servers are used to host all the websites.
TLG Websites hosted in AWS with EFS, auto-scaling and a secure VPC
Increased Agility, Increased Quality, and Reduced Outages: CT’s introduction of DevOps tools and methodologies for continuous integration and deployment enables instant, automated deployment across environments, constant feedback leading to increase in end user satisfaction and early detection of bugs preventing outages and downtime. Multiple deployments per day are seamlessly supported in an automated way across all the websites with email notifications on success or failure. This enables a faster time-to-market.
Cost Saving: CT’s initiatives to consolidate the source code into a single cloud hosted repository enabled shutting down multiple Git servers in different AWS accounts. The websites are hosted in a single server which allowed shutting down all the Beanstalk environments. These initiatives led to a significant cost saving for the client.
Enhanced Security and Control: CT’s initiatives to migrate the classic non-VPC AWS environment to VPC enhanced the security and controlled outbound and inbound traffic with security groups, and network Access Control Lists (ACLs). The instances have static private IP addresses that persist with the ability to assign multiple private IP addresses. Network interfaces are defined and one or more of them can be attached to the instances.
Scalability: Scalable and highly available network file storage system with Amazon EFS for all the websites. Ansible introduces IT automation with configuration management and application deployment on the web servers. This allows a consistent, repeatable, reliable and automated process to launch and update different websites with Ansible playbooks. Sending updates ensures zero-downtime deployments.
Continuous Integration: Continuous Integration and Delivery Pipeline with Jenkins provided a standardized, consistent, streamlined process to deploy and promote code automatically across QA and production environments for all the websites.
Continuous Integration and Delivery on AWS
On analyzing the challenges of TLG, CT developed DevOps tools and methodologies and introduced them to the client along with deployment setup. CT utilized various services such as Cloud Strategy Workshop, Architecture & Design, Application Development, Application Migration & Transformation, Application Testing, Application Hosting, and Application Monitoring to achieve swift time-to-market with continuous integration and delivery on AWS
The core challenges of TLG included higher AWS costs, slower time to market, with missing continuous integration and deployment mechanism, and demanding need support multiple deployments each day across all websites.
CT initiated migration of non-VPC to a secure VPC environment across all the Git repositories from different Gitlab accounts migrated to a single team account on Bitbucket. The project was executed from October 2016 to March 2017.
- Enhanced security and control on outbound and inbound traffic
- Unified Source Code Management system hosted on Bitbucket
- Scalable and highly available network file storage system with Amazon EFS for all the websites
- Continuous Integration and Delivery Pipeline with Jenkins
- Faster time to market, cost savings, increased quality and reduced outages, increased communication, collaboration and user feedback