Devops work - Mapme
Infrastructure overview - Current state
Mapme services are deployed on AWS
AWS resources in use:
Elastic beanstalk
S3
Cloudfront
Non-AWS resources in use:
IBM/Compose hosted mongodb instance
Azure Pipeline/Release
CircleCI (for legacy services)
Deployments stages:
Production: prd environment only ("prd" mongodb instance)
Staging: dev and qa1 environments ("stg" mongodb instance)
Local: developer runs a local mongodb instance against his locally running apps
Each stage is associated with a dedicated:
mongodb instance
S3 bucket (for viewer app static assets)
S3 bucket (for user uploaded media: images/video/audio)
Elastic beanstalk services per stage:
Editor (mapme-story-editor)
Viewer Proxy (main-viewer-proxy)
Api server (main-api)
Worker instance (for image processing, email sending, CF invalidation tracking, geolocation operations).
Due to legacy reasons some API calls (authentication, icon search) are implemented on the worker instance. We plan to move them to the API server and have the worker instance not accessible via public domain
CI/CD
Build
Azure Pipeline is used for building artifacts
Azure Release flow is used for deploying the artifact to the appropriate environment (there might be several environments per stage)
Flow
The developer pushes a branch to github `mapme-main` repo
If the branch has a predetermined pattern an Azure build is triggered
The build creates artifacts for elastic beanstalk services:
Api server (nodejs)
Viewer Proxy (nodejs)
Note:
Worker and Editor apps are a part of Mapme's legacy deployment
Both are built via CircleCI
It is planned to migrate both into the new build/deploy flow, but it isn't a part of current TBD scope
Deploy
Deploy is triggered via Azure's ‘Release' feature
Pulumi is currently used as the infrastructure-as-code framework (we want this change):
Nodejs bundles are deployed to elastic beanstalk environments
When the elastic beanstalk environments become ready, a Cloudfront invalidation is issued
Viewer deployment issue
The standard way to build vue.js apps is to have the environment variables compiled into the application via .env files (per-environment file is implied)
Since each environment requires different variables, a separate build is required prior to deploy. The build is run against the associated .env file
As a result, either all possible viewer bundles are built in advance, or the relevant viewer gets built during a specific environment deployment
Either way it's a problem. Either the same app is built several times or the deploy is longer than necessary because the viewer app is built
Expected Work
The following items are expected to be completed:
Switch to exclusively to AWS-managed infrastructure-as-code deployment (replace Pulumi)
Azure build pipeline
Viewer App
Build Viewer app once during build pipeline (not during deploy)
The viewer app cannot have compiled-in environment variables
Environment variables are fetched as a separate http resource using an html tag in viewer's index.html (served by the viewer-proxy):
The script is required to be "infinitely" cached
Env vars script url should include the build ID string. This is so that a new version is requested after the vue app is deployed (this is when environment variables may change)
Performance:
The added script tag should not cause a noticeable delay in the app load time from the user's point of view
An ‘app ready' event is defined as the event in which the mapboxgl map signals the ‘load' event
When the environment resource url is locally cached by the browser, there should be no delay in app ready
If the environment resource url isn't locally cached, the ‘load' event may be delayed by 20ms in comparison to compiled-in environment variables
Optimize Build time by taking advantage of parallel Azure jobs (we have 10 parallel jobs since our build pipeline is public)
Deployment:
Overview
The essence of the deployment process is: update elastic beanstalk apps, replace S3 bucket content, and trigger Cloudfront invalidations. All that needs to happen as optimally and fast as possible. All operations which can execute in parallel - should
Details
Do not build Viewer during deploy
Minimize overall deploy time by optimizing AWS CDK flow
Migrate Pulumi deploy scripts to AWS CDK + Cloud Formation
Pulumi config secrets are required to be moved to AWS SSM (Systems Manager Agent)
Availability
Services availability shouldn't be impaired. Use immutable elastic beanstalk updates if necessary
There should be absolutely no downtime from users point of view (as is the case today)
About the recuiterMember since May 20, 2018 Vaniloran Elysa
from Arad, Romania