Recently I was involved in setting up a new Node.js-based service on AWS. There are multiple compute options to choose from on AWS. The one I’d like to discuss here is the Fargate launch type on the AWS Elastic Container Service (ECS).
I won’t get into too much details of what Fargate is (there is the AWS documentation for that), but basically it allows you to run Docker containers without having to manage EC2 instances underneath. That said it is not bound to execution time constraints like AWS Lambda functions, which may also be a suitable compute option depending on your requirements. For the decision process, which of these to choose, I’d like to recommend this very good episode of the Cloudonaut Podcast.
So why this blog post? Well, there are a few pitfalls to run into when first setting up a Fargate based service, which I’d like to document so that I don’t forget about them, when coming back to it later 😉.
Load Balancer
The AWS Application Load Balancer (ALB) (or another API gateway or service middleware) is required to have a static DNS name (service URL) for the service running behind it, since Fargate re-assigns dynamic IPs to services. Luckily the ALB is tightly integrated with Fargate via the load balancer target group and listener mechanism. A listener is registered with ECS service and ECS manages routing from the load balancer to the container under the hood. If you want to run multiple services on Fargate which belong to the same business process, you can run a single load balancer and register multiple target groups and hence listeners to different ports. This spares you the base-cost coming with a each load balancer you deploy.
Pipeline
This morning I played around with AWS CodeBuild, CodeDeploy and hence CodePipeline and tried to get a simple node.js service built and deployed on Fargate. The nice thing I noticed is, that AWS CodeDeploy considers the time a rolling service update with a newly built container and the respective health checks take. This means the new container will be deployed, a new task for the ECS service is launched, health checks by the load balancer are performed and only after the new task is deemed healthy, the old task is being drained and killed. CodeDeploy will only succeed after all these steps succeed. This is something you would need to build yourself, when choosing another CI provier.
Health Checks
The health check URL endpoint configured for the load balancer target group to be connected to the service, should be a fairly simple endpoint with a simple response (e.g. “ok”, of course with HTTP code `200`). This will make health checks faster and more robust. When first dealing with Fargate tasks behind a load balancer, it was odd to me to see new tasks being spun up and immediately killed over and over again when something was broken inside the container or the configuration. This is due to health checks deciding to kill unhealthy tasks and then the service trying to satisfy its `DesiredCount` again. Over and over.
Configuration
Usually you would want to make your service configurable depending on the purpose it is deployed for (testing, production, etc.). Environment variables can be easily defined as part of the ECS task definition, which is used to describe how a task for a service should run. Important here is to distinguish between environment variables and secrets. The latter will not be visible in the AWS console and or log files, which is of course the desired behavior for secrets or access tokens. These parameters can be easily stored in AWS Systems Manager (SSM) and used in the taskdefinition (using `ValueFrom`), since ECS and SSM are tightly integrated.
Troubleshooting
You cannot access Fargate containers via SSH, like you can on EC2. However, thinking about it, that should not be required if you put the effort in proper alerts and logging. Hence this is just a reminder to log what you can (within regulations of course) and rely on robust deployment and roll back mechanisms in case of an issue.
Cheers.