Deploy NextJs React website to AWS ECS container via bitbucket pipelines

Settle back for a big post on nextJs variables, bitbucket pipeleines and ECS containers..

Recently I was given the challenge to work out the best way to host a NextJs React website in AWS infrastructure.
We chose to attempt to dockerize the react website and host this on an AWS ECS cluster using fargate.

We had the following requirements;

have a CI / CD deployment pipeline
support multiple environments, dev / qa / uat and production
be highly scaleable
cost effective

We chose to;

host the website as a docker container;
on a AWS ECS cluster
using fargate
deploying via bitbucket pipelines, we wanted to
1. build the docker images once
2. push to ECR once, but
3. deploy the image to mutliple different environments

This was difficult as there was lots of documentation but nothing that tied it all together.This post will not go into the setup of the ECS cluster, but focus on the CI / CD pipeline configuration.

NextJs Docker image

Dockerizing the NextJs website is very simple.I followed instructions provided on the following website;

https://medium.com/swlh/dockerize-your-next-js-application-91ade32baa6

Assuming you have your own nextJs website..

Run site locally

inside you repo to test and run your site do the following

npm install next react react-dom
To run the app in quick dev mode
1. npm run dev
to deploy
1. npm run build
2. npm start

Create a docker image of the site

This assumes you have docker desktop running on your computer

Create a Dockerfile as follows in the root of your website

# Dockerfile

# base image
FROM node:alpine

# create & set working directory
RUN mkdir -p /usr/src
WORKDIR /usr/src

# copy source files
COPY . /usr/src

# install dependencies
RUN npm install

# start app
RUN npm run build
EXPOSE 3000
CMD npm run start

This specifies to run you site on port 3000

build an image of you site

docker build -t my-website .

Run the image

docker run --name my-website-container -dp 5000:3000 my-website

This should show your website running on port 5000.

Bitbucket pipelines deployment

This is where it gets interesting.
The following is a full deployment pipeline.

This is configured to do the following;

Be triggered off the development branch
perform a build and push to the docker image to ECR.
automatically deploy to the development environment
manually trigger deployment to qa

Pre-requisites;

AWS

ECS Cluster – (eg. my-web-cluser)
ECS Service (eg – svc-my-website)
ECR – elastic container repository – (eg – my-website-ecr)
Task Definition – td-my-website
s3 bucket – s3-bitbucket-deployment

Bitbucket

repository for your code
development branch
pipelines enabled
Deployments
1. test environments
  1. dev
2. staging environments
  1. qa
3. prod environments
  1. prod

In the deployment we need to setup the following “global variables”These variables are used in the pipeline file

s3Bucket
- the path to your s3 bucket
- s3://s3-bitbucket-deployment/my-website
AWS_REGION
- the region you want to run in
- i use ap-southeast-2, so if see that in the examples change to your zone
AWS_ACCESS_KEY_ID
- the aws access key
AWS_SECRET_KEY
- the aws secret key
ECR_ARN
- the arn to the ECR tat you have configured
- {aws-account-number}.dkr.ecr.ap-southeast-2.amazonaws.com/my-website
ECS_CLUSTER
- the name of your ECS cluster
- my-web-cluster
ECS_ROLE
- the role used when setting up the task definition
- arn:aws:iam::{aws-account-number}:role/ecsTaskExecutionRole
SECURTY-GROUP
- The default security group being used for the ECS Service
- Note I am setting this as a default NON-Production grooup
- If you have different security config per environment then setup on the specific deployment environment
SUBNET_1
- the subnet of the ECS Service
- I used the 2a zone
SUBNET_2
- the secondary subnet of the ECS Service
- I used the 2c zone

Environment specific varaibles

env
- the name of the environment
- e.g. dev / qa / prod etc
ECS_SERVICE
- the name of your ecs service
- svc-dev-my-website
ECS_TASK_DEF_NAME
- the name of the task definition
- td-dev-my-website

Interesting points to note on these parametersIf you name the AWS Keys as above, they automatically get picked up as the AWS configuration so when you run any AWS CLI command you do NOT need to pass in the extra parameters or setup anything as it picks up these keys.
You can name your environments as you please, this is just the configuration for my example.

S3 Bucket setup

I will write an entire section of the environment config for the NextJs website, suffice to say my solution is to reference the .env config from the s3 Bucket

I assume you have a .env.local file for your nextJs website, so upload the following versions of it to your s3 bucket

.env – base definition used for the build
one per environment; eg
- dev.env
- qa.env
- uat.env
- prod.env
Obviously each one would contain environment specific environment settings

Deployment Pipeline

This is a full deployment file for dev and qa. You should be able to easily add the deployment for production

I will run through each section of the deployment in detail

pipleline syntax

syntax to understand;

Its a yaml fie so indenting and spacing matters
echo just prints output into the logs, useful for debugging and working out what is going wrong with your script.
export VAR=”hello”
1. export sets a variable that can be used
echo $VAR
1. $VAR is how you reference your variable.

image: atlassian/default-image:2

definitions: 
    services: 
      docker: 
        memory: 2048  # docker running out of memory

pipelines: 
  default: 
    - step: 
        name: startup pipeline 
        script: 
          - echo "do nothing"

  branches: 
    development:   
      - step: 
          name: Build and publish to ECR 
          services: 
            - docker 
          caches: 
            - docker 
          artifacts: 
            - tag.txt 
          script: 
            - echo "in build step" 
            # setup env config 
            - echo "setup .env configuration" 
            - aws s3 cp $s3Bucket/.env . 
            # setup build tags 
            - IMAGE_NAME=$BITBUCKET_REPO_SLUG 
            - export DATESTAMP="$(date +%Y-%m-%d)" 
            - export TAG="$DATESTAMP.${BITBUCKET_BUILD_NUMBER}" 
            - export ECR_IMAGE="$ECR_ARN:$TAG" 
            - echo $TAG 
            - echo $ECR_IMAGE 
            # docker build 
            - docker build -t $IMAGE_NAME:$TAG . 
            - docker tag $IMAGE_NAME:$TAG $ECR_ARN 
            # push to ecr - pipe 
            - pipe: atlassian/aws-ecr-push-image:1.4.2 
              variables: 
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID 
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY 
                AWS_DEFAULT_REGION: $AWS_REGION 
                IMAGE_NAME: $IMAGE_NAME:$TAG 
                TAGS: $TAG 
            # output artifacts 
            - echo $TAG > tag.txt 
      - step: 
          name: dev deployment 
          deployment: dev  
          script:  
            - echo "in $env deploy step" 
            # Get variables from build step 
            - export TAG=$(cat ./tag.txt) 
            - echo $TAG 
            # setup aws 
            - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" 
            - unzip awscliv2.zip 
            - ./aws/install -b ~/bin/aws 
            - export PATH=~/bin/aws:$PATH 
            - aws --version 
            # setup version 
            - export ECR_IMAGE="$ECR_ARN:$TAG" 
            - echo $ECR_IMAGE 
            - echo $ECS_TASK_DEF_NAME 
            # update task definition 
            - rm -rf task-definition.json 
            - file_contents=$(< task-definition-template.json ) 
            - echo "${file_contents//ECS_TASK_DEF_NAME/$ECS_TASK_DEF_NAME}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//ECR_IMAGE/$ECR_IMAGE}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//ECS_SERVICE/$ECS_SERVICE}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//#env/$env}" > task-definition.json 
            - cat task-definition.json 
            # register task  
            - export TASK_VERSION=$((aws ecs register-task-definition --execution-role-arn $ECS_ROLE --cli-input-json file:///opt/atlassian/pipelines/agent/build/task-definition.json) | grep revision | cut -d ":" -f2 | cut -d "," -f1 | tr -d ' ') 
            - echo $TASK_VERSION 
            # update ecs service 
            - aws ecs update-service --cluster $ECS_CLUSTER --service $ECS_SERVICE --task-definition $ECS_TASK_DEF_NAME:$TASK_VERSION 
      - step: 
          name: qa deployment 
          deployment: qa 
          trigger: manual 
          script:  
            - echo "in $env deploy step" 
            # Get variables from build step 
            - export TAG=$(cat ./tag.txt) 
            - echo $TAG 
            # setup aws 
            - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" 
            - unzip awscliv2.zip 
            - ./aws/install -b ~/bin/aws 
            - export PATH=~/bin/aws:$PATH 
            - aws --version 
            # setup version 
            - export ECR_IMAGE="$ECR_ARN:$TAG" 
            - echo $ECR_IMAGE 
            - echo $ECS_TASK_DEF_NAME 
            # update task definition 
            - rm -rf task-definition.json 
            - file_contents=$(< task-definition-template.json ) 
            - echo "${file_contents//ECS_TASK_DEF_NAME/$ECS_TASK_DEF_NAME}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//ECR_IMAGE/$ECR_IMAGE}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//ECS_SERVICE/$ECS_SERVICE}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//#env/$env}" > task-definition.json 
            - cat task-definition.json 
            # register task  
            - export TASK_VERSION=$((aws ecs register-task-definition --execution-role-arn $ECS_ROLE --cli-input-json file:///opt/atlassian/pipelines/agent/build/task-definition.json) | grep revision | cut -d ":" -f2 | cut -d "," -f1 | tr -d ' ') 
            - echo $TASK_VERSION 
            # update ecs service 
            - aws ecs update-service --cluster $ECS_CLUSTER --service $ECS_SERVICE --task-definition $ECS_TASK_DEF_NAME:$TASK_VERSION

Image

image: atlassian/default-image:2

This is the default build image which is an Ubuntu based linux image.

Details of what is there by default is here https://hub.docker.com/r/atlassian/default-image/

Docker Definition – Memory

definitions: 
    services: 
      docker: 
        memory: 2048  # docker running out of memory

When building my docker image I was running out of memory.

This definition allows you to control the amount of memory that the docker build image has access too.
By setting to 2GB this allowed my build to complete successfully

default

default:
    - step:
        name: startup pipeline
        script:
          - echo "do nothing"

Can someone explain why I need this?
Seems that I have issues if I do not run a default step, it seems to allow the build environment to come up and “checkout” the git repo, but I do nothing in this step.

I have managed to remove this block, in later versions.

What should really be here is a test block to allow the build to run tests, and then fail if the tests fail.Currently on my play React website I have no tests, so no test block

Build Step

artifacts

This is a good trick.

There is no way to pass calculated variables between steps.
The easiest way to do this is by artifacts.
You define an artifact in the step e.g. tag.txt
You create this file during the step and then this artifact is available for subsequent steps.

 branches: 
    development:   
      - step: 
          name: Build and publish to ECR 
          services: 
            - docker 
          caches: 
            - docker 
          artifacts: 
            - tag.txt

script

I will walk you through each section

Env config.
- my build was crashing without a required variable in the .env file during the docker build
- I don’t want sensitive data committed to my git repo, so
- copy down the default .env file from the s3 bucket during the build step

            # setup env config 
            - echo "setup .env configuration" 
            - aws s3 cp $s3Bucket/.env .

Setup the Tag Name
- The tag name is super important for versioning
- I am using a date stamp and build number to identify my build image
- $ECR_IMAGE is the full path to the image and the ECR
- $TAG is the tag of the build
- I echo the variables so I can make sure it is working correctly

            # setup build tags 
            - IMAGE_NAME=$BITBUCKET_REPO_SLUG 
            - export DATESTAMP="$(date +%Y-%m-%d)" 
            - export TAG="$DATESTAMP.${BITBUCKET_BUILD_NUMBER}" 
            - export ECR_IMAGE="$ECR_ARN:$TAG" 
            - echo $TAG 
            - echo $ECR_IMAGE

Build the docker image
- Build the image
- Tag the build with our tag

            # docker build 
            - docker build -t $IMAGE_NAME:$TAG . 
            - docker tag $IMAGE_NAME:$TAG $ECR_ARN

Push to the ecr
- I am using the Atlassian pipe command to push

            # push to ecr - pipe
            - pipe: atlassian/aws-ecr-push-image:1.4.2
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                AWS_DEFAULT_REGION: $AWS_REGION
                IMAGE_NAME: $IMAGE_NAME:$TAG
                TAGS: $TAG

You can deploy without using the pipe command, it would look something like this;

            # push to ecr
            - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_IMAGE
            - docker push $ECR_IMAGE

Artifacts
- The tag used to create the build is output to tag.txt
- this artifact is now available for subsequent steps

            # output artifacts 
            - echo $TAG > tag.txt

So where are we now?

The build has been completed and the docker image has been pushed up to aws ecr.

Next step is to deploy your task definition

Task Definition

The task definition is the spec of your task, which is the container running your application.
For the ability to deploy to different environments we need to be able to dynamically configure the task definition for each specific environment we are deploying too.

To do this we create a task definition file that has been “Marked Up” with keywords.
These keywords are used in the pipeline to do a “find replace” to replace with dynamic values that configure the task definition for your environment.
This is how we are able to build a single image but deploy to multiple environments

task-definition-template.json

{
    "family": "ECS_TASK_DEF_NAME", 
    "networkMode": "awsvpc", 
    "containerDefinitions": [
        {
            "name": "ECS_SERVICE", 
            "image": "ECR_IMAGE", 
            "portMappings": [
                {
                    "containerPort": 80, 
                    "hostPort": 80, 
                    "protocol": "tcp"
                },
                {
                    "containerPort": 3000, 
                    "hostPort": 3000, 
                    "protocol": "tcp"
                }
            ],
            "environmentFiles": [
                {
                    "value": "arn:aws:s3:::s3-bitbucket-deployment/my-website/#env.env", 
                    "type": "s3"
                }
            ], 
              "logConfiguration": {
                  "logDriver": "awslogs",
                  "options": {
                      "awslogs-group": "/ecs/ECS_TASK_DEF_NAME",
                      "awslogs-region": "ap-southeast-2",
                      "awslogs-stream-prefix": "ecs"
                  }
              },
            "essential": true
        }
    ], 
    "requiresCompatibilities": [
        "FARGATE"
    ], 
    "cpu": "256", 
    "memory": "512"
  }

ECS_TASK_DEF_NAME – the name of your task definition
ECS_SERVICE- the name of the ECS service
ECR_IMAGE – the full path to the ECR_IMAGE that has been passed
#env – the environment e.g. dev / qa / prod
- needs to match the name of your environment file in the s3 bucket

Important items to note:

family
- This MUST match the name of you task definition
name
- this MUST, MUST, MUST (REALLY IMPORTANT), match the name of your service.
  - If the first time you create the task definition this value is set incorrectly then this task definition will NEVER WORK.
  - you cannot edit the task definition as it seems that there is something special about the first definition that revisions do not change it and it will never work
  - you cannot delete a task definition you can only mark all the definition versions as INACTIVE
environmentFiles
- This is how you can pass in “RUNTIME” environment variables for your react website
- The references to an s3Bucket allows storing a single file with your variables.
- file must be {name}.env to work.
- I will discuss environment files below.

Deployment Step

step deployment

deployment: dev
- This lines up with the deployment environments
- This is triggered automatically

- step: 
          name: dev deployment 
          deployment: dev

script

TAG
- Get Tag variable from artifact
- Read it from the artifact passed in from the previous step
- echo the tag for debugging

            # Get variables from build step 
            - export TAG=$(cat ./tag.txt) 
            - echo $TAG

AWS Setup
- we download the awscliv2.zip file
- extract and install it
- add ~bin/aws to the PATH so that you can reference aws without needing to provide a path
- aws –version
  - this line confirms that aws is installed and can be run successfully

            # setup aws  
            - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" 
            - unzip awscliv2.zip 
            - ./aws/install -b ~/bin/aws 
            - export PATH=~/bin/aws:$PATH 
            - aws --version

Setup Version
- Recreate the version and variables to match the image that has been uploaded to the ECR

            # setup version 
            - export ECR_IMAGE="$ECR_ARN:$TAG" 
            - echo $ECR_IMAGE 
            - echo $ECS_TASK_DEF_NAME

Update the task definition
- ensure that any previous task definition does not exist
- Load the template into memory and then find replace out to task-definition.json
- replace the 4 different tags into the task-definition
- cat task definition writes the contents of the task definition to the log so we confirm the find/replace has worked

            # update task definition 
            - rm -rf task-definition.json 
            - file_contents=$(< task-definition-template.json ) 
            - echo "${file_contents//ECS_TASK_DEF_NAME/$ECS_TASK_DEF_NAME}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//ECR_IMAGE/$ECR_IMAGE}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//ECS_SERVICE/$ECS_SERVICE}" > task-definition.json 
            - file_contents=$(< task-definition.json ) 
            - echo "${file_contents//#env/$env}" > task-definition.json 
            - cat task-definition.json

Register the task definition
- this creates a new version of the task definition
- We get that result into a $TASK_VERSION variable

            # register task  
            - export TASK_VERSION=$((aws ecs register-task-definition --execution-role-arn $ECS_ROLE --cli-input-json file:///opt/atlassian/pipelines/agent/build/task-definition.json) | grep revision | cut -d ":" -f2 | cut -d "," -f1 | tr -d ' ') 
            - echo $TASK_VERSION

Update the ECS Service
- connects the service to the new version of the task definition

            # update ecs service 
            - aws ecs update-service --cluster $ECS_CLUSTER --service $ECS_SERVICE --task-definition $ECS_TASK_DEF_NAME:$TASK_VERSION

That is it, if you look at your service in the cluster you will see a new instance being provisioned and the new Task definition version. When the load balancer works out the new instance is healthy it, the old instance will be drained and then disposed of.

NextJs environment files and Docker Images

One of the key elements to understand about how the above deployment actually works is to understand how environment files work in NextJs and Docker.
When a .env is created in a NextJs React website you add content like

MY_VAR=build
You can access this variable with;

process.env.MY_VAR

Docker build

When a docker build is run the .env file is COMPILED into the docker image AT BUILD TIME!

But in our deployment we want to build a SINGLE image but deploy it to multiple environments.

Docker run passing in an environment file

Create a run.env file and put

MY_VAR=runtime

Run the following command

docker run --name my-website-container --env-file run.env -dp 5003:3000 my-website

This — env-file will pass in the contents of the run.env file and if you inspect the running image you will see that the MY_VAR will reflect the updated value runtime.

THE PROBLEM

However, try and access process.env.MY_VAR and it will still say build

So what is the problem?

The system needs to be adjusted to access the environment variables at runtime.
THE SOLUTION

https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration

You need to access the environment variables at runtime.

To do that you need to edit your next.config.js file

module.exports = { 
  serverRuntimeConfig: { 
    myServerVar: process.env.MY_VAR, // Pass through env variables but only available on the server side
  }, 
  publicRuntimeConfig: { 
    // Will be available on both server and client 
    myPublicVar: process.env.MY_VAR, // pass through and available both server and client side
  }, 
}

On a home page you would then be able to do the following;

import getConfig from "next/config"; 
const { publicRuntimeConfig } = getConfig();

export default function Home() { 
  return ( 
    <div className={styles.container}> 
      <Head> 
        <title>Create Next App</title> 
        <meta name="description" content="Generated by create next app" /> 
        <link rel="icon" href="/favicon.ico" /> 
      </Head> 
      <main className={styles.main}> 
        <h1 className={styles.title}> 
          Welcome to <a href="https://nextjs.org">Next.js!</a> 
        </h1> 
        <h3>MY_VAR</h3> 
        <div>{publicRuntimeConfig.myPublicVar}<div> 
          
      </main> 
      <footer className={styles.footer}> 
        <a 
          href="https://vercel.com?utm_source=create-next-app&utm_medium=default-template&utm_campaign=create-next-app" 
          target="_blank" 
          rel="noopener noreferrer" 
        > 
          Powered by{' '} 
          <span className={styles.logo}> 
            <Image src="/vercel.svg" alt="Vercel Logo" width={72} height={16} /> 
          </span> 
        </a> 
      </footer> 
    </div> 
  ) 
} 
export const getServerSideProps = () => { 
  return {props: {}}; 
}

If you have this working in your local docker images with the –env-file passing in the environment file..

Then when you create your task definition with;

            "environmentFiles": [ 
                { 
                    "value": "arn:aws:s3:::s3-bitbucket-deployment/my-website/#env.env",  
                    "type": "s3" 
                } 
            ],

This is exactly what AWS does it essentially runs the docker container with this -env-file variable but linking to your file in the S3 bucket.

Final Thoughts

Triggers

The QA deployment is a manual deployment as specified by the manual trigger

      - step:  
          name: qa deployment  
          deployment: qa  
          trigger: manual

Versioning

Using the task definition I can pass in individual values at runtime as well

{  
    "family": "",  
    "containerDefinitions": [  
        {  
            "name": "",  
            "image": "",  
            ...  
            "environment": [  
                {  
                    "name": "website-version",  
                    "value": "SITE_VERSION"  
                }  
            ],  
            ...  
        }  
    ],  
    ...  
}

I am going to pass through the $TAG into a variable SITE_VERSION.
Building on 1-Jan-2022 and pipeline build # 43 would create a TAG=”2020-01-01.43″ and this would be passed into the environment settings.

Branches

The build file as presented would promote a commit into development to the dev environment which could be promoted to QA and all the way to production, (COPY / PASTE the QA deployment and call it Prod).

I think this may cause problems. I think it may be better to leave the dev environment triggering off the development branch.QA should be triggered off the master branch, so this way the following would be the deployment approach;

Developers Pull Requests are merged into development and deployed to the dev environment
when we are confident with the build we can do a PR into master
master triggers the QA build.

By separating dev and QA environments by connecting QA to the master branch, you have the ability that if a change gets into development that breaks the build, you could create a release branch of the validated features only and then merge those features through into the master branch which then is able to be tested in QA.

Conclusion

Thats it. Hope these detailed instructions help. Like everything in software development there is no right or wrong way to do things, but this is my solution to handle the ability to build a docker image once but deploy it multiple times to multiple environments.

The beauty of this, is that the code you tested in development is the same that can end up in production as you are using the EXACT SAME ECR IMAGE, it is the same code. build once deploy multiple times.

M	T	W	T	F	S	S
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30