Containerize a python web application

Overview

containerize a python web application

introduction

agenda

  • you probably heard about docker, Kubernetes, and container.
  • create a web application using Django.
  • explore deployment methods. and development environment challenges.
  • docker ( what why how ?)
  • Dockerizing our lovely Django app.
  • what's next

.

Django web framework

what's Django

  • it's a free and open-source python web frameworks, that uses MTV - Model Template View ( derived from MVC )
  • what is mvc
    • you can get up and running quickly.
    • very good for MVPs ( minimum viable product )
    • scalable.
    • who uses Django ?
      • pintrest, instagram, coursera
      • me ! checkout cgine.app

NOTE : im using Linux, if you are using windows, installation process can be a little different, but if you are using mac they should be similar.

first of all, it's a good practice to create your own virtual environment.

$ virtualenv venv

activate venv

$ source venv/bin/activate

install django

$ pip install django

start a new project using django-admin, and a new application

$ django-admin startproject GDSC
$ python manage.py startapp main

cd to the GDSC file & migrate.

$ cd GDSC              
$ python manage.py migrate

run our python application

$ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
October 19, 2021 - 07:51:41
Django version 3.2.8, using settings 'GDSC.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

you can navigate to http://127.0.0.1:8000/ and confirm that your app is running.

now we need to create a requirements.txt , which is a file that lists all of the modules needed for the Django project to work. and write this in it ( it should at the same direcotry as manage.py)

 Django==3.2.8

so now we have up and running django project, it's very simple, but will we be containerize it.

explore deployment methods.

so now if we want to deploy our django app, we would get a linux server set up nginx and gunicorn ( HTTP servers), and pray to good that everything work, and to make that app scaleble and able to handle huge traffic, we would set up another server ( load balancing).

and during development, you would spend hours trying to setup your development environment, from installing dependencies to setting up a databse. postgres & redis ( DB & caching) .

docker.

coming back to the deployment example, the concept of Hardware server is vanishing thanks to the cloud, we now have some sort-of a software server called containers.

the concept of a server could be removed from the constraints of hardware and instead became, essentially, a piece of software. These software-based servers are called containers, and they're a hybrid mix of the Linux OS they're running on plus a hyper-localized runtime environment (the contents of the container). [open source] (https://opensource.com/resources/what-docker)

containers consist of three main elements: 1- builders : that build the container i.e dockerfile 2- engine : and application that runs the container i.e docker command 3-Orchestration : technology that manages many container i.e Kubernetes.

so using container will help us deploy the application and its configuration using one image, that can be reused anywhere, this is also helpful for developers because all team members will have consistent environments and dependencies.

now we will use docker to deploy and run our django application. first you will need docker ( obviously ) https://docs.docker.com/get-docker/ if you are using

  • windows : you can just download the desktop version, ( make sure to add it to PATH variable )
    • confirm installtion by typing docker in cmd.
  • mac : same as windows, get the desktop version.
  • linux : ...............

** NOTE: ** we will use docker CLI, you can use GUI but it's always better to learn using the cli first.

now we need to create a builder which is the docker file. docker file is : a text document that contains all the commands a user could call on the command line to assemble an image. its similar to setting up an environment on a Linux server.

docker file should be in the same directory as manage . py

the basic syntax of a docker file is

INSTRUCTION  arguments 
# the offical apline base image
FROM python:3.9.6-alpine

# set up a work directory ( for the app )
WORKDIR /usr/src/GDSC

#set up enviroment variables.
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# copy project
COPY . .

what is alpine? and why its a linux distribution that is small and fast, and we are using an image that is based on it. it uses less disk space and faster to deploy. refer to this talk given by micheal herman about best practices for docker python

now we create the docker compose file ( docker-compose.yml) what is docker-compose ? a tool for defining and running multi-container Docker applications and define the services that make up your app

docker-compose.yml should be in the root directory ( where venv is )

version: '3.8'

services:
  web:
    build: ./GDSC
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./GDSC/:/usr/src/GDSC/
    ports:
      - 8000:8000
    

to recap.

  • Dockerfile is a simple text file that contains the commands a user could call to assemble an image
  • Docker Compose is a tool for defining and running multi-container Docker applications (
  • what is the relation between them ?
    • docker compose defines the services that make up you app and uses the docker file.
  • what do you mean by services?
    • part of your application ( database, API, etc) and have to be in a single container.
    • for example web service which we used above, it uses an image that’s built from the Dockerfile in the current directory. It then binds the container and the host machine to the exposed port, 8000. so currently we have only one service. see docker compose docs

so your files should look like this

.
├── docker-compose.yml
├── GDSC
│   ├── db.sqlite3
│   ├── Dockerfile
│   ├── GDSC
│   │   ├── asgi.py
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-39.pyc
│   │   │   ├── settings.cpython-39.pyc
│   │   │   ├── urls.cpython-39.pyc
│   │   │   └── wsgi.cpython-39.pyc
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── main
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── __init__.py
│   │   ├── migrations
│   │   │   └── __init__.py
│   │   ├── models.py
│   │   ├── tests.py
│   │   └── views.py
│   ├── manage.py
│   └── requirements.txt

and then we build the image and run it. if you are running docker desktop you already have docker compose installed if not refer to this page https://docs.docker.com/compose/install/

$ docker-compose build
$ docker-compose up -d

go check http://localhost:8000/admin/ note that we can see all the static files served.

until now we have been working with Django development server, but what if we want to deploy our application using a production grade server ?

configuring for production

  • we can use the fast,easy,cheap but manual and not scalable way.
  • or we can use the more complex, reliable, scalable way.

for the sake of this tutorial we are going to use the first way.

adding Gunicorn

Gunicorn, a production-grade WSGI ( Web Server Gateway Interfac) server, that is reliable and fast. so we will need to add to our requirements.txt file

Django==3.2.8
gunicorn==20.1.0

so now we will create a new docker-compose file for the production environment, note that we can still use the old one for development. we will call it docker-compose.prod.yml for more information about why we added .prod refer to https://docs.docker.com/compose/extends/

version: '3.8'

services:
  web:
    build: ./GDSC
    command: gunicorn GDSC.wsgi:application --bind 0.0.0.0:8000
    ports:
      - 8000:8000
    

we changed the command from running default django server, to running gunicorn server.

if our development container are still running we need to spin them down before running this one

$ docker-compose down -v

and build & run our new docker compose

$ sudo docker-compose -f docker-compose.prod.yml up -d --build

navigate to http://localhost:8000/ you will see everything working but these is something odd. static files are not served (admin page)!! to do that we will need NGINX to act as a reverse proxy for Gunicorn to handle client requests as well as serve up static files.

what is a proxy server ? A proxy server is a go‑between or intermediary server that forwards requests for content from multiple clients to different servers across the Internet.

adding nginx

we are going to need a new service so where we should add that ? to the docker-compose.prod.yml right ? remember the job of docker compose is to define all of the app service. so it should look like this after adding nginx.

version: '3.8'

services:
  web:
    build: ./GDSC
    command: gunicorn GDSC.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - ./GDSC/:/usr/src/GDSC/
    ports:
      - 8000:8000
    
  nginx:
    build: ./nginx
    ports:
      - 1337:80
    depends_on:
        - web

now we need to add nginx configuration files in the root folder ( where docker-compose is ) so its should look like this

.
├── docker-compose.prod.yml
├── docker-compose.yml
├── .env.dev
├── .env.prod
├── GDSC
│   ├── db.sqlite3
│   ├── Dockerfile
│   ├── GDSC
│   │   ├── asgi.py
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-39.pyc
│   │   │   ├── settings.cpython-39.pyc
│   │   │   ├── urls.cpython-39.pyc
│   │   │   └── wsgi.cpython-39.pyc
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── main
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── __init__.py
│   │   ├── migrations
│   │   │   └── __init__.py
│   │   ├── models.py
│   │   ├── tests.py
│   │   └── views.py
│   ├── manage.py
│   └── requirements.txt
├── nginx
│   ├── Dockerfile
│   └── nginx.conf

in nginx/Dockerfile


FROM nginx:1.21-alpine

RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d

and in nginx.conf

upstream GDSC {
    server web:8000;
}
server {
    listen 80;
    location / {
        proxy_pass http://GDSC;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }
    
}

now we replace ports with expose in the docker-compose.prod.yml file Now, port 8000 is only exposed internally, to other Docker services

version: '3.8'

services:
  web:
    build:
      context: ./GDSC
      dockerfile: Dockerfile.prod
    command: gunicorn GDSC.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - static_volume:/home/GDSC/web/staticfiles
    expose:
      - 8000
   
  nginx:
    build: ./nginx
    volumes:
      - static_volume:/home/GDSC/web/staticfiles
    ports:
      - 1337:80
    depends_on:
        - web
volumes:
  static_volume:

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers.

now we will create Dockerfile.prod ( in the same directory as Dockerfile)

###########
# BUILDER #
###########

# pull official base image
FROM python:3.9.6-alpine as builder

# set work directory
WORKDIR /usr/src/GDSC

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1





# install dependencies
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/GDSC/wheels -r requirements.txt


#########
# FINAL #
#########

# pull official base image
FROM python:3.9.6-alpine

# create directory for the app user
RUN mkdir -p /home/GDSC

# create the app user
RUN addgroup -S GDSC && adduser -S GDSC -G GDSC

# create the appropriate directories
ENV HOME=/home/GDSC
ENV APP_HOME=/home/GDSC/web
RUN mkdir $APP_HOME
RUN mkdir $APP_HOME/staticfiles
WORKDIR $APP_HOME

# install dependencies
RUN apk update && apk add libpq
COPY --from=builder /usr/src/GDSC/wheels /wheels
COPY --from=builder /usr/src/GDSC/requirements.txt .
RUN pip install --no-cache /wheels/*


# copy project
COPY . $APP_HOME

# chown all the files to the app user
RUN chown -R GDSC:GDSC $APP_HOME

# change to the app user
USER GDSC

quick test to insure everything is running

$ docker-compose -f docker-compose.prod.yml down -v
$ docker-compose -f docker-compose.prod.yml up -d --build

now spin down the container one more time. because we need to add somethings .

$ docker-compose -f docker-compose.prod.yml down -v

to make sure that static files can be served in development environment add this to settings.py

STATIC_ROOT = BASE_DIR / "staticfiles"

lastly we need to add static files location to nginx.conf

upstream GDSC {
    server web:8000;
}
server {
    listen 80;
    location / {
        proxy_pass http://GDSC;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }
    location /static/ {
        alias /home/GDSC/web/staticfiles/;
    }
}

now we have a fully working django app !.

You might also like...
The leading native Python SSHv2 protocol library.

Paramiko Paramiko: Python SSH module Copyright: Copyright (c) 2009 Robey Pointer [email protected] Copyright: Copyright (c) 2020 Jeff Forcier

Official Python client library for kubernetes

Kubernetes Python Client Python client for the kubernetes API. Installation From source: git clone --recursive https://github.com/kubernetes-client/py

Python job scheduling for humans.

schedule Python job scheduling for humans. Run Python functions (or any other callable) periodically using a friendly syntax. A simple to use API for

A cron monitoring tool written in Python & Django
A cron monitoring tool written in Python & Django

Healthchecks Healthchecks is a cron job monitoring service. It listens for HTTP requests and email messages ("pings") from your cron jobs and schedule

Remote Desktop Protocol in Twisted Python

RDPY Remote Desktop Protocol in twisted python. RDPY is a pure Python implementation of the Microsoft RDP (Remote Desktop Protocol) protocol (client a

Python utility function to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed

iterable-subprocess Python utility function to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be

Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions
Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions

Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions

Push Container Image To Docker Registry In Python

push-container-image-to-docker-registry 概要 push-container-image-to-docker-registry は、エッジコンピューティング環境において、特定のエッジ端末上の Private Docker Registry に特定のコンテナイメー

Get Response Of Container Deployment Kube with python

get-response-of-container-deployment-kube 概要 get-response-of-container-deployment-kube は、例えばエッジコンピューティング環境のコンテナデプロイメントシステムにおいて、デプロイ元の端末がデプロイ先のコンテナデプロイ

Owner
abdullah mosibah
i live by algorithms
abdullah mosibah
Jenkins-AWS-CICD - Implement Jenkins CI/CD with AWS CodeBuild and AWS CodeDeploy, build a python flask web application.

Jenkins-AWS-CICD - Implement Jenkins CI/CD with AWS CodeBuild and AWS CodeDeploy, build a python flask web application.

Ning 1 Jan 1, 2022
A simple python application for running a CI pipeline locally This app currently supports GitLab CI scripts

?? Simple Local CI Runner ?? A simple python application for running a CI pipeline locally This app currently supports GitLab CI scripts ⚙️ Setup Inst

Tom Stowe 0 Jan 11, 2022
Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:

Latest Salt Documentation Open an issue (bug report, feature request, etc.) Salt is the world’s fastest, most intelligent and scalable automation engi

SaltStack 12.9k Jan 4, 2023
Phonebook application to manage phone numbers

PhoneBook Phonebook application to manage phone numbers. How to Use run main.py python file. python3 main.py Links Download Source Code: Click Here M

Mohammad Dori 3 Jul 15, 2022
Coding For Entrepreneurs 100 Jan 1, 2023
Bugbane - Application security tools for CI/CD pipeline

BugBane Набор утилит для аудита безопасности приложений. Основные принципы и осо

GardaTech 20 Dec 9, 2022
Universal Command Line Interface for Amazon Web Services

aws-cli This package provides a unified command line interface to Amazon Web Services. Jump to: Getting Started Getting Help More Resources Getting St

Amazon Web Services 13.3k Jan 1, 2023
Honcho: a python clone of Foreman. For managing Procfile-based applications.

___ ___ ___ ___ ___ ___ /\__\ /\ \ /\__\ /\ \ /\__\ /\

Nick Stenning 1.5k Jan 3, 2023
Cross-platform lib for process and system monitoring in Python

Home Install Documentation Download Forum Blog Funding What's new Summary psutil (process and system utilities) is a cross-platform library for retrie

Giampaolo Rodola 9k Jan 2, 2023
A Python library for the Docker Engine API

Docker SDK for Python A Python library for the Docker Engine API. It lets you do anything the docker command does, but from within Python apps – run c

Docker 6.1k Dec 31, 2022