Intro

Prometheus is an open source project that is part of the Cloud Native Computing Foundation(CNCF). It serves as a monitoring system and time series database. The project was second to be part of CNCF after Kubernetes. As such the two project play really nicely with one another.

In this post I'll take us through using Kubernetes(K8s for short) as our deployment platform for Prometheus, then setting up a simple python app for monitoring. I recommend checking out my previous post on standing up K8s on AWS first, here. Otherwise if you already have a K8s cluster, lets get started.

Prerequisites

• A running Kubernetes cluster
• Configured kubectl client

Architecture

Not going to do a complete architecture run through here, but I want to mention some of the basics.

Prometheus takes a particular stance on how it goes about monitoring. Most monitoring systems out there rely on the use of an agent, that sits on the client machine. This agent will gather metric from the host and shoot them over to the monitoring system. This can be seen as a push method.

Prometheus takes more of a pull method to this. The idea behind this is that you tell Prometheus what it needs to monitor, and it will go out and start scraping that endpoint for metric.

This approach provides a lot of flexibility as application KPIs can be added/modified/removed very quickly and provides a modern devops approach to application monitoring. It also means application monitoring is driven by the individuals/teams who understand the application(to be monitored) best.

Jump in

If you want to jump straight in, deploy the following in your k8s cluster:

• python-app - simple python app that incorporates Prometheus monitoring. Metrics provides by app: request_processing_seconds, index_request_processing_seconds, requests_for_host.

• Prometheus - The official Prometheus image.

• Assets - All assets described in the post can be retrieved from this repo

Deploy Prometheus

Deploying Prometheus into K8s is super easy. Lets take a look at the deployment file deployment.yml.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment-1.7.1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v1.7.1
          args:
            - "-config.file=/etc/prometheus/conf/prometheus.yml"
            # Metrics are stored in an emptyDir volume which
            # exists as long as the Pod is running on that Node.
            # The data in an emptyDir volume is safe across 
            # container crashes.
            - "-storage.local.path=/prometheus/"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/conf/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        # The config map we will create then bind to our volume mount.
        - name: prometheus-config-volume
          configMap:
            name: prometheus-server-conf
        # Create the actual volume for the metric data
        - name: prometheus-storage-volume
          emptyDir: {} # containers in the Pod can all read and write the same files here.

Before we deploy this let first create the configmap for the prometheus config. A configmap is a way we can decouple and store configuration files for use in different deployments. Lets looks at the prometheus.yml config.

global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'python-app'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: python-app
        action: keep

Here we have setup two scrape jobs. One is simply telling Prometheus to scrape itself. The other is scraping our example python-app. Take notice of the kubernetes_sd_configs directive, very handy for using the kubernetes api to find scrape targets. Here we are using the pods role to discover all pods with an app label of 'python-app'. Also it is worth mentioning that by default Prometheus will scrape the /metrics endpoint unless stated otherwise.

We can create the configmap like so:

kubectl create configmap prometheus-server-conf --from-file=prometheus.yml

Now that we have created that we can create our prometheus deployment.

kubectl create -f deployment.yml

Once you have deployed Prometheus you will be able to see your deployment via

kubectl get deploy
NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
prometheus-deployment-1.7.1   1         1         1            1           40s

Now that Prometheus has been deployed, we need a method of accessing it. For this we will need to create a service(svc). A service manifest file looks like so:

apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
spec:
  selector: # exposes any pods with the following labels as a service
    app: prometheus-server
  type: NodePort
  ports:
    - port: 80 # this Service's port (cluster-internal IP clusterIP)
      targetPort: 9090 # pods expose this port
      # Kubernetes master will allocate a port from a flag-configured range (default: 30000-32767),
      # or we can set a specific port number (in our case).
      # Each node will proxy 32514 port (the same port number on every node) into this service.
      # Note that this Service will be visible as both NodeIP:nodePort and clusterIp:port
      nodePort: 32514

We can deploy a service in a similar fashion as the deployment:

kubectl create -f svc.yml

Now that we have a service for prometheus we can access it simply by hitting the node on port 32514 as we specified. Take note, if you are using aws then use the public ip of the instance (make sure your security group allows you access over this port).

Deploy Python-app

Now that Prometheus has been deployed we can now deploy our simple python app and start scraping some metric in. Our deployment file my-app.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: python-app
spec:
  replicas: 1
  minReadySeconds: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: python-app
        image: vect0r/python-app:latest
        ports:
        - containerPort: 5000
          name: web-app

Lets deploy this:

kubectl create -f my-app.yml

As before we'll create a service manifest, my-app-svc.yml:

apiVersion: v1
kind: Service
metadata:
  name: my-app-svc
spec:
  selector: # exposes any pods with the following labels as a service
    app: python-app
  type: NodePort
  ports:
    - port: 5000 # this Service's port (cluster-internal IP clusterIP)
      targetPort: 5000 # pods expose this port

Notice here we haven't specified a nodePort. Kubernetes will pick one out for us. Create the service:

kubectl create -f my-app-svc.yml
kubectl get svc
NAME                  CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
prometheus-service    100.69.192.22   <nodes>       80:32514/TCP                    13d
my-app-svc            100.69.92.128   <nodes>      5000:32455/TCP   12d

Listing the services displays the nodePort selected, we can see the python app can be accessed on 32455. Remember back when we created our configmap for prometheus. We specified in our config to find pods with the 'python-app' app tag and scrape them. Hence now we have done this we should see the following targets in Prometheus.

App monitoring

We have two views we are monitoring in our web app. These are the '/' and '/host' endpoints. Lets take a look at our index view:

# Create a metric to track time spent and requests made.
INDEX_TIME = Summary('index_request_processing_seconds', 'DESC: INDEX time spent processing request')

# Create a metric to count the number of runs on process_request()
c = Counter('requests_for_host', 'Number of runs of the process_request method', ['method', 'endpoint'])

@app.route('/')
@INDEX_TIME.time()
def hello_world():
    path = str(request.path)
    verb = request.method
    label_dict = {"method": verb,
                 "endpoint": path}
    c.labels(**label_dict).inc()

    return 'Flask Dockerized'

This block of the code is doing serveral things, firstly we are creating a Summary metric type. This is one of the metric type the prometheus_client supports. The first parameter is the name of the metric, the second is a description text for the metric. The second metric we create is a Counter, similar to the Summary the first two parameters are name and description. However a third parameter is provided, this allows us to provide tags to differentiate targets. Think of it as allowing you to create another line on the graph.

Then we create our view using the flask decorator, notice we also use INDEX_TIME as a decorator and using the time() method to give us the time taken to execute this method. The view itself is just returning 'Flask Dockerized' to the webpage, however note that we are using flask to get the path requested and the HTTP verb used. We save this to a dictionary and pass it through to our counter.

Expose metrics

How do we expose this to Prometheus? Well as mentioned before the Prometheus by default looks at the /metrics endpoint for all it's scraping. So lets build that endpoint into our app.

@app.route('/metrics')
def metrics():
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)

What we have done here is simply used Flask's routing mechanism to create the metrics endpoint. This view is returning a Flask Response object. the generate_latest() method is provided by the Prometheus client and this will provide a string with all the metrics defined in the app and their values, in the Prometheus standard. We wrap this in the Response object that is passed back to the client. Output looks like below:

Every time Prometheus scrapes this app, this block of code will run providing Prometheus with the latest metrics values, which it stores in its TSDB.

Graph in action

Lets take a look at what the code does for us. Navigate to your Prometheus UI and search on our metric 'index_request_processing_seconds', you will notice there is a count and sum. Every time you hit the index endpoint you will see both these metrics change. The count will show you how many times the view has been called, the sum will show you the total execution time taken by this method. You can see this in action below:

The Counter metric we created can be found by searching requests_for_host.

Here notice we have two lines one for our '/index' view and another for our '/hosts' view.

Wrap up

So we've gone go over deploying Prometheus on Kubernetes and using the Prometheus_client to monitor our app. This has only scratched the surface of what you can do. I hope this has help you kick start your use of Prometheus.

Intro

So I've been using Sumologic for a while now. Its a great way to get up and running with centralised log monitoring very quickly. No need to install and maintain an ELK cluster. Just sign up and start throwing it logs and configure your dashboards!

Lately I had a need to integrate Sumologic's searching capabilities with a centralised alerting console I use. Here I am going to cover how I went about doing this with Sumologic and AWS. Note that the solution I am using here is not just applicable to Sumologic. Any system that has support for webhooks can use this solution.

Prerequisites

Before continuing with the rest of this article. Assumptions below have been made:

• You have an AWS account
• You have a Sumologic account
• Familiarity with Sumologic, AWS API Gateway, AWS Lambda, AWS SNS, AWS SQS and Boto3

Architecture

To achieve this I am utilising scheduled views in Sumologic to raise an alert based on my search and threshold, then using Webhooks to shoot off to an AWS API Gateway method backed by a Lambda function. This Lambda function will publishes to an SNS which in turn sends the message to SQS ready to get picked up by my alerting console. Below diagram describes this.

For the purpose of this example I will take us all the way to the SQS level. You may not want to go that far depending on your use case.

SNS topic and SQS

Starting from the end of the diagram we will create a standard SQS queue using the quick-create queue.

You should end up with a queue like below.

Then create your SNS topic and subscribe your SQS to it.

Lambda

Now we will take a look at the Lambda function powering the backend. See the code below:

import json
import boto3

sns = boto3.client('sns')

def lambda_handler(event, context):
    snsARN = '<ARN_OF_YOUR_SNS'
    data = event
    data["default"] = json.dumps(event)

    try:
        print("Publishing message to sns")
        sns.publish(TargetArn=snsARN, Message=json.dumps(data), MessageStructure='json', Subject="Alert")
        print("Published to sns")

    except Exception as e:
        print(e)
        raise e

This simple lambda is using boto3 to publish the webhook into an sns topic, which has our SQS subcribed to.

Important that you give this lambda function the correct permissions. To do this I have create a role in IAM with the following permissions and attached this role to my function:

API Gateway

The front end to this service is the AWS API Gateway. Firstly create your API by clicking the "Create API button" in the API Gateway console.
The API for simplicity and extensibility is very minimal by design. The below table describes the endpoint available:

POST /Sumo2Siren

This POST method provides the means for the Webhook from Sumologic to pass data in. Lets take a look at this method.

Nothing special here, you can set up your endpoint as you desire, I have opted to use API keys without any authorization. I will be using usage plans to specify connection limitations, we will see this soon.
Now lets take a look at the integration setup for your method:

Here I am using simple Lambda integration. The proxy integration is handy if you want the entire request object to get wrapped inside a JSON blob. In this use case the simple body of the request is enough.

You should end up with a workflow that looks like this:

API Keys and usage plan

Usage plans allow you to specify lower levels of control around what can be done with your API. You can enforce throttling, max hits, bursts and rate limits. If you haven't used Usage plans yet and don't see the option in your API console, you most likely need to enable it on your account.
In the Usage plan menu, create your plan. Like below:

Hit next when done, then skip on the next page as we still need to create our key.

Next create your API key. Go to the API Gateway console and select API keys, then create key. Once created click back on your key and you will see there is an "Add to Usage plan" option available. Hit that and you can now tie your key to the usage plan, like below:

Take a note of your API key by clicking on the "Show" button next to "API key"

Deploy API

Now you are ready to deploy your API. To do this you need to Stage it. Go to your API and click on "stage" menu option. Create a stage like below:

I advise you pick an appropriate stage name as this will make up part of your endpoint. I like using name like "vN" where N is the version iteration, as this allows for easy tracking of deployments. Once created you will get an "Invole URL" which is just an endpoint to your API, take note of this.

Sumologic Webhook

First you need to set up your Webhook connection in Sumologic. Log in and navigate to Manage --> Data Configuration --> Connections. Create a connection, in the wizard choose "Webhook":

Don't forget to place your URL in there, this will be the endpoint you noted earlier.
This payload comprises the body of the request, this is what will be passed to your Lambda function. For further payload examples see here.

Now that you have created your connection to your API, you need to set a scheduled view that will make use of this webhook once alerted.

Scheduled View

You can follow the instructions here. The last part of the puzzle is to connect your scheduled view to the webhook, this is done through the "connection" field:

Now sit back and watch those alerts fire when matching your condition.

Intro

So creating a new K8s cluster on AWS is easy as ever with Kops. It can help you save a lot of time, help you in setting up your autoscaling groups and HA config, as well as help you with maintaining central versioned cluster config. We'll go through setting up Kops and using it to create a HA, autoscaling K8s cluster.

Prerequisites

You will need the following:

• An AWS account
• A linux host or VM (sorry windows!)
• A resolvable DNS or subdomain (this is important!)

Install Kops

First thing is first, lets install the Kops tools. Its a Golang project so you can grab it from source via the following:

go get -d k8s.io/kops
cd ${GOPATH}/src/k8s.io/kops/
git checkout release
make

Caveats:

• Make sure you have set your GOPATH in your env. Kops expects the source to be checked out in $GOPATH/src/k8s.io/kops. You will get errors if this is not the case.
• You will require Go 1.6+

Alternatively if you are using a Mac and have Homebrew you can simply do:

brew update && brew install --HEAD kops

Now that is done you should get something like below when using Kops in your terminal:

kops version
Version 1.6.0-alpha.2 (git-3fe6e04)

Install Kubectl

Kubectl is the controller interface between you and your cluster. You will want this to talk to the API server and issue commands to your cluster.

For Mac users:

brew install kubernetes-cli

Linux users:

1. curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

2. Make the binary excutable chmod +x ./kubectl

3. Move the binary file to your PATH sudo mv ./kubectl /usr/local/bin/kubectl

Install AWSCLI

This is be used to create AWS resource quickly as part of the initial setup. Simply install AWSCLI via:

pip install awscli

We will use AWS CLI to create some initial resources in your AWS account. Make sure you have place an API key and secret pair for your AWS account in $ ~/.aws/credentials. An example of the file should look like so:

# AWS creds
[default]
aws_access_key_id = XXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Your awscli config can be found in ~/.aws/config. Set up as desired, example below:

[default]

# If web proxy is needed
#proxy=http://www-cache.my.web.proxy:80

# AWS regions
#London
#region = eu-west-2

# Frankfurt
region = eu-central-1

# Ireland
#region = eu-west-1

Prepare AWS Account

Now you need to prep your account. We will create the following initial resources before using Kops to setup the cluster.

• Create a dedicated group for Kops in your AWS account.
• Create a dedicated user for Kops in your AWS account.
• Assign the neccessary IAM policies

Firstly create the group with:

aws iam create-group --group-name kops-group

Next assign the following group policies:

aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops-group

Then create the Kops user:

aws iam create-user --user-name kops

Assign that user to the Kops group:

aws iam add-user-to-group --user-name kops --group-name kops

Lastly create an API key and secret for that user. These credentials will be used by the Kops tool.

aws iam create-access-key --user-name kops

This returns the pair to your terminal. Use these to update ~/.aws/credentials with the new pair. Also export the following in your term with your new key and secret:

export AWS_ACCESS_KEY_ID=<access key>
export AWS_SECRET_ACCESS_KEY=<secret key>

DNS configuration

This is the important part, without this the rest won't work. Kops will by default rely on communicating with your cluster through a public DNS.
As there are many providers out there I will talk about the high levels of what you need to do.

For a dedicated hosted DNS, all that is required is purchasing a domain with AWS. Simply go the Route53 console. Once this is done you will have a hosted zone. No other action is needed

For testing purposes you will probably want to use a subdomain, instead of purchasing and entire domain. Chances are this is hosted by a different DNS provider. Not an issue.

To do this, you will firstly need to create a hosted domain in Route53 for the subdomain you want to use. For example this my be, k8s.mydomain.com. In that hosted zone create an NS record set for your subdomain. Doing this will give 4 name servers. Make a note of these name servers. Log into your dns providers portal and create an NS record with them. Place the subdomain as your host and the name servers you took note of as the values. You should have something that looks like below for our example of k8s.mydomain.com.

Host: k8s
Value: aws.example.dns1.com, aws.example.dns2.org, aws.example.dns3.net, aws.example.dns4.co.uk

Refer to your Domain providers documenation around creating a subdomain.

You can test your subdomain setup is correct by doing a dig and looking for the 4 noted NS servers to be returned:

dig ns subdomain.example.com

Cluster state

Ok with DNS out the way, next we need to establish a place for our cluster state to live. Best place to put this is into an S3 bucket as it is a highly reliable source of truth for our cluster, bundled with the versioning capabilities S3 provides allows us to capture state changes to the cluster for auditing.

Create your S3 bucket in the region desired:

aws s3api create-bucket --bucket k8s-mydomain-com-state-store --region eu-central-1

Enable versioning on the bucket (optional)

aws s3api put-bucket-versioning --bucket k8s-mydomain-com-state-store  --versioning-configuration Status=Enabled

Let do this!

So still with us?.....Great as now we are ready to kick this thing off. Export the following variable to your env:

# simply a name you want to use for your cluster for my example it can be.
export NAME=myfirstcluster.k8s.mydomain.com

# The location of the S3 that holds the state of the cluster, we created this earlier
export KOPS_STATE_STORE=s3://k8s-mydomain-com-state-store

Now, lets create our cluster config.

kops create cluster --master-zones=eu-central-1a,eu-central-1b,eu-central-1c \
--master-size=m4.large \
--zones=eu-central-1a,eu-central-1b,eu-central-1c \
--node-count=4 --node-size=m4.large ${NAME}

So what is going on here, we are tell Kops to:

• --master-zones create our master nodes across 3 different availability zones for an HA deployment.
• --master-size tell it our EC2 size for the master node
  • --zones deploy our worker nodes between 3 different availability zones, again for HA.
• --node-count how many nodes you want in the cluster
• --node-size the size of the EC2 instances for the nodes.

Once this has successfully completed, take a look in your S3 bucket. You will see the cluster config Kops as created for you. Now that this is done, you can deploy your cluster with

kops update cluster ${NAME} --yes

This will take a little bit complete. Go take 20 mins, you've earned it at this point ^_^

Once done check out your cluster via kops get clusters or check out your instance groups for your cluster with kops get ig, you should see 4 groups split up between 3 separate masters and one node group.

Now you can start using kubectl to interact with our cluster. Luckily Kops has taken care of the kubectl config for us when we span up the cluster. Check out ~/.kube/config.

You can check your cluster is happy by doing kops validate cluster and get your cluster nodes via kubectl get nodes

Congratulations!

You've stuck with it and now you have a fully HA K8s cluster on AWS!

Introduction

OpenSSL is a library that provides cryptographic functionality, specifically SSL/TLS for popular applications such as secure web servers. When going for that A+ on Qualys SSL labs, you most proberly want to complie your own OpenSSL. We'll go through doing that.

Requirements

This is going to be Fedora base so grab yourself a RHEL 7 or Centos 7 box.

Installation

Get your current version with “openssl version”:

openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013 

and “yum info openssl” command :

yum info openssl

To download the latest version of OpenSSL, do as follows:

cd /usr/local/src
wget https://www.openssl.org/source/openssl-1.0.2-latest.tar.gz
tar -zxf openssl-1.0.2-latest.tar.gz

To manually compile OpenSSL and install/upgrade OpenSSL, do as follows:

cd openssl-1.0.2a
./config
make
make test
make install

Next you need to swap our your binaries, move or delete your old version and place it with you latest build one:

mv /usr/bin/openssl /root/
ln -s /usr/local/ssl/bin/openssl /usr/bin/openssl

Now verify the OpenSSL version.

openssl version
OpenSSL 1.0.2e 3 Dec 2015

That's it, job done. Hope this has been helpful.

Note: Compiling Openssl major version may case issues with other system binaries. So make sure you know what is depending on it.

Working behind a proxy is always a bit of a pain. Luckly if you are doing so work with Docker behind a proxy you've come to the right place. Below steps should allow your docker instance to shot out to the internet to get images. Note, I will go about how to do this with systemd. You will need to adapt this slightly if using a different init system.

Config

First make the following directory

mkdir /etc/systemd/system/docker.service.d

Then create a http-proxy.conf file under this directory. This conf file will hold the proxy details. The content as follows:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"

Take note, if you have a local docker registry that you don't want to go through the proxy you can set a no proxy option like so:

[Service]
Environment="NO_PROXY=localhost,127.0.0.0/8,docker-registry.local.com"

Essentially place the dns names or ips of the address you don't want to go through the proxy.

Once this is done simply flush your changes:

sudo systemctl daemon-reload

You can verify these changes have been applied via:

systemctl show --property Environment docker

Finally restart docker

systemctl restart docker

You should now be able to pull your images from your proxy.

Intro

Ok so I've been working with Docker in production for a while now. If like me your sitting in the same boat, then I'm sure you can appreciate the pain that comes along with Docker and disk space in production environments. In this post I'm going to list some tasks you may find yourself having to do to get yourself out of a pickle.

Tasks

So what are these tasks I mentioned. Well, ever asked yourself any of these questions?;

• Why am I getting log messages about a read only file system?
• I have a cluster of containers running, how can I find out which ones are read only?
• How can reclaim space back from my docker-pool LV?
• My docker-pool LV is full, how can I resize it?
• My docker-pool LV is corrupted how can I recreated it?

You will notice I've structured these questions in a logical way starting with the issue you might see and ending with the worst case scenario. We'll take a look at what tasks we can do to answer these questions.

Prerequisites

So going forwards, I'm expecting you to be;

• Familiar with Docker and comfortable with Docker terminology and concepts.
• Proficient Linux administration skills and understand storage terminologies in Linux systems.

Read Only

So your sitting back, drinking your ice tea. watching your container cluster at work..... Then you start seeing errors getting flagged up, lets say you see them coming into Logstash. Lets say you see an error "filesystem is readonly". Of course this error message is going to be unique to your environment but the key point is that you've deduce you have a disk related issue.
So what causes this? Could be a range of things, however there may be good chances that your docker-pool LV has run out of space. Do the below on your system;

# You should see output something like below when running lvs
$ lvs
LogVol00        VolGroup00 -wi-ao---- 37.97g
LogVol01        VolGroup00 -wi-ao----  1.50g
docker-pool     VolGroup00 -wi-a----- 40.00g          100%

The docker-pool lvs is where all your docker images, containers and ultimately data is stored in. So if you see 100% data utilisation against the docker-pool then you've just confirmed why you are seeing read only error messages.

Determine which containers are read only

From what I've seen in production, when this occurs don't assume all containers on your host become read only. As I've seen only a handful of containers turning read only in my cluster when hitting this issue. Having said that, DO expect to see all sorts of issues occurring in your cluster.
So at this point you will want to do some analysis to determine where all the space has gone. I'll always recommend you do this before jumping straight in and recreating/resizing your docker-pool. Blowing away your docker-pool and recreating it will resolve your problem in the short term, but if you haven't identified what has used up all your space, you are likely to run into this issue again.

Ok so before we can proceed, first we need to get the PID's of all the running containers. We can do this my running the following command on your host:

docker ps -q | xargs docker inspect --format '{{.ID}}, {{.State.Pid}}' | awk {'print $2'}

Now that you have a list of all the PIDs, go into your /Proc directory. Here you will find directories for each of the PIDs you go back. If you have a look in there you will find a mounts file. This file countains the mount points of your running container. It essentially displays the out from a 'mount -l'. Run the below command and you can see if that running container is readonly.

cat mounts | awk '/docker/'

I've created the below little script you can run to automate this process for all your running containers.

#! /bin/bash

echo "ContainerID  ContainerPID  Mounts"

for i in $(/bin/docker ps -q); do
  container_pid=$(/bin/docker inspect -f '{{ .State.Pid }}' ${i})
  /bin/cat /proc/${container_pid}/mounts | /bin/echo ${i} $container_pid $(/bin/awk '/docker/ {print "Container=" $4}')
done

Now that you have identified your RO containers, you can start debugging further. In production I've seen containers not freeing up space because a file is still in use even though it has been deleted, or heavy log output taking up a lot of disk space. These are just some of the things that can be the cause of your disk space issue. Its up to you to identify this and resolve the issue.

Reclaim some diskspace

Ok so the commands you will find here can be run against a running environment however I recommend bringing your services down before doing this, as if only a little bit of space is reclaimed, you could run the risk of instantly filling that back up again. Essentially what we are going to do here is run a fstrim on the filesystem for your containers. Now usually you would run your fstrim on the mountpoint in question. However for containers its a bit more tricky as you can't just do a df to find the mount point as you would on a normal linux host. To do this we need to employ a similar tactic as before. Use the command we saw earlier to get the PIDs of your running containers.

docker ps -q | xargs docker inspect --format '{{.ID}}, {{.State.Pid}}' | awk {'print $2'}

Then cd into the proc locations i.e.;

cd /proc/17756

In here you will see there is a root symlink. This is actually a symlink to your running container fs. Now that you found this you can do an fstrim on it like so;

fstrim -v /proc/17756/root

You can apply this logical to all of your running containers to trim down any empty blocks. As before I've provided a script that will automate doing this for all your running containers.

#! /bin/bash

for i in ‘docker ps -q’; do
  container_pid=$(/bin/docker inspect -f '{{ .State.Pid }}' ${i})
  fstrim -v /proc/${container_pid}/root
done

After running this you should have reclaimed so disk space back. Check your docker-pool and see if it back to reasonable levels with;

lvs

Resize it?

So trimming down has not helped and you need to resize. Ok no problem, if you have free space left in the volume group your docker-pool is sitting on you can do something like;

lvextend -l 100%FREE /dev/VolGroup00/docker-pool

This will take all available space left in your volume group and assign it to your docker-pool. Of course if you no longer have space left in your volume group you will first have to assign more space to it before performing the above.

Recreate it?

You might not care for the data in your docker-pool, or there is some sort of corruption that has gotten too messy to unpick. Well there is a last resort.....Nuke it.
It should go without saying that blowing away the docker-pool will wipe and data Docker will be holding on your cluster. This will be all container resources, metadata and container images..... proceed at your own peril...

So to do this first stop your cluster then stop docker;

systemctl stop docker

Once docker has stopped remove the below files;

rm -rf /var/lib/docker/*

Once these steps are done you will can now blow away the docker pool LV;

lvremove VolGroup00/docker-pool

Now that is done we can create the pool;

lvcreate -T --name docker-pool -l 90%FREE VolGroup00

Now your done, do an lvs and you should see your pool is clean and back down to zero. (note the 90%, leaving space just incase...paranoia is a good thing here. ^_^ )

Fin

Hope this post has helped. Till next time!

Intro

Sorry for the long absence. Been struggling to find some time to sit down and write, so glad to be finally doing it! Today's topic I am going to go through some work I've been doing with Control Center. (note American spelling!)

What is Control Center?

You probably would not of heard of Control Centre unless you've been working with Zenoss. The project seems to have stated, publicly anyway, in Aug 2013 and is actively being developed.
Control Centre is essentially a neat container management system. No, not the type you find at the Docklands.

It's key components are:

• Docker & Docker Registry
• OpentTSDB & Hbase
• Logstash - process logs
• Elasticsearch - Store logs
• Zookeeper - cluster coordination
• AngularJS, Bootstrap and nvd3.js for UI
• Linux kernel for NFS, Btrfs and cgroups.

Overview

At the heart of Control Centre's automation are JSON service files, it is in here where you define what Control Centre should do to deliver your application. As part of this article we will:

• Install Control Centre
• Build a Docker image for our app
• Provide the config files for our app to Control Centre
• Configure the Networking for our container via Control Centre
• Create persistent storage with Control Centre with the use of Docker Volumes

Installing Control Centre

Before we start, I am assuming you already have docker installed and set up on your system. If not, install it before continuing.

Hate to start this off with a caveat but according to the official docs, you have to make sure you have set a hostname/IP in your host file. Control Center will not function correctly if the hostname is mapped to the loopback adapter IP: 127.0.0.1. Once you've address this you are ready to start with the below:

$>sudo apt-key adv --keyserver keys.gnupg.net --recv-keys AA5A1AD7
$>sudo sh -c 'echo "deb [ arch=amd64 ] http://get.zenoss.io/apt/ubuntu trusty universe" \
    > /etc/apt/sources.list.d/zenoss.list'
$>sudo apt-get update
$>sudo apt-get install serviced           

Once you let that complete you should be able to start it up:

$>sudo start serviced

Control Center's GUI is accessible over https thus upon starting serviecd the webserver binds to port 443. Make sure you don't already have a daemon bound to this port or you will see start up errors in the logs /var/log/upstart/serviced.log.

Deploy an application to Control Center

Before showing you how to deploy an application in Control Center, lets first cover some theory. So firstly Control Center provides a layer of abstraction from Docker. There is no need to write Dockerfiles for your containers. Instead you provide are template files, that are in JSON format, to instruct Control Center. It is beneficial to also note that Control Center refers to containers as applications, we will use this terminology going forwards to avoid confusion.

Now lets look at and example Control Center application. I have taken uploaded an example called webserver.json. There is alot of configuration options in the template. I will covered the options we are using.

 2/ "ID": "",
 3/ "Name": "Python-Web-Server",
 4/ "Description": "Python Web Server",

The first couple of lines just define some metadata. This can be whatever you want. Here we have decided not to give an ID, in which case and automatic one is given. An appropriate description is given along with the name.

5/   "Services": [
6/ {
7/   "Name": "Python Web Server",
8/   "Command": "",
9/   "Description": "Python Web Server",
10/   "Tags": null,
11/   "ImageID": "",
12/   "Instances": {
13/     "Min": 0,
14/     "Max": 0
15/   },
16/   "ChangeOptions": null,
17/   "Launch": "auto",
18/   "HostPolicy": "",
19/   "Hostname": "",
20/   "Privileged": false,
21/   "ConfigFiles": {},
22/   "Context": null,
23/   "Endpoints": null,
24/   "Services": [

The "Services" definition accepts and array of objects and allows you to specify a hierarchy to your application. What we are doing here is creating a set of resources underneath the Python-Web-Server application. You can think of the Python-Web-Server application being a group that has a "Python Web Server" service within in. The config following on is pretty self explanatory, the ones to point out are;

• "Instances": Its possible to have multiple instances of a service, for example multiple nginx web service.
• "Launch": Specify whether the service starts automatically, manually etc.
• "Hostname": Hostname of the container spawned by Control Center.

Following on from the other Services directive we have the bulk of configuration:

25/     {
26/       "Name": "python",
27/       "Command": "/usr/local/bin/python /data/webserver.py -p 8000 -l /var/log/webserver.log",
28/       "Description": "",
29/       "Tags": [
30/         "daemon"
31/       ],
32/       "ImageID": "python:2.7.11",
33/       "Instances": {
34/         "Min": 1,
35/         "Max": 0
36/       },

We start off by giving a name to the docker image, and give the image a command to run in the container. Notice we feed in a webserver.py file, we will see how the container knows about this later.
On line 32 we give the imageID of the docker base image we want to use. I am using the 2.7.11 release of the python image found on dockerhub. By specifying a min of 1 instance ensures there is always at least one container running for this service.
So remember the webserver.py we spoke about. The next couple of lines down we actually define this file.

42/       "ConfigFiles": {
43/         "/data/webserver.py": {
44/           "Filename": "/data/webserver.py",
45/           "Owner": "root",
46/           "Permissions": "",
47/           "Content": 
48/         }
49/       }

Note the content directive is where you specify the actual file content. This will need to be provided as one line in the json template, so make sure you use the ascii characters for new lines etc.

51/       "Endpoints": [
52/         {
53/           "Name": "http",
54/           "Purpose": "export",
55/           "Protocol": "tcp",
56/           "PortNumber": 8000,
57/           "PortTemplate": "",
58/           "VirtualAddress": "",
59/           "Application": "http",
60/           "ApplicationTemplate": "",
61/           "AddressConfig": {
62/             "Port": 8000,
63/             "Protocol": "tcp"
64/           },
65/           "VHosts": [
66/             "web"
67/           ]
68/         }
69/       ]

The Endpoints directive allows you to sort out the networking mapping between your host and your container. The above snippet is exposing container port via the "PortNumber" and making it accessible via host port 8000, as can be seen from the "AddressConfig" object. The vhost object tells control center to expose an endpoint over a vhost name so you don't have to use the ip address, here we just gave the name "web".

Other important sections to note:

74/      "LogConfigs": [
75/         {
76/           "Path": "/var/log/webserver.log",
77/           "Type": "apache",
78/           "Filters": null,
79/           "LogTags": null
80/         }
81/       ],

The "LogConfig" directive allows your to provide the location of the logs we're interested within the container.

117/       "RAMCommitment": 1073741824,
118/       "CPUCommitment": 2,

"RAMCommitment" directive below specific how much RAM in bytes the container takes from the host."CPUCommitment" dedicates CPU cores from the host.

Right so we need to add this template into Control Center. Take the webserver.json template and do:

# add template to serviced
$>serviced template add webserver.json

# then deploy the template into Control Center
$>serviced template deploy <template_id>

Once the template is deployed you should be able to see the service in Control Center. You can run the service via:

$>serviced service start <serviced_id>

Once the service is up an running you can access the python web server via https://web. < hostname >

Conclusion

Control Center allows you to design your systems with service oriented architecture in mind. I hope this post provides you with a jump start into Control Center and how it can be used to manage and run your container based services.

Hello again, today I’m going to pick up where I left off with more Flask~ing (…..can I say that? o_0 ). As usual with my multi-part posts I suggest you start with the first, you can find that here. I spoke about setting up your Flask environment and touched on routing your web application. I want to now move onto showing you how to incorporate an API with Flask. API-First design has now become common practise with teams of developers, often the API is designed to serve your front end web site as a client with the scope of opening up the API for public use.

To do this I will be using Firebase. Firebase provides a great backend as a service (BaaS) product to quickly get up and running with you API. You can log in with your Google account to use the service, the free service is perfectly adequate for our purposes. The documentation on the firebase website is very good and easy to pick up so I won’t go through it. However I will provide you a base you can import to your app database, which is just one big JSON object.

    {
      "films" : {
        "film1" : {
          "Rating" : "9",
          "Title" : "The Matrix",
          "Year" : "1999"
           }
        }
    }

Now that you have Firebase set up we are ready to build our Flask API.

Theory######

Just a quick run through about what we are trying to achieve. In this example, we are going to create a form where someone can enter information about a movie. We will ask for title, the year of release and a rating. Upon submitting the form we are going to use the api service to shoot the information as JSON to our database on firebase. In order to do this we are going to use a few different packages. We will use;

• WTForms - a python library for form rendering and validation
• Flask-WTF – a library which intergrates WTForms with Flask
• python-firebase – a python interface to Firebase.

Let’s get to it######

To make our life easier there is a great python package for quickly allowing us to use the Firebase API. We will need this library for our example so please go ahead and get it, note – don’t use the package from the PIP repo as it has some bugs. Instead grab it from github. Credit to Ozgur.

Let pick up with the code we left off in the last post, view.py:

    from firebase import firebase
    from flask import Flask
    from .forms import FirePut

    app = Flask(__name__)
    firebase = firebase.FirebaseApplication(‘<path tofirebase app>’, None)

    @app.route(‘/’)
    def index():
    return “<h1>Hello, world!</h1>”

    @app.route(‘/testing’)
    def testing():
    return “<h1>This is another testing page</h1>”

    If __name__ == ‘__main__’:
    app.run(debug=True)

The astute among you would of noticed that the above code has three amendments. The first is importing the firebase package, the next is where we import a package called FirePut, this will be explained later. Lastly firebase is initisalised. The would be your own link to your firebase app. It will follow the convention of https://.firebaseIO.com. Now that we have a added the package, lets create the route to our api.

    count = 0
    @app.route(‘/api/put’, methods=[‘GET’, ‘POST’])
    def fireput():
      form = FirePut()
      If form.validate_on_submit():
        global count
        count += 1
        putData = {‘Title’ : form.title.data, ‘Year’ : form.year.data, ‘Rating’ : form.rating.data}
        firebase.put(‘/films’, ‘film’ + str(count), putData)
        return render_template(‘api-put-result.html’, form=form, putData=putData)
      return render_template(‘My-Form.html’)

Ok that’s a lot of new code to digest, let’s start at the top. I started off by declaring a global variable ‘count’, then we created a route for the api declaring GET and POST methods for the route. Next I declared my function fireput() but this can really be another of your choosing. Then I declared a form field and initialised the FirePut that was imported earlier.

Next I constructed an if statement that uses the wtform method validate_on_submit() to check that the form submitted meets it’s validators( we will see how this works later). The “global count” is telling our method that we want to access the count variable and we are simply incrementing the value by one every time the method runs, (of course in the real world this value would need to be persistent). The putData variable is a dictionary containing the data we want to send over to firebase. We are using the form variable to access our FirePut method, (we still need to see what this is) for now all you need to know is this is allowing us to access information that is typed on our form.

The putData variable is used in the next line, we call the firebase put method, specify the films object that we want to add to as the first parameter then the film id (in this case I have chosen to use filmN where N is the incrementing count). The third parameter is the data we want to send, in this case the putData variable. Lastly use flasks render_template method and pass our html page, along with creating parameters that contain the form and putData.

So in short we are saying:

1. If the form is posted correctly
2. Increase our count
3. Take the captured form data and place it in the putData dictionary
4. Shoot that data off to firebase
5. And return the api-put-result html page
6. Else return the My-Form.html page
   Now as it stands this alone won’t work. Remember the .forms import of FirePut? This will handle our form processing, we need to create the file before the above will work. Let us take a look at form.py;

    from flask.ext.wtf import Form
    from wtforms import StringField, BooleanField
    from wtforms.validators import DataRequired

    Class FirePut(Form):
    	title = StringField(‘title’, validators=[DataRequired()])
    	year = StringField(‘year’, validators=[DataRequired()])
	    rating = StringField(‘rating’, validators=[DataRequired()])

Here we are using wtform and its flask intergration library flask.ext.wtf along with form validation with wtf.forms.validators. So this class we defined takes the wtf Form as a parameter and within it we define variables of object type StringField, where we pass the name of the field we want to map to in the html as param1 and a validator of DataRequired in param2. This is a simple validator that says we won’t allow nothing to be entered in this field. We do the same for year and rating. Now lets look at the html for this. My-Form.html

{% block content %}
  <h3> Complete below</h3>
    <p>Please complete the form below<p>
    <form action="" method="post" name="api">
      <p> Please enter a movie you want to add.: <br>
      {{ form.title(size=40) }}<br>
      {% for error in form.title.errors %}
        <span style="color: red;">[{{ error }}]</span>
      {% endfor %}<br>
      <p> Please enter the year the movie was released.: <br>
      {{ form.year(size=40) }} <br>
      {% for error in form.year.errors %}
        <span style="color: red;">[{{ error }}]</span>
      {% endfor %}<br>
      <p> Please enter the rating you give the movie out of 10.: <br>
      {{ form.rating(size=40) }} <br>
      {% for error in form.rating.errors %}
        <span style="color: red;">[{{ error }}]</span>
      {% endfor %}<br>
      <p><input type ="submit" value="Add" class="btn btn-sm btn-primary"></p>
    </form>
{% endblock %}

You would have noticed that this doesn’t look like your conventional html code. What we are seeing here is the Jinja template engine at work. Here we are creating a form with 3 fields, these fields map to the variables we created in form.py - movie title, movie year and movie rating. Jinja allows us to map our wtform to the actual input from the client.

{{ form.title(size=40) }}

This is how we are accessing our form variables. Remember when we set up our route, we created a variable form = FirePut(). Our template has access to this variable so when we say form.title we are referencing the title variable we set up in our FirePut() method, the parameter size=40 is specifying the size of the input box. The validation has been handled on the next line.

{% for error in form.title.errors %}
  <span style=”color: red;”>[{{ error }}]</span>

Our validator of DataRequired runs if an error is found we produce the error message giving it a red styling. We repeat this for Year and Rating.
The last piece of the puzzle is the view the user will get after submitting the form. Lets take a look at the api-form-result.html:

{% block content %}
  <h3> Complete below</h3>
  <p> The movie you entered is: {{PutData[‘Title’] }} <br>
  <p> The year this movie was made is: {{ putData[‘Year’] }} <br>
  <p> The rating you gave this movie is: {{ putData[‘Rating’] }} <br>
{% endblock %}

Here we are simply presenting the data submitted back to the user. The putData dictionary is getting accessed to place the values on the page.

So lets see this in action######

If all has gone well, you should end up with a form looking like below:

Enter your movie credentials. Once you submit these, the firebase put() method will place the information in the database and then the api-form-result.html will get rendered to the browser

Fin######

We covered a lot of concepts from templating to form processing and using Firebase in flask. I hope this post has demonstrated how you can go about building a quick api for your web application.

Intro####

Hi all, belated xmas and new year salutations! Had a busy time over the period and have only just gotten things settled to carry on my tinkering ways!
So, todays post is going to be about Python and the Flask framework. I was reading someone’s comments the other day about how easy, straight forward and clean Flask is.

After reading this I stopped and pondered a bit.......

then muttered “I like easy and clean……”. So I decided to play around with it. ^_ ^

I haven’t really gotten far, spent about an hour and a half, but decided to stop and write this up, I've come across some points I really like already that I want to write down before I forget!

Installation####

To start off I’ll talk about the installation. I’ll begin with the virtual environment install first. Those of you new to Python may not know about the python tool Virtualenv. This tool is very handy for python developers. It allows you to have isolated pockets for your various different python projects. It is not a full blown virtualisation technology alongside the likes of VMware or VirtualBox. It is simply a clever way wrapping up your projects, all you get are independent lib directories, and project local binaries for python. The great thing about it is its really fast to get up and running and comes bolted together with Python3.

All you need to do to set up a env (with Python3) is:

python3 –m venv <environment-name>

This command would create a folder in your current dir with the title of the env-name you give. A list of this dir will show you something like:

bin   include  lib   lib64    pyvenv.cfg

All you’ll find is a python binary in bin and something like 3 lines of config in pyvenv.cfg which speaks for itself. Simple and clear...nice eh?

The lib folders will contain any modules you add to your environment as not to get mixed in with your global modules thus keeping it neatly tucked away.
So at this stage, you may of gotten an error already!?! Don’t panic, it happens to the best of us ^_^. After running the venv module you may seen an error message like:

Error: Command ‘[‘<your-pwd>/bin/python3’, ‘-Im’, ‘ensurepip’, ‘—upgrade’, ‘—default-pip’]’       return non-zero exit status 1

You may be wondering what pip has to do with anything?!?!

Well, when you create your venv by default the –default-pip argument is passed, this installs pip into your venv. For reasons I have not yet looked into, you may get this message indicating that pip was not installed ( believe it may be a bug with a particular version of python3). I'll leave a comment at the bottom once I've looked into this.

To fix this, delete your environment, then create it again using the above command, but use the "with_pip=False" option. This will create you env without any errors. However you will now have to install pip ( or any other package manager) into you new env manually.
To do this we first have to start up our environment. All you need to do is:

Source /<env-name>/bin/activate

Your prompt should change to look like:

(<env-name>) [b@localhost]

This is how you can tell you are now working within your new environment. You can also tell by using the which command on python:

which python

This will show you that your using the python binary under your project as oppose to your global.
Now you have your environment up and running you can install pip via:

curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py | python

and once you have install pip into your project you can start by installing flask:

pip install flask

……..installation complete. Now let’s use flask!
As is custom I have started off by creating a simple hello word page. I’ve create a file under my environment hello.py, which looks like this:

From flask import Flask
app = Flask(__name__)

@app.route(‘/’)
def index():
    return “<h1>Hello, world!</h1>”
@app.route(‘/testing’)
def testing():
    return “<h1>This is another testing page</h1>”
If __name__ == ‘__main__’:
    app.run(debug=True)

Lets run through this code. We've imported the Flask framework and initialised it under the 'app' parameter. The routing that Flask gives us can be utilised via the '@app.route()' method. Our first route demonstrates creating our index page. Flask will implement the method when the route is accessed. In this case the index method simply returns a string to the client accessing the '/' route, and of course your browser will interpret this as a headline tag and render it accordingly. The last line runs the simple app in debug mode.

Until next time######

Flask is a small compact framework. When you install Flask, all your getting really is a routing system - Werkzeug and a templating engine - Jinja2. Today I only touched on the routing system, My next post will build on this and show you how Jinja2 can be used to help you build you application.

The ideology that the Flask framework promotes is flexibility. The framework doesn't believe in force feeding you, Flask embraces that everyone has different needs, ideas and ways of working that is specific to you. It encourage you to take charge and decide your own way of doing things.
There is no database layer incorporated into the framework. This may seem odd at first glance, this is because Flask looks after the little man first ^_^, not all applications will need a database layer, especially small applications. So if you decide you need one, it’s up to you to pick the one you need and plug it in.
If I had to sum up Flask, I would use a quote by a wise man we all know:

“Be like water my friend…”

Docker Round 2…..Fight!

Back at ya with some more Docker fun. In this post I’ll build upon the Apache fun with Docker post. In the last post we saw an example of using Docker to set up a webserver. This post will show you how you can use Docker to produce a complete LAMP stack, which you could then use as a template to deliver a lot of different LAMP sites off this one Docker image. You sure your ready bub?

Prerequisites

What do you need to follow this post? Firstly you’ll need to install Docker, install instructions here. You’ll also need:

• A good understanding of Apache HTTP Server
• A good understanding of PHP
• A understanding of Supervisord
• A good understanding of MySQL
• A decent understanding of Docker, if you’ve completed my previous post then that would be adequate.

Jump in

As always, if you’re feeling pro, jump on ahead. You can use my pre-built Docker image or follow along with the post and build it yourself here.

Suit up

Once you’ve downloaded the files off Github, let’s quickly skim through the Dockerfile and look at what is going on.

    FROM ubuntu:trusty
    MAINTAINER vect0r

    # Install packages
    ENV DEBIAN_FRONTEND noninteractive
    RUN apt-get update && \
      apt-get -y install supervisor git apache2 libapache2-mod-php5 mysql-server php5-mysql pwgen php-apc php5-mcrypt && \  
      echo "ServerName localhost" >> /etc/apache2/apache2.conf

    # Add image configuration and scripts best practice to start service from start up script to avoid problems observed with starting via service directive
    ADD start-apache2.sh /start-apache2.sh
    ADD start-mysqld.sh /start-mysqld.sh
    ADD run.sh /run.sh
    RUN chmod 755 /*.sh          # */ delete this comment
    ADD my.cnf /etc/mysql/conf.d/my.cnf
    ADD supervisord-apache2.conf /etc/supervisor/conf.d/supervisord-apache2.conf
    ADD supervisord-mysqld.conf /etc/supervisor/conf.d/supervisord-mysqld.conf

    # Remove pre-installed database
    RUN rm -rf /var/lib/mysql/*   # */ delete this comment

    # Add MySQL utils
    ADD create_mysql_admin_user.sh /create_mysql_admin_user.sh
    RUN chmod 755 /*.sh    # */ delete this comment

    # config to enable .htaccess
    ADD apache_default /etc/apache2/sites-available/000-default.conf
    RUN a2enmod rewrite

    # Configure /app folder with sample app
    RUN git clone https://github.com/V3ckt0r/hello-docker.git /app
    RUN mkdir -p /app && rm -fr /var/www/html && ln -s /app /var/www/html

    #Enviornment variables to configure php
    ENV PHP_UPLOAD_MAX_FILESIZE 10M
    ENV PHP_POST_MAX_SIZE 10M

    # Add volumes for MySQL 
    VOLUME  ["/etc/mysql", "/var/lib/mysql" ]

    EXPOSE 80 3306
    CMD ["/run.sh"]

I will only go through the noteworthy points here as I expect you to be able to understand the majority of what is going on here without explanations. So let’s start with that odd ENV param on line 4. This is specific to the Ubuntu base image, it’ll be a bit long winded and slightly off topic to talk about this in great detail. For now just accept that this needs to be there so avoid seeing some fairly ominous error messages later on! You can do some background reading on this here.
Next we install all our LAMP stack dependencies and then move onto the configuration. What we are doing here is, as before, using start up scripts to start our services. However we are using Supervisord to actually execute these start up scripts. Supervisord benefits us in this context as it watches our services and automatically starts them up for us on boot, or in the event that the application crashes.
Our Dockerfile then clears out any default database files that get installed then configures the Mysql admin user. Let’s take a quick look at it:

#!/bin/bash

/usr/bin/mysqld_safe > /dev/null 2>&1 &

RET=1
while [[ RET -ne 0 ]]; do
    echo "=> Waiting for confirmation of MySQL service startup"
    sleep 5
    mysql -uroot -e "status" > /dev/null 2>&1
    RET=$?
done

PASS=${MYSQL_PASS:-$(pwgen -s 12 1)}
_word=$( [ ${MYSQL_PASS} ] && echo "preset" || echo "random" )
echo "=> Creating MySQL admin user with ${_word} password"

mysql -uroot -e "CREATE USER 'admin'@'%' IDENTIFIED BY '$PASS'"
mysql -uroot -e "GRANT ALL PRIVILEGES ON *.* TO 'admin'@'%' WITH GRANT OPTION"


echo "=> Done!"

echo "========================================================================"
echo "You can now connect to this MySQL Server using:"
echo ""
echo "    mysql -uadmin -p$PASS -h<host> -P<port>"
echo ""
echo "Please remember to change the above password as soon as possible!"
echo "MySQL user 'root' has no password but only allows local connections"
echo "========================================================================"

mysqladmin -uroot shutdown

In order for us to create a set up the root account we need to bring up myqld, hence we begin with mysqld_safe.

Next we have a simple loop that checks for a successful startup of MySQL, while RET is not equals to 0 then echo a message sleep for 5 seconds and check the status of MySQL. We keep doing this loop until we get a success code back from $? ($? Returns the status code of the last command, so if MySql status is successful a code of 0 is returned). It is worth a mention that you would of noticed we are routing all output from this command to /dev/null, this is because it will look very confusing on your terminal when bundled together with your Docker logs.

Then we create the variable PASS for our randomly generated password and use this variable in MySql when we create the admin user. Once the grants are given we can echo out the information so it gets captured in the Docker logs. Lastly we shutdown MySql, don’t forget that we want Supervisord to handle the MySql process.
Back to the DockerFile, we get to the point of configuring Apache. If you followed the last post this part will look familiar to you. Our Apache vhost configuration is in the apache_default file. This is all standard stuff. With the below line we add this into the 000-default.conf. Please note – the 000-default file is specific to Apache installations on Ubuntu, you won’t find this on Fedora based systems.
ADD apache_default /etc/apache2/sites-available/000-default.conf
RUN a2enmod rewrite

The a2emod rewrite link can be skipped if you are not using rewriting in Apache, however might as well include it as this is a quiet common requirement.
Once our Apache config is set up we need supply our content. There are multiple ways you can include your content, the method I prefer is using a git repo to pull the content into your container, remember when we installed git back at the top of the DockerFile.

RUN git clone https://github.com/V3ckt0r/hello-docker.git /app
RUN mkdir -p /app && rm -fr /var/www/html && ln -s /app /var/www/html

Here we’re telling Docker to clone a repo from Github then place it in the directory /app. On the next line we remove all the default html content that comes with apache and we create a symlink to our content location /app. On this line I included the mkdir –p /app option, in case you don’t want to use a git repo. If you don’t want to use it you can remote to git clone line and put a ADD command after the second line to add local content.
System environments can be setup with the ENV tag as demonstrated below;

ENV PHP_UPLOAD_MAX_FILESIZE 10M
ENV PHP_POST_MAX_SIZE 10M

Docker volumes is a new concept we haven’t see yet. The VOLUME tag allows us to tell Docker about a resistant path we want to use for our application. When Docker builds a container from an image it builds it out of a series of read only layers, you see these layers when creating and pulling images. Once the container is fully initialised, then Docker adds a read/write layer on top of it all and that becomes your active container. The combination of layers is referred to as a Union File System.
When you want to have a location where you are able to share resources between different containers or have files accessible outside of the container, this is where Volume’s come in handy. Volumes essentially allow you to map host filesystem paths into your containers. So the line

VOLUME ["/etc/mysql", "/var/lib/mysql" ]

Is mapping out host location /etc/mysql on our container location /var/lib/mysql. In my build we aren’t using this VOLUME but I have included it to provide give you an example
Lastly we EXPOSE both ports 80 and 3306 for our Apache and MySql instances. Then start the run.sh, that does some validation checks on the database before executing Supervisord.
Now that we’ve gone through what the Docker file is actually doing. We are ready to run our container.
If you have built your Dockerfile yourself then make sure you build your image:

Docker build –t <username/name>

And run your container with:

Docker run –d –p 80:80 –p 3306:3306 <username/name>

Then just open up a web browser and check it out http://localhost/

Conclusion

We went through a lot in the post. I hope this has demonstrated a possible use case for Docker. In the article we have essentially containerised our web application for deployment. If you are in an environment where you are managing multiple different web applications, this method of bundling all your application dependencies into one container will make managing your Docker host much easier. Not to say that this is the way to go every time, this is just one road of many. Go find yours ;-)

Forever NPM module

So you've heard of the Forever NPM module right? Its a very simple node module that makes sure your NodeJS script runs.....forever! Check it out here.

Normal use case

Most of you will be familiar with the notion of starting your NodeJS applications via;

$node start app.js

and by extension using Forever to start your application;

$forever start app.js

Starting app using NPM

Those of you who are more experienced in the NodeJS world may prefer starting your NodeJS applications using package.json to configure your entry point and NPM start from your application directory;

$npm start

So to use Forever to start up your application is this manner, you can do;

$forever start -c "npm start" ./

Of course this command needs to be run from your application directory, or just specify it at the end. Success! Your application has been started in the background, and will run forever. Happy days!

Intro

Hi all, in this post I will show you an example of how you can use Docker to deliver a simple Apache http server for your website or application. I plan to make another posts that builds on top of this for a more advance use case.

So first a little introduction to what Docker is – I’ll be brief as there is plenty of information out there on it. Docker is a containerisation technology that allows you to wrap and enclose an entire service into a container for deployment on any Linux infrastructure without having to worry about the hosts configuration, setup, software dependencies, the lot. Docker Inc describes their product as

"Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in."

Prerequisites

To get started you will need to have Docker installed on your system, I am using CentOS, however any Fedora or Debian based system can be used. install instructions here.
You also need a good understanding of Apache.

Jump in

Feeling a little pro? Jump straight in via github.

Let’s Start

Ok, so we’re going to build an Apache webserver to serve up content, however that is really easy so I’ll make it a little bit more challenging by showing you how to set up vhosts, so that you can host more than one website from your Docker container.

First lets have a look at our Dockerfile;

FROM centos:latest
Maintainer vect0r
LABEL Vendor="CentOS"
RUN yum -y update && yum clean all
RUN yum -y install httpd && yum clean all
EXPOSE 80 81
#Simple startup script to aviod some issues observed with container restart
ADD run-httpd.sh /run-httpd.sh
RUN chmod -v +x /run-httpd.sh
#Copy config file across
COPY ./httpd.conf /etc/httpd/conf/httpd.conf
COPY ./example.com /var/www/example.com
COPY ./example2.com /var/www/example2.com
COPY ./sites-available /etc/httpd/sites-available
COPY ./sites-enabled /etc/httpd/sites-enabled

When Docker builds a container it builds it in layers. So when you go to create your Dockerfile you would see the essential stuff at the top and as you go down the file you will start to see more lower level configuration. Hence the first line. “FROM centos:latest” here we are telling Docker to go off and download the latest image of CentOS. This will be the base flavour our container uses.

“But I’m not using Centos.” I hear you say ^_^. It doesn't matter! As long as your using a Linux kernel, this will work. One of the beauties of Docker. Next we just add some fluffy meta data to our image. MAINTAINER AND LABEL. Maintainer would typically be your email address. In my case I've just added my tag. For the LABEL I've just specified CentOS as I'm using their base image. This interesting stuff starts happening at the RUN command. The RUN command shockingly tells Docker to run the subsequent command on the base image when building. Here we update all packages first then on the next line install Apache and clean yum.

Now at this point before we look at the rest let’s start looking at our Apache config, there will be subtle differences between Linux distros. I will follow CentOS’s style however I’m sure you can easily adapt this if your using Ubuntu. Right in our httpd.conf (apache.conf for Ubuntu) - we are mainly interested in setting our server to listen to ports 80 and 81, a port for our respective sites. You might want to also double check that sites-enable directory is also included. That’s it! The result of the defaults could be left as they are;

#make sure you have the below entries in your config #file.
Listen 80
Listen 81
IncludeOptional sites-enabled/*.conf

Inside our sites-enabled directory is where we’ll find our vhost.conf files for our two sites, in my case example.com.conf and example2.com.conf. You can specify this file however you want, example of mine looks like;

<VirtualHost *:80>
  ServerName www.example.com
  ServerAlias example.com
  DocumentRoot /var/www/example.com/public_html
  ScriptAlias /cgi-bin/ /var/www/cgi-bin/

<Directory "/var/cgi-bin">
  AllowOverride None
  Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
  Require all granted
</Directory>
    
  ScriptAlias /httpd/ /etc/httpd/

  ReWriteEngine on
  RewriteRule '/home/b/test'

<Location "/status">
  SetHandler server-status
  Order deny,allow
  Require all granted
  Allow from all
</Location>

  ErrorLog /var/www/example.com/error.log
  CustomLog /var/www/example.com/requests.log combined
</VirtualHost>

That’s all we need to worry about so now lets get back to our Dockerfile. So now that we've seen that Apache is listening to ports 80 and 81 we are now going to tell Docker to EXPOSE ports 80 and 81 on the docker container we build, as Docker doesn't do this automatically (that’s a good thing!).

EXPOSE 80 81

Next now that is done, you’ll see I’m using an ADD directive. This allows us to tell Docker to ADD a file from my desktop/host and put it into the container;

ADD <source-file-location> <container-file-location>

The source file goes into the container. So in my example run-httpd.sh is in the directory where I am running the build command and I’m placing the file on the root level of the container /. Important note about the ADD function is that it does not simply add local files. I can be used with URL supports certain compression types. See the Docker website for a more extensive explanation.

Let’s have a look at run-httpd.sh and why we need it;

#!/bin/bash

rm –rf /run/httpd/* /tmp/httpd*
exec /usr/sbin/apachectl –D FOREGROUND

So you might be wondering why we are removing files. This is simply to clean up the system in the event that you server crashed or was interrupted and didn’t shutdown properly. If that does happen and these directories aren’t cleaned out Apache will fail to start as the system will think it’s already running.

You can see here we are starting up Apache with its cli command as oppose to using the more conventional service or systemctl. This is more of a best practice convention if anything. There is nothing stopping you from using these utilities, however with all the different base images out there some may not have these services available by default!
Another point to highlight here is that we need to start Apache in the FOREGROUND, this may sound unconventional as Apache is usually started as a background daemon. This is done because of the way Docker works. When you run a Docker container, the containers only remain up for the lifetime of the command you give it. This is determined by the stdout output. So if we started Apache in the usual manner, as a daemon, then our container would start then immediately stop as it would think its purpose is complete.

Our next command we give Docker is RUN, here we are telling Docker to run a command at build time. In our example we are simply making the script we just added executable;

RUN chmod -v +x /run-httpd.sh

On to the COPY function lines. COPY is similar to the ADD function in that it allows use to place files in our container. However COPY isn’t as sophisticated as ADD, copy simply puts local files into your container. Why not use ADD? Well COPY is just more transparent in its functionality so it’s usually preferred when copying config files.

COPY ./httpd.conf /etc/httpd/conf/httpd.conf
COPY ./example.com /var/www/example.com
COPY ./example2.com /var/www/example2.com
COPY ./sites-available /etc/httpd/sites-available
COPY ./sites-enabled /etc/httpd/sites-enabled

I have taken the apache files we look at earlier and placed them in the relevant places.
Last but not least. CMD this is going to execute our run-httpd.sh script once all the above is executed. You have just completed a Dockerfile – was easy huh!

In action

Let’s see this in container in action. First we need to build this sucker, make sure your in run the below command in the directory your Dockerfile is located;

docker build –rm –t <username>/httpd-proxy .

You can call your image whatever you want if you don’t like httpd-proxy. When you run this you will see a bunch of stuff output to your console, this is Docker building your container in the layered instructions we gave it. Once that’s finished if you type;

docker images

You will see the image you’ve just created listed. Once this is built you can use this image to create your container. Now to create the container we also need to bind our host ports to our container, otherwise our web request won’t go through! We can do this as part of our run command like so;

docker run –d –p 80-81:80-81 <username>/httpd-proxy

Here we are telling docker to create a container the –d tell its to daemonise the container and the –p is binding the ports. We are saying bind our host ports 80-81 to our containers ports 80-81. Then we are simply giving the name of the image we want to use. When our container is build it will be given a random name like “firey-scotty”, if we want to specify a name yourself we can use the --name option. In my example I’m happy with a random name. Once this command is executed you can see the container running via;

docker ps

Now you can open your browser and navigate to your sites via http://localhost and http://localhost:81

Conclusion

To conclude, we have gone through setting up, building, and then running our Docker container. I hope this demonstrates and makes you think about the fun that could be had with Docker.
Sorry if this went on for too long. Please leave a comment if this helped you, if you have any questions, or if you have any recommendations. I look forward to hearing from you.