How to Get Secure Webhook for SUSI Bots in Kubernetes Deployment

Webhook is a user-defined callback which gets triggered by any events in code like receiving a message from a user in SUSI bot is an event. Few bots need webhook URI for callback like in SUSI Viber bot we need to define a webhook URI in the code to receive callbacks and make our Viber bot work. In this blog, we will learn how can we get an SSL activated webhook while deploying our bot to Google container using Kubernetes. We will generate SSL certificate using kube lego service that is included in kubernetes and you will define that in yaml files below. We can also generate SSL certificate using third party services like CloudFlare but by using it we will be dependant on CloudFlare so we will use kube lego.

We will start off by registering a domain first on which we will activate SSL certificate and use that domain as a webhook. Go to freenom and register your account. After logging in, register a free domain of any name and check out that order. Next, you have to set IP for DNS of this domain. To do so we will reserve an IP address in our Google cloud project with this command:

gcloud compute addresses create IPname --region us-central1

You will get a created message. To see your IP go to VPC Network -> External IP addresses. Add this IP to DNS zone of your domain and save it for later use in yaml files that we will use for deployment. Now we will deploy our bot using yaml files but before deployment, we will create a cluster

gcloud container clusters create clusterName

After creating cluster add these yaml files to your bot repository and add your IP address that you have saved above to the yamls/nginx/service.yaml file for “loadBalancerIP” parameter. Replace domain name in yamls/application/ingress-notls.yaml and yamls/application/ingress-tls.yaml with your domain name that you have registered already. Add your email ID to yamls/lego/configmap.yaml for “lego.email” parameter. Replace “image” and “env” parameters in yamls/application/deployment.yaml with your docker image and your environment variables that you are using in your code. After changing yaml files we will use this deploy script to create a deployment. Change paths for yaml files in script according to your yaml files path.

In gcloud shell run the following command to deploy an application using given configurations.

bash ./path-to-deploy-script/deploy.sh create all

This will create the deployment as we have defined in the script. The Kubernetes master creates the load balancer and related Compute Engine forwarding rules, target pools, and firewall rules to make the service fully accessible from outside of Google Cloud Platform. Wait for a few minutes for all the containers to be created and the SSL Certificates to be generated and loaded.

You have successfully created a secure webhook. Test it by opening the domain that you have registered at the start.

Resources

Enabling SSL using CloudFlare: https://jonnyjordan.com/blog/how-to-setup-cloudflare-flexible-ssl-for-wordpress/
https://www.youtube.com/watch?v=qFvwEVkl5gk

Persistently Storing loklak Server Dumps on Kubernetes

In an earlier blog post, I discussed loklak setup on Kubernetes. The deployment mentioned in the post was to test the development branch. Next, we needed to have a deployment where all the messages are collected and dumped in text files that can be reused.

In this blog post, I will be discussing the challenges with such deployment and the approach to tackle them.

Volatile Disk in Kubernetes

The pods that hold deployments in Kubernetes have disk storage. Any data that gets written by the application stays only until the same version of deployment is running. As soon as the deployment is updated/relocated, the data stored during the application is cleaned up.


Due to this, dumps are written when loklak is running but they get wiped out when the deployment image is updated. In other words, all dumps are lost when the image updates. We needed to find a solution to this as we needed a permanent storage when collecting dumps.

Persistent Disk

In order to have a storage which can hold data permanently, we can mount persistent disk(s) on a pod at the appropriate location. This ensures that the data that is important to us stays with us, even
when the deployment goes down.


In order to add persistent disks, we first need to create a persistent disk. On Google Cloud Platform, we can use the gcloud CLI to create disks in a given region –

gcloud compute disks create --size=<required size> --zone=<same as cluster zone> <unique disk name>

After this, we can mount it on a Docker volume defined in Kubernetes configurations –

      ...
      volumeMounts:
        - mountPath: /path/to/mount
          name: volume-name
  volumes:
    - name: volume-name
      gcePersistentDisk:
        pdName: disk-name
        fsType: fileSystemType

But this setup can’t be used for storing loklak dumps. Let’s see “why” in the next section.

Rolling Updates and Persistent Disk

The Kubernetes deployment needs to be updated when the master branch of loklak server is updated. This update of master deployment would create a new pod and try to start loklak server on it. During all this, the older deployment would also be running and serving the requests.


The control will not be transferred to the newer pod until it is ready and all the probes are passing. The newer deployment will now try to mount the disk which is mentioned in the configuration, but it would fail to do so. This would happen because the older pod has already mounted the disk.


Therefore, all new deployments would simply fail to start due to insufficient resources. To overcome such issues, Kubernetes allows persistent volume claims. Let’s see how we used them for loklak deployment.

Persistent Volume Claims

Kubernetes provides Persistent Volume Claims which claim resources (storage) from a Persistent Volume (just like a pod does from a node). The higher level APIs are provided by Kubernetes (configurations and kubectl command line). In the loklak deployment, the persistent volume is a Google Compute Engine disk –

apiVersion: v1
kind: PersistentVolume
metadata:
  name: dump
  namespace: web
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: slow
  gcePersistentDisk:
    pdName: "data-dump-disk"
    fsType: "ext4"

[SOURCE]

It must be noted here that a persistent disk by the name of data-dump-index is already created in the same region.


The storage class defines the way in which the PV should be handled, along with the provisioner for the service –

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
  namespace: web
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  zone: us-central1-a

[SOURCE]

After having the StorageClass and PersistentVolume, we can create a claim for the volume by using appropriate configurations –

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dump
  namespace: web
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: slow

[SOURCE]

After this, we can mount this claim on our Deployment –

  ...
  volumeMounts:
    - name: dump
      mountPath: /loklak_server/data
volumes:
  - name: dump
    persistentVolumeClaim:
      claimName: dump

[SOURCE]

Verifying persistence of Dumps

To verify this, we can redeploy the cluster using the same persistent disk and check if the earlier dumps are still present there –

$ http http://link.to.deployment/dump/
HTTP/1.1 200 OK
Cache-Control: public, max-age=60
Content-Type: text/html;charset=utf-8
...


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

<h1>Index of /dump</h1>
<pre>      Name 
[gz ] <a href="messages_20170802_71562040.txt.gz">messages_20170802_71562040.txt.gz</a>   Thu Aug 03 00:07:21 GMT 2017   132M
[gz ] <a href="messages_20170803_69925009.txt.gz">messages_20170803_69925009.txt.gz</a>   Mon Aug 07 15:40:04 GMT 2017   532M
[gz ] <a href="messages_20170807_36357603.txt.gz">messages_20170807_36357603.txt.gz</a>   Wed Aug 09 10:26:24 GMT 2017   377M
[txt] <a href="messages_20170809_27974404.txt">messages_20170809_27974404.txt</a>      Thu Aug 10 08:51:49 GMT 2017  1564M
<hr></pre>
...

Conclusion

In this blog post, I discussed the process of deployment of loklak with persistent dumps on Kubernetes. This deployment is intended to work as root.loklak.org in near future. The changes were proposed in loklak/loklak_server#1377 by @singhpratyush (me).

Resources