Kubernetes

Read more about the article Deploying a Postgres-Based Open Event Server to Kubernetes

Deploying a Postgres-Based Open Event Server to Kubernetes

In this post, I will walk you through deploying the Open Event Server on Kubernetes, hosted on Google Cloud Platform’s Compute Engine. You’ll be needing a Google account for this, so create one if you don’t have one. First, I cd into the root of our project’s Git repository. Now I need to create a Dockerfile. I will use Docker to package our project into a nice image which can be then be “pushed” to Google Cloud Platform. A Dockerfile is essentially a text doc which simply contains the commands required to assemble an image. For more details on how to write one for your project specifically, check out Docker docs. For Open Event Server, the Dockerfile looks like the following: FROM python:3-slim ENV INSTALL_PATH /open_event RUN mkdir -p $INSTALL_PATH WORKDIR $INSTALL_PATH # apt-get update and update some packages RUN apt-get update && apt-get install -y wget git ca-certificates curl && update-ca-certificates && apt-get clean -y # install deps RUN apt-get install -y --no-install-recommends build-essential python-dev libpq-dev libevent-dev libmagic-dev && apt-get clean -y # copy just requirements COPY requirements.txt requirements.txt COPY requirements requirements # install requirements RUN pip install --no-cache-dir -r requirements.txt RUN pip install eventlet # copy remaining files COPY . . CMD bash scripts/docker_run.sh These commands simply install the dependencies and set up the environment for our project. The final CMD command is for running our project, which, in our case, is a server. After our Dockerfile is configured, I go to Google Cloud Platform’s console and create a new project: Once I enter the product name and other details, I enable billing in order to use Google’s cloud resources. A credit card is required to set up a billing account, but Google doesn’t charge any money for that. Also, one of the perks of being a part of FOSSASIA was that I had about $3000 in Google Cloud credits! Once billing is enabled, I then enable the Container Engine API. It is required to support Kubernetes on Google Compute Engine. Next step is to install Google Cloud SDK. Once that is done, I run the following command to install Kubernetes CLI tool: gcloud components install kubectl Then I configure the Google Cloud Project Zone via the following command: gcloud config set compute/zone us-west1-a Now I will create a disk (for storing our code and data) as well as a temporary instance for formatting that disk: gcloud compute disks create pg-data-disk --size 1GB gcloud compute instances create pg-disk-formatter gcloud compute instances attach-disk pg-disk-formatter --disk pg-data-disk Once the disk is attached to our instance, I SSH into it and list the available disks: gcloud compute ssh "pg-disk-formatter" Now, ls the available disks: ls /dev/disk/by-id This will list multiple disks (as shown in the Terminal window below), but the one I want to format is "google-persistent-disk-1". Now I format that disk via the following command: sudo mkfs.ext4 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/disk/by-id/google-persistent-disk-1 Finally, after the formatting is done, I exit the SSH session and detach the disk from the instance: gcloud…

Continue ReadingDeploying a Postgres-Based Open Event Server to Kubernetes
Read more about the article Setting Loklak Server with SSL

Setting Loklak Server with SSL

Loklak Server is based on embedded Jetty Server which can work both with or without SSL encryption. Lately, there was need to setup Loklak Server with SSL. Though the need was satisfied by CloudFlare. Alternatively, there are 2 ways to set up Loklak Server with SSL. They are:- 1) Default Jetty Implementation There is pre-existing implementation of Jetty libraries. The http mode can be set in configuration file. There are 4 modes on which Loklak Server can work: http mode, https mode, only https mode and redirect to https mode. Loklak Server listens to port 9000 when in http mode and to port 9443 when in https mode. There is also a need of SSL certificate which is to be added in configuration file. 2) Getting SSL certificate with Kube-Lego on Kubernetes Deployment I got to know about Kube-Lego by @niranjan94. It is implemented in Open-Event-Orga-Server. The approach is to use: a) Nginx as ingress controller For setting up Nginx ingress controller, a yml file is needed which downloads and configures the server. The configurations for data requests and response are: proxy-connect-timeout: "15" proxy-read-timeout: "600" proxy-send-imeout: "600" hsts-include-subdomains: "false" body-size: "64m" server-name-hash-bucket-size: "256" server-tokens: "false" Nginx is configured to work on both http and https ports in service.yml ports: - port: 80 name: http - port: 443 name: https   b) Kube-Lego for fetching SSL certificates from Let's Encrypt Kube-Lego was set up with default values in yml. It uses the host-name, email address and secretname of the deployment to validate url and fetch SSL certificate from Let's Encrypt. c) Setup configurations related to TLS and no-TLS connection These configuration files mentions the path and service ports for Nginx Server through which requests are forwarded to backend Loklak Server. Here for no-TLS and TLS requests, the requests are directly forwarded to localhost at port 80 of Loklak Server container. rules: - host: staging.loklak.org http: paths: - path: / backend: serviceName: server servicePort: 80 For TLS requests, the secret name is also mentioned. Kube-Lego fetches host-name and secret-name from here for the certificate tls: - hosts: - staging.loklak.org secretName: loklak-api-tls d) Loklak Server, ElasticSearch and Mosquitto at backend These containers work at backend. ElasticSearch and Mosquitto are only accessible to Loklak Server. Loklak Server can be accessed through Nginx server. Loklak Server is configured to work at http mode and is exposed at port 80. ports: - port: 80 protocol: TCP targetPort: 80 To deploy the Loklak Server, all these are deployed in separate pods and they interact through service ports. To deploy, we use deploy script: # For elasticsearch, accessible only to api-server kubectl create -R -f ${path-to-config-file}/elasticsearch/ # For mqtt, accessible only to api-server kubectl create -R -f ${path-to-config-file}/mosquitto/ # Start KubeLego deployment for TLS certificates kubectl create -R -f ${path-to-config-file}/lego/ kubectl create -R -f ${path-to-config-file}/nginx/ # Create web namespace, this acts as bridge to Loklak Server kubectl create -R -f ${path-to-config-file}/web/ # Create API server deployment and expose the services kubectl create -R -f ${path-to-config-file}/api-server/ # Get the…

Continue ReadingSetting Loklak Server with SSL
Read more about the article How to Debug SUSI Bots Deployment on Google Container Engine Using Kubernetes

How to Debug SUSI Bots Deployment on Google Container Engine Using Kubernetes

You can learn how to deploy SUSI bots on to Google container engine using Kubernetes from this tutorial. In this blog, we will learn how to debug SUSI bots deployment to keep them running continuously. Whenever we create a deployment on Google container using Kubernetes a pod is created in which our deployment keeps running. Whenever we deploy bots to any platform to check if it is working right or not we refer to logs of that bot. To get logs first we will get pod for our deployment with this kubectl get pods --namespace={your-namespace-of-deployment-here} This will show us the pod for our deployment like this Copy the name of this pod and enter this command to get logs kubectl get logs {your-pod-here} --namespace={your-namespace-of-deployment-here} This will show us the logs of our deployment. In Google cloud console you will not get running logs. You will get logs of the everything that has happened before you requested for logs. Now if there is some error in logs and you need to restart the deployment but in Kubernetes you can not restart your pod directly but to restart pod we will need to enter the following command kubectl replace --force -f {path-to-your-deployment-config-file} If everything goes well you will get to see the following with your deployment name in it After deployment, if you want to see the services and deployment in detail follow the approach given below To get services write this command kubectl get service --namespace={your-namespace-of-deployment-here} When you will get service it will look like If you don’t get your external IP then check your service config file and after fixing it make a new deployment after deleting previous one. To check deployment in detail write following command kubectl describe deployments --namespace={your-namespace-of-deployment-here} This will show us details about deployment like this You can now easily solve issues with deployments now. Resources Debugging Kubernetes service locally using telepresence: https://www.telepresence.io/tutorials/kubernetes.html Debug Services: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ Troubleshooting Kuberetes: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/getting_started_with_kubernetes/troubleshooting_kubernetes  

Continue ReadingHow to Debug SUSI Bots Deployment on Google Container Engine Using Kubernetes
Read more about the article

Continuous Integration and Deployment of Yacy Grid We have deployed Yacy Grid on Google cloud recently, and we have achieved this using kubernetes and Travis for auto deployment. How we have deployed it: Firstly, it is advised to have different containers for each service your application requires, and follow a multi container architecture. Using multi container architecture you can allocate fixed size of power to each application and also replicate individual services, whichever is required. Presently, Yacy has two main applications which are required to be deployed in separate containers - Yacy_grid_mcp and ElasticSearch. We took the official kubernetes YAML files of ElasticSearch and followed the instructions at https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md for deployment of elastic search on the google cloud. With this we are able to run pods, volumes required for elastic search and services for connecting Yacy with elastic search. The pull request regarding deployment of separate elasticsearch component is at https://github.com/yacy/yacy_grid_mcp/pull/27/files Below figure shows different services and external endpoints present pods use for elastic search. Now elastic search can be accessed at 35.202.154.219:9300 and http://35.193.124.253:9200/ Continuous deployment of Yacy_grid_mcp: Please make sure that you have created a cluster on google container engine for deploying our containers on it. Regarding starting a project and cluster please read https://cloud.google.com/container-engine/docs/ 1.Initially, Travis.yml initiates and sets up the required environment for Yacy deployment by installing Google cloud cli and kubectl components. Source code regarding the Travis setup could be found at https://github.com/yacy/yacy_grid_mcp/blob/master/.travis.yml 2.Later Travis runs the depoy_staging.sh file, which builds the docker image of yacy o the present build and pushes it to hub.docker.com if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then echo "Skipping deploy; The request or commit is not on master" exit 0 fi set -e docker build -t nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT ./docker docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD" docker tag nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT nikhilrayaprolu/yacygridmcp:latest docker push nikhilrayaprolu/yacygridmcp Later with service key, we authenticate with google cloud and set the required environments and variables echo $GCLOUD_SERVICE base64 --decode -i > ${HOME}/gcloud-service-key.json gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json gcloud --quiet config set project $PROJECT_NAME_STG gcloud --quiet config set container/cluster $CLUSTER_NAME_STG gcloud --quiet config set compute/zone ${CLOUDSDK_COMPUTE_ZONE} gcloud --quiet container clusters get-credentials $CLUSTER_NAME_STG And Later we push the docker image built to google cloud and deploy it kubectl config view kubectl config current-context kubectl set image deployment/${KUBE_DEPLOYMENT_NAME} ${KUBE_DEPLOYMENT_CONTAINER_NAME}=nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT Presently Yacy runs on 5vCPUs With the following pods and services: Also one can use kubectl cli for getting information regarding the cluster and pods as shown below Pull request regarding deployment of yacy on google cloud is available at: https://github.com/yacy/yacy_grid_mcp/pull/16/files References: 1.A Medium Blog on CD to Google Container: https://medium.com/google-cloud/continuous-delivery-in-a-microservice-infrastructure-with-google-container-engine-docker-and-fb9772e81da7 2.Another Blog on CD to Google Container: https://engineering.hexacta.com/automatic-deployment-of-multiple-docker-containers-to-google-container-engine-using-travis-e5d9e191d5ad 3.Deploying ElasticSearch to Cloud using Kubernetes: https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md

Continue Reading
Read more about the article How to Get Secure Webhook for SUSI Bots in Kubernetes Deployment

How to Get Secure Webhook for SUSI Bots in Kubernetes Deployment

Webhook is a user-defined callback which gets triggered by any events in code like receiving a message from a user in SUSI bot is an event. Few bots need webhook URI for callback like in SUSI Viber bot we need to define a webhook URI in the code to receive callbacks and make our Viber bot work. In this blog, we will learn how can we get an SSL activated webhook while deploying our bot to Google container using Kubernetes. We will generate SSL certificate using kube lego service that is included in kubernetes and you will define that in yaml files below. We can also generate SSL certificate using third party services like CloudFlare but by using it we will be dependant on CloudFlare so we will use kube lego. We will start off by registering a domain first on which we will activate SSL certificate and use that domain as a webhook. Go to freenom and register your account. After logging in, register a free domain of any name and check out that order. Next, you have to set IP for DNS of this domain. To do so we will reserve an IP address in our Google cloud project with this command: gcloud compute addresses create IPname --region us-central1 You will get a created message. To see your IP go to VPC Network -> External IP addresses. Add this IP to DNS zone of your domain and save it for later use in yaml files that we will use for deployment. Now we will deploy our bot using yaml files but before deployment, we will create a cluster gcloud container clusters create clusterName After creating cluster add these yaml files to your bot repository and add your IP address that you have saved above to the yamls/nginx/service.yaml file for “loadBalancerIP” parameter. Replace domain name in yamls/application/ingress-notls.yaml and yamls/application/ingress-tls.yaml with your domain name that you have registered already. Add your email ID to yamls/lego/configmap.yaml for “lego.email” parameter. Replace “image” and “env” parameters in yamls/application/deployment.yaml with your docker image and your environment variables that you are using in your code. After changing yaml files we will use this deploy script to create a deployment. Change paths for yaml files in script according to your yaml files path. In gcloud shell run the following command to deploy an application using given configurations. bash ./path-to-deploy-script/deploy.sh create all This will create the deployment as we have defined in the script. The Kubernetes master creates the load balancer and related Compute Engine forwarding rules, target pools, and firewall rules to make the service fully accessible from outside of Google Cloud Platform. Wait for a few minutes for all the containers to be created and the SSL Certificates to be generated and loaded. You have successfully created a secure webhook. Test it by opening the domain that you have registered at the start. Resources Enabling SSL using CloudFlare: https://jonnyjordan.com/blog/how-to-setup-cloudflare-flexible-ssl-for-wordpress/ https://www.youtube.com/watch?v=qFvwEVkl5gk

Continue ReadingHow to Get Secure Webhook for SUSI Bots in Kubernetes Deployment
Read more about the article Optimising Docker Images for loklak Server

Optimising Docker Images for loklak Server

The loklak server is in a process of moving to Kubernetes. In order to do so, we needed to have different Docker images that suit these deployments. In this blog post, I will be discussing the process through which I optimised the size of Docker image for these deployments. Initial Image The image that I started with used Ubuntu as base. It installed all the components needed and then modified the configurations as required - FROM ubuntu:latest # Env Vars ENV LANG=en_US.UTF-8 ENV JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 ENV DEBIAN_FRONTEND noninteractive WORKDIR /loklak_server RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y git openjdk-8-jdk RUN git clone https://github.com/loklak/loklak_server.git /loklak_server RUN git checkout development RUN ./gradlew build -x test -x checkstyleTest -x checkstyleMain -x jacocoTestReport RUN sed -i.bak 's/^\(port.http=\).*/\180/' conf/config.properties ... # More configurations RUN echo "while true; do sleep 10;done" >> bin/start.sh # Start CMD ["bin/start.sh", "-Idn"] The size of images built using this Dockerfile was quite huge - REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE loklak_server       latest              a92f506b360d        About a minute ago   1.114 GB ubuntu              latest              ccc7a11d65b1        3 days ago           120.1 MB But since this size is not acceptable, we needed to reduce it. Moving to Apline Alpine Linux is an extremely lightweight Linux distro, built mainly for the container environment. Its size is so tiny that it hardly puts any impact on the overall size of images. So, I replaced Ubuntu with Alpine - FROM alpine:latest ... RUN apk update RUN apk add git openjdk8 bash ... And now we had much smaller images - REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE loklak_server       latest              54b507ee9187        17 seconds ago      668.8 MB alpine              latest              7328f6f8b418        6 weeks ago         3.966 MB As we can see that due to no caching and small size of Alpine, the image size is reduced to almost half the original. Reducing Content Size There are many things in a project which are no longer needed while running the project, like the .git folder (which is huge in case of loklak) - $ du -sh loklak_server/.git 236M loklak_server/.git We can remove such files from the Docker image and save a lot of space - rm -rf .[^.] .??* Optimizing Number of Layers The number of layers also affect the size of the image. More the number of layers, more will be the size of image. In the Dockerfile, we can club together the RUN commands for lower number of images. RUN apk update && apk add openjdk8 git bash && \ git clone https://github.com/loklak/loklak_server.git /loklak_server && \ ... After this, the effective size is again reduced by a major factor - REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE loklak_server       latest              54b507ee9187        17 seconds ago      422.3 MB alpine              latest              7328f6f8b418        6 weeks ago         3.966 MB Conclusion In this blog post, I discussed the process of optimising the size of Dockerfile for Kubernetes deployments of loklak server. The size was reduced to 426 MB from 1.234 GB and this provided much faster push/pull time for Docker images, and therefore, faster updates for Kubernetes deployments.…

Continue ReadingOptimising Docker Images for loklak Server
Read more about the article How to Deploy Node js App to Google Container Engine Using Kubernetes

How to Deploy Node js App to Google Container Engine Using Kubernetes

There are many ways to host node js apps. A popular way is to host your app on Heroku. We can also deploy our app to Google cloud platform on container engine using Kubernetes. In this blog, we will learn how to deploy node js app on container engine with kubernetes. There are many resources on the web but we will use YAML files to create a deployment and to build docker image we will not use Google container registry (GCR) as it will cost us more for the deployment. To deploy we will start by creating an account on Google cloud and you can get a free tier of Google cloud platform worth 300$ for 12 months. After creating account create a project with any name of your choice. Enable Google cloud shell from an icon on right top. We will also need a docker image of our app for deployment. To create a docker image first create an account on https://www.docker.com and create a repository with any name on it. Now, we will add docker file into our repository so that we can build docker image. Dockerfile will contain this code: FROM node:boron # Create app directory RUN mkdir -p /usr/src/app WORKDIR /usr/src/app # Install app dependencies COPY package.json /usr/src/app RUN npm install # Bundle app source COPY . /usr/src/app EXPOSE 8080 CMD [ "npm", "start" ] You can see an example of docker file in SUSI telegram repository. After pushing docker file to your repository we will now build docker image of the app. In Google cloud shell clone, your repository with git clone {your-repository-link here }and change your current directory to the cloned app. We will use --no-cache -t arguments as described above it will be better for building docker image and it will use fewer resources. Run these two commands to build docker image and pushing it to your docker hub. docker build --no-cache -t {your-docker-username-here}/{repository-name-on-docker here} . docker push {your-docker-username-here}/{repository-name-on-docker here} We have successfully created docker image for our app. Now we will deploy our app to container engine using this image. To deploy it we will use configuration files. Add a yaml folder in your repository and we will add four files into it now. In the first file, we will specify the namespace and name it as 00-namespace.yml It will contain following code: apiVersion: v1 kind: Namespace metadata:  name: web In second file we will configure our namespace that we specified and name it as configmap.yml It will contain following code: apiVersion: v1 metadata:  name:{name-of-your-deployment-here}  namespace: web kind: ConfigMap In third file, we will define our deployment and name it as deployment.yml It will contain following code: kind: Deployment apiVersion: apps/v1beta1 metadata:  name: {name-of-your-deployment-here}  namespace: web spec:  replicas: 1  template:    metadata:      labels:        app: {name-of-your-deployment-here}    spec:      containers:      - name: {name-of-your-deployment-here}        image: {your-docker-username-here}/{repository-name-on-docker here}:latest        ports:        - containerPort: 8080          protocol: TCP        envFrom:        - configMapRef:            name: {name-of-your-deployment-here}      restartPolicy: Always In fourth file, we will define service for our deployment and name it as service.yml It will…

Continue ReadingHow to Deploy Node js App to Google Container Engine Using Kubernetes
Read more about the article Auto Deploying loklak Server on Google Cloud Using Travis

Auto Deploying loklak Server on Google Cloud Using Travis

This is a setup for loklak server which want to check in only the source files, but have the development branch in Kubernetes deployment automatically updated with some compiled output every time the push using details from Travis build. How to achieve it? Unix commands and shell script is one of the best option to automate all deployment and build activities. I explored Kubernetes Gcloud which can be accessed through unix command. 1.Checking for Travis build details before deployment: Firstly check whether the repository is loklak_server, pull request is available and branches are either master or development, and then decide to update the docker image or not. The code of the aforementioned things is as follows: if [ "$TRAVIS_REPO_SLUG" != "loklak/loklak_server" ]; then echo "Skipping image update for repo $TRAVIS_REPO_SLUG" exit 0 fi if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then echo "Skipping image update for pull request" exit 0 fi if [ "$TRAVIS_BRANCH" != "master" ] && [ "$TRAVIS_BRANCH" != "development" ]; then echo "Skipping image update for branch $TRAVIS_BRANCH" exit 0 fi 2. Setting up Tag and Decrypting the credentials: For the Kubernetes deployment, each time the travis build is successful, it takes the commit details from travis and appended into tag details for deployment and gcloud credentials is decrypted from the json file. openssl aes-256-cbc -K $encrypted_48d01dc243a6_key -iv $encrypted_48d01dc243a6_iv -in kubernetes/gcloud-credentials.json.enc -out kubernetes/gcloud-credentials.json -d 3. Install, Authenticate and Configure GCloud details with Kubernetes: In this step, initially Google Cloud SDK should be installed with Kubernetes- curl https://sdk.cloud.google.com | bash > /dev/null source ~/google-cloud-sdk/path.bash.inc gcloud components install kubectl   Then, Authenticate Google Cloud using the above mentioned decrypted credentials and finally configure the Google Cloud with the details like zone, project name, cluster details, number of nodes etc. 4. Update the Kubernetes deployment: Since, in this issue it is specific to the loklak_server/development branch, so in here it checks if the branch is development or not and then updates the deployment using following command: if [ $TRAVIS_BRANCH == "development" ]; then kubectl set image deployment/server --namespace=web server=$TAG fi   Conclusion: In this post, how to write a script in such a way that with each successful push after travis build how to update the deployment on Kubernetes GCloud. Resources: Read more about Kubernetes GCloud deployment here: http://thylong.com/ci/2016/deploying-from-travis-to-gce/ Documentation available on Kubernetes: https://kubernetes.io/docs/tutorials/

Continue ReadingAuto Deploying loklak Server on Google Cloud Using Travis
Read more about the article Persistently Storing loklak Server Dumps on Kubernetes

Persistently Storing loklak Server Dumps on Kubernetes

In an earlier blog post, I discussed loklak setup on Kubernetes. The deployment mentioned in the post was to test the development branch. Next, we needed to have a deployment where all the messages are collected and dumped in text files that can be reused. In this blog post, I will be discussing the challenges with such deployment and the approach to tackle them. Volatile Disk in Kubernetes The pods that hold deployments in Kubernetes have disk storage. Any data that gets written by the application stays only until the same version of deployment is running. As soon as the deployment is updated/relocated, the data stored during the application is cleaned up. Due to this, dumps are written when loklak is running but they get wiped out when the deployment image is updated. In other words, all dumps are lost when the image updates. We needed to find a solution to this as we needed a permanent storage when collecting dumps. Persistent Disk In order to have a storage which can hold data permanently, we can mount persistent disk(s) on a pod at the appropriate location. This ensures that the data that is important to us stays with us, even when the deployment goes down. In order to add persistent disks, we first need to create a persistent disk. On Google Cloud Platform, we can use the gcloud CLI to create disks in a given region - gcloud compute disks create --size=<required size> --zone=<same as cluster zone> <unique disk name> After this, we can mount it on a Docker volume defined in Kubernetes configurations - ... volumeMounts: - mountPath: /path/to/mount name: volume-name volumes: - name: volume-name gcePersistentDisk: pdName: disk-name fsType: fileSystemType But this setup can’t be used for storing loklak dumps. Let’s see “why” in the next section. Rolling Updates and Persistent Disk The Kubernetes deployment needs to be updated when the master branch of loklak server is updated. This update of master deployment would create a new pod and try to start loklak server on it. During all this, the older deployment would also be running and serving the requests. The control will not be transferred to the newer pod until it is ready and all the probes are passing. The newer deployment will now try to mount the disk which is mentioned in the configuration, but it would fail to do so. This would happen because the older pod has already mounted the disk. Therefore, all new deployments would simply fail to start due to insufficient resources. To overcome such issues, Kubernetes allows persistent volume claims. Let’s see how we used them for loklak deployment. Persistent Volume Claims Kubernetes provides Persistent Volume Claims which claim resources (storage) from a Persistent Volume (just like a pod does from a node). The higher level APIs are provided by Kubernetes (configurations and kubectl command line). In the loklak deployment, the persistent volume is a Google Compute Engine disk - apiVersion: v1 kind: PersistentVolume metadata: name: dump namespace: web spec: capacity: storage:…

Continue ReadingPersistently Storing loklak Server Dumps on Kubernetes
Read more about the article Using Mosquitto as a Message Broker for MQTT in loklak Server

Using Mosquitto as a Message Broker for MQTT in loklak Server

In loklak server, messages are collected from various sources and indexed using Elasticsearch. To know when a message of interest arrives, users can poll the search endpoint. But this method would require a lot of HTTP requests, most of them being redundant. Also, if a user would like to collect messages for a particular topic, he would need to make a lot of requests over a period of time to get enough data. For GSoC 2017, my proposal was to introduce stream API in the loklak server so that we could save ourselves from making too many requests and also add many use cases. Mosquitto is Eclipse’s project which acts as a message broker for the popular MQTT protocol. MQTT, based on the pub-sub model, is a lightweight and IOT friendly protocol. In this blog post, I will discuss the basic setup of Mosquitto in the loklak server. Installation and Dependency for Mosquitto The installation process of Mosquitto is very simple. For Ubuntu, it is available from the pre installed PPAs - sudo apt-get install mosquitto Once the message broker is up and running, we can use the clients to connect to it and publish/subscribe to channels. To add MQTT client as a project dependency, we can introduce following line in Gradle dependencies file - compile group: 'net.sf.xenqtt', name: 'xenqtt', version: '0.9.5' [SOURCE] After this, we can use the client libraries in the server code base. The MQTTPublisher Class The MQTTPublisher class in loklak would provide an interface to perform basic operations in MQTT. The implementation uses AsyncClientListener to connect to Mosquitto broker - AsyncClientListener listener = new AsyncClientListener() { // Override methods according to needs }; [SOURCE] The publish method for the class can be used by other components of the project to publish messages on the desired channel - public void publish(String channel, String message) { this.mqttClient.publish(new PublishMessage(channel, QoS.AT_LEAST_ONCE, message)); } [SOURCE] We also have methods which allow publishing of multiple messages to multiple channels in order to increase the functionality of the class. Starting Publisher with Server The flags which signal using of streaming service in loklak are located in conf/config.properties. These configurations are referred while initializing the Data Access Object and an MQTTPublisher is created if needed - String mqttAddress = getConfig("stream.mqtt.address", "tcp://127.0.0.1:1883"); streamEnabled = getConfig("stream.enabled", false); if (streamEnabled) { mqttPublisher = new MQTTPublisher(mqttAddress); } [SOURCE] The mqttPublisher can now be used by other components of loklak to publish messages to the channel they want. Adding Mosquitto to Kubernetes Since loklak has also a nice Kubernetes setup, it was very simple to introduce a new deployment for Mosquitto to it. Changes in Dockerfile The Dockerfile for master deployment has to be modified to discover Mosquitto broker in the Kubernetes cluster. For this purpose, corresponding flags in config.properties have to be changed to ensure that things work fine - sed -i.bak 's/^\(stream.enabled\).*/\1=true/' conf/config.properties && \ sed -i.bak 's/^\(stream.mqtt.address\).*/\1=mosquitto.mqtt:1883/' conf/config.properties && \ [SOURCE] The Mosquitto broker would be available at mosquitto.mqtt:1883 because of the service that is created…

Continue ReadingUsing Mosquitto as a Message Broker for MQTT in loklak Server