Continuous Integration and Deployment of Yacy Grid
We have deployed Yacy Grid on Google cloud recently, and we have achieved this using kubernetes and Travis for auto deployment.
How we have deployed it:
Firstly, it is advised to have different containers for each service your application requires, and follow a multi container architecture. Using multi container architecture you can allocate fixed size of power to each application and also replicate individual services, whichever is required. Presently, Yacy has two main applications which are required to be deployed in separate containers – Yacy_grid_mcp and ElasticSearch.
We took the official kubernetes YAML files of ElasticSearch and followed the instructions at https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md for deployment of elastic search on the google cloud.
With this we are able to run pods, volumes required for elastic search and services for connecting Yacy with elastic search.
The pull request regarding deployment of separate elasticsearch component is at https://github.com/yacy/yacy_grid_mcp/pull/27/files
Below figure shows different services and external endpoints present pods use for elastic search.
Now elastic search can be accessed at 18.104.22.168:9300 and http://22.214.171.124:9200/
Continuous deployment of Yacy_grid_mcp:
Please make sure that you have created a cluster on google container engine for deploying our containers on it. Regarding starting a project and cluster please read https://cloud.google.com/container-engine/docs/
1.Initially, Travis.yml initiates and sets up the required environment for Yacy deployment by installing Google cloud cli and kubectl components.
Source code regarding the Travis setup could be found at https://github.com/yacy/yacy_grid_mcp/blob/master/.travis.yml
2.Later Travis runs the depoy_staging.sh file, which builds the docker image of yacy o the present build and pushes it to hub.docker.com
Later with service key, we authenticate with google cloud and set the required environments and variables
And Later we push the docker image built to google cloud and deploy it
Presently Yacy runs on 5vCPUs
With the following pods and services:
Also one can use kubectl cli for getting information regarding the cluster and pods as shown below
Pull request regarding deployment of yacy on google cloud is available at: https://github.com/yacy/yacy_grid_mcp/pull/16/files
1.A Medium Blog on CD to Google Container: https://medium.com/google-cloud/continuous-delivery-in-a-microservice-infrastructure-with-google-container-engine-docker-and-fb9772e81da7
2.Another Blog on CD to Google Container: https://engineering.hexacta.com/automatic-deployment-of-multiple-docker-containers-to-google-container-engine-using-travis-e5d9e191d5ad
3.Deploying ElasticSearch to Cloud using Kubernetes: https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md