docker-compose

Read more about the article Creating A Dockerfile For Yacy Grid MCP

Creating A Dockerfile For Yacy Grid MCP

The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine. A YaCy Grid installation consists of a set of micro-services which communicate with each other using a common infrastructure for data persistence. The task was to deploy the second-generation of YaCy Grid. To do so, we first had created a Dockerfile. This dockerfile should start the micro services such as rabbitmq, Apache ftp and elasticsearch in one docker instance along with MCP. The microservices perform following tasks:

  • Apache ftp server for asset storage.
  • RabbitMQ message queues for the message system.
  • Elasticsearch for database operations.

To launch these microservices using Dockerfile, we referred to following documentations regarding running these services locally: https://github.com/yacy/yacy_grid_mcp/blob/master/README.md

For creating a Dockerfile we proceeded as follows:

FROM ubuntu:latest
MAINTAINER Harshit Prasad# Update
RUN apt-get update
RUN apt-get upgrade -y# add packages
# install jdk package for java
RUN apt-get install -y git openjdk-8-jdk

#install gradle required for build
RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:cwchien/gradle
RUN apt-get update
RUN apt-get install -y wget
RUN wget https://services.gradle.org/distributions/gradle-3.4.1-bin.zip
RUN mkdir /opt/gradle
RUN apt-get install -y unzip
RUN unzip -d /opt/gradle gradle-3.4.1-bin.zip
RUN PATH=$PATH:/opt/gradle/gradle-3.4.1/bin
ENV GRADLE_HOME=/opt/gradle/gradle-3.4.1
ENV PATH=$PATH:$GRADLE_HOME/bin
RUN gradle -v

# install apache ftp server 1.1.0
RUN wget http://www-eu.apache.org/dist/mina/ftpserver/1.1.0/dist/apache-ftpserver-1.1.0.tar.gz
RUN tar xfz apache-ftpserver-1.1.0.tar.gz

# install RabbitMQ server
RUN wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-generic-unix-3.6.6.tar.xz
RUN tar xf rabbitmq-server-generic-unix-3.6.6.tar.xz

# install erlang language for RabbitMQ
RUN apt-get install -y erlang

# install elasticsearch
RUN wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
RUN sha1sum elasticsearch-5.5.0.tar.gz
RUN tar -xzf elasticsearch-5.5.0.tar.gz

# clone yacy_grid_mcp repository
RUN git clone https://github.com/nikhilrayaprolu/yacy_grid_mcp.git
WORKDIR /yacy_grid_mcp

RUN cat docker/config–ftp.properties > ../apache–ftpserver–1.1.0/res/conf/users.properties

# compile
RUN gradle build
RUN mkdir data/mcp-8100/conf/ -p
RUN cp docker/config-mcp.properties data/mcp-8100/conf/config.properties
RUN chmod +x ./docker/start.sh

# Expose web interface ports
# 2121: ftp, a FTP server to be used for mass data / file storage
# 5672: rabbitmq, a rabbitmq message queue server to be used for global messages, queues and stacks
# 9300: elastic, an elasticsearch server or main cluster address for global database storage
EXPOSE 2121 5672 9300 9200 15672 8100

# Define default command.
ENTRYPOINT [“/bin/bash”, “./docker/start.sh”]

 

We have created a start.sh file to start RabbitMQ and Apache FTP services. At the end, for compilation gradle run will be executed.

adduser –disabled-password –gecos ” r
adduser r sudo
echo ‘%sudo ALL=(ALL) NOPASSWD:ALL’ >> /etc/sudoers
chmod a+rwx /elasticsearch-5.5.0 -R
su -m r -c ‘/elasticsearch-5.5.0/bin/elasticsearch -Ecluster.name=yacygrid &’
cd /apache–ftpserver–1.1.0
./bin/ftpd.sh res/conf/ftpd–typical.xml &
/rabbitmq_server-3.6.6/sbin/rabbitmq-server -detached
sleep 5s;
/rabbitmq_server-3.6.6/sbin/rabbitmq-plugins enable rabbitmq_management
/rabbitmq_server–3.6.6/sbin/rabbitmqctl add_user yacygrid password4account
echo [{rabbit, [{loopback_users, []}]}]. >> /rabbitmq_server-3.6.6/etc/rabbitmq/rabbitmq.config
/rabbitmq_server-3.6.6/sbin/rabbitmqctl set_permissions -p / yacygrid “.*” “.*” “.*”
cd /yacy_grid_mcp
sleep 5s;
gradle run

 

start.sh will first add username and then password. Then it will start RabbitMQ along with Apache FTP.  For username and password, we have created a separate files to configure their properties during Docker run which can be found here:

The logic behind running all the microservices in one docker instance was: creating each container for microservice and then link those containers with the help of docker-compose.yml file.

The Dockerfile which we have created was corresponding to one image. Another image was elasticsearch which was linked to this Dockerfile. The latest version of elasticsearch image was already available on their site: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

We configured the docker-compose.yml file according to the reference link provided above. The docker-compose file can be found here: https://github.com/yacy/yacy_grid_mcp/blob/master/docker/docker-compose.yml

The source code for the implementation of whole structure can be found here: https://github.com/yacy/yacy_grid_mcp/tree/master/docker

Resources

 

Continue ReadingCreating A Dockerfile For Yacy Grid MCP
Read more about the article How to use Digital Ocean and Docker to setup Test CMS for Phimpme

How to use Digital Ocean and Docker to setup Test CMS for Phimpme

One of the core feature of Phimpme app is sharing images to other different accounts, including various open source CMS such as WordPress, Drupal etc and other open source data storage account such as OwnCloud, NextCloud etc.

One can not have everything at place, but for development and testing purpose it is required in our end. So problem I was facing to get things done in most optimize way. I thought setting things on hosted server would be good, because it saves lots of time in setting locally in our system, adding all the dependencies. And also we cannot share the account as it is limited to our local environment.

Digital Ocean caught my attention in providing hosting service. It is very easy to use with their droplet creation. Select as per your system requirement and service requirement, the droplet will be ready in few moments and we can use it anywhere.

Note: DigitalOcean is a paid service. Student can use Github Education Pack for free credits on Digital Ocean. I used the same.

I currently worked on Nextcloud integration so here in this blog I will tell how to quickly create nextcloud server using Digital Ocean and Docker.

Step 1: Creating Droplet

DigitalOcean completely work on droplets and one can anytime create and destroy different droplets associated with their account.

Choose an Image

So there are three options of choosing the image of Droplet.

Distributions : Which is other operating systems you want to use

One Click app: It is a very good feature as it creates everything for use in just one click. But again, it doesn’t provide everything, like there is no NextCloud. That’s why I used docker to take its image.

Snapshots: This is if you saved your droplet already, so it will pick it and creates similar to the saved image. Here I selected Docker from one-click apps section.

Selecting the size

This is for selecting the size of the server we are creating, For small development purpose $5 plan is good. There is a good thing in DigitalOcean as it didn’t charge on the monthly basis to the use. It works on hourly basis and charge according to that. Also providing the SSD Disk for fast IO operations.

Choose a datacenter Region

Add SSH

This is very important to add a ssh key. Otherwise you have to setup root password or used the shell they provide. To work on your computer terminal, its good that you setup an ssh key already and it to the account.

How to get ssh key in your system: https://help.github.com/

Rename the number of droplet and name of the droplet and create.

Now it will show there in your droplet section

Step 2: Access the Server

As we have already added the ssh key to our droplet. Now we can access it from our terminal. Open the terminal and type this 

➜  ~ ssh root@<your IP>

It will logged in to you 

root@docker-512mb-blr1-01:~#

Our objective is setting a NextCloud Account.

Here now I will use Docker. Firstly, What is Docker?

Go here to read: https://www.docker.com/what-docker

I will explain docker in other words. Like I setted up everything which I need. Now If I have to destroy this all and want to use it after some days. Or if my friends wants to use the setted platform. What is the option here?

Recreate and everything everytime? NO.

Just create docker image, save it pull the image when you want, and run it to serve on the serve. Your friends need, provide them the docker image.

Isn’t it cool and much time saving.

Browse the Docker Hub

In the hub we can find docker images for various platforms officially maintained by the authors.

Nextcloud have their official account on Docker to provide latest images to the developers.

Here is the link : https://hub.docker.com/_/nextcloud/

Pull the image in your server.

root@docker-512mb-blr1-01:~# docker pull nextcloud
Using default tag: latest
latest: Pulling from library/nextcloud
9f0706ba7422: Pull complete
4c407763908f: Pull complete
82e2bc3a45c1: Pull complete
c84e1013aed1: Pull complete
a3b5e03d7e24: Pull complete
917f836a88be: Pull complete
b2dc54431819: Pull complete
a60b574790b8: Pull complete
49ef0f1aff88: Pull complete
7773a865ee49: Pull complete
9e0e5cc56a9d: Pull complete
bfade1c7421e: Pull complete
ece8ceb33bed: Pull complete
c691d2747a3e: Pull complete
4b5e96bf54c9: Pull complete
6fbe30ae456b: Pull complete
e0c534b35a6b: Pull complete
4d2687f4b6f3: Pull complete
00197422846a: Pull complete
6ab57168c49c: Pull complete
9e1260db005f: Pull complete
Digest: sha256:1bb5c256f19dcec60d8468c00bc7dc74efdf93390666cb82e20bcacbbbd9746c
Status: Downloaded newer image for nextcloud:latest
root@docker-512mb-blr1-01:~#

Following the documentation

I need to run this command $ docker run -d -p 8080:80 nextcloud

It serves the account on localhost.

Check on your https://<IP>:8080

So in this way I easily setup different account for testing and integration purpose in Phimpme Android app. It really saves lots of time and speed up the process.

You can easily destroy the droplet after work is done.

Student can use the free credits from GitHub Education Pack.

Source:

 

Continue ReadingHow to use Digital Ocean and Docker to setup Test CMS for Phimpme

Getting started with Docker Compose

In this post, I will talk about running multiple containers at once using Docker Compose.

The problem ?

Suppose you have a complex app with Database containers, Redis and what not. How are you going to start the app ? One way is to write a shell script that starts the containers one by one.

docker run postgres:latest --name mydb -d
docker run redis:3-alpine --name myredis -d
docker run myapp -d

Now suppose these containers have lots of configurations (links, volumes, ports, environment variables) that they need to function. You will have to write those parameters in the shell script.

docker network create myapp_default
docker run postgres:latest --name db -d -p 5432:5432 --net myapp_default
docker run redis:3-alpine --name redis -d -p 6379:6379 \
    --net myapp_default -v redis:/var/lib/redis/data
docker run myapp -d -p 5000:5000 --net myapp_default -e SOMEVAR=value \
    --link db:db --link redis:redis -v storage:/myapp/static

Won’t it get un-manageable ? Won’t it be great if we had a cleaner way to running multiple containers. Here comes docker-compose to the rescue.

Docker compose

Docker compose is a python package which does the job of handling multiple containers for an application very elegantly. The main file of docker-compose is docker-compose.yml which is a YAML like syntax file with the settings/components required to run your app. Once you define that file, you can just do docker-compose up to start your app with all the components and settings. Pretty cool, right ?

So let’s see the docker-compose.yml for the fictional app we have considered above.

version: '2'

services:
  db:
    image: postgres:latest
    ports:
      - '5432:5432'

  redis:
    image: 'redis:3-alpine'
    command: redis-server
    volumes:
      - 'redis:/var/lib/redis/data'
    ports:
      - '6379:6379'

  web:
    build: .
    environment:
      SOMEVAR: value
    links:
      - db:db
      - redis:redis
    volumes:
      - 'storage:/myapp/static'
    ports:
      - '5000:5000'

volumes:
  redis:
  storage:

Once this file is in the project’s root directory, you can use docker-compose up to start the application. It will run the services in the order in which they have been defined in the YAML file.

Docker compose has a lot of commands that generally correspond to the parameters that docker runaccepts. You can see a full list on the official docker-compose reference.

Conclusion

It’s no doubt that docker-compose is a boon when you have to run complex applications. It personally use Compose in every dockerized application that I write. In GSoC 16, I dockerized Open Event. Here is the docker-compose.yml file if you are interested.

PS – If you liked this post, you might find my other posts on Docker interesting. Do take a look and let me know your views.

 

{{ Repost from my personal blog http://aviaryan.in/blog/gsoc/docker-compose-starting.html }}

Continue ReadingGetting started with Docker Compose