docker-compose

Read more about the article Creating A Dockerfile For Yacy Grid MCP

Creating A Dockerfile For Yacy Grid MCP

The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine. A YaCy Grid installation consists of a set of micro-services which communicate with each other using a common infrastructure for data persistence. The task was to deploy the second-generation of YaCy Grid. To do so, we first had created a Dockerfile. This dockerfile should start the micro services such as rabbitmq, Apache ftp and elasticsearch in one docker instance along with MCP. The microservices perform following tasks: Apache ftp server for asset storage. RabbitMQ message queues for the message system. Elasticsearch for database operations. To launch these microservices using Dockerfile, we referred to following documentations regarding running these services locally: https://github.com/yacy/yacy_grid_mcp/blob/master/README.md For creating a Dockerfile we proceeded as follows: FROM ubuntu:latest MAINTAINER Harshit Prasad# Update RUN apt-get update RUN apt-get upgrade -y# add packages # install jdk package for java RUN apt-get install -y git openjdk-8-jdk #install gradle required for build RUN apt-get update && apt-get install -y software-properties-common RUN add-apt-repository ppa:cwchien/gradle RUN apt-get update RUN apt-get install -y wget RUN wget https://services.gradle.org/distributions/gradle-3.4.1-bin.zip RUN mkdir /opt/gradle RUN apt-get install -y unzip RUN unzip -d /opt/gradle gradle-3.4.1-bin.zip RUN PATH=$PATH:/opt/gradle/gradle-3.4.1/bin ENV GRADLE_HOME=/opt/gradle/gradle-3.4.1 ENV PATH=$PATH:$GRADLE_HOME/bin RUN gradle -v # install apache ftp server 1.1.0 RUN wget http://www-eu.apache.org/dist/mina/ftpserver/1.1.0/dist/apache-ftpserver-1.1.0.tar.gz RUN tar xfz apache-ftpserver-1.1.0.tar.gz # install RabbitMQ server RUN wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-generic-unix-3.6.6.tar.xz RUN tar xf rabbitmq-server-generic-unix-3.6.6.tar.xz # install erlang language for RabbitMQ RUN apt-get install -y erlang # install elasticsearch RUN wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz RUN sha1sum elasticsearch-5.5.0.tar.gz RUN tar -xzf elasticsearch-5.5.0.tar.gz # clone yacy_grid_mcp repository RUN git clone https://github.com/nikhilrayaprolu/yacy_grid_mcp.git WORKDIR /yacy_grid_mcp RUN cat docker/config-ftp.properties > ../apache-ftpserver-1.1.0/res/conf/users.properties # compile RUN gradle build RUN mkdir data/mcp-8100/conf/ -p RUN cp docker/config-mcp.properties data/mcp-8100/conf/config.properties RUN chmod +x ./docker/start.sh # Expose web interface ports # 2121: ftp, a FTP server to be used for mass data / file storage # 5672: rabbitmq, a rabbitmq message queue server to be used for global messages, queues and stacks # 9300: elastic, an elasticsearch server or main cluster address for global database storage EXPOSE 2121 5672 9300 9200 15672 8100 # Define default command. ENTRYPOINT ["/bin/bash", "./docker/start.sh"]   We have created a start.sh file to start RabbitMQ and Apache FTP services. At the end, for compilation gradle run will be executed. adduser --disabled-password --gecos '' r adduser r sudo echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers chmod a+rwx /elasticsearch-5.5.0 -R su -m r -c '/elasticsearch-5.5.0/bin/elasticsearch -Ecluster.name=yacygrid &' cd /apache-ftpserver-1.1.0 ./bin/ftpd.sh res/conf/ftpd-typical.xml & /rabbitmq_server-3.6.6/sbin/rabbitmq-server -detached sleep 5s; /rabbitmq_server-3.6.6/sbin/rabbitmq-plugins enable rabbitmq_management /rabbitmq_server-3.6.6/sbin/rabbitmqctl add_user yacygrid password4account echo [{rabbit, [{loopback_users, []}]}]. >> /rabbitmq_server-3.6.6/etc/rabbitmq/rabbitmq.config /rabbitmq_server-3.6.6/sbin/rabbitmqctl set_permissions -p / yacygrid ".*" ".*" ".*" cd /yacy_grid_mcp sleep 5s; gradle run   start.sh will first add username and then password. Then it will start RabbitMQ along with Apache FTP.  For username and password, we have created a separate files to configure their properties during Docker run which can be found here: Configuration of FTP server: https://github.com/yacy/yacy_grid_mcp/blob/master/docker/config-ftp.properties Configuration of MCP service: https://github.com/yacy/yacy_grid_mcp/blob/master/docker/config-mcp.properties The logic behind running all the microservices in one docker instance was: creating each container for microservice and then link those containers…

Continue ReadingCreating A Dockerfile For Yacy Grid MCP
Read more about the article How to use Digital Ocean and Docker to setup Test CMS for Phimpme

How to use Digital Ocean and Docker to setup Test CMS for Phimpme

One of the core feature of Phimpme app is sharing images to other different accounts, including various open source CMS such as Wordpress, Drupal etc and other open source data storage account such as OwnCloud, NextCloud etc. One can not have everything at place, but for development and testing purpose it is required in our end. So problem I was facing to get things done in most optimize way. I thought setting things on hosted server would be good, because it saves lots of time in setting locally in our system, adding all the dependencies. And also we cannot share the account as it is limited to our local environment. Digital Ocean caught my attention in providing hosting service. It is very easy to use with their droplet creation. Select as per your system requirement and service requirement, the droplet will be ready in few moments and we can use it anywhere. Note: DigitalOcean is a paid service. Student can use Github Education Pack for free credits on Digital Ocean. I used the same. I currently worked on Nextcloud integration so here in this blog I will tell how to quickly create nextcloud server using Digital Ocean and Docker. Step 1: Creating Droplet DigitalOcean completely work on droplets and one can anytime create and destroy different droplets associated with their account. Choose an Image So there are three options of choosing the image of Droplet. Distributions : Which is other operating systems you want to use One Click app: It is a very good feature as it creates everything for use in just one click. But again, it doesn’t provide everything, like there is no NextCloud. That’s why I used docker to take its image. Snapshots: This is if you saved your droplet already, so it will pick it and creates similar to the saved image. Here I selected Docker from one-click apps section. Selecting the size This is for selecting the size of the server we are creating, For small development purpose $5 plan is good. There is a good thing in DigitalOcean as it didn’t charge on the monthly basis to the use. It works on hourly basis and charge according to that. Also providing the SSD Disk for fast IO operations. Choose a datacenter Region Add SSH This is very important to add a ssh key. Otherwise you have to setup root password or used the shell they provide. To work on your computer terminal, its good that you setup an ssh key already and it to the account. How to get ssh key in your system: https://help.github.com/ Rename the number of droplet and name of the droplet and create. Now it will show there in your droplet section Step 2: Access the Server As we have already added the ssh key to our droplet. Now we can access it from our terminal. Open the terminal and type this ➜  ~ ssh root@<your IP> It will logged in to you root@docker-512mb-blr1-01:~# Our objective is setting a NextCloud Account.…

Continue ReadingHow to use Digital Ocean and Docker to setup Test CMS for Phimpme

Getting started with Docker Compose

In this post, I will talk about running multiple containers at once using Docker Compose. The problem ? Suppose you have a complex app with Database containers, Redis and what not. How are you going to start the app ? One way is to write a shell script that starts the containers one by one. docker run postgres:latest --name mydb -d docker run redis:3-alpine --name myredis -d docker run myapp -d Now suppose these containers have lots of configurations (links, volumes, ports, environment variables) that they need to function. You will have to write those parameters in the shell script. docker network create myapp_default docker run postgres:latest --name db -d -p 5432:5432 --net myapp_default docker run redis:3-alpine --name redis -d -p 6379:6379 \ --net myapp_default -v redis:/var/lib/redis/data docker run myapp -d -p 5000:5000 --net myapp_default -e SOMEVAR=value \ --link db:db --link redis:redis -v storage:/myapp/static Won’t it get un-manageable ? Won’t it be great if we had a cleaner way to running multiple containers. Here comes docker-compose to the rescue. Docker compose Docker compose is a python package which does the job of handling multiple containers for an application very elegantly. The main file of docker-compose is docker-compose.yml which is a YAML like syntax file with the settings/components required to run your app. Once you define that file, you can just do docker-compose up to start your app with all the components and settings. Pretty cool, right ? So let’s see the docker-compose.yml for the fictional app we have considered above. version: '2' services: db: image: postgres:latest ports: - '5432:5432' redis: image: 'redis:3-alpine' command: redis-server volumes: - 'redis:/var/lib/redis/data' ports: - '6379:6379' web: build: . environment: SOMEVAR: value links: - db:db - redis:redis volumes: - 'storage:/myapp/static' ports: - '5000:5000' volumes: redis: storage: Once this file is in the project’s root directory, you can use docker-compose up to start the application. It will run the services in the order in which they have been defined in the YAML file. Docker compose has a lot of commands that generally correspond to the parameters that docker runaccepts. You can see a full list on the official docker-compose reference. Conclusion It’s no doubt that docker-compose is a boon when you have to run complex applications. It personally use Compose in every dockerized application that I write. In GSoC 16, I dockerized Open Event. Here is the docker-compose.yml file if you are interested. PS - If you liked this post, you might find my other posts on Docker interesting. Do take a look and let me know your views.   {{ Repost from my personal blog http://aviaryan.in/blog/gsoc/docker-compose-starting.html }}

Continue ReadingGetting started with Docker Compose