Author: Pratyush

This author has written 16 articles
Read more about the article Adding Tweet Streaming Feature in World Mood Tracker loklak App

Adding Tweet Streaming Feature in World Mood Tracker loklak App

The World Mood Tracker was added to loklak apps with the feature to display aggregated data from the emotion classifier of loklak server. The next step in the app was adding the feature to display the stream of Tweets from a country as they are discovered by loklak. With the addition of stream servlet in loklak, it was possible to utilise it in this app. In this blog post, I will be discussing the steps taken while adding to introduce this feature in World Mood Tracker app. Props for WorldMap component The WorldMap component holds the view for the map displayed in the app. This is where API calls to classifier endpoint are made and results are displayed on the map. In order to display tweets on clicking a country, we need to define react props so that methods from higher level components can be called. In order to enable props, we need to change the constructor for the component - export default class WorldMap extends React.Component { constructor(props) { super(props); ... } ... } [SOURCE] We can now pass the method from parent component to enable streaming and other components can close the stream by using props in them - export default class WorldMoodTracker extends React.Component { ... showStream(countryName, countryCode) { /* Do something to enable streaming component */ ... } render() { return ( ... <WorldMap showStream={this.showStream}/> ... ) } } [SOURCE] Defining Actions on Clicking Country Map As mentioned in an earlier blog post, World Mood Tracker uses Datamaps to visualize data on a map. In order to trigger a piece of code on clicking a country, we can use the "done" method of the Datamaps instance. This is where we use the props passed earlier - done: function(datamap) { datamap.svg.selectAll('.datamaps-subunit').on('click', function (geography) { props.showStream(geography.properties.name, reverseCountryCode(geography.id)); }) } [SOURCE] The name and ID for the country will be used to display name and make API call to stream endpoint respectively. The StreamOverlay Component The StreamOverlay components hold all the utilities to display the stream of Tweets from loklak. This component is used from its parent components whose state holds info about displaying this component - export default class WorldMoodTracker extends React.Component { ... getStreamOverlay() { if (this.state.enabled) { return (<StreamOverlay show={true} channel={this.state.channel} country={this.state.country} onClose={this.onOverlayClose}/>); } } render() { return ( ... {this.getStreamOverlay()} ... ) } } [SOURCE] The corresponding props passed are used to render the component and connect to the stream from loklak server. Creating Overlay Modal On clicking the map, an overlay is shown. To display this overlay, react-overlays is used. The Modal component offered by the packages provides a very simple interface to define the design and interface of the component, including style, onclose hook, etc. import {Modal} from 'react-overlays'; <Modal aria-labelledby='modal-label' style={modalStyle} backdropStyle={backdropStyle} show={true} onHide={this.close}> <div style={dialogStyle()}> ... </div> </Modal> [SOURCE] It must be noted that modalStyle and backdropStyle are React style objects. Dialog Style The dialog style is defined to provide some space at the top, clicking where, the overlay is closed.…

Continue ReadingAdding Tweet Streaming Feature in World Mood Tracker loklak App
Read more about the article Introducing Stream Servlet in loklak Server

Introducing Stream Servlet in loklak Server

A major part of my GSoC proposal was adding stream API to loklak server. In a previous blog post, I discussed the addition of Mosquitto as a message broker for MQTT streaming. After testing this service for a few days and some minor improvements, I was in a position to expose the stream to outside users using a simple API. In this blog post, I will be discussing the addition of /api/stream.json endpoint to loklak server. HTTP Server-Sent Events Server-sent events (SSE) is a technology where a browser receives automatic updates from a server via HTTP connection. The Server-Sent Events EventSource API is standardized as part of HTML5 by the W3C. - Wikipedia This API is supported by all major browsers except Microsoft Edge. For loklak, the plan was to use this event system to send messages, as they arrive, to the connected users. Apart from browser support, EventSource API can also be used with many other technologies too. Jetty Eventsource Plugin For Java, we can use Jetty’s EventSource plugin to send events to clients. It is similar to other Jetty servlets when it comes to processing the arguments, handling requests, etc. But it provides a simple interface to send events as they occur to connected users. Adding Dependency To use this plugin, we can add the following line to Gradle dependencies - compile group: 'org.eclipse.jetty', name: 'jetty-eventsource-servlet', version: '1.0.0' [SOURCE] The Event Source An EventSource is the object which is required for EventSourceServlet to send events. All the logics for emitting events needs to be defined in the related class. To link a servlet with an EventSource, we need to override the newEventSource method - public class StreamServlet extends EventSourceServlet { @Override protected EventSource newEventSource(HttpServletRequest request) { String channel = request.getParameter("channel"); if (channel == null) { return null; } if (channel.isEmpty()) { return null; } return new MqttEventSource(channel); } } [SOURCE] If no channel is provided, the EventSource object will be null and the request will be rejected. Here, the MqttEventSource would be used to handle the stream of Tweets as they arrive from the Mosquitto message broker. Cross Site Requests Since the requests to this endpoint can’t be of JSONP type, it is necessary to allow cross site requests on this endpoint. This can be done by overriding the doGet method of the servlet - @Override protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setHeader("Access-Control-Allow-Origin", "*"); super.doGet(request, response); } [SOURCE] Adding MQTT Subscriber When a request for events arrives, the constructor to MqttEventSource is called. At this stage, we need to connect to the stream from Mosquitto for the channel. To achieve this, we can set the class as MqttCallback using appropriate client configurations - public class MqttEventSource implements MqttCallback { ... MqttEventSource(String channel) { this.channel = channel; } ... this.mqttClient = new MqttClient(address, "loklak_server_subscriber"); this.mqttClient.connect(); this.mqttClient.setCallback(this); this.mqttClient.subscribe(this.channel); ... } [SOURCE] By setting the callback to this, we can override the messageArrived method to handle the arrival of a new message on the channel. Just to…

Continue ReadingIntroducing Stream Servlet in loklak Server
Read more about the article Optimising Docker Images for loklak Server

Optimising Docker Images for loklak Server

The loklak server is in a process of moving to Kubernetes. In order to do so, we needed to have different Docker images that suit these deployments. In this blog post, I will be discussing the process through which I optimised the size of Docker image for these deployments. Initial Image The image that I started with used Ubuntu as base. It installed all the components needed and then modified the configurations as required - FROM ubuntu:latest # Env Vars ENV LANG=en_US.UTF-8 ENV JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 ENV DEBIAN_FRONTEND noninteractive WORKDIR /loklak_server RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y git openjdk-8-jdk RUN git clone https://github.com/loklak/loklak_server.git /loklak_server RUN git checkout development RUN ./gradlew build -x test -x checkstyleTest -x checkstyleMain -x jacocoTestReport RUN sed -i.bak 's/^\(port.http=\).*/\180/' conf/config.properties ... # More configurations RUN echo "while true; do sleep 10;done" >> bin/start.sh # Start CMD ["bin/start.sh", "-Idn"] The size of images built using this Dockerfile was quite huge - REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE loklak_server       latest              a92f506b360d        About a minute ago   1.114 GB ubuntu              latest              ccc7a11d65b1        3 days ago           120.1 MB But since this size is not acceptable, we needed to reduce it. Moving to Apline Alpine Linux is an extremely lightweight Linux distro, built mainly for the container environment. Its size is so tiny that it hardly puts any impact on the overall size of images. So, I replaced Ubuntu with Alpine - FROM alpine:latest ... RUN apk update RUN apk add git openjdk8 bash ... And now we had much smaller images - REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE loklak_server       latest              54b507ee9187        17 seconds ago      668.8 MB alpine              latest              7328f6f8b418        6 weeks ago         3.966 MB As we can see that due to no caching and small size of Alpine, the image size is reduced to almost half the original. Reducing Content Size There are many things in a project which are no longer needed while running the project, like the .git folder (which is huge in case of loklak) - $ du -sh loklak_server/.git 236M loklak_server/.git We can remove such files from the Docker image and save a lot of space - rm -rf .[^.] .??* Optimizing Number of Layers The number of layers also affect the size of the image. More the number of layers, more will be the size of image. In the Dockerfile, we can club together the RUN commands for lower number of images. RUN apk update && apk add openjdk8 git bash && \ git clone https://github.com/loklak/loklak_server.git /loklak_server && \ ... After this, the effective size is again reduced by a major factor - REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE loklak_server       latest              54b507ee9187        17 seconds ago      422.3 MB alpine              latest              7328f6f8b418        6 weeks ago         3.966 MB Conclusion In this blog post, I discussed the process of optimising the size of Dockerfile for Kubernetes deployments of loklak server. The size was reduced to 426 MB from 1.234 GB and this provided much faster push/pull time for Docker images, and therefore, faster updates for Kubernetes deployments.…

Continue ReadingOptimising Docker Images for loklak Server
Read more about the article Persistently Storing loklak Server Dumps on Kubernetes

Persistently Storing loklak Server Dumps on Kubernetes

In an earlier blog post, I discussed loklak setup on Kubernetes. The deployment mentioned in the post was to test the development branch. Next, we needed to have a deployment where all the messages are collected and dumped in text files that can be reused. In this blog post, I will be discussing the challenges with such deployment and the approach to tackle them. Volatile Disk in Kubernetes The pods that hold deployments in Kubernetes have disk storage. Any data that gets written by the application stays only until the same version of deployment is running. As soon as the deployment is updated/relocated, the data stored during the application is cleaned up. Due to this, dumps are written when loklak is running but they get wiped out when the deployment image is updated. In other words, all dumps are lost when the image updates. We needed to find a solution to this as we needed a permanent storage when collecting dumps. Persistent Disk In order to have a storage which can hold data permanently, we can mount persistent disk(s) on a pod at the appropriate location. This ensures that the data that is important to us stays with us, even when the deployment goes down. In order to add persistent disks, we first need to create a persistent disk. On Google Cloud Platform, we can use the gcloud CLI to create disks in a given region - gcloud compute disks create --size=<required size> --zone=<same as cluster zone> <unique disk name> After this, we can mount it on a Docker volume defined in Kubernetes configurations - ... volumeMounts: - mountPath: /path/to/mount name: volume-name volumes: - name: volume-name gcePersistentDisk: pdName: disk-name fsType: fileSystemType But this setup can’t be used for storing loklak dumps. Let’s see “why” in the next section. Rolling Updates and Persistent Disk The Kubernetes deployment needs to be updated when the master branch of loklak server is updated. This update of master deployment would create a new pod and try to start loklak server on it. During all this, the older deployment would also be running and serving the requests. The control will not be transferred to the newer pod until it is ready and all the probes are passing. The newer deployment will now try to mount the disk which is mentioned in the configuration, but it would fail to do so. This would happen because the older pod has already mounted the disk. Therefore, all new deployments would simply fail to start due to insufficient resources. To overcome such issues, Kubernetes allows persistent volume claims. Let’s see how we used them for loklak deployment. Persistent Volume Claims Kubernetes provides Persistent Volume Claims which claim resources (storage) from a Persistent Volume (just like a pod does from a node). The higher level APIs are provided by Kubernetes (configurations and kubectl command line). In the loklak deployment, the persistent volume is a Google Compute Engine disk - apiVersion: v1 kind: PersistentVolume metadata: name: dump namespace: web spec: capacity: storage:…

Continue ReadingPersistently Storing loklak Server Dumps on Kubernetes
Read more about the article Using Mosquitto as a Message Broker for MQTT in loklak Server

Using Mosquitto as a Message Broker for MQTT in loklak Server

In loklak server, messages are collected from various sources and indexed using Elasticsearch. To know when a message of interest arrives, users can poll the search endpoint. But this method would require a lot of HTTP requests, most of them being redundant. Also, if a user would like to collect messages for a particular topic, he would need to make a lot of requests over a period of time to get enough data. For GSoC 2017, my proposal was to introduce stream API in the loklak server so that we could save ourselves from making too many requests and also add many use cases. Mosquitto is Eclipse’s project which acts as a message broker for the popular MQTT protocol. MQTT, based on the pub-sub model, is a lightweight and IOT friendly protocol. In this blog post, I will discuss the basic setup of Mosquitto in the loklak server. Installation and Dependency for Mosquitto The installation process of Mosquitto is very simple. For Ubuntu, it is available from the pre installed PPAs - sudo apt-get install mosquitto Once the message broker is up and running, we can use the clients to connect to it and publish/subscribe to channels. To add MQTT client as a project dependency, we can introduce following line in Gradle dependencies file - compile group: 'net.sf.xenqtt', name: 'xenqtt', version: '0.9.5' [SOURCE] After this, we can use the client libraries in the server code base. The MQTTPublisher Class The MQTTPublisher class in loklak would provide an interface to perform basic operations in MQTT. The implementation uses AsyncClientListener to connect to Mosquitto broker - AsyncClientListener listener = new AsyncClientListener() { // Override methods according to needs }; [SOURCE] The publish method for the class can be used by other components of the project to publish messages on the desired channel - public void publish(String channel, String message) { this.mqttClient.publish(new PublishMessage(channel, QoS.AT_LEAST_ONCE, message)); } [SOURCE] We also have methods which allow publishing of multiple messages to multiple channels in order to increase the functionality of the class. Starting Publisher with Server The flags which signal using of streaming service in loklak are located in conf/config.properties. These configurations are referred while initializing the Data Access Object and an MQTTPublisher is created if needed - String mqttAddress = getConfig("stream.mqtt.address", "tcp://127.0.0.1:1883"); streamEnabled = getConfig("stream.enabled", false); if (streamEnabled) { mqttPublisher = new MQTTPublisher(mqttAddress); } [SOURCE] The mqttPublisher can now be used by other components of loklak to publish messages to the channel they want. Adding Mosquitto to Kubernetes Since loklak has also a nice Kubernetes setup, it was very simple to introduce a new deployment for Mosquitto to it. Changes in Dockerfile The Dockerfile for master deployment has to be modified to discover Mosquitto broker in the Kubernetes cluster. For this purpose, corresponding flags in config.properties have to be changed to ensure that things work fine - sed -i.bak 's/^\(stream.enabled\).*/\1=true/' conf/config.properties && \ sed -i.bak 's/^\(stream.mqtt.address\).*/\1=mosquitto.mqtt:1883/' conf/config.properties && \ [SOURCE] The Mosquitto broker would be available at mosquitto.mqtt:1883 because of the service that is created…

Continue ReadingUsing Mosquitto as a Message Broker for MQTT in loklak Server
Read more about the article Deploying loklak Server on Kubernetes with External Elasticsearch

Deploying loklak Server on Kubernetes with External Elasticsearch

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. - kubernetes.io Kubernetes is an awesome cloud platform, which ensures that cloud applications run reliably. It runs automated tests, flawless updates, smart roll out and rollbacks, simple scaling and a lot more. So as a part of GSoC, I worked on taking the loklak server to Kubernetes on Google Cloud Platform. In this blog post, I will be discussing the approach followed to deploy development branch of loklak on Kubernetes. New Docker Image Since Kubernetes deployments work on Docker images, we needed one for the loklak project. The existing image would not be up to the mark for Kubernetes as it contained the declaration of volumes and exposing of ports. So I wrote a new Docker image which could be used in Kubernetes. The image would simply clone loklak server, build the project and trigger the server as CMD - FROM alpine:latest ENV LANG=en_US.UTF-8 ENV JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 WORKDIR /loklak_server RUN apk update && apk add openjdk8 git bash && \ git clone https://github.com/loklak/loklak_server.git /loklak_server && \ git checkout development && \ ./gradlew build -x test -x checkstyleTest -x checkstyleMain -x jacocoTestReport && \ # Some Configurations and Cleanups CMD ["bin/start.sh", "-Idn"] [SOURCE] This image wouldn’t have any volumes or exposed ports and we are now free to configure them in the configuration files (discussed in a later section). Building and Pushing Docker Image using Travis To automatically build and push on a commit to the master branch, Travis build is used. In the after_success section, a call to push Docker image is made. Travis environment variables hold the username and password for Docker hub and are used for logging in - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD [SOURCE] We needed checks there to ensure that we are on the right branch for the push and we are not handling a pull request - # Build and push Kubernetes Docker image KUBERNETES_BRANCH=loklak/loklak_server:latest-kubernetes-$TRAVIS_BRANCH KUBERNETES_COMMIT=loklak/loklak_server:kubernetes-$TRAVIS_COMMIT if [ "$TRAVIS_BRANCH" == "development" ]; then docker build -t loklak_server_kubernetes kubernetes/images/development docker tag loklak_server_kubernetes $KUBERNETES_BRANCH docker push $KUBERNETES_BRANCH docker tag $KUBERNETES_BRANCH $KUBERNETES_COMMIT docker push $KUBERNETES_COMMIT elif [ "$TRAVIS_BRANCH" == "master" ]; then # Build and push master else echo "Skipping Kubernetes image push for branch $TRAVIS_BRANCH" fi [SOURCE] Kubernetes Configurations for loklak Kubernetes cluster can completely be configured using configurations written in YAML format. The deployment of loklak uses the previously built image. Initially, the image tagged as latest-kubernetes-development is used - apiVersion: apps/v1beta1 kind: Deployment metadata: name: server namespace: web spec: replicas: 1 template: metadata: labels: app: server spec: containers: - name: server image: loklak/loklak_server:latest-kubernetes-development ... [SOURCE] Readiness and Liveness Probes Probes act as the top level tester for the health of a deployment in Kubernetes. The probes are performed periodically to ensure that things are working fine and appropriate steps are taken if they fail. When a new image is updated, the older pod still runs and servers the requests. It is replaced by the new ones only when the probes are…

Continue ReadingDeploying loklak Server on Kubernetes with External Elasticsearch
Read more about the article Caching Elasticsearch Aggregation Results in loklak Server

Caching Elasticsearch Aggregation Results in loklak Server

To provide aggregated data for various classifiers, loklak uses Elasticsearch aggregations. Aggregated data speaks a lot more than a few instances from it can say. But performing aggregations on each request can be very resource consuming. So we needed to come up with a way to reduce this load. In this post, I will be discussing how I came up with a caching model for the aggregated data from the Elasticsearch index. Fields to Consider while Caching At the classifier endpoint, aggregations can be requested based on the following fields - Classifier Name Classifier Classes Countries Start Date End Date But to cache results, we can ignore cases where we just require a few classes or countries and store aggregations for all of them instead. So the fields that will define the cache to look for will be - Classifier Name Start Date End Date Type of Cache The data structure used for caching was Java’s HashMap. It would be used to map a special string key to a special object discussed in a later section. Key The key is built using the fields mentioned previously - private static String getKey(String index, String classifier, String sinceDate, String untilDate) { return index + "::::" + classifier + "::::" + (sinceDate == null ? "" : sinceDate) + "::::" + (untilDate == null ? "" : untilDate); } [SOURCE] In this way, we can handle requests where a user makes a request for every class there is without running the expensive aggregation job every time. This is because the key for such requests will be same as we are not considering country and class for this purpose. Value The object used as key in the HashMap is a wrapper containing the following - json - It is a JSONObject containing the actual data. expiry - It is the expiry of the object in milliseconds. class JSONObjectWrapper { private JSONObject json; private long expiry; ... } Timeout The timeout associated with a cache is defined in the configuration file of the project as “classifierservlet.cache.timeout”. It defaults to 5 minutes and is used to set the eexpiryof a cached JSONObject - class JSONObjectWrapper { ... private static long timeout = DAO.getConfig("classifierservlet.cache.timeout", 300000); JSONObjectWrapper(JSONObject json) { this.json = json; this.expiry = System.currentTimeMillis() + timeout; } ... }   Cache Hit For searching in the cache, the previously mentioned string is composed from the parameters requested by the user. Checking for a cache hit can be done in the following manner - String key = getKey(index, classifier, sinceDate, untilDate); if (cacheMap.keySet().contains(key)) { JSONObjectWrapper jw = cacheMap.get(key); if (!jw.isExpired()) { // Do something with jw } } // Calculate the aggregations ... But since jw here would contain all the data, we would need to filter out the classes and countries which are not needed. Filtering results For filtering out the parts which do not contain the information requested by the user, we can perform a simple pass and exclude the results that are not needed. Since…

Continue ReadingCaching Elasticsearch Aggregation Results in loklak Server
Read more about the article Using Elasticsearch Aggregations to Analyse Classifier Data in loklak Server

Using Elasticsearch Aggregations to Analyse Classifier Data in loklak Server

Loklak uses Elasticsearch to index Tweets and other social media entities. It also houses a classifier that classifies Tweets based on emotion, profanity and language. But earlier, this data was available only with the search API and there was no way to get aggregated data out of it. So as a part of my GSoC project, I proposed to introduce a new API endpoint which would allow users to access aggregated data from these classifiers. In this blog post, I will be discussing how aggregations are performed on the Elasticsearch index of Tweets in the loklak server. Structure of index The ES index for Twitter is called messages and it has 3 fields related to classifiers - classifier_emotion classifier_language classifier_profanity With each of these classifiers, we also have a probability attached which represents the confidence of the classifier for assigned class to a Tweet. The name of these fields is given by suffixing the emotion field by _probability (e.g. classifier_emotion_probability). Since I will also be discussing aggregation based on countries in this blog post, there is also a field named place_country_code which saves the ISO 3166-1 alpha-2 code for the country of creation of Tweet. Requesting aggregations using Elasticsearch Java API Elasticsearch comes with a simple Java API which can be used to perform any desired task. To work with data, we need an ES client which can be built from a ES Node (if creating a cluster) or directly as a transport client (if connecting remotely to a cluster) - // Transport client TransportClient tc = TransportClient.builder() .settings(mySettings) .build(); // From a node Node elasticsearchNode = NodeBuilder.nodeBuilder() .local(false).settings(mySettings) .node(); Client nc = elasticsearchNode.client(); [SOURCE] Once we have a client, we can use ES AggregationBuilder to get aggregations from an index - SearchResponse response = elasticsearchClient.prepareSearch(indexName) .setSearchType(SearchType.QUERY_THEN_FETCH) .setQuery(QueryBuilders.matchAllQuery()) // Consider every row .setFrom(0).setSize(0) // 0 offset, 0 result size (do not return any rows) .addAggregation(aggr) // aggr is a AggregatoinBuilder object .execute().actionGet(); // Execute and get results [SOURCE] AggregationBuilders are objects that define the properties of an aggregation task using ES’s Java API. This code snippet is applicable for any type of aggregation that we wish to perform on an index, given that we do not want to fetch any rows as a response. Performing simple aggregation for a classifier In this section, I will discuss the process to get results from a given classifier in loklak’s ES index. Here, we will be targeting a class-wise count of rows and stats (average and sum) of probabilities. Writing AggregationBuilder An AggregationBuilder for this task will be a Terms AggregationBuilder which would dynamically generate buckets for all the different values of fields for a given field in index - AggregationBuilder getClassifierAggregationBuilder(String classifierName) { String probabilityField = classifierName + "_probability"; return AggregationBuilders.terms("by_class").field(classifierName) .subAggregation( AggregationBuilders.avg("avg_probability").field(probabilityField) ) .subAggregation( AggregationBuilders.sum("sum_probability").field(probabilityField) ); } [SOURCE] Here, the name of aggregation is passed as by_class. This will be used while processing the results for this aggregation task. Also, sub-aggregation is used to get average and sum probability by the…

Continue ReadingUsing Elasticsearch Aggregations to Analyse Classifier Data in loklak Server
Read more about the article Making loklak Server’s Kaizen Harvester Extendable

Making loklak Server’s Kaizen Harvester Extendable

Harvesting strategies in loklak are something that the end users can’t see, but they play a vital role in deciding the way in which messages are collected with loklak. One of the strategies in loklak is defined by the Kaizen Harvester, which generates queries from collected messages. The original strategy used a simple hash queue which drops queries once it is full. This effect is not desirable as we tend to lose important queries in this process if they come up late while harvesting. To overcome this behaviour without losing important search queries, we needed to come up with new harvesting strategy(ies) that would provide a better approach for harvesting. In this blog post, I am discussing the changes made in the kaizen harvester so it can be extended to create different flavors of harvesters. What can be different in extended harvesters? To make the Kaizen harvester extendable, we first needed to decide that what are the parts in the original Kaizen harvester that can be changed to make the strategy different (and probably better). Since one of the most crucial part of the Kaizen harvester was the way it stores queries to be processed, it was one of the most obvious things to change. Another thing that should be allowed to configure across various strategies was the decision of whether to go for harvesting the queries from the query list. Query storage with KaizenQueries To allow different methods of storing the queries, KaizenQueries class was introduced in loklak. It was configured to provide basic methods that would be required for a query storing technique to work. A query storing technique can be any data structure that we can use to store search queries for Kaizen harvester. public abstract class KaizenQueries { public abstract boolean addQuery(String query); public abstract String getQuery(); public abstract int getSize(); public abstract int getMaxSize(); public boolean isEmpty() { return this.getSize() == 0; } } [SOURCE] Also, a default type of KaizenQueries was introduced to use in the original Kaizen harvester. This allowed the same interface as the original queue which was used in the harvester. Another constructor was introduced in Kaizen harvester which allowed setting the KaizenQueries for an instance of its derived classes. It solved the problem of providing an interface of KaizenQueries inside the Kaizen harvester which can be used by any inherited strategy - private KaizenQueries queries = null; public KaizenHarvester(KaizenQueries queries) { ... this.queries = queries; ... } public void someMethod() { ... this.queries.addQuery(someQuery); ... } [SOURCE] This being added, getting new queries or adding new queries was a simple. We just need to use getQuery() and addQuery() methods without worrying about the internal implementations. Configurable decision for harvesting As mentioned earlier, the decision taken for harvesting should also be configurable. For this, a protected method was implemented and used in harvest() method - protected boolean shallHarvest() { float targetProb = random.nextFloat(); float prob = 0.5F; if (this.queries.getMaxSize() > 0) { prob = queries.getSize() / (float)queries.getMaxSize(); } return !this.queries.isEmpty() && targetProb…

Continue ReadingMaking loklak Server’s Kaizen Harvester Extendable
Read more about the article Adding React based World Mood Tracker to loklak Apps

Adding React based World Mood Tracker to loklak Apps

loklak apps is a website that hosts various apps that are built by using loklak API. It uses static pages and angular.js to make API calls and show results from users. As a part of my GSoC project, I had to introduce the World Mood Tracker app using loklak’s mood API. But since I had planned to work on React, I had to go off from the track of typical app development in loklak apps and integrate a React app in apps.loklak.org. In this blog post, I will be discussing how I introduced a React based app to apps.loklak.org and solved the problem of country-wise visualisation of mood related data on a World map. Setting up development environment inside apps.loklak.org After following the steps to create a new app in apps.loklak.org, I needed to add proper tools and libraries for smooth development of the World Mood Tracker app. In this section, I’ll be explaining the basic configuration that made it possible for a React app to be functional in the angular environment. Pre-requisites The most obvious prerequisite for the project was Node.js. I used node v8.0.0 while development of the app. Instead of npm, I decided to go with yarn because of offline caching and Internet speed issues in India. Webpack and Babel To begin with, I initiated yarn in the app directory inside project and added basic dependencies - $ yarn init $ yarn add webpack webpack-dev-server path $ yarn add babel-loader babel-core babel-preset-es2015 babel-preset-react --dev   Next, I configured webpack to set an entry point and output path for the node project in webpack.config.js - module.exports = { entry: './js/index.js', output: { path: path.resolve('.'), filename: 'index_bundle.js' }, ... }; This would signal to look for ./js/index.js as an entry point while bundling. Similarly, I configured babel for es2015 and React presets - { "presets":[ "es2015", "react" ] }   After this, I was in a state to define loaders for module in webpack.config.js. The loaders would check for /\.js$/ and /\.jsx$/ and assign them to babel-loader (with an exclusion of node_modules). React After configuring the basic presets and loaders, I added React to dependencies of the project - $ yarn add react react-dom   The React related files needed to be in ./js/ directory so that the webpack can bundle it. I used the file to create a simple React app - import React from 'react'; import ReactDOM from 'react-dom'; ReactDOM.render( <div>World Mood Tracker</div>, document.getElementById('app') );   After this, I was in a stage where it was possible to use this app as a part of apps.loklak.org app. But to do this, I first needed to compile these files and bundle them so that the external app can use it. Configuring the build target for webpack In apps.loklak.org, we need to have a file by the name of index.html in the app’s root directory. Here, we also needed to place the bundled js properly so it could be included in index.html at app’s root. HTML Webpack Plugin Using…

Continue ReadingAdding React based World Mood Tracker to loklak Apps