Using Open Layers 3 to Render a loklak Emoji Heatmap

In Emoji Heatmapper App I am implementing a heatmap with the help of Open Layers 3. Open Layers 3 contains a really handy class, ol.layer.Heatmap. In this blog post am going to tell you how the heatmap actually works in the backend.

A heatmap is an impressive way to visualize data. For a given matrix of data in which each value is represented by a color. The heatmap implementation is usually expensive in computation terms: For each grid’s pixel we need to compute its color from a set of known values. It is not a feasible method to be implemented on the client side because map rendering would take so much of time.

Open Layers 3 contains an easy-to-use class called ol.layer.Heatmap, which allows to render vector data as a heatmap. So how is this implemented?

The ol.layer.Heatmap layer uses a smart estimation to the design which produces relatively good results and which is also fast. The steps can be outlined as:

  • A gradient of colors is created as an image.
  • Each value is rendered in a canvas as a blurred point using the default radius and gradient. This produces a canvas where the blurred points may overlap each other and create more fuzzy zones.
  • Finally, an image is obtained from the canvas. The color is obtained from the previous image and the obtained color value may vary from 0 to 255.

Example usage of ol.layer.Heatmap class:

var heatMap = new ol.layer.Heatmap({
  source: vector,
  blur: parseInt(15, 10),
  radius: parseInt(5, 10),
  opacity: 0.9,
  gradient: ['#0000ff', '#f00', '#f00', '#ff0', '#f00']
});

 

The colored image is then rendered in the map canvas, obtaining a nice effect suited to be used for density maps. The ol.layer.Heatmap offers some properties we can use to visualize the map in a better way. The properties include blur, radius, gradient, shadow and weight. This can be configured as per feature, according to the level of importance to each feature determining in more or less measure of the final color we want.

Fig: Default colors of gradient property

Fig: Used gradient property for different colors

Resources

Continue ReadingUsing Open Layers 3 to Render a loklak Emoji Heatmap

Sentiment Data in Emoji-Heatmapper loklak App

Analysing emojis can uncover meaning and sentiment in ways regular text analytics cannot. So this was the main idea to introduce sentiment data into the Emoji-Heatmapper app. LokLak Search API has features such as classification and categorization of tweets. The emotions, for instance, can be joy, anticipation, sad etc.

So, in the Emoji-heatmapper app, I am displaying the occurrence of emojis on the map according to the location traced and also the sentiment related to the emoji i.e., search query as follows:

How to get the sentiment data:

One should simply enter the emoji into the search box for the results. The following code shows part of the LokLak Search API results (JSONObject):

      "hashtags_count": 0,
      "classifier_emotion": "anger",
      "classifier_emotion_probability": 
      1.2842921170985733E-9,
      "classifier_language": "english",
      "classifier_language_probability": 
      1.4594549568869297E-8,
      "without_l_len": 49,
      "without_lu_len": 49,
      "without_luh_len": 49,

I am using the above field name ”classifier_emotion” to display the results.

Till here getting the data relevant to query part is done. Next, the classifier_emotion of each tweet containing the query is collected into an array and sorted to get a unique list.

var emotion = [];
var emotions_array = [];

for (var i = 0; i  <  tweets.statuses.length; i++) {
    if (tweets.statuses[i].classifier_emotion) {
        emotion = tweets.statuses[i].classifier_emotion;
        emotions_array.push(emotion);
    }

    emotions_array.sort();
    emotion_array = jQuery.unique( emotions_array );

Loading the Sentiment data onto the Screen

When the query has a single emotion, or if multiple emotions or no emotions. These use cases/situations are displayed as follows:

Fig: Single Emotion

Fig: Multiple Emotions

Fig: No Emotions data

The code which creates the data dynamically on the output screen is as follows:

//Loading the sentiment
$(document).ready(function() {
    var listItems= "";
    if (emotions_array.length == 0) {
        listItems = "No Sentiment data is available for " + query
    }
    if (emotion_array.length == 1) {
        listItems += "<h3> Sentiment of " + query + " is ";
    } else if (emotion_array.length > 1) {
        listItems += "<h3> Sentiments of " + query + " are ";
    }

    var emotion_data = emotion_array.join(", ") + "."
    listItems += emotion_data + "</h3>"

    $("#sentiment").html(listItems);
});

Conclusion

The Emoji-Heatmapper app displays the sentiment data of the query being searched for which populates data dynamically using LokLak Search API.

Resources

Continue ReadingSentiment Data in Emoji-Heatmapper loklak App

Emoticon Map Markers in Emoji Heatmapper App

As I’ve been exploring and trying to learn what’s possible in the maps of OpenLayers 3 using LokLak API, I wondered about map markers. The markers which i used earlier seem to be so dull on the map, and as I am working on Emoji Heatmapper I couldn’t help but think about 🍩,  😻, 🍦 and 🔮. And sure, the more practical 🏡 , 🏢, ☕ and 🌆.

Emojis as map markers? I had to give it a try.

I didn’t know how one acquires the emoji trove, so I searched around Github. Sure enough, I found many solutions on GitHub. I sifted through all of them until Emoji-picker caught my attention. So i tried giving a dropdown using the emoji-picker as searching would be lot more easier for the user.

Emoji-picker will convert an emoji keyword to the image internally. That is why when you hover over an emoji in the drop-down menu, it shows the corresponding keyword. For instance, the image 🚀 when hovered on it, it displays :rocket: .

 

All the emojis are saved as data URIs, so I don’t need to worry about lugging around hundreds of images. All I need is emoji-picker.js, and few more *.js files  hooked up on my page, and a little JavaScript to get everything working accordingly.

Armed with hundreds of emojis, my next step was to swap markers with emoji keywords. After a few clicks around emoji-picker documentation, I landed on data-emoji-input=”unicode” . It allows you to replace the traditional marker with a unicode emojis so the search outputs data. You can add a class to that lead emoji-picker-container and data-emoji-input=”unicode” for the HTML option.

Style the Open Layers 3 map:

var style = new ol.style.Style({
    stroke: new ol.style.Stroke({
        color: [64, 200, 200, 0.5],
        width: 5
    }),
    text: new ol.style.Text({
        font: '30px sans-serif',
        text: document.getElementById('searchField').value !== '' ? document.getElementById('searchField').value : '',
        fill: new ol.style.Fill({
            color: [64, 64, 64, 0.75]
        })
    })
});

 

and 🎇 I have an emoji map marker.

Resources:

Continue ReadingEmoticon Map Markers in Emoji Heatmapper App

OpenLayers 3 Map that Animates Emojis Using LokLak API

OpenLayers3 maps are fully functional maps which offer additional interactive features. In the Emoji Heatmapper app in Loklak Apps, I am using interactive OpenLayers3 maps to visualize the data. In this blog post, I am going to show you how to build an OpenLayers 3 map that animates emojis according to the query entered and location tracked from the LokLak Search API.

We start with a simple map using just one background layer in a clean style.

var map = new ol.Map({
target: 'map',  // The DOM element that will contains the map
renderer: 'canvas', // Force the renderer to be used
layers: [
// Add a new Tile layer getting tiles from OpenStreetMap source
new ol.layer.Tile({
    source: new ol.source.OSM()
}),
vectorLayer
],
// Create a view centered on the specified location and zoom level
view: new ol.View({
    center: ol.proj.transform([2.1833, 41.3833], 'EPSG:4326', 'EPSG:3857'),
    zoom: 2
})
});

 

Sample Output which displays map:

The data set for the locations of tweets containing emoji in them are tracked using search API of LokLak, which is in the form of simplified extract as JSON file. The file contains a list of coordinates named as location_point, the coordinate consists of lat and long values. With the coordinates, we will create a circle point i.e.,marker on the map showing where the emoji have been recently used from the tweets posted.

In the callback of the AJAX request we loop through the list of coordinates. The coordinate of the resulting line string are in EPSG:4326. Usually, when loading vector data with a different projection, OpenLayers will automatically re-project the geometries to the projection of the map. Because we are loading loading the data ourself, we manually have to transform the line to EPSG:3857. Then we could add the feature to the vector source.

for(var i = 0; i < tweets.statuses.length; i++) {
        if(tweets.statuses[i].location_point !== undefined){
            // Creation of the point with the tweet's coordinates
            //  Coords system swap is required: OpenLayers uses by default
            //  EPSG:3857, while loklak's output is EPSG:4326
            var point = new ol.geom.Point(ol.proj.transform(tweets.statuses[i].location_point, 'EPSG:4326', 'EPSG:3857'));
            vectorSource.addFeature(new ol.Feature({  // Add the point to the data vector
                geometry: point
            }));
        }
    }
});

 

Markers on the Map:

We can also style the markers which gets rendered onto the map using the feature ol.style.Style provided by OpenLayers.

var style = new ol.style.Style({
    stroke: new ol.style.Stroke({
        color: [64, 200, 200, 0.5],
        width: 5
    }),
    text: new ol.style.Text({
        font: '30px sans-serif',
        text: document.getElementById('searchField').value !== '' ? document.getElementById('searchField').value : '', //any text can be given here
        fill: new ol.style.Fill({
            color: [64, 64, 64, 0.75]
        })
    })
});

 

Styled Markers on the Map:

So these were a few tips and tricks to use the interactive OpenLayers3 Maps.

The full code of the example is available here.

Resources:

Continue ReadingOpenLayers 3 Map that Animates Emojis Using LokLak API

How to Geolocate Tweets for Emoji Heatmapper App using LokLak API

Geolocating tweets i.e., getting the location of the tweets posted, is one of the major task in the app Emoji Heatmapper. As I have to plot a marker on the map according to the query searched and the location tracked from the tweet containing that query. This is easily done in the app using the LokLak Search API.
Social media, such as Twitter, are filled with millions of micro messages by people around the world. My geolocation task is to get the geographic location with a Twitter message (having emoji for Emoji Heatmapper) based on the information we have about the user and the message. Twitter provides various data APIs, and tweets are obtained as JSON objects that include the tweet text along with metadata, such as location coordinates (lat and long).

Solutions

So there are few ways to geolocate tweets:

  • Place object from tweets Tweets usually delivered by the Twitter API include a JSON “Place” object which has a location associated with the tweet. These can have fields such as the country and city related to the place, as well as lat and long coordinates. Twitter Users have the option to tag their twitter statuses i.e., tweets with a place; the tagging can also be done automatically based on matches to the user’s current GPS position, if the user allows this (if GPS is switched ON). For the tweets containing place objects, the geolocation has already been done by Twitter.
  • Coordinates from tweets Some tweets are tagged with the coordinates (lat and long) of the user when the message was written, based on the user’s current GPS position. Reverse geocoding using APIs from Google Maps can be done to know the detailed information about the place.
  • Location from user profile Many users provide a location in their profile, a field with values such as “IND”. These locations are mostly always the same(i.e.,static), corresponding to the user’s primary location rather than the location at the time of the message posting. We assume that the users are not avid travelers.
  • Content-based Geo-location Geo-location can also be done on a message or set of messages based on the textual content of the messages.A user’s primary location can be detected based on their dialect or the mention of regional issues like sports teams, for example, as well as the mention of landmarks.

Implementation

So, here comes about the usage of LokLak search API in Emoji Heatmapper app:
I have used the LokLak Search API to get the tweets, which contains the query(i.e., emoji) which is being searched for and the location of the tweets. In LokLak we perform geolocation using location information from tweet metadata and user profiles.
Try this simple query:

http://loklak.org/api/search.json?q=😄

Which shows the data related to search query in which we also have geolocation information. Below is the output of the query, in which a single Twitter status is displayed which has the search query in it and the highlighted part shows the information about the location:

 {
      "timestamp": "2017-06-01T17:46:41.874Z",
      "created_at": "2017-06-01T17:46:25.000Z",
      "screen_name": "CerdaEnterprise",
      "text": "\"These boots were made for wearing... & that's just what we'll do.\" 🎶😄 Coral + guest pins = 💟 http://Www.TfjByCei.com https://pic.twitter.com/469uVWumSN",
      "link": "https://twitter.com/CerdaEnterprise/status/870335741561151488",
      "id_str": "870335741561151488",
      "canonical_id": "",
      "parent": "",
      "source_type": "TWITTER",
      "provider_type": "SCRAPED",
      "retweet_count": 0,
      "favourites_count": 0,
      "place_name": "Made",
      "place_id": "",
      "text_length": 155,
      "place_context": "ABOUT",
      "place_country": "Netherlands",
      "place_country_code": "NL",
      "place_country_center": [
        -3.5756949584817903,
        26.74012558382941
      ],
      "location_point": [
        4.7930598188159195,
        51.67666995510302
      ],
      "location_radius": 0,
      "location_mark": [
        4.793226608645116,
        51.676940788684036
      ],
      "location_source": "ANNOTATION",
      "hosts": [
        "www.tfjbycei.com",
        "pic.twitter.com"
      ],
      "hosts_count": 2,
      "links": [
        "http://Www.TfjByCei.com",
        "https://pic.twitter.com/469uVWumSN"
      ],
      "links_count": 2,
      "unshorten": {},
      "images": [
        "https://pbs.twimg.com/media/DBQNqkDUQAAkntY.jpg",
        "https://pic.twitter.com/469uVWumSN"
      ],
      "images_count": 2,
      "audio": [],
      "audio_count": 0,
      "videos": [],
      "videos_count": 0,
      "mentions": [],
      "mentions_count": 0,
      "hashtags": [],
      "hashtags_count": 0,
      "classifier_language": "english",
      "classifier_language_probability": 3.246518732845919E-16,
      "without_l_len": 96,
      "without_lu_len": 96,
      "without_luh_len": 96,
      "user": {
        "appearance_first": "2017-06-01T17:46:41.874Z",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/769877281497944064/PgsRK9C7_bigger.jpg",
        "screen_name": "CerdaEnterprise",
        "user_id": "529052548",
        "name": "The Fashion Junkies",
        "appearance_latest": "2017-06-01T17:46:41.874Z"
      }
    }

As you can see the above JSON output from the “Search API of LokLak”, the location we get in the JSON output i.e., location_mark/location_point is used to plot the point on the map. If in case the location from where the tweet posted is missing, location from the user profile is used.

Location is pointed with a marker on the map as shown below:

You can find the information about Emoji Heatmapper app here and even can try it out:

http://apps.loklak.org/details.html?q=emojiHeatmapper.

Resources

Continue ReadingHow to Geolocate Tweets for Emoji Heatmapper App using LokLak API

Using LokLak to Scrape Profiles from Quora, GitHub, Weibo and Instagram

Most of us are really so curious to know about one’s social life. So taking this as a key point, LokLak has many profile scrapers in it. Profile scraper which are now available in LokLak  helps us to know about the posts, followers one has. Few of the profile scrapers available in LokLak are Quora Profile, GitHub Profile, Weibo Profile and Instagram Profile.

How do the scrapers work?

In loklak now we are using java to get the json objects of the scraped profile from different websites as mentioned above. So here is a simple explanation how one of the scraper works. In this post I am going to give you a gist about how Github Profile scraper API works:

In the github profile scraper one can search for a profile without logging in and know the contents like the followers, repositories, gists of that profile and many more.

The simple query which can be used is:

To scrape individual profiles:

https://loklak.org/api/githubprofilescraper.json?profile=kavithaenair

To scrape organization profiles:

https://loklak.org/api/githubprofilescraper.json?profile=fossasia

Jsoup is an API and it is a easiest way used by java developers for scraping the web i.e.,web scraping. This API is used for manipulating and extracting data using DOM, CSS like methods. So in here, the Jsoup API is helping us to extract the html data and with the help of the tags used in the html extracted data we are trying to get the relevant data which is needed.

How do we get the matching elements?

We here are using special methods like getElementsByAttributeValueContaining() of the org.jsoup.nodes.Element class to get the data. For instance, to get the email from the extracted data the code is written as:

String email = html.getElementsByAttributeValueContaining("itemprop", "email").text();
            if (!email.contains("@"))
                  email = "";
            githubProfile.put("email", email);

Code:

Here is the java code which imports and extracts data:

Imports the html file:

html = Jsoup.connect("https://github.com/" + profile).get();

Extracts the html file for individual user:

/*If Individual*/
           if (html.getElementsByAttributeValueContaining("class", "user-profile-nav").size() != 0) {
               scrapeGithubUser(githubProfile, terms, profile, html);
           }
           if (terms.contains("gists") || terms.contains("all")) {
               String gistsUrl = GITHUB_API_BASE + profile + GISTS_ENDPOINT;
               JSONArray gists = getDataFromApi(gistsUrl);
               githubProfile.put("gists", gists);
           }
           if (terms.contains("subscriptions") || terms.contains("all")) {
               String subscriptionsUrl = GITHUB_API_BASE + profile + SUBSCRIPTIONS_ENDPOINT;
               JSONArray subscriptions = getDataFromApi(subscriptionsUrl);
               githubProfile.put("subscriptions", subscriptions);
           }
           if (terms.contains("repos") || terms.contains("all")) {
               String reposUrl = GITHUB_API_BASE + profile + REPOS_ENDPOINT;
               JSONArray repos = getDataFromApi(reposUrl);
               githubProfile.put("repos", repos);
           }
           if (terms.contains("events") || terms.contains("all")) {
               String eventsUrl = GITHUB_API_BASE + profile + EVENTS_ENDPOINT;
               JSONArray events = getDataFromApi(eventsUrl);
               githubProfile.put("events", events);
          }
          if (terms.contains("received_events") || terms.contains("all")) {
              String receivedEventsUrl = GITHUB_API_BASE + profile + RECEIVED_EVENTS_ENDPOINT;
              JSONArray receivedEvents = getDataFromApi(receivedEventsUrl);
              githubProfile.put("received_events", receivedEvents);
          }

Extracts the html file for organization:

/*If organization*/
if (html.getElementsByAttributeValue("class", "orgnav").size() != 0) {
    scrapeGithubOrg(profile, githubProfile, html);
}

And this is the sample output:

For query: https://loklak.org/api/githubprofilescraper.json?profile=kavithaenair
 
{
  "data": [{
    "joining_date": "2016-04-12",
    "gists_url": "https://api.github.com/users/kavithaenair/gists",
    "repos_url": "https://api.github.com/users/kavithaenair/repos",
    "user_name": "kavithaenair",
    "bio": "GSoC'17 @loklak @fossasia ; Developer @fossasia ; Intern @amazon",
    "subscriptions_url": "https://api.github.com/users/kavithaenair/subscriptions",
    "received_events_url": "https://api.github.com/users/kavithaenair/received_events",
    "full_name": "Kavitha E Nair",
    "avatar_url": "https://avatars0.githubusercontent.com/u/18421291",
    "user_id": "18421291",
    "events_url": "https://api.github.com/users/kavithaenair/events",
    "organizations": [
      {
        "img_link": "https://avatars1.githubusercontent.com/u/6295529?v=3&s=70",
        "link": "https://github.com/fossasia",
        "label": "fossasia",
        "img_Alt": "@fossasia"
      },
      {
        "img_link": "https://avatars0.githubusercontent.com/u/10620750?v=3&s=70",
        "link": "https://github.com/coala",
        "label": "coala",
        "img_Alt": "@coala"
      },
      {
        "img_link": "https://avatars1.githubusercontent.com/u/11370631?v=3&s=70",
        "link": "https://github.com/loklak",
        "label": "loklak",
        "img_Alt": "@loklak"
      },
      {
        "img_link": "https://avatars2.githubusercontent.com/u/24720168?v=3&s=70",
        "link": "https://github.com/bvrit-wise-django-team",
        "label": "bvrit-wise-django-team",
        "img_Alt": "@bvrit-wise-django-team"
      }
    ],
    "home_location": "\n    Hyderabad, India\n",
    "works_for": "",
    "special_link": "https://www.overleaf.com/read/ftnvcphnwzhp",
    "email": "",
    "atom_feed_link": "https://github.com/kavithaenair.atom"
  }],
  "metadata": {"count": 1},
  "session": {"identity": {
    "type": "host",
    "name": "162.158.46.18",
    "anonymous": true
  }}
}
Continue ReadingUsing LokLak to Scrape Profiles from Quora, GitHub, Weibo and Instagram