Autolinker Component in Loklak Search

In Loklak Search the post items contain links, which are either internal or external. These links include the hashtags, mentions, and URLs. From the backend server we just received the message in the plain text format, and thus there is need to parse the plain text and render it as clickable links. These clickable links can be either internal or external. Thus we need an auto-linker component, which takes the text and render it as links.

The API of the Component

The component takes as the input the plain text, then four arrays of strings. Each containing the text to be linked. These are hashtags, mentions, links and the unshorten attribute which is used to unshorten the shortened URLs in the post. These attributes are used by the component to render the text in the appropriate format.

export class FeedLinkerComponent implements OnInit {
@Input() text: string;
@Input() hashtags: string[] = new Array<string>();
@Input() mentions: string[] = new Array<string>();
@Input() links: string[] = new Array<string>();
@Input() unshorten: Object = {};

The Logic of the Component

The basic logic of the component works as the following, we divide the text into chunks known as shards, we have three basic data structures for the component to work

  • The ShardType which is the type of the chunk it specifies whether it is plain, hashtags, mentions, and links.
  • The Shard which is the simple object containing the text to show, its type and the link it refers to

The StringIndexdChunks, they are utilized to index the chunks in the order in which they appear in the text.

const enum ShardType {
plain, // 0
link, // 1
hashtag, // 2
mention // 3

class Shard {
constructor (
public type: ShardType = ShardType.plain,
public text: String = '',
public linkTo: any = null,
public queryParams: any = null
) { }

interface StringIndexedChunks {
index: number;
str: string;
type: ShardType;

First we have a private method of the component which searches for all the elements (strings) in the text. Here we have an array which maintains the index of those chunks in the text.

private generateShards() {
const indexedChunks: StringIndexedChunks[] = [];

this.hashtags.forEach(hashtag => {
const indices = getIndicesOf(this.text, `#${hashtag}`, false);
indices.forEach(idx => {
indexedChunks.push({index: idx, str: `#${hashtag}`, type: ShardType.hashtag});

this.mentions.forEach(mention => {
const indices = getIndicesOf(this.text, `@${mention}`, false);
indices.forEach(idx => {
indexedChunks.push({index: idx, str: `@${mention}`, type: ShardType.mention});

Then we sort the chunks according to their indexes in the text. This gives us sorted array which consists of all the chunks sorted according to the indexes as they appear in the text.

indexedChunks.sort((a, b) => { return (a.index > b.index) ? 1 : (a.index < b.index) ? -1 : 0; });

The next part of the logic is to generate the shard array, an array which contains each chunk, once. To do this we iterate over the Sorted Indexed array created in the previous step and use it split the text into chunks. We iterate over the text and take substrings using the indexes of each element.

let startIndex = 0;
const endIndex = this.text.length;

indexedChunks.forEach(element => {
if (startIndex !== element.index) {
const shard = new Shard(ShardType.plain, this.text.substring(startIndex, element.index));
startIndex = element.index;
if (startIndex === element.index) {
const str = this.text.substring(startIndex, element.index + element.str.length);
const shard = new Shard(element.type, str);
switch (element.type) {
case {
if (this.unshorten[element.str]) {
shard.linkTo = str;
shard.text = this.unshorten[element.str];
else {
shard.linkTo = str;

case ShardType.hashtag: {
shard.linkTo = ['/search'];
shard.queryParams = { query : str };

case ShardType.mention: {
shard.linkTo = ['/search'];
shard.queryParams = { query : `from:${str.substring(1)}` };
startIndex += element.str.length;

if (startIndex !== endIndex) {
const shard = new Shard(ShardType.plain, this.text.substring(startIndex));

After this we have generated the chunks of the text, now the only task is to write the view of the component which uses this Shard Array to render the linked elements.

<div class="textWrapper">
<span *ngFor="let shard of shardArray">
<span *ngIf="shard.type === 0"> <!-- Plain -->
<span *ngIf="shard.type === 1"> <!-- URL Links -->
<span *ngIf="shard.type === 2"> <!-- Hashtag -->
<a [routerLink]="shard.linkTo" [queryParams]="shard.queryParams">{{shard.text}}</a>
<span *ngIf="shard.type === 3"> <!-- Mention -->
<a [routerLink]="shard.linkTo" [queryParams]="shard.queryParams">{{shard.text}}</a>
  • This renders the chunks and handles the links of both internal and external type.
  • It also also makes sure that the links get unshortened properly using the unshorten API property.
  • Uses routerLink, angular property to link in application URLs, for asynchronous reloading while clicking links.

Resources and Links

This component is inspired from the two main open source libraries.

Earlier these libraries were used in the project, but as the need of unshortening and asynchronous linking appeared in the application, a custom implementation was needed to be implemented.

Deleting Meilix Github Releases

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. Meilix-Generator is the webapp which uses Meilix to generate ISO and deploy it on Meilix Github Release. Then the webapp mail the link of the ISO to the user.
Increasing number of ISO will increase the number of releases which results in dirty looking of Meilix repository. So we need to delete older releases after certain interval of time to make the repository release page looks good and decrease unwanted space.
This script will do this work for us.

#!/usr/bin/env bash
set -e
echo "This is a script to delete obsolete meilix iso builds by Abishek V Ashok"
echo "You have to add an authorization token to make it functional."

# jq is the JSON parser we will be using
sudo apt-get -y install jq

# Storing the response to a variable for future usage
response=`curl | jq '.[] | .id, .published_at'`

index=1  # when index is odd, $i contains id and when it is even $i contains published_date
delete=0 # Should we delete the release?
current_year=`date +%Y`  # Current year eg) 2001
current_month=`date +%m` # Current month eg) 2
current_day=`date +%d`   # Current date eg) 24

for i in $response; do
    if [ $((index % 2)) -eq 0 ]; then # We get the published_date of the release as $i's value here

        if [ $published_year -lt $current_year ]; then
             let "delete=1"
            if [ $published_month -lt $current_month ]; then
                let "delete=1"
                if [ $((current_day-$published_day)) -gt 10 ]; then
                    let "delete=1"
    else # We get the id of the release as $i`s value here
        if [ $delete -eq 1 ]; then
            curl -X DELETE -H "Authorization: token $KEY"$i
            let "delete=0"
    let "index+=1"

This code uses Github API to curl the Meilix releases. Github API is very useful in providing lots of information but here we are only concerned with the release date and time of the build.
Then we setup a condition if that satisfies then the release will automatically will get deleted.

For taking care of the authentication, a token has been uploaded to the Travis settings of Meilix of FOSSASIA.

The personal token has been generated by a user with write access to the repository with repo scope token.

This sort out the issue of having bulk of releases in the Meilix repository of FOSSASIA.

Users Github API  by REST API v3
Repo Github API   by REST API v3

Adding additional information to store listing page of Loklak apps site

Loklak apps site has now got a completely functional store listing page where users can find all relevant information about the app which they want to view. The page has a left side bar which shows various categories to switch between, a right sidebar for suggesting similar kind of apps to users and a middle section to provide users with various important informations about the app like getting started, use of app, promo images, preview images, test link and various other details. In this blog I will be describing how the bottom section of the middle column has been created (related issue: #209).

The bottom section

The bottom section provides various informations like updated, version, app source, developer information, contributors, technology stack, license. All these informations has to be dynamically loaded for each selected app. As I had previously mentioned here, no HTML content can be hard coded in the store listing page. So how do we show the above mentioned informations for the different apps? Well, for this we will once again use the app.json of the corresponding app like we had done for the middle section here.

At first, for a given app we need to define some extra fields in the app.json file as shown below.

"appSource": "",
  "contributors": [{"name": "djmgit", "url": ""}],
  "techStack": ["HTML", "CSS", "AngularJs", "Morris.js", "Bootstrap", "Loklak API"],
  "license": {"name": "LGPL 2.1", "url": ""},
  "version": "1.0",
  "updated": "June 10,2017",

The above code snippet shows the new fields included in app.json. The fields are as described below.

  • appSource: Stores link to the source code of the app.
  • Contributors: Stores a list containing objects. Each object stores name of the contributor and an url corresponding to that contributor.
  • techStack: A list containing names of the technologies used.
  • License: Name and link of the license.
  • Version: The current version of the app.
  • Updated: Date on which the app was last updated.

These fields provide the source for the informations present in the bottom section of the app.

Now we need to render these information on the store listing page. Let us take an example. Let us see how version is rendered.

<div ng-if="appData.version !== undefined && appData.version !== ''" class="col-md-4 add-info">
                  <div class="info-type">
                    <h5 class="info-header">
                  <div class="info-body">

We first check if version field is defined and version is not empty. Then we print a header (Version in this case) and then we print the value. This is how updated, appSource and license are also displayed. What about technology stack and contributors? Technology stack is basically an list and it may contain quite a number of strings(technology names). If we display all the values at once the bottom section will get crowded and it may degrade the UI of the page.To avoid this a popup dialog has been used. When user clicks on the technology stack label, a popup dialogue appears which shows the various technologies used in the app.

<div class="info-body">
                    <div class="dropdown">
                      <div class="dropdown-toggle" type="button" data-toggle="dropdown">
                        View technology stack
                      <ul class="dropdown-menu">
                        <li ng-repeat="item in appData.techStack" class="tech-item">

After displaying a header, we iterate over the techStack list and populate our popup dialogue. This popup dialogue is attached to the label ‘View technology stack‘. Whenever a user clicks on this label, the popup is shown. The same technique technique is also applied for rendering contributors. A popup dialogue is used to display all the contributors. Thus technology stack and contributors list is shown only on demand.

For developer information, name of the developer is shown which is linked to his/her website and there is an option to send email or copy email id if present.

<div class="info-body">
                    <span ng-if=" !== undefined && !== ''">
                      <a href="{{}}"> {{}} </a>
                    <a ng-if=" !== undefined && !== ''" class="mail"
                      <span class="glyphicon glyphicon-envelope"></span>

For email id, bootstrap’s email glyphicon is used along with a mailto link pointing to the developer’s email id. What does mailto do? It simply opens your default mail client. For example if you are on linux, it might open Thunderbird. If you do not have a mail client installed, but your default browser is google chrome, it will open gmail mail composer. If you are viewing the site on android device, it will open gmail app directly.

The bottom section can be viewed here.

Important resources


Discount Codes in Open Event Server

The Open Event System allows usage of discount codes with tickets and events. This blogpost describes what types of discount codes are present and what endpoints can be used to fetch and update details.

In Open Event API Server, each event can have two types of discount codes. One is ‘event’ discount code, while the other is ‘ticket’ discount code. As the name suggests, the event discount code is an event level discount code and the ticket discount code is ticket level.

Now each event can have only one ‘event’ discount code and is accessible only to the server admin. The Open Event server admin can create, view and update the ‘event’ discount code for an event. The event discount code followsDiscountCodeEvent Schema. This schema is inherited from the parent class DiscountCodeSchemaPublic. To save the unique discount code associated with an event, the event model’s discount_code_id field is used.

The ‘ticket’ discount is accessible by the event organizer and co-organizer. Each event can have any number of ‘ticket’ discount codes. This follows the DiscountCodeTicket schema, which is also inherited from the same base class ofDiscountCodeSchemaPublic. The use of the schema is decided based on the value of the field ‘used_for’ which can have the value either ‘events’ or ‘tickets’. Both the schemas have different relationships with events and marketer respectively.

We have the following endpoints for Discount Code events and tickets:

The first endpoint is based on the DiscountCodeDetail class. It returns the detail of one discount code which in this case is the event discount code associated with the event.

The second endpoint is based on the DiscountCodeList class which returns a list of discount codes associated with an event. Note that this list also includes the ‘event’ discount code, apart from all the ticket discount codes.

class DiscountCodeFactory(factory.alchemy.SQLAlchemyModelFactory):
   class Meta:
       model = DiscountCode
       sqlalchemy_session = db.session
event_id = None
user = factory.RelatedFactory(UserFactory)
user_id = 1

Since each discount code belongs to an event(either directly or through the ticket), the factory for this has event as related factory, but to check for 
/events/<int:event_id>/discount-code endpoint we first need the event and then pass the discount code id to be 1 for dredd to check this. Hence, event is not included as a related factory, but added as a different object every time a discount code object is to be used.

@hooks.before("Discount Codes > Get Discount Code Detail of an Event > Get Discount Code Detail of an Event")
def event_discount_code_get_detail(transaction):
   GET /events/1/discount-code
   :param transaction:
   with stash['app'].app_context():
       discount_code = DiscountCodeFactory()
       event = EventFactoryBasic(discount_code_id=1)

The other tests and extended documentation can be found 


Marker Click Management in Android Google Map API Version 2

We could display a marker on Google map to point to a particular location. Although it is a simple task sometimes we need to customise it a bit more. Recently I customised marker displayed in Connfa app displaying the location of the sessions on the map loaded from Open Event format. In this blog manipulation related to map marker is explored.

Markers indicate single locations on the map. You can customize your markers by changing the default colour, or replace the marker icon with a custom image. Info windows can provide additional context to a marker. You can place a marker on the map by using following code.

MarkerOptions marker = new MarkerOptions().position(new LatLng(latitude, longitude)).title("Dalton Hall");

But as you can see this may not be enough, we need to do operations on clicking the marker too, so we define them in the Marker Click Listener. We declare marker null initially so we check if the marker colour is changed previously or not.

private Marker previousMarker = null;

We check if the marker is initialized to change its colour again to initial colour, we can do other related manipulation like changing the map title here,

Note: the first thing that happens when a marker is clicked or tapped is that any currently showing info window is closed, and the GoogleMap.OnInfoWindowCloseListener is triggered. Then the OnMarkerClickListener is triggered. Therefore, calling isInfoWindowShown() on any marker from the OnMarkerClickListener will return false.

mGoogleMap.setOnMarkerClickListener(new GoogleMap.OnMarkerClickListener() {
   public boolean onMarkerClick(Marker marker) {
       String locAddress = marker.getTitle();
       if (previousMarker != null) {
       previousMarker = marker;

       return true;

It’s possible to customize the colour of the default marker image by passing a BitmapDescriptor object to the icon() method. You can use a set of predefined colours in the BitmapDescriptorFactory object, or set a custom marker colour with the BitmapDescriptorFactory.defaultMarker(float hue) method. The hue is a value between 0 and 360, representing points on a colour wheel. We use red colour when the marker is not clicked and blue when it is clicked so a user knows which one is clicked.

To conclude you can use an OnMarkerClickListener to listen for click events on the marker. To set this listener on the map, call GoogleMap.setOnMarkerClickListener(OnMarkerClickListener). When a user clicks on a marker, onMarkerClick(Marker) will be called and the marker will be passed through as an argument. This method returns a boolean that indicates whether you have consumed the event (i.e., you want to suppress the default behaviour). If it returns false, then the default behaviour will occur in addition to your custom behaviour. The default behaviour for a marker click event is to show its info window (if available) and move the camera such that the marker is centered on the map.

The final result looks like this, so you the user can see which marker is clicked as its colour is changed,




  • Google Map APIs Documentation –

Link Preview Service from SUSI Server

 SUSI Webchat, SUSI Android app, SUSI iOS app are various SUSI clients which depend on response from SUSI Server. The most common response of SUSI Server is in form of links. Clients usually need to show the preview of the links to the user. This preview may include featured image, description and the title of the link.  Clients show this information by using various 3rd party APIs and libraries. We planned to create an API endpoint for this on SUSI Server to give the preview of the link. This service is called LinkPreviewService.
String url = post.get("url", "");
        if(url==null || url.isEmpty()){
            jsonObject.put("message","URL Not given");
            return new ServiceResponse(jsonObject);

This API Endpoint accept only 1 get parameter which is the URL whose preview is to be shown.

Here we also check if no parameter or wrong URL parameter was sent. If that was the the case then we return an error message to the user.

 SourceContent sourceContent =     TextCrawler.scrape(url,3);
        if (sourceContent.getImages() != null) jsonObject.put("image", sourceContent.getImages().get(0));
        if (sourceContent.getDescription() != null) jsonObject.put("descriptionShort", sourceContent.getDescription());
        if(sourceContent.getTitle()!=null)jsonObject.put("title", sourceContent.getTitle());
        return new ServiceResponse(jsonObject);

The TextCrawler function accept two parameters. One is the url of the website which is to be scraped for the preview data and the other is depth. To get the images, description and title there are methods built in. Here we just call those methods and set them in our JSON Object.

 private String htmlDecode(String content) {
        return Jsoup.parse(content).text();

Text Crawler is based on Jsoup. Jsoup is a java library that is used to scrape HTML pages.

To get anything from Jsoup we need to decode the content of HTML to Text.

public List<String> getImages(Document document, int imageQuantity) {
        Elements media ="[src]");
        while(var5.hasNext()) {
            Element srcElement = (Element);
            if(srcElement.tagName().equals("img")) {

 The getImages method takes the HTML document from the JSoup and find the image tags in that. We have given the imageQuantity parameter in the function, so accordingly it returns the src attribute of the first n images it find.

This API Endpoint can be seen working on<ANY URL>

A real working example of this endpoint would be


Web Crawlers:


JSoup Api Docs:

Parsing HTML with JSoup:

Deleting SUSI Skills from Server

SUSI Skill CMS is a web application to create and edit skills. In this blog post I will be covering how we made the skill deleting feature in Skill CMS from the SUSI Server.
The deletion of skill was to be made in such a way that user can click a button to delete the skill. As soon as they click the delete button the skill is deleted it is removed from the directory of SUSI Skills. But admins have an option to recover the deleted skill before completion of 30 days of deleting the skill.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.


We need to move the skill to a directory called deleted_skills_dir. So we check if the directory exists or not. If it not exists then we create a directory for the deleted skills.

  if (skill.exists()) {
   File file = new File(DAO.deleted_skill_dir.getPath()+path);
   Boolean changed =  new File(DAO.deleted_skill_dir.getPath()+path).setLastModified(System.currentTimeMillis());

This is the part where the real deletion happens. We get the path of the skill and rename that to a new path which is in the directory of deleted skills.

Also here change the last modified time of the skill as the current time. This time is used to check if the skill deleted is older than 30 days or not.

    try (Git git = DAO.getGit()) {
                DAO.pushCommit(git, "Deleted " + skill_name, rights.getIdentity().isEmail() ? rights.getIdentity().getName() : "[email protected]");
                json.put("accepted", true);
                json.put("message", "Deleted " + skill_name);
            } catch (IOException | GitAPIException e) {

Finally we add the changes to Git. DAO.pushCommit pushes to commit to the Susi Skill Data repository. If the user is logged in we get the email of the user and set that email as the commit author. Else we set the username “[email protected]”.

Then in the caretaker class there is a method deleteOldFiles that checks for all the files whose last modified time was older than 30 days. If there is any file whose last modified time was older than 30 days then it quietly delete the files.

public void deleteOldFiles() {
     Collection<File> filesToDelete = FileUtils.listFiles(new         File(DAO.deleted_skill_dir.getPath()),
            TrueFileFilter.TRUE);    // include sub dirs
        for (File file : filesToDelete) {
               boolean success = FileUtils.deleteQuietly(file);
            if (!success) {
                System.out.print("Deleted skill older than 30 days.");

To test this API endpoint, we need to call http://localhost:4000/cms/deleteSkill.txt?model=general&group=Knowledge&language=en&skill=<skill_name>


JGit Documentation:

Commons IO:

Age Filter:

JGit User Guide:

JGit Repository access:

Getting SUSI Skill at a Commit ID

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID.

In this blog post I will tell about making an API endpoint which works similar to git show.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");
        String commitID  = call.get("commitID", null);

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution.

    Repository repository = DAO.getRepository();
    ObjectId CommitIdObject = repository.resolve(commitID);

Then we get the git repository of the skill from the DAO and initialize the repository object.

From the commitID that we got in the request parameters we create a CommitIdObject.

   (RevWalk revWalk = new RevWalk(repository)) {
   RevCommit commit = revWalk.parseCommit(CommitIdObject);
   RevTree tree = commit.getTree();

Now using commit’s tree, we will find the find the path and get the tree of the commit.

From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders.

                revWalk = new RevWalk(repository)) {
                try (TreeWalk treeWalk = new TreeWalk(repository)) {
                    if (! {
                        throw new IllegalStateException("Did not find expected file");

If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID.

       ObjectId objectId = treeWalk.getObjectId(0);
       ObjectLoader loader =;
       OutputStream output = new OutputStream();

And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it.


This Servlet can be seen working


JGit Documentation:

JGit User Guide:

JGit Repository access:

JGit Github: