Setting Loklak Server with SSL

Loklak Server is based on embedded Jetty Server which can work both with or without SSL encryption. Lately, there was need to setup Loklak Server with SSL. Though the need was satisfied by CloudFlare. Alternatively, there are 2 ways to set up Loklak Server with SSL. They are:-

1) Default Jetty Implementation

There is pre-existing implementation of Jetty libraries. The http mode can be set in configuration file. There are 4 modes on which Loklak Server can work: http mode, https mode, only https mode and redirect to https mode. Loklak Server listens to port 9000 when in http mode and to port 9443 when in https mode.

There is also a need of SSL certificate which is to be added in configuration file.

2) Getting SSL certificate with Kube-Lego on Kubernetes Deployment

I got to know about Kube-Lego by @niranjan94. It is implemented in Open-Event-Orga-Server. The approach is to use:

a) Nginx as ingress controller

For setting up Nginx ingress controller, a yml file is needed which downloads and configures the server.

The configurations for data requests and response are:

proxy-connect-timeout: "15"
 proxy-read-timeout: "600"
 proxy-send-imeout: "600"
 hsts-include-subdomains: "false"
 body-size: "64m"
 server-name-hash-bucket-size: "256"
 server-tokens: "false"

Nginx is configured to work on both http and https ports in service.yml

ports:
- port: 80
  name: http
- port: 443
  name: https

 

b) Kube-Lego for fetching SSL certificates from Let’s Encrypt

Kube-Lego was set up with default values in yml. It uses the host-name, email address and secretname of the deployment to validate url and fetch SSL certificate from Let’s Encrypt.

c) Setup configurations related to TLS and no-TLS connection

These configuration files mentions the path and service ports for Nginx Server through which requests are forwarded to backend Loklak Server. Here for no-TLS and TLS requests, the requests are directly forwarded to localhost at port 80 of Loklak Server container.

rules:
- host: staging.loklak.org
  http:
  paths:
  - path: /
    backend:
    serviceName: server
    servicePort: 80

For TLS requests, the secret name is also mentioned. Kube-Lego fetches host-name and secret-name from here for the certificate

tls:
- hosts:
- staging.loklak.org
  secretName: loklak-api-tls

d) Loklak Server, ElasticSearch and Mosquitto at backend

These containers work at backend. ElasticSearch and Mosquitto are only accessible to Loklak Server. Loklak Server can be accessed through Nginx server. Loklak Server is configured to work at http mode and is exposed at port 80.

ports:
- port: 80
  protocol: TCP
  targetPort: 80

To deploy the Loklak Server, all these are deployed in separate pods and they interact through service ports. To deploy, we use deploy script:

# For elasticsearch, accessible only to api-server
kubectl create -R -f ${path-to-config-file}/elasticsearch/

# For mqtt, accessible only to api-server
kubectl create -R -f ${path-to-config-file}/mosquitto/

# Start KubeLego deployment for TLS certificates
kubectl create -R -f ${path-to-config-file}/lego/
kubectl create -R -f ${path-to-config-file}/nginx/

# Create web namespace, this acts as bridge to Loklak Server
kubectl create -R -f ${path-to-config-file}/web/

# Create API server deployment and expose the services
kubectl create -R -f ${path-to-config-file}/api-server/

# Get the IP address of the deployment to be used
kubectl get services --namespace=nginx-ingress

References

Continue ReadingSetting Loklak Server with SSL

Link Preview Service from SUSI Server

 SUSI Webchat, SUSI Android app, SUSI iOS app are various SUSI clients which depend on response from SUSI Server. The most common response of SUSI Server is in form of links. Clients usually need to show the preview of the links to the user. This preview may include featured image, description and the title of the link.  Clients show this information by using various 3rd party APIs and libraries. We planned to create an API endpoint for this on SUSI Server to give the preview of the link. This service is called LinkPreviewService.
String url = post.get("url", "");
        if(url==null || url.isEmpty()){
            jsonObject.put("message","URL Not given");
            jsonObject.put("accepted",false);
            return new ServiceResponse(jsonObject);
        }

This API Endpoint accept only 1 get parameter which is the URL whose preview is to be shown.

Here we also check if no parameter or wrong URL parameter was sent. If that was the the case then we return an error message to the user.

 SourceContent sourceContent =     TextCrawler.scrape(url,3);
        if (sourceContent.getImages() != null) jsonObject.put("image", sourceContent.getImages().get(0));
        if (sourceContent.getDescription() != null) jsonObject.put("descriptionShort", sourceContent.getDescription());
        if(sourceContent.getTitle()!=null)jsonObject.put("title", sourceContent.getTitle());
        jsonObject.put("accepted",true);
        return new ServiceResponse(jsonObject);
    }

The TextCrawler function accept two parameters. One is the url of the website which is to be scraped for the preview data and the other is depth. To get the images, description and title there are methods built in. Here we just call those methods and set them in our JSON Object.

 private String htmlDecode(String content) {
        return Jsoup.parse(content).text();
    }

Text Crawler is based on Jsoup. Jsoup is a java library that is used to scrape HTML pages.

To get anything from Jsoup we need to decode the content of HTML to Text.

public List<String> getImages(Document document, int imageQuantity) {
        Elements media = document.select("[src]");
        while(var5.hasNext()) {
            Element srcElement = (Element)var5.next();
            if(srcElement.tagName().equals("img")) {
                ((List)matches).add(srcElement.attr("abs:src"));
            }
        }

 The getImages method takes the HTML document from the JSoup and find the image tags in that. We have given the imageQuantity parameter in the function, so accordingly it returns the src attribute of the first n images it find.

This API Endpoint can be seen working on

http://127.0.0.1:4000/susi/linkPreview.json?url=<ANY URL>

A real working example of this endpoint would be http://api.susi.ai/susi/linkPreview.json?url=https://techcrunch.com/2017/07/23/dear-tech-dudes-stop-being-such-idiots-about-women/

Resources:

Web Crawlers: https://www.promptcloud.com/data-scraping-vs-data-crawling/

JSoup: https://jsoup.org/

JSoup Api Docs: https://jsoup.org/apidocs/

Parsing HTML with JSoup: http://www.baeldung.com/java-with-jsoup

Continue ReadingLink Preview Service from SUSI Server

Deleting SUSI Skills from Server

SUSI Skill CMS is a web application to create and edit skills. In this blog post I will be covering how we made the skill deleting feature in Skill CMS from the SUSI Server.
The deletion of skill was to be made in such a way that user can click a button to delete the skill. As soon as they click the delete button the skill is deleted it is removed from the directory of SUSI Skills. But admins have an option to recover the deleted skill before completion of 30 days of deleting the skill.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

 if(!DAO.deleted_skill_dir.exists()){
            DAO.deleted_skill_dir.mkdirs();
   }

We need to move the skill to a directory called deleted_skills_dir. So we check if the directory exists or not. If it not exists then we create a directory for the deleted skills.

  if (skill.exists()) {
   File file = new File(DAO.deleted_skill_dir.getPath()+path);
   file.getParentFile().mkdirs();
   if(skill.renameTo(file)){
   Boolean changed =  new File(DAO.deleted_skill_dir.getPath()+path).setLastModified(System.currentTimeMillis());
     }

This is the part where the real deletion happens. We get the path of the skill and rename that to a new path which is in the directory of deleted skills.

Also here change the last modified time of the skill as the current time. This time is used to check if the skill deleted is older than 30 days or not.

    try (Git git = DAO.getGit()) {
                DAO.pushCommit(git, "Deleted " + skill_name, rights.getIdentity().isEmail() ? rights.getIdentity().getName() : "anonymous@");
                json.put("accepted", true);
                json.put("message", "Deleted " + skill_name);
            } catch (IOException | GitAPIException e) {

Finally we add the changes to Git. DAO.pushCommit pushes to commit to the Susi Skill Data repository. If the user is logged in we get the email of the user and set that email as the commit author. Else we set the username “anonymous@”.

Then in the caretaker class there is a method deleteOldFiles that checks for all the files whose last modified time was older than 30 days. If there is any file whose last modified time was older than 30 days then it quietly delete the files.

public void deleteOldFiles() {
     Collection<File> filesToDelete = FileUtils.listFiles(new         File(DAO.deleted_skill_dir.getPath()),
     new 
(DateTime.now().withTimeAtStartOfDay().minusDays(30).toDate()),
            TrueFileFilter.TRUE);    // include sub dirs
        for (File file : filesToDelete) {
               boolean success = FileUtils.deleteQuietly(file);
            if (!success) {
                System.out.print("Deleted skill older than 30 days.");
            }
      }
}

To test this API endpoint, we need to call http://localhost:4000/cms/deleteSkill.txt?model=general&group=Knowledge&language=en&skill=<skill_name>

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

Commons IO: https://commons.apache.org/proper/commons-io/

Age Filter: https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/filefilter/AgeFileFilter.html

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

Continue ReadingDeleting SUSI Skills from Server

Getting SUSI Skill at a Commit ID

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID.

In this blog post I will tell about making an API endpoint which works similar to git show.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");
        String commitID  = call.get("commitID", null);

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution.

    Repository repository = DAO.getRepository();
    ObjectId CommitIdObject = repository.resolve(commitID);

Then we get the git repository of the skill from the DAO and initialize the repository object.

From the commitID that we got in the request parameters we create a CommitIdObject.

   (RevWalk revWalk = new RevWalk(repository)) {
   RevCommit commit = revWalk.parseCommit(CommitIdObject);
   RevTree tree = commit.getTree();


Now using commit’s tree, we will find the find the path and get the tree of the commit.

From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders.

                revWalk = new RevWalk(repository)) {
                try (TreeWalk treeWalk = new TreeWalk(repository)) {
                    treeWalk.addTree(tree);
                    treeWalk.setRecursive(true);
                    treeWalk.setFilter(PathFilter.create(path));
                    if (!treeWalk.next()) {
                        throw new IllegalStateException("Did not find expected file");
                    }

If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID.

       ObjectId objectId = treeWalk.getObjectId(0);
       ObjectLoader loader = repository.open(objectId);
       OutputStream output = new OutputStream();
       loader.copyTo(output);

And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it.

       json.put("file",output);

This Servlet can be seen working api.susi.ai: http://api.susi.ai/cms/getFileAtCommitID.json?model=general&group=knowledge&language=en&skill=bitcoin&commitID=214791f55c19f24d7744364495541b685539a4ee

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

JGit Github: https://github.com/eclipse/jgit

Continue ReadingGetting SUSI Skill at a Commit ID

Introducing Stream Servlet in loklak Server

A major part of my GSoC proposal was adding stream API to loklak server. In a previous blog post, I discussed the addition of Mosquitto as a message broker for MQTT streaming. After testing this service for a few days and some minor improvements, I was in a position to expose the stream to outside users using a simple API.

In this blog post, I will be discussing the addition of /api/stream.json endpoint to loklak server.

HTTP Server-Sent Events

Server-sent events (SSE) is a technology where a browser receives automatic updates from a server via HTTP connection. The Server-Sent Events EventSource API is standardized as part of HTML5 by the W3C.

Wikipedia

This API is supported by all major browsers except Microsoft Edge. For loklak, the plan was to use this event system to send messages, as they arrive, to the connected users. Apart from browser support, EventSource API can also be used with many other technologies too.

Jetty Eventsource Plugin

For Java, we can use Jetty’s EventSource plugin to send events to clients. It is similar to other Jetty servlets when it comes to processing the arguments, handling requests, etc. But it provides a simple interface to send events as they occur to connected users.

Adding Dependency

To use this plugin, we can add the following line to Gradle dependencies –

compile group: 'org.eclipse.jetty', name: 'jetty-eventsource-servlet', version: '1.0.0'

[SOURCE]

The Event Source

An EventSource is the object which is required for EventSourceServlet to send events. All the logics for emitting events needs to be defined in the related class. To link a servlet with an EventSource, we need to override the newEventSource method –

public class StreamServlet extends EventSourceServlet {
    @Override
    protected EventSource newEventSource(HttpServletRequest request) {
        String channel = request.getParameter("channel");
        if (channel == null) {
            return null;
        }
        if (channel.isEmpty()) {
            return null;
        }
        return new MqttEventSource(channel);
    }
}

[SOURCE]

If no channel is provided, the EventSource object will be null and the request will be rejected. Here, the MqttEventSource would be used to handle the stream of Tweets as they arrive from the Mosquitto message broker.

Cross Site Requests

Since the requests to this endpoint can’t be of JSONP type, it is necessary to allow cross site requests on this endpoint. This can be done by overriding the doGet method of the servlet –

@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
     response.setHeader("Access-Control-Allow-Origin", "*");
    super.doGet(request, response);
}

[SOURCE]

Adding MQTT Subscriber

When a request for events arrives, the constructor to MqttEventSource is called. At this stage, we need to connect to the stream from Mosquitto for the channel. To achieve this, we can set the class as MqttCallback using appropriate client configurations –

public class MqttEventSource implements MqttCallback {
    ...
    MqttEventSource(String channel) {
        this.channel = channel;
    }
    ...
    this.mqttClient = new MqttClient(address, "loklak_server_subscriber");
    this.mqttClient.connect();
    this.mqttClient.setCallback(this);
    this.mqttClient.subscribe(this.channel);
    ...
}

[SOURCE]

By setting the callback to this, we can override the messageArrived method to handle the arrival of a new message on the channel. Just to mention, the client library used here is Eclipse Paho.

Connecting MQTT Stream to SSE Stream

Now that we have subscribed to the channel we wish to send events from, we can use the Emitter to send events from our EventSource by implementing it –

public class MqttEventSource implements EventSource, MqttCallback {
    private Emitter emitter;


    @Override
    public void onOpen(Emitter emitter) throws IOException {
        this.emitter = emitter;
        ...
    }

    @Override
    public void messageArrived(String topic, MqttMessage message) throws Exception {
        this.emitter.data(message.toString());
    }
}

[SOURCE]

Closing Stream on Disconnecting from User

When a client disconnects from the stream, it doesn’t makes sense to stay connected to the server. We can use the onClose method to disconnect the subscriber from the MQTT broker –

@Override
public void onClose() {
    try {
        this.mqttClient.close();
        this.mqttClient.disconnect();
    } catch (MqttException e) {
        // Log some warning 
    }
}

[SOURCE]

Conclusion

In this blog post, I discussed connecting the MQTT stream to SSE stream using Jetty’s EventSource plugin. Once in place, this event system would save us from making too many requests to collect and visualize data. The possibilities of applications of such feature are huge.

This feature can be seen in action at the World Mood Tracker app.

The changes were introduced in pull request loklak/loklak_server#1474 by @singhpratyush (me).

Resources

Continue ReadingIntroducing Stream Servlet in loklak Server

Integrating Swagger with SUSI Server

The goal of Swagger is to define a standard interface for REST APIs which allows humans to understand the capabilities of the APIs without access to source code or documentation.

SUSI Server is now completely API centric. As more and more people make their own bots using SUSI Server they will be needing documentation for the APIs. We can use swagger so that without looking at the javadocs or documentation people can consume the REST APIs.

In this I post will walk you through the steps to integrate Swagger with SUSI Server which is running on Jetty.

Add the following dependencies to build.gradle file. These add Swagger Annotations, Swagger Models, Swagger UI and Glassfish containers.

  compile group: 'io.swagger', name: 'swagger-annotations',version: swaggerVersion
  compile group: 'io.swagger', name: 'swagger-core', version: swaggerVersion
  compile group: 'io.swagger', name: 'swagger-jersey2-jaxrs', version: swaggerVersion
  compile group: 'io.swagger', name: 'swagger-models', version: swaggerVersion
  compile group: 'org.glassfish.jersey.core', name: 'jersey-server', version: versionJersey
  compile group: 'org.glassfish.jersey.containers', name: 'jersey-container-servlet-core', version: versionJersey
  compile group: 'org.webjars', name: 'swagger-ui', version: '2.1.4'

Now SusiServer.class is the main file which initializes all the servlets and server handlers.

Here, we need to tell the SusiServer to look for the swagger annotations and use them to build the documentation.

In the main function, before starting the server we set the serverHandlers from setServerHandler function.

public static void main(String[] args) throws Exception {
  // init the http server
        try {
            setupHttpServer(httpPort, httpsPort);
        } catch (Exception e) {
            Log.getLog().warn(e.getMessage());
            System.exit(-1);
        }
        setServerHandler(dataFile);
        
        SusiServer.server.start();
        SusiServer.caretaker = new Caretaker();
        SusiServer.caretaker.start();

Now, we will modify the setServetHandler function for registering the Swagger Handler.

There are already 2 handlers so I used a handerCollection object, so that we can give multiple handlers to handerCollection and set the serverHandler as handerCollection.

private static void setServerHandler(File dataFile){
         ServletContextHandler servletHandler = new ServletContextHandler();
       handlerCollection.addHandler(ipaccess);
        ContextHandlerCollection contexts = new ContextHandlerCollection();
        ServletContainer sc = new ServletContainer(resourceConfig);
        ServletHolder holder = new ServletHolder(sc);

        servletHandler.addServlet(holder, "/docs/*");
        handlerCollection.addHandler(contexts);

        SusiServer.server.setHandler(handlerCollection);
}

servletHandler.addServlet(holder, “/docs/*”), this line in the setServerHandler method sets the default swagger.json path to api.susi.ai/docs/swagger.json

This is all the basic setup to initialize swagger and now we need to modify our API endpoints and set annotations for the base URL and parameters.

Now I will discuss how to add swagger annotations to our Servlets.

For the demo, I will use GetAllUsers.class file, which returns the list of all users to Admins.

@Path("/aaa")
@Api(value = "/AAA")
@Produces({"application/json"})
public class GetAllUsers extends AbstractAPIHandler implements APIHandler {

Just before the class starts we will add the Path of the API endpoint and the result it produces. In this case, GetAllUsers returns JSON and is a part of aaa group.

@GET
@Path("/getAllUsers.json")
@ApiOperation(value = "Get All Users Registered on SUSI.AI",
        notes = "This API Endpoint returns the list of all users registered on SUSI Server",
        responseContainer = "Object")
@ApiResponses(value = { @ApiResponse(code = 200, message = "Success"),
@ApiResponse(code = 401, message = "Base user role not sufficient") })

Inside the class, we will declare the path of the API endpoint. A description of this endpoint and the sample response code.

In this file, there are only two possible responses.
1. Response code 200 – when everything goes well.

2. Response code 401 – when base user role is not sufficient, i.e when user is not Admin

When we save the file and browse to 127.0.0.1/docs/swagger.json
We get a response something like –

The Swagger UI is used to parse this JSON and create a UI for the documentation, where we can see the sample responses of the API endpoints, error codes and other things.

Swagger UI presents us the API documentation something like this image below.

Resources

Susi Server:  http://github.com/fossasia/susi_server/

Swagger: https://swagger.io/

Swagger and Jetty Tutorial: http://robferguson.org/2016/12/11/resteasy-embedded-jetty-fat-jars-swagger-and-swagger-ui/

Continue ReadingIntegrating Swagger with SUSI Server