Setting Loklak Server with SSL

Loklak Server is based on embedded Jetty Server which can work both with or without SSL encryption. Lately, there was need to setup Loklak Server with SSL. Though the need was satisfied by CloudFlare. Alternatively, there are 2 ways to set up Loklak Server with SSL. They are:- 1) Default Jetty Implementation There is pre-existing implementation of Jetty libraries. The http mode can be set in configuration file. There are 4 modes on which Loklak Server can work: http mode, https mode, only https mode and redirect to https mode. Loklak Server listens to port 9000 when in http mode and to port 9443 when in https mode. There is also a need of SSL certificate which is to be added in configuration file. 2) Getting SSL certificate with Kube-Lego on Kubernetes Deployment I got to know about Kube-Lego by @niranjan94. It is implemented in Open-Event-Orga-Server. The approach is to use: a) Nginx as ingress controller For setting up Nginx ingress controller, a yml file is needed which downloads and configures the server. The configurations for data requests and response are: proxy-connect-timeout: "15" proxy-read-timeout: "600" proxy-send-imeout: "600" hsts-include-subdomains: "false" body-size: "64m" server-name-hash-bucket-size: "256" server-tokens: "false" Nginx is configured to work on both http and https ports in service.yml ports: - port: 80 name: http - port: 443 name: https   b) Kube-Lego for fetching SSL certificates from Let's Encrypt Kube-Lego was set up with default values in yml. It uses the host-name, email address and secretname of the deployment to validate url and fetch SSL certificate from Let's Encrypt. c) Setup configurations related to TLS and no-TLS connection These configuration files mentions the path and service ports for Nginx Server through which requests are forwarded to backend Loklak Server. Here for no-TLS and TLS requests, the requests are directly forwarded to localhost at port 80 of Loklak Server container. rules: - host: staging.loklak.org http: paths: - path: / backend: serviceName: server servicePort: 80 For TLS requests, the secret name is also mentioned. Kube-Lego fetches host-name and secret-name from here for the certificate tls: - hosts: - staging.loklak.org secretName: loklak-api-tls d) Loklak Server, ElasticSearch and Mosquitto at backend These containers work at backend. ElasticSearch and Mosquitto are only accessible to Loklak Server. Loklak Server can be accessed through Nginx server. Loklak Server is configured to work at http mode and is exposed at port 80. ports: - port: 80 protocol: TCP targetPort: 80 To deploy the Loklak Server, all these are deployed in separate pods and they interact through service ports. To deploy, we use deploy script: # For elasticsearch, accessible only to api-server kubectl create -R -f ${path-to-config-file}/elasticsearch/ # For mqtt, accessible only to api-server kubectl create -R -f ${path-to-config-file}/mosquitto/ # Start KubeLego deployment for TLS certificates kubectl create -R -f ${path-to-config-file}/lego/ kubectl create -R -f ${path-to-config-file}/nginx/ # Create web namespace, this acts as bridge to Loklak Server kubectl create -R -f ${path-to-config-file}/web/ # Create API server deployment and expose the services kubectl create -R -f ${path-to-config-file}/api-server/ # Get the…

Continue ReadingSetting Loklak Server with SSL

Link Preview Service from SUSI Server

 SUSI Webchat, SUSI Android app, SUSI iOS app are various SUSI clients which depend on response from SUSI Server. The most common response of SUSI Server is in form of links. Clients usually need to show the preview of the links to the user. This preview may include featured image, description and the title of the link.  Clients show this information by using various 3rd party APIs and libraries. We planned to create an API endpoint for this on SUSI Server to give the preview of the link. This service is called LinkPreviewService. String url = post.get("url", ""); if(url==null || url.isEmpty()){ jsonObject.put("message","URL Not given"); jsonObject.put("accepted",false); return new ServiceResponse(jsonObject); } This API Endpoint accept only 1 get parameter which is the URL whose preview is to be shown. Here we also check if no parameter or wrong URL parameter was sent. If that was the the case then we return an error message to the user. SourceContent sourceContent = TextCrawler.scrape(url,3); if (sourceContent.getImages() != null) jsonObject.put("image", sourceContent.getImages().get(0)); if (sourceContent.getDescription() != null) jsonObject.put("descriptionShort", sourceContent.getDescription()); if(sourceContent.getTitle()!=null)jsonObject.put("title", sourceContent.getTitle()); jsonObject.put("accepted",true); return new ServiceResponse(jsonObject); } The TextCrawler function accept two parameters. One is the url of the website which is to be scraped for the preview data and the other is depth. To get the images, description and title there are methods built in. Here we just call those methods and set them in our JSON Object. private String htmlDecode(String content) { return Jsoup.parse(content).text(); } Text Crawler is based on Jsoup. Jsoup is a java library that is used to scrape HTML pages. To get anything from Jsoup we need to decode the content of HTML to Text. public List<String> getImages(Document document, int imageQuantity) { Elements media = document.select("[src]"); while(var5.hasNext()) { Element srcElement = (Element)var5.next(); if(srcElement.tagName().equals("img")) { ((List)matches).add(srcElement.attr("abs:src")); } }  The getImages method takes the HTML document from the JSoup and find the image tags in that. We have given the imageQuantity parameter in the function, so accordingly it returns the src attribute of the first n images it find. This API Endpoint can be seen working on http://127.0.0.1:4000/susi/linkPreview.json?url=<ANY URL> A real working example of this endpoint would be http://api.susi.ai/susi/linkPreview.json?url=https://techcrunch.com/2017/07/23/dear-tech-dudes-stop-being-such-idiots-about-women/ Resources: Web Crawlers: https://www.promptcloud.com/data-scraping-vs-data-crawling/ JSoup: https://jsoup.org/ JSoup Api Docs: https://jsoup.org/apidocs/ Parsing HTML with JSoup: http://www.baeldung.com/java-with-jsoup

Continue ReadingLink Preview Service from SUSI Server

Deleting SUSI Skills from Server

SUSI Skill CMS is a web application to create and edit skills. In this blog post I will be covering how we made the skill deleting feature in Skill CMS from the SUSI Server. The deletion of skill was to be made in such a way that user can click a button to delete the skill. As soon as they click the delete button the skill is deleted it is removed from the directory of SUSI Skills. But admins have an option to recover the deleted skill before completion of 30 days of deleting the skill. First we will accept all the request parameters from the GET request. String model_name = call.get("model", "general"); String group_name = call.get("group", "Knowledge"); String language_name = call.get("language", "en"); String skill_name = call.get("skill", "wikipedia"); In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository. if(!DAO.deleted_skill_dir.exists()){ DAO.deleted_skill_dir.mkdirs(); } We need to move the skill to a directory called deleted_skills_dir. So we check if the directory exists or not. If it not exists then we create a directory for the deleted skills. if (skill.exists()) { File file = new File(DAO.deleted_skill_dir.getPath()+path); file.getParentFile().mkdirs(); if(skill.renameTo(file)){ Boolean changed = new File(DAO.deleted_skill_dir.getPath()+path).setLastModified(System.currentTimeMillis()); } This is the part where the real deletion happens. We get the path of the skill and rename that to a new path which is in the directory of deleted skills. Also here change the last modified time of the skill as the current time. This time is used to check if the skill deleted is older than 30 days or not. try (Git git = DAO.getGit()) { DAO.pushCommit(git, "Deleted " + skill_name, rights.getIdentity().isEmail() ? rights.getIdentity().getName() : "anonymous@"); json.put("accepted", true); json.put("message", "Deleted " + skill_name); } catch (IOException | GitAPIException e) { Finally we add the changes to Git. DAO.pushCommit pushes to commit to the Susi Skill Data repository. If the user is logged in we get the email of the user and set that email as the commit author. Else we set the username “anonymous@”. Then in the caretaker class there is a method deleteOldFiles that checks for all the files whose last modified time was older than 30 days. If there is any file whose last modified time was older than 30 days then it quietly delete the files. public void deleteOldFiles() { Collection<File> filesToDelete = FileUtils.listFiles(new File(DAO.deleted_skill_dir.getPath()), new (DateTime.now().withTimeAtStartOfDay().minusDays(30).toDate()), TrueFileFilter.TRUE); // include sub dirs for (File file : filesToDelete) { boolean success = FileUtils.deleteQuietly(file); if (!success) { System.out.print("Deleted skill older than 30 days."); } } } To test this API endpoint, we need to call http://localhost:4000/cms/deleteSkill.txt?model=general&group=Knowledge&language=en&skill=<skill_name> Resources JGit Documentation: https://eclipse.org/jgit/documentation/ Commons IO: https://commons.apache.org/proper/commons-io/ Age Filter: https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/filefilter/AgeFileFilter.html JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

Continue ReadingDeleting SUSI Skills from Server

Getting SUSI Skill at a Commit ID

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID. In this blog post I will tell about making an API endpoint which works similar to git show. First we will accept all the request parameters from the GET request. String model_name = call.get("model", "general"); String group_name = call.get("group", "Knowledge"); String language_name = call.get("language", "en"); String skill_name = call.get("skill", "wikipedia"); String commitID = call.get("commitID", null); In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository. This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution. Repository repository = DAO.getRepository(); ObjectId CommitIdObject = repository.resolve(commitID); Then we get the git repository of the skill from the DAO and initialize the repository object. From the commitID that we got in the request parameters we create a CommitIdObject. (RevWalk revWalk = new RevWalk(repository)) { RevCommit commit = revWalk.parseCommit(CommitIdObject); RevTree tree = commit.getTree(); Now using commit's tree, we will find the find the path and get the tree of the commit. From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders. revWalk = new RevWalk(repository)) { try (TreeWalk treeWalk = new TreeWalk(repository)) { treeWalk.addTree(tree); treeWalk.setRecursive(true); treeWalk.setFilter(PathFilter.create(path)); if (!treeWalk.next()) { throw new IllegalStateException("Did not find expected file"); } If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID. ObjectId objectId = treeWalk.getObjectId(0); ObjectLoader loader = repository.open(objectId); OutputStream output = new OutputStream(); loader.copyTo(output); And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it. json.put("file",output); This Servlet can be seen working api.susi.ai: http://api.susi.ai/cms/getFileAtCommitID.json?model=general&group=knowledge&language=en&skill=bitcoin&commitID=214791f55c19f24d7744364495541b685539a4ee Resources JGit Documentation: https://eclipse.org/jgit/documentation/ JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/ JGit Github: https://github.com/eclipse/jgit

Continue ReadingGetting SUSI Skill at a Commit ID

Introducing Stream Servlet in loklak Server

A major part of my GSoC proposal was adding stream API to loklak server. In a previous blog post, I discussed the addition of Mosquitto as a message broker for MQTT streaming. After testing this service for a few days and some minor improvements, I was in a position to expose the stream to outside users using a simple API. In this blog post, I will be discussing the addition of /api/stream.json endpoint to loklak server. HTTP Server-Sent Events Server-sent events (SSE) is a technology where a browser receives automatic updates from a server via HTTP connection. The Server-Sent Events EventSource API is standardized as part of HTML5 by the W3C. - Wikipedia This API is supported by all major browsers except Microsoft Edge. For loklak, the plan was to use this event system to send messages, as they arrive, to the connected users. Apart from browser support, EventSource API can also be used with many other technologies too. Jetty Eventsource Plugin For Java, we can use Jetty’s EventSource plugin to send events to clients. It is similar to other Jetty servlets when it comes to processing the arguments, handling requests, etc. But it provides a simple interface to send events as they occur to connected users. Adding Dependency To use this plugin, we can add the following line to Gradle dependencies - compile group: 'org.eclipse.jetty', name: 'jetty-eventsource-servlet', version: '1.0.0' [SOURCE] The Event Source An EventSource is the object which is required for EventSourceServlet to send events. All the logics for emitting events needs to be defined in the related class. To link a servlet with an EventSource, we need to override the newEventSource method - public class StreamServlet extends EventSourceServlet { @Override protected EventSource newEventSource(HttpServletRequest request) { String channel = request.getParameter("channel"); if (channel == null) { return null; } if (channel.isEmpty()) { return null; } return new MqttEventSource(channel); } } [SOURCE] If no channel is provided, the EventSource object will be null and the request will be rejected. Here, the MqttEventSource would be used to handle the stream of Tweets as they arrive from the Mosquitto message broker. Cross Site Requests Since the requests to this endpoint can’t be of JSONP type, it is necessary to allow cross site requests on this endpoint. This can be done by overriding the doGet method of the servlet - @Override protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setHeader("Access-Control-Allow-Origin", "*"); super.doGet(request, response); } [SOURCE] Adding MQTT Subscriber When a request for events arrives, the constructor to MqttEventSource is called. At this stage, we need to connect to the stream from Mosquitto for the channel. To achieve this, we can set the class as MqttCallback using appropriate client configurations - public class MqttEventSource implements MqttCallback { ... MqttEventSource(String channel) { this.channel = channel; } ... this.mqttClient = new MqttClient(address, "loklak_server_subscriber"); this.mqttClient.connect(); this.mqttClient.setCallback(this); this.mqttClient.subscribe(this.channel); ... } [SOURCE] By setting the callback to this, we can override the messageArrived method to handle the arrival of a new message on the channel. Just to…

Continue ReadingIntroducing Stream Servlet in loklak Server

Integrating Swagger with SUSI Server

The goal of Swagger is to define a standard interface for REST APIs which allows humans to understand the capabilities of the APIs without access to source code or documentation. SUSI Server is now completely API centric. As more and more people make their own bots using SUSI Server they will be needing documentation for the APIs. We can use swagger so that without looking at the javadocs or documentation people can consume the REST APIs. In this I post will walk you through the steps to integrate Swagger with SUSI Server which is running on Jetty. Add the following dependencies to build.gradle file. These add Swagger Annotations, Swagger Models, Swagger UI and Glassfish containers. compile group: 'io.swagger', name: 'swagger-annotations',version: swaggerVersion compile group: 'io.swagger', name: 'swagger-core', version: swaggerVersion compile group: 'io.swagger', name: 'swagger-jersey2-jaxrs', version: swaggerVersion compile group: 'io.swagger', name: 'swagger-models', version: swaggerVersion compile group: 'org.glassfish.jersey.core', name: 'jersey-server', version: versionJersey compile group: 'org.glassfish.jersey.containers', name: 'jersey-container-servlet-core', version: versionJersey compile group: 'org.webjars', name: 'swagger-ui', version: '2.1.4' Now SusiServer.class is the main file which initializes all the servlets and server handlers. Here, we need to tell the SusiServer to look for the swagger annotations and use them to build the documentation. In the main function, before starting the server we set the serverHandlers from setServerHandler function. public static void main(String[] args) throws Exception { // init the http server try { setupHttpServer(httpPort, httpsPort); } catch (Exception e) { Log.getLog().warn(e.getMessage()); System.exit(-1); } setServerHandler(dataFile); SusiServer.server.start(); SusiServer.caretaker = new Caretaker(); SusiServer.caretaker.start(); Now, we will modify the setServetHandler function for registering the Swagger Handler. There are already 2 handlers so I used a handerCollection object, so that we can give multiple handlers to handerCollection and set the serverHandler as handerCollection. private static void setServerHandler(File dataFile){ ServletContextHandler servletHandler = new ServletContextHandler(); handlerCollection.addHandler(ipaccess); ContextHandlerCollection contexts = new ContextHandlerCollection(); ServletContainer sc = new ServletContainer(resourceConfig); ServletHolder holder = new ServletHolder(sc); servletHandler.addServlet(holder, "/docs/*"); handlerCollection.addHandler(contexts); SusiServer.server.setHandler(handlerCollection); } servletHandler.addServlet(holder, "/docs/*"), this line in the setServerHandler method sets the default swagger.json path to api.susi.ai/docs/swagger.json This is all the basic setup to initialize swagger and now we need to modify our API endpoints and set annotations for the base URL and parameters. Now I will discuss how to add swagger annotations to our Servlets. For the demo, I will use GetAllUsers.class file, which returns the list of all users to Admins. @Path("/aaa") @Api(value = "/AAA") @Produces({"application/json"}) public class GetAllUsers extends AbstractAPIHandler implements APIHandler { Just before the class starts we will add the Path of the API endpoint and the result it produces. In this case, GetAllUsers returns JSON and is a part of aaa group. @GET @Path("/getAllUsers.json") @ApiOperation(value = "Get All Users Registered on SUSI.AI", notes = "This API Endpoint returns the list of all users registered on SUSI Server", responseContainer = "Object") @ApiResponses(value = { @ApiResponse(code = 200, message = "Success"), @ApiResponse(code = 401, message = "Base user role not sufficient") }) Inside the class, we will declare the path of the API endpoint. A description of this endpoint and the sample response code. In this file,…

Continue ReadingIntegrating Swagger with SUSI Server