Generating Map Action Responses in SUSI AI

SUSI AI responds to location related user queries with a Map action response. The different types of responses are referred to as actions which tell the client how to render the answer. One such action type is the Map action type. The map action contains latitude, longitude and zoom values telling the client to correspondingly render a map with the given location. Let us visit SUSI Web Chat and try it out. Query: Where is London Response: (API Response) The API Response actions contain text describing the specified location, an anchor with text ‘Here is a map` linked to openstreetmaps and a map with the location coordinates. Let us look at how this is implemented on server. For location related queries, the key where is used as an identifier. Once the query is matched with this key, a regular expression `where is (?:(?:a )*)(.*)` is used to parse the location name. "keys"   : ["where"], "phrases": [ {"type":"regex", "expression":"where is (?:(?:a )*)(.*)"}, ] The parsed location name is stored in $1$ and is used to make API calls to fetch information about the place and its location. Console process is used to fetch required data from an API. "process": [ { "type":"console", "expression":"SELECT location[0] AS lon, location[1] AS lat FROM locations WHERE query='$1$';"}, { "type":"console", "expression":"SELECT object AS locationInfo FROM location-info WHERE query='$1$';"} ], Here, we need to make two API calls : For getting information about the place For getting the location coordinates First let us look at how a Console Process works. In a console process we provide the URL needed to fetch data from, the query parameter needed to be passed to the URL and the path to look for the answer in the API response. url = <url>   - the url to the remote json service which will be used to retrieve information. It must contain a $query$ string. test = <parameter> - the parameter that will replace the $query$ string inside the given url. It is required to test the service. For getting the information about the place, we used Wikipedia API. We name this console process as location-info and added the required attributes to run it and fetch data from the API. "location-info": { "example":"http://127.0.0.1:4000/susi/console.json?q=%22SELECT%20*%20FROM%20location-info%20WHERE%20query=%27london%27;%22", "url":"https://en.wikipedia.org/w/api.php?action=opensearch&limit=1&format=json&search=", "test":"london", "parser":"json", "path":"$.[2]", "license":"Copyright by Wikipedia, https://wikimediafoundation.org/wiki/Terms_of_Use/en" } The attributes used are : url : The Media WIKI API endpoint test : The Location name which will be appended to the url before making the API call. parser : Specifies the response type for parsing the answer path : Points to the location in the response where the required answer is present The API endpoint called is of the following format : https://en.wikipedia.org/w/api.php?action=opensearch&limit=1&format=json&search=LOCATION_NAME For the query where is london, the API call made returns [ "london", ["London"], ["London  is the capital and most populous city of England and the United Kingdom."], ["https://en.wikipedia.org/wiki/London"] ] The path $.[2] points to the third element of the array i.e "London  is the capital and most populous city of England and the United Kingdom.” which…

Continue ReadingGenerating Map Action Responses in SUSI AI

Avoiding Nested Callbacks using RxJS in Loklak Scraper JS

Loklak Scraper JS, as suggested by the name, is a set of scrapers for social media websites written in NodeJS. One of the most common requirement while scraping is, there is a parent webpage which provides links for related child webpages. And the required data that needs to be scraped is present in both parent webpage and child webpages. For example, let’s say we want to scrape quora user profiles matching search query “Siddhant”. The matching profiles webpage for this example will be https://www.quora.com/search?q=Siddhant&type=profile which is the parent webpage, and the child webpages are links of each matched profiles. Now, a simplistic approach is to first obtain the HTML of parent webpage and then synchronously fetch the HTML of child webpages and parse them to get the desired data. The problem with this approach is that, it is slower as it is synchronous. A different approach can be using request-promise-native to implement the logic in asynchronous way. But, there are limitations with this approach. The HTML of child webpages that needs to be fetched can only be obtained after HTML of parent webpage is obtained and number of child webpages are dynamic. So, there is a request dependency between parent and child i.e. if only we have data from parent webpage we can extract data from child webpages. The code would look like this request(parent_url) .then(data => { ... request(child_url) .then(data => { // again nesting of child urls }) .catch(error => { }); }) .catch(error => { });   Firstly, with this approach there is callback hell. Horrible, isn’t it? And then we don’t know how many nested callbacks to use as the number of child webpages are dynamic. The saviour: RxJS The solution to our problem is reactive extensions in JavaScript. Using rxjs we can obtain the required data without callback hell and asynchronously! The promise-request object of the parent webpage is obtained. Using this promise-request object an observable is generated by using Rx.Observable.fromPromise. flatmap operator is used to parse the HTML of the parent webpage and obtain the links of child webpages. Then map method is used transform the links to promise-request objects which are again transformed into observables. The returned value - HTML - from the resulting observables is parsed and accumulated using zip operator. Finally, the accumulated data is subscribed. This is implemented in getScrapedData method of Quora JS scraper. getScrapedData(query, callback) { // observable from parent webpage Rx.Observable.fromPromise(this.getSearchQueryPromise(query)) .flatMap((t, i) => { // t is html of parent webpage // request-promise object of child webpages let profileLinkPromises = this.getProfileLinkPromises(t); // request-promise object to observable transformation let obs = profileLinkPromises.map(elem => Rx.Observable.fromPromise(elem)); // each Quora profile is parsed return Rx.Observable.zip( // accumulation of data from child webpages ...obs, (...profileLinkObservables) => { let scrapedProfiles = []; for (let i = 0; i < profileLinkObservables.length; i++) { let $ = cheerio.load(profileLinkObservables[i]); scrapedProfiles.push(this.scrape($)); } return scrapedProfiles; // accumulated data returned } ) }) .subscribe( // desired data is subscribed scrapedData => callback({profiles: scrapedData}), error => callback(error) ); }…

Continue ReadingAvoiding Nested Callbacks using RxJS in Loklak Scraper JS

Posting Tweet from Loklak Wok Android

Loklak Wok Android is a peer harvester that posts collected tweets to the Loklak Server. Not only it is a peer harvester, but also provides users to post their tweets from the app. Images and location of the user can also be attached in the tweet. This blog explains Adding Dependencies to the project In app/build.gradle: apply plugin: 'com.android.application' apply plugin: 'me.tatarka.retrolambda' android { ... packagingOptions { exclude 'META-INF/rxjava.properties' } } dependencies { ... compile 'com.google.code.gson:gson:2.8.1' compile 'com.squareup.retrofit2:retrofit:2.3.0' compile 'com.squareup.retrofit2:converter-gson:2.3.0' compile 'com.squareup.retrofit2:adapter-rxjava2:2.3.0' compile 'io.reactivex.rxjava2:rxjava:2.0.5' compile 'io.reactivex.rxjava2:rxandroid:2.0.1' }   In build.gradle project level: dependencies { classpath 'com.android.tools.build:gradle:2.3.3' classpath 'me.tatarka:gradle-retrolambda:3.2.0' }   Implementation User first authorize the application, so that they are able to post tweet from the app. For posting tweet statuses/update API endpoint of twitter is used and for attaching images with tweet media/upload API endpoint is used. As, photos and location can be attached in a tweet, for Android Marshmallow and above we need to ask runtime permissions for camera, gallery and location. The related permissions are mentioned in Manifest file first <uses-permission android:name="android.permission.CAMERA" /> <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" /> <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" /> // for location <uses-feature android:name="android.hardware.location.gps"/> <uses-feature android:name="android.hardware.location.network"/>   If, the device is using an OS below Android Marshmallow, there will be no runtime permissions, the user will be asked permissions at the time of installing the app. Now, runtime permissions are asked, if the user had already granted the permission the related activity (camera, gallery or location) is started. For camera permissions, onClickCameraButton is called @OnClick(R.id.camera) public void onClickCameraButton() { int permission = ContextCompat.checkSelfPermission( getActivity(), Manifest.permission.CAMERA); if (isAndroidMarshmallowAndAbove && permission != PackageManager.PERMISSION_GRANTED) { String[] permissions = { Manifest.permission.CAMERA, Manifest.permission.WRITE_EXTERNAL_STORAGE, Manifest.permission.READ_EXTERNAL_STORAGE }; requestPermissions(permissions, CAMERA_PERMISSION); } else { startCameraActivity(); } }   To start the camera activity if the permission is already granted, startCameraActivity method is called private void startCameraActivity() { Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE); File dir = getActivity().getExternalFilesDir(Environment.DIRECTORY_PICTURES); mCapturedPhotoFile = new File(dir, createFileName()); Uri capturedPhotoUri = getImageFileUri(mCapturedPhotoFile); intent.putExtra(MediaStore.EXTRA_OUTPUT, capturedPhotoUri); startActivityForResult(intent, REQUEST_CAPTURE_PHOTO); }   If the user decides to save the photo clicked from camera activity, the photo should be saved by creating a file and its uri is required to display the saved photo. The filename is created using createFileName method private String createFileName() { String timeStamp = new SimpleDateFormat("ddMMyyyy_HHmmss").format(new Date()); return "JPEG_" + timeStamp + ".jpg"; }   and uri is obtained using getImageFileUri private Uri getImageFileUri(File file) { if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) { return Uri.fromFile(file); } else { return FileProvider.getUriForFile(getActivity(), "org.loklak.android.provider", file); } }   Similarly, for the gallery, onClickGalleryButton method is implemented to ask runtime permissions and launch gallery activity if the permission is already granted. @OnClick(R.id.gallery) public void onClickGalleryButton() { int permission = ContextCompat.checkSelfPermission( getActivity(), Manifest.permission.WRITE_EXTERNAL_STORAGE); if (isAndroidMarshmallowAndAbove && permission != PackageManager.PERMISSION_GRANTED) { String[] permissions = { Manifest.permission.WRITE_EXTERNAL_STORAGE, Manifest.permission.READ_EXTERNAL_STORAGE }; requestPermissions(permissions, GALLERY_PERMISSION); } else { startGalleryActivity(); } }   For starting the gallery activity, startGalleryActivity is used private void startGalleryActivity() { Intent intent = new Intent(); intent.setType("image/*"); intent.setAction(Intent.ACTION_GET_CONTENT); intent.putExtra(Intent.EXTRA_ALLOW_MULTIPLE, true); startActivityForResult( Intent.createChooser(intent, "Select images"), REQUEST_GALLERY_MEDIA_SELECTION); }   And finally for location onClickAddLocationButton…

Continue ReadingPosting Tweet from Loklak Wok Android

Optimising Docker Images for loklak Server

The loklak server is in a process of moving to Kubernetes. In order to do so, we needed to have different Docker images that suit these deployments. In this blog post, I will be discussing the process through which I optimised the size of Docker image for these deployments. Initial Image The image that I started with used Ubuntu as base. It installed all the components needed and then modified the configurations as required - FROM ubuntu:latest # Env Vars ENV LANG=en_US.UTF-8 ENV JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 ENV DEBIAN_FRONTEND noninteractive WORKDIR /loklak_server RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y git openjdk-8-jdk RUN git clone https://github.com/loklak/loklak_server.git /loklak_server RUN git checkout development RUN ./gradlew build -x test -x checkstyleTest -x checkstyleMain -x jacocoTestReport RUN sed -i.bak 's/^\(port.http=\).*/\180/' conf/config.properties ... # More configurations RUN echo "while true; do sleep 10;done" >> bin/start.sh # Start CMD ["bin/start.sh", "-Idn"] The size of images built using this Dockerfile was quite huge - REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE loklak_server       latest              a92f506b360d        About a minute ago   1.114 GB ubuntu              latest              ccc7a11d65b1        3 days ago           120.1 MB But since this size is not acceptable, we needed to reduce it. Moving to Apline Alpine Linux is an extremely lightweight Linux distro, built mainly for the container environment. Its size is so tiny that it hardly puts any impact on the overall size of images. So, I replaced Ubuntu with Alpine - FROM alpine:latest ... RUN apk update RUN apk add git openjdk8 bash ... And now we had much smaller images - REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE loklak_server       latest              54b507ee9187        17 seconds ago      668.8 MB alpine              latest              7328f6f8b418        6 weeks ago         3.966 MB As we can see that due to no caching and small size of Alpine, the image size is reduced to almost half the original. Reducing Content Size There are many things in a project which are no longer needed while running the project, like the .git folder (which is huge in case of loklak) - $ du -sh loklak_server/.git 236M loklak_server/.git We can remove such files from the Docker image and save a lot of space - rm -rf .[^.] .??* Optimizing Number of Layers The number of layers also affect the size of the image. More the number of layers, more will be the size of image. In the Dockerfile, we can club together the RUN commands for lower number of images. RUN apk update && apk add openjdk8 git bash && \ git clone https://github.com/loklak/loklak_server.git /loklak_server && \ ... After this, the effective size is again reduced by a major factor - REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE loklak_server       latest              54b507ee9187        17 seconds ago      422.3 MB alpine              latest              7328f6f8b418        6 weeks ago         3.966 MB Conclusion In this blog post, I discussed the process of optimising the size of Dockerfile for Kubernetes deployments of loklak server. The size was reduced to 426 MB from 1.234 GB and this provided much faster push/pull time for Docker images, and therefore, faster updates for Kubernetes deployments.…

Continue ReadingOptimising Docker Images for loklak Server

Working of One Click Deployment Buttons in loklak

Today’s topic is deployment. It’s called one-click deployment for a reason: Developers are lazy. It’s hard to do less than clicking on one button, so that’s our goal to make use of one click button in loklak. For one click buttons we only need a central build server, which is our loklak_server. Everything written here was based on Apache ant, but later on ant build was deprecated and loklak server started to use gradle build. We wanted to make the process of provisioning and setting up a complete infrastructure of your own, from server to continuous integration tasks, as easy as possible. These button allows you to do all of that in one click. How does it work? You can see the one click buttons in the README page of loklak_server repository. These repositories may include a different files like scalingo.json for scalingo, docker-compose.yml and docker-cloud.yml for docker cloud etc files at their root, allowing them to define a few things like a name, description, logo and build environment (Gradle build in the case of loklak server). Once you've clicked on any of the buttons, you will be redirected to respective apps and prompted with this information for you to review before confirming the fork. This will effectively fork the repository in your account. Once the repo is ready, you can click on it. You will then be asked to "activate" or “deploy” your branch, allowing it to provision actual servers and run tasks. At the same time, you will be asked to review and potentially modify a few variables that were defined in the predefined files (for eg: app.json for heroku) of the apps. These are usually things like the Git URL of the repo for loklak, or some of the details related to the cloud provider you want to use (eg: Digital Ocean). Once you confirmed this last step, your branch i.e., most probably master branch of loklak server repo is activated and the button will start provisioning and configuring your servers, along with the tasks which may allow you to build and deploy your app. In most of the cases, you can go to the tasks/setup section and run the build task that will fetch loklak server's code, build it and deploy it on your server, all configurations included and will get a public IP. What's next In loklak we are also introducing new one click “AZURE” button, then the users can also start deploying loklak in azure platform. Resources About Bluemix one click: https://console.bluemix.net/docs/develop/deploy_button.html About Scalingo one click: http://doc.scalingo.com/deployment/one-click-deploy.html About Docker cloud one click: https://docs.docker.com/docker-cloud/apps/deploy-to-cloud-btn/ About Heroku one click: https://devcenter.heroku.com/articles/heroku-button

Continue ReadingWorking of One Click Deployment Buttons in loklak

Implementing 3 legged Authorization in Loklak Wok Android for Twitter

Loklak Wok Android is a peer harvester that posts collected tweets to the Loklak Server. Not only it is a peer harvester, but also provides users to post their tweets from the app. Posting tweets from the app requires users to authorize the Loklak Wok app, the client app created https://apps.twitter.com/ . This blog explains in detail about the authorization process. Adding Dependencies to the project In app/build.gradle: apply plugin: 'com.android.application' apply plugin: 'me.tatarka.retrolambda' android { ... packagingOptions { exclude 'META-INF/rxjava.properties' } } dependencies { ... compile 'com.google.code.gson:gson:2.8.1' compile 'com.squareup.retrofit2:retrofit:2.3.0' compile 'com.squareup.retrofit2:converter-gson:2.3.0' compile 'com.squareup.retrofit2:adapter-rxjava2:2.3.0' compile 'io.reactivex.rxjava2:rxjava:2.0.5' compile 'io.reactivex.rxjava2:rxandroid:2.0.1' }   In build.gradle project level: dependencies { classpath 'com.android.tools.build:gradle:2.3.3' classpath 'me.tatarka:gradle-retrolambda:3.2.0' }   Steps of Authorization Step 1: Create client app in Twitter Create a twitter client app at https://apps.twitter.com/. Provide the mandatory entries and also Callback url (would be used in next steps). Then go to “Keys and Access Token” and save your consumer key and consumer secret. In case you want to use Twitter API for yourself, click on “Create my access token”, which provides access token and access token secret. Step 2: Obtaining a request token Using the “consumer key” and “consumer secret” request token is obtained by sending a POST request to oauth/request_token. As Twitter API are Oauth1 based the sent request needs to be signed by generating oauth_signature. The oauth_signature is generated by intercepting the network request sent by retrofit rest API client, the oauth interceptor used in Loklak Wok Android is a modified version of this snippet. The retrofit TwitterAPI interface is defined public interface TwitterAPI { String BASE_URL = "https://api.twitter.com/"; @POST("/oauth/request_token") Observable<ResponseBody> getRequestToken(); @FormUrlEncoded @POST("/oauth/access_token") Observable<ResponseBody> getAccessTokenAndSecret(@Field("oauth_verifier") String oauthVerifier); }   And the retrofit REST client is implemented in TwitterRestClient. createTwitterAPIWithoutAccessToken method returns a twitter API client which can be called without providing access keys, this is used as we don’t have access tokens right now. public static TwitterAPI createTwitterAPIWithoutAccessToken() { if (sWithoutAccessTokenRetrofit == null) { sLoggingInterceptor.setLevel(HttpLoggingInterceptor.Level.BODY); // uncomment to debug network requests // sWithoutAccessTokenClient.addInterceptor(sLoggingInterceptor); sWithoutAccessTokenRetrofit = sRetrofitBuilder .client(sWithoutAccessTokenClient.build()).build(); } return sWithoutAccessTokenRetrofit.create(TwitterAPI.class); }   So, getRequestToken method is used to obtain the request token, if the request is successful oauth_token is returned. @OnClick(R.id.twitter_authorize) public void onClickTwitterAuthorizeButton(View view) { mTwitterApi.getRequestToken() .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .subscribe(this::parseRequestTokenResponse, this::onFetchRequestTokenError); }   Step 3: Redirecting the user Using the oauth_token obtained in Step 2, the user is redirected to login page using WebView. private void setAuthorizationView() { ... webView.setVisibility(View.VISIBLE); webView.loadUrl(mAuthorizationUrl); }   A WebView client is created by extending WebViewClient, this is used to keep track of which webpage is opened by overriding shouldOverrideUrlLoading. @Override public boolean shouldOverrideUrlLoading(WebView view, String url) { if (url.contains("github")) { String[] tokenAndVerifier = url.split("&"); mOAuthVerifier = tokenAndVerifier[1].substring(tokenAndVerifier[1].indexOf('=') + 1); getAccessTokenAndSecret(); return true; } return false; }   As the link provided in callback url while creating our twitter app is a github page. The WebViewClient checks if it is a github page or not. If yes, then it parses the oauth_verifier from the github url. Step 4: Converting the request token to an access token A…

Continue ReadingImplementing 3 legged Authorization in Loklak Wok Android for Twitter

Route Based Chunking in Loklak Search

The loklak search application running at loklak.org is growing in size as the features are being added into the application, this growth is a linear one, and traditional SPA, tend to ship all the code is required to run the application in one pass, as a single monolithic JavaScript file, along with the index.html. This approach is suitable for the application with few pages which are frequently used, and have context switching between those logical pages at a high rate and almost simultaneously as the application loads. But generally, only a fraction of code is what is accessed most frequently by most users, so as the application size grows it does not make sense to include all the code for the entire application at the first request, as there is always some code in the application, some views, are rarely accessed. The loading of such part of the application can be delayed until they are accessed in the application. The angular router provides an easy way to set up such system and is used in the latest version of loklak search. The technique which is used here is to load the content according to the route. This makes sure only the route which is viewed is loaded on the initial load, and subsequent loading is done at the runtime as and when required. Old setup for baseline Here are the compiled file sizes, of the project without the chunking the application. Now as we can see that the file sizes are huge, especially the vendor bundle, which is of 5.4M and main bundle which is about 0.5M now, these are the files which are loaded on the first load and due to their huge sizes, the first paint of the application suffers, to a great extent. These numbers will act as a baseline upon which we will measure the impact of route based chunking. Setup for route based chunking The setup for route based chunking is fairly simple, this is our routing configuration, the part of the modules which we want to lazy load are to be passed as loadChildren attribute of the route, this attribute is a string which is a path of the feature module which, and part after the hash symbol is the actual class name of the module, in that file. This setup enables the router to load that module lazily when accessed by the user. const routes: Routes = [ { path: '', pathMatch: 'full', loadChildren: './home/home.module#HomeModule', data: { preload: true } }, { path: 'about', loadChildren: './about/about.module#AboutModule' }, { path: 'contact', loadChildren: './contact/contact.module#ContactModule' }, { path: 'search', loadChildren: './feed/feed.module#FeedModule', data: { preload: true } }, { path: 'terms', loadChildren: './terms/terms.module#TermsModule' }, { path: 'wall', loadChildren: './media-wall/media-wall.module#MediaWallModule' } ]; Preloading of some routes As we can see that in two of the configurations above, there is a data attribute, on which preload: true attribute is specified. Sometimes we need to preload some part of theapplication, which we know we will access, soon enough. So angular…

Continue ReadingRoute Based Chunking in Loklak Search

Lazy Loading Images in Loklak Search

In last blog post, I discussed the basic Web API’s which helps us to create the lazy image loader component. I also discussed the structure which is used in the application, to load the images lazily. The core idea is to wrap the <img> element in a wrapper, <app-lazy-img> element. This enables us the detection of the element in the viewport and corresponding loading only if the image is present in the viewport. In this blog post, I will be discussing the implementation details about how this is achieved in Loklak search in an optimized manner. The logic for lazy loading of images in the application is divided into a Component and a corresponding Service. The reason for this splitting of logic will be explained as we discuss the core parts of the code for this feature. Detecting the Intersection with Viewport The lazy image service is a service for the lazy image component which is registered at the by the modules which intend to use this app lazy image component. The task of this service is to register the elements with the intersection observer, and, then emit an event when the element comes in the viewport, which the element can react on and then use the other methods of services to actually fetch the image. @Injectable() export class LazyImgService { private intersectionObserver: IntersectionObserver = new IntersectionObserver(this.observerCallback.bind(this), { rootMargin: '50% 50%' }); private elementSubscriberMap: Map<Element, Subscriber<boolean>> = new Map<Element, Subscriber<boolean>>(); } The service has two member attributes, one is IntersectionObserver, and the other is a Map which stores the the reference of the subscribers of this intersection observer. This reference is then later used to emit the event when the element comes in viewport. The rootMargin of the intersection observer is set to 50% this makes sure that when the element is 50% away from the viewport. The obvserve public method of the service, takes an element and pass it to intersection observer to observe, also put the element in the subscriber map. public observe(element: Element): Observable<boolean> { const observable: Observable<boolean> = new Observable<boolean>(subscriber => { this.elementSubscriberMap.set(element, subscriber); }); this.intersectionObserver.observe(element); return observable; } Then there is the observer callback, this method, as an argument receives all the objects intersecting the root of the observer, when this callback is fired, we find all the intersecting elements and emit the intersection event. Indicating that the element is nearby the viewport and this is the time to load it. private observerCallback(entries: IntersectionObserverEntry[], observer: IntersectionObserver) { entries.forEach(entry => { if (this.elementSubscriberMap.has(entry.target)) { if (entry.intersectionRatio > 0) { const subscriber = this.elementSubscriberMap.get(entry.target); subscriber.next(true); this.elementSubscriberMap.delete(entry.target); } } }); } Now, our LazyImgComponent enables us to uses this service to register its element, with the intersection observer and then reacting to it later, when the event is emitted. This method sets up the IO, to load the image, and subscribes to the event emittes by the service and eventually calls the loadImage method when the element intersects with the viewport. private setupIntersectionObserver() { this.lazyImgService .observe(this.elementRef.nativeElement) .subscribe(value =>…

Continue ReadingLazy Loading Images in Loklak Search

Enhancing LoklakWordCloud app present on Loklak apps site

LoklakWordCloud app is presently hosted on loklak apps site. Before moving into the content of this blog, let us get a brief overview of the app. What does the app do? The app generates a word cloud using twitter data returned by loklak based on the query word provided by the user. The user enters a word in the input field and presses the search button. After that a word cloud is created using the content (text body, hashtags and mentioned) of the various tweets which contains the user provided query word. In my previous post I wrote about creating the basic functional app. In this post I will be describing the next steps that have been implemented in the app. Making the word cloud clickable This is one of the most important and interesting features added to the app. The words in the cloud are now clickable.Whenever an user clicks on a word present in the cloud, the cloud is replaced by the word cloud of that selected word. How do we achieve this behaviour? Well, for this we use Jqcloud’s handler feature. While creating the list of objects for each word and its frequency, we also specify a handler corresponding to each of the word. The handler is supposed to handle a click event. Whenever a click event occurs, we set the value of $scope.tweet to the selected word and invoke the search function, which calls the loklak API and regenerates the word cloud. for (var word in $scope.wordFreq) { $scope.wordCloudData.push({ text: word, weight: $scope.wordFreq[word], handlers: { click: function(e) { $scope.tweet = e.target.textContent; $scope.search(); } } }); } As it can be seen in the above snippet, handlers is simply an JavaScript object, which takes a function for the click event. In the function we pass the word selected as value of the tweet variable and call search method. Adding filters to the app Previously the app generated word cloud using the entire tweet content, that is, hashtags, mentions and tweet body. Thus the app was not flexible. User was not able to decide on which field he wants his word cloud to be generated. User might want to generate his  word cloud using only the hashtags or the mentions or simply the tweet body. In order to make this possible, filters have been introduced. Now we have filters for hashtags, mentions, tweet body and date. <div class="col-md-6 tweet-filters"> <strong>Filters</strong> <hr> <div class="filters"> <label class="checkbox-inline"><input type="checkbox" value="" ng-model="hashtags">Hashtags</label> <label class="checkbox-inline"><input type="checkbox" value="" ng-model="mentions">Mentions</label> <label class="checkbox-inline"><input type="checkbox" value="" ng-model="tweetbody">Tweet body</label> </div> <div class="filter-all"> <span class="select-all" ng-click="selectAll()"> Select all </span> </div> </div> We have used checkboxes for the individual filters and have kept an option to select all the filters at once. Next we require to hook this HTML to AngularJS code to make the filters functional. if ($scope.hashtags) { tweet.hashtags.forEach(function (hashtag) { $scope.filteredWords.push("#" + hashtag); }); } if ($scope.mentions) { tweet.mentions.forEach(function (mention) { $scope.filteredWords.push("@" + mention); }); } In the above snippet, before adding the hashtags to the list…

Continue ReadingEnhancing LoklakWordCloud app present on Loklak apps site

One Click Deployment Button for loklak Using Heroku with Gradle Build

The one click deploy button makes it easy for the users of loklak to get their own cloud instance created and deployed in their heroku account and can be used according to their flexibility. Heroku uses an app.json manifest in the code repo to figure out what add-ons, config and other deployment steps are required to make the code run. This is used to configure and deploy the app. Once you have provide the app name and then click on deploy button, Heroku will start deploying the loklak server to a new app on your account: When setup is complete, you can open the deployed app in your browser or inspect it in Dashboard. All these steps and requirements can now be encoded in an app.json file and placed in a repo alongside a button that kicks off the setup with a single click. App.json is a manifest format for describing apps and specifying what their config requirements are. Heroku uses this file to figure out how code in a particular repo should be deployed on the platform. Here is the loklak’s app.json file which used gradle build pack: { "name": "Loklak Server", "description": "Distributed Tweet Search Server", "logo": "https://raw.githubusercontent.com/loklak/loklak_server/master/html/images/loklak_anonymous.png", "website": "http://api.loklak.org", "repository": "https://github.com/loklak/loklak_server.git", "image": "loklak/loklak_server:latest-master", "env": { "BUILDPACK_URL": "https://github.com/heroku/heroku-buildpack-gradle.git" } }   If you are interested you can try deploying the peer from here itself. Checkout how simple it can be to deploy. Deploy button: Resources: Read more about heroku one click deploy button: https://devcenter.heroku.com/articles/heroku-button How to write app.json file for your application: https://blog.heroku.com/introducing_the_app_json_application_manifest

Continue ReadingOne Click Deployment Button for loklak Using Heroku with Gradle Build