Adding Unit Test for Reducer in loklak search

Ngrx/store components are an integral part of the loklak search. All the components are dependent on how the data is received from the reducers. Reducer is like a client-side database which stores up all the data received from the API response. It is responsible for changing the state of the application. Reducers also supplies data to the Angular components from the central Store. If correct data is not received by the components, the application would crash. Therefore, we need to test if the reducer is storing the data in a correct way and changes the state of the application as expected. Reducer also stores the current state properties which are fetched from the APIs. We need to check if reducers store the data in a correct way and if the data is received from the reducer when called from the angular components. In this blog, I would explain how to build up different components for unit testing reducers. Reducer to test This reducer is used for store the data from suggest.json API from the loklak server.The data received from the server is further classified into three properties which can be used by the components to show up auto- suggestions related to current search query. metadata: - This property stores the metadata from API suggestion response. entities: - This property stores the array of suggestions corresponding to the particular query received from the server. valid: - This is a boolean which keeps a check if the suggestions are valid or not. We also have two actions corresponding to this reducer. These actions, when called, changes the state properties which , further, supplies data to the components in a more classified manner. Moreover, state properties also causes change in the UI of the component according to the action dispatched. SUGGEST_COMPLETE_SUCCESS: - This action is called when the data is received successfully from the server. SUGGEST_COMPLETE_FAIL: - This action is called when the retrieving data from the server fails. export interface State { metadata: SuggestMetadata; entities: SuggestResults[]; valid: boolean; }export const initialState: State = { metadata: null, entities: [], valid: true };export function reducer(state: State = initialState, action: suggestAction.Actions): State { switch (action.type) { case suggestAction.ActionTypes.SUGGEST_COMPLETE_SUCCESS: { const suggestResponse = action.payload;return { metadata: suggestResponse.suggest_metadata, entities: suggestResponse.queries, valid: true }; }case suggestAction.ActionTypes.SUGGEST_COMPLETE_FAIL: { return Object.assign({}, state, { valid: false }); }default: { return state; } } } Unit tests for reducers Import all the actions, reducers and mocks import * as fromSuggestionResponse from './suggest-response'; import * as suggestAction from '../actions/suggest'; import { SuggestResponse } from '../models/api-suggest'; import { MockSuggestResponse } from '../shared/mocks/suggestResponse.mock';   Next, we are going to test if the undefined action doesn’t a cause change in the state and returns the initial state properties. We will be creating an action by const action = {} as any;  and call the reducer by const result = fromSuggestionResponse.reducer(undefined, action);. Now we will be making assertions with expect() block to check if the result is equal to initialState and all the initial state properties are…

Continue ReadingAdding Unit Test for Reducer in loklak search

CSS Styling Tips Used for loklak Apps

Cascading Style Sheets (CSS) is one of the main factors which is valuable to create beautiful and dynamic websites. So we use CSS for styling our apps in apps.loklak.org. In this blog post am going to tell you about few rules and tips for using CSS when you style your App: 1.Always try something new - The loklak apps website is very flexible according to the user whomsoever creates an app. The user is always allowed to use any new CSS frameworks to create an app. 2.Strive for Simplicity - As the app grows, we’ll start developing a lot more than we imagine like many CSS rules and elements etc. Some of the rules may also override each other without we noticing it. It’s good practice to always check before adding a new style rule—maybe an existing one could apply. 3.Proper Structured file - Maintain uniform spacing. Always use semantic or “familiar” class/id names. Follow DRY (Don’t Repeat Yourself) Principle. CSS file of Compare Twitter Profiles App: #searchBar { width:500px; } table { border-collapse: collapse; width: 70%; } th, td { padding: 8px; text-align: center; border-bottom: 1px solid#ddd; }   The output screen of the app: Do’s and Don'ts while using CSS: Pages must continue to work when style sheets are disabled. In this case this means that the apps which are written in apps.loklak.org should run in any and every case. Let's say for instance, when a user uses a old browsers or bugs or either because of style conflicts. Do not use the !important attribute to override the user's settings. Using the !important declaration is often considered bad practice because it has side effects that mess with one of CSS's core mechanisms: specificity. In many cases, using it could indicate poor CSS architecture. If you have multiple style sheets, then make sure to use the same CLASS names for the same concept in all of the style sheets. Do not use more than two fonts. Using a lot of fonts simply because you can will result in a messy look. A firm rule for home page design is more is less : the more buttons and options you put on the home page, the less users are capable of quickly finding the information they need. Resources: See more apps in apps.loklak.org. Checkout the code of the apps at: https://github.com/fossasia/apps.loklak.org . More about CSS and styling at https://www.w3.org/Style/CSS/Overview.en.html .

Continue ReadingCSS Styling Tips Used for loklak Apps

How the Compare Twitter Profiles loklak App works

People usually have a tendency to compare their profiles with others, So this is what exactly this app is used for: To compare Twitter profiles. loklak provides so many API’s which serves different functionalities. One among those API’s which I am using to implement this app is loklak’s User Details API. This API actually help in getting all the details of the user we search giving the user name as the query. In this app am going to implement a comparison between two twitter profiles which is shown in the form of tables on the output screen. Usage of loklak’s User Profile API in the app: In this app when the user given in the user names in the search fields as seen below: The queries entered into the search field are taken and used as query in the User Profile API. The query in the code is taken in the following form: var userQueryCommand = 'http://api.loklak.org/api/user.json?' + 'callback=JSON_CALLBACK&screen_name=' + $scope.query; var userQueryCommand1 = 'http://api.loklak.org/api/user.json?' + 'callback=JSON_CALLBACK&screen_name=' + $scope.query1; The query return a json output from which we fetch details which we need. A simple query and its json output: http://api.loklak.org/api/user.json?screen_name=fossasia Sample json output: { "search_metadata": {"client": "162.158.50.42"}, "user": { "$P": "I", "utc_offset": -25200, "friends_count": 282, "profile_image_url_https": "https://pbs.twimg.com/profile_images/1141238022/fossasia-cubelogo_normal.jpg", "listed_count": 185, "profile_background_image_url": "http://pbs.twimg.com/profile_background_images/882420659/14d1d447527f8524c6aa0c568fb421d8.jpeg", "default_profile_image": false, "favourites_count": 1877, "description": "#FOSSASIA #OpenTechSummit 2017, March 17-19 in Singapore https://t.co/aKhIo2s1Ck #OpenTech community of developers & creators #Code #Hardware #OpenDesign", "created_at": "Sun Jun 20 16:13:15 +0000 2010", "is_translator": false, "profile_background_image_url_https": "https://pbs.twimg.com/profile_background_images/882420659/14d1d447527f8524c6aa0c568fb421d8.jpeg", "protected": false, "screen_name": "fossasia", "id_str": "157702526", "profile_link_color": "DD2E44", "is_translation_enabled": false, "translator_type": "none", "id": 157702526, "geo_enabled": true, "profile_background_color": "F50000", "lang": "en", "has_extended_profile": false, "profile_sidebar_border_color": "000000", "profile_location": null, "profile_text_color": "333333", "verified": false, "profile_image_url": "http://pbs.twimg.com/profile_images/1141238022/fossasia-cubelogo_normal.jpg", "time_zone": "Pacific Time (US & Canada)", "url": "http://t.co/eLxWZtqTHh", "contributors_enabled": false, "profile_background_tile": true, }   I am getting data from the json outputs as shown above, I use different fields from the json output like screen_name, favourites_count etc. Injecting data from loklak API response using Angular: As the loklak’s user profile API returns a json format file, I am using Angular JS to align the data according to the needs in the app. I am using JSONP to retrieve the data from the API. JSONP or "JSON with padding" is a JSON extension wherein a prefix is specified as an input argument of the call itself. This how it is written in code: $http.jsonp(String(userQueryCommand)).success(function (response) { $scope.userData = response.user; }); Here the response is stored into a $scope is an application object here. Using the $scope.userData variable , we access the data and display it on the screen using Javascript, HTML and CSS. <div id="contactCard" style="pull-right"> <div class="panel panel-default"> <div class="panel-heading clearfix"> <h3 class="panel-title pull-left">User 1 Profile</h3> </div> <div class="list-group"> <div class="list-group-item"> <img src="{{userData.profile_image_url}}" alt="" style="pull-left"> <h4 class="list-group-item-heading" >{{userData.name}}</h4> </div> In this app am also adding keyboard action and validations of fields which will not allow users to search for an empty query using this simple line in the input field. ng-keyup="$event.keyCode == 13 && query1 != '' && query != '' ? Search()…

Continue ReadingHow the Compare Twitter Profiles loklak App works

Introducing Priority Kaizen Harvester for loklak server

In the previous blog post, I discussed the changes made in loklak’s Kaizen harvester so it could be extended and other harvesting strategies could be introduced. Those changes made it possible to introduce a new harvesting strategy as PriorityKaizen harvester which uses a priority queue to store the queries that are to be processed. In this blog post, I will be discussing the process through which this new harvesting strategy was introduced in loklak. Background, motivation and approach Before jumping into the changes, we first need to understand that why do we need this new harvesting strategy. Let us start by discussing the issue with the Kaizen harvester. The produce consumer imbalance in Kaizen harvester Kaizen uses a simple hash queue to store queries. When the queue is full, new queries are dropped. But numbers of queries produced after searching for one query is much higher than the consumption rate, i.e. the queries are bound to overflow and new queries that arrive would get dropped. (See loklak/loklak_server#1156) Learnings from attempt to add blocking queue for queries As a solution to this problem, I first tried to use a blocking queue to store the queries. In this implementation, the producers would get blocked before putting the queries in the queue if it is full and would wait until there is space for more. This way, we would have a good balance between consumers and producers as the consumers would be waiting until producers can free up space for them - public class BlockingKaizenHarvester extends KaizenHarvester {    ...    public BlockingKaizenHarvester() {        super(new KaizenQueries() {            ...            private BlockingQueue<String> queries = new ArrayBlockingQueue<>(maxSize);            @Override            public boolean addQuery(String query) {                if (this.queries.contains(query)) {                    return false;                }                try {                    this.queries.offer(query, this.blockingTimeout, TimeUnit.SECONDS);                    return true;                } catch (InterruptedException e) {                    DAO.severe("BlockingKaizen Couldn't add query: " + query, e);                    return false;                }            }            @Override            public String getQuery() {                try {                    return this.queries.take();                } catch (InterruptedException e) {                    DAO.severe("BlockingKaizen Couldn't get any query", e);                    return null;                }            }            ...        });    } } [SOURCE, loklak/loklak_server#1210] But there is an issue here. The consumers themselves are producers of even higher rate. When a search is performed, queries are requested to be appended to the KaizenQueries instance for the object (which here, would implement a blocking queue). Now let us consider the case where queue is full and a thread requests a query from the queue and scrapes data. Now when the scraping is finished, many new queries are requested to be inserted to most of them get blocked (because the queue would be full again after one query getting inserted). Therefore, using a blocking queue in KaizenQueries is not a good thing to do. Other considerations After the failure of introducing the Blocking Kaizen harvester, we looked for other alternatives for storing queries. We came across multilevel queues, persistent disk queues and priority queues. Multilevel queues sounded like a good idea at first where we would have multiple queues for storing queries. But eventually, this would just boil down to how…

Continue ReadingIntroducing Priority Kaizen Harvester for loklak server

Fetching URL for Embedded Twitter Videos in loklak server

The primary web service that loklak scrapes is Twitter. Being a news and social networking service, Twitter allows its users to post videos directly to Twitter and they convey more thoughts than what text can. But for an automated scraper, getting the links is not a simple task. Let us see that what were the problems we faced with videos and how we solved them in the loklak server project. Previous setup and embedded videos In the previous version of loklak server, the TwitterScraper searched for videos in 2 ways - Youtube links HTML5 video links To fetch the video URL from HTML5 video, following snippet was used - if ((p = input.indexOf("<source video-src")) >= 0 && input.indexOf("type=\"video/") > p) {    String video_url = new prop(input, p, "video-src").value;    videos.add    continue; } Here, input is the current line from raw HTML that is being processed and prop is a class defined in loklak that is useful in parsing HTML attributes. So in this way, the HTML5 videos were extracted. The Problem - Embedded videos Though the previous setup had no issues, it was useless as Twitter embeds the videos in an iFrame and therefore, can’t be fetched using simple HTML5 tag extraction. If we take the following Tweet for example, the requested HTML from the search page contains video in following format - <src="https://twitter.com/i/videos/tweet/881946694413422593?embed_source=clientlib&player_id=0&rpc_init=1" allowfullscreen="" id="player_tweet_881946694413422593" style="width: 100%; height: 100%; position: absolute; top: 0; left: 0;"> So we needed to come up with a better technique to get those videos. Parsing video URL from iFrame The <div> which contains video is marked with AdaptiveMedia-videoContainer class. So if a Tweet has an iFrame containing video, it will also have the mentioned class. Also, the source of iFrame is of the form https://twitter.com/i/videos/tweet/{Tweet-ID}. So now we can programmatically go to any Tweet’s video and parse it to get results. Extracting video URL from iFrame source Now that we have the source of iFrame, we can easily get the video source using the following flow - public final static Pattern videoURL = Pattern.compile("video_url\\\":\\\"(.*?)\\\""); private static String[] fetchTwitterIframeVideos(String iframeURL) {    // Read fron iframeURL line by line into BufferReader br    while ((line = br.readLine()) != null ) {        int index;        if ((index = line.indexOf("data-config=")) >= 0) {            String jsonEscHTML = (new prop(line, index, "data-config")).value;            String jsonUnescHTML = HtmlEscape.unescapeHtml(jsonEscHTML);            Matcher m = videoURL.matcher(jsonUnescHTML);            if (!m.find()) {                return new String[]{};            }            String url = m.group(1);            url = url.replace("\\/", "/");  // Clean URL            /*             * Play with url and return results             */        }    } } MP4 and M3U8 URLs If we encounter mp4 URLs, we’re fine as it is the direct link to video. But if we encounter m3u8 URL, we need to process it further before we can actually get to the videos. For Twitter, the hosted m3u8 videos contain link to further m3u8 videos which are of different resolution. These m3u8 videos again contain link to various .ts files that contain actual video in parts of 3 seconds length each to support better streaming experience on…

Continue ReadingFetching URL for Embedded Twitter Videos in loklak server

Create Scraper in Javascript for Loklak Scraper JS

Loklak Scraper JS is the latest repository in Loklak project. It is one of the interesting projects because of expected benefits of Javascript in web scraping. It has a Node Javascript engine and is used in Loklak Wok project as bundled package. It has potential to be used in different repositories and enhance them. Scraping in Python is easy (at least for Pythonistas) as one needs to just import Request library and BeautifulSoup library (lxml as better option), write some lines of code using Request library to get webpage and some lines of bs4 to walk through html and scrape data. This sums up to about less than a hundred lines of coding, where as Javascript coding isn't easily readable (at least to me) as compared to Python. But it has an advantage, it can easily deal with Javascript in the pages we are scraping. This is one of the motive, Loklak Scraper JS repository was created and we contributed and worked on it. I recently coded a Javascript scraper in loklak_scraper_js repository. While coding, I found it’s libraries similar to the libraries, I use to code in Python. Therefore, this blog is for Pythonistas how they can start scraping in Javascript as they finish reading and also contribute to Loklak Scraper JS. First, replace Python interpreter, Request and Beautifulsoup library with Node JS interpreter, Request and Cheerio JS library. 1) Node JS Interpreter: Node JS Interpreter is used to interpret Javascript files. This is different from Python as it deals with the project instead of a module in case of Python. The most compatible Node for most of the libraries is 6.0.0 , where as latest version available(as I checked) is 8.0.0 TIP: use `--save` with npm like here while installing a library. 2) Request Library :- This is used to load webpage to be processed. Similar to one in Python. Request-promise library, a wrapper around Request with implementation of Bluebird library, improves readability and makes code cleaner (how?).   3) Cheerio Library:- A Pythonista (a rookie one) can call it twin of BeautifulSoup Library. But this is faster and is Javascript. It's selector implementation is nearly identical to jQuery's. Let us code a basic Javascript scraper. I will take TimeAndDate scraper from loklak_scraper_js as example here. It inputs place and outputs its local time. Step#1: fetching HTML from webpage with the help of Request library. We input url to Request function to fetch the webpage and is saved to `html` variable. This scrapeTimeAndDate() function scrapes data from html url = "http://www.timeanddate.com/worldclock/results.html?query=London"; request(url, function(error, response, body) { if(error) { console.log("Error: " + error); process.exit(-1); } html = body; scrapeTimeAndDate() });   Step#2: To scrape important data from html using Cheerio JS list of date and time of locations is embedded in table tag, So we will iterate through <td> and extract text. a) Load html to Cheerio as we do in beautifulsoup In Python soup = BeautifulSoup(html,'html5lib')   In Cheerio JS $ = cheerio.load(html);   b) This line finds…

Continue ReadingCreate Scraper in Javascript for Loklak Scraper JS

Best Practices when writing Tests for loklak Server

Why do we write unit-tests? We write them to ensure that developers’ implementation doesn't change the behaviour of parts of the project. If there is a change in the behaviour, unit-tests throw errors. This keep developers in ease during integration of the software and ensure lower chances of unexpected bugs. After setting up the tests in Loklak Server, we were able to check whether there is any error or not in the test. Test failures didn’t mention the error and the exact test case at which they failed. It was YoutubeScraperTest that brought some of the best practices in the project. We modified the tests according to it. The following are some of the best practices in 5 points that we shall follow while writing unit tests: Assert the assertions There are many assert methods which we can use like assertNull, assertEquals etc. But we should use one which describes the error well (being more descriptive) so that developer's effort is reduced while debugging. Using these assertions related preferences help in getting to the exact errors on test fails, thus helping in easier debugging of the code. Some examples can be:- Using assertThat() over assertTrue assertThat() give more descriptive errors over assertTrue(). Like:- When assertTrue() is used: java.lang.AssertionError: Expected: is <true> but: was <false> at org.loklak.harvester.TwitterScraperTest.testSimpleSearch(TwitterScraperTest.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at org.hamcr..........   When assertThat() is used: java.lang.AssertionError: Expected: is <true> but: was <false> at org.loklak.harvester.TwitterScraperTest.testSimpleSearch(TwitterScraperTest.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at org.hamcr...........   NOTE:- In many cases, assertThat() is preferred over other assert method (read this), but in some cases other methods are used to give better descriptive output (like in next examples) Using assertEquals() over assertThat() For assertThat() java.lang.AssertionError: Expected: is "ar photo #test #car https://pic.twitter.com/vd1itvy8Mx" but: was "car photo #test #car https://pic.twitter.com/vd1itvy8Mx" at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Ass........   For assertEquals() org.junit.ComparisonFailure: expected:<[c]ar photo #test #car ...> but was:<[]ar photo #test #car ...> at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at org.loklak.harvester.Twitter.........   We can clearly see that second example gives better error description than the first one.(An SO link) One Test per Behaviour Each test shall be independent of other with none having mutual dependencies. It shall test only a specific behaviour of the module that is tested. Have a look of this snippet. This test checks the method that creates the twitter url by comparing the output url method with the expected output url. @Test public void testPrepareSearchURL() { String url; String[] query = { "fossasia", "from:loklak_test", "spacex since:2017-04-03 until:2017-04-05" }; String[] filter = {"video", "image", "video,image", "abc,video"}; String[] out_url = { "https://twitter.com/search?f=tweets&vertical=default&q=fossasia&src=typd", "https://twitter.com/search?f=tweets&vertical=default&q=fossasia&src=typd", }; // checking simple urls for (int i = 0; i < query.length; i++) { url = TwitterScraper.prepareSearchURL(query[i], ""); //compare urls with urls created assertThat(out_url[i], is(url)); } }   This unit-test tests whether the method-under-test is able to create twitter link according to query or not. Selecting test cases for the test We shall remember that testing is a very costly task in terms of processing. It takes time to execute. That is why, we need to keep the…

Continue ReadingBest Practices when writing Tests for loklak Server

Fetching URL for Complete Twitter Videos in loklak server

In the previous blog post, I discussed how to fetch the URLs for Twitter videos in parts (.ts extension). But getting a video in parts is not beneficial as the loklak users have to carry out the following task in order to make sense out of it: Placing the videos in correct order (the videos are divided into 3-second sections). Having proper libraries and video player to play the .ts extension. This would require fairly complex loklak clients and hence the requirement was to have complete video in a single link with a popular extension. In this blog post, I’ll be discussing how I managed to get links to complete Twitter videos. Guests and Twitter Videos Most of the content on Twitter is publicly accessible and we don’t need an account to access it. And this public content includes videos too. So, there should be some way in which Twitter would be handling guest users and serving them the videos. We needed to replicate the same flow in order to get links to those videos. Problem with Twitter video and static HTML In Twitter, the videos are not served with the static HTML of a page. It is generally rendered using a front-end JavaScript framework. Let us take an example of mobile.twitter.com website. Let us consider the video from a tweet of @HiHonourIndia - We can see that the page is rendered using ReactJS and we also have the direct link for the video - “So what’s the problem then? We can just request the web page and parse HTML to get video link, right?” Wrong. As I mentioned earlier, the pages are rendered using React and when we initially request it, it looks something like this - The HTML contains no link to video whatsoever, and keeping in mind that we would be getting the previously mentioned HTML, the scraper wouldn’t be getting any video link either. We, therefore, need to mimic the flow which is followed internally in the web app to get the video link and play them. Mimicking the flow of Twitter Mobile to get video links After tracking the XHR requests made to by the Twitter Mobile web app, one can come up with the forthcoming mentioned flow to get video URLs. Mobile URL for a Tweet Getting mobile URL for a tweet is very simple - String mobileUrl = "https://mobile.twitter.com" + tweetUrl; Here, tweet URL is of the type /user/tweetID. Guest Token and Bearer JS URL The Bearer JS is a file which contains Bearer Token which along with a Guest Token is used to authenticate Twitter API to get details about a conversation. The guest token and bearer script URL can be extracted from the static mobile page - Pattern bearerJsUrlRegex = Pattern.compile(“showFailureMessage\\(\\'(.*?main.*?)\\’\\);”); Pattern guestTokenRegex = Pattern.compile(“document\\.cookie \\= decodeURIComponent\\(\\\”gt\\=([0-9]+);”); ClientConnection conn = new ClientConnection(mobileUrl); BufferedReader br = new BufferedReader(new InputStreamReader(conn.inputStream, StandardCharsets.UTF_8)); String line; while ((line = br.readLine()) != null) { if (bearerJsUrl != null && guestToken != null) { // Both the entities…

Continue ReadingFetching URL for Complete Twitter Videos in loklak server

Improving Performance of the Loklak Search with Lazy Loading Images

Loklak Search initially faced a problem of huge load time because of a high number of parallel HTTP requests. Feeds page of the search engine made near to 50 parallel HTTP requests on every new search. One of the possible ways to reduce load time in long pages is to lazily load images. Since loklak is a multi-platform web application, a majority of your site's visitors might use the application from high-latency devices (i.e. mobiles, tablets), then lazy-loading is required for a smooth user experience (as website speed equals User Experience). I am explaining in this blog about how I took advantage of a directive to implement lazy loading of images and how the performance of the application improved. What is lazy loading? As this project website states, “Lazy loading is just the opposite of ‘pre-loading images’. The basic idea behind lazy loading is to keep a number of parallel request low. The amount of request is kept low until the user scrolls down then the images will load.” This idea is used by Google images to reduce the number of parallel requests. As we can look in the image below, the amount of parallel request for images sent to the server without lazy loading: Using viewport directive in loklak to lazily load images Lazy loading can be implemented in Angular using viewport directive. We need to setup viewport and start using it in feed component to lazy load the profile pictures of the users and the images. Using viewport directive in Components In this directive, we have two observables :- Scroll which keeps a check on the scroll position of the page this.scroll = Observable.fromEvent(window, 'scroll').subscribe((event) => { this.check(); });   Resize which keeps a check on the resize event when the browser window changes size this.resize = Observable.fromEvent(window, 'resize').subscribe((event) => { this.check(); });   Now, whenever the resize and scroll event occurs, it calls a method check which calculates the dimensional parameters of the element and whenever element has entered the viewport it emits an event (which is of type boolean) with value true. check() function takes up reference of the element. Next, it calculates area occupied by this element i.e. elementSize. Next, it looks for the position of the element within the viewport with getBoundingClientRect(). Parameter partial can be set to true if the we need to check if image appears in viewport partially. Moreover, we can specify parameter direction to specify the direction from which the reference element will be entering. Now, we would use conditional statements to check if element exists within the dimensions of viewport and would emit an event. check(partial:boolean = true, direction:string = 'both') { const el = this._el.nativeElement;const elSize = (el.offsetWidth * el.offsetHeight);const rec = el.getBoundingClientRect();const vp = { width: window.innerWidth, height: window.innerHeight };const tViz = rec.top >= 0 && rec.top < vp.height; const bViz = rec.bottom > 0 && rec.bottom <= vp.height;const lViz = rec.left >= 0 && rec.left < vp.width; const rViz = rec.right > 0 && rec.right <=…

Continue ReadingImproving Performance of the Loklak Search with Lazy Loading Images

Implementing Auto-Suggestions in loklak search

Auto-suggestions can add a friendly touch to the search engine. Loklak provides suggest.json API to give suggestions based on the previous search terms entered by the users. Moreover, suggest results needs to be reactive and quick enough to be displayed as soon as the user types a new character. The main demand of the auto-suggestion feature was to make it really quick so as to make it look reactive. I will explain how I implemented auto-suggestion feature and try to explain issues I faced and solution for that. Ngrx Effects The cycle for implementing auto-suggest goes like this: The most important component in this cycle is effects as it is the event listener and it recognises the action immediately after it is dispatched and makes a call to loklak suggest.json API. We will look at how effects should look like for a reactive implementation of auto-suggestion. Making effects run effectively can make Rate Determining Step to be the API response time instead of any other component. @Injectable() export class SuggestEffects {@Effect() suggest$: Observable<Action> = this.actions$ .ofType(suggestAction.ActionTypes.SUGGEST_QUERY) .debounceTime(300) .map((action: suggestAction.SuggestAction) => action.payload) .switchMap(query => { const nextSuggest$ = this.actions$.ofType(suggestAction.ActionTypes.SUGGEST_QUERY).skip(1);return this.suggestService.fetchQuery(query) .takeUntil(nextSuggest$) .map(response => { return new suggestAction.SuggestCompleteSuccessAction(response); }) .catch(() => of(new suggestAction.SuggestCompleteFailAction(''))); });constructor( private actions$: Actions, private suggestService: SuggestService, ) { }}   This effect basically listens to the action  SUGGEST_QUERY and recognises the action, next it makes a call to the Suggestion service which receives data from the server. Now, as the data is received, it maps the response and passes the response to the SuggestCompleteSuccessAction so that it could change the considered state properties. The debounce time is kept low (equal to 300) so as to detect next SUGGEST_QUERY within next 300ms of the API suggest.json call. This will help to whole cycle of suggest response reactive and help in early detection of the action. Angular Components In this component, I will explain what changes I made to the feed header component for autocomplete to run effectively. @Component({ selector: 'feed-header-component', templateUrl: './feed-header.component.html', styleUrls: ['./feed-header.component.scss'], changeDetection: ChangeDetectionStrategy.OnPush }) export class FeedHeaderComponent { public suggestQuery$: Observable<Query>; public isSuggestLoading$: Observable<boolean>; public suggestResponse$: Observable<SuggestResults[]>; public searchInputControl = new FormControl(); constructor( private store: Store<fromRoot.State>, ) { }ngOnInit() { this.getDataFromStore(); this.setupSearchField(); }private getDataFromStore(): void { this.suggestQuery$ = this.store.select(fromRoot.getSuggestQuery); this.isSuggestLoading$ = this.store.select(fromRoot.getSuggestLoading); this.suggestResponse$ = this.store.select(fromRoot.getSuggestResponseEntities); }private setupSearchField(): void { this.__subscriptions__.push( this.searchInputControl .valueChanges .subscribe(query => { this.store.dispatch(new suggestAction.SuggestAction(value)); }) ); } }   We have created a FormControl searchInputControl which would keep a check on the input value of the search box and would dispatch an action SuggestAction() as soon as its value changes. Next, we have to subscribe to the observables of suggestion entities from the store as in the function “this.getDataFromStore()”.The data is received from the store and now we can proceed with displaying suggestion data in the template. Template HTML language has an tag datalist which can be used to display auto-suggestion in the template but datalist shows less compatibility with the Angular application and makes rendering HTML slow with next suggestion based on…

Continue ReadingImplementing Auto-Suggestions in loklak search