Autolinker Component in Loklak Search

In Loklak Search the post items contain links, which are either internal or external. These links include the hashtags, mentions, and URLs. From the backend server we just received the message in the plain text format, and thus there is need to parse the plain text and render it as clickable links. These clickable links can be either internal or external. Thus we need an auto-linker component, which takes the text and render it as links. The API of the Component The component takes as the input the plain text, then four arrays of strings. Each containing the text to be linked. These are hashtags, mentions, links and the unshorten attribute which is used to unshorten the shortened URLs in the post. These attributes are used by the component to render the text in the appropriate format. export class FeedLinkerComponent implements OnInit { @Input() text: string; @Input() hashtags: string[] = new Array<string>(); @Input() mentions: string[] = new Array<string>(); @Input() links: string[] = new Array<string>(); @Input() unshorten: Object = {}; } The Logic of the Component The basic logic of the component works as the following, we divide the text into chunks known as shards, we have three basic data structures for the component to work The ShardType which is the type of the chunk it specifies whether it is plain, hashtags, mentions, and links. The Shard which is the simple object containing the text to show, its type and the link it refers to The StringIndexdChunks, they are utilized to index the chunks in the order in which they appear in the text. const enum ShardType { plain, // 0 link, // 1 hashtag, // 2 mention // 3 } class Shard { constructor ( public type: ShardType = ShardType.plain, public text: String = '', public linkTo: any = null, public queryParams: any = null ) { } } interface StringIndexedChunks { index: number; str: string; type: ShardType; } First we have a private method of the component which searches for all the elements (strings) in the text. Here we have an array which maintains the index of those chunks in the text. private generateShards() { const indexedChunks: StringIndexedChunks[] = []; this.hashtags.forEach(hashtag => { const indices = getIndicesOf(this.text, `#${hashtag}`, false); indices.forEach(idx => { indexedChunks.push({index: idx, str: `#${hashtag}`, type: ShardType.hashtag}); }); }); this.mentions.forEach(mention => { const indices = getIndicesOf(this.text, `@${mention}`, false); indices.forEach(idx => { indexedChunks.push({index: idx, str: `@${mention}`, type: ShardType.mention}); }); }); } Then we sort the chunks according to their indexes in the text. This gives us sorted array which consists of all the chunks sorted according to the indexes as they appear in the text. indexedChunks.sort((a, b) => { return (a.index > b.index) ? 1 : (a.index < b.index) ? -1 : 0; }); The next part of the logic is to generate the shard array, an array which contains each chunk, once. To do this we iterate over the Sorted Indexed array created in the previous step and use it split the text into chunks. We iterate over the…

Continue ReadingAutolinker Component in Loklak Search

Service Workers in Loklak Search

Loklak search is a web application which is built on latest web technologies and is aiming to be a progressive web application. A PWA is a web application which has a rich, reliable, fast, and engaging web experience, and web API which enables us to get these are Service Workers. This blog post describes the basics of service workers and their usage in the Loklak Search application to act as a Network Proxy to and the programmatical cache controller for static resources. What are Service Workers? In the very formal definition, Matt Gaunt describes service workers to be a script that the browser runs in the background, and help us enable all the modern web features. Most these features include intercepting network requests and caching and responding from the cache in a more programmatical way, and independent from native browser based caching. To register a service worker in the application is a really simple task, there is just one thing which should be kept in mind, that service workers need the HTTPS connection, to work, and this is the web standard made around the secure protocol. To register a service worker if ('serviceWorker' in navigator) { window.addEventListener('load', function() { navigator.serviceWorker.register('/sw.js').then(function(registration) { // Registration was successful console.log('ServiceWorker registration successful with scope: ', registration.scope); }, function(err) { // registration failed :( console.log('ServiceWorker registration failed: ', err); }); }); } This piece of javascript, if the browser supports, registers the service worker defined by sw.js. The service worker then goes through its lifecycle, and gets installed and then it takes control of the page it gets registered with. What does service workers solve in Loklak Search? In loklak search, service workers currently work as a, network proxy to work as a caching mechanism for static resources. These static resources include the all the bundled js files and images. These bundled chunks are cached in the service workers cache and are responded with from the cache when requested. The chunking of assets have an advantage in this caching strategy, as the cache misses only happen for the chunks which are modified, and the parts of the application which are unmodified are served from the cache making it possible for lesser download of assets to be served. Service workers and Angular As the loklak search is an angular application we, have used the @angular/service-worker library to implement the service workers. This is simple to integrate library and works with the, CLI, there are two steps to enable this, first is to download the Service Worker package npm install --save @angular/service-worker And the second step is to enable the service worker flag in .angular-cli.json "apps": [   {      // Other Configurations      serviceWorker: true   } ] Now when we generate the production build from the CLI, along with all the application chunks we get, The three files related to the service workers as well sw-register.bundle.js : This is a simple register script which is included in the index page to register the service worker. worker-basic.js : This is…

Continue ReadingService Workers in Loklak Search

Analyzing Production Build Size in Loklak Search

Loklak search being a web application it is critical to keep the size of the application in check to ensure that we are not transferring any non-essential bytes to the user so that application load is faster, and we are able to get the minimal first paint time. This requires a mechanism for the ability to check the size of the build files which are generated and served to the user. Alongside the ability to check sizes it is also critically important to analyze the distribution of the modules along with their sizes in various chunks. In this blog post, I discuss the analysis of the application code of loklak search and the generated build files. Importance of Analysis The chunk size analysis is critical to any application, as the chunk size of any application directly determines the performance of any application, at any scale. The smaller the application the lesser is the load time, thus faster it becomes usable at the user side. The time to first-paint is the most important metric to keep in mind while analyzing any web application for performance, though the first paint time consists of many critical parts from loading, parsing, layout and paint, but still the size of any chunk determines all the time it will take to render it on the screen. Also as we use the 3rd party libraries and components it becomes crucially important to inspect the impact on the size of the application upon the inclusion of those libraries and components. Development Phase Checking Angular CLI provides a clean mechanism to track and check the size of all the chunks always at the runtime, these stats simply show the size of each chunk in the application in the terminal on every successful compilation, and this provides us a broad idea about the chunks to look and address. Deep Analysis using Webpack Bundle Analyzer The angular cli while generating the production build provides us with an option to generates the statistics about the chunks including the size and namespaces of the each module which is part of that chunk. These stats are directly generated by the webpack at the time of bundling, code splitting, and tree shaking. These statistics thus provide us to peek into the actual deeper level of chunk creation in webpack to analyze sizes of its various components. To generate the statistics we just need to enable the --stats-json flag while building. ng serve --prod --aot --stats-json This will generate the statistics file for the application in the /dist directory, alongside all the bundles. Now to have the visual and graphical analysis of these statistics we can use a tool like webpack-bundle-analyzer to analyze the statistics. We can install the webpack-bundle-analyzer via npm, npm install --save-dev webpack-bundle-analyzer Now, to our package.json we can add a script, running this script will open up a web page which contains graphical visualization of all the chunks build in the application // package.json {   …   …   {      “scripts”: {         …         …         "analyze":…

Continue ReadingAnalyzing Production Build Size in Loklak Search

Using CSS Grid in Loklak Search

CSS Grid is the latest web standard for the layouts in the web applications. This is the web standard which allows the HTML page to be viewed as 2-dimensional for laying out the elements in the page. It is thus used in parts of loklak search for layout. In this blog post, I will discuss the basic naming convention for CSS grid and its usage in Loklak Search for layout structuring and responsiveness. CSS Grid Basics There are some basic terminologies regarding grid few major ones are the following Grid Container The grid container is the container which is the wrapper of all the grid items. It is declared by display: grid, this makes all the direct children of that element to become grid items. Grid Tracks We define rows and columns of the grid as the lines, the area between any two lines is called a grid track. Tracks can be defined using any length unit. Grid also introduces an additional length unit to help us create flexible grid tracks. The new fr unit represents a fraction of the available space in the grid container. Grid Cells The area between any two horizontal and vertical lines is called a grid cell. Grid Area The area formed by the combination of two or more cells is called a grid area. Using CSS grid in Loklak Search The CSS grid is used in loklak search uses CSS grid in the feeds page to align elements in a responsive way on mobile and desktop. Earlier there was the issue that on small displays the info box of the results appeared after the feed results, and we needed to make sure that it appears on top on smaller displays. This is the outline of the structure of the feed page. <div class=”feed-wrapper”> <div class=”feed-results”> <!-- Feed Results --> </div> <div class=”feed-info-box”> <!-- Feed Info Box --> </div> </div> Now we use the CSS grid to position the items according to the display width. First we declare the “feed-wrapper” as display:grid to make it a Grid Container, and we associate the rows and columns accordingly. .feed-wrapper { display: grid; grid-template-columns: 150px 632px 455px 1fr; grid-template-rows: auto; } This defines the grid to be consisting of 4 columns of width 150px, 632px,  455px and one remaining unit i.e. 1fr. The rows are set to be auto. Now we define the grid areas i.e. the names of the areas using the grid-area:<area> css property. This gives names to the elements in the CSS grid. .feed-results { grid-area: feed-results; } .feed-info-box { grid-area: feed-info-box; } The last thing which remains now is to specify the position of these grid elements in the grid cells according to the display width, we use simple media queries along with simple grid area positioning property, i.e. grid-template-areas. .feed-wrapper { /* Other Properties */   @media(min-width: 1200px) { grid-template-areas: ". feed-results feed-info-box .";   } @media(max-width: 1199px) { grid-template-columns: 1fr; grid-template-areas: "feed-info-box" "feed-results"; } } This positions both the boxes according to the display…

Continue ReadingUsing CSS Grid in Loklak Search

Reactive Side Effects of Actions in Loklak Search

In a Redux based application, every component of the application is state driven. Redux based applications manage state in a predictable way, using a centralized Store, and Reducers to manipulate various aspects of the state. Each reducer controls a specific part of the state and this allows us to write the code which is testable, and state is shared between the components in a stable way, ie. there are no undesired mutations to the state from any components. This undesired mutation of the shared state is prevented by using a set of predefined functions called reducers which are central to the system and updates the state in a predictable way. These reducers to update the state require some sort triggers to run. This blog post concentrates on these triggers, and how in turn these triggers get chained to form a Reactive Chaining of events which occur in a predictable way, and how this technique is used in latest application structure of Loklak Search. In any state based asynchronous application, like, Loklak Search the main issue with state management is to handle the asynchronous action streams in a predictable manner and to chain asynchronous events one after the other.  The technique of reactive action chaining solves the problem of dealing with asynchronous data streams in a predictable and manageable manner. Overview Actions are the triggers for the reducers, each redux action consists of a type and an optional payload. Type of the action is like its ID which should be purposely unique in the application. Each reducer function takes the current state which it controls and action which is dispatched. The reducer decides whether it needs to react to that action or not. If the user reacts to the action, it modifies the state according to the action payload and returns the modified state, else, it returns the original state. So at the core, the actions are like the triggers in the application, which make one or more reducers to work. This is the basic architecture of any redux application. The actions are the triggers and reducers are the state maintainers and modifiers. The only way to modify the state is via a reducer, and a reducer only runs when a corresponding action is dispatched. Now, who dispatches these actions? This question is very important. The Actions can be technically dispatched from anywhere in the application, from components, from services, from directives, from pipes etc. But we almost in every situation will always want the action to be dispatched by the component. Component who wishes to modify the state dispatch the corresponding actions. Reactive Effects If the components are the one who dispatch the action, which triggers a reducer function which modifies the state, then what are these effects, cause the cycle of events seem pretty much complete. The Effects are the Side Effects, of a particular action. The term “side effect” means these are the piece of code which runs whenever an action is dispatched. Don’t confuse them with the…

Continue ReadingReactive Side Effects of Actions in Loklak Search

Route Based Chunking in Loklak Search

The loklak search application running at loklak.org is growing in size as the features are being added into the application, this growth is a linear one, and traditional SPA, tend to ship all the code is required to run the application in one pass, as a single monolithic JavaScript file, along with the index.html. This approach is suitable for the application with few pages which are frequently used, and have context switching between those logical pages at a high rate and almost simultaneously as the application loads. But generally, only a fraction of code is what is accessed most frequently by most users, so as the application size grows it does not make sense to include all the code for the entire application at the first request, as there is always some code in the application, some views, are rarely accessed. The loading of such part of the application can be delayed until they are accessed in the application. The angular router provides an easy way to set up such system and is used in the latest version of loklak search. The technique which is used here is to load the content according to the route. This makes sure only the route which is viewed is loaded on the initial load, and subsequent loading is done at the runtime as and when required. Old setup for baseline Here are the compiled file sizes, of the project without the chunking the application. Now as we can see that the file sizes are huge, especially the vendor bundle, which is of 5.4M and main bundle which is about 0.5M now, these are the files which are loaded on the first load and due to their huge sizes, the first paint of the application suffers, to a great extent. These numbers will act as a baseline upon which we will measure the impact of route based chunking. Setup for route based chunking The setup for route based chunking is fairly simple, this is our routing configuration, the part of the modules which we want to lazy load are to be passed as loadChildren attribute of the route, this attribute is a string which is a path of the feature module which, and part after the hash symbol is the actual class name of the module, in that file. This setup enables the router to load that module lazily when accessed by the user. const routes: Routes = [ { path: '', pathMatch: 'full', loadChildren: './home/home.module#HomeModule', data: { preload: true } }, { path: 'about', loadChildren: './about/about.module#AboutModule' }, { path: 'contact', loadChildren: './contact/contact.module#ContactModule' }, { path: 'search', loadChildren: './feed/feed.module#FeedModule', data: { preload: true } }, { path: 'terms', loadChildren: './terms/terms.module#TermsModule' }, { path: 'wall', loadChildren: './media-wall/media-wall.module#MediaWallModule' } ]; Preloading of some routes As we can see that in two of the configurations above, there is a data attribute, on which preload: true attribute is specified. Sometimes we need to preload some part of theapplication, which we know we will access, soon enough. So angular…

Continue ReadingRoute Based Chunking in Loklak Search

Lazy Loading Images in Loklak Search

In last blog post, I discussed the basic Web API’s which helps us to create the lazy image loader component. I also discussed the structure which is used in the application, to load the images lazily. The core idea is to wrap the <img> element in a wrapper, <app-lazy-img> element. This enables us the detection of the element in the viewport and corresponding loading only if the image is present in the viewport. In this blog post, I will be discussing the implementation details about how this is achieved in Loklak search in an optimized manner. The logic for lazy loading of images in the application is divided into a Component and a corresponding Service. The reason for this splitting of logic will be explained as we discuss the core parts of the code for this feature. Detecting the Intersection with Viewport The lazy image service is a service for the lazy image component which is registered at the by the modules which intend to use this app lazy image component. The task of this service is to register the elements with the intersection observer, and, then emit an event when the element comes in the viewport, which the element can react on and then use the other methods of services to actually fetch the image. @Injectable() export class LazyImgService { private intersectionObserver: IntersectionObserver = new IntersectionObserver(this.observerCallback.bind(this), { rootMargin: '50% 50%' }); private elementSubscriberMap: Map<Element, Subscriber<boolean>> = new Map<Element, Subscriber<boolean>>(); } The service has two member attributes, one is IntersectionObserver, and the other is a Map which stores the the reference of the subscribers of this intersection observer. This reference is then later used to emit the event when the element comes in viewport. The rootMargin of the intersection observer is set to 50% this makes sure that when the element is 50% away from the viewport. The obvserve public method of the service, takes an element and pass it to intersection observer to observe, also put the element in the subscriber map. public observe(element: Element): Observable<boolean> { const observable: Observable<boolean> = new Observable<boolean>(subscriber => { this.elementSubscriberMap.set(element, subscriber); }); this.intersectionObserver.observe(element); return observable; } Then there is the observer callback, this method, as an argument receives all the objects intersecting the root of the observer, when this callback is fired, we find all the intersecting elements and emit the intersection event. Indicating that the element is nearby the viewport and this is the time to load it. private observerCallback(entries: IntersectionObserverEntry[], observer: IntersectionObserver) { entries.forEach(entry => { if (this.elementSubscriberMap.has(entry.target)) { if (entry.intersectionRatio > 0) { const subscriber = this.elementSubscriberMap.get(entry.target); subscriber.next(true); this.elementSubscriberMap.delete(entry.target); } } }); } Now, our LazyImgComponent enables us to uses this service to register its element, with the intersection observer and then reacting to it later, when the event is emitted. This method sets up the IO, to load the image, and subscribes to the event emittes by the service and eventually calls the loadImage method when the element intersects with the viewport. private setupIntersectionObserver() { this.lazyImgService .observe(this.elementRef.nativeElement) .subscribe(value =>…

Continue ReadingLazy Loading Images in Loklak Search

Lazy loading images in Loklak Search

Loklak Search delivers the media rich content to the users. Most of the media delivered to the users are in the form of images. In the earlier versions of loklak search, these images were delivered to the users imperatively, irrespective of their need. What this meant is, whether the image is required by the user or not it was delivered, consuming the bandwidth and slowing down the initial load of the app as large amount of data was required to be fetched before the page was ready. Also, the 404 errors were also not being handled, thus giving the feel of a broken UI. So we required a mechanism to control this loading process and tap into its various aspects, to handle the edge cases. This, on the whole, required few new Web Standard APIs to enable smooth working of this feature. These API’s are IntersectionObserver API Fetch API   As the details of this feature are involving and comprise of new API standards, I have divided this into two posts, one with the basics of the above mentioned API’s and the outline approach to the module and its subcomponents and the difficulties which we faced. The second post will mostly comprise of the details of the code which went into making this feature and how we tackled the corner cases in the path. Overview Our goal here was to create a component which can lazily load the images and provide UI feedback to the user in case of any load or parse error. As mentioned above the development of this feature depends on the relatively new web standards API’s so it’s important to understand the functioning of these AP’s we understand how they become the intrinsic part of our LazyImgComponent. Intersection Observer If we see history, the problem of intersection of two elements on the web in a performant way has been impossible since always, because it requires DOM polling constantly for the ClientRect the element which we want to check for intersection, as these operations run on main thread these kinds of polling has always been a source of bottlenecks in the application performance. The intersection observer API is a web standard to detect when two DOM elements intersect with each other. The intersection observer API helps us to configure a callback whenever an element called target intersects with another element (root) or viewport. To create an intersection observer is a simple task we just have to create a new instance of the observer. var observer = new IntersectionObserver(callback, options); Here the callback is the function to run whenever some observed element intersect with the element supplied to the observer. This element is configured in the options object passed to the Intersection Observer var options = { root: document.querySelector('#root'), // Defaults to viewport if null rootMargin: '0px', // The margin around root within which the callback is triggered threshold: 1.0 } The target element whose intersection is to be tested with the main element can be setup using…

Continue ReadingLazy loading images in Loklak Search

Updating Page Titles Dynamically in Loklak Search

Page titles are native in the web platform, and are prime ways to identify any page. The page titles have been in the web platform since ages. They tell the browsers, the web scrapers and search engines about the page content in 1-2 words. Since the titles are used for wide variety of things from presentation of the page, history references and most importantly by the search engines to categorise the pages, it becomes very important for any web application to update the title of the page appropriately. In earlier implementation of loklak search the page title was a constant and was not updated regularly and this was not a good from presentation and SEO perspective. Problem of page titles with SPA Since loklak search is a single page application, there are few differences in the page title implementation in comparison to a server served multi page application. In a server served multi page application, the whole application is divided into pages and the server knows what page it is serving thus can easily set the title of the page while rendering the template. A simple example will be a base django template which holds the place to have a title block for the application. <!-- base.html --> <title>{% block title %} Lokalk Search {% endblock %}</title> <!-- Other application blocks --> Now for any other page extending this base.html it is very simple to update the title block by simply replacing it with it’s own title. <!-- home.html --> {% extends ‘base.html’ %} {% block title %} Home Page - Loklak Search {% endblock %} <!-- Other page blocks --> When the above template is rendered by the templating engine it replaces the title block of the base.html with the updated title block specific to the page, thus for each page at the rendering time server is able to update the page title, appropriately. But in a SPA, the server just acts as REST endpoints, and all the templating is done at the client side. Thus in an SPA the page title never changes automatically, from the server, as only the client is in control of what page (route) it is showing. Thus it becomes the duty of the client side to update the title of the page, appropriately, and thus this issue of static non informative page titles is often overlooked. Updating page titles in Loklak Search Before being able to start solving the issue of updating the page titles it is certainly very important to understand what all are the points of change in the application where we need to update the page title. Whenever the route in the application changes. Whenever new query is fetched from the server. These two are the most important places where we definitely want to update the titles. The way we achieved is using the Angular Title Service. The title service is a platform-browser service by angular which abstracts the workflow to achieve the title updation. There are are two main…

Continue ReadingUpdating Page Titles Dynamically in Loklak Search

Search Engine Optimization and Meta Tags in Loklak Search

Ranking higher in search results is very important for any website’s, productivity and reach. These days modern search engines use algorithms and scrape the sites for relevant data points, then these data points are processed to result in a relevance number aka the page ranking. Though these days the algorithms which search engines use are very advanced and are able to generate the context of the page by the page content, but still there are some key points which should be followed by the developers to enable a higher page ranking on search results along with better presentation of search results on the pages. Loklak Search is also a web application thus it is very important to get these crucial points correct. So the first thing which search engines see on a website is their landing or index page, this page gives information to the search engine what the data is about in this site and then search engines follow links to to crawl the details of the page. So the landing page should be able to provide the exact context for the page. The page can provide the context using the meta tags which store the metadata information about the page, which can be used by the search engines and social sites Meta Charset The first and foremost important tag is the charset tag. It specifies the character set being used on the page and declares the page's character encoding which is used by browsers encoding algorithm. It very important to determine a correct and compatible charset for security and presentation reasons. Thus the most widely used charset UTF-8 is used in loklak search. <meta charset="utf-8"> Meta Viewport The mobile browsers often load the page in a representative viewport which is usually larger than the actual screen of the device so that the content of the screen is not crammed into small space. This makes the user to pan-zoom around the page to reach the desired content. But this approach often undesirable design for the most of the mobile websites. For this at loklak search we use <meta name="viewport" content="width=device-width, initial-scale=1">   This specifies the relation between CSS pixel and device pixel. Here the relationship is actually computed by the browser itself, but this meta tag says that the width for calculating that ratio should be equal to device width and the initial-scale should be 1 (or zoom should be 0). Meta Description The meta description tag is the most important tad for the SEO as this is the description which is used by search engines and social media sites while showing the description of the page <meta name="description"    content="Search social media on Loklak Search. Our mission is to make the world’s social media information openly accessible and useful generating open knowledge for all">   This is how this description tag is used by google on the Google Search Social Media meta tags The social media meta tags are important for the presentation of the content of the page…

Continue ReadingSearch Engine Optimization and Meta Tags in Loklak Search