Resource Injection Using ButterKnife in Loklak Wok Android

Loklak Wok Android being a sophisticated Android app uses a lot of views, and of those most are manipulated at runtime. In Android to play with a View or ViewGroup defined in XML at runtime requires developers to add the following line:

(TypeOfView) parentView.findViewById(R.id.id_of_view);

 

This leads to lengthy code. And very often, more than one Views respond to a particular event. For example, hiding Views if a network request fails and showing a message to the user to “Try Again!”. Let’s say you have to hide 4 Views, are you going to do the following:

view1.setVisibility(View.GONE);
view2.setVisibility(View.GONE);
view3.setVisibility(View.GONE);
view4.setVisibility(View.GONE);
textView.setVisibility(View.VISIBLE); // has "Try Again!" message.

// more 5 lines of code when hiding textView and displaying 4 other Views

 

Surely not! And the old fashioned way to get a string value defined as a resource in string.xml

String appName = getActivity().getResources().getString(R.id.app_name);

 

Surely, all this works good, but being a developer while working on a sophisticated app you would like to focus on the logic of the app, rather than scratching your head to debug whether you properly did a findViewById or not, did you typecast it to the proper View, or where did you miss to change the visibility of a view in response to an event.

Well, all of this can be easily handled by using a library which provides you the dependency, here resources. All you need to do is just declare your resources, and that’s it, the library provides the resources to you, yes you don’t need to initialize it using findViewById. So let’s dive in and see how ButterKnife is used in Loklak Wok Android to handle these issues.

Adding ButterKnife to Android Project

In the app/build.gradle:

dependencies {
    compile 'com.jakewharton:butterknife:8.6.0'
    annotationProcessor 'com.jakewharton:butterknife-compiler:8.6.0'
    ...
}

Dealing with Views in Fragments

When views are declared, BindView annotation is used with its parameter as the ID of the view, for example, views in TweetHarvestingFragment :

@BindView(R.id.toolbar)
Toolbar toolbar;
@BindView(R.id.harvested_tweets_count)
TextView harvestedTweetsCountTextView;
@BindView(R.id.harvested_tweets_container)
RecyclerView recyclerView;
@BindView(R.id.network_error)
TextView networkErrorTextView;

NOTE: Views declared can’t be private.

Once Views are declared, then it needs to be injected, it is done using ButterKnife.bind(Object target, View Source). Here in TweetHarvestingFragment the target will be the fragment itself and source i.e. the parent view will be rootView (obtained by inflating the layout file of fragment). All this needs to be done in onCreateView method

View rootView = inflater.inflate(R.layout.fragment_tweet_harvesting, container, false);
ButterKnife.bind(this, rootView);

That’s it, we are done!

The same paradigm can be used to bind views to a ViewHolder of a RecyclerView, as implemented in HarvestTweetViewHolder:

@BindView(R.id.user_fullname)
TextView userFullname;
@BindView(R.id.username)
TextView username;
@BindView(R.id.tweet_date)
TextView tweetDate;
@BindView(R.id.harvested_tweet_text)
TextView harvestedTweetTextView;

public HarvestedTweetViewHolder(View itemView) {
   super(itemView);
   ButterKnife.bind(this, itemView);
}

 

Injecting resources like strings, dimensions, colors, drawables etc. is even easier, only the related annotation and ID needs to be provided. Example the string app_name is used in TweetHarvestingFragment to display the app name i.e. “Loklak Wok” in toolbar

@BindString(R.string.app_name)
String appName;

// directly used inside onCreateView to set the title in toolbar
toolbar.setTitlet(appName);

 

Using ButterKnife onClickListeners can be implemented in separate method, a clean way to define click events rather than polluting onCreate(in Activity) or onCreateView(in Fragment), as implemented in SuggestFragment

@OnClick(R.id.clear_image_button)
public void onClickedClearImageButton(View view) {
   tweetSearchEditText.setText("");
}

 

Multiple Views responding to a single Event

Using @BindViews annotation a list of multiple views can be created, and then a common ButterKnife.Action can be defined to act on the list of views. In TweetHarvestingFragment visibility of some views are changed if a network request fails or succeeds using this feature of ButterKnife:

// declaring List of Views
@BindViews({
       R.id.harvested_tweets_count,
       R.id.harvested__tweet_count_message,
       R.id.harvested_tweets_container})
List<View> networkViews;

// defining action for views
private final ButterKnife.Action<View> VISIBLE = (view, index) -> view.setVisibility(View.VISIBLE);
private final ButterKnife.Action<View> GONE = (view, index) -> view.setVisibility(View.GONE);

// applying action on the views
ButterKnife.apply(networkViews, VISIBLE);
ButterKnife.apply(networkViews, GONE);

Resources

Continue ReadingResource Injection Using ButterKnife in Loklak Wok Android

Implement Caching in the Live Feed of Open Event Android App

In the Open Event Android App, a live feed from the event’s Facebook page was recently implemented. Since it was a live feed, it was decided that it was futile to store it in the Realm database of the app. The data of the live feed didn’t persist anywhere, hence the feed used to be empty when the app ran without the internet connection.

To solve the problem of data persistence, it was decided to store the feed in the cache. Now, there were two paths before us – use retrofit okhttp cache management or use volley. Since retrofit is used to make the API requests in the app, we used the former. To implement caching with retrofit, its API response should include the cache control header. Since it was not a response generated by a personal server, interceptors were needed to force change the request.

Interceptors

Interceptors are a powerful mechanism that can monitor, rewrite, and retry calls. The solution was to use interceptors to rewrite the calls to force use of cache. Two interceptors were added, application interceptor for the request and the network interceptor for the response.

Implementation

Create a cache file to store the response.

private static Cache provideCache() {
   Cache cache = null;
   try {
       cache = new Cache(new File(OpenEventApp.getAppContext().getCacheDir(), "facebook-feed-cache"),
               10 * 1024 * 1024); // 10 MB
   } catch (Exception e) {
       Timber.e(e, "Could not create Cache!");
   }
   return cache;
}

 

Create a network interceptor by chaining the response with the cache control header and removing the pragma header to force use of cache.

private static Interceptor provideCacheInterceptor() {
   return chain -> {
       Response response = chain.proceed(chain.request());

       // re-write response header to force use of cache
       CacheControl cacheControl = new CacheControl.Builder()
               .maxAge(2, TimeUnit.MINUTES)
               .build();

       return response.newBuilder()
               .removeHeader("Pragma")
               .header(CACHE_CONTROL, cacheControl.toString())
               .build();
   };
}

 

Create an application interceptor by chaining the request with the cache control header for stale responses and removing the pragma header to make the feed available for offline usage.

private static Interceptor provideOfflineCacheInterceptor() {
   return chain -> {
       Request request = chain.request();

       if (!NetworkUtils.haveNetworkConnection(OpenEventApp.getAppContext())) {
           CacheControl cacheControl = new CacheControl.Builder()
                   .maxStale(7, TimeUnit.DAYS)
                   .build();

           request = request.newBuilder()
                   .removeHeader("Pragma")
                   .cacheControl(cacheControl)
                   .build();
       }

       return chain.proceed(request);
   };
}

 

Finally add the cache and the two interceptors while building the okhttp client.

OkHttpClient okHttpClient = okHttpClientBuilder.addInterceptor(new HttpLoggingInterceptor()
       .setLevel(HttpLoggingInterceptor.Level.BASIC))
       .addInterceptor(provideOfflineCacheInterceptor())
       .addNetworkInterceptor(provideCacheInterceptor())
       .cache(provideCache())
       .build();

 

Conclusion

Working of apps without the internet connection builds up a strong case for corner cases while testing. It is therefore critical to persist data however small to avoid crashes and bad user experience.

Resources

Continue ReadingImplement Caching in the Live Feed of Open Event Android App

Add Autocomplete SearchView in Open Event Android App

The Open Event Android App has a map for showing all locations of sessions. All the locations have a marker in the map. It is difficult to find a particular location on the map because to know the name of location user has to click on the marker. Adding autocomplete SearchView will improve user experience by providing an ability to search the location by name and by suggesting name according to the search query. In this post I explain how to add autocomplete SearchView in the fragment or activity.

Add search icon in actionbar

The first step to do is to create a menu xml file and add a search menu item in it. Then inflate this menu xml file in Fragment in onCreateOptionsMenu() method.

1. Create menu.xml file

In this file add search menu element. Inside menu element add search menu item. Define id, title, and icon of search menu item. Add android.support.v7.widget.SearchView” as actionViewClass which will be used as action view when the user clicks on the icon.

<?xml version="1.0" encoding="utf-8"?>
<menu xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools">
    
   <item
        android:id="@+id/action_search"
        android:icon="@drawable/ic_search_white_24dp"
        android:title="@string/search"
        app:actionViewClass="android.support.v7.widget.SearchView"
        app:showAsAction="ifRoom | collapseActionView"/>
</menu>

2. Inflate menu.xml file in Fragment

In the fragment’s onCreateOptionsMenu() method inflate menu.xml file using MenuInflater’s inflate() method. Then find search menu item using menu’s findItem() method by passing id of search menu item as parameter.

public void onCreateOptionsMenu(Menu menu, MenuInflater inflater) {
        super.onCreateOptionsMenu(menu, inflater);
        inflater.inflate(R.menu.menu_map, menu);
        MenuItem item = menu.findItem(R.id.action_search);
}

Add and initialize SearchView  

Now after adding search icon we need to add SearchView and SearchAutoComplete fields in the fragment.

private SearchView searchView;
private SearchView.SearchAutoComplete   mSearchAutoComplete;

Initialize SearchView in onCreateOptionMenu() method by passing search menu item in the getActionView() method of MenuItemCompat.

Here SearchAutoComplete is a child object of SearchView so initialize it using findViewById method of SearchView by passing the id as parameter.

searchView = (SearchView) MenuItemCompat.getActionView(item);
mSearchAutoComplete = (SearchView.SearchAutoComplete) searchView.findViewById(android.support.v7.appcompat.R.id.search_src_text);

Define properties of SearchAutoCompleteView

By default background of drop down menu in SearchAutoComplete is black. You can change background using setDropDownBackgroundResource() method. Here i’m making it white by providing white drawable resource.

mSearchAutoComplete.setDropDownBackgroundResource(R.drawable.background_white);
mSearchAutoComplete.setDropDownAnchor(R.id.action_search);
mSearchAutoComplete.setThreshold(0)

The setDropDownAnchor() method sets the view to which the auto-complete drop down list should anchor. The setThreshold() method specifies the minimum number of characters the user has to type in the edit box before the drop down list is shown.

Create array adapter

Now it’s time to make the ArrayAdapter object which will provide the data set (strings) which will be used to run search queries.

ArrayAdapter<String> adapter = new ArrayAdapter<>(getActivity(), android.R.layout.simple_list_item_1, searchItems);

Here searchItems is List of strings. Now set this adapter to the mSearchAutoComplete object using setAdapter() method.

mSearchAutoComplete.setAdapter(adapter);

Now we are all set to run the app on device or emulator. Here’s demo how it will look

Conclusion

The SearchView with an ability to give suggestions serves the great user experience in the application.

Additional resources:

Continue ReadingAdd Autocomplete SearchView in Open Event Android App

Communicate between Child and Parent Components in React JS of SUSI Web Chat

When we were developing SUSI AI web chat  some components became huge. So the team wanted to break some components into parts. Since the Login dialog-box is used in several  places we decided to make a separate component for Login Dialog-box. In this post I am discussing how we implemented the feature as a separate component and how we have changed the state of the parent component of the child component.

Login Dialog-box contains all the things of the login dialog-box component.

Child-component (Login Dialog-box component) is here:

This method executes the ‘switchDialog’ function of the parent component.

export default class LoginDialog extends React.Component {

   handleClose = () => {
      this.props.switchDialog(false);
   };

   render() {
       this.state = { open: this.props.open }
       const actions = <RaisedButton
           label="Cancel"
           backgroundColor={
               UserPreferencesStore.getTheme() === 'light' ? '#607D8B' : '#19314B'}
           labelColor="#fff"
           width='200px'
           keyboardFocused={true}
           onTouchTap={this.handleClose}
       />;

       return (
           <Dialog
               actions={actions}
               modal={false}
               open={this.props.open}
               autoScrollBodyContent={true}
               bodyStyle={bodyStyle}
               contentStyle={{ width: '35%', minWidth: '300px' }}
               onRequestClose={this.handleClose}>
               <Login {...this.props} />
           </Dialog>
       );
   }

};

In this part we validate property types that has passed from the parent component.

LoginDialog.propTypes = {
   open: PropTypes.bool,
   switchDialog: PropTypes.func
};

In render() method I have returned the element.
To open and close dialog we have to communicate with parent component. We can send an instruction as an attribute of the element and we can refer it inside the element as “props”. This is how I have sent an instruction to the child element.
Parent-component:
‘handleOpen’ function opens the dialog when user hit on the login button.

   handleOpen = () => {
       this.setState({ open: true });
   };

‘switchDialog’ function is using for change the state of parent component from child Component (Login Dialog-box component).

   switchDialog=(dialogState)=>{
       this.setState({open:dialogState});
   };

   render() {

       const styles = {
           'margin': '60px auto',
           'width': '100%',
           'padding': '20px',
           'textAlign': 'center'
       }

       return (
           <div className="signUpForm">
               <Paper zDepth={1} style={styles}>
                   <h1>Sign Up with SUSI</h1>
                   <form onSubmit={this.handleSubmit}>
                       <div>
                           <h4>If you have an Account Please Login</h4>
                           <RaisedButton
                               onTouchTap={this.handleOpen}
                               label='Login'
                               backgroundColor={
                                   UserPreferencesStore.getTheme()==='light'
                                   ? '#607D8B' : '#19314B'}
                               labelColor="#fff" />
                       </div>
                   </form>
               </Paper>

               <LoginDialog {...this.props} open={this.state.open} switchDialog={this.switchDialog} />
           </div>
       );
   };

To open and close the dialog-box we have to send the state of the parent component to child component. To close the dialog-box we have to update the parent component’s state from child component.

To change the parent component’s state we have used this in element.

switchDialog={this.switchDialog}

To send the state to the child component we used this.

open={this.state.open}

To send other properties to the element we used this.

{...this.props}

After closing the dialog-box it calls this method and it updates the state of the parent component.

handleClose = () => {
      this.props.switchDialog(false);
};

This is how we can communicate between child and parent components using react.

Resources:

Component Communication: http://andrewhfarmer.com/component-communication/
Material UI Dialogs: http://www.material-ui.com/#/components/dialog

Continue ReadingCommunicate between Child and Parent Components in React JS of SUSI Web Chat

Using CoreLocation in SUSI iOS

The SUSI Server responds with intelligent answers to the user’s queries. To make these answers better, the server makes use of the user’s location which is sent as a parameter to the query request each time. To implement this feature in the SUSI iOS client, we use the CoreLocation framework provided by Apple which helps us to get the user’s location coordinates and add them as a parameter to each request made.

In order to start with using the CoreLocation framework, we first import it inside the view controller.

import CoreLocation

Now, we create a variable of type CLLocationManager which will help us to use the actual functionality.

// Location Manager
var locationManager = CLLocationManager()

The location manager has some delegate methods which give an option to get the maximum accuracy for a user’s location.  To set that, we need the controller to conform to the CLLocationManagerDelegate, so we create an extension of the view controller conforming to this.

extension MainViewController: CLLocationManagerDelegate {

   // use functionality

}

Next, we set the manager delegate.

locationManager.delegate = self

And create a method to ask for using the user’s location and set the delegate properties.

func configureLocationManager() {
       locationManager.delegate = self
       if CLLocationManager.authorizationStatus() == .notDetermined || CLLocationManager.authorizationStatus() == .denied {
           self.locationManager.requestWhenInUseAuthorization()
       }

       locationManager.distanceFilter = kCLDistanceFilterNone
       locationManager.desiredAccuracy = kCLLocationAccuracyBest
}

Here, we ask for the user location if it was previously denied or is not yet determined and following that, we set the `distanceFilter` as kCLDistanceFilterNone  and `desiredAccuray` as kCLLocationAccuracyBest.. Finally, we are left with starting to update the location which we do by:

locationManager.startUpdatingLocation()

We call this method inside viewDidLoad to start updation of the location when the view first loads. The complete extension looks like below:

extension MainViewController: CLLocationManagerDelegate {

   // Configures Location Manager
   func configureLocationManager() {
       locationManager.delegate = self
       if CLLocationManager.authorizationStatus() == .notDetermined || CLLocationManager.authorizationStatus() == .denied {
           self.locationManager.requestWhenInUseAuthorization()
       }

       locationManager.distanceFilter = kCLDistanceFilterNone
       locationManager.desiredAccuracy = kCLLocationAccuracyBest
       locationManager.startUpdatingLocation()
   }

}

Now, it’s very easy to use the location manager and get the coordinates and add it to the params for each request.

if let location = locationManager.location {
   params[Client.ChatKeys.Latitude] = location.coordinate.latitude as AnyObject
   params[Client.ChatKeys.Longitude] = location.coordinate.longitude as AnyObject
}

Now the params which is a dictionary object is added to each request made so that the user get’s the most accurate results for each query he makes.

References:

Continue ReadingUsing CoreLocation in SUSI iOS

Simplifying Scrapers using BaseScraper

Loklak Server‘s main function is to scrape data from websites and other sources and output in different formats like JSON, xml and rss. There are many scrapers in the project that scrape data and output them, but are implemented with different design and libraries which makes them different from each other and a difficult to fix changes.

Due to variation in scrapers’ design, it is difficult to modify them and fix the same issue (any issue, if it appears) in each of them. This issue signals fault in design. To solve this problem, Inheritance can be brought into application. Thus, I created BaseScraper abstract class so that scrapers are more concentrated on fetching data from HTML and all supportive tasks like creating connection with the help of url are defined in BaseScraper.

The concept is pretty easy to implement, but for a perfect implementation, there is a need to go through the complete list of tasks a scraper does.

These are the following tasks with descriptions and how they are implemented using BaseScraper:

  1. Endpoint that triggers the scraper

Every search scraper inherits class AbstractAPIHandler. This is used to fetch get parameters from the endpoint according to which data is scraped from the scraper. The arguments from serviceImpl method is used to generate output and is returned to it as JSONObject.

For this task, the method serviceImpl has been defined in BaseScraper and method getData is implemented to return the output. This method is the driver method of the scraper.

public JSONObject serviceImpl(Query call, HttpServletResponse response, Authorization rights, JSONObjectWithDefault permissions) throws APIException {
    this.setExtra(call);
    return this.getData().toJSON(false, "metadata", "posts");
}

 

  1. Constructor

The constructor of Scraper defines the base URL of the website to be scraped, name of the scraper and data structure to fetch all get parameters input to the scraper. For get parameters, the Map data structure is used to fetch them from Query object.

Since every scraper has it’s own different base URL, scraper name and get parameters used, so it is implemented in respective Scrapers. QuoraProfileScraper is an example which has these variables defined.

  1. Get all input variables

To get all input variables, there are setters and getters defined for fetching them as Map from Query object in BaseScraper. There is also an abstract method getParam(). It is defined in respective scrapers to fetch the useful parameters for scraper and set them to the scraper’s class variables.

// Setter for get parameters from call object
protected void setExtra(Query call) {
    this.extra = call.getMap();
    this.query = call.get("query", "");
    this.setParam();
}

// Getter for get parameter wrt to its key
public String getExtraValue(String key) {
    String value = "";
    if(this.extra.get(key) != null) {
        value = this.extra.get(key).trim();
    }
    return value;
}

// Defination in QuoraProfileScraper
protected void setParam() {
    if(!"".equals(this.getExtraValue("type"))) {
        this.typeList = Arrays.asList(this.getExtraValue("type").trim().split("\\s*,\\s*"));
    } else {
        this.typeList = new ArrayList<String>();
        this.typeList.add("all");
        this.setExtraValue("type", String.join(",", this.typeList));
    }
}

 

  1.  URL creation for web scraper

The URL creation shall be implemented in a separate method as in TwitterScraper. The following is the rough implementation adapted from one of my pull request:

protected String prepareSearchUrl(String type) {
    URIBuilder url = null;
    String midUrl = "search/";

    try {
        switch(type) {
            case "question":
                url = new URIBuilder(this.baseUrl + midUrl);
                url.addParameter("q", this.query);
                url.addParameter("type", "question");
        .
        .
    }
    .
    .
    return url.toString();
}

 

  1. Get BufferedReader object from InputStream

getDataFromConnection method fetches the BufferedReader object from ClientConnection. This object reads the web page line by line by the scrape method to fetch data. See here.

ClientConnection connection = new ClientConnection(url);
BufferedReader br = getHtml(connection);
.
.
.
public BufferedReader getHtml(ClientConnection connection) {

    if (connection.inputStream == null) {
        return null;
    }

    BufferedReader br = new BufferedReader(new InputStreamReader(connection.inputStream, StandardCharsets.UTF_8));
    return br;
}

 

  1. Scraping of data from HTML

The Scraper method for scraping data is declared abstract in BaseScraper and defined in the scraper. This can be a perfect example of implementation for BaseScraper (See code the here) and scraper (here).

  1. Output of data

The output of scrape method is fetched in Post data objects that are implemented for the respective scraper. These Post objects are added to Timeline iterator and which outputs data as JSONArray. Later the objects are output in enclosed Post object wrapper.

This data can be directly output as Post object, but adding it to iterator makes the Post Objects capable to be sorted in an order and be indexed to ElasticSearch.

 

Resources

Continue ReadingSimplifying Scrapers using BaseScraper
Read more about the article Iterating the Loklak Server data
Iterating the Loklak Server data

Iterating the Loklak Server data

Loklak Server is amazing for what it does, but it is more impressive how it does the tasks. Iterators are used for and how to use them, but this project has a customized iterator that iterates Twitter data objects. This iterator is Timeline.java .

Timeline implements an interface iterable (isn’t it iterator?). This interface helps in using Timeline as an iterator and add methods to modify, use or create the data objects. At present, it only iterates Twitter data objects. I am working on it to modify it to iterate data objects from all web scrapers.

The following is a simple example of how an iterator is used.

// Initializing arraylist
List<String> stringsList = Arrays.asList("foo", "bar", "baz");

// Using iterator to display contents of stringsList
System.out.print("Contents of stringsList: ");

Iterator iter = al.iterator();
while(iter.hasNext()) {
    System.out.print(iter.next() + " ");
}

 

This iterator can only iterate data the way array does. (Then why do we need it?) It does the task of iterating objects perfectly, but we can add more functionality to the iterator.

 

Timeline iterator iterates the MessageEntry objects i.e. superclass of TwitterTweet objects. According to Javadocs, “Timeline is a structure which holds tweet for the purpose of presentation, There is no tweet retrieval method here, just an iterator which returns the tweets in reverse appearing order.”

Following are some of the tasks it does:

  1. As an iterator:

This basic use of Timeline is to iterate the MessageEntry objects. It not only iterates the data objects, but also fetches them (See here).

// Declare Timeline object according to order the data object has been created
Timeline tline = new Timeline(Timeline.parseOrder("created_at"));

// Adding data objects to the timeline
tline.add(me1);
tline.add(me2);
.
.
.
// Outputing all data objects as array of JSON objects
for (MessageEntry me: tline) {
    JSONArray postArray = new JSONArray();
    for (MessageEntry post : this) {
        postArray.put(post.toJSON());
    }
}

 

  1. The order of iterating the data objects

Timeline can arrange and iterate the data objects according to the date of creation of the twitter post, number of retweets or number of favourite counts. For this there is an Enum declaration of Order in the Timeline class which is initialized during creation of Timeline object. [link]

    Timeline tline = new Timeline(Timeline.parseOrder("created_at"));

 

  1. Pagination of data objects

There is an object cursor, some methods, including getter and setters to support pagination of the data objects. It is only internally implemented, but can also be used to return a section of the result.

  1. writeToIndex method

This method can be used to write all data fetched by Timeline iterator to ElasticSearch for indexing and to dump that can be used for testing. Thus, indexing of data can concurrently be done while it is iterated. It is implemented here.

  1. Other methods

It also has methods to output all data as JSON and customized method to add data to Timeline keeping user object and Data separate, etc. There are a bit more things in this iterable class which shall be explored instead.

 

Resources:

Continue ReadingIterating the Loklak Server data

Multithreading implementation in Loklak Server

Loklak Server is a near-realtime system. It performs a large number of tasks and are very costly in terms of resources. Its basic function is to scrape all data from websites and output it at the endpoint. In addition to scraping data, there is also a need to perform other tasks like refining and cleaning of data. That is why, multiple threads are instantiated. They perform other tasks like:

  1. Refining of data and extract more data

The data fetched needs to be cleaned and refined before outputting it. Some of the examples are:

a) Removal of html tags from tweet text:

After extracting text from html data and feeding to TwitterTweet object, it concurrently runs threads to remove all html from text.

b) Unshortening of url links:

The url links embedded in the tweet text may track the users with the help of shortened urls. To prevent this issue, a thread is instantiated to unshorten the url links concurrently while cleaning of tweet text.

  1. Indexing all JSON output data to ElasticSearch

While extracting JSON data as output, there is a method here in Timeline.java that indexes data to ElasticSearch.

Managing multithreading

To manage multithreading, Loklak Server applies following objects:

1. ExecutorService

To deal with large numbers of threads ExecutorService object is used to handle threads as it helps JVM to prevent any resource overflow. Thread’s lifecycle can be controlled and its creation cost can be optimized. This is the best example of ExecutorService application is here:

.
.
public class TwitterScraper {
    // Creation of at max 40 threads. This sets max number of threads to 40 at a time
    public static final ExecutorService executor = Executors.newFixedThreadPool(40);
    .
    .
    .
    .
    // Feeding of TwitterTweet object with data
    TwitterTweet tweet = new TwitterTweet(
        user.getScreenName(),
        Long.parseLong(tweettimems.value),
        props.get("tweettimename").value,
        props.get("tweetstatusurl").value,
        props.get("tweettext").value,
        Long.parseLong(tweetretweetcount.value),
        Long.parseLong(tweetfavouritecount.value),
        imgs, vids, place_name, place_id,
        user, writeToIndex,  writeToBackend
    );
    // Starting thread to refine TwitterTweet data
    if (tweet.willBeTimeConsuming()) {
       executor.execute(tweet);
    }    .
    .
    .

 

2. basic Thread class

Thread class can also be used instead of ExecutorService in cases where there is no resource crunch. But it is always suggested to use ExecutorService due to its benefits. Thread implementation can be used as an anonymous class like here.

3. Runnable interface

Runnable interface can be used to create an anonymous class or classes which does more task than just a task concurrently. In Loklak Server, TwitterScraper concurrently indexes the data to ElasticSearch, unshortens link and cleans data. Have a look at implementation here.

Resources:

Continue ReadingMultithreading implementation in Loklak Server

Create an AutocompleteTextView dropdown for the email input in the Open Event Orga Android App

In the first version of the Open Event Organizer App, the event organizer was required to enter his full email each time he logged out of his account and therefore it was hindering the user experience. AutoCompleteTextView with shared preferences is a solution to this problem. This feature provides an editable text view that shows completion suggestions automatically while the user is typing. The list of suggestions is displayed in a drop down menu. The user can choose an item to replace the content of the edit box with. It is extremely useful in enhancing user experience.

The solution we implemented was to create an autocomplete textview for the email input, store the email address of the user on a successful login in the shared preference in a set of strings to prevent duplicacy and display it in the dropdown on subsequent login attempts.

Implementation

Change your TextInputLayout structure to accommodate the autocompletetextview. Remember to create a separate autocompletetextview object with the specific id of the view.

<android.support.v7.widget.AppCompatAutoCompleteTextView
       android:id="@+id/email_dropdown"
       android:layout_width="match_parent"
       android:layout_height="wrap_content"
       android:hint="@string/email"
       android:inputType="textEmailAddress" />

 

Create Utility methods to get/store the emails in the shared preferences. The set data structure has been used here so that there is no duplicacy while storing the emails in the shared preferences.

public Set<String> getStringSet(String key, Set<String> defaultValue) {
   return sharedPreferences.getStringSet(key, defaultValue);
}

public void saveStringSet(String key, Set<String> value) {
   SharedPreferences.Editor editor = sharedPreferences.edit();
   editor.putStringSet(key, value);
   editor.apply();
}

public void addStringSetElement(String key, String value) {
   Set<String> set = getStringSet(key, new HashSet<>());
   set.add(value);
   saveStringSet(key, set);
}

 

Create helper methods to add an email and retrieve the list of emails from the shared preferences to provide it to the views.

private void saveEmail(String email) {
   utilModel.addStringSetElement(Constants.SHARED_PREFS_SAVED_EMAIL, email);
}

private Set<String> getEmailList() {
   return utilModel.getStringSet(Constants.SHARED_PREFS_SAVED_EMAIL, null);
}

 

Create an autocompleteTextView object in your activity with the help of the R id from the layout and set the adapter with the set of strings retrieved from the shared preferences. You could create a custom adapter for this case too, but as far as the Open Event Orga App was concerned, using the array adapter made sense.

autoCompleteEmail.setAdapter(new ArrayAdapter<>(this, android.R.layout.simple_list_item_1,
   new ArrayList<String>(emails)));

 

Conclusion

It is important that the user is served with the best possible experience of the application and the autocomplete text view for the email serves just that.

Resources

Continue ReadingCreate an AutocompleteTextView dropdown for the email input in the Open Event Orga Android App

Unifying Data from Different Scrapers of loklak server using Post

Loklak Server project is a software that scrapes data from different websites through different endpoints. It is difficult to create a single endpoint. For a single endpoint, there is a need of a decent design for using multiple scrapers. For such a task, multiple changes are needed. That is why one of the changes I introduced was Post class that acts as both wrapper and an interface for data objects of search scrapers (though implementation in scrapers is in progress).

Post is a subclass of JSONObject that helps in working with JSON data in Java. In other words, Post is a JSONObject with an identity (we call it postId) and and a timestamp of the data scraped. It is used to capture data fetched by the web-scrapers. Benefit of JSONObject as superclass is that it provides methods to capture and access data efficiently.

Why Post?

At present there is a Class MessageEntry which is the superclass of TwitterTweet (data object of TwitterScraper). It has numerous methods that can be used by data objects to clean and analyse data. But it has a disadvantage, it is a specialized for social websites like Twitter, but will become redundant for different types websites like Quora, Github, etc.

Whereas Post object is a small but powerful and flexible object with its ability to deal with data like JSONObject. It contains getter and setter methods, identity members used to provide each Post object a unique identity. It doesn’t have any methods for analysis and cleaning of data, but MessageEntry class’ methods can be used for this purpose.

Uses of Post Object

When I started working on Post Object, it could be used as marker interface for data objects. Following are the advantages I came up with it:

1) Accessing the data object of any scraper using its variable. And yes, this is the primary reason it is an interface.

2) But in addition to accessing the data objects, one can also directly use it to fetch, modify or use data without knowing the scraper it belongs. This feature is useful in Timeline iterator.

This is an example how Post interface is used to append two lists of Posts (maybe carrying different type of data) into one.

public void mergePost(PostTimeline list) {
    for (Post post: list) {
        this.add(post);
    }
}

 

Post as a wrapper object

While working on Post object, I converted it into a class to also use it as a wrapper. But why a wrapper? Wrapper can be used to wrap a list of Post objects into one object. It doesn’t have any identity or timestamp. It is just a utility to dump a pack of data objects with homogeneous attributes.

This is an example implementation of Post object as wrapper. typeArray is a wrapper which is used to store 2 arrays of data objects in it. These data object arrays are timeline objects that are saved as JSONArray objects in the Post wrapper.

    Post typeArray = new Post(true);
    switch(type) {
        case "users":
            typeArray.put("users", scrapeProfile(br, url).toArray());
            break;
        case "question":
            typeArray.put("question", scrapeQues(br, url).toArray());
            break;
        default:
            break;
    }

 

Resources:

 

Continue ReadingUnifying Data from Different Scrapers of loklak server using Post