Author: Deepjyoti Mondal

The Loklak applications site now has a functional store listing page where developers can showcase their apps and users and other developers can get all sorts of information about an app. However, for an app to be showcased properly on store listing page, the app must contain a properly configured app.json file and some necessary assets. But while creating an app a developer might miss out some vital information from app.json or forget to provide some of the required assets, which he will later come to know from his co-developers or reviewers.

Now this will cause inconvenience, both for the developer and reviewer. So to overcome this apps.loklak.org has now got a new script to review a given app. It checks whether the necessary fields are present in app.json or not. If present then it checks whether the fields are empty or not. If any invalid or missing information problem is encountered then it is reported to the developer along with information on what should be the actual case. If the app passes all the checks then the developer is informed that his app is ready to be published.

In order to use this script all the developer needs to do is open his app directory in terminal and execute the following command

../bin/review.sh

This will present the developer with all the necessary informations.

How the script works?

Now let us delve into the working of the script. The initial call to the shell script review.sh calls a python script review.py. This python script performs all the checks on the app and displays necessary informations.

The first thing the script does is, it checks whether there is an app.json present in the app directory or not, if yes then the process continues otherwise it ends immediately with an error.

if "app.json" not in dir_contents:
    print_info("Please include a well configured app.json")
    print_problem("review failed")
    exit(0)

Next it performs one of the most important checks. It verifies whether the name of the app and app directory name is same or not. If it is same then there is no problem else it shows the corresponding error message.

app_dir_name = os.getcwd().split("/")[-1]
    if app_dir_name != app_json.get("name"):
      print_problem("app directory name and name mentioned in app.json are different")
      print_info("app directory name and name mentioned in app.json must be same")
      problems_no += 1

Next the script checks whether there is an index.html present or not. Index.html is a must as it serves as the entry point of the app.

if “index.html” not in dir_contents:
print_problem(“index.html is missing”)
print_info(“app must contain index.html”)
problems_no += 1

After this the script checks whether a number of key fields are present or not. These field includes applicationCategory, oneLineDescription, author. If they are present, it checks whether the fields are empty or not. If the fields are either absent or empty, error is shown to the developer.

if app_json.get("applicationCategory") == None:
    print_problem("key applicationCategory missing in app.json")
    print_info("app.json must contain key applicationCategory with category as value")
    problems_no += 1
  else:
    if app_json.get("applicationCategory") == "":
      print_problem("applicationCategory cannot be an empty string")
      problems_no += 1
  
  if app_json.get("oneLineDescription") == None:
    print_problem("key oneLineDescription missing in app.json")
    print_info("app.json must contain key oneLineDescription with a one line description of the app")
    problems_no += 1
  else:
    if app_json.get("oneLineDescription") == "":
      print_problem("oneLineDescription cannot be an empty string")
      problems_no += 1
  
  if app_json.get("author") == None:
    print_problem("author object missing in app.json")
    print_info("app.json must contain author object containing information about author")
    problems_no += 1
  else:
    author = app_json.get("author")
    if author.get("name") == None:
      print_problem("name of author is mssing in author object")
      print_info("name of author must be mentioned in author object")
      problems_no += 1
    else:
      if author.get("name") == "":
        print_problem("author name cannot be an empty string")
        problems_no += 1

After the above checks are performed, it checks whether the fields corresponding to the image assets are present or not and whether the assets being referred in app.json exist in the directory. If not, then the developer is informed about the same.

if app_json.get("promoImage") == None:
    print_problem("key promoImage missing in app.json")
    print_info("app.json must contain key promoImage with path of promo image as value")
    problems_no += 1
  else:
    if os.path.isfile(app_json.get("promoImage")) == False:
      if app_json.get("promoImage") == "":
        print_problem("promoImage cannot be an empty string")
      else:
        print_problem(app_json.get("promoImage") + " does not exists")
      problems_no += 1
  
  if app_json.get("appImages") == None:
    print_problem("key appImages missing in app.json")
    print_info("app.json must contain key appImages with paths of preview images as value")
    problems_no += 1
  else:
    for image in app_json.get("appImages"):
      if os.path.isfile(image) == False:
        if image == "":
          print_problem("appImages cannot contain empty strings")
        else:
          print_problem(image +" does not exists")
        problems_no += 1

Finally after all the checks are performed, the developer is informed whether his or her app is ready to be published or not. The number of errors are also presented along with all other details.

if problems_no > 0:
    print_problem("Number of problems detected : " + str(problems_no))
    print_problem("App cannot be published")
    print_info("Please check https://github.com/fossasia/apps.loklak.org/blob/master/docs/tutorial.md")
  else:
    print_success("App is ready for publishing")
    print_info("Check https://github.com/fossasia/apps.loklak.org/blob/master/docs/tutorial.md to publish your app")

Here is an sample output for a faulty app

Here is an sample output for app which is ready to go.

The script is presently functional and can be used to review your apps before sending it for being published on apps.loklak.org. However there are a number of things which are presently not checked like version, getStarted.md, appUse.md, etc. These checks will be included soon.

There are various apps presently hosted on apps.loklak.org, which shows several interesting and awesome stuff that can be done using the loklak API. Now the question is how to create a loklak app? Well previously there were two ways to create a loklak app, you could either create an app from scratch creating all the necessary files including index.html (app entry point), app.json (required for maintaining app metadata and store listing), getStarted.md, appUse.md and other necessary files, or you could use the boilerplate app which provides you with a ready to go app template which you have to edit to create your app.

Recently there has been a new addition to the repository, a command line tool called loklakinit, built with python which will automate the process of initiating a loklak app on the go. The tool creates the entire loklak app directory structure including app.json with all the necessary details (as provided by the user) and defaults, index.html, a css and js subdirectory with a style.css and script.js in the respective folders, getStarted.md file, appUse.md file and others.md file with proper references in app.json.

All that the developer needs to do is, execute the following from the root of the apps.loklak.org repository.

bin/loklakinit.sh

This will start the process and initiate the app.

Creating loklakinit tool

Now let us delve into the code. So what actually the script does? How it works? Well the tool acts very much like the popular npm init command. At first the script creates a dictionary which stores the defaults for app.json. Next the script takes a number of inputs from the user and predicts some default values, if the user refuses to provide any input for a particular parameter then the default value for that parameter is used.

app_json = collections.OrderedDict()

app_json["@context"] = "http://schema.org"
app_json["@type"] = "SoftwareApplication"
app_json["permissions"] = "/api/search.json"
app_json["name"] = "myloklakapp"
app_json["headline"] = "My first loklak app"
app_json["alternativeHeadline"] = app_json["headline"]
app_json["applicationCategory"] = "Misc"
app_json["applicationSubCategory"] = ""
app_json["operatingSystem"] = "http://loklak.org"
app_json["promoImage"] = "promo.png"
app_json["appImages"] = ""
app_json["oneLineDescription"] = ""
app_json["getStarted"] = "getStarted.md"
app_json["appUse"] = "appUse.md"
app_json["others"] = "others.md"

author = collections.OrderedDict()
author["@type"] = "Person"
author["name"] = ""
author["email"] = ""
author["url"] = ""
author["sameAs"] = "" 

app_json["author"] = author

The first part of the script inserts some default values into the app_json dictionary. Now what is an OrderedDict? Well, it is nothing but a python dictionary in which the order in which the keys are inserted is maintained. A ordinary python dictionary does not maintain the order of the keys.

while True :

  app_name = raw_input("name (" + app_json["name"] + ") : ")
  if app_name :
    app_json["name"] = app_name

  app_context = raw_input("@context (" + app_json["@context"] + ") : ")
  if app_context :
    app_json["@context"] = app_context

  app_type = raw_input("@type (" + app_json["@type"] + ") : ")
  if app_type :
    app_json["@type"] = app_type

  app_permissions = raw_input("permissions (" + app_json["permissions"] + ") : ")
  if app_permissions :
    app_json["permissions"] = app_permissions.split(",")

  app_headline = raw_input("headline (" + app_json["headline"] + ") : ")
  if app_headline :
    app_json["headline"] = app_headline
    app_json["alternativeHeadline"] = app_headline


  app_alternative_headline = raw_input("alternative headline (" + app_json["alternativeHeadline"] + ") : ")
  if app_alternative_headline :
    app_json["alternativeHeadline"] = app_alternative_headline

  app_category = raw_input("application category (" + app_json["applicationCategory"] + ") : ")
  if app_category :
    app_json["applicationCategory"] = app_category

  app_sub_category = raw_input("sub category : ")
  if app_sub_category :
    app_json["applicationSubCategory"] = app_sub_category

  app_os = raw_input("application operating system (" + app_json["operatingSystem"] + ") : ")
  if app_os :
    app_json["operatingSystem"] = app_os

  app_promo_image = raw_input("promo image (" + app_json["promoImage"] + ") : ")
  if app_promo_image :
    app_json["promoImage"] = app_promo_image

  app_images = raw_input("app preview images (values can be separted by comma) : ")
  if app_images :
    app_json["appImages"] = app_images.split(",")

  app_description = raw_input("one line description : ")
  if app_description :
    app_json["oneLineDescription"] = app_description

  author["name"] = raw_input("author name : ")

  author_type = raw_input("author type (" + author["@type"] + ") : ")
  if author_type :
    author["@type"] = author_type

  author["email"] = raw_input("email : ")

  author["url"] = raw_input("url : ")
  author["sameAs"] = author["url"]

  author["sameAs"] = raw_input("same as : ")

  app_json["author"] = author

  print json.dumps(app_json, indent=2)

  confirm = raw_input("confirm (yes) : ")

  if confirm.lower() != "no" :
break

The next part of the script asks for various inputs from the developer which he or she can either provide or skip, if an input skipped then the default value is used. However for some parameters like author name, url etc, no default value is used, the developer must provide those values later by editing the app.json file. Once all the inputs are taken, the script outputs the json object created, if the output is alright then the developer may press enter and the process will continue, otherwise the developer may enter ‘no’ and once again the script will start taking inputs.

Once the final app_json is created, next part is creating the actual directory structure with the default files. This is done as shown below.

os.mkdir(app_json["name"])
os.chdir(app_json["name"])
os.mkdir("css")
os.mkdir("js")
index_html = open("index.html", 'w')
index_html.write("<!-- entry point of the app, on launching the appliccation" +
                  "this page will be displayed -->")
index_html.close()

app_json_file = open("app.json", 'w')
app_json_file.write(json.dumps(app_json, indent=2))
app_json_file.close()

get_started_md = open("getStarted.md", 'w')
get_started_md.write("<!-- getting started with the app -->")
get_started_md.close()

app_use_md = open("appUse.md", 'w')
app_use_md.write("<!-- use of this app -->")
app_use_md.close()

others_md = open("others.md", 'w')
others_md.write("<!-- other relevant information about this app -->")
others_md.close()

style_css = open("css/style.css", 'w')
style_css.write("/* css to style the app */")
style_css.close()

script_js = open("js/script.js", 'w')
script_js.write("// javascript logic for the app")
script_js.close()

print "done"

A new directory is created in the root of the repository, the name of the directory is same as the app name. Python’s os module is used to create the directory and subdirectories. Inside the directory, app.json file is created in which app_json dictionary is dumped as a json object.
After this the other default files are created. These include index.html, js/script.js, css/style.css
getStarted.md, appUse.md and others.md. After everything is done script exits and the developer is all set to start with his development.

Finally loklakinit.sh is used to call loklakinit.py

#!/usr/bin/env bash

python bin/loklakinit.py

This allows users to initiate the script by just executing bin/loklakinit.sh from root of the repository. The entire source code for this script can be found here (python code) and here (bash script)

The Github profile scraper is one of the several scrapers present in the loklak project, some of the other scrapers being Quora profile scraper, WordPress profile scraper, Instagram profile scraper, etc.Github profile scraper scrapes the profile of a given github user and provides the data in json format. The scraped data contains user_id, followers, users following the given user, starred repositories of an user, basic information like user’s name, bio and much more. It uses the popular Java Jsoup library for scraping.

The entire source code for the scraper can be found here.

Changes made

The scraper previously provided very limited information like, name of the user, description, starred repository url, followers url, following url, user_id, etc. One of the major problem was the api accepted only the profile name as a parameter and it returned the entire data, that is, the scraper scraped all the data even if the user did not ask for it. Moreover the service provided certain urls as data for example starred repository url, followers url, following url instead of providing the actual data present at those urls.

The scraper contained only one big function where all the scraping was performed. The code was not modular.

The scraper has been enhanced in the ways mentioned below.

• The entire code has been refactored.Code scraping user specific data has been separated from code scraping organization specific data. Also separate method has been created for accessing the Github API.
• Apart from profile parameter, the api now accepts another parameter called terms. It is a list of fields the user wants information on. In this way the scraper scrapes only that much data which is required by the user. This allows better response time and prevents unnecessary scraping.
• The scraper now provides more information like user gists, subscriptions, events, received_events and repositories.

Code refactoring

The code has been refactored to smaller methods. ‘getDataFromApi’ method has been designed to access Github API. All the other methods which want to make request to api.github.com/ now calls ‘getDatFromApi’ method with required parameter. The method is shown below.

private static JSONArray getDataFromApi(String url) {
       URI uri = null;
       try {
           uri = new URI(url);
       } catch (URISyntaxException e1) {
           e1.printStackTrace();
       }

       JSONTokener tokener = null;
       try {
           tokener = new JSONTokener(uri.toURL().openStream());
       } catch (Exception e1) {
           e1.printStackTrace();
       }
       JSONArray arr = new JSONArray(tokener);
       return arr;
   }

For example if we want to make a request to the endpoint https://api.github.com/users/djmgit/followers then we can use the above method and we will get a JSONArray in return.

All the code which scrapes user related data has been moved to ‘scrapeGithubUser’ method. This method scrapes basic user information like full name of the user, bio, user name, atom feed link, and location.

String fullName = html.getElementsByAttributeValueContaining("class", "vcard-fullname").text();
githubProfile.put("full_name", fullName);
 
String userName = html.getElementsByAttributeValueContaining("class", "vcard-username").text();
githubProfile.put("user_name", userName);
 
String bio = html.getElementsByAttributeValueContaining("class", "user-profile-bio").text();
githubProfile.put("bio", bio);
 
String atomFeedLink = html.getElementsByAttributeValueContaining("type", "application/atom+xml").attr("href");
githubProfile.put("atom_feed_link", "https://github.com" + atomFeedLink);
 
String worksFor = html.getElementsByAttributeValueContaining("itemprop", "worksFor").text();
githubProfile.put("works_for", worksFor);
 
String homeLocation = html.getElementsByAttributeValueContaining("itemprop", "homeLocation").attr("title");
githubProfile.put("home_location", homeLocation);

Next it returns other informations like starred repositories information, followers information, following information and information about the organizations the user belongs to but before fetching such data it checks whether such data is really needed. In this way unnecessary api calls are prevented.

if (terms.contains("starred") || terms.contains("all")) {
            String starredUrl = GITHUB_API_BASE + profile + STARRED_ENDPOINT;
            JSONArray starredData = getDataFromApi(starredUrl);
            githubProfile.put("starred_data", starredData);
 
            int starred = Integer.parseInt(html.getElementsByAttributeValue("class", "Counter").get(1).text());
            githubProfile.put("starred", starred);
        }
 
        if (terms.contains("follows") || terms.contains("all")) {
            String followersUrl = GITHUB_API_BASE + profile + FOLLOWERS_ENDPOINT;
            JSONArray followersData = getDataFromApi(followersUrl);
            githubProfile.put("followers_data", followersData);
 
            int followers = Integer.parseInt(html.getElementsByAttributeValue("class", "Counter").get(2).text());
            githubProfile.put("followers", followers);
        }
 
        if (terms.contains("following") || terms.contains("all")) {
            String followingUrl = GITHUB_API_BASE + profile + FOLLOWING_ENDPOINT;
            JSONArray followingData = getDataFromApi(followingUrl);
            githubProfile.put("following_data", followingData);
 
            int following = Integer.parseInt(html.getElementsByAttributeValue("class", "Counter").get(3).text());
            githubProfile.put("following", following);
        }
 
        if (terms.contains("organizations") || terms.contains("all")) {
            JSONArray organizations = new JSONArray();
            Elements orgs = html.getElementsByAttributeValue("itemprop", "follows");
            for (Element e : orgs) {
                JSONObject obj = new JSONObject();
 
                String label = e.attr("aria-label");
                obj.put("label", label);
 
                String link = e.attr("href");
                obj.put("link", "https://github.com" + link);
 
                String imgLink = e.children().attr("src");
                obj.put("img_link", imgLink);
 
                String imgAlt = e.children().attr("alt");
                obj.put("img_Alt", imgAlt);
 
                organizations.put(obj);
            }
            githubProfile.put("organizations", organizations);
        }

Similarly ‘scrapeGithubOrg’ is used to scrape information related to Github organization.

API usage

The API can be used in the ways shown below.

• api/githubprofilescraper.json?profile=<profile_name> : This is equivalent to api/githubprofilescraper.json?profile=<profile_name>&terms=all :This will return all the data the scraper is currently capable of collecting.
• api/githubprofilescraper.json?profile=<profile_name>&terms=<list_containing_required_fields> : This will return the basic information about a profile and data on selective fields as mentioned by the user.

For example – api/githubprofilescraper.json?profile=djmgit&terms=followers,following,organizations

The above request will return data about followers, following, and organizations apart from basic profile information. Thus the scraper will scrape only those data that is required by the user.

Future scope

‘scrapeGithubUser’ is still lengthy and it can be further broken down into smaller methods. The part of the method which scrapes basic user information like name, bio, etc can be moved to a separate method. This will further increase code readability and maintainability.

Now we have a much more functional and enhanced Github profile scraper which provides more data than the previous one.