Mailing Attachments Using Terminal in Open Event Android

The latest version of Open Event Android App Generator, v2 lacked the feature of mailing the generated APK to the email ID that is entered at the start of the app generation process. This also included mailing the error logs in case of APK failure.

This is an important feature for app generator because the process of app generation is a time taking one. The users have to wait for the app to be generated so that they can download the generated APK. To avoid this, the generator can automatically email the APK as soon as it is generated.

I took up this issue a few days back and started working on it. I started with thinking about the ways through which it will be implemented. This required some discussions with the mentors and co-developers. We finalised on the following ways:

  • Using Sendgrid
  • Using SMTP

I will be discussing the implementation of both of them in this blog. The code for APK mailing starts with the function call Notification.send in generator.py

if completed and apk_path and not error:
   Notification.send(
       to=self.creator_email,
       subject='Your android application for %s has been generated ' % self.event_name,
       message='Hi,<br><br>'
               'Your android application for the \'%s\' event has been generated. '
               'And apk file has been attached along with this email.<br><br>'
               'Thanks,<br>'
               'Open Event App Generator' % self.event_name,
       file_attachment=apk_path,
       via_api=self.via_api
   )
else:
   Notification.send(
       to=self.creator_email,
       subject='Your android application for %s could not generated ' % self.event_name,
       message='Hi,<br><br> '
               'Your android application for the \'%s\' event could not generated. '
               'The error message has been provided below.<br><br>'
               '<code>%s</code><br><br>'
               'Thanks,<br>'
               'Open Event App Generator' % (self.event_name, str(error) if error else ''),
       file_attachment=apk_path,
       via_api=self.via_api
   )

This leads me to the class Notification.py. It has three functions:-

1. send(to, subject, message, file_attachment, via_api)
2. send_mail_via_smtp(payload):
3. send_email_via_sendgrid(payload):

As the name suggests, the first function:

send(to, subject, message, file_attachment, via_api)

mainly decides which service (out of smtp and sendgrid) should be used to send the email, on the basis of the input parameters (especially, the ‘EMAIL_SERVICE’ parameter that has to be set in config.py).
The function looks like as follows:

send(to, subject, message, file_attachment, via_api)

It is in the send() that the other two functions are called. If the email_service is smtp, it calls the Notification.send_mail_via_smtp(payload). Otherwise, the Notification.send_email_via_sendgrid(payload) is called.
The sendgrid function is pretty straightforward:

@staticmethod
def send_email_via_sendgrid(payload):

   key = current_app.config['SENDGRID_KEY']
   if not key:
       logger.info('Sendgrid key not defined')
       return
   headers = {
       "Authorization": ("Bearer " + key)
   }
   requests.post(
       "https://api.sendgrid.com/api/mail.send.json",
       data=payload,
       headers=headers
   )

It requires a personalised sendgrid key which is accessed from the config.py file. Apart from that it handles some errors by giving logs in celery tasks. The main line in the function that initiates the email is a POST request made using the python library ‘requests’. The request is made as follows:

 requests.post(
       "https://api.sendgrid.com/api/mail.send.json",
       data=payload,
       headers=headers
   )

The send_mail_via_smtp(payload): function looks for some configurations before sending the mail:

@staticmethod
def send_mail_via_smtp(payload):
   """
   Send email via SMTP
   :param config:
   :param payload:
   :return:
   """
   smtp_encryption = current_app.config['SMTP_ENCRYPTION']
   if smtp_encryption == 'tls':
       smtp_encryption = 'required'
   elif smtp_encryption == 'ssl':
       smtp_encryption = 'ssl'
   elif smtp_encryption == 'tls_optional':
       smtp_encryption = 'optional'
   else:
       smtp_encryption = 'none'
   config = {
       'host': current_app.config['SMTP_HOST'],
       'username': current_app.config['SMTP_USERNAME'],
       'password': current_app.config['SMTP_PASSWORD'],
       'encryption': smtp_encryption,
       'port': current_app.config['SMTP_PORT'],
   }
   mailer_config = {
       'transport': {
           'use': 'smtp',
           'host': config['host'],
           'username': config['username'],
           'password': config['password'],
           'tls': config['encryption'],
           'port': config['port']
       }
   }

   mailer = Mailer(mailer_config)
   mailer.start()
   message = Message(author=payload['from'], to=payload['to'])
   message.subject = payload['subject']
   message.plain = strip_tags(payload['message'])
   message.rich = payload['message']
   message.attach(payload['attachment'])
   mailer.send(message)
   mailer.stop()

It is using the Marrow Mailer Python library to email with attachments(APK). This Python library can be installed using
pip install marrow.mailer
To use Marrow Mailer you instantiate a marrow.mailer.Mailer object with the configuration, then pass Message instances to the Mailer instance’s send() method.

You can refer to the following guides for more information about sending emails through command line:
https://github.com/marrow/mailer is the official repo of Marrow Mailer repository.
https://pypi.python.org/pypi/marrow.mailer
More detailled information on sending emails using Sendgrid can be found here https://www.daveperrett.com/articles/2013/03/19/setting-up-sendmail-with-sendgrid-on-ubuntu/

Displaying Upcoming Sessions at a Microlocation Open Event Android

When I am attending a session in a room, I don’t get information on what is coming up.”

The issue that the user expressed was that he wanted to know what were the upcoming sessions at a microlocation. While I took up this issue in Open Event Android a few days back, I was thinking of ways about how this can be implemented. The app should be easy-to-use even for non-developers and thus, any new feature shouldn’t be too complex in its implementation. We decided upon doing the following:

  • Adding an “upcoming” option in the options menu of the Location activity.
  • This option’s purpose was to trigger the app to show information about the upcoming session in that microlocation.

Initial changes in LocationActivity.java

First of all, we added a new icon in the options menu of LocationActivity.java. One of the things that we learnt there was to use ifRoom|collapseActionView option for the app:showAsAction  

attribute as frequently as possible. This option ensures that the title in the option’s menu is visible at all times irrespective of the options being visible along with their icons.

So in case, the title is too big and there is very little room for the options to appear individually, then instead of squeezing down the title, the “ifRoom” attribute will collapse the option icons and insert a 3-dotted drop-down option list with all the options appearing in the drop-down.

Something like this:

The icon’s XML element and UI looked something like this:

<item
       android:id="@+id/upcoming_sessions"
       android:icon="@drawable/ic_timeline_white_24dp"
       android:title="@string/upcoming"
       app:showAsAction="ifRoom|collapseActionView"
app:actionViewClass="android.support.v7.widget.Button"/>

About the drawable icon that you see in the screenshot above, it was a tough find. Before I talk about how I came across this icon, I will talk about adding an icon in Android Studio.

How to add an icon in Android Studio?

Adding an item in Android studio means adding a drawable at a basic level. You can find all drawables under the app/src/main/res/drawable folder.

To add a new drawable, right-click on the drawable folder and go to new –>Vector asset. A window similar to what is shown below will appear.

Now, on selecting the “icon” option you will be taken to a huge list of icons that you can add in your app and then use them subsequently. But the problem here is that it is tough at times to find the icon that will be fit for your purpose. Like in my case, there was no direct icon for “upcoming”. This is when we had to do something more. We had to browse to this amazing site by Google: https://material.io/icons/ This site shows all the available icons in a much more interactive way and it was a lot more easier for me to come across the icon we wanted using this site.

The vector drawable file for the icon we chose looks like this:

<vector xmlns:android="http://schemas.android.com/apk/res/android"
       android:width="24dp"
       android:height="24dp"
       android:viewportWidth="24.0"
       android:viewportHeight="24.0">
   <path
       android:fillColor="#FFFFFF"
       android:pathData="M23,8c0,1.1 -0.9,2 -2,2 -0.18,0 -0.35,-0.02 -0.51,-0.07l-3.56,3.55c0.05,0.16 0.07,0.34 0.07,0.52 0,1.1 -0.9,2 -2,2s-2,-0.9 -2,-2c0,-0.18 0.02,-0.36 0.07,-0.52l-2.55,-2.55c-0.16,0.05 -0.34,0.07 -0.52,0.07s-0.36,-0.02 -0.52,-0.07l-4.55,4.56c0.05,0.16 0.07,0.33 0.07,0.51 0,1.1 -0.9,2 -2,2s-2,-0.9 -2,-2 0.9,-2 2,-2c0.18,0 0.35,0.02 0.51,0.07l4.56,-4.55C8.02,9.36 8,9.18 8,9c0,-1.1 0.9,-2 2,-2s2,0.9 2,2c0,0.18 -0.02,0.36 -0.07,0.52l2.55,2.55c0.16,-0.05 0.34,-0.07 0.52,-0.07s0.36,0.02 0.52,0.07l3.55,-3.56C19.02,8.35 19,8.18 19,8c0,-1.1 0.9,-2 2,-2s2,0.9 2,2z"/>
</vector>

What would the upcoming icon do?

Keeping in mind the necessity for the feature to be less complex, I decided that the upcoming icon will lead the user to a dialog box that shows the status of upcoming sessions in that micro location. The implementation for this feature involved 2 main things:

  1. Finding out the upcoming session from the list of sessions in the microlocation.
  2. Generate a dialog box that shows information about that session.

Finding position of upcoming session in Recycler View

Upcoming session will be a session whose starting time comes after the current time. So the approach was simple.

  1. Run a loop on a sorted list of all sessions in a microlocation.
  2. Find out every session’s start time.
  3. Compare the start time of every session with the current time.
  4. Find the first session whose start time comes after the current time.
  5. Store that session’s position, name, ID and other stuff like track name and track color.
  6. Break out of the loop.

This was the basic logic or algorithm, so to say. Here’s the implementation in the upcomingSession() function:

public void upcomingSession(){
   String upcomingTitle = "";
   String track = "";
   String color = null;
   Date current = new Date();
   for (Session sess:sortedSessions){
       try {
           Date start = DateUtils.getDate(sess.getStartsAt());
           if (start.after(current)){
               upcomingTitle = sess.getTitle();
               track = sess.getTrack().getName();
               color = sess.getTrack().getColor();
               break;
           }
       } catch (ParseException e) {
           e.printStackTrace();
       }
   }

Now, displaying a dialog box consisting of all the necessary information is an easy thing to do once you have the required information. So, I’ll just provide some code for it here without explaining much about it.

The initialisations:

public void upcomingSessionsInitial(){
   upcomingDialogBox = new Dialog(this);
           upcomingDialogBox.setContentView(R.layout.upcoming_dialogbox);
           trackImageIcon = (ImageView)upcomingDialogBox.findViewById(R.id.track_image_drawable);
           upcomingSessionText = (TextView) upcomingDialogBox.findViewById(R.id.upcoming_session_textview);
           upcomingSessionTitle = (TextView) upcomingDialogBox.findViewById(R.id.upcoming_Session_title);
           Button dialogButton = (Button) upcomingDialogBox.findViewById(R.id.upcoming_button);
           dialogButton.setOnClickListener(new View.OnClickListener() {
               @Override
               public void onClick(View view) {
                   upcomingDialogBox.dismiss();
               }
           });
}

The calling:

switch (item.getItemId()){
       case R.id.action_map_location:
           FragmentManager fragmentManager = getSupportFragmentManager();
           FragmentTransaction fragmentTransaction = fragmentManager.beginTransaction();

           Bundle bundle = new Bundle();
           bundle.putBoolean(ConstantStrings.IS_MAP_FRAGMENT_FROM_MAIN_ACTIVITY, false);
           bundle.putString(ConstantStrings.LOCATION_NAME, location);

           Fragment mapFragment = ((OpenEventApp)getApplication())
                   .getMapModuleFactory()
                   .provideMapModule()
                   .provideMapFragment();
           mapFragment.setArguments(bundle);
           fragmentTransaction.replace(R.id.content_frame_location, mapFragment, FRAGMENT_TAG_LOCATION).addToBackStack(null).commit();

           sessionRecyclerView.setVisibility(View.GONE);
           noSessionsView.setVisibility(View.GONE);
           menu.setGroupVisible(R.id.menu_group_location_activity, false);
           return true;
       case android.R.id.home:
           onBackPressed();
           getSupportFragmentManager().popBackStack();
           sessionRecyclerView.setVisibility(View.VISIBLE);
           return true;
       case R.id.upcoming_sessions:
           upcomingDialogBox.show();
           return true;
       default:
           return true;
   }
}

Final result:

This is the final result or solution that we generated for the issue that was addressed by one of the users:

Some useful links are:

Handlebars.js used in Open Event Web App

I recently started working in the Open Event Webapp project. One of the initial issues that I took up was a trivial UI bug. It was about adding sponsor names beneath sponsor images for better representation. The issue can be found here. On reading up the code base and exploring the project a bit, I came across a new template – Handlebars.js. Handlebars is a template which has it’s base with the Mustache templating language. One of the early discoveries that I made with Handlebars.js was the use of {{ }} and {{{ }}} and the basic difference between them. In general, all Handlebar.js expressions, just like in Mustache templating, are written between {{ }} or {{{ }}} type of brackets. That is how I learned to identify and distinguish Handlebars from core HTML, even though they are inter-linked. The official Handlebars documentation describes Handlebars expressions in the following way:

A handlebars expression is a {{, some contents, followed by a }} ”

Getting started with Handlebars.js

Installation:

For a basic Linux installation, type the following in your command line:

npm install --save handlebars

Including Handlebars in HTML:

<script src="handlebars-v4.0.10.js"></script>

Handlebars templates are often stored in .hbs files for better readability and accessibility. The Open Event Webapp project consists of a handlebars .hbs file for each of the tracks, events, rooms, schedule, sessions and speakers templates. These can be found here, that is under src/backend/templates folder.

Difference between {{ }} and {{{ }}}: 

Handlebars enables developers to print raw HTML tags or code with the help of {{{ }}}. On the contrary, if you don’t want to print HTML (which is usually the case), use {{ }}. For better understanding, let’s take an example.

If our JS has an object that looks something like:

$(function () {
   var templateScript = $("#title-template").html();

   var temp = Handlebars.compile(templateScript);

var Title= {
“title”: <a> Handlebars</a>
}

Then, HTML of the following kind will help to distinguish the {{ }} and {{{ }}} brackets.

<script id=”title-template” type=”text/x-handlebars-template”>
{{title}}
{{{title}}}
</script>

//the first line will contain an anchor tag with the name “Handlebars”
//the second line will contain “<a>Handlebars</a>”

Block helpers in Handlebars:

Block helpers are identified by a ‘ #’ and they help to define and access custom iterators.

Handlebars allow calling JavaScript functions with the help of ‘helpers’. It doesn’t allow direct JavaScript code in the HTML with templates. We can create our own helpers using Handlebars.registerHelper () in our JavaScript. We generally pass a function to the helper. A good example was provided in the Handlebars.js documentation:

Handlebars.registerHelper('noop', function(options) {
  return options.fn(this);
});

By default, Handlebars helpers take the current context as the context to pass(“this”). Other fields are overshadowed. Incase, we want to access one of the fields that is masked by the default “this” context, we have to use a path reference.

Iterations using helpers:

Helpers can be a great way  to iterate over lists or objects. I will demonstrate it with an example from the Open Event Webapp project. To display all the sponsors of an event in the home page of the event Webapp, we use the following handlebars code, where we iterate over the object list “sponsorpics” that we have. It looks something like this:

{'1': ['Oreilly', 'Amazon'], '2': ['Huawei', 'Google'],'3': ['RedHat', 'GitHub']}
     
{{#if eventurls.sponsorsection}}
<div class="sponsor-container">
       <section class="sponsorscont">
         <div class="row sponsor-row">
           <div class="col-sm-12 col-md-12 col-xs-12 text-center">
             <h1 class="section-header">Proudly supported by</h1><br>
           </div>
         </div>
         <div class="row">
           <div class="col-sm-10 col-sm-offset-1">



             <div class="row">
               {{#each sponsorpics}}
                 {{#each this}}
                   <div class="{{{divclass}}}">
                     <div class=" {{{sponsorimg}}} text-center">
                       <a href="{{{url}}}" data-toggle="tooltip" title="{{{type}}}">
                         <img class="lazy centre {{{imgsize}}}" alt="{{{name}}}" data-original="{{{logo}}}">
                       </a>
                       {{{name}}}
                     </div>
                   </div>
                 {{/each}}
               {{/each}}
             </div> <!-- sponsor-row -->
           </div>
         </div>
       </section>
     </div>
   {{/if}}

For your reference, you can view a sample Webapp for the OSCON 2017 event here.
For further information, please refer to Handlebars.js .
An interesting tutorial about Handlebars in 10 mins or less can be found here.

Serialisation, Deserialisation and gson in Open Event Android

JSON is a format to exchange and inter-change data. It has almost become a universal means for transferring data because of it being lightweight, human understandable, machine readable and most importantly, it’s language independence. It can be used with, and in any programming language. (Reference)

FOSSASIA’s Open Event project makes extensive use of JSON for transferring information about events, their speakers, sessions and other event information. The Open Event Android application itself loads all it’s data in JSON format. This JSON is uploaded by the user in the form of a .zip compressed file or by giving an API link. Now before we use this data in the app, we have to parse the data to get Java objects that can be used in Android. Deserialisation is the process of converting JSON to Java objects. In the same way, Serialisation refers to converting Java objects into JSON.

In most applications and frameworks, JSON is serialized and deserialized on multiple instances. The most common approach is to create objects corresponding to the JSON format and then use functions to convert them to JSON line-by-line, attribute by attribute. While this approach will work, it will mean writing unnecessary code and spending a lot more time on it. In Open-event-Android, we are using Google’s gson library to  serialise and deserialise JSON and corresponding Java objects respectively.

Why are we using gson specifically? And why do we need any library in the first place?

Yes, we can obviously make functions in Java by defining all the JSON parameters and converting the JSON into Java objects manually. It is indeed the obvious approach, but the entire process will be time-consuming, complex and definitely not error-free. Also, as projects become bigger, it is inevitable to take care of the code size and reduce it to only what is necessary.

Let’s take a look at the one of the Open Event JSON file as a sample to try and understand things in a better way. This is an example of event.json in the FBF8’2017 sample.

{

"id": 180,
"name": "F8-Facebook Developer Conference 2017",
"latitude": 37.329008,
"longitude": -121.888794,
"location_name": "San Jose Convention Center, 150 W San Carlos St  San Jose  CA USA 95113 ",
"start_time": "2017-04-18T10:00:00-07:00",
"end_time": "2017-04-19T10:00:00-07:00",
"timezone": "US / Pacific",
"description": "Join us for our annual 2-day event, where developers and businesses come together to explore the future of technology. Learn how Facebook connects the world through new products and innovation. This year's event is bigger than ever – with more than 50 sessions, interactive experiences, and the opportunity to meet one-on-one with members of the Facebook team.",
"background_image": "/images/background.jpg",
"logo": "/images/fbf8.png",
"organizer_name": "Facebook",
"organizer_description": "Join us for our annual 2-day event, where developers and businesses come together to explore the future of technology. Learn how Facebook connects the world through new products and innovation. This year's event is bigger than ever – with more than 50 sessions, interactive experiences, and the opportunity to meet one-on-one with members of the Facebook team.",
"event_url": "https://www.fbf8.com",
"social_links": [
{"id": 1,
"link": "https://www.facebook.com/FacebookForDevelopers",
"name": "Facebook"},
{"id": 2,
"link": "https://twitter.com/fbplatform",
"name": "Twitter"},
{"id": 3,
"link": "https://www.instagram.com/explore/tags/fbf8/",
"name": "Instagram"},
{"id": 4,
"link": "https://github.com/Facebook",
"name": "GitHub"}],
"ticket_url": null,
"privacy": "public",
"type": "",
"topic": "Science & Technology",
"sub_topic": "High Tech",
"code_of_conduct": "",
"copyright": {
"logo": "",
"licence_url": "https://en.wikipedia.org/wiki/All_rights_reserved",
"holder": "Facebook",
"licence": "All rights reserved",
"holder_url": null,
"year": 2017},
"call_for_papers": null,
"email": null,
"has_session_speakers": false,
"identifier": "7d16c124",
"large": null,
"licence_details": {
"logo": "",
"compact_logo": "",
"name": "All rights reserved",
"url": "https://en.wikipedia.org/wiki/All_rights_reserved",
"long_name": "All rights reserved",
"description": "The copyright holder reserves, or holds for their own use, all the rights provided by copyright law under one specific copyright treaty."},
"placeholder_url": null,
"schedule_published_on": null,
"searchable_location_name": null,
"state": "Published",
"version": {
"event_ver": 0,
"microlocations_ver": 0,
"sessions_ver": 0,
"speakers_ver": 0,
"sponsors_ver": 0,
"tracks_ver": 0}}

For a file with so many attributes, it can be very tedious to make a parser class in Java. Instead, we simply use the gson library to do the job.

To install gson in maven central, add the following dependency:

<dependencies>
   <!--  Gson: Java to Json conversion -->
   <dependency>
     <groupId>com.google.code.gson</groupId>
     <artifactId>gson</artifactId>
     <version>2.8.0</version>
     <scope>compile</scope>
   </dependency>
</dependencies>                                             

[source:https://github.com/google/gson/blob/master/UserGuide.md]

To use gson in Android, we need to add the following dependency in build.gradle of the project.

compile ‘com.google.code.gson:gson:2.6.2’

Using gson in your code is as simple as using any built-in class in java. There are two main functions – toJson()and fromJson() that will be used throughout the processes of serialisation and deserialisation.

First off, let’s declare a basic Java object. I will take an example of a Java object that represents a car to simplify things a bit. This object car contains 3 variables- Name, Model number and color of the car. For example:

class Car(){
String  name;
int model_no;
String colour;
Car(){}
}

Serialisation:

Serialising the car object now, that is converting the object into JSON format:

Car one = new Car();
Gson gson = new GSON();
String serialised_output= gson.toJson(car);//serialised_output=>JSON

Deserialisation:

Deserializing the JSON now, that is converting the JSON into java objects:

Car car = gson.fromJson(serialised_output,Car.class);

Finally, you can find the GitHub link for gson library here in case you want to know more about it or if you are facing some issues with the library, you can report them there. This is the link to the official Google group for discussions related to gson.

Debugging JSON Files of Sample Events for Open Event Android using Stetho

In this blog, I will talk about data validation and debugging in Android using the Stetho-A debug bridge for Android by Facebook. Most Android applications have JSON files to populate the elements like RecyclerViews, ListViews and different types of Layouts. Stetho is a debug tool for Android which uses the well-known Chrome Developer tools as it’s User Interface. With Stetho, we can see all our incoming JSON data in spreadsheets making debugging much easier and fun. What’s more, it’s completely Open source.

Getting started with Stetho

To start with, we need to enable Stetho by adding it as one of the dependencies in the build.gradle file.

compile 'com.facebook.stetho:stetho:1.4.2'

It is already enabled in the Open Event Android app.

Now, you need to add the api endpoint for your sample in the config.json file in the project(if you are using the Android repo for debugging). The config.json file is at app/assets/

In case, you are using an Open Event generated app then there’s no need to do that.

Now connect your phone through the USB cable to your laptop and start your application.

Next, you have to browse to chrome://inspect where you will see your device in the window as shown below.

 

On clicking “Inspect”, you will be shown the Stetho debug tool interface. Something like this:

When you download the details by clicking “Yes” on the starting of the application, be sure to keep it connected to the Stetho tool in the ‘inspect’ mode.

Once the data has loaded, go to Web SQL-> default.realm. You will see tabs like:-

class_speaker, class_sponsor, class_session, class_track among others.

On clicking them, you will see well-organised tables that show all linked attributes of the class that you selected.

Sessions can be seen along with all the related information in a tabular format.

Stetho displays speakers of the event and their information in an easy-to-read tabular format.

Micro locations of the event along with related information in a tabular format.

This is how you can view all the attributes of your sample in a tabular layout and hence, debug them easily. Although, Stetho is about much more than this, this is all I will talk about in this blog.

You can read more about Stetho and it’s functionalities here.

The official repo for Stetho can be found here for the source code.

Creating A Sample Event Web App Using the Open Event Framework

I have been creating sample events for the Open Event Framework by creating JSON files for them. In this blog, I will walk you through 4 easy steps to create a sample event for FOSSASIA Open Event. These are the steps that I followed while I contributed in making sample web apps:

Let’s start by taking the example of a sample event. I recently created a sample for FBF8’2017. F8 is conference held by Facebook annually.

I started by visiting the official F8 site. I personally used Google Chrome for it as I was almost positive that the official site will be having an API which will make it easier for me to extract JSON for the event. Also, Google Chrome developer tools make it easier to access the API. I have previously explained how to access JSON through APIs in one of my previous blog posts that can be found here.

So in the F8 site, I looked for the page that will lead me to the full schedule of the conference:

After clicking on the “SEE FULL SCHEDULE” button, this is where it leads us, with the developer tools ON.

That weird named file on that appears in under the XHR tab contains our JSONs!

[{1492502400: [{day: 1, sessionTitle: "Registration", sessionSlug: "day-1-registration",…},…],…}, {,…}]

0:{1492502400: [{day: 1, sessionTitle: "Registration", sessionSlug: "day-1-registration",…},…],…}

1:{,…}

This is how the file looks on clicking it at first. You must be seeing small clickable arrows on the sides which expand the JSON to reveal all the contents. But okay, we have crossed our first step: Obtaining JSON from the website’s API!

Step 2: Tailoring the JSON according to the Open Event format needs to be done now. To view the latest API format, visit here. Now this step is obviously most time-consuming and tedious. We need to modify the API we obtained from the F8 site to meet the Open Event requirements.

I mainly used Sublime text editor to do this. It has this great feature to find and replace text in text files.

Taking an example of a session object. This is what we will get from the API:-

allDay:false

createdAt:"2017–01–28T01:24:12.770Z"

day:1

displayTime:"3:00pm"

endTime:{__type: "Date", iso: "2017–04–18T15:50:00.000Z"}

iso:"2017–04–18T15:50:00.000Z"

__type:"Date"

hasDetails:true

objectId:"62yCeJMvAf"

ogImage:{__type: "File", name: "296643cca9aad196bbd3534324c52049_Making Great Camera Effects.png",…}

name:"296643cca9aad196bbd3534324c52049_Making Great Camera Effects.png"

url:"https://f82017-parse.s3.amazonaws.com/296643cca9aad196bbd3534324c52049_Making%20Great%20Camera%20Effects.png"

__type:"File"

onMySchedule:false

sessionDescription:"With AI that transforms images, Facebook's Camera Effects Platform lets you create animated masks, interactive overlays, frames, and other effects that people can apply to their photos and videos in the Facebook camera. Learn best technical and design practices for creating effects using this innovative technology, and see examples of successful effects."

sessionLocation:"G"

sessionSlug:"making-great-camera-effects"

sessionTitle:"Making Great Camera Effects"

sortTime:"1492527600"

speakers:[{speakerName: "Volodymyr Giginiak", createdAt: "2017–01–28T01:27:49.292Z",…},…]

0:{speakerName: "Volodymyr Giginiak", createdAt: "2017–01–28T01:27:49.292Z",…}

1:{speakerName: "Stef Smet", createdAt: "2017–01–28T01:30:16.022Z",…}

startTime:{__type: "Date", iso: "2017–04–18T15:00:00.000Z"}

iso:"2017–04–18T15:00:00.000Z"

__type:"Date"

tags:["Camera"]

0:"Camera"

updatedAt:"2017–04–18T19:19:16.501Z"

Now, this is where I used the find & replace feature in Sublime text to make it look like the open-event standard JSON. For instance, replacing “sessionLocation” with “microlocation”, “sessionDescription” with “long_abstract” and so on. Yes, there were a few manual changes as well that I had to make.

This is how the final JSON session object looked like after completing step 2:

{
     "short_abstract": "",
     "id": 26,
     "subtitle": "making-great-camera-effects",
     "title": "Making Great Camera Effects",
     "long_abstract": "With AI that transforms images, Facebook's Camera Effects Platform lets you create animated masks, interactive overlays, frames, and other effects that people can apply to their photos and videos in the Facebook camera. Learn best technical and design practices for creating effects using this innovative technology, and see examples of successful effects.",
     "end_time": "2017-04-18T15:50:00-07:00",
     "microlocation": {
        "id": 5,
        "name": "Hall G"
     },
     "start_time": "2017-04-18T15:00:00-07:00",
     "comments": "",
     "language": "",
     "slides": "",
     "audio": "",
     "video": "",
     "signup_url": "",
     "state": "accepted",
     "session_type": {
        "id": 6,
        "name": "50 minutes session",
        "length": "00.50"
     },
     "track": {
        "id": 2,
        "name": "Camera",
        "color": "#FF3046"
     },
     "speakers": [{
           "city": "",
           "heard_from": "",
           "icon": "",
           "long_biography": "",
           "small": "",
           "photo": "",
           "thumbnail": "",
           "short_biography": "",
           "speaking_experience": "",
           "sponsorship_required": "",
           "organisation": "Facebook",
           "id": 40,
           "name": "Volodymyr Giginiak"
        }
     ]
  }

Apart from this, there were places where I had to dig for information about speakers, their social website links, their images and so on because the open-event JSON format expects all this.

Step 3: JSON Validation

It is very likely that one of your JSON files will contain invalid JSON strings. So it is important to validate them. I used this online tool and I liked it very much. Once the JSON is validated, everything is set. All you need to now do is-move to step 4!

Step 4: Web app generation

Compress your JSON archive in .zip format and then go to the webapp generator. Enter your email ID and upload the zip file. Now, you can deploy, download and preview your personalised web-app, generated by FOSSASIA open-event web app generator.

The official web-app generator repo is here.

If you want to contribute to FOSSASIA by creating a sample, you can create an issue in the official open-event repo, along with a PR which contains the JSON sample files, the zipped file and a link to the web-app deployment on GitHub pages.

Data Scraping using Python with BeautifulSoup4 for Open Event samples

was creating samples for Open Event Android and Open Event Webapp when the idea of web scraping through scripts stuck me.

I chose Python for web scraping. But what’s more to using Python other than it’s user-friendliness, easy syntax and speed? It’s the wide variety of open-source libraries that come along with it!

One of them is the BeautifulSoup library for extracting data from XML and HTML tags and files. It provides with ways to search and sort through webpages, find specific elements that you need and extract them in objects of your preference.

Apart from BeautifulSoup, another module that can be easily neglected but is of great use is:- urllib2. This module is to open URLs that are fed to it.

Hence, with a combination of the above 2 Python modules, I was able to scrape off the main sample data like speakers, sessions, tracks, sponsors and so on for PyCon 2017 (sample that I created recently) in a quicker and more efficient way.

I will now talk about these modules in greater detail and their use in my web scraping scripts.

Web Scraping with BeautifulSoup

Start by installing Beautifulsoup module in your Python environment. I believe that it will be best to install it using  pip.  Run the following command to install Beautifulsoup:-

pip install beautifulsoup4

Next up, you need to import the module to your script by adding the following line:-

from bs4 import BeautifulSoup

Now you need to open a webpage using the module in the following way. For that we first have to open the URL of the page that is being scraped. We use urllib2 to do that.

import urllib2
site=urllib2.urlopen("https://us.pycon.org/2017/sponsors/")

Next we use BeautifulSoup to get the contents of the webpage in the following way:-

website=BeautifulSoup(site)

Suppose that, next I want to obtain the data inside all the <div> tags in the webpage that have their class=”sponsor-level”.

I will use BeautifulSoup in the following way along with the ‘find_all’ method to do so :-

divs=website.find_all("div",{'class':'sponsor-level'})

Here, divs will now be a special <list> type element containing all the HTML in the div tags of class=’sponsor-level’. Now I can do whatever with I want with this list. Parse it in any way, store it any kind of data type without any difficulty.

BeautifulSoup also helps in accessing the children tags or elements of different HTML or XML tags using the ‘dot’ (.) operator. Following example will make it more clear:-

tag = soup.b

Here, ‘tag’ will contain all the  <b> elements of the parent element that is stored in ‘soup’ variable.

To access attributes of HTML or XML tags, there is a special way as demonstrated below:-

To access the ‘href’ attribute of the first <a> tag in the webpage:-

soup.find_all(“a”)[0][‘href’]

A sample script…

Following is a script that I used to obtain sponsors from the PyCon 2017 site in json format.

import urllib2

site=urllib2.urlopen("https://us.pycon.org/2017/sponsors/")

from bs4 import BeautifulSoup

s=BeautifulSoup(site)

divs=s.find_all("div",{'class':'sponsor-level'})

spons=[]

for level in divs:

levelname=level.find_all("h2")[0].string

level_sponsors=level.find_all("div",{'class':'sponsor'})

for level_sponsor in level_sponsors:

anchor=level_sponsor.find_all("a")

dictionary={}

dictionary['id']=""

dictionary['name']=anchor[1].string

dictionary['level']=""

dictionary['sponsor_Type']=levelname

dictionary['url']=anchor[0]['href']

dictionary['description']=level_sponsor.find_all("p")[1].string

spons.append(dictionary)

print spons

There are still a lot of other functionalities of BeautifulSoup that are available and can make your life easier. These can be found here in the official BeautifulSoup documentation.

Data Scraping with Selenium

There still exists websites without any APIs. Scraping data from such sites can be very time-consuming and manual. I created samples for Open Event App generator. One of the samples that I created was for All Hands Hawaii 2016. This site didn’t have any API to enable easy data scraping.

How do we find out if a website is using an API or not?

Using Google Chrome, go to View → Developer → Developer Tools. Under the Network →XHR look for API endpoint with a bit of Hit and Trial method. (XHR stands for XMLHttpRequest)

However, what if there is no API being used in the site? How would you scrape data in that case? Will you now manually click onto every hyperlink on the site and visit every page to get the data by manually copying and pasting it? Could there be someone doing that manual job for you? Or better could there be “something” doing that job for you? Yes, It’s selenium.

Selenium Web Browser Automation

Selenium is a tool that automates the task of browsing through the internet. Although, technically it is used for web testing purposes but there is no restriction to it’s utility.

Let’s get started with basics of Selenium:-

INSTALLATION:
  1. Run the following command pip install selenium (Quick Tip: It is advised to use virtualenv)
  2. Selenium requires drivers to run. Different browsers use different drivers. Choose an appropriate driver for your browser. some common drivers are shown below (Source)-

Chrome: https://sites.google.com/a/chromium.org/chromedriver/downloads

Firefox: https://github.com/mozilla/geckodriver/releases

Safari: https://webkit.org/blog/6900/webdriver-support-in-safari-10/

BASIC FUNCTIONALITIES-SELENIUM:

Visit a page ( using the get() ):

 driver.get(urlofpage)

Navigating to various elements on the visited/current webpage:

  1. BY ID: 
    WebElement element = driver.findElement(By.id(“ui_elementid”));
  2. BY CLASS NAME:
    List<WebElement> cheeses = driver.findElements(By.className(“cheese”));
  3. BY TAG NAME:
    WebElement tag = driver.findElement(By.tagName(“tag_name”));
  4. BY CSS: WebElement cs = driver.findElement(By.cssSelector(“#”));
  5. BY LINK TEXT:
    WebElement cheese = driver.findElement(By.linkText(“blog”));

    If the element href is something like urlofpage?q=blog.

  6. BY XPATH:
  7.  List<WebElement> xp = driver.findElements(By.xpath(“//input”))

 

We can also use JavaScript along with Selenium. This might be a helpful thread for the same.

Another really good link for the same is http://www.marinamele.com/selenium-tutorial-web-scraping-with-selenium-and-python

ButterKnife for Open Event Android

Open Event Android, by FOSSASIA has been coded in a very clean and systematic manner. One of the great things used in it is the ButterKnife library. It has made increased the readability and understand ability of the app’s code.

ButterKnife, developed and maintained by Jake Wharton(Square Inc.) is an Android library designed to make Java code less complex and efficient. The library operates with the use of annotations and hence, binds classes to the relevant annotations in Java and do the job.

First off, here’s the very first thing that you want to do before starting to use ButterKnife — Adding it’s dependency in the build.gradle file of your project.

compile 'com.jakewharton:butterknife:8.6.0
annotationProcessor 'com.jakewharton:butterknife-compiler:8.6.0'

At FOSSASIA, we have extensively used this amazing library to simplify the code for our Open Event Android application. I hope this blog will help newcomers in understanding our code base in a better way.

We have the famous @Bindview annotation by ButterKnife to bind views to Android components without having to call R.FindViewById() method for every view. The following is a code sample from Open Event Android about the same.

@BindView(R.id.toolbar) Toolbar toolbar;

@BindView(R.id.nav_view) NavigationView navigationView;

@BindView(R.id.progress) ProgressBar downloadProgress;

@BindView(R.id.layout_main) CoordinatorLayout mainFrame;

@BindView(R.id.drawer) DrawerLayout drawerLayout;

@BindView(R.id.appbar) AppBarLayout appBarLayout;

Similarly, the ButterKnife library also handles events in an Android application and has annotations like @onClick, @onLongClick among others. So while using ButterKnife for an event, choose the appropriate event and the corresponding annotation followed by the Java method that is to be executed. Here is an example for the same.

@OnClick(R.id.submit)

public void submit(View view) {

// TODO submit data to server...

}

The ButterKnife library also helps to bind resources in your project.

It simplifies the View holder pattern inside a list adapter as illustrated in the following example.

public class MyAdapter extends BaseAdapter {

@Override public View getView(int position, View view, ViewGroup parent) {

   ViewHolder holder;

   if (view != null) {

     holder = (ViewHolder) view.getTag();

   } else {

     view = inflater.inflate(R.layout.whatever, parent, false);

     holder = new ViewHolder(view);

     view.setTag(holder);

   }

   holder.name.setText("John Doe");

   // etc...

   return view;

 }

static class ViewHolder {

   @BindView(R.id.title) TextView name;

   @BindView(R.id.job_title) TextView jobTitle;

   public ViewHolder(View view) {

     ButterKnife.bind(this, view);

   }

 }

}

The above code sample has been taken from http://jakewharton.github.io/butterknife/ and you can visit the site for further in-depth knowledge on ButterKnife.

How ButterKnife works?

I will be talking a bit about the working of the ButterKnife library. Firstly, I would like to introduce annotations concept in Java briefly before proceeding.

“In the Java, computer programming language, an annotation is a form of syntactic metadata that can be added to Javasource code. Classes, methods, variables, parameters and packages may be annotated.” As it is rightly said, Annotations can be of various uses like Getting information for the compiler, Compile-time and deployment-time processing, Runtime processing. However, please note that annotations can NOT modify and edit the existing classes. They can simply make more classes.

With this, I’m gonna continue on the working of ButterKnife. From the above discussion it must be clear now that ButterKnife is not changing any of our classes. It makes new classes to handle the code.

ButterKnife goes through all the files in your project and identifies the ButterKnife annotations used, if any. It then creates new classes for every annotation encountered, based on the annotations used. These new classes contain the general methods that would have been used in a project without ButterKnife. When we finally call ButterKnife.inject(this) , the inject()of all the new classes made by ButterKnife are called which then perform the desired function during execution.

Finally, try to go through the complete documentation for Butterknife and Java annotations for more information. 

Recyclerview in Open Event Android: A great upgradation to ListView

Recently, I was fixing a trivial bug in the Open Event Android app, an interesting thing caught my attention. Most of the lists in the app where actually not implemented using the traditional ListView, but instead used something called RecyclerView. The beautiful transitions that make this app the beauty it is, is due to the amazing and easy to code Recycler Views used in it.

A screen shot showcasing the simplicity of the FOSSASIA Open event android app.

 

List View is one among the various views that Android provides. In a nutshell, it is literally a group of generic views in Android. Why then use an entire new and more complex view, if all that it offers is a number of views displayed together? Well, it certainly has certain add-ons to the most generic set of views like Image and TextViews in Android.

I encourage you to think about what different could the ListView be as compared to multiple declarations of TextView. The thing with multiple declarations of a Text or Image View is that they aren’t “scrollable”, that is, if you want numerous Views in a single activity, without List View, you will have to fit them on one screen, else make another activity. So, if you want a scrollable view with same basic view items, List View is the way to go about it in Android.

The next question that may arise is:

How does a Listview work?

Google defines Adapters as “a bridge between an AdapterView and the underlying data for that view”. List Views use Adapters that retrieve data from Arrays, Lists and other data structures and queries, plugs them into our List View and thus, creates a beautiful looking List in no time. In Spite of all this, List View certainly lags in various fields that deal with performance, memory and compactness issues. As a great improvement Recycler View was introduced.

Using Recyclerview you are not restricted to vertical scrollable views. It has the ability to provide horizontal, vertical, grid and staggered layouts among others, with the help of different layout managers:-

recyclerView.setLayoutManager(new StaggeredGridLayoutManager(2,StaggeredGridLayoutManager.VERTICAL));
recyclerView.setLayoutManager(new LinearLayoutManager(RecyclerView_Activity.this,LinearLayoutManager.HORIZONTAL, false));

RecyclerView.ItemDecoration class allows us to add special drawings to our view. Thus, we can add dividers, borders and much more using this class.

public class DividerItemDecoration extends RecyclerView.ItemDecoration {
 
     private Drawable mDivider;
 
     public DividerItemDecoration(Drawable divider) {
         mDivider = divider;
     }

Adding animation used to be a devil of a task in Listview days, however with the release of Recyclerview, it has become a lot more easy. RecyclerView.ItemAnimator class helps in animating items while the adapters are being changed.

Next, we have something related to the performance:-

In a List View, you might have come across the term, ‘Viewholder’, but if you haven’t, that’s okay too. However, in Recyclerview, using View Holders have been made mandatory and for a good reason. The class goes by the name RecyclerView.ViewHolder.

What exactly are View Holders?
The clean dividers and minimal design in the Recyclerview makes up for a subtle UI.

In a broader sense, it is used to provide information about the identity of an itemView and it’s position within a Recyclerview. Without View Holders we have an overhead of calling the function findViewById() for every item in our Recyclerview! This upgradation from the typical findViewById() to View Holders awards us with better and more smooth scrolling through the list.

How does Recyclerview work?

List View is an ancestor to Recyclerview. Recyclerview has everything that List View offers to us (except for a few things like setchoicemode(), which I might take up in my upcoming blog posts), and much more.

A great achievement that Android achieved through Recyclerview has been it’s ability to “recycle” views on the way. In laymen terms, if you scroll through a Recyclerview, the views that go away from the screen during scrolling are the same views that come into the screen. The only difference is that the data in the views leaving the screen is changed to the data of the views entering your screen and so they are presented as new views. In this way, a Recyclerview helps saving memory for sure as compared to List Views.

There are still a lot of other things to learn about Recycler and List Views and you find more resources here.