Handling Data Requests in Open Event Organizer Android App

Open Event Organizer is a client side application of Open Event API Server created for event organizers and entry managers. The app maintains a local database and syncs it with the server when required. I will be talking about handling data requests in the app in this blog.

The app uses ReactiveX for all the background tasks including data accessing. When a user requests any data, there are two possible ways the app can perform. The one where app fetches the data directly from the local database maintained and another where it requests data from the server. The app has to decide one of the ways. In the Organizer app, AbstractObservableBuilder class takes care of this. The relevant code is:

final class AbstractObservableBuilder<T> {

   private final IUtilModel utilModel;
   private boolean reload;
   private Observable<T> diskObservable;
   private Observable<T> networkObservable;

   ...
   ...

   @NonNull
   private Callable<Observable<T>> getReloadCallable() {
       return () -> {
           if (reload)
               return Observable.empty();
           else
               return diskObservable
                   .doOnNext(item -> Timber.d("Loaded %s From Disk on Thread %s",
                       item.getClass(), Thread.currentThread().getName()));
       };
   }

   @NonNull
   private Observable<T> getConnectionObservable() {
       if (utilModel.isConnected())
           return networkObservable
               .doOnNext(item -> Timber.d("Loaded %s From Network on Thread %s",
                   item.getClass(), Thread.currentThread().getName()));
       else
           return Observable.error(new Throwable(Constants.NO_NETWORK));
   }

   @NonNull
   private <V> ObservableTransformer<V, V> applySchedulers() {
       return observable -> observable
           .subscribeOn(Schedulers.io())
           .observeOn(AndroidSchedulers.mainThread());
   }

   @NonNull
   public Observable<T> build() {
       if (diskObservable == null || networkObservable == null)
           throw new IllegalStateException("Network or Disk observable not provided");

       return Observable
               .defer(getReloadCallable())
               .switchIfEmpty(getConnectionObservable())
               .toList()
               .flatMap(items -> diskObservable.toList())
               .flattenAsObservable(items -> items)
               .compose(applySchedulers());
   }
}

 

DiskObservable is a data request to the local database and networkObservable is a data request to the server. The build function decides which one to use and returns a correct observable accordingly. The class object takes a boolean field reload which is used to decide which observable to subscribe. If reload is true, that means the user wants data from the server, hence networkObservable is returned to subscribe. Also switchIfEmpty in the build method checks whether the data fetched using diskObservable is empty, if found empty it switches the observable to the networkObservable to subscribe.

This class object is used for every data access in the app. For example, this is a code snippet of the gettEvents method in EventRepository class.

@Override
public Observable<Event> getEvents(boolean reload) {
   Observable<Event> diskObservable = Observable.defer(() ->
       databaseRepository.getAllItems(Event.class)
   );

   Observable<Event> networkObservable = Observable.defer(() ->
       eventService.getEvents(JWTUtils.getIdentity(getAuthorization()))
           ...
           ...
           .flatMapIterable(events -> events));

   return new AbstractObservableBuilder<Event>(utilModel)
       .reload(reload)
       .withDiskObservable(diskObservable)
       .withNetworkObservable(networkObservable)
       .build();
}

 

Links:
1. Documentation of ReactiveX API
2. Github repository link of RxJava – Reactive Extension for JVM

Automatic Signing and Publishing of Android Apps from Travis

As I discussed about preparing the apps in Play Store for automatic deployment and Google App Signing in previous blogs, in this blog, I’ll talk about how to use Travis Ci to automatically sign and publish the apps using fastlane, as well as how to upload sensitive information like signing keys and publishing JSON to the Open Source repository. This method will be used to publish the following Android Apps:

Current Project Structure

The example project I have used to set up the process has the following structure:

It’s a normal Android Project with some .travis.yml and some additional bash scripts in scripts folder. The update-apk.sh file is standard app build and repo push file found in FOSSASIA projects. The process used to develop it is documented in previous blogs. First, we’ll see how to upload our keys to the repo after encrypting them.

Encrypting keys using Travis

Travis provides a very nice documentation on encrypting files containing sensitive information, but a crucial information is buried below the page. As you’d normally want to upload two things to the repo – the app signing key, and API JSON file for release manager API of Google Play for Fastlane, you can’t do it separately by using standard file encryption command for travis as it will override the previous encrypted file’s secret. In order to do so, you need to create a tarball of all the files that need to be encrypted and encrypt that tar instead. Along with this, before you need to use the file, you’ll have to decrypt in in the travis build and also uncompress it for use.

So, first install Travis CLI tool and login using travis login (You should have right access to the repo and Travis CI in order to encrypt the files for it)

Then add the signing key and fastlane json in the scripts folder. Let’s assume the names of the files are key.jks and fastlane.json

Then, go to scripts folder and run this command to create a tar of these files:

tar cvf secrets.tar fastlane.json key.jks

 

secrets.tar will be created in the folder. Now, run this command to encrypt the file

travis encrypt-file secrets.tar

 

A new file secrets.tar.enc will be created in the folder. Now delete the original files and secrets tar so they do not get added to the repo by mistake. The output log will show the the command for decryption of the file to be added to the .travis.yml file.

Decrypting keys using Travis

But if we add it there, the keys will be decrypted for each commit on each branch. We want it to happen only for master branch as we only require publishing from that branch. So, we’ll create a bash script prep-key.sh for the task with following content

#!/bin/sh
set -e

export DEPLOY_BRANCH=${DEPLOY_BRANCH:-master}

if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_REPO_SLUG" != "iamareebjamal/android-test-fastlane" -o "$TRAVIS_BRANCH" != "$DEPLOY_BRANCH" ]; then
    echo "We decrypt key only for pushes to the master branch and not PRs. So, skip."
    exit 0
fi

openssl aes-256-cbc -K $encrypted_4dd7_key -iv $encrypted_4dd7_iv -in ./scripts/secrets.tar.enc -out ./scripts/secrets.tar -d
tar xvf ./scripts/secrets.tar -C scripts/

 

Of course, you’ll have to change the commands and arguments according to your need and repo. Specially, the decryption command keys ID

The script checks if the repo and branch are correct, and the commit is not of a PR, then decrypts the file and extracts them in appropriate directory

Before signing the app, you’ll need to store the keystore password, alias and key password in Travis Environment Variables. Once you have done that, you can proceed to signing the app. I’ll assume the variable names to be $STORE_PASS, $ALIAS and $KEY_PASS respectively

Signing App

Now, come to the part in upload-apk.sh script where you have the unsigned release app built. Let’s assume its name is app-release-unsigned.apk.Then run this command to sign it

cp app-release-unsigned.apk app-release-unaligned.apk
jarsigner -verbose -tsa http://timestamp.comodoca.com/rfc3161 -sigalg SHA1withRSA -digestalg SHA1 -keystore ../scripts/key.jks -storepass $STORE_PASS -keypass $KEY_PASS app-release-unaligned.apk $ALIAS

 

Then run this command to zipalign the app

${ANDROID_HOME}/build-tools/25.0.2/zipalign -v -p 4 app-release-unaligned.apk app-release.apk

 

Remember that the build tools version should be the same as the one specified in .travis.yml

This will create an apk named app-release.apk

Publishing App

This is the easiest step. First install fastlane using this command

gem install fastlane

 

Then run this command to publish the app to alpha channel on Play Store

fastlane supply --apk app-release.apk --track alpha --json_key ../scripts/fastlane.json --package_name com.iamareebjamal.fastlane

 

You can always configure the arguments according to your need. Also notice that you have to provide the package name for Fastlane to know which app to update. This can also be stored as an environment variable.

This is all for this blog, you can read more about travis CLI, fastlane features and signing process in these links below:

Keeping Order of tickets in Event Wizard in Sync with API on Open Event Frontend

This blog article will illustrate how the various tickets are stored and displayed in order the event organiser decides  on  Open Event Frontend and also, how they are kept in sync with the backend.

First we will take a look at how the user is able to control the order of the tickets using the ticket widget.

{{#each tickets as |ticket index|}}
  {{widgets/forms/ticket-input ticket=ticket
  timezone=data.event.timezone
  canMoveUp=(not-eq index 0)
  canMoveDown=(not-eq ticket.position (dec
  data.event.tickets.length))
  moveTicketUp=(action 'moveTicket' ticket 'up')
  moveTicketDown=(action 'moveTicket' ticket 'down')
  removeTicket=(confirm 'Are you sure you  wish to delete this 
  ticket ?' (action 'removeTicket' ticket))}}
{{/each}}

The canMoveUp and canMoveDown are dynamic properties and are dependent upon the current positions of the tickets in the tickets array.  These properties define whether the up or down arraow or both should be visible alongside the ticket to trigger the moveTicket action.

There is an attribute called position in the ticket model which is responsible for storing the position of the ticket on the backend. Hence it is necessary that the list of the ticket available should always be ordered by position. However, it should be kept in mind, that even if the position attribute of the tickers is changed, it will not actually change the indices of the ticket records in the array fetched from the API. And since we want the ticker order in sync with the backend, i.e. user shouldn’t have to refresh to see the changes in ticket order, we are going to return the tickets via a computed function which sorts them in the required order.

tickets: computed('data.event.tickets.@each.isDeleted', 'data.event.tickets.@each.position', function() {
   return this.get('data.event.tickets').sortBy('position').filterBy('isDeleted', false);
 })

The sortBy method ensures that the tickets are always ordered and this computed property thus watches the position of each of the tickets to look out for any changes. Now we can finally define the moveTicket action to enable modification of position for tickets.

moveTicket(ticket, direction) {
     const index = ticket.get('position');
     const otherTicket = this.get('data.event.tickets').find(otherTicket => otherTicket.get('position') === (direction === 'up' ? (index - 1) : (index + 1)));
     otherTicket.set('position', index);
     ticket.set('position', direction === 'up' ? (index - 1) : (index + 1));
   }

The moveTicket action takes two arguments, ticket and direction. It temporarily stores the position of the current ticket and the position of the ticket which needs to be swapped with the current ticket.Based on the direction the positions are swapped. Since the position of each of the tickets is being watched by the tickets computed array, the change in order becomes apparent immediately.

Now when the User will trigger the save request, the positions of each of the tickets will be updated via a PATCH or POST (if the ticket is new) request.

Also, the positions of all the tickets maybe affected while adding a new ticket or deleting an existing one. In case of a new ticket, the position of the new ticket should be initialised while creating it and it should be below all the other tickets.

addTicket(type, position) {
     const salesStartDateTime = moment();
     const salesEndDateTime = this.get('data.event.startsAt');
     this.get('data.event.tickets').pushObject(this.store.createRecord('ticket', {
       type,
       position,
       salesStartsAt : salesStartDateTime,
       salesEndsAt   : salesEndDateTime
     }));
   }

Deleting a ticket requires updating positions of all the tickets below the deleted ticket. All of the positions need to be shifted one place up.

removeTicket(deleteTicket) {
     const index = deleteTicket.get('position');
     this.get('data.event.tickets').forEach(ticket => {
       if (ticket.get('position') > index) {
         ticket.set('position', ticket.get('position') - 1);
       }
     });
     deleteTicket.deleteRecord();
   }

The tickets whose position is to be updated are filtered by comparison of their position from the position of the deleted ticket.

Resources

Updating the UI of the generator form in Open Event Webapp

  • Add a pop-up menu bar similar to the one shown in Google/Susper6be8d972-72bc-4e12-b27a-219e46608cfc.png
  • Add a version deployment link at the bottom of the page like the one shown in staging.loklak.org.

29072668-b1db-4ef5-8865-c71ef2438433.png

 

  • Implementing the top-bar and the pop-up menu bar

The first task was to introduce a top-bar and a pop-up menu bar in Generator. The top-bar would contain the text Open Event Webapp Generator and an icon button on the right side of it which would show a pop-up menu. The pop-up menu would contain a number of icons which would link to different pages like FOSSASIA blogs and it’s official website, different projects like loklak, SUSI and Eventyay and also to the Webapp Project Readme and issues page.

Creating a top navbar is easy but the pop-up menu is a comparatively tougher. The first step was to gather the gather the small images of the different services. Since this feature had already been implemented in Susper project, we just copied all the icon images from there and copy it into a folder named icons in the open event webapp. Then we create a custom menu div which would hold all the different icons and present it an aesthetic manner. Write the HTML code for the menu and then CSS to decorate and position it! Also, we have to create a click event handler on the pop-up menu button for toggling the menu on and off.

Here is an excerpt of the code. The whole file can be seen here

<div class="custom-navbar">
 <a href='.' class="custom-navtitle">
   <strong>Open Event Webapp Generator</strong> <!-- Navbar Title -->
 </a>
 <div class="custom-menubutton">
   <i class="glyphicon glyphicon-th"></i> <!-- Pop-up Menu button -->
 </div>
 <div class="custom-menu"> <!-- Custom pop-up menu containing different links -->
   <div class="custom-menu-item">
     <a class="custom-icon" href="http://github.com/fossasia/open-event-webapp" target="_blank"><img src="./icons/code.png">
       <p class="custom-title">Code</p></a>
   </div>
   <!-- Code for other links to different projects-->
 </div>
</div>

Here is a screenshot of how the top-bar and the pop-up menu looks!

bb82ba88-4317-46b0-91ec-514499c5cfde.png

  • Adding version deployment info to the bottom

The next task was to add a footer to the page which would contain the version deployment info. The user can click on that link and we can then be taken to the latest version of the code which is currently deployed.

To show the version info, we make use of the Github API. We need to get the hash of the latest commit made on the development branch. We send an API request to the Github requesting for the latest hash and then dynamically add the info and the link received to the footer. The user can then click on that link and will be taken to the latest deployment page of the webapp!

var apiUrl = "https://api.github.com/repos/fossasia/open-event-webapp/git/refs/heads/development";
$.ajax({url: apiUrl, success: function(result){
 var version = result['object']['sha'];
 var versionLink = 'https://github.com/fossasia/open-event-webapp/tree/' + version;
 var deployLink = $('#deploy-link');
 deployLink.attr('href', versionLink);
 deployLink.html(version);
}});

This is how the footer looks after the API Response

44385ad8-e094-490b-8575-a47932aa75c5.png

References:

Implementing Order Statistics API on Tickets Route in Open Event Frontend

The order statistics API endpoints are used to display the statistics related to tickets, orders, and sales. It contains the details about the total number of orders, the total number of tickets sold and the amount of the sales. It also gives the detailed information about the pending, expired, placed and completed orders, tickets, and sales.

This article will illustrate how the order statistics can be displayed using the Order Statistics API in Open Event Frontend. The primary end point of Open Event API with which we are concerned with for statistics is

GET /v1/events/{event_identifier}/order-statistics

First, we need to create a model for the order statistics, which will have the fields corresponding to the API, so we proceed with the ember CLI command:

ember g model order-statistics-tickets

Next, we need to define the model according to the requirements. The model needs to extend the base model class. The code for the model looks like this:

import attr from 'ember-data/attr';
import ModelBase from 'open-event-frontend/models/base';

export default ModelBase.extend({
  orders  : attr(),
  tickets : attr(),
  sales   : attr()
});

As we need to display the statistics related to orders, tickets, and sales so we have their respective variables inside the model which will fetch and store the details from the API.

Now, after creating a model, we need to make an API call to get the details. This can be done using the following:

return this.modelFor('events.view').query('orderStatistics', {});

Since the tickets route is nested inside the event.view route so, first we are getting the model for event.view route and then we’re querying order statistics from the model.

The complete code can be seen here.

Now, we need to call the model inside the template file to display the details. To fetch the total orders we can write like this

{{model.orders.total}}

 

In a similar way, the total sales can be displayed like this.

{{model.sales.total}}

 

And total tickets can be displayed like this

{{model.tickets.total}}

 

If we want to fetch other details like the pending sales or completed orders then the only thing we need to replace is the total attribute. In place of total, we can add any other attribute depending on the requirement. The complete code of the template can be seen here.

The UI for the order statistics on the tickets route looks like this.

Fig. 1: The user interface for displaying the statistics

The complete source code can be seen here.

Resources:

Implementing Pages API in Open Event Frontend

The pages endpoints are used to create static pages which such as about page or any other page that doesn’t need to be updated frequently and only a specific content is to be shown. This article will illustrate how the pages can be added or removed from the /admin/content/pages route using the pages API in Open Event Frontend. The primary end point of Open Event API with which we are concerned with for pages is

GET /v1/pages

First, we need to create a model for the pages, which will have the fields corresponding to the API, so we proceed with the ember CLI command:

ember g model page

Next, we need to define the model according to the requirements. The model needs to extend the base model class. The code for the page model looks like this:

import attr from 'ember-data/attr';
import ModelBase from 'open-event-frontend/models/base';

export default ModelBase.extend({
  name        : attr('string'),
  title       : attr('string'),
  url         : attr('string'),
  description : attr('string'),
  language    : attr('string'),
  index       : attr('number', { defaultValue: 0 }),
  place       : attr('string')
});

As the page will have name, title, url which will tell the URL of the page, the language, the description, index and the place of the page where it has to be which can be either a footer or an event.

The complete code for the model can be seen here.

Now, after creating a model, we need to make an API call to get and post the pages created. This can be done using the following:

return this.get('store').findAll('page');

The above line will check the store and find all the pages which have been cached in and if there is no record found then it will make an API call and cache the records in the store so that when called it can return it immediately.

Since in the case of pages we have multiple options like creating a new page, updating a new page, deleting an existing page etc. For creating and updating the page we have a form which has the fields required by the API to create the page.  The UI of the form looks like this.

Fig. 1: The user interface of the form used to create the page.

Fig. 2: The user interface of the form used to update and delete the already existing page

The code for the above form can be seen here.

Now, if we click the items which are present in the sidebar on the left, it enables us to edit and update the page by displaying the information stored in the form and then the details be later updated on the server by clicking the Update button. If we want to delete the form we can do so using the delete button which first shows a pop up to confirm whether we actually want to delete it or not. The code for displaying the delete confirmation pop up looks like this.

<button class="ui red button" 
{{action (confirm (t 'Are you sure you would like to delete this page?') (action 'deletePage' data))}}>
{{t 'Delete'}}</button>

 

The code to delete the page looks like this

deletePage(data) {
    if (!this.get('isCreate')) {
      data.destroyRecord();
      this.set('isFormOpen', false);
    }
  }

In the above piece of code, we’re checking whether the form is in create mode or update mode and if it’s in create mode then we can destroy the record and then close the form.

The UI for the pop up looks like this.

Fig.3: The user interface for delete confirmation pop up

The code for the entire process of page creation to deletion can be checked here

To conclude, this is how we efficiently do the process of page creation, updating and deletion using the Open-Event-Orga pages API  ensuring that there is no unnecessary API call to fetch the data and no code duplication.

Resources:

Using Android Palette with Glide in Open Event Organizer Android App

Open Event Organizer is an Android Application for the Event Organizers and Entry Managers. The core feature of the App is to scan a QR code from the ticket to validate an attendee’s check in. Other features of the App are to display an overview of sales, ticket management and basic editing in the Event Details. Open Event API Server acts as a backend for this App. The App uses Navigation Drawer for navigation in the App. The side drawer contains menus, event name, event start date and event image in the header. Event name and date is shown just below the event image in a palette. For a better visibility Android Palette is used which extracts prominent colors from images. The App uses Glide to handle image loading hence GlidePalette library is used for palette generation which integrates Android Palette with Glide. I will be talking about the implementation of GlidePalette in the App in this blog.

The App uses Data Binding so the image URLs are directly passed to the XML views in the layouts and the image loading logic is implemented in the BindingAdapter class. The image loading code looks like:

GlideApp
   .with(imageView.getContext())
   .load(Uri.parse(url))
   ...
   .into(imageView);

 

So as to implement palette generation for event detail label, it has to be implemented with the event image loading. GlideApp takes request listener which implements methods on success and failure where palette can be generated using the bitmap loaded. With GlidePalette most of this part is covered in the library itself. It provides GlidePalette class which is a sub class of GlideApp request listener which is passed to the GlideApp using the method listener. In the App, BindingAdapter has a method named bindImageWithPalette which takes a view container, image url, a placeholder drawable and the ids of imageview and palette. The relevant code is:

@BindingAdapter(value = {"paletteImageUrl", "placeholder", "imageId", "paletteId"}, requireAll = false)
public static void bindImageWithPalette(View container, String url, Drawable drawable, int imageId, int paletteId) {
   ImageView imageView = (ImageView) container.findViewById(imageId);
   ViewGroup palette = (ViewGroup) container.findViewById(paletteId);

   if (TextUtils.isEmpty(url)) {
       if (drawable != null)
           imageView.setImageDrawable(drawable);
       palette.setBackgroundColor(container.getResources().getColor(R.color.grey_600));
       for (int i = 0; i < palette.getChildCount(); i++) {
           View child = palette.getChildAt(i);
           if (child instanceof TextView)
               ((TextView) child).setTextColor(Color.WHITE);
       }
       return;
   }
   GlidePalette<Drawable> glidePalette = GlidePalette.with(url)
       .use(GlidePalette.Profile.MUTED)
       .intoBackground(palette)
       .crossfade(true);

   for (int i = 0; i < palette.getChildCount(); i++) {
       View child = palette.getChildAt(i);
       if (child instanceof TextView)
           glidePalette
               .intoTextColor((TextView) child, GlidePalette.Swatch.TITLE_TEXT_COLOR);
   }
   setGlideImage(imageView, url, drawable, null, glidePalette);
}

 

The code is pretty obvious. The method checks passed URL for nullability. If null, it sets the placeholder drawable to the image view and default colors to the text views and the palette. The GlidePalette object is generated using the initializer method with which takes the image URL. The request is passed to the method setGlideImage which loads the image and passes the GlidePalette to the GlideApp as a listener. Accordingly, the palette is generated and the colors are set to the label and text views accordingly. The container view in the XML layout looks like:

<LinearLayout
   android:layout_width="match_parent"
   android:layout_height="wrap_content"
   android:orientation="vertical"
   app:paletteImageUrl="@{ event.largeImageUrl }"
   app:placeholder="@{ @drawable/header }"
   app:imageId="@{ R.id.image }"
   app:paletteId="@{ R.id.eventDetailPalette }">

 

Links:
1. Documentation for Glide Image Loading Library
2. GlidePalette Github Repository
3. Android Palette Official Documentation

Making App Name Configurable for Open Event Organizer App

Open Event Organizer is a client side android application of Open Event API server created for event organizers and entry managers. The application provides a way to configure the app name via environment variable app_name. This allows the user to change the app name just by setting the environment variable app_name to the new name. I will be talking about its implementation in the application in this blog.

Generally, in an android application, the app name is stored as a static string resource and set in the manifest file by referencing to it. In the Organizer application, the app name variable is defined in the app’s gradle file. It is assigned to the value of environment variable app_name and the default value is assigned if the variable is null. The relevant code in the manifest file is:

def app_name = System.getenv('app_name') ?: "eventyay organizer"

app/build.gradle

The default value of app_name is kept, eventyay organizer. This is the app name when the user doesn’t set environment variable app_name. To reference the variable from the gradle file into the manifest, manifestPlaceholders are defined in the gradle’s defaultConfig. It is a map of key value pairs. The relevant code is:

defaultConfig {
   ...
   ...
   manifestPlaceholders = [appName: app_name]
}

app/build.gradle

This makes appName available for use in the app manifest. In the manifest file, app name is assigned to the appName set in the gradle.

<application
   ...
   ...
   android:label="${appName}"

app/src/main/AndroidManifest.xml

By this, the application name is made configurable from the environment variable.

Links:
1. ManifestPlaceholders documentation
2. Stackoverflow answer about getting environment variable in gradle

Implementing Notifications in Open Event Server

In FOSSASIA’s Open Event Server project, along with emails, almost all actions have necessary user notifications as well. So, when a new session is created or a session is accepted by the event organisers, along with the email, a user notification is also sent. Though showing the user notification is mainly implemented in the frontend site but the content to be shown and on which action to show is strictly decided by the server project.

A notification essentially helps an user to get the necessary information while staying in the platform itself and not needing to go to check his/her email for every action he performs. So unlike email which acts as a backup for the informations, notification is more of an instant thing.

The API

The Notifications API is mostly like all other JSON API endpoints in the open event project. However in Notifications API we do not allow any to send a POST request. The admin of the server is able to send a GET a request to view all the notifications that are there in the system while a user can only view his/her notification. As of PATCH we allow only the user to edit his/her notification to mark it as read or not read. Following is the schema for the API:

class NotificationSchema(Schema):
    """
    API Schema for Notification Model
    """

    class Meta:
        """
        Meta class for Notification API schema
        """
        type_ = 'notification'
        self_view = 'v1.notification_detail'
        self_view_kwargs = {'id': '<id>'}
        self_view_many = 'v1.microlocation_list_post'
        inflect = dasherize

    id = fields.Str(dump_only=True)
    title = fields.Str(allow_none=True, dump_only=True)
    message = fields.Str(allow_none=True, dump_only=True)
    received_at = fields.DateTime(dump_only=True)
    accept = fields.Str(allow_none=True, dump_only=True)
    is_read = fields.Boolean()
    user = Relationship(attribute='user',
                        self_view='v1.notification_user',
                        self_view_kwargs={'id': '<id>'},
                        related_view='v1.user_detail',
                        related_view_kwargs={'notification_id': '<id>'},
                        schema='UserSchema',
                        type_='user'
                        )


The main things that are shown in the notification from the frontend are the
title and message. The title is the text that is shown without expanding the entire notification that gives an overview about the message in case you don’t want to read the entire message. The message however provides the entire detail that is associated with the action performed. The user relationship stores which user the particular notification is related with. It is a one-to-one relationship where one notification can only belong to one user. However one user can have multiple notifications. Another important attribute is the is_read attribute. This is the only attribute that is allowed to be changed. By default, when we make an entry in the database, is_read is set to FALSE. Once an user has read the notification, a request is sent from the frontend to change is_read to TRUE.

The different actions for which we send notification are stored in the models/notification.py file as global variables.

USER_CHANGE_EMAIL = "User email"'
NEW_SESSION = 'New Session Proposal'
PASSWORD_CHANGE = 'Change Password'
EVENT_ROLE = 'Event Role Invitation'
TICKET_PURCHASED = 'Ticket(s) Purchased'
TICKET_PURCHASED_ATTENDEE = 'Ticket(s) purchased to Attendee    '
EVENT_EXPORTED = 'Event Exported'
EVENT_EXPORT_FAIL = 'Event Export Failed'
EVENT_IMPORTED = 'Event Imported'

HTML Templates

The notification title and message that is stored in the database and later served via the Notification API is created using some string formatting HTML templates. We firstly import all the global variables that represent the various actions from the notification model. Then we declare a global dict type variable named NOTIFS which stores all title and messages to be stored in the notification table.

NEW_SESSION: {
        'title': u'New session proposal for {event_name}',
        'message': u"""The event <strong>{event_name}</strong> has received
             a new session proposal.<br><br>
            <a href='{link}' class='btn btn-info btn-sm'>View Session</a>""",
        'recipient': 'Organizer',
    },


This is an example of the contents stored inside the dict. For every action, there is a dict with attributes
title, message and recipient. Title contains the brief overview of the entire notification whereas message contains a more vivid description with proper links. Recipient contains the one who receives the notification. So for example in the above code snippet, it is a notification for a new session created. The notification goes to the organizer. The title and message contains named placeholders which are later replaced by particular values using python’s .format() function.

Notification Helper

Notification helper module contains two main parts :-

  1. A parent function which saves the notification to the table related to the user to whom the notification belongs.
  2. Individual notification helper functions that are used by the APIs to save notification after various actions are performed.

Parent Function

def send_notification(user, action, title, message):
    if not current_app.config['TESTING']:
        notification = Notification(user_id=user.id,
                                    title=title,
                                    message=message,
                                    action=action
                                    )
        save_to_db(notification, msg="Notification saved")
        record_activity('notification_event', user=user, action=action, title=title)


send_notification
() is the parent function which takes as parameters user, action, title and message and stores them in the notification table in the database. The user is the one to whom the notification belongs to, action represents the particular action in an API which triggered the notification. Title and message are the contents that are shown in the frontend in the form of a notification. The frontend can implement it as a dropdown notification like facebook or a desktop notification like gitter or whatsapp. After the notification is saved we also update the activity table with the action that a notification has been saved for a user with the following action and title. Later, the Notification API mentioned in the very beginning of the blog uses this data that is being stored now and serves it as a JSON response.

Individual Functions

Apart from this, we have individual functions that uses the parent function to store notifications particular to a particular actions. For example, we have a send_notif_new_session_organizer() function which is used to save notification for all the organizers of an event that a new session has been added to their particular event. This function is called when a POST request is made in the Sessions API and the data is saved successfully. The function is executed for all the organizers of the event for which the session has been created.

def send_notif_new_session_organizer(user, event_name, link):
    message_settings = MessageSettings.query.filter_by(action=NEW_SESSION).first()
    if not message_settings or message_settings.notification_status == 1:
        notif = NOTIFS[NEW_SESSION]
        action = NEW_SESSION
        title = notif['title'].format(event_name=event_name)
        message = notif['message'].format(event_name=event_name, link=link)

        send_notification(user, action, title, message)


In the above function, we take in 3 parameters, user, event_name and link. The value of the user parameter is used to link the notification to that particular user. Event_name and link are used in the title and message of the notification that is saved in the database. Firstly in the function we check if there is certain message setting which tells that the user doesn’t want to receive notifications related to new sessions being created. If not, we proceed. We get the title and message strings from the NOTIFS dict from the system_notification.py file.

After that, using string formatting we get the actual message. For example,

u'New session proposal for {event_name}'.format(‘FOSSASIA’)

would give us a resulting string of the form:

u'New session proposal for FOSSASIA'

After this, we use this variables and send them as parameters to the send_notification() parent function to save the notification properly.

 

Reference:

Implement Email in Open Event Server

In FOSSASIA’s Open Event Server project, we send out emails when various different actions are performed using the API. For example, when a new user is created, he/she receives an email welcoming him to the server as well as an email verification email. Users get role invites from event organisers in the form of emails, when someone buys a ticket he/she gets a PDF link to the ticket as email. So as you can understand all the important informations that are necessary to be notified to the user are sent as an email to the user and sometimes to the organizer as well.

In FOSSASIA, we use sendgrid’s API or an SMTP server depending on the admin settings for sending emails. You can read more about how we use sendgrid’s API to send emails in FOSSASIA here. Now let’s dive into the modules that we have for sending the emails. The three main parts in the entire email sending are:

  1. Model – Storing the Various Actions
  2. Templates – Storing the HTML templates for the emails
  3. Email Functions – Individual functions for various different actions

Let’s go through each of these modules one by one.

Model

USER_REGISTER = 'User Registration'
USER_CONFIRM = 'User Confirmation'
USER_CHANGE_EMAIL = "User email"
INVITE_PAPERS = 'Invitation For Papers'
NEXT_EVENT = 'Next Event'
NEW_SESSION = 'New Session Proposal'
PASSWORD_RESET = 'Reset Password'
PASSWORD_CHANGE = 'Change Password'
EVENT_ROLE = 'Event Role Invitation'
SESSION_ACCEPT_REJECT = 'Session Accept or Reject'
SESSION_SCHEDULE = 'Session Schedule Change'
EVENT_PUBLISH = 'Event Published'
AFTER_EVENT = 'After Event'
USER_REGISTER_WITH_PASSWORD = 'User Registration during Payment'
TICKET_PURCHASED = 'Ticket(s) Purchased'


In the Model file, named as
mail.py, we firstly declare the various different actions for which we send the emails out. These actions are globally used as the keys in the other modules of the email sending service. Here, we define global variables with the name of the action as strings in them. These are all constant variables, which means that there value remains throughout and never changes. For example, USER_REGISTER has the value ‘User Registration’, which essentially means that anything related to the USER_REGISTER key is executed when the User Registration action occurs. Or in other words, whenever an user registers into the system by signing up or creating a new user through the API, he/she receives the corresponding emails.
Apart from this, we have the model class which defines a table in the database. We use this model class to store the actions performed while sending emails in the database. So we store the action, the time at which the email was sent, the recipient and the sender. That way we have a record about all the emails that were sent out via our server.

class Mail(db.Model):
    __tablename__ = 'mails'
    id = db.Column(db.Integer, primary_key=True)
    recipient = db.Column(db.String)
    time = db.Column(db.DateTime(timezone=True))
    action = db.Column(db.String)
    subject = db.Column(db.String)
    message = db.Column(db.String)

    def __init__(self, recipient=None, time=None, action=None, subject=None,
                 message=None):
        self.recipient = recipient
        self.time = time
        if self.time is None:
            self.time = datetime.now(pytz.utc)
        self.action = action
        self.subject = subject
        self.message = message

    def __repr__(self):
        return '<Mail %r to %r>' % (self.id, self.recipient)

    def __str__(self):
        return unicode(self).encode('utf-8')

    def __unicode__(self):
        return 'Mail %r by %r' % (self.id, self.recipient,)


The table name in which all the information is stored is named as mails. It stores the recipient, the time at which the email is sent (timezone aware), the action which initiated the email sending, the subject of the email and the entire html body of the email. In case a datetime value is sent, we use that, else we use the current time in the time field.

HTML Templates

We store the html templates in the form of key value pairs in a file called system_mails.py inside the helpers module of the API. Inside the system_mails, we have a global dict variable named MAILS as shown below.

MAILS = {
    EVENT_PUBLISH: {
        'recipient': 'Organizer, Speaker',
        'subject': u'{event_name} is Live',
        'message': (
            u"Hi {email}<br/>" +
            u"Event, {event_name}, is up and running and ready for action. Go ahead and check it out." +
            u"<br/> Visit this link to view it: {link}"
        )
    },
    INVITE_PAPERS: {
        'recipient': 'Speaker',
        'subject': u'Invitation to Submit Papers for {event_name}',
        'message': (
            u"Hi {email}<br/>" +
            u"You are invited to submit papers for event: {event_name}" +
            u"<br/> Visit this link to fill up details: {link}"
        )
    },
    SESSION_ACCEPT_REJECT: {
        'recipient': 'Speaker',
        'subject': u'Session {session_name} has been {acceptance}',
        'message': (
            u"Hi {email},<br/>" +
            u"The session <strong>{session_name}</strong> has been <strong>{acceptance}</strong> by the organizer. " +
            u"<br/> Visit this link to view the session: {link}"
        )
    },
    SESSION_SCHEDULE: {
        'recipient': 'Organizer, Speaker',
        'subject': u'Schedule for Session {session_name} has been changed',
        'message': (
            u"Hi {email},<br/>" +
            u"The schedule for session <strong>{session_name}</strong> has been changed. " +
            u"<br/> Visit this link to view the session: {link}"
        )
    },


Inside the MAILS dict, we have key-value pairs, where in keys we use the global variables from the Model to define the action related to the email template. In the value, we again have 3 different key-value pairs – recipient, subject and message. The recipient defines the group who should receive this email, the subject goes into the subject part of the email while message forms the body for the email. For subject and message we use unicode strings with named placeholders that are used later for formatting using python’s
.format() function.

Email Functions

This is the most important part of the entire email sending system since this is the place where the entire email sending functionality is implemented using the above two modules. We have all these functions inside a single file namely mail.py inside the helpers module of the API. Firstly, we import two things in this file – The global dict variable MAILS defined in the template file above, and the various global action variables defined in the model. There is one main module which is used by every other individual modules for sending the emails defined as send_email(to, action, subject, html). This function takes as parameters the email to which the email is to be sent, the subject string, the html body string along with the action to store it in the database.

Firstly we ensure that the email address for the recipient is present and isn’t an empty string. After we have ensured this, we retrieve the email service as set in the admin settings. It can either be “smtp” or “sendgrid”. The email address for the sender has different formatting depending on the email service we are using. While sendgrid uses just the email say for example “[email protected]”, smtp uses a format  a little different like this: Medozonuo Suohu<[email protected]>. So we set that as well in the email_from variable.

def send_email(to, action, subject, html):
    """
    Sends email and records it in DB
    """
    if not string_empty(to):
        email_service = get_settings()['email_service']
        email_from_name = get_settings()['email_from_name']
        if email_service == 'smtp':
            email_from = email_from_name + '<' + get_settings()['email_from'] + '>'
        else:
            email_from = get_settings()['email_from']
        payload = {
            'to': to,
            'from': email_from,
            'subject': subject,
            'html': html
        }

        if not current_app.config['TESTING']:
            if email_service == 'smtp':
                smtp_encryption = get_settings()['smtp_encryption']
                if smtp_encryption == 'tls':
                    smtp_encryption = 'required'
                elif smtp_encryption == 'ssl':
                    smtp_encryption = 'ssl'
                elif smtp_encryption == 'tls_optional':
                    smtp_encryption = 'optional'
                else:
                    smtp_encryption = 'none'

                config = {
                    'host': get_settings()['smtp_host'],
                    'username': get_settings()['smtp_username'],
                    'password': get_settings()['smtp_password'],
                    'encryption': smtp_encryption,
                    'port': get_settings()['smtp_port'],
                }

                from tasks import send_mail_via_smtp_task
                send_mail_via_smtp_task.delay(config, payload)


After this we create the payload containing the email address for the recipient, the email address of the sender, the subject of the email and the html body of the email.
For unittesting and any other testing we avoid email sending since that is really not required in the flow. So we check that the current app is not configured to run in a testing environment. After that we have two different implementation depending on the email service used.

SMTP

There are 3 kind of possible encryptions for the email that can be used with smtp server – tls, ssl and optional. We determine this based on the admin settings again. Also, from the admin settings we collect the host, username, password and port for the smtp server.

After this we start a celery task for sending the email. Since email sending to a number of clients can be time consuming so we do it using the celery queueing service without disturbing the main workflow of the entire system.

@celery.task(name='send.email.post.smtp')
def send_mail_via_smtp_task(config, payload):
    mailer_config = {
        'transport': {
            'use': 'smtp',
            'host': config['host'],
            'username': config['username'],
            'password': config['password'],
            'tls': config['encryption'],
            'port': config['port']
        }
    }

    mailer = Mailer(mailer_config)
    mailer.start()
    message = Message(author=payload['from'], to=payload['to'])
    message.subject = payload['subject']
    message.plain = strip_tags(payload['html'])
    message.rich = payload['html']
    mailer.send(message)
    mailer.stop()

Inside the celery task, we use the Mailer and Message classes from the marrow module of python. We configure the Mailer according to the various settings received from the admin and then use the payload to send the email.

Sendgrid

For sending email using the sendgrid API, we need to set the Bearer key which is used for authenticating the email service. This key is also defined in the admin settings. After we have set the Bearer key as the authorization header, we again initiate the celery task corresponding to the sendgrid email sending service.

@celery.task(name='send.email.post')
def send_email_task(payload, headers):
    requests.post(
        "https://api.sendgrid.com/api/mail.send.json",
        data=payload,
        headers=headers
    )


For sending the email service, all we need to do is make a POST request to the api endpoint “
https://api.sendgrid.com/api/mail.send.json” with the headers which contains the Bearer Key and the data which contains the payload containing all the information related to the recipient, sender, subject of email and the body of the email.

Apart from these, this module implements all the individual functions that are called based on the various functions that occur. For example, let’s look into the email sending function in case a new session is created.

def send_email_new_session(email, event_name, link):
    """email for new session"""
    send_email(
        to=email,
        action=NEW_SESSION,
        subject=MAILS[NEW_SESSION]['subject'].format(
            event_name=event_name
        ),
        html=MAILS[NEW_SESSION]['message'].format(
            email=email,
            event_name=event_name,
            link=link
        )
    )


This function is called inside the Sessions API, for every speaker of the session as well as for every organizer of the event to which the session is submitted. Inside this function, we use the
send_email().  But firstly we need to create the subject of the email and the message body of the email using the templates and by replacing placeholders by actual value using python formatting. MAILS[NEW_SESSION] returns a unicode string: u’New session proposal for {event_name}’ . So what we do is use the .format() function to replace {event_name} by the actual event_name received as parameter. So it is equivalent to doing something like:

u'New session proposal for {event_name}'.format(‘FOSSASIA’)

which would give us a resulting string of the form:

u'New session proposal for FOSSASIA'

Similarly, we create the html message body using the templates and the parameters received. After this is done, we make a function call to send_email()  which then sends the final email.

References: