Open Event Frontend – Settings Service

This blog illustrates how the settings of a particular user are obtained in the Open Event Frontend web app. To access the settings of the user a service has been created which fetches the settings from the endpoint provided by Open Event Server.

Let’s see why a special service was created for this.

Problem

In the first step of the event creation wizard, the user has the option to link Paypal or Stripe to accept payments. The option to accept payment through Paypal or Stripe was shown to the user without checking if it was enabled by the admin in his settings. To solve this problem, we needed to access the settings of the admin and check for the required conditions. But since queryRecord() returned a promise we had to re-render the page for the effect to show which resulted in this code:

canAcceptPayPal: computed('data.event.paymentCurrency', function() {     this.get('store').queryRecord('setting', {}) .then(setting => { this.set('canAcceptPayPal', (setting.paypalSandboxUsername || setting.paypalLiveUsername) && find(paymentCurrencies, ['code', this.get('data.event.paymentCurrency')]).paypal); this.rerender(); });

This code was setting a computed property inside it and then re-rendering which is bad programming and can result in weird bugs.

Solution

The above problem was solved by creating a service for settings. This made sense as settings would be required at other places as well. The file was called settings.js and was placed in the services folder. Let me walk you through its code.

  • Extend the default Service provided by Ember.js and initialize store, session, authManager and _lastPromise.
import Service, { inject as service } from '@ember/service';
import { observer } from '@ember/object';

export default Service.extend({

 store       : service(),
 session     : service(),
 authManager : service(),

_lastPromise: Promise.resolve(),
  • The main method which fetches results from the server is called _loadSettings(). It is an async method. It queries setting from the server and then iterates through every attribute of the setting model and stores the corresponding value from the fetched result.
/**
* Load the settings from the API and set the attributes as properties on the service
*
* @return {Promise<void>}
* @private
*/
async _loadSettings() {
 const settingsModel = await this.get('store').queryRecord('setting', {});
 this.get('store').modelFor('setting').eachAttribute(attributeName => {
   this.set(attributeName, settingsModel.get(attributeName));
 });
},
  • The initialization of the settings service is handled by initialize(). This method returns a promise.
/**
* Initialize the settings service
* @return {*|Promise<void>}
*/
initialize() {
 const promise = this._loadSettings();
 this.set('_lastPromise', promise);
 return promise;
}
  • _authenticationObserver observes for changes in authentication changes and reloads the settings as required.
/**
* Reload settings when the authentication state changes.
*/
_authenticationObserver: observer('session.isAuthenticated', function() {
 this.get('_lastPromise')
   .then(() => this.set('_lastPromise', this._loadSettings()))
   .catch(() => this.set('_lastPromise', this._loadSettings()));
}),

The service we created can be directly used in the app to fetch the settings for the user. To solve the Paypal and Stripe payment problem described above, we use it as follows:

canAcceptPayPal: computed('data.event.paymentCurrency', 'settings.paypalSandboxUsername', 'settings.paypalLiveUsername', function() {
 return (this.get('settings.paypalSandboxUsername') || this.get('settings.paypalLiveUsername')) && find(paymentCurrencies, ['code', this.get('data.event.paymentCurrency')]).paypal;
}),

canAcceptStripe: computed('data.event.paymentCurrency', 'settings.stripeClientId', function() {
 return this.get('settings.stripeClientId') && find(paymentCurrencies, ['code', this.get('data.event.paymentCurrency')]).stripe;
}),

Thus, there is no need to re-render the page and dangerously set the property inside its computed method.

References

Continue ReadingOpen Event Frontend – Settings Service

Open Event Server – Export Speakers as CSV File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he/she makes his event public, others can view it and buy tickets if interested.

The organizer can see all the speakers in a very detailed view in the event management dashboard. He can see the statuses of all the speakers. The possible statuses are pending, accepted and rejected. He/she can take actions such as editing/viewing speakers.

If the organizer wants to download the list of all the speakers as a CSV file, he or she can do it very easily by simply clicking on the Export As CSV button in the top right-hand corner.

Let us see how this is done on the server.

Server side – generating the Speakers CSV file

Here we will be using the csv package provided by python for writing the csv file.

import csv
  • We define a method export_speakers_csv which takes the speakers to be exported as a CSV file as the argument.
  • Next, we define the headers of the CSV file. It is the first row of the CSV file.
def export_speakers_csv(speakers):
   headers = ['Speaker Name', 'Speaker Email', 'Speaker Session(s)',
              'Speaker Mobile', 'Speaker Bio', 'Speaker Organisation', 'Speaker Position']
  • A list is defined called rows. This contains the rows of the CSV file. As mentioned earlier, headers is the first row.
rows = [headers]
  • We iterate over each speaker in speakers and form a row for that speaker by separating the values of each of the columns by a comma. Here, every row is one speaker.
  • As a speaker can contain multiple sessions we iterate over each session for that particular speaker and append each session to a string. ‘;’ is used as a delimiter. This string is then added to the row. We also include the state of the session – accepted, rejected, confirmed.
  • The newly formed row is added to the rows list.
for speaker in speakers:
   column = [speaker.name if speaker.name else '', speaker.email if speaker.email else '']
   if speaker.sessions:
       session_details = ''
       for session in speaker.sessions:
           if not session.deleted_at:
               session_details += session.title + ' (' + session.state + '); '
       column.append(session_details[:-2])
   else:
       column.append('')
   column.append(speaker.mobile if speaker.mobile else '')
   column.append(speaker.short_biography if speaker.short_biography else '')
   column.append(speaker.organisation if speaker.organisation else '')
   column.append(speaker.position if speaker.position else '')
   rows.append(column)
  • rows contains the contents of the CSV file and hence it is returned.
return rows
  • We iterate over each item of rows and write it to the CSV file using the methods provided by the csv package.
with open(file_path, "w") as temp_file:
   writer = csv.writer(temp_file)
   from app.api.helpers.csv_jobs_util import export_speakers_csv
   content = export_speakers_csv(speakers)
   for row in content:
       writer.writerow(row)

Obtaining the Speakers CSV file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/speakers/csv

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the speakers of the event as a CSV file. It returns the URL of the task to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above-returned URL and check the status of his/her Celery task. If the task completed successfully he/she will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above-mentioned URL.

Resources

Continue ReadingOpen Event Server – Export Speakers as CSV File

Open Event Server – Export Sessions as CSV File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he/she makes his event public, others can view it and buy tickets if interested.

The organizer can see all the sessions in a very detailed view in the event management dashboard. He can see the statuses of all the sessions. The possible statuses are pending, accepted, confirmed and rejected. He/she can take actions such as accepting/rejecting the sessions.

If the organizer wants to download the list of all the sessions as a CSV file, he or she can do it very easily by simply clicking on the Export As CSV button in the top right-hand corner.

Let us see how this is done on the server.

Server side – generating the Sessions CSV file

Here we will be using the csv package provided by python for writing the csv file.

import csv
  • We define a method export_sessions_csv which takes the sessions to be exported as a CSV file as the argument.
  • Next, we define the headers of the CSV file. It is the first row of the CSV file.
def export_sessions_csv(sessions):
   headers = ['Session Title', 'Session Speakers',
              'Session Track', 'Session Abstract', 'Created At', 'Email Sent']
  • A list is defined called rows. This contains the rows of the CSV file. As mentioned earlier, headers is the first row.
rows = [headers]
  • We iterate over each session in sessions and form a row for that session by separating the values of each of the columns by a comma. Here, every row is one session.
  • As a session can contain multiple speakers we iterate over each speaker for that particular session and append each speaker to a string. ‘;’ is used as a delimiter. This string is then added to the row.
  • The newly formed row is added to the rows list.
for session in sessions:
   if not session.deleted_at:
       column = [session.title + ' (' + session.state + ')' if session.title else '']
       if session.speakers:
           in_session = ''
           for speaker in session.speakers:
               if speaker.name:
                   in_session += (speaker.name + '; ')
           column.append(in_session[:-2])
       else:
           column.append('')
       column.append(session.track.name if session.track and session.track.name else '')
       column.append(strip_tags(session.short_abstract) if session.short_abstract else '')
       column.append(session.created_at if session.created_at else '')
       column.append('Yes' if session.is_mail_sent else 'No')
       rows.append(column)
  • rows contains the contents of the CSV file and hence it is returned.
return rows
  • We iterate over each item of rows and write it to the CSV file using the methods provided by the csv package.
writer = csv.writer(temp_file)
from app.api.helpers.csv_jobs_util import export_sessions_csv
content = export_sessions_csv(sessions)
for row in content:
   writer.writerow(row)

Obtaining the Sessions CSV file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/sessions/csv

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the sessions of the event as a CSV file. It returns the URL of the task to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above-returned URL and check the status of his/her Celery task. If the task completed successfully he/she will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above-mentioned URL.

Resources

Continue ReadingOpen Event Server – Export Sessions as CSV File

Setting an Event Favorite in Open Event Android

The favorite events feature in Open Event Android allows any user to favorite an event and those events can are available in the favorite events fragment which is easily accessible from the bottom navigation bar. This blog article will walk you through on how favorite works in Open Event Android. There are a couple of different ways to do it, in Open Event Android we are keeping track of favorite events using a favorite Boolean stored along with the information of the event ie we have added a favorite boolean field inside the event entity class. However, this method has its own pros and cons. The big plus point of this method is the fact that we can simply check the favorite field of the event object to check if that particular event is favorite or not. But since we have set onConflict strategy as replace in the event insert DAO method (shown below)

 @Insert(onConflict = REPLACE)
   fun insertEvents(events: List<Event>)

This leads to a big problem when the same event is fetched or is present in the response while inserting the favorite field will be set to default value (false) and the favorite information would be lost. Hence extra care needs to be taken while updating events.

Given following is the event model class for serializing / deserializing JSON responses. Note the extra field favorite added here which helps us to provide user favorite feature locally.

@Type(“event”)
@JsonNaming(PropertyNamingStrategy.KebabCaseStrategy::class)
@Entity
data class Event(
      @Id(LongIdHandler::class)
      @PrimaryKey
      val id: Long,
      val name: String,
      val identifier: String,
      val isMapShown: Boolean = false,
      val favorite: Boolean = false
)

Since we added a new field to the Event model class, to access the favorite events we will have to create DAO methods. To fetch all the favorite events we will have to create a Query method which returns all the events wherever favorite property is set. This can be done using a simple select statement which looks for events with favorite boolean set ie. 1, the method implementation is shown below.

@Query(“SELECT * from Event WHERE favorite = 1”)
   fun getFavoriteEvents(): Flowable<List<Event>>

To set an event favorite we will have to add one more method to the EventDao. We can create a generalized method which sets according to the boolean passed as a parameter. The function implementation is shown below, setFavorite takes in EventId, Id of the event which has to be updated and the boolean favorite which is what the new value for the favorite field is. setFavorite makes an SQL update query and matches for EventId and sets favorite for that event.

@Query(“UPDATE Event SET favorite = :favorite WHERE id = :eventId”)
   fun setFavorite(eventId: Long, favorite: Boolean)

In Open Event Android we use the service class which can be used to expose the DAO methods to the rest of the project. The idea behind creating service layer is the separation of concerns this service method is then called from the view model function (following the MVVM approach). Given following is the implementation of the get and set favorite methods.

fun getFavoriteEvents(): Flowable<List<Event>> {
       return eventDao.getFavoriteEvents()
   }
fun setFavorite(eventId: Long, favourite: Boolean): Completable {
       return Completable.fromAction {
           eventDao.setFavorite(eventId, favourite)
       }
   }

The method getFavoriteEvents calls the DAO method to fetch the favorite evens and returns a flowable list of favorite events and setFavorite method generates a completable from the DAO action to favorite any event.

EventViewModel uses these methods and specifies where the observables should be observedOn and subscribedOn also defines what should happen when some error occurs. Given below is implementation for the two methods.

fun setFavorite(eventId: Long, favourite: Boolean) {
       compositeDisposable.add(eventService.setFavorite(eventId, favourite)
               .subscribeOn(Schedulers.io())
               .observeOn(AndroidSchedulers.mainThread())
               .subscribe({
                   Timber.d(“Success”)
               }, {
                   Timber.e(it, “Error”)
                   error.value = “Error”
               }))
   }

For every event, we have a floating action bar to toggle the favorite state of the same whenever user clicks on the FAB it checks the current state of the event and calls the view model function setFavorite passing the Event Id of the clicked event and the negation of current event.favorite state.

val favouriteFabClickListener = object : FavoriteFabListener {
           override fun onClick(eventId: Long, isFavourite: Boolean) {
               eventsViewModel.setFavorite(eventId, !isFavourite)
           }
       }

What we discussed in this blog post works well unless we are not inserting the events with the same event id, this will cause replace on the stored events and current favorite information will be lost. To overcome this limitation we follow the following procedure

  • Fetch event id’s of all the events returned by the API response
  • Use the DAO method to find out the id’s of favorite events out of these
  • Set the favorite in the new events where IDs are in the ids returned in the second point
  • Insert the events into the database
  • Return the events from the database with ID’s as of point one

References

Continue ReadingSetting an Event Favorite in Open Event Android

Implementing Endpoint to Resend Email Verification

Earlier, when a user registered via Open Event Frontend, s/he received a verification link via email to confirm their account. However, this was not enough in the long-term. If the confirmation link expired, or for some reasons the verification mail got deleted on the user side, there was no functionality to resend the verification email, which prevented the user from getting fully registered. Although the front-end already showed the option to resend the verification link, there was no support from the server to do that, yet.

So it was decided that a separate endpoint should be implemented to allow re-sending the verification link to a user. /resend-verification-email was an endpoint that would fit this action. So we decided to go with it and create a route in `auth.py` file, which was the appropriate place for this feature to reside. First step was to do the necessary imports and then definition:

from app.api.helpers.mail import send_email_confirmation
from app.models.mail import USER_REGISTER_WITH_PASSWORD
...
...
@auth_routes.route('/resend-verification-email', methods=['POST'])
def resend_verification_email():
...

Now we safely fetch the email mentioned in the request and then search the database for the user corresponding to that email:

def resend_verification_email():
    try:
        email = request.json['data']['email']
    except TypeError:
        return BadRequestError({'source': ''}, 'Bad Request Error').respond()

    try:
        user = User.query.filter_by(email=email).one()
    except NoResultFound:
        return UnprocessableEntityError(
{'source': ''}, 'User with email: ' + email + ' not found.').respond()
    else:

    ...

Once a user has been identified in the database, we proceed further and create an essentially unique hash for the user verification. This hash is in turn used to generate a verification link that is then ready to be sent via email to the user:

else:
    serializer = get_serializer()
    hash_ = str(base64.b64encode(str(serializer.dumps(
[user.email, str_generator()])).encode()), 'utf-8')
    link = make_frontend_url(
'/email/verify'.format(id=user.id), {'token': hash_})

Finally, the email is sent:

send_email_with_action(
user, USER_REGISTER_WITH_PASSWORD,
app_name=get_settings()['app_name'], email=user.email)
    if not send_email_confirmation(user.email, link):
        return make_response(jsonify(message="Some error occured"), 500)
    return make_response(jsonify(message="Verification email resent"), 200)

But this was not enough. When the endpoint was tested, it was found that actual emails were not being delivered, even after correctly configuring the email settings locally. So, after a bit of debugging, it was found that the settings, which were using Sendgrid to send emails, were using a deprecated Sendgrid API endpoint. A separate email function is used to send emails via Sendgrid and it contained an old endpoint that was no longer recommended by Sendgrid:

@celery.task(name='send.email.post')
def send_email_task(payload, headers):
   requests.post(
       "https://api.sendgrid.com/api/mail.send.json",
       data=payload,
       headers=headers
   )

The new endpoint, as per Sendgrid’s documentation, is:

https://api.sendgrid.com/v3/mail/send

But this was not the only change required. Sendgrid had also modified the structure of requests they accepted, and the new structure was different from the existing one that was used in the server. Following is the new structure:

'{"personalizations": [{"to": [{"email": "example@example.com"}]}],"from": {"email": "example@example.com"},"subject": "Hello, World!","content": [{"type": "text/plain", "value": "Heya!"}]}'

The header structure was also changed, so the structure in the server was also updated to

headers = {
"Authorization": ("Bearer " + key),
"Content-Type": "application/json"
}

The Sendgrid function (which is executed as a Celery task) was modified as follows, to incorporate the changes in the API endpoint and structure:

import json
...
@celery.task(name='send.email.post')
def send_email_task(payload, headers):
    data = {"personalizations": [{"to": []}]}
    data["personalizations"][0]["to"].append({"email": payload["to"]})
    data["from"] = {"email": payload["from"]}
    data["subject"] = payload["subject"]
    data["content"] = [{"type": "text/html", "value": payload["html"]}]
    requests.post(
        "https://api.sendgrid.com/v3/mail/send",
        data=json.dumps(data),
        headers=headers,
        verify=False  # doesn't work with verification in celery context
    )

 

As can be seen, there is a bug that doesn’t allow SSL verification within the celery context. However, the verification is successful when the functionality is executed independent of the celery context. But now email sending via Sendgrid actually works, which makes our verification resend endpoint functional:Screen Shot 2018-08-10 at 10.04.12 PM.pngEmail is received successfully by the recipient:

Screen Shot 2018-08-10 at 10.04.30 PM.png

Thus, a working email verification endpoint is implemented, which can be easily integrated in the frontend.


Resources:

Continue ReadingImplementing Endpoint to Resend Email Verification

Creating an apk in the apk branch using Travis CI

All Android apps in FOSSASIA have an apk branch where after a pull request is merged a new apk gets created so that even people who do not know how to setup the app locally can access it. Let’s see how it is done.

Lets get started

Create a bash file in the scripts folder and name it upload-apk.sh

Here is all the code that you require to create an apk in the apk branch.

First thing that we need to do is setup git. So we’ll set the user name and email just like we do when we setup git for the first time in our own systems.

git config --global user.name "Travis CI"
git config --global user.email "noreply+travis@fossasia.org"

 

Next we will clone the apk branch of the repository and copy all the files that we need ie the apk and the JSON files.

git clone --quiet --branch=apk https://fossasia:$GITHUB_API_KEY@github.com/fossasia/open-event-android apk > /dev/null
cd apk
\cp -r ../app/build/outputs/apk/*/**.apk .
\cp -r ../app/build/outputs/apk/debug/output.json debug-output.json
\cp -r ../app/build/outputs/apk/release/output.json release-output.json

 

Next we will create a new branch that contains only the latest apk. After that we will add the APK and then commit all those changes. You can see that the current date and time are printed out in the commit message.

git checkout --orphan temporary

git add --all .
git commit -am "[Auto] Update Test Apk ($(date +%Y-%m-%d.%H:%M:%S))"

 

We will delete the current apk branch and then rename the current branch to apk. In the end we will force push to origin since histories are unrelated.

git branch -D apk
git branch -m apk

git push origin apk --force --quiet > /dev/null

 

If you have already integrated Travis CI in your repository then you just need to add this line your travis.yml file.

after_success:
- bash scripts/update-apk.sh

 

Now every time a PR gets merged in the repository a new apk file is created in the apk branch of your repository.

Resources

  1. Travis CLI – https://github.com/travis-ci/travis.rb#readme
  2. Travis official documentation – https://travis-ci.org/
Continue ReadingCreating an apk in the apk branch using Travis CI

Adding Check-in Attributes to Tickets

Recently, it was decided by the Open Event Orga App team that the event ticket API response from Open Event Server should have two additional attributes for specifying event check-in access. At first sight, it seemed that adding these options will only require changes in the orga app, but it turned out that the entire Ticket API from the server will need this addition.

Implementing these attributes turned out to be quite straightforward. Specifically, the fields to be added were boolean is_checkin_restricted and auto_checkin_enabled. By default, checkin is not automatic and is restricted. Therefore, the default values for these fields were chosen to be True and False respectively. To add them, the ticket model file was changed first – due to the addition of these two columns:

class Ticket(SoftDeletionModel):
    ...
    is_checkin_restricted = db.Column(db.Boolean)  # <--
    auto_checkin_enabled = db.Column(db.Boolean)  # <--
    ...
    def __init__(self,
        name=None,
        event_id=None,
        ...
        is_checkin_restricted=True,
        auto_checkin_enabled=False):

        self.name = name
        ...
        self.is_checkin_restricted = is_checkin_restricted
        self.auto_checkin_enabled = auto_checkin_enabled
        ...

Since the ticket database model was updated, a migration had to be performed. Following shell commands (at the open event server project root) did the migration and database update and a migration file was then generated:

$ python manage.py db migrate
$ python manage.py db upgrade

Here’s the generated migration file:

from alembic import op
import sqlalchemy as sa

revision = '6440077182f0'
down_revision = 'eaa029ebb260'

def upgrade():
    op.add_column('tickets', sa.Column('auto_checkin_enabled', sa.Boolean(), nullable=True))
    op.add_column('tickets', sa.Column('is_checkin_restricted', sa.Boolean(), nullable=True))

def downgrade():
    op.drop_column('tickets', 'is_checkin_restricted')
    op.drop_column('tickets', 'auto_checkin_enabled')

The next code change was required in the ticket API schema. The change was essentially the same as the one added in the model file – just these 2 new fields were added:

class TicketSchemaPublic(SoftDeletionSchema):
    ...
    id = fields.Str(dump_only=True)
    name = fields.Str(required=True)
    ...
    is_checkin_restricted = fields.Boolean(default=True)  # <--
    auto_checkin_enabled = fields.Boolean(default=False)  # <--
    event = Relationship(attribute='event',
    self_view='v1.ticket_event',
    self_view_kwargs={'id': '<id>'},
    related_view='v1.event_detail',
    related_view_kwargs={'ticket_id': '<id>'},
    schema='EventSchemaPublic',
    type_='event')
    ...

Now all that remained were changes in the API documentation, which were made accordingly. This completed the addition of these two checkin attributes in the ticket API, and eventually made way to the orga app. And, these can be requested as usual by the front-end and user app as well.


Resources and Links:

Continue ReadingAdding Check-in Attributes to Tickets

Adding Port Specification for Static File URLs in Open Event Server

Until now, static files stored locally on Open Event server did not have port specification in their URLs. This opened the door for problems while consuming local APIs. This would have created inconsistencies, if two server processes were being served on the same machine but at different ports. In this blog post, I will explain my approach towards solving this problem, and describe code snippets to demonstrate the changes I made in the Open Event Server codebase.

The first part in this process involved finding the source of the bug. For this, my open-source integrated development environment, Microsoft Visual Studio Code turned out to be especially useful. It allowed me to jump from function calls to function definitions quickly:

Screen Shot 2018-08-10 at 12.29.01 PM

I started at events.py and jumped all the way to storage.py, where I finally found out the source of this bug, in upload_local() function:

def upload_local(uploaded_file, key, **kwargs):
    """
    Uploads file locally. Base dir - static/media/
    """
    filename = secure_filename(uploaded_file.filename)
    file_relative_path = 'static/media/' + key + '/' + generate_hash(key) + '/' + filename
    file_path = app.config['BASE_DIR'] + '/' + file_relative_path
    dir_path = file_path.rsplit('/', 1)[0]
    # delete current
    try:
        rmtree(dir_path)
    except OSError:
        pass
    # create dirs
    if not os.path.isdir(dir_path):
        os.makedirs(dir_path)
        uploaded_file.save(file_path)
        file_relative_path = '/' + file_relative_path
    if get_settings()['static_domain']:
        return get_settings()['static_domain'] + \
    file_relative_path.replace('/static', '')
    url = urlparse(request.url)
    return url.scheme + '://' + url.hostname + file_relative_path

Look closely at the return statement:

return url.scheme + '://' + url.hostname + file_relative_path

Bingo! This is the source of our bug. A straightforward solution is to simply concatenate the port number in between, but that will make this one-liner look clumsy – unreadable and un-pythonic. We therefore use Python string formatting:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path)

But this statement isn’t perfect. There’s an edge case that might give unexpected URL. If the port isn’t originally specified, Python’s string formatting heuristic will substitute url.port with None. This will result in a URL like http://localhost:None/some/file_path.jpg, which is obviously something we don’t desire. We therefore append a call to Python’s string replace() method: replace(‘:None’, ”)

The resulting return statement now looks like the following:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path).replace(':None', '')

This should fix the problem. But that’s not enough. We need to ensure that our project adapts well with the change we made. We check this by running the project tests locally:

$ nosetests tests/unittests

Unfortunately, the tests fail with the following traceback:

======================================================================
ERROR: test_create_save_image_sizes (tests.unittests.api.helpers.test_files.TestFilesHelperValidation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 138, in test_create_save_image_sizes
resized_width_large, _ = self.getsizes(resized_image_file_large)
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 22, in getsizes
im = Image.open(file)
File "/usr/local/lib/python3.6/site-packages/PIL/Image.py", line 2312, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/open-event-server:5000/static/media/events/53b8f572-5408-40bf-af97-6e9b3922631d/large/UFNNeW5FRF/5980ede1-d79b-4907-bbd5-17511eee5903.jpg'

It’s evident from this traceback that the code in our test framework is not converting the image url to file path correctly. The port specification part is working fine, but it should not affect file names, they should be independent of port number. The files saved originally do not have port specified in their name, but the code in test framework is expecting port to be involved, hence the above error.

Using the traceback, I went to the code in the test framework where this problem occurred:

def test_create_save_image_sizes(self):
       with app.test_request_context():
           image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'

           image_sizes_type = "event"
           width_large = 1300
           width_thumbnail = 500
           width_icon = 75
           image_sizes = create_save_image_sizes(image_url_test, image_sizes_type)

           resized_image_url = image_sizes['original_image_url']
           resized_image_url_large = image_sizes['large_image_url']
           resized_image_url_thumbnail = image_sizes['thumbnail_image_url']
           resized_image_url_icon = image_sizes['icon_image_url']

           resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
           resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large.split('/localhost')[1]
           resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail.split('/localhost')[1]
           resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon.split('/localhost')[1]

           resized_width_large, _ = self.getsizes(resized_image_file_large)
           resized_width_thumbnail, _ = self.getsizes(resized_image_file_thumbnail)
           resized_width_icon, _ = self.getsizes(resized_image_file_icon)

           self.assertTrue(os.path.exists(resized_image_file))
           self.assertEqual(resized_width_large, width_large)
           self.assertEqual(resized_width_thumbnail, width_thumbnail)
           self.assertEqual(resized_width_icon, width_icon)

 

Obviously, resized_image_url.split(‘/localhost’)[1] will involve port number. So we have to change this line. But this means we also have to change the subsequent lines involving thumbnail, icon and large images. Instead of stripping the port for each of these, we can simply do this collectively at an earlier stage. So we redefine the image_sizes dictionary after the create_save_image_sizes() function call:

image_sizes = {
url_name: urlparse(image_sizes[url_name]).path
for url_name in image_sizes
} # Now file names don't contain port (this gives relative urls).

Now we can simplify the lines each of which earlier required port-stripping code:

resized_image_file = app.config.get('BASE_DIR') + resized_image_url
resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large
resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail
resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon

We now do a similar modification in test_create_save_resized_image() test method as it also involves URL to file path conversion. We break the line

resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]

to 2 lines:

resized_image_path = urlparse(resized_image_url).path
resized_image_file = app.config.get('BASE_DIR') + resized_image_path

Now let’s run the tests (which failed earlier) again:

Screen Shot 2018-08-10 at 12.29.15 PM.pngFinally, the tests pass without errors! Now, we can add some extra convenience functionality: we can also strip the port when it corresponds with the protocol we’re using. For example, if we’re using https protocol, then we need not specify the port if it is 443, as 443 corresponds to that protocol. We can add this functionality by creating a mapping of such correspondence and checking for it before generating the URL. To do this, we now go back to storage.py and add the following:

SCHEMES = {80: 'http', 443: 'https'}

And add

# No need to specify scheme-corresponding port
port = url.port
if port and url.scheme == SCHEMES.get(url.port, None):
    port = None

just before

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=port,
file_relative_path=file_relative_path).replace(':None', '')

And that finishes our work! The tests again pass successfully, plus on top of that we have this new functionality of mentioning ports only when they don’t correspond with the URL scheme!


Resources:

Continue ReadingAdding Port Specification for Static File URLs in Open Event Server

Patching an Attribute Type Across a Flask App

Recently, it was discovered by a contributor that the rating attribute for event feedbacks in Open Event was of type String. The type was incorrect, indeed. After a discussion, developers came concluded that it should be of type Float. In this post, I explain how to perform this simple migration task of changing a data type across a typical Flask app’s stack.

To begin this change, we first, we modify the database model. The model file for feedbacks (feedback.py) looks like the following:

from app.models import db


class Feedback(db.Model):
"""Feedback model class"""
__tablename__ = 'feedback'
id = db.Column(db.Integer, primary_key=True)
rating = db.Column(db.String, nullable=False)  # ←-- should be float
comment = db.Column(db.String, nullable=True)
user_id = db.Column(db.Integer,
db.ForeignKey('users.id', ondelete='CASCADE'))
event_id = db.Column(db.Integer,
db.ForeignKey('events.id', ondelete='CASCADE'))

def __init__(self, rating=None, comment=None, event_id=None,                                        user_id=None):
self.rating = rating  # ←-- cast here for safety
self.comment = comment
self.event_id = event_id
self.user_id = user_id





The change here is quite straightforward, and spans just 2 lines:

rating = db.Column(db.Float, nullable=False)

and

self.rating = float(rating)

We now perform the database migration using a couple of manage.py commands on the terminal. This file is different for different projects, but the migration commands essentially look the same. For Open Event Server, the manage.py file is at the root of the project directory (as is conventional). After cd’ing to the root, we execute the following commands:

$ python manage.py db migrate

and then

$ python manage.py db upgrade

These commands update our Open Event database so that the rating is now stored as a Float. However, if we execute these commands one after the other, we note that an exception is thrown:

sqlalchemy.exc.ProgrammingError: column "rating" cannot be cast automatically to type float
HINT:  Specify a USING expression to perform the conversion.
'ALTER TABLE feedback ALTER COLUMN rating TYPE FLOAT USING rating::double precision'

This happens because the migration code is ambiguous about what precision to use after converting to type float. It hints us to utilize the USING clause of PostgreSQL to do that. We accomplish this manually by using the psql client to connect to our database and command it the type change:

$ psql oevent
psql (10.1)
Type "help" for help.

oevent=# ALTER TABLE feedback ALTER COLUMN rating TYPE FLOAT USING rating::double precision

We now exit the psql shell and run the above migration commands again. We see that the migration commands pass successfully this time, and a migration file is generated. For our migration, the file looks like the following:

from alembic import op
import sqlalchemy as sa


# These values would be different for your migrations.
revision = '194a5a2a44ef'

down_revision = '4cac94c86047'


def upgrade():
op.alter_column('feedback', 'rating',
existing_type=sa.VARCHAR(),
type_=sa.Float(),
existing_nullable=False)


def downgrade():
op.alter_column('feedback', 'rating',
existing_type=sa.Float(),
type_=sa.VARCHAR(),
existing_nullable=False)

This is an auto-generated file (built by the database migration tool Alembic) and we need to specify the extra commands we used while migrating our database. Since we did use an extra command to specify the precision, we need to add it here. PostgreSQL USING clause can be added to alembic migration files via the postgresql_using keyword argument. Thus, the edited version of the upgrade function looks like the following:

def upgrade():
op.alter_column('feedback', 'rating',
existing_type=sa.VARCHAR(),
type_=sa.Float(),
existing_nullable=False,
postgresql_using='rating::double precision')

This completes our work on database migration. Migration files are useful for a variety of purposes – they allow us to easily get to a previous database state, or a new database state as suggested by a project collaborator. Just like git, they allow for easy version control and collaboration.

We didn’t finish this work after database migration. We also decided to impose limits on the rating value. We concluded that 0-5 would be a good range for rating. Furthermore, we also decided to round off the rating value to the “nearest 0.5”, so if the input rating is 2.3, it becomes 2.5. Also, if it is 4.1, it becomes 4.0. This was decided because such values are conventional for ratings across numerous web and mobile apps. So this will hopefully enable easier adoption for new users.

For the validation part, marshmallow came to rescue. It is a simple object serialization and deserialization tool for Python. So it basically allows to convert complex Python objects to JSON data for communicating over HTTP and vice-versa, among other things. It also facilitates pre-processing input data and therefore, allows clean validation of payloads. In our case, marshmallow was specifically used to validate the range of the rating attribute of feedbacks. The original feedbacks schema file looked like the following:

from marshmallow_jsonapi import fields
from marshmallow_jsonapi.flask import Schema, Relationship

from app.api.helpers.utilities import dasherize


class FeedbackSchema(Schema):
"""
Api schema for Feedback Model
"""
class Meta:
"""
Meta class for Feedback Api Schema
"""
type_ = 'feedback'
self_view = 'v1.feedback_detail'
self_view_kwargs = {'id': '<id>'}
inflect = dasherize

id = fields.Str(dump_only=True)
rating = fields.Str(required=True)  # ← need to validate this
comment = fields.Str(required=False)
event = Relationship(attribute='event',
self_view='v1.feedback_event',
self_view_kwargs={'id': '<id>'},
related_view='v1.event_detail',
related_view_kwargs={'feedback_id': '<id>'},
schema='EventSchemaPublic',
type_='event')




To validate the rating attribute, we use marshmallow’s Range class:

from marshmallow.validate import Range

Now we change the line

rating = fields.Str(required=True)

to

rating = fields.Float(required=True, validate=Range(min=0, max=5))

So with marshmallow, just about 2 lines of work implements rating validation for us!

After the validation part, what’s remaining is the rounding-off business logic. This is simple mathematics, and for getting to the “nearest 0.5” number, the formula goes as follows:

rating * 2 --> round off --> divide the result by 2

We will use Python’s built-in function (BIF) to accomplish this. To implement the business logic, we go back to the feedback model class and modify its constructor. Before this type change, the constructor looked like the following:

def __init__(self, rating=None, comment=None, event_id=None, user_id=None):
self.rating = rating
self.comment = comment
self.event_id = event_id
self.user_id = user_id

We change this by first converting the input rating to float, rounding it off and then finally assigning the result to feedback’s rating attribute. The new constructor is shown below:

def __init__(self, rating=None, comment=None, event_id=None, user_id=None):
rating = float(rating)
self.rating = round(rating*2, 0) / 2  # Rounds to nearest 0.5

self.comment = comment
self.event_id = event_id
self.user_id = user_id

This completes the rounding-off part and ultimately concludes rating’s type change from String to Float. We saw how a simple high-level type change requires editing code across multiple files and the use of different tools in between. In doing so, we thus also learned the utility of alembic and marshmallow in database migration and data validation, respectively.


Resources

Continue ReadingPatching an Attribute Type Across a Flask App

Attendee details in the Open Event Android App

To be able to create an order we first need to create an attendee with whom we can associate an order. Let’s see how in Open Event Android App we are creating an attendee.

We are loading the event details from our local database using the id variable. Since only logged in users can create an attendee, if the user is not logged in then the user is redirected to the login screen. If any errors are encountered while creating an attendee then they are shown in a toast message to the user. When the user clicks on the register button a POST request is sent to the server with the necessary details of the attendee. In the POST request we are passing an attendee object which has the id, first name, last name and email of the attendee. The ticket id and event id is also sent.

attendeeFragmentViewModel.loadEvent(id)

if (!attendeeFragmentViewModel.isLoggedIn()) {
redirectToLogin()
Toast.makeText(context, "You need to log in first!", Toast.LENGTH_LONG).show()
}

 

attendeeFragmentViewModel.message.observe(this, Observer {
Toast.makeText(context, it, Toast.LENGTH_LONG).show()
})

attendeeFragmentViewModel.progress.observe(this, Observer {
it?.let { Utils.showProgressBar(rootView.progressBarAttendee, it) }
})

 

attendeeFragmentViewModel.event.observe(this, Observer {
it?.let { loadEventDetails(it) }
})

rootView.register.setOnClickListener {
val attendee = Attendee(id = attendeeFragmentViewModel.getId(),
firstname = firstName.text.toString(),
lastname = lastName.text.toString(),
email = email.text.toString(),
ticket = ticketId,
event = eventId)

attendeeFragmentViewModel.createAttendee(attendee)

 

We are using a method called loadEvent in the above code which is defined in the View Model let’s have a look. We are throwing an IllegalStateException if the id is equal to -1 because this should never happen. Then we are fetching the event from the database in a background thread. If we face any errors while fetching the event we report it to the user

 

fun loadEvent(id: Long) {
if (id.equals(-1)) {
throw IllegalStateException("ID should never be -1")
}
compositeDisposable.add(eventService.getEvent(id)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe({
event.value = it
}, {
Timber.e(it, "Error fetching event %d", id)
message.value = "Error fetching event"
}))
}

 

This method is used to create an attendee. We are checking if the user has filled all the fields if any of the fields is empty a toast message is shown. Then we send a POST request to the server in a background thread. The progress bar starts loading as soon as the request is made and then finally when the attendee has been created successfully, the progress bar stops loading and a success message is shown to the user. If we face any errors while creating an attendee, an error message is shown to the user.

fun createAttendee(attendee: Attendee) {
if (attendee.email.isNullOrEmpty() || attendee.firstname.isNullOrEmpty() || attendee.lastname.isNullOrEmpty()) {
message.value = "Please fill all the fields"
return
}

compositeDisposable.add(attendeeService.postAttendee(attendee)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.doOnSubscribe {
progress.value = true
}.doFinally {
progress.value = false
}.subscribe({
message.value = "Attendee created successfully!"
Timber.d("Success!")
}, {
message.value = "Unable to create Attendee!"
Timber.d(it, "Failed")
}))
}

 

This function sends a POST request to the server and stores the attendee details in the local database.

fun postAttendee(attendee: Attendee): Single<Attendee> {
return attendeeApi.postAttendee(attendee)
.map {
attendeeDao.insertAttendee(it)
it
}

 

This is how the attendee details are inserted into the local database. In case of a conflict the attendee object gets replaced.

@Insert(onConflict = OnConflictStrategy.REPLACE)
fun insertAttendee(attendee: Attendee)

 

Resources

  1. ReactiveX official documentation : http://reactivex.io/
  2. Vogella RxJava 2 – Tutorial : http://www.vogella.com/tutorials/RxJava/article.html
  3. Androidhive RxJava Tutorial : https://www.androidhive.info/RxJava/
Continue ReadingAttendee details in the Open Event Android App