Implementing Event Export API in Open Event Frontend

In Open Event Frontend, a user can export a particular event in the zip format and download that. While dealing with an issue, we had to implement the facility of exporting the event and downloading it with a single click of button. We achieved it as follows:

The endpoints for the event export API return the responses which are not in the format of the JSON API response as we have for others like tickets, events, etc. Their responses are just the JSON objects which are not having any relationship with any model. We have four checkboxes in our template component which are used to customise the data to be present in the zip file which is to be exported. The component name is ‘download-zip’. The content of which are the checkboxes as follows:

<div class="ui form">
  <div class="field">
    {{ui-checkbox class='toggle' label=(t 'Image') checked=data.exportData.dataImage onChange=(action (mut data.exportData.dataImage))}}
  </div>
  <div class="field">
    {{ui-checkbox class='toggle' label=(t 'Video') checked=data.exportData.dataVideo onChange=(action (mut data.exportData.dataVideo))}}
  </div>
  <div class="field">
    {{ui-checkbox class='toggle' label=(t 'Audio') checked=data.exportData.dataAudio onChange=(action (mut data.exportData.dataAudio))}}
  </div>
  <div class="field">
    {{ui-checkbox class='toggle' label=(t 'Document') checked=data.exportData.dataDocument onChange=(action (mut data.exportData.dataDocument))}}
  </div>
  <div class="ui basic segment less left padding">
    <button class="ui blue button" {{action 'startGeneration'}}>
      {{t 'Start'}}
    </button>
    <button class="ui button">
      {{t 'Download'}}
    </button>
  </div>
</div>

Thus, the above code shows the four checkboxes namely audio, video, image, document used to customise the zip file generated. We also have a ‘start’ button which is used to trigger the event export. On clicking the ‘start’ button, we are handling an action called ‘startGeneration’ where we make the requests to the server which returns the event download links in response. The action is being handled in the parent controller i.e export.js.

startGeneration() {
      this.set('isLoading', true);
      let payload = this.get('data');
      this.get('loader')
        .post(`/events/${this.get('model.id')}/export/json`, payload)
        .then(exportJobInfo => {
          this.requestLoop(exportJobInfo);
        })
        .catch(() => {
          this.get('notify').error(this.l10n.t('Unexpected error occurred.'));
        });
}

As we can see, we are getting the payload from the form in the template which is shown previously above. Since the response we get from the server is not JSON API formatted, we cannot use the ember data to make requests and get a response. Thus, we use an add on called ‘loader’ which is used to make requests and get responses.
As per the server, to obtain the download URL of an event, first, we make a POST request to the URL shown in the code above with the payload that we get from the form.
On getting the response, we resolve the promise by calling the method on the same controller called ‘requestLoop’ and pass the response returned by the POST request we made which is nothing but the ‘task_url’.

The ‘requestLoop’ method makes a GET request to the task_url that we got from the previous POST to get the ‘download_url’ for the event.

requestLoop(exportJobInfo) {
    run.later(() => {
      this.get('loader')
        .load(exportJobInfo.task_url, { withoutPrefix: true })
        .then(exportJobStatus => {
          if (exportJobStatus.state === 'SUCCESS') {
            this.set('isLoading', false);
            this.set('isDownloadDisabled', false);
            this.set('eventDownloadUrl', exportJobStatus.result.download_url);
            this.set('eventExportStatus', exportJobStatus.state);
            this.get('notify').success(this.l10n.t('Event exported.'));
          } else if (exportJobStatus.state === 'WAITING') {
            this.requestLoop(exportJobInfo);
            this.set('eventExportStatus', exportJobStatus.state);
            this.get('notify').alert(this.l10n.t('Event export is going on.'));
          } else {
            this.set('isLoading', false);
            this.set('eventExportStatus', exportJobStatus.state);
            this.get('notify').error(this.l10n.t('Event export failed.'));
          }
        })
        .catch(() => {
          this.set('isLoading', false);
          this.set('eventExportStatus', 'FAILURE');
          this.get('notify').error(this.l10n.t('Event export failed.'));
        });
    }, 3000);
  }

Thus, the above code shows the ‘requestLoop’ method which runs according to the response returned by the GET to the ‘task_url’. Thus, we have three states as the response of the GET to ‘task_url’. They are:
‘FAILURE’
‘WAITING’
‘SUCCESS’
As we can see in the method, we use the logic that once the event is exported successfully or if there is any failure, we stop the loop and set the status. If the server returns ‘WAITING’ then we keep on running loop until the server returns the state ‘SUCCESS’ or ‘FAILURE’.

Thus, once the event is exported successfully, we pass the download URL returned by the server to the template and link it with the download button. Thus clicking the download button, the user can download the event as a zip file.

Resources:
Github documentation of loader.js

GET and POST requests

If you wonder how to get or update page resource, you have to read this article.

It’s trivial if you have basic knowledge about HTTP protocol. I’d like to get you little involved to this subject.

So GET and POST are most useful methods in HTTP protocol.

What is HTTP?

Hypertext transfer protocol – allow us to communicate between client and server side. In Open Event project we use web browser as client and for now we use Heroku for server side.

Difference between GET and POST methods

GET – it allows to get data from specified resources

POST – it allows to submit new data to specified resources for example by html form.

GET samples:

For example we use it to get details about event

curl http://open-event-dev.herokuapp.com/api/v2/events/95

Response from server:

Of course you can use this for another needs, If you are a poker player I suppose that you’d like to know how many percentage you have on hand.

curl http://www.propokertools.com/simulations/show?g=he&s=generic&b&d&h1=AA&h2=KK&h3&h4&h5&h6&_

POST samples:

curl -X POST https://example.com/resource.cgi

You can often find this action in a contact page or in a login page.

How does request look in python?

We use Requests library to communication between client and server side. It’s very readable for developers. You can find great documentation  and a lot of code samples on their website. It’s very important to see how it works.

>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200

I know that samples are very important, but take a look how Requests library fulfils our requirements in 100%. We have decided to use it because we would like to communicate between android app generator and orga server application. We have needed to send request with params(email, app_name, and api of event url) by post method to android generator resource. It executes the process of sending an email – a package of android application to a provided email address.

data = {
    "email": login.current_user.email,
    "app_name": self.app_name,
    "endpoint": request.url_root + "api/v2/events/" + str(self.event.id)
}
r = requests.post(self.app_link, json=data)

 

Downloading Files from URLs in Python

This post is about how to efficiently/correctly download files from URLs using Python. I will be using the god-send library requests for it. I will write about methods to correctly download binaries from URLs and set their filenames.

Let’s start with baby steps on how to download a file using requests –

import requests

url = 'http://google.com/favicon.ico'
r = requests.get(url, allow_redirects=True)
open('google.ico', 'wb').write(r.content)

The above code will download the media at http://google.com/favicon.ico and save it as google.ico.

Now let’s take another example where url is https://www.youtube.com/watch?v=9bZkp7q19f0. What do you think will happen if the above code is used to download it ? If you said that a HTML page will be downloaded, you are spot on. This was one of the problems I faced in the Import module of Open Event where I had to download media from certain links. When the URL linked to a webpage rather than a binary, I had to not download that file and just keep the link as is. To solve this, what I did was inspecting the headers of the URL. Headers usually contain a Content-Type parameter which tells us about the type of data the url is linking to. A naive way to do it will be –

r = requests.get(url, allow_redirects=True)
print r.headers.get('content-type')

It works but is not the optimum way to do so as it involves downloading the file for checking the header. So if the file is large, this will do nothing but waste bandwidth. I looked into the requests documentation and found a better way to do it. That way involved just fetching the headers of a url before actually downloading it. This allows us to skip downloading files which weren’t meant to be downloaded.

import requests

def is_downloadable(url):
    """
    Does the url contain a downloadable resource
    """
    h = requests.head(url, allow_redirects=True)
    header = h.headers
    content_type = header.get('content-type')
    if 'text' in content_type.lower():
        return False
    if 'html' in content_type.lower():
        return False
    return True

print is_downloadable('https://www.youtube.com/watch?v=9bZkp7q19f0')
# >> False
print is_downloadable('http://google.com/favicon.ico')
# >> True

To restrict download by file size, we can get the filesize from the Content-Length header and then do suitable comparisons.

content_length = header.get('content-length', None)
if content_length and content_length > 2e8:  # 200 mb approx
	return False

So using the above function, we can skip downloading urls which don’t link to media.

Getting filename from URL

We can parse the url to get the filename. Example – http://aviaryan.in/images/profile.png.

To extract the filename from the above URL we can write a routine which fetches the last string after backslash (/).

url = 'http://aviaryan.in/images/profile.png'
if url.find('/'):
	print url.rsplit('/', 1)[1]

This will be give the filename in some cases correctly. However, there are times when the filename information is not present in the url. Example, something like http://url.com/download. In that case, the Content-Disposition header will contain the filename information. Here is how to fetch it.

import requests
import re

def get_filename_from_cd(cd):
    """
    Get filename from content-disposition
    """
    if not cd:
        return None
    fname = re.findall('filename=(.+)', cd)
    if len(fname) == 0:
        return None
    return fname[0]


url = 'http://google.com/favicon.ico'
r = requests.get(url, allow_redirects=True)
filename = get_filename_from_cd(r.headers.get('content-disposition'))
open(filename, 'wb').write(r.content)

The url-parsing code in conjuction with the above method to get filename from Content-Dispositionheader will work for most of the cases. Use them and test the results.

These are my 2 cents on downloading files using requests in Python. Let me know of other tricks I might have overlooked.

{{ Repost from my personal blog http://aviaryan.in/blog/gsoc/downloading-files-from-urls.html }}