Last week, I was playing with a scraper program in Loklak Server project when I came across a library Boilerpipe. There were some issues in the program related to it’s implementation. It worked well. I implemented it, pulled a request but was rejected due to it’s maintenance issues. This wasn’t the first time an API(or a library) has let me down, but this added one more point to my ‘Linear Selection Algorithm’ to select one.
Once Libraries revolutionized the Software Projects and now API‘s are taking abstraction to a greater level. One can find many API’s and libraries on GitHub or on their respective websites, but they may be buggy. This may lead to waste of one’s time and work. I am not blogging to suggest which one to choose between the two, but what to check before getting them into use in development.
So let us select a bunch of these and give score +1 if it satisfies the point, 0 for Don’t care condition and -1 , a BIG NO.
Now initialize the variable score to zero and lets begin.
1. First thing first. is it easy to understand
Does this library code belongs to your knowledge domain? Can you use it without any issue? Also consider your project’s platform compatibility with the library. If you are developing a prototype or a small software(like for an event like Hackathon), you shall choose easy-to-read tutorial as higher priority and score++. But if you are working on a project, you shouldn’t shy going an extra mile and retain the value of score.
2. Does it have any documentation or examples of implementation
It shall have to be well written, well maintained documentation. If it doesn’t, I am ok with examples. Choose well according to your comfort. If none, at least code shall be easy to understand.
3. Does it fulfill all my needs?
Test and try to implement all the methods/ API calls needed for the project. Sometimes it may not have all the methods you need for your application or may be some methods are buggy. Take care of this point, a faulty library can ruin all your hard work.
4. Efficiency and performance (BONUS POINT for this one)
Really important for projects with high capacity/performance issues.
5. See for the Apps where they are implemented
If you are in a hackathon or a dev sprint, Checking for applications working on this API shall work. Just skip the rest of the steps (except the first).
6. Can you find blogs, Stack Overflow questions and tutorials?
If yes, This is a score++
7. An Active Community, a Super GO!
Yaay! An extra plus with the previous point.
8. Don’t tell me it isn’t maintained
This is important as if the library isn’t maintained, you are prone to bugs that may pop up in future and couldn’t be solved. Also it’s performance can never be improved. If there is no option, It is better to use it’s parts in your code so that you can work on it, if needed.
Now calculate the scores, choose the fittest one and get to work.
So with the deserving library in your hand, my first blog post here ends.
Google released the new Awareness API for everyone, a suite of signals concurring to gives the developer the context in which our user is, all as part of the Play Services 9.2.0, already available on Android devices around the globe.
This library combines 7 different feeds managing both the battery drain and the used memory, providing us with data such as user location, weather, time, headphones connection status, places, nearby beacons and currently performing activity.
INTRO
The Awareness API comes with two different versions, one is a callback based one (Fence API) and the other one is a polling based one (Snapshot API): while the first one will notify us when the conditions we specified are met, the second expects us to query the suite for the data from a unified interface.
There are 7 kinds of context that we can work with in Awareness API such as
Time — Local time at current user’s location
Location — Latitude/Longitude
Place — Places around user
Activity — Detected user activity (biking, walking, running, etc.)
Beacons — Check nearby beacon(s)
Headphones — Are headphones plugged in?
Weather — Current weather conditions
Now the Two set’s of API’s :
Snapshot API — Allows you to “request an information based on user’s context” as listed above.
Fence API — Allows you to “receive a signal when user’s context has changed and reaches the condition” through callback function, for example, if user moves closed to the specific coordinate with headphones plugged in, Fench API will call the registered BroadcastReceiver and let you do your job.
And then browse to API Manager page and search for Awareness and click at Awareness API and Click Enable and wait until it finishes enabling
3. Go to Credentials tab and click at Create credentials -> API key -> Android key. Enter the name you project, for example, Android key and click Create (or if you have already created Android key previously, you could skip this step and use the existed one)
So this ends the setup part. Now we move on to the actual data retrieval from the API.
Accessing Snapshot API
Awareness.SnapshotApi.getDetectedActivity(mGoogleApiClient)
.setResultCallback(new ResultCallback<DetectedActivityResult>() {
@Override
public void onResult(@NonNull DetectedActivityResult detectedActivityResult) {
if (!detectedActivityResult.getStatus().isSuccess()) {
Log.e(TAG, "Could not get the current activity.");
return;
}
ActivityRecognitionResult ar = detectedActivityResult.getActivityRecognitionResult();
DetectedActivity probableActivity = ar.getMostProbableActivity();
Log.i(TAG, probableActivity.toString());
}
});
This will return the most probable activity done by the user. We can get a particular activity like walking, running, driving as well by integers defined below
public static final int IN_VEHICLE = 0;
public static final int ON_BICYCLE = 1;
public static final int ON_FOOT = 2;
public static final int STILL = 3;
public static final int UNKNOWN = 4;
public static final int TILTING = 5;
public static final int WALKING = 7;
public static final int RUNNING = 8;
Similarly we can also get Headphone state, location, weather, beacon etc.
For example let’s see headphone state:
Awareness.SnapshotApi.getHeadphoneState(mGoogleApiClient)
.setResultCallback(new ResultCallback<HeadphoneStateResult>() {
@Override
public void onResult(@NonNull HeadphoneStateResult headphoneStateResult) {
if (!headphoneStateResult.getStatus().isSuccess()) {
Log.e(TAG, "Could not get headphone state.");
return;
}
HeadphoneState headphoneState = headphoneStateResult.getHeadphoneState();
if (headphoneState.getState() == HeadphoneState.PLUGGED_IN) {
Log.i(TAG, "Headphones are plugged in.\n");
} else {
Log.i(TAG, "Headphones are NOT plugged in.\n");
}
}
});
This is same as acquiring activity and headphoneState.getState() will give you if it is plugged in or not
Now we take a look at the Fence API
Fence is similar to the geofence but in addition to geofence, the fence API also allows us to set awareness conditions and check if both conditions are true.
Here when we get message in onReceive() and we can detect if headphone is connected or not. Something like this
FenceState fenceState = FenceState.extract(intent);
Log.d(TAG, "Fence Receiver Received");
if (TextUtils.equals(fenceState.getFenceKey(), "headphoneFenceKey")) {
switch (fenceState.getCurrentState()) {
case FenceState.TRUE:
Log.i(TAG, "Fence > Headphones are plugged in.");
break;
case FenceState.FALSE:
Log.i(TAG, "Fence > Headphones are NOT plugged in.");
break;
case FenceState.UNKNOWN:
Log.i(TAG, "Fence > The headphone fence is in an unknown state.");
break;
}
}
So as you can see this is pretty straight forward and very useful. You can build so many apps with multiple fences. You can think of a lot of usecases for this and make a lot of amazing apps. Happy Tinkering with the Awareness API!
If you wonder how to get or update page resource, you have to read this article.
It’s trivial if you have basic knowledge about HTTP protocol. I’d like to get you little involved to this subject.
So GET and POST are most useful methods in HTTP protocol.
What is HTTP?
Hypertext transfer protocol – allow us to communicate between client and server side. In Open Event project we use web browser as client and for now we use Heroku for server side.
Difference between GET and POST methods
GET – it allows to get data from specified resources
POST – it allows to submit new data to specified resources for example by html form.
You can often find this action in a contact page or in a login page.
How does request look in python?
We use Requests library to communication between client and server side. It’s very readable for developers. You can find great documentation and a lot of code samples on their website. It’s very important to see how it works.
I know that samples are very important, but take a look how Requests library fulfils our requirements in 100%. We have decided to use it because we would like to communicate between android app generator and orga server application. We have needed to send request with params(email, app_name, and api of event url) by post method to android generator resource. It executes the process of sending an email – a package of android application to a provided email address.
data = {
"email": login.current_user.email,
"app_name": self.app_name,
"endpoint": request.url_root + "api/v2/events/" + str(self.event.id)
}
r = requests.post(self.app_link, json=data)
One of the key components of my GSoC Project was to have a POST API for the Android App generator.
This was required so that the app generator could be plugged into the server and can be called directly instead of someone manually visiting the webpage and entering his/her details.
It takes in a JSON input and compiles and emails the app to the organizer based on his email address in the input JSON.
The input to the API will look something like this :
Google map is one of the most widely used API of Google as most of the websites use Google map for showing address location. For a static address it’s pretty simple. All you need to do is mention the address and the map will show the nearest location. Problem arrives when the address is dynamically putted by the user. Suppose for an event in event organizer server, one enters the location. The main component used while taking input location is Google Autocomplete. But we went a step further and parsed the entire address based on city, state, country, etc. and allowed user to input the details as well which gave them the nearest location marked in Map if autocomplete couldn’t find the address.
Autocomplete Location
As we can see, in the above input box we get suggestions by Google Map on entering first few letters of our address. To this, we need the API https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&libraries=places&callback=initMap. You can find an example code of how to do this here.
After this is done, what we wanted is not to just include this address, but to provide a form to fill up the entire address in case some parts were missing on this address. The function that the autocomplete listens to is “place_changed” . So once we click on one of the options, this event is triggered. Once the event is triggered, we use the autocomplete.getPlace() to get the complete address json. The json looks something like this:
Now what we do is we create a form with input fields having the id same as the ones we require(e.g., country, administrative_area_level_1, locality, etc.). After that we select the long_name or the short_name from this json and put the value in the corresponding input field with the ids. The code for the process after getting the json is elaborated here.
Editing Address Form
After doing this it looks something like this:
However, now the important part is to show the map according to this fields. Also, every time we update a field, the map should be updated. For this we use a hack. Instead of removing the initial location field completely, we hide the field but keep the binding to autocomplete intact. As a result the map is shown when we select a particular address.
Now when we update the fields in the address form, we append the value of this field to the value in the initial location field. Though the field is hidden but it is still bound to autocomplete. As a result as soon as we append something to the string contained in the field, the map gets updated. Also, the updated value gets stored to the DB. Thus, with every update in field, the pointer is moved to the nearest location formed by appending all the data from the form.
After saving the location data to DB, if we wish to edit it, we can get back the json by making the same request with the location value. And then we will get back the same address form and the map accordingly.
Many client applications require caching of data to work with low bandwidth connections. Many of them do it to provide faster loading time to the client user. The Webapp and Android app had similar requirements. Previously they provided caching using a versions API that would keep track of any modifications made to Events or Services. The response of the API would be something like this:
The number corresponding to "*_ver" tells the number of modifications done for that resource list. For instance, "tracks_ver": 3 means there were three revisions for tracks inside the event (/events/:event_id/tracks). So when the client user starts his app, the app would make a request to the versions API, check if it corresponds to the local cache and update accordingly. It had some shortcomings, like checking modifications for a individual resources. And if a particular service (microlocation, track, etc.) resource list inside an event needs to be checked for updates, a call to the versions API would be needed.
ETag based caching for GET APIs
The concept of ETag (Entity Tag) based caching is simple. When a client requests (GET) a resource or a resource list, a hash of the resource/resource list is calculated at the server. This hash, called the ETag is sent with the response to the client, preferably as a header. The client then caches the response data and the ETag alongside the resource. Next time when the client makes a request at the same endpoint to fetch the resource, he sets an If-None-Match header in the request. This header contains the value of ETag the client saved before. The server grabs the resource requested by the client, calculates its hash and checks if it is equal to the value set for If-None-Match. If the value of the hash is same, then it means the resource has not changed, so a response with resource data is not needed. If it is different, then the server returns the response with resource data and a new ETag associated with that resource.
Little modifications were needed to deal with ETags for GET requests. Flask-Restplus includes a Resource class that defines a resource. It is a pluggable view. Pluggable views need to define a dispatch_request method that returns the response.
import json
from hashlib import md5
from flask.ext.restplus import Resource as RestplusResource
# Custom Resource Class
class Resource(RestplusResource):
def dispatch_request(self, *args, **kwargs):
resp = super(Resource, self).dispatch_request(*args, **kwargs)
# ETag checking.
# Check only for GET requests, for now.
if request.method == 'GET':
old_etag = request.headers.get('If-None-Match', '')
# Generate hash
data = json.dumps(resp)
new_etag = md5(data).hexdigest()
if new_etag == old_etag:
# Resource has not changed
return '', 304
else:
# Resource has changed, send new ETag value
return resp, 200, {'ETag': new_etag}
return resp
To add support for ETags, I sub-classed the Resource class to extend the dispatch_request method. First, I grabbed the response for the arguments provided to RestplusResource‘s dispatch_request method. old_etag contains the value of ETag set in the If-None-Match header. Then hash for the resp response is calculated. If both ETags are equal then an empty response is returned with 304 HTTP status (Not Modified). If they are not equal, then a normal response is sent with the new value of ETag.
[smg:~] $ curl -i http://127.0.0.1:8001/api/v2/events/1/tracks/1
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 1061
ETag: ada4d057f76c54ce027aaf95a3dd436b
Server: Werkzeug/0.11.9 Python/2.7.6
Date: Thu, 21 Jul 2016 09:01:01 GMT
{"description": "string", "sessions": [{"id": 1, "title": "Fantastische Hardware Bauen & L\u00f6ten Lernen mit Mitch (TV-B-Gone) // Learn to Solder with Cool Kits!"}, {"id": 2, "title": "Postapokalyptischer Schmuck / Postapocalyptic Jewellery"}, {"id": 3, "title": "Query Service Wikidata "}, {"id": 4, "title": "Unabh\u00e4ngige eigene Internet-Suche in wenigen Schritten auf dem PC installieren"}, {"id": 5, "title": "Nitrokey Sicherheits-USB-Stick"}, {"id": 6, "title": "Heart Of Code - a hackspace for women* in Berlin"}, {"id": 7, "title": "Free Software Foundation Europe e.V."}, {"id": 8, "title": "TinyBoy Project - a 3D printer for education"}, {"id": 9, "title": "LED Matrix Display"}, {"id": 11, "title": "Schnittmuster am PC erstellen mit Valentina / Valentina Digital Pattern Design"}, {"id": 12, "title": "PC mit Gedanken steuern - Brain-Computer Interfaces"}, {"id": 14, "title": "Functional package management with GNU Guix"}], "color": "GREEN", "track_image_url": "http://website.com/item.ext", "location": "string", "id": 1, "name": "string"}
[smg:~] $ curl -i --header 'If-None-Match: ada4d057f76c54ce027aaf95a3dd436b' http://127.0.0.1:8001/api/v2/events/1/tracks/1
HTTP/1.0 304 NOT MODIFIED
Connection: close
Server: Werkzeug/0.11.9 Python/2.7.6
Date: Thu, 21 Jul 2016 09:01:27 GMT
ETag based caching has a drawback. Since the hash is calculated for every GET request it increases the load on servers. So if four clients request the same resource, the server calcuates hashes four times. This can be solved by calculating and saving the ETag during creation and modification of resources, and then getting and sending this ETag directly.
I recently had a requirement to create permission decorators for use in our REST APIs. There had to be separate decorators for Event and Services.
Event Permission Decorators
Understanding Event permissions is simple: Any user can create an event. But access to an event is restricted to users that have Event specific Roles (e.g. Organizer, Co-organizer, etc) for that event. The creator of an event is its Organizer, so he immediately gets access to that event. You can read about these roles in the aforementioned post.
So for Events, create operation does not require any permissions, but read/update/delete operations needed a decorator. This decorator would restrict access to users with event roles.
def can_access(func):
"""Check if User can Read/Update/Delete an Event.
This is done by checking if the User has a Role in an Event.
"""
@wraps(func)
def wrapper(*args, **kwargs):
user = UserModel.query.get(login.current_user.id)
event_id = kwargs.get('event_id')
if not event_id:
raise ServerError()
# Check if event exists
get_object_or_404(EventModel, event_id)
if user.has_role(event_id):
return func(*args, **kwargs)
else:
raise PermissionDeniedError()
return wrapper
The has_role(event_id) method of the User class determines if the user has a Role in an event.
# User Model class
def has_role(self, event_id):
"""Checks if user has any of the Roles at an Event.
"""
uer = UsersEventsRoles.query.filter_by(user=self, event_id=event_id).first()
if uer is None:
return False
else:
return True
Reading one particular event (/events/:id [GET]) can be restricted to users, but a GET request to fetch all the events (/events [GET]) should only be available to staff (Admin and Super Admin). So a separate decorator to restrict access to Staff members was needed.
def staff_only(func):
@wraps(func)
def wrapper(*args, **kwargs):
user = UserModel.query.get(login.current_user.id)
if user.is_staff:
return func(*args, **kwargs)
else:
raise PermissionDeniedError()
return wrapper
Service Permission Decorators
Service Permissions for a user are defined using Event Roles. What Role a user has in an Event determines what Services he has access to in that Event. Access here means permission to Create, Read, Update and Delete services. The User model class has four methods to determine the permissions for a Service in an event.
So four decorators were needed to put alongside POST, GET, PUT and DELETE method handlers. I’ve pasted snippet for the can_update decorator. The rest are similar but with their respective permission methods for User class object.
def can_update(DAO):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
user = UserModel.query.get(login.current_user.id)
event_id = kwargs.get('event_id')
if not event_id:
raise ServerError()
# Check if event exists
get_object_or_404(EventModel, event_id)
service_class = DAO.model
if user.can_update(service_class, event_id):
return func(*args, **kwargs)
else:
raise PermissionDeniedError()
return wrapper
return decorator
This decorator is a little different than can_access event decorator in a way that it takes an argument, DAO. DAO is Data Access Object. A DAO includes a database Model and methods to create, read, update and delete object of that model. The db model for a DAO would be the Service class for the object. You can look that the model class is taken from the DAO and used as the service class.
The can_create, can_read and can_delete decorators look exactly the same except they use their (obvious) permission methods on the User class object.
First of all you need to know that Instagram API uses OAuth 2.0 protocol. OAuth 2.0 provides a specific authorization flow for web apps, desktop apps and mobile apps. Instagram requires authentication before getting information from their API, but don’t be afraid it’s very simple.
Pre Requirements:
Created account in Instagram
Registered Client(You can create your own client here)
Errors are an essential part of a REST API system. Error instances must follow a particular structure so the client developer can correctly handle them at the client side. We had set a proper error structure at the beginning of creating REST APIs. It’s as follows:
Any error occurring during client server communication would follow the above format. Code is the returned status code and message is a brief description of the error. To raise an error we used an _error_abort() function which was an abstraction over Flask-RESTplus abort(). We defined an error structure inside _error_abort() and passed it to abort().
This method had its limitations. Since the error handlers were not being overidden, only errors raised through _error_abort() had the defined structure. So if an Internal Server error occurred, the returned error response wouldn’t follow the format.
To overcome this, we wrote our own exceptions for errors and created error handlers to handle them. We first made the response structure more detailed, so the client developer can understand what kind of error is being returned.
{
"error": {
"code": 400,
"message": "'name' is a required parameter",
"status": "INVALID_FIELD",
"field": "name"
}
}
The above is an example of a validation error.
The “status” key helps making the error more unique. For example we use 400 status code for both validation errors and invalid service errors (“Service not belonging to said event”). But both have different statuses: “INVALID_FIELD” and “INVALID_SERVICE”.
The “field” key is only useful in the case validation errors, where it names the field that did not pass the validation. For other errors it remains null.
I first documented what kind of errors we would need.
Code
Status
Field
Description
401
NOT_AUTHORIZED
null
Invalid token or user not authenticated
400
INVALID_FIELD
“field_name”
Missing or invalid field
400
INVALID_SERVICE
null
Service ID mentioned in path does not belong to said Event
404
NOT_FOUND
null
Event or Service not found
403
PERMISSION_DENIED
null
User role not allowed to perform such action
500
SERVER_ERROR
null
Internal server error
Next part was creating exception classes for each one these. I created a base error class that extended the python Exception class.
Swagger is a specification for describing REST APIs. The main aim of Swagger is to provide a REST API definition format that is readable by both machines and humans.
You can think of two entities in a REST API: One the provider of API, and another the client using the API. Swagger essentially covers the gap between them by providing a format that is easy to use by the client and easy for the provider to define.
If not using Swagger, one would most certainly be creating the API first, writing the documentation (human-readable) with it. The client developer would read the documentation and use APIs as required. With Swagger, the specification can be considered as the document itself, helping both the client developer and the provider.
Here’s an example spec I wrote for our GET APIs at Organizer Server: https://gist.github.com/shivamMg/dacada0b45585bcd9cd0fbe4a722eddf
The format, although readable doesn’t really look what a client developer would be asking for.
Remember that the format is machine readable? What does it mean exactly?
Since the Swagger spec is a defined format, the provider can document it and people can write programs that understand the Swagger specification. Swagger itself comes with a set of tools (http://swagger.io/tools/) that use Swagger definitions created by the API provider to create SDKs for the clients to use.
Swagger-UI
One of our most used tools at our server is the Swagger UI (http://swagger.io/swagger-ui/).
It reads an API spec written in Swagger to generate corresponding UI that people can use to explore the APIs. Every API endpoint can have responses and parameters associated with it. For instance our Event endpoint at Server (“/events/:event_id”).
You can see how the UI displays the Model Schema, required parameters and possible response message.
Apart from documentation, Swagger UI provides the “Try it out!” tool that lets you make requests to the server for the corresponding API. This feature is incredibly useful for POST requests. No need for long curl commands in the terminal.
Here’s the Swagger config from our demo application: https://open-event.herokuapp.com/api/v2/swagger.json
The Swagger UI for this config can be found at https://open-event.herokuapp.com/api/v2
The UI for the example spec (gist) I linked before can be browsed here.
Swagger-js
https://github.com/swagger-api/swagger-js
Swagger-js is JS library that reads an API spec written in Swagger and provides an interface to the client developer to interact with the API. We will be using Swagger-js for Open Event Webapp.
Here’s an example to show you how it works. Let’s say the following endpoint returns (json) a list of events.
/events
The client developer can create a GET request and render the list with HTML.
What if the provider moves the endpoint to “/event/all”? The client developer would need to change every instance of the URL to http://example.com/event/all.
Let’s now take the case of Swagger-js. With Swagger-js the client developer would essentially be writing programs to interact with the API Swagger spec defined by the provider, instead of directly consuming the API.
window.client = new SwaggerClient({
url: "http://example.com/api/v2/swagger.json",
success: function() {
client.event.getEvents({
responseContentType: 'application/json'
}, function(data) {
/* Create `events` string same as before */
$("ul#events").html(events);
});
}
});
The APIs at the Organizer server are getting more complex. Swagger helps in keeping the spec well defined.
You must be logged in to post a comment.