Recent Posts

Read more about the article Testing Errors and Exceptions Using Unittest in Open Event Server

Testing Errors and Exceptions Using Unittest in Open Event Server

Like all other helper functions in FOSSASIA‘s Open Event Server, we also need to test the exception and error helper functions and classes. The error helper classes are mainly used to create error handler responses for known errors. For example we know error 403 is Access Forbidden, but we want to send a proper source message along with a proper error message to help identify and handle the error, hence we use the error classes. To ensure that future commits do not mismatch the error, we implemented the unit tests for errors.

There are mainly two kind of error classes, one are HTTP status errors and the other are the exceptions. Depending on the type of error we get in the try-except block for a particular API, we raise that particular exception or error.

Unit Test for Exception

Exceptions are written in this form:

@validates_schema
    def validate_quantity(self, data):
        if 'max_order' in data and 'min_order' in data:
            if data['max_order'] < data['min_order']:
                raise UnprocessableEntity({'pointer': '/data/attributes/max-order'},
                                          "max-order should be greater than min-order")

 

This error is raised wherever the data that is sent as POST or PATCH is unprocessable. For example, this is how we raise this error: 

raise UnprocessableEntity({'pointer': '/data/attributes/min-quantity'},

           "min-quantity should be less than max-quantity")

This exception is raised due to error in validation of data where maximum quantity should be more than minimum quantity.

To test that the above line indeed raises an exception of UnprocessableEntity with status 422, we use the assertRaises() function. Following is the code:

 def test_exceptions(self):
        # Unprocessable Entity Exception
        with self.assertRaises(UnprocessableEntity):
            raise UnprocessableEntity({'pointer': '/data/attributes/min-quantity'},
                                      "min-quantity should be less than max-quantity")


In the above code, with self.assertRaises() creates a context of exception type, so that when the next line raises an exception, it asserts that the exception that it was expecting is same as the exception raised and hence ensures that the correct exception is being raised

Unit Test for Error

In error helper classes, what we do is, for known HTTP status codes we return a response that is user readable and understandable. So this is how we raise an error:

ForbiddenError({'source': ''}, 'Super admin access is required')

This is basically the 403: Access Denied error. But with the “Super admin access is required” message it becomes far more clear. However we need to ensure that status code returned when this error message is shown still stays 403 and isn’t modified in future unwantedly.

Here, errors and exceptions work a little different. When we declare a custom error class, we don’t really raise that error. Instead we show that error as a response. So we can’t use the assertRaises() function. However what we can do is we can compare the status code and ensure that the error raised is the same as the expected one. So we do this:

def test_errors(self):
        with app.test_request_context():
            # Forbidden Error
            forbidden_error = ForbiddenError({'source': ''}, 'Super admin access is required')
            self.assertEqual(forbidden_error.status, 403)

            # Not Found Error
            not_found_error = NotFoundError({'source': ''}, 'Object not found.')
            self.assertEqual(not_found_error.status, 404)


Here we firstly create an object of the error class ForbiddenError with a sample source and message. We then assert that the status attribute of this object is 403 which ensures that this error is of the Access Denied type using the assertEqual() function, which is what was expected.
The above helps us maintain that no one in future unknowingly or by mistake changes the error messages and status code so as to maintain the HTTP status codes in the response.


Resources:
Continue ReadingTesting Errors and Exceptions Using Unittest in Open Event Server
Read more about the article Implementing JSON API for ‘settings/contact-info’ route in Open Event Frontend

Implementing JSON API for ‘settings/contact-info’ route in Open Event Frontend

In Open Event Frontend, under the settings route, there is a ‘contact-info’ route which allows the user to change his info (email and contact). Previously to achieve this we were using the mock response from the server. But since we have the JSON API now we could integrate and use the JSON API for it so as to let the user modify his/her email and contact info. In the following section, I will explain how it is built:

The first thing to do is to create the model for the user so as to have a skeleton of the database and include our required fields in it. The user model looks like:

export default ModelBase.extend({
  email        : attr('string'),
  password     : attr('string'),
  isVerified   : attr('boolean', { readOnly: true }),
  isSuperAdmin : attr('boolean', { readOnly: true }),
  isAdmin      : attr('boolean', { readOnly: true }),

  firstName : attr('string'),
  lastName  : attr('string'),
  details   : attr('string'),
  contact   : attr('string'),
});

Above is the user’s model, however just the related fields are included here(there are more fields in user’s model). The above code shows that email and contact are two attributes which will accept ‘string’ values as inputs. We have a contact-info form located at ‘settings/contact-info’ route. It has two input fields as follows:

<form class="ui form {{if isLoading 'loading'}}" {{action 'submit' on='submit'}} novalidate>
  <div class="field">
    <label>{{t 'Email'}}</label>
    {{input type='email' name='email' value=data.email}}
  </div>
  <div class="field">
    <label>{{t 'Phone'}}</label>
    {{input type='text' name='phone' value=data.contact}}
  </div>
  <button class="ui teal button" type="submit">{{t 'Save'}}</button>
</form>

The form has a submit button which triggers the submit action. We redirect the submit action from the component to the controller so as to maintain ‘Data down, actions up’. The code is irrelevant, hence not shown here. Following is the action which is used to update the user which we are handling in the contact-info.js controller.

updateContactInfo() {
      this.set('isLoading', true);
      let currentUser = this.get('model');
      currentUser.save({
        adapterOptions: {
          updateMode: 'contact-info'
        }
      })
        .then(user => {
          this.set('isLoading', false);
          let userData = user.serialize(false).data.attributes;
          userData.id = user.get('id');
          this.get('authManager').set('currentUserModel', user);
          this.get('session').set('data.currentUserFallback', userData);
          this.get('notify', 'Updated information successfully');
        })
        .catch(() => {
        });
    }

We are returning the current user’s model from the route’s model method and storing it into ‘currentUser’ variable. Since we have data binding ember inputs in our contact-info form, the values will be grabbed automatically once the form submits. Thus, we can call ‘save’ method on the ‘currentUser’ model and pass an object called ‘adapterOptions’ which has key ‘updateMode’. We send this key to the ‘user’ serializer so that it picks only the attributes to be updated and omits the other ones. We have customized our user serializer as:

if (snapshot.adapterOptions && snapshot.adapterOptions.updateMode === 'contact-info') {
        json.data.attributes = pick(json.data.attributes, ['email', 'contact']);
}

The ‘save’ method on the ‘currentUser’ ‘currentUser.save()’ returns a promise. We resolve the promise by setting the ‘currentUserModel’ as the updated ‘user’ as returned by the promise. Thus, we are able to update the email and contact-info using the JSON API.

Resources:

Ember data official guide

Blog on models and Ember data by Embedly

Continue ReadingImplementing JSON API for ‘settings/contact-info’ route in Open Event Frontend
Read more about the article Checking Image Size to Avoid Crash in Android Apps

Checking Image Size to Avoid Crash in Android Apps

In Giggity app a user can create a shortcut for the event by clicking on “home shortcut” button in the navigation drawer. Open Event format provides the logo URL in the return data so we do not need to provide it separately in the app’s raw file.

Sometimes the image can be too big to be put on screen as icon for shortcut. In this blog I describe a very simple method to check if we should use the image or not to avoid the crash and pixelation due to resolution.

We can store the image received in bitmap format. A bitmap is a type of memory organization or image file format used to store digital images. The term bitmap comes from the computer programming terminology, meaning just a map of bits, a spatially mapped array of bits. By storing it in bitmap format we can easily get the necessary information about the image to check if it is suitable for use.

We can use the BitmapFactory class which provides several decoding methods like (decodeByteArray(), decodeFile(), decodeResource(), etc.) for creating a Bitmap from various sources. Choose the most appropriate decode method based on your image data source. These methods attempt to allocate memory for the constructed bitmap and therefore can easily result in an OutOfMemory exception. Each type of decode method has additional signatures that let you specify decoding options via the BitmapFactory.Options class. Setting the inJustDecodeBounds property to true while decoding avoids memory allocation, returning null for the bitmap object but setting outWidth, outHeight and outMimeType. This technique allows you to read the dimensions and type of the image data prior to construction (and memory allocation) of the bitmap.

BitmapFactory.Options options = new BitmapFactory.Options();
options.inJustDecodeBounds = true;
BitmapFactory.decodeResource(getResources(), R.id.myimage, options);
int imageHeight = options.outHeight;
int imageWidth = options.outWidth;
String imageType = options.outMimeType;

To avoid java.lang.OutOfMemory exceptions, check the dimensions of a bitmap before decoding it, unless you absolutely trust the source to provide you with predictably sized image data that comfortably fits within the available memory.

So here is the particular example from Giggity app, it avoids crash on the recieving a large image for the icon. So once we store the the image in bitmap format we check if the height and width of the icon is exceeding the maximum limit.

public Bitmap getIconBitmap() {

 InputStream stream = getIconStream();
 Bitmap ret = null;

 if (stream != null) {
 ret = BitmapFactory.decodeStream(stream);
 if (ret == null) {
 Log.w("getIconBitmap", "Discarding unparseable file");
 return null;
 }
 if (ret.getHeight() > 512 || ret.getHeight() != ret.getWidth()) {
 Log.w("getIconBitmap", "Discarding, icon not square or >512 pixels");
 return null;
 }
 if (!ret.hasAlpha()) {
 Log.w("getIconBitmap", "Discarding, no alpha layer");
 return null;
 }
 }
 
 return ret;
}

If it does then we can avoid the icon. In this case we check if the icon is more than 512 pixels in height and width. If it is so then we could avoid it.

We could also check if the icon has a transparent background by using “hasAlpha” so we could have uniformity in the icons displayed on the screen. In the final result you can see the icon of the TUBIX 2017 conference added on the screen as it was following all those defined criterias.

Now that the image dimensions are known, they can be used to decide if the full image should be loaded into memory or if a subsampled version should be loaded instead. Here are some factors to consider:

  • Estimated memory usage of loading the full image in memory.
  • Amount of memory you are willing to commit to loading this image given any other memory requirements of your application.
  • Dimensions of the target ImageView or UI component that the image is to be loaded into.
  • Screen size and density of the current device.

For example, it’s not worth loading a 1024×768 pixel image into memory if it will eventually be displayed in a 128×96 pixel thumbnail in an ImageView.

 

References:

Continue ReadingChecking Image Size to Avoid Crash in Android Apps
Read more about the article Upgrading the Style and Aesthetic of an Android App using Material Design

Upgrading the Style and Aesthetic of an Android App using Material Design

I often encounter apps as I add Open Event format support that don’t follow current design guidelines. Earlier styling an app was a tough task as the color and behaviour of the views needed to be defined separately. But now as we move forward to advanced styling methods we can easily style our app.

I recently worked on upgrading the user interface of Giraffe app after adding our Open Event support. See the repository to view the code for more reference. Here I follow the same procedure to upgrade the user interface.

First we add essential libraries to move with our material aesthetic. The Appcompat library provides backward compatibility.

//Essential Google libraries
compile 'com.android.support:appcompat-v7:25.3.1'
compile 'com.android.support:design:25.3.1

Then we define an XML file in the values folder for the style of the app which we get through Appcompat library. We could inherit same style in the entire app or separate style for the particular activity.

<resources>

   <!-- Base application theme. -->
   <style name="AppTheme" parent="Theme.AppCompat.Light.NoActionBar">
       <!-- Customize your theme here. -->
       <item name="colorPrimary">@color/colorPrimary</item>
       <item name="colorPrimaryDark">@color/colorPrimaryDark</item>
       <item name="colorAccent">@color/colorAccent</item>
   </style>


   <style name="AlertDialogCustom" parent="Theme.AppCompat.Light.Dialog.Alert">
       <item name="colorPrimary">@color/colorPrimary</item>
       <item name="colorAccent">@color/colorAccent</item>
   </style>

</resources>

So now we can see the views made following the same color scheme and behaviour throughout the app following current design guidelines without any particular manipulation to each of them.

Tip: Don’t define values of colors separately for different views. Define them in colors.xml to use them everywhere. It becomes easier then to change in future if needed.

The app now uses Action Bar for the frequently used operations unlike the custom layout that was made earlier.

This is how Action Bar is implemented,

First declare the action bar in XML layout,

Tip: Define color of the bar two shades lighter than the status bar.

 <android.support.design.widget.AppBarLayout
             android:layout_width="match_parent"
             android:layout_height="wrap_content"
             android:background="@android:color/transparent"
             android:theme="@style/ThemeOverlay.AppCompat.Dark.ActionBar">
             <android.support.v7.widget.Toolbar

                 xmlns:app="http://schemas.android.com/apk/res-auto"         
                 android:id="@+id/toolbar_options"
                 android:layout_width="match_parent"
                 android:layout_height="?attr/actionBarSize"
                 android:background="@color/colorPrimary"
                 app:popupTheme="@style/ThemeOverlay.AppCompat.Dark">
                 
                 <TextView
                     android:layout_width="wrap_content"
                     android:layout_height="wrap_content"
                     android:text="@string/options"
                     android:textColor="@color/colorAccent"
                     android:textSize="20sp" />
              </android.support.v7.widget.Toolbar>

</android.support.design.widget.AppBarLayout>

Then you can use the action bar in the activity, use onCreateOptionsMenu() method to inflate options in the toolbar.

@Override
    public void onCreate(Bundle savedInstanceState) {
        ...

        setTitle("");
        title = (TextView) findViewById(R.id.titlebar);
        Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar_main);
        setSupportActionBar(toolbar);

        ...
    }

The menu that needs to be inflated will be like this for two button at the right end of the action bar for bookmarks and filter respectively,

<menu xmlns:android="http://schemas.android.com/apk/res/android"
      xmlns:app="http://schemas.android.com/apk/res-auto">
     <item
        android:id = "@+id/action_bookmark"
        android:icon = "@drawable/ic_bookmark"
        android:menuCategory = "secondary"
        android:title = "Bookmark"
        app:showAsAction = "ifRoom" />
 
     <item
         android:id = "@+id/action_filter"
         android:icon = "@drawable/ic_filter"
         android:menuCategory = "secondary"
         android:title = "Filter"
         app:showAsAction = "ifRoom" />
</menu>

To adapt the declared style further, Alert Dialogs are also modified to match the app’s theme, it’s style is defined along with the app’s style. See below

AlertDialog.Builder noFeedBuilder = new AlertDialog.Builder(context,R.style.AlertDialogCustom);
            noFeedBuilder.setMessage(R.string.main_no_feed_text)
                    .setTitle(R.string.main_no_feed_title)
                    .setPositiveButton(R.string.common_yes, new DialogInterface.OnClickListener() {
                  ...
            noFeedBuilder.show();

Here is an example of improvement, before and after we update the user interface and aesthetic of app in easy steps defined,

   

See this for all the changes made to step up the user interface of the app.

References:

 

Continue ReadingUpgrading the Style and Aesthetic of an Android App using Material Design
Read more about the article How RSS Action Type is Implemented in SUSI Android

How RSS Action Type is Implemented in SUSI Android

Important skills of SUSI.AI are to display web search queries, a map of any location and provide a list of relevant information of a topic. RSS action type is similar to websearch action type but when the web search is to be performed on the client side, it is denoted by websearch action type and when the web search is performed by the server itself, it is denoted by rss action type. In this blog, I will show you how rss action type is implemented in SUSI Android.

In case of RSS action type server searches the internet and using RSS feeds, returns an array of objects containing :

  • Title
  • Description
  • Link
{
   “title”: “dog-doh: Definitions Index”,
   “description”: “dog-doh: Definitions Index. dog dog and pony show dog biscuit dog collar dog days …”,
  “link”: “http://websters.yourdictionary.com/index/dog-doh/”,
 }

title: Title related to user query

description: Description of user query.

link: If user want to know more information then user can use link to find more information.

How rss action type is parsed in SUSI Android

SUSI  reply in json format. It should be parsed properly to show it in android app. We used retrofit library developed by square to parse json data. Retrofit library parse data according to model class. We code model class according to expected json reply. For example, each susi response contains answer jsonarray. There are two jsonarray data and action inside answer jsonarray. We made a different model class for each jsonarray.

First model class is SusiResponse. We used this model class to parse ‘answers’ jsonarray.

@SerializedName(“answers”)
 private List<Answer> answers = new ArrayList<>();

Here we used List<>  because ‘answers’ jsonarray contain a list of jsonarray and jsonobject. Answer is second model class. We used it to parse two important jsonarray ‘data’ and ‘action’. The ‘action’ attribute has information about action type.

public class Answer {
    @SerializedName(“data”)
    private RealmList<Datum> data = new RealmList<>();
}

Here also we used the list because data jsonarray also contains a list of jsonobject but instead of simple list we used RealmList<> because after parsing we save data using realm. ‘data’ jsonarray contain multiple jsonobject and each jsonobject contain three important information ‘title’, ‘description’ and ‘link’.

Datum class is the main model class which is used to save and retrieve ‘title’, ‘description’ and ‘link’. setTitle, setLink and setDescription method of Datum class are used to save ‘title’, ‘description’ and ‘link’ and getTitle, getDescription and getLink method are use to retrieve ‘title’, ‘description’ and ‘link’.

How rss action type data is retrieved and saved

As already mentioned we used retrofit to retrieve data and realm to save data. susiResponse is response we received from SUSI server. We used susiResponse to retrieve a list of Datum class type data.

List<Datum> datumList = susiResponse.getAnswers().get(0).getData();

We then loop through datumList and from each element we extract ‘title’, ‘description’ and ‘link’ using getTitle(), getDescription() and getLink() method respectively. Datum class is model class for both retrofit and realm. realmDatum is instance of Datum class and datumRealmList is an instance of RealmList of Datum class type. After extracting data we save data using setTitle(), setDescription() and setLink().

for (Datum datum : datumList) {
          Datum realmDatum = bgRealm.createObject(Datum.class);
          realmDatum.setDescription(datum.getDescription());
          realmDatum.setLink(datum.getLink());
          realmDatum.setTitle(datum.getTitle());
         datumRealmList.add(realmDatum);
       }      

Layout design to show rss action type reply

There are three textview with id ‘title’, ‘description’ and ‘link’ to show ‘title’, ‘description’ and ‘link’ retrieved from SUSI’s reply. We used recyclerview to show list of results.

Datum datum = datumList.get(position);

holder.titleTextView.setText(Html.fromHtml(datum.getTitle()));

holder.descriptionTextView.setText(Html.fromHtml(datum.getDescription()));

holder.descriptionTextView.setText(Html.fromHtml(datum.getDescription()));

Resources

Continue ReadingHow RSS Action Type is Implemented in SUSI Android
Read more about the article Importing the Open Event format in Giraffe

Importing the Open Event format in Giraffe

Giraffe is a personal conference schedule tool for Android. Most conferences, bar camps and similar events offer their plan of sessions and talks in the iCal format for importing into your calendar. However importing a whole session plan into your standard calendar renders it pretty much useless for anything else. Giraffe allows users to import the schedule into a separate list giving you a simple overview on what happens on the conference. Besides the session, title, date and  time it also lists the speaker, location and description if available in the iCal URL. Sessions can be bookmarked and the list can be filtered by favourites and upcoming talks.

Recently I added the support for Open Event JSON format along with iCal. In this blog I describe the simple steps you need to follow to see the event that is created in the Open Event server in the Giraffe app. The initial steps are similar to Giggity app,

 1. Go to your event dashboard

2. Click on the export button.

3. Select sessions from the dashboard and copy the URL.

 

4. Click on the “Giraffe” button on the toolbar and paste the link in the box following. App will ask you to paste it when the first time you open it. Here the app loads the data and checks few initial character to see which kind of data is received. Find my other blog post to solve that problem here.

The app uses separate data models for iCal and JSON to store the informations received and then save them in SQL database for CRUD options. See the database activity here

   

5. Now you can see the sessions. Click on them to see more information or bookmark them if needed. The data is loaded from the database so when app is offline so we don’t need to worry about connection once the data is being loaded.

   

Resources:

 

Continue ReadingImporting the Open Event format in Giraffe
Read more about the article Open Event Server: Testing Image Resize Using PIL and Unittest

Open Event Server: Testing Image Resize Using PIL and Unittest

FOSSASIA‘s Open Event Server project uses a certain set of functions in order to resize image from its original, example to thumbnail, icon or larger image. How do we test this resizing of images functions in Open Event Server project? To test image dimensions resizing functionality, we need to verify that the the resized image dimensions is same as the dimensions provided for resize.  For example, in this function, we provide the url for the image that we received and it creates a resized image and saves the resized version.

def create_save_resized_image(image_file, basewidth, maintain_aspect, height_size, upload_path,
                              ext='jpg', remove_after_upload=False, resize=True):
    """
    Create and Save the resized version of the background image
    :param resize:
    :param upload_path:
    :param ext:
    :param remove_after_upload:
    :param height_size:
    :param maintain_aspect:
    :param basewidth:
    :param image_file:
    :return:
    """
    filename = '{filename}.{ext}'.format(filename=get_file_name(), ext=ext)
    image_file = cStringIO.StringIO(urllib.urlopen(image_file).read())
    im = Image.open(image_file)

    # Convert to jpeg for lower file size.
    if im.format is not 'JPEG':
        img = im.convert('RGB')
    else:
        img = im

    if resize:
        if maintain_aspect:
            width_percent = (basewidth / float(img.size[0]))
            height_size = int((float(img.size[1]) * float(width_percent)))

        img = img.resize((basewidth, height_size), PIL.Image.ANTIALIAS)

    temp_file_relative_path = 'static/media/temp/' + generate_hash(str(image_file)) + get_file_name() + '.jpg'
    temp_file_path = app.config['BASE_DIR'] + '/' + temp_file_relative_path
    dir_path = temp_file_path.rsplit('/', 1)[0]

    # create dirs if not present
    if not os.path.isdir(dir_path):
        os.makedirs(dir_path)

    img.save(temp_file_path)
    upfile = UploadedFile(file_path=temp_file_path, filename=filename)

    if remove_after_upload:
        os.remove(image_file)

    uploaded_url = upload(upfile, upload_path)
    os.remove(temp_file_path)

    return uploaded_url


In this function, we send the image url, the width and height to be resized to, and the aspect ratio as either True or False along with the folder to be saved. For this blog, we are gonna assume aspect ratio is False which means that we don’t maintain the aspect ratio while resizing. So, given the above mentioned as parameter, we get the url for the resized image that is saved.
To test whether it has been resized to correct dimensions, we use Pillow or as it is popularly know, PIL. So we write a separate function named getsizes() within which get the image file as a parameter. Then using the Image module of PIL, we open the file as a JpegImageFile object. The JpegImageFile object has an attribute size which returns (width, height). So from this function, we return the size attribute. Following is the code:

def getsizes(self, file):
        # get file size *and* image size (None if not known)
        im = Image.open(file)
        return im.size


As we have this function, it’s time to look into the unit testing function. So in unit testing we set dummy width and height that we want to resize to, set aspect ratio as false as discussed above. This helps us to test that both width and height are properly resized. We are using a creative commons licensed image for resizing. This is the code:

def test_create_save_resized_image(self):
        with app.test_request_context():
            image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'
            width = 500
            height = 200
            aspect_ratio = False
            upload_path = 'test'
            resized_image_url = create_save_resized_image(image_url_test, width, aspect_ratio, height, upload_path, ext='png')
            resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
            resized_width, resized_height = self.getsizes(resized_image_file)


In the above code from create_save_resized_image, we receive the url for the resized image. Since we have written all the unittests for local settings, we get a url with localhost as the server set. However, we don’t have the server running so we can’t acces the image through the url. So we build the absolute path to the image file from the url and store it in resized_image_file. Then we find the sizes of the image using the getsizes function that we have already written. This  gives us the width and height of the newly resized image. We make an assertion now to check whether the width that we wanted to resize to is equal to the actual width of the resized image. We make the same check with height as well. If both match, then the resizing function had worked perfectly. Here is the complete code:

def test_create_save_resized_image(self):
        with app.test_request_context():
            image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'
            width = 500
            height = 200
            aspect_ratio = False
            upload_path = 'test'
            resized_image_url = create_save_resized_image(image_url_test, width, aspect_ratio, height, upload_path, ext='png')
            resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
            resized_width, resized_height = self.getsizes(resized_image_file)
            self.assertTrue(os.path.exists(resized_image_file))
            self.assertEqual(resized_width, width)
            self.assertEqual(resized_height, height)


In open event orga server, we use this resize function to basically create 3 resized images in various modules, such as events, users,etc. The 3 sizes are names – Large, Thumbnail and Icon. Depending on the one more suitable we use it avoiding the need to load a very big image for a very small div. The exact width and height for these 3 sizes can be changed from the admin settings of the project. We use the same technique as mentioned above. We run a loop to check the sizes for all these. Here is the code:

def test_create_save_image_sizes(self):
        with app.test_request_context():
            image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'
            image_sizes_type = "event"
            width_large = 1300
            width_thumbnail = 500
            width_icon = 75
            image_sizes = create_save_image_sizes(image_url_test, image_sizes_type)

            resized_image_url = image_sizes['original_image_url']
            resized_image_url_large = image_sizes['large_image_url']
            resized_image_url_thumbnail = image_sizes['thumbnail_image_url']
            resized_image_url_icon = image_sizes['icon_image_url']

            resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
            resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large.split('/localhost')[1]
            resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail.split('/localhost')[1]
            resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon.split('/localhost')[1]

            resized_width_large, _ = self.getsizes(resized_image_file_large)
            resized_width_thumbnail, _ = self.getsizes(resized_image_file_thumbnail)
            resized_width_icon, _ = self.getsizes(resized_image_file_icon)

            self.assertTrue(os.path.exists(resized_image_file))
            self.assertEqual(resized_width_large, width_large)
            self.assertEqual(resized_width_thumbnail, width_thumbnail)
            self.assertEqual(resized_width_icon, width_icon)

Resources:
Continue ReadingOpen Event Server: Testing Image Resize Using PIL and Unittest
Read more about the article Know the Usage of all Emoji’s on Twitter with loklak Emoji Heatmap

Know the Usage of all Emoji’s on Twitter with loklak Emoji Heatmap

Loklak apps page now has a new app in its store, Emoji Heatmap. This app can be used to see the usage of all the emoji’s in the tweets all over the world in the form of heatmap. So, the major difference between the emoji heatmap and emoji heatmapper apps are heatmapper shows the tweets related to specific search query whereas this heatmapper app, it displays all the tweets which contains emojis.

How do the App fetches and stores the locations

The emoji heatmap uses the loklak search API . The search API needs a query in order to search and output the JSON data. But this app takes no input from the user end to search any query. To make the search dynamic, we are using an existing JSON file from emojiHeatmapper app and loklak-fetcher-client javascript file. From the emoji.json file, we collect the each query item and search it using loklak-fetcher-client. The output json file which we get from loklak-fetcher-client is retrieved into the emojiHeatmap and we extract the location parameter. The location parameter is then stored into the “feature” option of open layers 3 maps.

So, here in the emoji Heatmap app, we iterate over the emoji.json, get different search query each time when we search for it using loklak search API.

Code which adds the location retrieved into feature

  $.getJSON("../emojiHeatmapper/emoji.json", function(json) {
    for (var i = 0; i < json.data.length; i++){
      var query = json.data[i][1];
      // Fetch loklak API data, and fill the vector
      loklakFetcher.getTweets(query, function(tweets) {
        for(var i = 0; i < tweets.statuses.length; i++) {
          if(tweets.statuses[i].location_point !== undefined){
          // Creation of the point with the tweet's coordinates
          //  Coords system swap is required: OpenLayers uses by default
          //  EPSG:3857, while loklak's output is EPSG:4326
            var point = new ol.geom.Point(ol.proj.transform(tweets.statuses[i].location_point, 'EPSG:4326', 'EPSG:3857'));
            vector.addFeature(new ol.Feature({  // Add the point to the data vector
              geometry: point,
              weight: 20
            }));
          }
        }
      });
    }
  });

 

The above function gets has two variables query and point. The query variable stores the data that is being retrieved from the emoji.json file each time it iterates and that query is being sent into the loklak-fetcher-client. Then the point variable is in which the location tracked using the loklak search API is converted into the co-ordinates system followed by the Open Layers 3. Then the point is added as a feature to the map vector. The map vector is the place where all the features are stored and then appended onto the map as a heatmap.

Resources

Continue ReadingKnow the Usage of all Emoji’s on Twitter with loklak Emoji Heatmap
Read more about the article Utilizing Readiness Probes for loklak Dependencies in Kubernetes

Utilizing Readiness Probes for loklak Dependencies in Kubernetes

When we use any application and fail to connect to it, we do not give up and retry connecting to it again and again. But in the reality we often face this kind of obstacles like application that break instantly or when connecting to an API or database that is not ready yet, the app gets upset and refuses to continue to work. So, something similar to this was happening with api.loklak.org.
In such cases we can’t really re-write the whole application again every time the problem occurs.So for this we need to define dependencies of some kind that can handle the situation rather than disappointing the users of loklak app.

Solution:

We will just wait until a dependent API or backend of loklak is ready and then only start the loklak app. For this to be done, we used Kubernetes Health Checks.

Kubernetes health checks are divided into liveness and readiness probes.
The purpose of liveness probes are to indicate that your application is running.
Readiness probes are meant to check if your application is ready to serve traffic.
The right combination of liveness and readiness probes used with Kubernetes deployments can:
Enable zero downtime deploys
Prevent deployment of broken images
Ensure that failed containers are automatically restarted

Pod Ready to be Used?

A Pod with defined readiness probe won’t receive any traffic until a defined request can be successfully fulfilled. This health checks are defined through the Kubernetes, so we don’t need any changes to be made in our services (APIs). We just need to setup a readiness probe for the APIs that loklak server is depending on. Here you can see the relevant part of the container spec you need to add (in this example we want to know when loklak is ready):

        readinessProbe:
          httpGet:
            path: /api/status.json
            port: 80
          initialDelaySeconds: 30
          timeoutSeconds: 3

 

Readiness Probes

Updating deployments of loklak when something pushed into development without readiness probes can result in downtime as old pods are replaced by new pods in case of Kubernetes deployment. If the new pods are misconfigured or somehow broken, that downtime extends until you detect the problem and rollback.
With readiness probes, Kubernetes will not send traffic to a pod until the readiness probe is successful. When updating a loklak deployment, it will also leave old one’s running until probes have been successful on new copy. That means that if loklak server new pods are broken in some way, they’ll never see traffic, instead old pods of loklak server will continue to serve all traffic for the deployment.

Conclusion

Readiness probes is a simple solution to ensure that Pods with dependencies do not get started before their dependencies are ready (in this case for loklak server). This also works with more than one dependency.

Resources

Continue ReadingUtilizing Readiness Probes for loklak Dependencies in Kubernetes
Read more about the article Creating Unit Tests for File Upload Functions in Open Event Server with Python Unittest Library

Creating Unit Tests for File Upload Functions in Open Event Server with Python Unittest Library

In FOSSASIA‘s Open Event Server, we use the Python unittest library for unit testing various modules of the API code. Unittest library provides us with various assertion functions to assert between the actual and the expected values returned by a function or a module. In normal modules, we simply use these assertions to compare the result since the parameters mostly take as input normal data types. However one very important area for unittesting is File Uploading. We cannot really send a particular file or any such payload to the function to unittest it properly, since it expects a request.files kind of data which is obtained only when file is uploaded or sent as a request to an endpoint. For example in this function:

def uploaded_file(files, multiple=False):
    if multiple:
        files_uploaded = []
        for file in files:
            extension = file.filename.split('.')[1]
            filename = get_file_name() + '.' + extension
            filedir = current_app.config.get('BASE_DIR') + '/static/uploads/'
            if not os.path.isdir(filedir):
                os.makedirs(filedir)
            file_path = filedir + filename
            file.save(file_path)
            files_uploaded.append(UploadedFile(file_path, filename))

    else:
        extension = files.filename.split('.')[1]
        filename = get_file_name() + '.' + extension
        filedir = current_app.config.get('BASE_DIR') + '/static/uploads/'
        if not os.path.isdir(filedir):
            os.makedirs(filedir)
        file_path = filedir + filename
        files.save(file_path)
        files_uploaded = UploadedFile(file_path, filename)

    return files_uploaded


So, we need to create a mock uploading system to replicate this check. So inside the unittesting function we create an api route for this particular scope to accept a file as a request. Following is the code:

@app.route("/test_upload", methods=['POST'])
        def upload():
            files = request.files['file']
            file_uploaded = uploaded_file(files=files)
            return jsonify(
                {'path': file_uploaded.file_path,
                 'name': file_uploaded.filename})


In the above code, it creates an app route with endpoint test_upload. It accepts a request.files. Then it sends this object to the uploaded_file function (the function to be unittested), gets the result of the function, and returns the result in a json format.
With this we have the endpoint to mock a file upload ready. Next we need to send a request with file object. We cannot send a normal data which would then be treated as a normal request.form. But we want to receive it in request.files. So we create 2 different classes inheriting other classes.

def test_upload_single_file(self):

        class FileObj(StringIO):

            def close(self):
                pass

        class MyRequest(Request):
            def _get_file_stream(*args, **kwargs):
                return FileObj()

        app.request_class = MyRequest


MyRequest class inherits the Request class of Flask framework. We define the file stream of the Request class as the FileObj. Then, we set the request_class attribute of the Flask app to this new MyRequest class.
After we have it all setup, we need to send the request and see if the uploaded file is being saved properly or not. For this purpose we take help of StringIO library. StringIO creates a file-like class which can be then used to replicate a file uploading system. So we send the data as {‘file’: (StringIO(‘1,2,3,4’), ‘test_file.csv’)}. We send this as data to the /test_upload endpoint that we have created previously. As a result, the endpoint receives the function, saves the file, and returns the filename and file_path for the stored file.

 with app.test_request_context():
            client = app.test_client()
            resp = client.post('/test_upload', data = {'file': (StringIO('1,2,3,4'), 'test_file.csv')})
            data = json.loads(resp.data)
            file_path = data['path']
            filename = data['name']
            actual_file_path = app.config.get('BASE_DIR') + '/static/uploads/' + filename
            self.assertEqual(file_path, actual_file_path)
            self.assertTrue(os.path.exists(file_path))


After this is done, we need to check if the file_path that we receive is the expected file path that we should get. Secondly, we also check whether the file was really created or is this just some dummy data sent. We get the expected path by this:

actual_file_path = app.config.get('BASE_DIR') + '/static/uploads/' + filename.

Then we assert that actual_file_path is same as the resulting path we received using the assertEqual. Thirdly, we use assertTrue to ensure that there is a file in that path. That is,

self.assertTrue(os.path.exists(file_path))

Which gives a True if file exists or False if not.

So that basically sums up the unittesting.
1) If the file is saved in the correct path, and
2) The file actually exist
The the unittest passes only if both is True and is thus successful. Else we get either an error or a failure.

Following is the entire code snippet for this unit testing function:

def test_upload_single_file(self):

        class FileObj(StringIO):

            def close(self):
                pass

        class MyRequest(Request):
            def _get_file_stream(*args, **kwargs):
                return FileObj()

        app.request_class = MyRequest

        @app.route("/test_upload", methods=['POST'])
        def upload():
            files = request.files['file']
            file_uploaded = uploaded_file(files=files)
            return jsonify(
                {'path': file_uploaded.file_path,
                 'name': file_uploaded.filename})

        with app.test_request_context():
            client = app.test_client()
            resp = client.post('/test_upload', data = {'file': (StringIO('1,2,3,4'), 'test_file.csv')})
            data = json.loads(resp.data)
            file_path = data['path']
            filename = data['name']
            actual_file_path = app.config.get('BASE_DIR') + '/static/uploads/' + filename
            self.assertEqual(file_path, actual_file_path)
            self.assertTrue(os.path.exists(file_path))

Resources:
Continue ReadingCreating Unit Tests for File Upload Functions in Open Event Server with Python Unittest Library