Join Codeheat Coding Contest 2019/20

Master Git, contribute to Open Source, and win a trip to the FOSSASIA Summit Singapore with Codeheat! Codeheat is the annual coding contest for developers to contribute to Free and Open Source software (FOSS) and open hardware projects of FOSSASIA. Join development of real world software applications and win awesome prizes, build up your developer profile, learn new coding skills, collaborate with the community and make new friends from around the world! Sign up now for the fourth edition of Codeheat on the website and follow Codeheat on Twitter.

Start date: September 15, 2019

End date: February 2, 2020

Which Projects Participate

Open Event – Eventyay / Code / Chat

SUSI AI – Website / Code / Chat

Pocket Science Lab (PSLab) – Website / Code / Chat

Phimpme Android – App / Code / Chat

Meilix Linux Distribution – Code / Chat

Voicerepublic – Website / Code / Chat

Badge Magic- App / Code / Chat

Neurolab – Code / Chat

Badgeyay – Website / Code / Chat

How to Join the Contest

  • The contest is open to everyone.
  • Participants can join at any time
  • Register on the site and check out the Frequently Asked Questions for more details.
  • Also join the FOSSASIA Gitter chat and communicate with mentors and follow developers on project specific channels. 

What are the Prizes

  • Winners (3 prizes): Listed on website, certificate, 600SGD travel voucher, 5-night accommodation in Singapore, Tshirt and FOSSASIA limited edition swags. 
  • Finalist (7 prizes): Listed on website, certificate, travel voucher of 100 SGD, Tshirt and FOSSASIA limited edition swags. 
  • Active Contributors (unlimited): Certificate, CodeHeat Tshirt and FOSSASIA limited edition of swags (with at least 10 merged pull requests)
  • Community Participants (unlimited): Digital Certificate of Participation (with at least 5 merged pull requests)

What are the Judging Criteria

Our jury will review the work of the 10 developers who have the highest number of quality contributions during the contest. Contributions include pull requests/code commits, scrum reports, articles, screencasts, community engagement and outreach activities. The mentors will look at the:  

Sustainability, which means that we specifically value contributions that make the project sustainable by building a community where developers collaborate with each other in a friendly way and support the project development through peer reviews, on-boarding new members, and helping fellow contributors. It also means that, while code is the most important success criteria for winning the contest, furthermore we are looking for contributions in other areas to make projects easy to join, to deploy and to use. This includes:

  • creating and enhancing documentation
  • developing how-tos
  • writing technical blog posts
  • sharing work in regular scrum updates to enhance communication
  • organizing local meetups, workshops, presentations 

Quality vs. Quantity: The sheer number of pull requests is not the only criteria for choosing the winners. Quality work is appreciated – some issues are more challenging than others just by their nature (for example, heavy coding versus solving a text typo bug). It is entirely possible that someone who completed 53 issues could be chosen as a winner over someone who completed 88 issues.

How Are the Winners Decided

  • Grand Prize Winners: Three developers will be selected by mentors from the top 10 contributors according to code quality, relevance of commits and contributions that help to bring the project forward.  
  • Finalist Winners: After the grand prize winners are selected, the remaining seven contributors of top the 10 will receive finalist winner prizes.
  • Other contributors who have more than 10 merged pull requests during the contest will receive a Thank you package. Anyone who has 5 pull requests merged will receive a digital certificate.

Links

Website: codeheat.org

Codeheat Twitter: twitter.com/codeheat_

FOSSASIA Twitter: twitter.com/fossasia

Codeheat Facebook: facebook.com/codeheat.org

Continue Reading

Neurolab data transfer – Establishing serial communication between Arduino and Android

In the development process of the Neurolab Android, we needed an Arduino-Android connection for transfer of data from datasets which included String and float data type values. In this blog post, I will show you how to establish a serial communication channel between

Android and Arduino through USB cable through which we can transmit data bidirectionally.

Requirements

Hardware:

  1. Android Phone
  2. Arduino (theoretically from any type, but I’ll be using Arduino Uno)
  3. USB 2.0 Cable Type A/B (for Arduino)
  4. OTG Cable (On The Go)
  5. Normal USB Cable (for transferring the data from Android Studio to your phone)

Software:

  1. Android Studio
  2. Arduino IDE

Wiring and Setup

Wiring must be established in the following way:

                                                                                 Figure: Android-Arduino Setup

Working on the Android Side

We would be using the UsbSerial Library by felHR85 for establishing serial

communication.

1. Adding the dependency:

a) Add the following line of code to your app level build.gradle file.

implementation "com.github.felHR85:UsbSerial:$rootProject.usbSerialLibraryVersion"

Note: The ‘usbSerialLibraryVersion’ may change from time to time. Please keep your project with the latest library version. Updates can be found here.

b) Add jitpack to your project.build.gradle file.

allprojects {
   repositories {
       jcenter()
       maven { url "https://jitpack.io" }
   }
}

2. Working with Arduino:

We need to program the Arduino to send and receive data. We achieve that with the help of Arduino IDE as mentioned above. Verify, compile and upload a sketch to the Arduino for sending and receiving data. If you are a complete beginner in Arduino programming, there are example sketches for this same purpose. Load an example sketch from under the communication segment and choose the serial communication sketch. Here, we will be working with a simple sketch for the Arduino such that it simply echoes whatever it receives on the serial port. Here is sketch code:

// the setup routine runs once when you press reset:
void setup() {
  // initialize serial communication at 9600 bits per second:
  Serial.begin(9600);
}

// the loop routine runs over and over again forever:
void loop() {
  char incomingByte;
   // If there is a data stored in the serial receive buffer, read it and print it to the serial port as human-readable ASCII text.
  if(Serial.available()){  
    incomingByte = Serial.read();
    Serial.print(incomingByte);  
  }
}

Feel free to compile and upload it to your own Arduino.

2. Working with Android:

  1. Firstly, we need an USBManager instance initialized with the system service – ‘USB_SERVICE’. This needs to be done in an Activity (preferably main) so that this instance can be passed to the Communication Handler class, which we are going to create next.
  2. Now, we will be working with a class for handling the serial communications with our Android device and the Arduino board. We would pass the USBManager instance to this class wherein work will be done with that to find the USBDevice and USBDeviceConnection

Starting with, we need to search for any attached Arduino devices to the Android device. We create a method for this and use the ‘getDevicesList’ callback to achieve this in the following way:

public void searchForArduinoDevice(Context context) {
        HashMap usbDevices = usbManager.getDeviceList();

        if (!usbDevices.isEmpty()) {
            boolean keep = true;
            for (Object object : usbDevices.entrySet()) {
                Map.Entry<String, UsbDevice> entry = (Map.Entry<String, UsbDevice>) object;
                device = entry.getValue();

                int deviceVID = device.getVendorId();
                if (deviceVID == ARDUINO_DEVICE_ID) { //Arduino Vendor ID = 0x2341
                    PendingIntent pi = PendingIntent.getBroadcast(context, 0,
                            new Intent(ACTION_USB_PERMISSION), 0);
                    usbManager.requestPermission(device, pi);
                    keep = false;
                } else {
                    connection = null;
                    device = null;
                }
                if (!keep)
                    break;
            }
        }
    }

c. Now, in the Activity where we will be testing or intend to work with the serial connection, we check for the usb permission in a broadcast receiver which is registered in the onCreate method of the activity along with an Intent Filter. The Intent filter has the usp permission as an action added to it.

usbCommunicationHandler = USBCommunicationHandler.getInstance(this, NeuroLab.getUsbManager());

        deviceConnector = new DeviceConnector(NeuroLab.getUsbManager());

        IntentFilter intentFilter = new IntentFilter();
 
        intentFilter.addAction(ACTION_USB_PERMISSION);
        registerReceiver(broadcastReceiver, intentFilter);

d. In the onReceive callback of the broadcast receiver, if the usb permission is granted, we initialize the serial connection with a baud rate for our Arduino device. In this initialization method, we get the connection and serial port of the connected Arduino to the Android device with which we can work. The method is implemented in the following way:

public boolean initializeSerialConnection(int baudRate) {
        connection = usbManager.openDevice(device);
        serialPort = UsbSerialDevice.createUsbSerialDevice(device, connection);
        if (serialPort != null) {
            if (serialPort.open()) { 
                serialPort.setBaudRate(baudRate);
                serialPort.setDataBits(UsbSerialInterface.DATA_BITS_8);
                serialPort.setStopBits(UsbSerialInterface.STOP_BITS_1);
                serialPort.setParity(UsbSerialInterface.PARITY_NONE);
                serialPort.setFlowControl(UsbSerialInterface.FLOW_CONTROL_OFF);
            } else {
                Log.d("SERIAL", "PORT NOT OPEN");
                return false;
            }
        } else {
            Log.d("SERIAL", "PORT IS NULL");
            return false;
        }
        setSerialPort(serialPort);
        return true;
    }

e. Now, with this ‘serialPort’ we can read data from the Arduino in the UsbReadCallback callback from the USBSerialInterface. The data is read in the form of array of bytes. This read data can be used to carry out the various functionalities we want to achieve.

With the above steps we can establish a serial connection between Android and Arduino, transmit data from Arduino to Android device for processing.

The whole working source code of the Neurolab Android project can be found here:

https://github.com/fossasia/neurolab-android/

Thanks for taking the time to read this blog. Hope it was able to make some good contributions to your knowledge base.

References

  1. https://github.com/felHR85/UsbSerial
  2. https://www.arduino.cc/reference/en/language/functions/communication/serial/

Tags: FOSSASIA, Neurolab, GSOC19, Open-source, Arduino, Serial terminal

Continue Reading

How to transfer data in files from different locations to app directory

In Android apps, we might have seen various instances in-app where we can import, export or save files. Now, where does the files go to or come from? There needs to be a specific folder or directory (maybe root) for a particular app which inturn contains the necessary files, the user has worked with from the app. In this blog, we will be learning about the backend of how to create directories and store files in them dynamically with proper content.

Context

I have been working with FOSSASIA on the project Neurolab-Android. In the app, we have various program modes, which has the option to save/record data. The saved data gets logged in a new file in the Neurolab directory/folder. Feel free to go ahead and explore the feature.

The Save/Record feature in Neurolab can be found in the app bar or as an option in the drop down menu present in the app bar itself in any program mode. The feature becomes functional, once the data is imported with the import data feature which is also present in the app bar.

                                              Figure: Demonstration of features in Neurolab

Tutorial

Now, starting off, there are apps out there wherein users can save files from different segments in the app and those saved files can be used by the app itself at other times as they belong to that app itself specifically.

First off, we need to make sure we have a directory for our app, wherein the files will get stored. If the directory or folder is not present, it needs to be created.

File directory = new File(
                Environment.getExternalStorageDirectory().getAbsolutePath() +
                        File.separator + DIRECTORY_NAME);
        if (!directory.exists()) {
            try {
                directory.mkdir();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

Now, once the directory is present, we can go ahead to keep the saved files in it. Also we will be implementing category-wise storage of the files. Simply put, we will be storing csv files in the CSV folder, xls files in the Excel folder, etc. In the below code example, we see the use case of CSV ‘category’.

private void categoryWise() {File categoryDirectory = new File(
                Environment.getExternalStorageDirectory().getAbsolutePath() +
                        File.separator + DIRECTORY_NAME + File.separator + category);
        if (!categoryDirectory.exists()) {
            try {
                categoryDirectory.mkdir();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }}

For saving a file in our app, we need to get (import) the file into our app. Once the file is imported, we can get the path of the file with the help of its URI. Let’s assign the path to a variable named ‘importedFilePath’. Then we place the imported file in the required category directory within our parent app directory depending and deciding upon the extension of the imported file.

File importedFile = new File(importedFilePath);
        FilePathUtil.setupPath();
        Date currentTime = Calendar.getInstance().getTime();
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String fileName = sdf.format(currentTime);
        File dst = new File(Environment.getExternalStorageDirectory().getAbsolutePath() +
                File.separator + DIRECTORY_NAME + File.separator + categoryDirectory + File.separator + fileName + ".$extension");
        if (!dst.exists()) {
            try {                categoryWise()
                transfer(importedFile, dst);
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

Now, we have the ‘importedFile’ and the destination file path (dst) where the file needs to be stored for our app. The ‘extension’ can be of any type you feel the file should be. Here, for the fileName we are using the current date and time together.

Then, we can come to the function ‘transfer’ which has been called above.

private static void transfer(File src, File dst) throws IOException {
        InputStream in = new FileInputStream(src);
        try {
            OutputStream out = new FileOutputStream(dst);
            try {
                // Transfer bytes from in to out
                byte[] buf = new byte[1024];
                int len;
                while ((len = in.read(buf)) > 0) {
                    out.write(buf, 0, len);
                }
            } finally {
                out.close();
            }
        } finally {
            in.close();
        }
    }

In the ‘transfer’ function, we initialize an input stream with the source file path and the output stream with the destination file path. We read the content in the form of  a certain chunk of bytes from the source file and write to the output stream (destination file).

Finally, we close the output and input streams simultaneously.

Thus, we have our code ready to be bound by UI actions/buttons. Once, the user interacts with the action in your app, the imported file will get saved in the specific directory of your app.

That’s it. Hope this blog enhanced your Android development and Java skillset. 

Resources:

  1. Author – Google Android Developers, Article – Data and File storage, Website – https://developer.android.com/guide/topics/data/data-storage
  2. Author – Rakshi and Thomas, Article – How to make a copy of file in android, Source – Stack overflow, Website – https://stackoverflow.com/questions/9292954/how-to-make-a-copy-of-a-file-in-android

Tags: FOSSASIA. Neurolab, GSOC19, Open-source, File-storage

Continue Reading

How to fix undetected Arduino boards in Android

In the development process of the Neurolab Android app, we needed an Arduino-Android connection. This blog explains how to  establish the connection and getting the Arduino board detected in my Android device

Context-connecting the board and getting it detected

Arduino boards are primarily programmed from the Desktop using the Arduino IDE, but they are not limited to the former. Android devices can be used to program the circuit boards using an application named Arduinodroid.

Arduino is basically a platform for building various types of electronic projects and the best part about it is that, it is open-sourced. Arduino, the company has got two products The physical programmable circuit board (often referred to as a microcontroller). 

Examples of Arduino circuit boards – UNO, UNO CH340G, Mega, etc. Find more here.

Connecting the board and getting it detected

Arduino boards are primarily programmed from the Desktop using the Arduino IDE, but they are not limited to the former. Android devices can be used to program the circuit boards using an application named Arduinodroid.

In this blog, we are going to use Arduinodroid app for establishing a connection between the Arduino board and the Android device, getting the board detected in the Android phone and uploading a sketch to it.

Materials/Gadgets required:-

  1. Arduino board (UNO preferably)
  2. Arduino-USB Cable
  3. OTG Cable
  4. Android device

Now, one of the most frequent issues, while establishing a connection and getting the Arduino board detected with the Android device, is the error message of: “No Arduino boards detected” in the Arduinodroid app. There can be a few core reasons for this –

  1. Your Android mobile device isn’t USB-OTG supported – Probably because it is an old model or it might be a company/brand-specific issue.
  2. Disabled OTG Mode – Be sure to enable USB-OTG mode (only if your device has one) from the Developer options in your Android device settings.

Even after trying and making sure of these above points, if you still continue to get an error while uploading a sketch from the Arduinodroid app like this:

                                                            Figure 1: The Error Message

Follow the steps below carefully and simultaneously one after the other:

  1. Look for any external module attached to your Arduino board using jumper wires. If so, remove those connections completely and press the reset button on the Arduino circuit board. The attached modules can be one of the following: Micro SD Card module, Bluetooth module, etc.
  2. Remove pin connections, if any from the TX and RX pin-slots in the Arduino board. These pre-attached pins can cause unnecessary signal transfers which can hinder and make the actual port of Arduino board busy.
  3. Before connecting the Arduino to the Android device, go to the drop down menu in the app at the top-right corner -> Settings -> Board Type -> Arduino -> UNO
  4. Now, you need to code a sketch and make it ready for compile and upload to the circuit board. We will use a basic example sketch for this case. Feel free to try out your own custom coded Arduino sketches. Go to the drop-down menu -> Sketch -> Examples -> Basics -> AnalogReadSignal
  5. Don’t compile the sketch yet because we haven’t connected any Arduino circuit board to our Android device. So first, connect the Arduino circuit board to the Android device through the OTG cable connected to the Arduino-USB cable.
  6. You should see some LEDs lit up on the circuit board (indicates power is flowing to the board). Go ahead to compile the sketch. Click the ‘lightning’ icon on the top in the toolbar of the app. You should see the code/sketch getting compiled. Once done you should see a toast message saying “Compilation finished”. This signifies that your code/sketch has been verified by the compiler.

                                              Figure 2: Successful Compilation of sketch

This process is inevitable and there is hardly any issue while compiling a sketch.

       7. Upload the sketch: Click on the upload icon from the toolbar in the app. Upload             should start once you get a pop-up dialog like this:

                                           Figure 3: Arduino board detected successfully

Once you click Okay, the upload shall start and if your code is correct and matches the particular Arduino circuit board, you shall get a successful upload, which was not the case earlier for the error : “no Arduino boards found” on clicking the upload button.

So, that’s it then. Hope this blog adds value to your development skills and you can continue working bug free with your Android-Arduino connections.

Resources:

  1. Author – Nick Gamon, Article – Have I bricked my Arduino uno problems with uploading to board, Date – Nov’16 2016, Website – https://arduino.stackexchange.com/questions/13292/have-i-bricked-my-arduino-uno-problems-with-uploading-to-board
  2. Author – Arduino Products, Article – Arduino boards, Website – https://www.arduino.cc/en/Main/Boards

3. Author – Anton Smirnov, App name – ArduinoDroid, Website – https://play.google.com/store/apps/details?id=name.antonsmirnov.android.arduinodroid2&hl=en_IN

Tags: FOSSASIA, Neurolab, GSOC19, Open-source, Arduino, Serial terminal

Continue Reading

Open is Becoming the New Common Foundation across Business, Government, Science, and Industry

Interview with Shanker V Selvadurai, Vice President & Chief Technology Officer of Cloud and Cognitive Software for IBM Asia Pacific

Could you briefly introduce yourself?

I am Shanker Selvadurai and currently the Vice President & Chief Technology Officer of Cloud and Cognitive Software for IBM Asia Pacific. I am based in Singapore and lead the technical organization that helps clients across Asia Pacific to explore and co-create cloud-based solutions that leverage data, analytics and artificial intelligence (AI) capabilities to deliver better decisions and outcomes.

I joined IBM in 2006.  Prior to IBM, I held key leadership positions in areas of research, development, consulting, sales and marketing with technology companies like AT&T, NCR and Fujitsu as well as start-up BlueGill Technologies. During this period I lead teams varying in size from 6 to over 1,000 while being based in North America, Europe and Asia.

I have a Bachelor of Science degree and a Master of Business Administration. I am also an Open Group Certified Distinguished Architect. Besides having published international patents/papers, I have actively contributed to international technology standards committees that include the IFX Forum, OFX Consortium and the Microsoft Advisory Council. I was also an adjunct lecturer at the Singapore Management University, teaching courses related to Services Science and Advanced Business Technology.

Tell us about your session at the FOSSASIA Summit, what will you cover?

At the FOSSASIA Summit, I am participating in a panel on “Business, Government, Science – What Opportunities Does “Open” Bring to Society”.  I hope to share IBM’s involvement in the open source movement, how businesses like IBM benefit from open source, as well as share thoughts about approaching open contribution and open governance in the future.

Shanker V Selvadurai handing over IBM Cloud Prize for Hackathon Winners

Business, Government, Science – What Opportunities Does “Open” Bring to Society from your point of view?

Open is becoming the new common foundation across business, government, science, and industry today.  For example, companies that still compete head-to-head in the marketplace are coming together to collaborate in open source communities.  They contribute to open source software and use it in their own IT systems and applications. They gain a competitive advantage — even though they may be helping their competitors in the short run.  The data demonstrates that companies with open source programs see more benefits from open source code and community participation.

IBM Connecting with Open Source Community at FOSSASIA Summit
International IBM Team Participating in FOSSASIA Summit

What is the role of Free Open Source Software in cloud and AI solutions in IBM?

Our offering portfolio, especially for our growth initiatives such as cloud and AI, is based on a solid foundation of open technologies.  Most of our strategic initiatives are founded on open source projects or communities, and we work across a wide variety of internal stakeholders to ensure that that the contributions we make to the community also provide greater value to our clients.

IBM Training on Open Source Cloud and AI Technologies at FOSSASIA OpenTechSummit
Many Open Source Developers Connect with IBM At the Booth

What was your motivation to work for IBM and to participate in the FOSSASIA Summit?

IBM has a long history as a leader in, and supporter of open source communities, most notably in the Apache, Linux, and Eclipse Foundations.  I joined IBM to help continue this tradition and I am looking forward to discussing new ideas to help build the future of open source at FOSSASIA Summit.

Which FOSS projects in the area of cloud and AI can interested developers contribute in IBM? In which domains could you use some help?

IBM believes that communities with open governance and an inclusive philosophy will attract the largest ecosystems and markets.  For a listing of some of the top open source projects that IBM believes represent significant opportunity areas, I would like to share with information how IBM supports high-impact open source projects here.

Participants from Around the World Following IBM Keynote on FOSS Collaboration

Which new features can we expect in IBM cloud and AI this year? Do you plan to release any new projects as FOSS?

Most of our strategic initiatives today are founded on open source projects or communities (Cloud Native Computing, Hyperledger, CD Foundation, etc.), and we work across a wide variety of internal stakeholders to ensure that that the contributions we make to the community also provide greater value to our clients. For a specific example of open source innovation from IBM, please check out the Egeria Project, which was founded in part by IBM.

Shanker V Selvadurai with Panelists and FOSSASIA Founder Ms. Hong Phuc Dang at the Singapore Summit 2019

As well, at the FOSSASIA Summit in Singapore, IBM shares the “Call for Code Global Challenge, which IBM is the Founding Partner.  

This multi-year global initiative rallies developers to create practical, effective, and high-quality applications based on cloud, data, and artificial intelligence that can have an immediate and lasting impact on humanitarian issues.  Call for Code brings startup, academic, and enterprise developers together and inspires them to solve the most pressing societal issues of our time. Building on the success of the 2018 competition, the 2019 Call for Code Global Challenge again asks developers to create solutions that significantly improve preparedness for natural disasters and accelerate relief when they hit. This year’s challenge introduces an emphasis on individual health and community wellbeing. This includes solutions that can reduce the risk of disease, improve access to data and the availability of resources, and address the mental health needs of those impacted before, during, and after disasters.

Hands-on Training with IBM Cloud and AI Experts at FOSSASIA Summit
Hands-on Training with IBM Cloud and AI Experts at FOSSASIA Summit
Continue Reading

Enforcing Constraints Throughout a Flask Back-End

Recently it was discovered that Open Event Server does not validate attendees’ tickets. Specifically, it was possible to create an arbitrary number of attendees who’d be attending an event on the same ticket! To fix this, a constraint had to be set up across different layers of Open Event Server, which is based on Flask and Postgres. This post will demonstrate how the constraint was added in the server, and these steps should apply in general to any Flask-based server with a relational back-end.

First of all, the immediate idea that comes after investigating such an issue, is to add a UNIQUE constraint to the database. For this specific case, the problem was in ticket_holders table of the Open Event database. There was originally no check imposed on the ticket_id and event_id columns.

As can be seen in the ticket_holders schema (using the \d+ ticket_holders command), there is no mention of uniqueness on either column. The initial guess was that the combination of ticket_id and event_id should be unique throughout the table to avoid multiple holders attending on the same ticket. However,imposing uniqueness on just the ticket_id column would’ve also worked. So, to be on the safer side, I moved ahead by adding uniqueness on both the columns.

To fix this, we need to make changes to the ticket_holder model. So, in the ticket_holder model file, we add a __table_args__ attribute to the TicketHolder class. This attribute represents the various constraints imposed on the ticket_holders table:

class TicketHolder(db.Model):
    __tablename__ = "ticket_holders"
    __table_args__ = (
db.UniqueConstraint('ticket_id', 'event_id', name='ticket_event'),
) # this is the constraint we add

    id = db.Column(db.Integer, primary_key=True)
    firstname = db.Column(db.String, nullable=False)
    lastname = db.Column(db.String, nullable=False)






The TicketHolder class has attributes named ticket_id and event_id, so to add a unique constraint over them, we pass their names to the UniqueConstraint constructor. Also, any suitable name can be given to the constraint, I chose ‘ticket_event’ to simply emphasize the relationship. Now that we’ve edited the database model file, we have to perform a database migration.

Before we command the migration, we have to remove the entries that potentially violate the constraint we just imposed. As a temporary fix, I connected to the database and deleted all non-unique rows via plain SQL. For a more consistent fix, I will implement this simple deletion code in the database migration file, if need be. So, once the non-unique rows are gone, we perform the database migration as follows:

$ python manage.py db migrate

And then,

$ python manage.py db upgrade

These commands may be different for different projects, but their purpose is the same – to update the database. The upgrade command generates a migration file which looks as follows:

from alembic import op
import sqlalchemy as sa
import sqlalchemy_utils


# revision identifiers, used by Alembic.
revision = '9d21de792967'
down_revision = '194a5a2a44ef'


def upgrade():
op.create_unique_constraint('ticket_event', 'ticket_holders', ['ticket_id', 'event_id'])

def downgrade():
op.drop_constraint('ticket_event', 'ticket_holders', type_='unique')

We can see that the upgrade() function has the command for adding our constraint. Once the database has been upgraded, we can revisit the schema of ticket_holders table (using the \d+ ticket_holders command again). Now we can see that our constraint is added very well in the table schema.

Now, if one tries to create multiple attendees that attend on the same ticket, s/he gets a 500 server error. Here are the related server logs:

2018-06-05 22:04:03.824 IST [46705] ERROR:  duplicate key value violates unique constraint "ticket_event"
2018-06-05 22:04:03.824 IST [46705] DETAIL:  Key (ticket_id, event_id)=(2, 6) already exists.
2018-06-05 22:04:03.824 IST [46705] STATEMENT:  UPDATE ticket_holders SET event_id=6 WHERE ticket_holders.id = 16
127.0.0.1 - - [05/Jun/2018 22:04:03] "POST /v1/attendees HTTP/1.1" 500 -
INFO:werkzeug:127.0.0.1 - - [05/Jun/2018 22:04:03] "POST /v1/attendees HTTP/1.1" 500 -

To get a more graceful error, we also need to make changes in the API schema. This will also allow to validate the data before it gets to the database. So, in the attendees.py file, we need to add a check. This check should extract the ticket and event ids from the data posted and see whether there is already an attendee in the database attending that event on the same ticket. If such an attendee is discovered, the check should raise an error and report it back to the API caller. The suitable place for this check is the before_post() method of the AttendeeListPost class. In any Flask app serving a REST API, such a method (perhaps of a different name) should exist in the API file corresponding to a model. Our check looks like the following within the before_post() method:

from flask_rest_jsonapi import ResourceList
from app.api.helpers.exceptions import ConflictException
from app.models import db
from app.models.ticket_holder import TicketHolder






class AttendeeListPost(ResourceList):
"""
List and create Attendees through direct URL
"""

def before_post(self, args, kwargs, data):
"""
Before post method to check for required relationship and proper permissions
:param args:
:param kwargs:
:param data:
:return:
"""
require_relationship(['ticket', 'event'], data)







if db.session.query(TicketHolder.id).filter_by(
ticket_id=int(data['ticket']), event_id=int(data['event'])
).scalar() is not None:
raise ConflictException(
{'pointer': '/data/attributes/ticket_id'},
"Attendee with this ticket already exists for the same event"
)

Once this check is implemented, we’re all good to go. Now, if an attendee is created that maps to a ticket belonging to an already existing attendee, the following error is sent back to the API caller:

{
"errors": [
{
"status": 409,
"source": {
"pointer": "/data/attributes/ticket_id"
},
"title": "Conflict",
"detail": "Attendee with this ticket already exists for the same event"
}
],
"jsonapi": {
"version": "1.0"
}
}

This completes our work of enforcing this constraint throughout our Flask server. This leads to a more consistent database and potentially avoids confusion at actual events!

Resources:

Continue Reading

Adding Port Specification for Static File URLs in Open Event Server

Until now, static files stored locally on Open Event server did not have port specification in their URLs. This opened the door for problems while consuming local APIs. This would have created inconsistencies, if two server processes were being served on the same machine but at different ports. In this blog post, I will explain my approach towards solving this problem, and describe code snippets to demonstrate the changes I made in the Open Event Server codebase.

The first part in this process involved finding the source of the bug. For this, my open-source integrated development environment, Microsoft Visual Studio Code turned out to be especially useful. It allowed me to jump from function calls to function definitions quickly:

Screen Shot 2018-08-10 at 12.29.01 PM

I started at events.py and jumped all the way to storage.py, where I finally found out the source of this bug, in upload_local() function:

def upload_local(uploaded_file, key, **kwargs):
    """
    Uploads file locally. Base dir - static/media/
    """
    filename = secure_filename(uploaded_file.filename)
    file_relative_path = 'static/media/' + key + '/' + generate_hash(key) + '/' + filename
    file_path = app.config['BASE_DIR'] + '/' + file_relative_path
    dir_path = file_path.rsplit('/', 1)[0]
    # delete current
    try:
        rmtree(dir_path)
    except OSError:
        pass
    # create dirs
    if not os.path.isdir(dir_path):
        os.makedirs(dir_path)
        uploaded_file.save(file_path)
        file_relative_path = '/' + file_relative_path
    if get_settings()['static_domain']:
        return get_settings()['static_domain'] + \
    file_relative_path.replace('/static', '')
    url = urlparse(request.url)
    return url.scheme + '://' + url.hostname + file_relative_path

Look closely at the return statement:

return url.scheme + '://' + url.hostname + file_relative_path

Bingo! This is the source of our bug. A straightforward solution is to simply concatenate the port number in between, but that will make this one-liner look clumsy – unreadable and un-pythonic. We therefore use Python string formatting:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path)

But this statement isn’t perfect. There’s an edge case that might give unexpected URL. If the port isn’t originally specified, Python’s string formatting heuristic will substitute url.port with None. This will result in a URL like http://localhost:None/some/file_path.jpg, which is obviously something we don’t desire. We therefore append a call to Python’s string replace() method: replace(‘:None’, ”)

The resulting return statement now looks like the following:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path).replace(':None', '')

This should fix the problem. But that’s not enough. We need to ensure that our project adapts well with the change we made. We check this by running the project tests locally:

$ nosetests tests/unittests

Unfortunately, the tests fail with the following traceback:

======================================================================
ERROR: test_create_save_image_sizes (tests.unittests.api.helpers.test_files.TestFilesHelperValidation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 138, in test_create_save_image_sizes
resized_width_large, _ = self.getsizes(resized_image_file_large)
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 22, in getsizes
im = Image.open(file)
File "/usr/local/lib/python3.6/site-packages/PIL/Image.py", line 2312, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/open-event-server:5000/static/media/events/53b8f572-5408-40bf-af97-6e9b3922631d/large/UFNNeW5FRF/5980ede1-d79b-4907-bbd5-17511eee5903.jpg'

It’s evident from this traceback that the code in our test framework is not converting the image url to file path correctly. The port specification part is working fine, but it should not affect file names, they should be independent of port number. The files saved originally do not have port specified in their name, but the code in test framework is expecting port to be involved, hence the above error.

Using the traceback, I went to the code in the test framework where this problem occurred:

def test_create_save_image_sizes(self):
       with app.test_request_context():
           image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'

           image_sizes_type = "event"
           width_large = 1300
           width_thumbnail = 500
           width_icon = 75
           image_sizes = create_save_image_sizes(image_url_test, image_sizes_type)

           resized_image_url = image_sizes['original_image_url']
           resized_image_url_large = image_sizes['large_image_url']
           resized_image_url_thumbnail = image_sizes['thumbnail_image_url']
           resized_image_url_icon = image_sizes['icon_image_url']

           resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
           resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large.split('/localhost')[1]
           resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail.split('/localhost')[1]
           resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon.split('/localhost')[1]

           resized_width_large, _ = self.getsizes(resized_image_file_large)
           resized_width_thumbnail, _ = self.getsizes(resized_image_file_thumbnail)
           resized_width_icon, _ = self.getsizes(resized_image_file_icon)

           self.assertTrue(os.path.exists(resized_image_file))
           self.assertEqual(resized_width_large, width_large)
           self.assertEqual(resized_width_thumbnail, width_thumbnail)
           self.assertEqual(resized_width_icon, width_icon)

 

Obviously, resized_image_url.split(‘/localhost’)[1] will involve port number. So we have to change this line. But this means we also have to change the subsequent lines involving thumbnail, icon and large images. Instead of stripping the port for each of these, we can simply do this collectively at an earlier stage. So we redefine the image_sizes dictionary after the create_save_image_sizes() function call:

image_sizes = {
url_name: urlparse(image_sizes[url_name]).path
for url_name in image_sizes
} # Now file names don't contain port (this gives relative urls).

Now we can simplify the lines each of which earlier required port-stripping code:

resized_image_file = app.config.get('BASE_DIR') + resized_image_url
resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large
resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail
resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon

We now do a similar modification in test_create_save_resized_image() test method as it also involves URL to file path conversion. We break the line

resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]

to 2 lines:

resized_image_path = urlparse(resized_image_url).path
resized_image_file = app.config.get('BASE_DIR') + resized_image_path

Now let’s run the tests (which failed earlier) again:

Screen Shot 2018-08-10 at 12.29.15 PM.pngFinally, the tests pass without errors! Now, we can add some extra convenience functionality: we can also strip the port when it corresponds with the protocol we’re using. For example, if we’re using https protocol, then we need not specify the port if it is 443, as 443 corresponds to that protocol. We can add this functionality by creating a mapping of such correspondence and checking for it before generating the URL. To do this, we now go back to storage.py and add the following:

SCHEMES = {80: 'http', 443: 'https'}

And add

# No need to specify scheme-corresponding port
port = url.port
if port and url.scheme == SCHEMES.get(url.port, None):
    port = None

just before

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=port,
file_relative_path=file_relative_path).replace(':None', '')

And that finishes our work! The tests again pass successfully, plus on top of that we have this new functionality of mentioning ports only when they don’t correspond with the URL scheme!


Resources:

Continue Reading

Publish an Open Source app on Fdroid

Fdroid is a famous software repository hosted with numerous free and open source Android apps. They have a main repository where they allow developers hosting free and ad free software after a thorough check up on the app. This blog will tell you how to get your project hosted in their repository using steps I followed to publish the PSLab Android app.

Before you get started, make sure you have the consent from your developer community to publish their app on Fdroid. Fdroid requires your app to use all kind of open resources to implement features. If there is any closed source libraries in your app and you still want to publish it on Fdroid, you may have to reimplement that feature by any other mean without using closed source resources. They will also not allow to have Google’s proprietary “play-services” in your app along with proprietary ad services. You can find the complete inclusion policy document from their official page.

When your app is fully ready, you can get started with the inclusion procedure. Unlike how we are publishing apps on Google Play, publishing an app on Fdroid is as simple as sending a pull request to their main repository. That’s exactly what we have to do. In simple terms all we have to do is:

  1. Fork the Fdroid main data repository
  2. Make changes to their files to include our app
  3. Do a pull request

First of all you need a GitLab account as the Fdroid repository is hosted in GitLab. Once you are ready with a GitLab account, fork and clone the f-droid-data repository. The next step is to install the fdroid-server. This can be simply done using apt:

$ sudo apt install fdroidserver

 
Once that is done, go into the directory where you cloned the repository and run the following command to check if the initiation is complete.

$ fdroid init

 
Then run the following command to read current meta data where it saves all the information related to existing apps on Fdroid;

$ fdroid readmeta

 
This will list out various details about the current meta files. Next step is to add our app details into this meta file. This can be done easily using following command or you can manually create folders and files. But the following is safer;

$ fdroid import --url https://github.com/fossasia/pslab-android --subdir app

 
Replace the link to repository from the –url tag in the above command. For instance the following will be the link for fossasia-phimpme android;

$ fdroid import --url https://github.com/fossasia/phimpme-android --subdir app

 
This will create a file named as “org.fossasia.pslab” in the metadata directory. Open up this text file and we have to fill in our details.

  1. Categories
  2. License
  3. Web Site
  4. Summary
  5. Description

Description needs to be terminated with a newline and a dot to avoid build failures.

Once the file is filled up, run the following command to make sure that the metadata file is complete.

$ fdroid readmeta

 
Then run the following command to clean up the file

$ fdroid rewritemeta org.fossasia.pslab

 
We can automatically add version details using the following command:

$ fdroid checkupdates org.fossasia.pslab

 
Now run the lint test to see if the app is building correctly.

$ fdroid lint org.fossasia.pslab

 
If there are any errors thrown, fix them to get to the next step where we actually build the app:

$ fdroid build -v -l org.fossasia.pslab

 
Now you are ready to make the pull request which will then get reviewed by developers in Fdroid community to get it merged into their main branch. Make a commit and then push to your fork. From there it is pretty straightforward to make a pull request to the main repository. Once that is done, they will test the app for any insecurities. If all of them are passed, the app will be available in Fdroid!

Reference:

  1. Quick Start: https://gitlab.com/fdroid/fdroiddata/blob/master/README.md#quickstart
  2. Making merge requests: https://gitlab.com/fdroid/fdroiddata/blob/master/CONTRIBUTING.md#merge-requests
Continue Reading

Variable Font Size Badgeyay

Badgeyay is a simple badge generator that aims for promoting an open-source tool for generation of badges in PDF format. The project has options to choose from predefined set of images or upload a background image. User can choose from set of fonts and color of the same. But now Badgeyay also has option to choose custom font-size in generation of badges.

To implement font size feature,  first, the component that is determining the font of the label has to be identified. The label that determines the text on the badge is the <text> label and within it, the label that determines the properties of the text is <tspan>. So mainly we need to alter the properties in the tspan.

The property that determines the font size for the badge is font-size and its default value is set to 31.25 px. If the property in the labels changed, then we can see the corresponding changes in the PDF generated from the svg.

Now the challenges were:

  • To Determine the font value from the frontend.
  • Using the same for the font-config.
  • Changing the built svg accordingly.

Procedure

  1. Firstly frontend component has to be changed to incorporate a slider to give input for the variable font size. So a range input is inserted with range from 15 px to 45 px and default as 30 px. The size_print label gets changed dynamically to show the value selected from the range slider.
<li>
<input type="radio" name="fontsize" id="font-size-picker"> Choose font size
</li>
<section id="font-size-input" style="display:none;">
<label for="inputFile" id="size_print"></label>
<div>
<input type="range" id="font-size" max=45 min=15 step=5 value=30  class="form-control" name="font_size">
</div>
</section>
  1. After adding the component, form script is changed to add toggle behaviour to the button. For adding the toggling behaviour in the component, checkbox is used and the value of the label is updated dynamically as the slider value is changed.
$("#size_print").text($("#font-size").val() + " px");

      $("#font-size-picker").click(function () {

          if ($(this).is(":checked")) {

              $("#font-size-input").css("display", "block");

          } else {

              $("#font-size-input").css("display", "none");

          }

      });

      $("#font-size").on('input', function () {

          $("#size_print").text($(this).val() + " px");

      });
  1. After completing the work on the frontend, it is necessary to modify the backend too. The method for choosing custom font has to be refactored. It now checks whether the custom font is set or font size variable is set, and creates a config file for fonts which after use gets deleted.
font_config = {}
   # custom font is specified
   if custom_font != '':
       font_config['font'] = custom_font
   if font_size != '':
       font_config['font_size'] = font_size
   if custom_font != '' or font_size != '':
       json_str = json.dumps(font_config)
       print(json_str)
       f = open(os.path.join(app.config['UPLOAD_FOLDER'], 'fonts.json'), "w+")
       f.write(json_str)
       f.close()
  1. The generator class is modified as well to accommodate the changes, by adding a new class attribute called font_size. We find the keys in the dict object loaded from the file and assign the same to class attribute.
if 'font_size' in self.DATA.keys():
               self.font_size = self.DATA['font_size']
  1. Make the necessary change in the svg, so that font size change can be represented in the generated PDF. Replace the old font size with the new font size specified.
if self.font_size:
           content = content.replace("font-size:31.25px",
                                     "font-size:" + str(self.font_size) + "px")
  1. After all the changes, badge generated will have a different font size.

The Pull request for the above change is at this Link

Topics Involved

Working on this Issue (Link) involve following topics:

  • SVG Label manipulation
  • Sending data from Ember frontend to Backend.
  • Javascript for the toggle radio button.

References

  • Extracting map information from the SVG (Link)
  • Python Documentation for class (Link)
  • About Github Pages- (Link)
  • Ajax Serialize method to serialize the form contents – (Link)
Continue Reading

What is Open Source and why you should do it?

Since Codeheat is going on and Google Code-in has started, I would like to share some knowledge with the new contributors with the help of this blog.

What is an Open Source software?

When googled, you will see:

“Open-source software is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change, and distribute the software to anyone and for any purpose.”

To put it in layman terms, “A software whose source code is made available to everyone to let them change/improve provided that the contributor who changes the code cannot claim the software to be his own.”

Thus, you don’t own the software thoroughly. All you can do is change the code of the software to make it better. Now, you may be thinking what’s there in for you? There are all pros according to me and I have explained them in the latter half of this article.

Why am I writing this?

I was just in the freshman’s year of my college when I came to know about the web and how it works. I started my journey as a developer, building things, started doing some projects and keeping it with myself. Those days,  exploring more, I first came to know about the Open Source software.

Curiously, wanting to know more about the same, I got to know that anyone can make his/her software Open so as to make it available to others for use and development. Thus, learning more about the same led me to explore other’s projects on GitHub and I went through the codebases of the softwares and started contributing. I remember my first contribution was to correct a “typo” i.e correcting a spelling mistake in the README of the project. That said, I went on exploring more and more and got my hands on Open Source which made me share some of my thoughts with you.

What’s there in for you doing Open Source Contribution?

1) Teaches you how to structure code:

Now a days, nearly many of the software projects are Open Sourced and the community of developer works on the projects to constantly improve them. Thus, big projects have big codebases too which are really hard to understand at first but after giving some time to understand and contribute, you will be fine with those. The thing with such projects is they have a structured code, by “structured”, I mean to say there are strict guidelines for the project i.e they have good tests written which make you write the code as they want, i.e clean and readable. Thus, by writing such code, you will learn how to structure it which ultimately is a great habit that every developer should practice.

2) Team Work:

Creating and maintaining a large project requires team work. When you contribute to a project, you have to work in a team where you have to take others opinions, give your opinions, ask teammates for improvisations or ask anything whichever you are stuck with. Thus, working in team increases productivity, community interaction, your own network, etc.

3) Improves the developer you:

Okay, so I think, one of the most important part of your developer journey is and should be “LEARNING ALWAYS”. Thus, when you contribute, your code is reviewed by others (experts or maintainers of project) who eventually point out the mistakes or the improvisations to be done in the code so that the code can be written much cleaner than you had written. Also, you start to think a problem widely. While solving the problem, you ensure that the code you have written makes the app scalable for a large number of users, also prolonging the life of code.

4) Increases your Network:

One advantage of Open Source contribution is that it also increases your network in the developer community. Thus, you get to know about the things that you have never heard of, you get to explore them, you get to meet people, you get to know what is going in what parts of the world, etc. Having connections with other developers sitting in different countries is always a bonus.

5) Earn some bucks too:

At the end of the day, money matters. Earlier days, people used to think that contributing to Open Source projects won’t earn you money, etc. But if you are a maintainer or a continuous contributor of a great project, you get donations to get continuing the project and making it available to people.

For students in college, doing Open Source is a bonus. There are programmes like:

These programmes offer high incentives and stipends to the fellow students. FOSSASIA participates in GSoC so you can go ahead and try getting in GSoC under FOSSASIA.

6) Plus point for job seekers:

When it comes to applying for job, if you have a good Open Source profile, the recruiter finds a reason to take you out and offer you an interview since you already know how to “manage a project”, “work in team”, “get work done”, “solve a problem efficiently”, etc. Now a days, many companies mention on their job application page as “Open Source would be a bonus”.

7) Where can you start:

We have many projects at FOSSASIA to start with. There are no restrictions on the language since we have projects available for most of the languages.

Currently, we are having a couple of programs open at FOSSASIA. They are:

Feel free to check out the programs and the projects under FOSSASIA at https://github.com/fossasia.

Conclusion

So, yeah. This was it. Hope you understood what Open Source is and how would it benefit you. Keep contributing to FOSSASIA and you will see the effects in no time.

Continue Reading
Close Menu
%d bloggers like this: