Saving Sensor Data in CSV format

PSLab Android app by FOSSASIA provides a variety of features to its users. One of them is accessing various types of sensors both built into mobile phone and external sensors connected with PSLab device. In earlier versions users were only able to view the captured data. Moving forward, adding improvements to the app, now there is a feature to save those data displayed in graphs in csv format.

This feature is important in many ways. One is educational. In a classroom, teachers can ask students to perform an experiment and prepare a report using the data collected. By just visualizing they cannot do this. Actual data points must be made available. Another use is sharing data sets related to say environmental data over different demographics.

CSV, or comma-separated values file is a text file where stored data are separated by commas. The file stores these tabular data (numbers and text) in plain text format. Each line of the file represents a data record. Each data record consists of one or more fields, separated by commas. CSV files are commonly used to store sensor data because of its easy use. This post is about how PSLab device uses CSV file to write sensor data in it.

In PSLab android source code, there is a dedicated class to handle read sensor data from different instruments called “CSVLogger”. Developers can easily instantiate this class wherever they want a data logging as follows;

CSVLogger logger = new CSVLogger(<SUBFOLDER>); 
logger .writeCSVFile("Heading1,Heading2,Heading3\n");

 
This will create a blank folder in “PSLab” folder in device storage.  The CSV file is generated with the following convention according to the date and time where data is saved in the file.

yyyymmdd-hhmmss.csv

A sample file would have a name like 20180710-07:30:28.csv inside the SUBFOLDER which is specific to each instrument. Folder name will be the one used when initiating the CSVLogger.

With this method, logging data is pretty easy. Simply create a string which is a comma seperated and ended with a new line character. Then simply call the writeCSVFile(data) method with the string as a parameter added to it. It will keep appending string data until a new file is created. File creation can be handled by developers at their own interests and preferences.

String data = String.valueOf(System.currentTimeMillis()) + "," + item.getX() + "," + item.getY() + "\n";
logger.writeCSVFile(data);

 

To bring out an example let’s view how it’s implemented in Lux Meter instrument. This is a good source one can refer to when adding this feature in fragments

inside a main activity. In Lux Meter, there is the parent activity named Lux Meter and inside that there are two fragments, one is fragmentdata and the other one is fragmentsettings. Data capturing and saving occurs inside fragmentdata.

Menu icon controlling happens in the parent activity and we have bound a variable across the main activity and child fragment as follows;

LuxMeterActivity parent = (LuxMeterActivity) getActivity();
if (parent.saveData) {/* Save Data */}

 
This makes it easier listening menu icon clicks and start/stop recording accordingly. How to handle menu icons is beyond the scope of this blog and you can find tutorials on how to do that in the Resources section at the bottom of this blog post.

Once these CSV files are available, users can easily integrate them with advanced software like Matlab or Octave to do further analysis and processing to captured data sets.

Resources:

  1. CSV Logger: https://github.com/fossasia/pslab-android/blob/development/app/src/main/java/org/fossasia/pslab/others/CSVLogger.java
  2. Android Menu options: https://stackoverflow.com/questions/27984041/android-correct-use-of-invalidateoptionsmenu
Continue Reading Saving Sensor Data in CSV format

Uploading Files via APIs in the Open Event Server

There are two file upload endpoints. One is endpoint for image upload and the other is for all other files being uploaded. The latter endpoint is to be used for uploading files such as slides, videos and other presentation materials for a session. So, in FOSSASIA’s Orga Server project, when we need to upload a file, we make an API request to this endpoint which is turn uploads the file to the server and returns back the url for the uploaded file. We then store this url for the uploaded file to the database with the corresponding row entry.

Sending Data

The endpoint /upload/file  accepts a POST request, containing a multipart/form-data payload. If there is a single file that is uploaded, then it is uploaded under the key “file” else an array of file is sent under the key “files”.

A typical single file upload cURL request would look like this:

curl -H “Authorization: JWT <key>” -F [email protected] -x POST http://localhost:5000/v1/upload/file

A typical multi-file upload cURL request would look something like this:

curl -H “Authorization: JWT <key>” -F [email protected] -F [email protected] -x POST http://localhost:5000/v1/upload/file

Thus, unlike other endpoints in open event orga server project, we don’t send a json encoded request. Instead it is a form data request.

Saving Files

We use different services such as S3, google cloud storage and so on for storing the files depending on the admin settings as decided by the admin of the project. One can even ask to save the files locally by passing a GET parameter force_local=true. So, in the backend we have 2 cases to tackle- Single File Upload and Multiple Files Upload.

Single File Upload

if 'file' in request.files:
        files = request.files['file']
        file_uploaded = uploaded_file(files=files)
        if force_local == 'true':
            files_url = upload_local(
                file_uploaded,
                UPLOAD_PATHS['temp']['event'].format(uuid=uuid.uuid4())
            )
        else:
            files_url = upload(
                file_uploaded,
                UPLOAD_PATHS['temp']['event'].format(uuid=uuid.uuid4())
            )


We get the file, that is to be uploaded using
request.files[‘file’] with the key as ‘file’ which was used in the payload. Then we use the uploaded_file() helper function to convert the file data received as payload into a proper file and store it in a temporary storage. After this, if force_local is set as true, we use the upload_local helper function to upload it to the local storage, i.e. the server where the application is hosted, else we use whatever service is set by the admin in the admin settings.

In uploaded_file() function of helpers module, we extract the filename and the extension of the file from the form-data payload. Then we check if the suitable directory already exists. If it doesn’t exist, we create a new directory and then save the file in the directory

extension = files.filename.split('.')[1]
        filename = get_file_name() + '.' + extension
        filedir = current_app.config.get('BASE_DIR') + '/static/uploads/'
        if not os.path.isdir(filedir):
            os.makedirs(filedir)
        file_path = filedir + filename
        files.save(file_path)


After that the upload function gets the settings key for either s3 or google storage and then uses the corresponding functions to upload this temporary file to the storage.

Multiple File Upload

 elif 'files[]' in request.files:
        files = request.files.getlist('files[]')
        files_uploaded = uploaded_file(files=files, multiple=True)
        files_url = []
        for file_uploaded in files_uploaded:
            if force_local == 'true':
                files_url.append(upload_local(
                    file_uploaded,
                    UPLOAD_PATHS['temp']['event'].format(uuid=uuid.uuid4())
                ))
            else:
                files_url.append(upload(
                    file_uploaded,
                    UPLOAD_PATHS['temp']['event'].format(uuid=uuid.uuid4())
                ))


In case of multiple files upload, we get a list of files instead of a single file. Hence we get the list of files sent as form data using
request.files.getlist(‘files[]’). Here ‘files’ is the key that is used and since it is an array of file content, hence it is written as files[]. We again use the uploaded_file() function to get back a list of temporary files from the content that has been uploaded as form-data. After that we loop over all the temporary files that are stored in the variable files_uploaded in the above code. Next, for every file in the list of temporary files, we use the upload() helper function to save these files in the storage system of the application.

In the uploaded_file() function of the helpers module, since this time there are multiple files and their content sent, so things work differently. We loop over all the files that are received and for each of these files we find their filename and extension. Then we create directories to save these files in and then save the content of the file with the corresponding filename and extension. After the file has been saved, we append it to a list and finally return the entire list so that we can get a list of all files.

if multiple:
        files_uploaded = []
        for file in files:
            extension = file.filename.split('.')[1]
            filename = get_file_name() + '.' + extension
            filedir = current_app.config.get('BASE_DIR') + '/static/uploads/'
            if not os.path.isdir(filedir):
                os.makedirs(filedir)
            file_path = filedir + filename
            file.save(file_path)
            files_uploaded.append(UploadedFile(file_path, filename))


The
upload() function then finally returns us the urls for the files after saving them.

API Response

The file upload endpoint either returns a single url or a list of urls depending on whether a single file was uploaded or multiple files were uploaded. The url for the file depends on the storage system that has been used. After the url or list of urls is received, we jsonify the entire response so that we can send a proper JSON response that can be parsed properly in the frontend and used for saving corresponding information to the database using the other API services.

A typical single file upload response looks like this:

{
     "url": "https://xyz.storage.com/asd/fgh/hjk/12332233.docx"
 }

Multiple file upload response looks like this:

{
     "url": [
         "https://xyz.storage.com/asd/fgh/hjk/12332233.docx",
         "https://xyz.storage.com/asd/fgh/hjk/66777777.ppt"
     ]
 }

You can find the related documentations and example payloads on how to use this endpoint to upload files here: http://open-event-api.herokuapp.com/#upload-file-upload.

 

Reference:

Continue Reading Uploading Files via APIs in the Open Event Server