Sensor Data Logging in the PSLab Android App

The PSLab Android App allows users to log data from sensors connected to the PSLab hardware device. The Connected sensors should support I2C, SPI communication protocols to communicate with the PSLab device successfully. The only prerequisite is the additional support for the particular sensor plugin in Android App. The user can log data from various sensors and measure parameters like temperature, humidity, acceleration, magnetic field, etc. These parameters are useful in predicting and monitoring the environment and in performing many experiments.

The support for the sensor plugins was added during the porting python communication library code to Java. In this post,  we will discuss how we logged real time sensor data from the PSLab Hardware Device. We used Realm database to store the sensor data locally. We have taken the MPU6050 sensor as an example to understand the complete process of logging sensor data.

Creating Realm Object for MPU6050 Sensor Data

The MPU6050 sensor gives the acceleration and gyroscope readings along the three axes X, Y and Z. So the data object storing the readings of the mpu sensor have variables to store the acceleration and gyroscope readings along all three axes.

public class DataMPU6050 extends RealmObject {

   private double ax, ay, az;
   private double gx, gy, gz;
   private double temperature;

   public DataMPU6050() {  }

   public DataMPU6050(double ax, double ay, double az, double gx, double gy, double gz, double temperature) {
       this.ax = ax;
       this.ay = ay;
       this.az = az;
       this.gx = gx;
       this.gy = gy;
       this.gz = gz;
       this.temperature = temperature;
   }

  // getter and setter for all variables
}

Creating Runnable to Start/Stop Data Logging

To sample the sensor data at 500ms interval, we created a runnable object and passed it to another thread which would prevent lagging of the UI thread. We can start/stop logging by changing the value of the boolean loggingThreadRunning on button click. TaskMPU6050 is an AsyncTask which reads each sample of sensor data from the PSLab device, it gets executed inside a while loop which is controlled by boolean loggingThreadRunning. Thread.sleep(500) pauses the thread for 500ms, this is also one of the reason to transfer the logging to another thread instead of logging the sensor data in UI thread. If such 500ms delays are incorporated in UI thread, app experience won’t be smooth for the users.

Runnable loggingRunnable = new Runnable() {
   @Override
   public void run() {
       try {
           MPU6050 sensorMPU6050 = new MPU6050(i2c);
           while (loggingThreadRunning) {
               TaskMPU6050 taskMPU6050 = new TaskMPU6050(sensorMPU6050);
               taskMPU6050.execute();
              // use lock object to synchronize threads
               Thread.sleep(500);
           }
       } catch (IOException   InterruptedException e) {
           e.printStackTrace();
       }
   }
};

Sampling of Sensor Data

We created an AsyncTask to read each sample of the sensor data from the PSLab device in the background thread. The getRaw() method read raw values from the sensor and returned an ArrayList containing the acceleration and gyro values. After the values were read successfully, they were updated in the data card in the foreground which was visible to the user. This data card acts as a real-time screen for the user. All the samples read are appended to ArrayList mpu6050DataList, when the user clicks on button Save Data, the collected samples are saved to the local realm database.

private ArrayList<DataMPU6050> mpu6050DataList = new ArrayList<>();

private class TaskMPU6050 extends AsyncTask<Void, Void, Void> {

   private MPU6050 sensorMPU6050;
   private ArrayList<Double> dataMPU6050 = new ArrayList<>();

   TaskMPU6050(MPU6050 mpu6050) {
       this.sensorMPU6050 = mpu6050;
   }

   @Override
   protected Void doInBackground(Void... params) {
       try {
           dataMPU6050 = sensorMPU6050.getRaw();
       } catch (IOException e) {
           e.printStackTrace();
       }
       return null;
   }

   @Override
   protected void onPostExecute(Void aVoid) {
       super.onPostExecute(aVoid);
       // update data card TextViews with data read.
       DataMPU6050 tempObject = new DataMPU6050(dataMPU6050.get(0), dataMPU6050.get(1), dataMPU6050.get(2),
               dataMPU6050.get(4), dataMPU6050.get(5), dataMPU6050.get(6), dataMPU6050.get(3));
       mpu6050DataList.add(tempObject);
       synchronized (lock) {
           lock.notify();
       }
   }
}
Source: PSLab Android App

There is an option for Start/Stop Logging, clicking on which will change the value of boolean loggingThreadRunning which stops starts/stops the logging thread.

When the Save Data button is clicked, all the samples of sensor data collected from the  PSLab device till that point are saved to the local realm database.

realm.beginTransaction();
for (DataMPU6050 tempObject : mpu6050DataList) {
   realm.copyToRealm(tempObject);
}
realm.commitTransaction();

Data can also be written asynchronously to the local realm database. For other methods to write to a real database refer write section of Realm docs.

Resources

Continue ReadingSensor Data Logging in the PSLab Android App

Multithreading implementation in Loklak Server

Loklak Server is a near-realtime system. It performs a large number of tasks and are very costly in terms of resources. Its basic function is to scrape all data from websites and output it at the endpoint. In addition to scraping data, there is also a need to perform other tasks like refining and cleaning of data. That is why, multiple threads are instantiated. They perform other tasks like:

  1. Refining of data and extract more data

The data fetched needs to be cleaned and refined before outputting it. Some of the examples are:

a) Removal of html tags from tweet text:

After extracting text from html data and feeding to TwitterTweet object, it concurrently runs threads to remove all html from text.

b) Unshortening of url links:

The url links embedded in the tweet text may track the users with the help of shortened urls. To prevent this issue, a thread is instantiated to unshorten the url links concurrently while cleaning of tweet text.

  1. Indexing all JSON output data to ElasticSearch

While extracting JSON data as output, there is a method here in Timeline.java that indexes data to ElasticSearch.

Managing multithreading

To manage multithreading, Loklak Server applies following objects:

1. ExecutorService

To deal with large numbers of threads ExecutorService object is used to handle threads as it helps JVM to prevent any resource overflow. Thread’s lifecycle can be controlled and its creation cost can be optimized. This is the best example of ExecutorService application is here:

.
.
public class TwitterScraper {
    // Creation of at max 40 threads. This sets max number of threads to 40 at a time
    public static final ExecutorService executor = Executors.newFixedThreadPool(40);
    .
    .
    .
    .
    // Feeding of TwitterTweet object with data
    TwitterTweet tweet = new TwitterTweet(
        user.getScreenName(),
        Long.parseLong(tweettimems.value),
        props.get("tweettimename").value,
        props.get("tweetstatusurl").value,
        props.get("tweettext").value,
        Long.parseLong(tweetretweetcount.value),
        Long.parseLong(tweetfavouritecount.value),
        imgs, vids, place_name, place_id,
        user, writeToIndex,  writeToBackend
    );
    // Starting thread to refine TwitterTweet data
    if (tweet.willBeTimeConsuming()) {
       executor.execute(tweet);
    }    .
    .
    .

 

2. basic Thread class

Thread class can also be used instead of ExecutorService in cases where there is no resource crunch. But it is always suggested to use ExecutorService due to its benefits. Thread implementation can be used as an anonymous class like here.

3. Runnable interface

Runnable interface can be used to create an anonymous class or classes which does more task than just a task concurrently. In Loklak Server, TwitterScraper concurrently indexes the data to ElasticSearch, unshortens link and cleans data. Have a look at implementation here.

Resources:

Continue ReadingMultithreading implementation in Loklak Server