Using external UART modules to debug PSLab operations

Pocket Science Lab by FOSSASIA is a compact tool that can be used for circuit analytics and debugging. To make things more interesting, this device can be accessed via the user interface using an Android app or also a desktop app. Both these apps use the UART protocol (Universal Asynchronous Receiver-Transmitter) to transmit commands to the PSLab device from mobile phone or PC and receive commands vice versa. The peculiar thing about hardware is that the developer cannot simply log data just like developing and debugging a software program. He needs some kind of an external mechanism or a tool to visualize those data packets travelling through the wires.

Figure 1: UART Interface in PSLab

PSLab has a UART interface extracted out simply for this reason and also to connect external sensors that use the UART protocol. With this, a developer who is debugging any of the Android app or the desktop app can view the command and data packets transmitted between the device and the user end application.

This  requires some additional components. UART interface has two communication related pins: Rx(Receiver) and Tx(Transmitter). We will need to monitor both these pin signals for input and output data packets. It should be kept in mind that PSLab is using 3.3V signals. This voltage level is important to mention here because if someone uses 5V signals on these pins, it will damage the main IC. There are FTDI modules available in market. FTDI stands for Future Technology Devices International which is a company name and their main product is this USB transceiver chip. These chips play a major role in electronic industry due to product reliability and multiple voltage support. PSLab uses 3.3V USB Tx Rx pins and modules other than FTDI wouldn’t support it.

Figure 2: FTDI Module from SparkFun

The module shown in Fig.2 is a FTDI module which you can simply plug in to computer and have a serial monitor interface. There are cheaper versions in shopping websites like eBay or Alibaba and they will also work fine. Both Tx and Rx pins will require two of these modules and connectivity is as follows;

PSLab [Rx Pin] → FTDI Module 1 [Rx Pin]
PSLab [Tx Pin] → FTDI Module 2 [Rx Pin]

This might look strange because everywhere we see a UART module is connected Rx → Tx and Tx → Rx. Notice that our idea is to monitor data packets. Not communicate with PSLab device directly. We want to see if our mobile phone Android app is sending correct commands to PSLab device or not and if PSLab device is transmitting back the expected result or not. This method helped a lot when debugging resistance measurement application in PSLab Android app.

Monitoring these UART data packets can be done simply using a serial monitor. You can either download and install some already built serial monitors or you can simply write a python script as follows which does the trick.

import serial

ser = serial.Serial(
    port='/dev/ttyUSB1',
    baudrate=1000000000
)

ser.isOpen()
while 1 :
    data = ''
    while ser.inWaiting() > 0:
        data += ser.read(1)

    if data != '':
        print ">>" + data

Once you are getting commands and responses, it will look like debugging a software using console.

References:

  1. PySerial Library : http://pyserial.readthedocs.io/en/latest/shortintro.html
  2. UART Protocol : https://en.wikipedia.org/wiki/Universal_asynchronous_receiver-transmitter
  3. FTDI : https://en.wikipedia.org/wiki/FTDI
Continue Reading

Adding Port Specification for Static File URLs in Open Event Server

Until now, static files stored locally on Open Event server did not have port specification in their URLs. This opened the door for problems while consuming local APIs. This would have created inconsistencies, if two server processes were being served on the same machine but at different ports. In this blog post, I will explain my approach towards solving this problem, and describe code snippets to demonstrate the changes I made in the Open Event Server codebase.

The first part in this process involved finding the source of the bug. For this, my open-source integrated development environment, Microsoft Visual Studio Code turned out to be especially useful. It allowed me to jump from function calls to function definitions quickly:

Screen Shot 2018-08-10 at 12.29.01 PM

I started at events.py and jumped all the way to storage.py, where I finally found out the source of this bug, in upload_local() function:

def upload_local(uploaded_file, key, **kwargs):
    """
    Uploads file locally. Base dir - static/media/
    """
    filename = secure_filename(uploaded_file.filename)
    file_relative_path = 'static/media/' + key + '/' + generate_hash(key) + '/' + filename
    file_path = app.config['BASE_DIR'] + '/' + file_relative_path
    dir_path = file_path.rsplit('/', 1)[0]
    # delete current
    try:
        rmtree(dir_path)
    except OSError:
        pass
    # create dirs
    if not os.path.isdir(dir_path):
        os.makedirs(dir_path)
        uploaded_file.save(file_path)
        file_relative_path = '/' + file_relative_path
    if get_settings()['static_domain']:
        return get_settings()['static_domain'] + \
    file_relative_path.replace('/static', '')
    url = urlparse(request.url)
    return url.scheme + '://' + url.hostname + file_relative_path

Look closely at the return statement:

return url.scheme + '://' + url.hostname + file_relative_path

Bingo! This is the source of our bug. A straightforward solution is to simply concatenate the port number in between, but that will make this one-liner look clumsy – unreadable and un-pythonic. We therefore use Python string formatting:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path)

But this statement isn’t perfect. There’s an edge case that might give unexpected URL. If the port isn’t originally specified, Python’s string formatting heuristic will substitute url.port with None. This will result in a URL like http://localhost:None/some/file_path.jpg, which is obviously something we don’t desire. We therefore append a call to Python’s string replace() method: replace(‘:None’, ”)

The resulting return statement now looks like the following:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path).replace(':None', '')

This should fix the problem. But that’s not enough. We need to ensure that our project adapts well with the change we made. We check this by running the project tests locally:

$ nosetests tests/unittests

Unfortunately, the tests fail with the following traceback:

======================================================================
ERROR: test_create_save_image_sizes (tests.unittests.api.helpers.test_files.TestFilesHelperValidation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 138, in test_create_save_image_sizes
resized_width_large, _ = self.getsizes(resized_image_file_large)
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 22, in getsizes
im = Image.open(file)
File "/usr/local/lib/python3.6/site-packages/PIL/Image.py", line 2312, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/open-event-server:5000/static/media/events/53b8f572-5408-40bf-af97-6e9b3922631d/large/UFNNeW5FRF/5980ede1-d79b-4907-bbd5-17511eee5903.jpg'

It’s evident from this traceback that the code in our test framework is not converting the image url to file path correctly. The port specification part is working fine, but it should not affect file names, they should be independent of port number. The files saved originally do not have port specified in their name, but the code in test framework is expecting port to be involved, hence the above error.

Using the traceback, I went to the code in the test framework where this problem occurred:

def test_create_save_image_sizes(self):
       with app.test_request_context():
           image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'

           image_sizes_type = "event"
           width_large = 1300
           width_thumbnail = 500
           width_icon = 75
           image_sizes = create_save_image_sizes(image_url_test, image_sizes_type)

           resized_image_url = image_sizes['original_image_url']
           resized_image_url_large = image_sizes['large_image_url']
           resized_image_url_thumbnail = image_sizes['thumbnail_image_url']
           resized_image_url_icon = image_sizes['icon_image_url']

           resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
           resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large.split('/localhost')[1]
           resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail.split('/localhost')[1]
           resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon.split('/localhost')[1]

           resized_width_large, _ = self.getsizes(resized_image_file_large)
           resized_width_thumbnail, _ = self.getsizes(resized_image_file_thumbnail)
           resized_width_icon, _ = self.getsizes(resized_image_file_icon)

           self.assertTrue(os.path.exists(resized_image_file))
           self.assertEqual(resized_width_large, width_large)
           self.assertEqual(resized_width_thumbnail, width_thumbnail)
           self.assertEqual(resized_width_icon, width_icon)

 

Obviously, resized_image_url.split(‘/localhost’)[1] will involve port number. So we have to change this line. But this means we also have to change the subsequent lines involving thumbnail, icon and large images. Instead of stripping the port for each of these, we can simply do this collectively at an earlier stage. So we redefine the image_sizes dictionary after the create_save_image_sizes() function call:

image_sizes = {
url_name: urlparse(image_sizes[url_name]).path
for url_name in image_sizes
} # Now file names don't contain port (this gives relative urls).

Now we can simplify the lines each of which earlier required port-stripping code:

resized_image_file = app.config.get('BASE_DIR') + resized_image_url
resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large
resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail
resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon

We now do a similar modification in test_create_save_resized_image() test method as it also involves URL to file path conversion. We break the line

resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]

to 2 lines:

resized_image_path = urlparse(resized_image_url).path
resized_image_file = app.config.get('BASE_DIR') + resized_image_path

Now let’s run the tests (which failed earlier) again:

Screen Shot 2018-08-10 at 12.29.15 PM.pngFinally, the tests pass without errors! Now, we can add some extra convenience functionality: we can also strip the port when it corresponds with the protocol we’re using. For example, if we’re using https protocol, then we need not specify the port if it is 443, as 443 corresponds to that protocol. We can add this functionality by creating a mapping of such correspondence and checking for it before generating the URL. To do this, we now go back to storage.py and add the following:

SCHEMES = {80: 'http', 443: 'https'}

And add

# No need to specify scheme-corresponding port
port = url.port
if port and url.scheme == SCHEMES.get(url.port, None):
    port = None

just before

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=port,
file_relative_path=file_relative_path).replace(':None', '')

And that finishes our work! The tests again pass successfully, plus on top of that we have this new functionality of mentioning ports only when they don’t correspond with the URL scheme!


Resources:

Continue Reading

Debugging JSON Files of Sample Events for Open Event Android using Stetho

In this blog, I will talk about data validation and debugging in Android using the Stetho-A debug bridge for Android by Facebook. Most Android applications have JSON files to populate the elements like RecyclerViews, ListViews and different types of Layouts. Stetho is a debug tool for Android which uses the well-known Chrome Developer tools as it’s User Interface. With Stetho, we can see all our incoming JSON data in spreadsheets making debugging much easier and fun. What’s more, it’s completely Open source.

Getting started with Stetho

To start with, we need to enable Stetho by adding it as one of the dependencies in the build.gradle file.

compile 'com.facebook.stetho:stetho:1.4.2'

It is already enabled in the Open Event Android app.

Now, you need to add the api endpoint for your sample in the config.json file in the project(if you are using the Android repo for debugging). The config.json file is at app/assets/

In case, you are using an Open Event generated app then there’s no need to do that.

Now connect your phone through the USB cable to your laptop and start your application.

Next, you have to browse to chrome://inspect where you will see your device in the window as shown below.

 

On clicking “Inspect”, you will be shown the Stetho debug tool interface. Something like this:

When you download the details by clicking “Yes” on the starting of the application, be sure to keep it connected to the Stetho tool in the ‘inspect’ mode.

Once the data has loaded, go to Web SQL-> default.realm. You will see tabs like:-

class_speaker, class_sponsor, class_session, class_track among others.

On clicking them, you will see well-organised tables that show all linked attributes of the class that you selected.

Sessions can be seen along with all the related information in a tabular format.

Stetho displays speakers of the event and their information in an easy-to-read tabular format.

Micro locations of the event along with related information in a tabular format.

This is how you can view all the attributes of your sample in a tabular layout and hence, debug them easily. Although, Stetho is about much more than this, this is all I will talk about in this blog.

You can read more about Stetho and it’s functionalities here.

The official repo for Stetho can be found here for the source code.

Continue Reading

Debugging Using Stetho in Open Event Android

The Open Event Android project helps event organizers to generator Apps (apk format) for their events/conferences by providing api end point or zip generated using Open Event server. In this android app data is fetched from the internet using network calls to the Open Event server and the data of the event is stored in a database. It is difficult to debug an app with so many network calls and database connectivity. A way to approach this is using  Stetho which is very helpful tool for debugging an app which deals with network calls and database.  

Stetho is Facebook’s open source project works as debug bridge for android applications which gives powerful Chrome Developers Tools for debugging android applications using Chrome desktop browser.

What can you do using stetho ?

  • Analyze network traffic
  • Inspect elements(layouts/views)  
  • View SQLite database in table format
  • Run queries on SQLite database
  • View shared preference and edit it
  • Manipulate android app from command line

Setup

1. Add Gradle dependency

To add stetho in your app add ‘com.facebook.stetho:stetho:1.5.0’ dependency in your app  module’s build.gradle file. This dependency is strictly required.

dependencies{
    compile 'com.facebook.stetho:stetho:1.5.0'
}

For network inspection add one of the following dependency according to which you will be using

'com.facebook.stetho:stetho-okhttp:1.5.0'
'com.facebook.stetho:stetho-okhttp3:1.5.0'
'com.facebook.stetho:stetho-urlconnection:1.5.0'

2. Initialize stetho

Initialize stetho in class MyApplication which extends Application class by overriding  onCreate() method. Make sure you have added MyAppication in manifest.

public class MyApplication extends Application {
    public void onCreate() {
        super.onCreate()
        Stetho.initializeWithDefaults(this);
    }
}

Stetho.initializeWithDefaults(this) initializes stetho with defaults configuration(without network inspection and more). It will be able to debug database.

Manifest file

<Application   android:name=”.MyApplication    …   />

For enabling network inspection add StethoInterceptor  in OkHttpClient

new OkHttpClient.Builder()
    .addNetworkInterceptor(new StethoInterceptor())
    .build();

Using Chrome Developer Tools

1. Open Inspect window

Run stetho initialized app on device or emulator then start Google chrome and type chrome://inspect. You will see you device with your app’s package name. Click on “inspect”2. Network Inspection

Go to Network tab. It will show all network calls. With almost all info like url(path), method, returned status code, returned data type, size, time taken etc.

You can also see preview of image and preview, response of returned json data by clicking on Name.

3. SQLite Database

Go to “Resources”  tab and select “Web SQL”. You will see database file(.db). By clicking on database file you will see all tables in that database file. And by clicking on table name you will see data in row-column format for that table.

4. Run queries on SQLite database

Same as above go to “Resources”  tab and select “Web SQL”. You will see database file(.db). By clicking on database file you will see console on right side, where you can run queries on SQLite database. Example,

SELECT * FROM tracks ;

5. Shared Preferences Inspection

Go to “Resources”  tab and select “Local Storage”. You will show all files that your app used to save key-value pairs in shared preference and by clicking on file you will see all key-value pairs.

6. Element(View/Layout) Inspection

Go to “Elements” tab. You will see top layout/view in view hierarchy. By clicking it you will see child layout/view of that layout/view. On hover on layout/view you view will be inspected in your device/emulator.

 

In this blog the most important have been put forward, but there are still  some nice stuff available,like:

  • An integration with JavaScript Console : Enables JavaScript code execution that can interact with the application
  • Dumpapp  : It allows an integration higher than the Developer Tools, enabling the development of custom plugins.

By now, you must have realized that stetho can significantly improve your debugging experience. To learn more about stetho, refer to

Continue Reading

Debugging with Node

Nodejs is a powerful Event-driven I/O server-side JavaScript environment based on V8. Debugging of Node applications is not as easy as the browser code. But the node wonderful debugger can make it easy if it is used effectively.

How to start the debugger

For debugging node applications, we have to only run a debug command. Here for a quick demonstration, I have used Open-Event-Webapp fold.js code.

Suppose there is a problem in the script file and we have to debug it. All we have to do is to run the debug command from the directory containing the file.

node debug ./fold.js

The output screen will show something like this:

25

This brings us into the debug mode. When we are in debug mode, we can try various commands like ‘n’ for next, ‘s’ for the step into a function, ‘list(k)’ where k is the number of lines of the code you want on the screen.

26

The ‘n’ command always takes us to next instruction, hence in long codebases, it is always recommended to use ‘c’ or Continue Execution command for going to the next breakpoint.

To set the breakpoint, we can use the command:

setBreakpoint() or sb()

The snapshot shows the output once the breakpoint is set. We can check the values at the breakpoint by using command repl.

4

Debugging the code correctly can save a lot of time and effort. These techniques provided by the debugger are necessary to learn.

Alternates

Continue Reading
Close Menu