PSLab Communication Function Calls

Prerequisite reading:

Communicating with Pocket Science Lab via USB and capturing and plotting sine waves

Interfacing with the hardware of PSLab, fetching the data and plotting it is very simple and straight forward. Various sensors can be connected to PSLab and data can be fetched with a simple python code as shown in the following example…

>>> from PSL import sciencelab
>>> I = sciencelab.connect()     # Initializing: Returns None if device isn't found. The initialization process connects to tty device and loads calibration values.
# An example function that measures voltage present at the specified analog input
>>> print I.get_average_voltage('CH1')
# An example to capture and plot data
>>> I.set_gain('CH1', 3) # set input CH1 to +/-4V range 
>>> I.set_sine1(1000) # generate 1kHz sine wave on output W1 
>>> x,y = I.capture1('CH1', 1000, 10) # digitize CH1 1000 times, with 10 usec interval 
>>> plot(x,y) 
>>> show()
# An example function to get data from magnetometer sensor connected to PSLab
>>> from PSL.SENSORS import HMC5883L #A 3-axis magnetometer >>> M = HMC5883L.connect() >>> Gx,Gy,Gz = M.getRaw() 

The module sciencelab.py contains all the functions required for communicating with PSLab hardware. It also contains some utility functions. The class ScienceLab() contains methods that can be used to interact with the PSLab.

After initiating this class, all the features built into the device can be accessed  using various function calls.


Capture1 : for capturing one trace

capture1(ch, ns, tg)

Arguments

  • ch  : Channel to select as input. [‘CH1′..’CH3′,’SEN’]
  • ns  :  Number of samples to fetch. Maximum 10000
  • tg   :  Time gap between samples in microseconds
#Example >>> x,y = I.capture1('CH1', 1000, 10) # digitize CH1 1000 times, with 10 usec interval

Returns : Arrays X(timestamps),Y(Corresponding Voltage values)


Capture2 : for capturing two traces

capture2(ns, tg, TraceOneRemap='CH1')

Arguments

  • ns :  Number of samples to fetch. Maximum 5000
  • tg  :  Time gap between samples in microseconds
  • TraceOneRemap :   Choose the analogue input for channel 1 (Like MIC OR SEN). It is connected to CH1 by default. Channel 2 always reads CH2.
#Example 
>>> x,y1,y2 = I.capture2(1600,1.75,'CH1') # digitize CH1 and CH2, 1600 times, with 1.75 usec interval

Returns: Arrays X(timestamps),Y1(Voltage at CH1),Y2(Voltage at CH2)


Capture4 : for capturing four taces

capture4(ns, tg, TraceOneRemap='CH1')

Arguments

  • ns:   Number of samples to fetch. Maximum 2500
  • tg :   Time gap between samples in microseconds. Minimum 1.75uS
  • TraceOneRemap :   Choose the analogue input for channel 1 (Like MIC OR SEN). It is connected to CH1 by default. Channel 2 always reads CH2, channel 3 always reads CH3 and MIC is channel 4 (CH4)
#Example
>>> x,y1,y2,y3,y4 = I.capture4(800,1.75) # digitize CH1-CH4, 800 times, with 1.75 usec interval

Returns: Arrays X(timestamps),Y1(Voltage at CH1),Y2(Voltage at CH2),Y3(Voltage at CH3),Y4(Voltage at CH4)


Capture_multiple : for capturing multiple traces

capture_multiple(samples, tg, *args)

Arguments

  • samples:   Number of samples to fetch. Maximum 10000/(total specified channels)
  • tg :   Time gap between samples in microseconds.
  • *args :   channel names
# Example 
>>> from pylab import * 
>>> I=interface.Interface() 
>>> x,y1,y2,y3,y4 = I.capture_multiple(800,1.75,'CH1','CH2','MIC','SEN') 
>>> plot(x,y1) 
>>> plot(x,y2) 
>>> plot(x,y3) 
>>> plot(x,y4) 
>>> show()

Returns: Arrays X(timestamps),Y1,Y2 …


Capture_fullspeed : fetches oscilloscope traces from a single oscilloscope channel at a maximum speed of 2MSPS

capture_fullspeed(chan, amples, tg, *args)

Arguments

  • chan:   channel name ‘CH1’ / ‘CH2’ … ‘SEN’
  • tg :   Time gap between samples in microseconds. minimum 0.5uS
  • *args :   specify if SQR1 must be toggled right before capturing. ‘SET_LOW’ will set it to 0V, ‘SET_HIGH’ will set it to 5V. if no arguments are specified, a regular capture will be executed.
# Example
>>> from pylab import *
>>> I=interface.Interface()
>>> x,y = I.capture_fullspeed('CH1',2000,1)
>>> plot(x,y)               
>>> show()

Returns: timestamp array ,voltage_value array


Set_gain : Set the gain of selected PGA

set_gain(channel, gain)

Arguments

  • channel:   ‘CH1’ , ‘CH2’
  • gain :   (0-7) -> (1x,2x,4x,5x,8x,10x,16x,32x)

Note: The gain value applied to a channel will result in better resolution for small amplitude signals.

# Example
>>> I.set_gain('CH1',7)  #gain set to 32x on CH1


Get_average_voltage : Return the voltage on the selected channel
get_average_voltage(channel_name, **kwargs)
Arguments

  • channel_name:    ‘CH1’,’CH2’,’CH3’, ‘MIC’,’IN1’,’SEN’
  • **kwargs :   Samples to average can be specified. eg. samples=100 will average a hundred readings
# Example 
>>> print I.get_average_voltage('CH4')
1.002

Get_freq : Frequency measurement on IDx. Measures time taken for 16 rising edges of input signal. returns the frequency in Hertz

get_average_voltage(channel='Fin', timeout=0.1)
Arguments

  • channel :    The input to measure frequency from. ‘ID1’ , ‘ID2’, ‘ID3’, ‘ID4’, ‘Fin’
  • timeout :   This is a blocking call which will wait for one full wavelength before returning the calculated frequency. Use the timeout option if you’re unsure of the input signal. returns 0 if timed out
# Example
>>> I.sqr1(4000,25)
>>> print I.get_freq('ID1')
4000.0

Return float: frequency


Get_states : Gets the state of the digital inputs. returns dictionary with keys ‘ID1’,’ID2’,’ID3’,’ID4’
get_states()
#Example
>>> print get_states()
{'ID1': True, 'ID2': True, 'ID3': True, 'ID4': False}

Get_state : Returns the logic level on the specified input (ID1,ID2,ID3, or ID4)
get_state(input_id)
Arguments

  • input_id :    The input channel ‘ID1’ -> state of ID1 ‘ID4’ -> state of ID4
#Example
>>> print I.get_state(I.ID1)
False

Set_state : Set the logic level on digital outputs SQR1,SQR2,SQR3,SQR4
set_state(**kwargs)
Arguments

  • **kwargs :    SQR1,SQR2,SQR3,SQR4 states(0 or 1)
#Example
>>> I.set_state(SQR1=1, SQR2=0) #sets SQR1 HIGH, SQR2 LOw, but leave SQR3,SQR4 untouched.


 

Flask-SocketIO Notifications

In the previous post I explained about configuring Flask-SocketIO, Nginx and Gunicorn. This post includes integrating Flask-SocketIO library to display notifications to users in real time.

Flask Config

For development we use the default web server that ships with Flask. For this, Flask-SocketIO fallsback to long-polling as its transport mechanism, instead of WebSockets. So to properly test SocketIO I wanted to work directly with Gunicorn (hence the previous post about configuring development environment). Also, not everyone needs to be bothered with the changes required to run it.

class DevelopmentConfig(Config):
    DEVELOPMENT = True
    DEBUG = True

    # If Env Var `INTEGRATE_SOCKETIO` is set to 'true', then integrate SocketIO
    socketio_integration = os.environ.get('INTEGRATE_SOCKETIO')
    if socketio_integration == 'true':
        INTEGRATE_SOCKETIO = True
    else:
        INTEGRATE_SOCKETIO = False

    # Other stuff

SocketIO is integrated (in development env) if the developer has set the INTEGRATE_SOCKETIO environment variable to “true”. In Production, our application runs on Gunicorn, and SocketIO integration must always be there.

Flow

To send message to a particular connection (or a set of connections) Flask-SocketIO provides Rooms. The connections are made to join a room and the message is sent in the room. So to send message to a particular user we need him to join a room, and then send the message in that room. The room name needs to be unique and related to just one user. The User database Ids could be used. I decided to keep user_{id} as the room name for a user with id {id}. This information (room name) would be needed when making the user join a room, so I stored it for every user that logged in.

@expose('/login/', methods=('GET', 'POST'))
    def login_view(self):
        if request.method == 'GET':
            # Render template
        if request.method == 'POST':
            # Take email and password from form and check if 
            # user exists. If he does, log him in.
            login.login_user(user)

            # Store user_id in session for socketio use
            session['user_id'] = login.current_user.id

            # Redirect

After the user logs in, a connection request from the client is sent to the server. With this connection request the connection handler at server makes the user join a room (based on the user_id stored previously).

@socketio.on('connect', namespace='/notifs')
def connect_handler():
    if current_user.is_authenticated():
        user_room = 'user_{}'.format(session['user_id'])
        join_room(user_room)
        emit('response', {'meta': 'WS connected'})

The client side is somewhat similar to this:

<script src="{{ url_for('static', filename='path/to/socket.io-client/socket.io.js') }}"></script>
<script type="text/javascript">
$(document).ready(function() {
    var namespace = '/notifs';

    var socket = io.connect(location.protocol + "//" + location.host + namespace, {reconnection: false});

    socket.on('response', function(msg) {
        console.log(msg.meta);
        // If `msg` is a notification, display it to the user.
    });
});
</script>

Namespaces helps when making multiple connections over the same socket.

So now that the user has joined a room we can send him notifications. The notification data sent to the client should be standard, so the message always has the same format. I defined a get_unread_notifs method for the User class that fetches unread notifications.

class User(db.Model):
    # Other stuff

    def get_unread_notifs(self, reverse=False):
        """Get unread notifications with titles, humanized receiving time
        and Mark-as-read links.
        """
        notifs = []
        unread_notifs = Notification.query.filter_by(user=self, has_read=False)
        for notif in unread_notifs:
            notifs.append({
                'title': notif.title,
                'received_at': humanize.naturaltime(datetime.now() - notif.received_at),
                'mark_read': url_for('profile.mark_notification_as_read', notification_id=notif.id)
            })

        if reverse:
            return list(reversed(notifs))
        else:
            return notifs

This class method is used when a notification is added in the database and has to be pushed into the user SocketIO room.

def create_user_notification(user, action, title, message):
    """
    Create a User Notification
    :param user: User object to send the notification to
    :param action: Action being performed
    :param title: The message title
    :param message: Message
    """
    notification = Notification(user=user,
                                action=action,
                                title=title,
                                message=message,
                                received_at=datetime.now())
    saved = save_to_db(notification, 'User notification saved')

    if saved:
        push_user_notification(user)

def push_user_notification(user):
    """
    Push user notification to user socket connection.
    """
    user_room = 'user_{}'.format(user.id)
    emit('response',
         {'meta': 'New notifications',
          'notif_count': user.get_unread_notif_count(),
          'notifs': user.get_unread_notifs()},
         room=user_room,
         namespace='/notifs')

Programmer principles

As programmers we develop our programming skills and learn something every single day. We write code and solve many troubles. But is our aim to simply write code? I am sure it is not. I think writing code just for doing it is not interesting, and it’s definitely not Open Event team’s objective. Personally, I like reading code like a poem. We should always try to eliminate bad practises and ugly code. There are a few principles how to do it. Let me share them with you now.

SOLID principle

SOLID  is a mnemonic acronym introduced by Michael Feathers, and it simply means five basic principles of object oriented programming. These principles, when applied together, make it more likely that a programmer will create a system that is easy to maintain and extend over time. They are guidelines that can be applied while working on software to remove code smells by causing the programmer to refactor the software’s source code.  It is also a part of an overall strategy of agile. So, here they are:

S – Single responsibility principle

This principle means that there should never be more than one reason for a class to change.

In other words, a class should have only one potential change in a software’s specification. You should not add everything into your class. The best practise here is to check if the logic you are introducing should be in this class or not. Responsibility is the heart of this principle, so to rephrase there should never be more than one responsibility per class. Use layers for a help. And try to divide big classes into smaller ones.

O – Open/closed principle

Software entities like classes, module and functions should be open for extension, but closed for modification.

All of them should be private by default.

To make an object behaving differently without modifying it use abstractions, or place behavior(responsibility) in derivative classes. If properties of the abstracted class need to be compared or organized together, another abstraction should handle this. This is the basis of the “keep all object variables private” argument.

L – Liskov substitution principle

Functions that use pointers or references to base classes have to be able to use objects of derived classes without knowing/alerting the correctness of a program

A great example you can find here. If you are using a method defined at a base class upon an abstracted class, the function must be implemented properly on the subtype class. A great example provided here http://williamdurand.fr/2013/07/30/from-stupid-to-solid-code/  you can find below.

“ A rectangle is a plane figure with four right angles. It has a width, and a height. Now, take a look at the following pseudo-code:

rect = new Rectangle();

rect.width  = 10;
rect.height = 20;

assert 10 == rect.width
assert 20 == rect.height

We simply set a width and a height on a Rectangle instance, and then we assert that both properties are correct. So far, so good.

Now we can improve our definition by saying that a rectangle with four sides of equal length is called a square. A square is a rectangle so we can create aSquare class that extends the Rectangle one, and replace the first line above by the one below:

rect = new Square();

According to the definition of a square, its width is equal to its height. Can you spot the problem? The first assertion will fail because we had to change the behavior of the setters in the Square class to fit the definition “

I – Interface segregation principle

Many client-specific interfaces are better than one general-purpose interface.

Implementing methods that you don’t use is not recommended in this way. The idea here is to keep your components focused and try to minimize the dependencies between them. Enforcing that principle gives you low coupling, and high cohesion.

D – Dependency inversion principle

This means that “one should depends upon abstractions, do not depend upon concretions”

Interfaces should depend on other interfaces. Don’t add concrete classes to method signatures of an interface. However, use interfaces in your class methods.

So, we can also say that rather than working with classes that are tight coupled, use interfaces. This reduces dependency on implementation specifics and makes code more reusable.

Why SOLID?

I hope all of you understand the importance of using SOLID principles in your everyday code practise. Finally, let me underline again the main arguments why you should starting following them now. The most important thing is that thanks to them you can create easy to maintain software, then you can reuse your code, and finally it helps you to test easier. Do you need anymore to be  persuaded  to do it? I think it’s that’s crucial advantages and they are enough.

Source:

https://pl.wikipedia.org/wiki/SOLID_(programowanie_obiektowe)

https://scotch.io/bar-talk/s-o-l-i-d-the-first-five-principles-of-object-oriented-design

http://williamdurand.fr/2013/07/30/from-stupid-to-solid-code/

http://www.codeproject.com/Articles/60845/The-S-O-L-I-D-Object-Oriented-Programming-OOP-Prin

Mark Notifications Read on Click

Screenshot from 2016-08-01 07:31:22

Notification has become a really important way of informing users about the various activities related to them in web apps. There are different types of notification such as web app notification, email notification, desktop notification, push notification, etc. We are going to primarily talk about web app notification and mainly about how to mark them as read.

Create Notification

Creating a notification is plain and simple. You have a json or an object which stores the notification message corresponding to a particular activity. Whenever that activity occurs in the backend, you call the send notification module, which adds the information to the database and shows it in the notification page. As simple as that.

Screenshot from 2016-08-01 07:48:08

Marking Notification as Read

The main functioning of this is plain and simple as well. You have a URL, which on getting a request from the user, marks the notification as read in the database. That’s it.

Screenshot from 2016-08-01 07:48:17

We know how to do this using a button or a link. But the question here is how to mark a notification as read on clicking any part of the notification?? The obvious answer is, well, put the entire notification inside an anchor tag and you are done, right? Well, it would work in many cases. But what if the design structure is such that this doesn’t work somehow. Somehow enclosing the notification inside a particular anchor tag doesn’t solve the purpose. What do we do then?

Identify Whether Inside a DIV

The main problem here actually is how to identify whether the click is inside the enclosing div or somewhere else. Once we solve this problem, we can send an ajax request to the mark read URL and our job is done.

Screenshot from 2016-08-01 07:52:58

So, to identify that a click is indeed inside a div, we use the event.target property of the event clicked. The target event property returns the element that triggered the event. So we check whether event.target has the “notification” class in our case. If it does not have the “notification” class we check in all it’s parent nodes. We get the parent nodes using the “parent()” function and check whether any of that has notification. If either of the 2 occurs, we consider that the click is inside the div. And thus mark the notification as read.

Screenshot from 2016-08-01 07:51:09

So, once this is done, we mark the notification as read in the backend and our job is done…

Communicating with Pocket Science Lab via USB and capturing and plotting sine waves

Design of PSLab combines the flexibility of Python programming language and the real-time measurement capability of micro-controllers.

PSLab, with its simple and open architecture allows users to use the tool for various measurements and to develop new experiments with simple functions written in python.

PSLab is interfaced and powered by USB port of the computer. For connecting external signals it has several input/output terminals as shown in the figure.

pslabdesign

Interfacing with the real world

Connecting to PSLab is as simple and straight forward as this…

>>> from PSL import sciencelab
>>> I = sciencelab.connect()     #Returns None if device isn't found
# An example function that measures voltage present at the specified analog input
>>> print I.get_average_voltage('CH1')

Various sensors can be connected to PSLab and data can be fetched with a simple python code as shown below…

>>> from PSL.SENSORS import HMC5883L #A 3-axis magnetometer
>>> M = HMC5883L.connect()
>>> Gx,Gy,Gz = M.getRaw()

The module sciencelab.py contains all the functions required for communicating with PSLab hardware. It also contains some utility functions. The class ScienceLab() contains methods that can be used to interact with the PSLab. The connect() function returns an object of this class if PSLab hardware is detected.

The initialization process does the following

* connects to tty device

* loads calibration values.

>>> from PSL import sciencelab
>>> I = sciencelab.connect()
>>> print I
<PSL.sciencelab.ScienceLab instance at 0x7fe9a7bf0e18>

After initiating this class, its various function calls will allow access to all the features built into the device. Some examples showing the use of few function calls are given below…

Example 1: Capturing and plotting a sine wave

The function call used,

capture1(self,ch,ns,tg,*args,**kwargs)

Arguments

  • ch  : Channel to select as input. [‘CH1′..’CH3′,’SEN’]
  • ns  :  Number of samples to fetch. Maximum 10000
  • tg   :  Time gap between samples in microseconds

Example Program

Connect WG1 to CH1 and run the following code.

>>> from pylab import *
>>> from PSL import sciencelab
>>> I=sciencelab.connect()
>>> I.set_gain('CH1', 3) # set input CH1 to +/-4V range
>>> I.set_sine1(1000) # generate 1kHz sine wave on output W1
>>> x,y = I.capture1('CH1', 1000, 10) # digitize CH1 1000 times, with 10 usec interval
>>> plot(x,y)
>>> show()

For running the script in IDE, one should define source code encoding, add this to the top of your script:

# -*- coding: utf-8 -*-

The output of the program is here…

sine1

Example 2 : Capturing two sine waves and plotting

The function call used,

capture2(self,ns,tg,TraceOneRemap='CH1')

Arguments

  • ns :  Number of samples to fetch. Maximum 5000
  • tg  :  Time gap between samples in microseconds
  • TraceOneRemap :   Choose the analogue input for channel 1 (Like MIC OR SEN). It is connected to CH1 by default. Channel 2 always reads CH2.

Example Program

Connect WG1 to CH1, WG2 to CH2 and run the following code.

# -*- coding: utf-8 -*-

from pylab import *
from PSL import sciencelab
I=sciencelab.connect()
I.set_gain('CH1', 2) # set input CH1 to +/-4V range
I.set_gain('CH2', 3) # set input CH2 to +/-4V range
I.set_sine1(1000) # generate 1kHz sine wave on output W1
I.set_sine2(1000) # generate 1kHz sine wave on output W2

x,y1,y2 = I.capture2(1600,1.75,'CH1') 
plot(x,y1) #Plot of analog input CH1
plot(x,y2) #plot of analog input CH2
show()

The output of the program is here…sine2

Example 3 : Capturing four traces and plotting

The function call used,

capture4(self,ns,tg,TraceOneRemap='CH1')

Arguments

  • ns:   Number of samples to fetch. Maximum 2500
  • tg :   Time gap between samples in microseconds. Minimum 1.75uS
  • TraceOneRemap :   Choose the analogue input for channel 1 (Like MIC OR SEN). It is connected to CH1 by default. Channel 2 always reads CH2.

Example Program

Connect WG1 to CH1, WG2 to CH2, SQR1 to CH3 and transducer mic to MIC (CH4) and run the following code.

# -*- coding: utf-8 -*-

from pylab import *
from PSL import sciencelab
I=sciencelab.connect()
I.set_gain('CH1', 2) # set input CH1 to +/-4V range
I.set_gain('CH2', 3) # set input CH2 to +/-4V range
I.set_sine1(1000) # generate 1kHz sine wave on output W1
I.set_sine2(1000) # generate 1kHz sine wave on output W2
I.sqr1(2000,duty_cycle=50) # generate 1kHz square wave on output SQR1

x,y1,y2,y3,y4 = I.capture4(800,1.75)
plot(x,y1) #Plot of analog input CH1
plot(x,y2) #plot of analog input CH2
plot(x,y3) #plot of analog input CH3
plot(x,y4) #plot of analog input CH4 : MIC
show()

The output of the program is here…waves

Next To Do for GSoC-16

A detailed User manual and programmers manual with description of all function calls. ( Work in progress 🙂  )

Read:
  1. Post about installing PSLab
  2. PSLab and ExpEYES and GSoC-16 work

Can solving lint bugs be interesting?

Today I am going to present you how we’ve changed monotonous solving bugs into motivating process.

PEP

Most developers need to improve their code quality. To do  that they can use style guide for e.g for Python code (PEP). PEP contains an index of all Python Enhancement Proposals.

Below you can find which logs PEP returned in a command line.

Do you think that this logs’ presentation is  good enough to interest a developer? Will he solve these  thousands of bugs?

Undoubtedly, there are much information about errors and warnings so PEP returns long logs. But developer can not even know how to start solving bugs. And even if she/he finally starts, after each commit he/she needs to run that script again to check if quantity of bugs are increased or decreased. It seems to be endless, exhausting and very monotonous.  Nobody is encouraged to do it.

logi.png

Quality monitoring

Open Event team wants to increase our productivity and code quality. Therefore we use a tool which allow us to check code style, security, duplication complexity and test coverage on every commit. That tool is Codacy and it fulfils our requirements in 100%. It is very helpful because it adds comments to pull requests and enables developer quickly find where a bug is located. It’s very comfortable, because you don’t need to check issues in above awful logs results. Take a look how it looks in Codacy.

-DO NOT MERGE  Ticketing Flow by niranjan94 · Pull Request  1927 · fossasia open event orga server.png

Isn’t it clear? Of course that it’s. Codacy shows in which line issue ocurres and which type of issue it’s.

Awesome statistics dashboard

I’d like to give an answer how you can engage your team to solve issues and make this process more interesting. On the main page codacy tool welcomes you with great statistics about your project.

open event orga server   Codacy   Dashboard

You can see number of issues, category like code complexity, code style, compatibility, documentation, error prone, performance, security and unused code. That params show in which stage of code quality your project is. I think that every developer’s aim is to have the highest code quality and increasing these statistics. But if project has many issues, developer sees only a few changes in project charts.

Define Goals

Recently I’ve discovered how you can motivate yourself more. You can define a goal which you’d like achive. It can be goal of category or goal of file. For example Open Event team has defined goal for a specific file to achieve. If you define small separate goals, you can quicker see the results of your work.

open event orga server_2   Codacy   Goals

On the left sidebar you can find a item which is named “Goals”. In this area you can easily add your projects goals. Everything is user friendly so you shouldn’t have a problem  to create own goals.

Downloading Files from URLs in Python

This post is about how to efficiently/correctly download files from URLs using Python. I will be using the god-send library requests for it. I will write about methods to correctly download binaries from URLs and set their filenames.

Let’s start with baby steps on how to download a file using requests –

import requests

url = 'http://google.com/favicon.ico'
r = requests.get(url, allow_redirects=True)
open('google.ico', 'wb').write(r.content)

The above code will download the media at http://google.com/favicon.ico and save it as google.ico.

Now let’s take another example where url is https://www.youtube.com/watch?v=9bZkp7q19f0. What do you think will happen if the above code is used to download it ? If you said that a HTML page will be downloaded, you are spot on. This was one of the problems I faced in the Import module of Open Event where I had to download media from certain links. When the URL linked to a webpage rather than a binary, I had to not download that file and just keep the link as is. To solve this, what I did was inspecting the headers of the URL. Headers usually contain a Content-Type parameter which tells us about the type of data the url is linking to. A naive way to do it will be –

r = requests.get(url, allow_redirects=True)
print r.headers.get('content-type')

It works but is not the optimum way to do so as it involves downloading the file for checking the header. So if the file is large, this will do nothing but waste bandwidth. I looked into the requests documentation and found a better way to do it. That way involved just fetching the headers of a url before actually downloading it. This allows us to skip downloading files which weren’t meant to be downloaded.

import requests

def is_downloadable(url):
    """
    Does the url contain a downloadable resource
    """
    h = requests.head(url, allow_redirects=True)
    header = h.headers
    content_type = header.get('content-type')
    if 'text' in content_type.lower():
        return False
    if 'html' in content_type.lower():
        return False
    return True

print is_downloadable('https://www.youtube.com/watch?v=9bZkp7q19f0')
# >> False
print is_downloadable('http://google.com/favicon.ico')
# >> True

To restrict download by file size, we can get the filesize from the Content-Length header and then do suitable comparisons.

content_length = header.get('content-length', None)
if content_length and content_length > 2e8:  # 200 mb approx
	return False

So using the above function, we can skip downloading urls which don’t link to media.

Getting filename from URL

We can parse the url to get the filename. Example – http://aviaryan.in/images/profile.png.

To extract the filename from the above URL we can write a routine which fetches the last string after backslash (/).

url = 'http://aviaryan.in/images/profile.png'
if url.find('/'):
	print url.rsplit('/', 1)[1]

This will be give the filename in some cases correctly. However, there are times when the filename information is not present in the url. Example, something like http://url.com/download. In that case, the Content-Disposition header will contain the filename information. Here is how to fetch it.

import requests
import re

def get_filename_from_cd(cd):
    """
    Get filename from content-disposition
    """
    if not cd:
        return None
    fname = re.findall('filename=(.+)', cd)
    if len(fname) == 0:
        return None
    return fname[0]


url = 'http://google.com/favicon.ico'
r = requests.get(url, allow_redirects=True)
filename = get_filename_from_cd(r.headers.get('content-disposition'))
open(filename, 'wb').write(r.content)

The url-parsing code in conjuction with the above method to get filename from Content-Dispositionheader will work for most of the cases. Use them and test the results.

These are my 2 cents on downloading files using requests in Python. Let me know of other tricks I might have overlooked.

{{ Repost from my personal blog http://aviaryan.in/blog/gsoc/downloading-files-from-urls.html }}


Building a logger interface for FlightGear using Python: Part One

{ Repost from my personal blog @ https://blog.codezero.xyz/python-logger-interface-for-flightgear-part-one/ }

The FlightGear flight simulator is an open-source, multi-platform, cooperative flight simulator developed as a part of the FlightGear project. I have been using this Flight simulator for a year for Virtual Flight testing, running simulations and measuring flight parameters during various types of maneuvers. I have noticed that, logging the data, (figuring out how to log in the first place) has been quite difficult for users with less technical knowledge in such softwares.

Also, the Property Tree of FlightGear is pretty extensive making it difficult to properly traverse the huge tree to get the parameters that are actually required.

That’s when I got the idea of making a simple, easy to use, user friendly logging interface for FlightGear. I gave it a name ‘FlightGear Command Center’:wink: and the project was born at github.com/niranjan94/flightgear-cc.

After 44 commits, this is what I have now.

1. A simple dashboard to connect to FlightGear, open FlightGear with a default plane, Getting individual parameter values or to log a lot of parameters continuously

2. An interface to choose the parameters to log and the interval

  1. The User interface is a web application written in HTML/javascript.
  2. The Web application communicates with a python bridge using WebSockets.
  3. The python bridge communicates with FlightGear via telnet.
  4. The data is logged to a csv file continuously (until the user presses stop) by the bridge once the web application requests it.
The interface with FlightGear

FlightGear has an internal “telnet” command server which provides us “remote shell” into the running FlightGear process which we can exploit to interactively view or modify any property/variable of the simulation.

FlightGear can be instructed to start the server and listen for commands by passing the --telnet=socket,out,60,localhost,5555,udp command line argument while starting FlightGear. (The argument is of format --telnet=medium,direction,speed_in_hertz,localhost,PORT,style.)

Communication with that server can be done using any simple telnet interface. But FlightGear also provides us with a small wrapper class that makes retrieving and setting properties using the telnet server even more easier.

The wrapper can be obtained from the official repository atsourceforge.net/p/flightgear/flightgear/ci/master/tree/scripts/python/FlightGear.py

Using the wrapper is straightforward. Initialize an instance of the class with the hostname and port. The class will then make a connection to the telnet server.

from FlightGear import FlightGear

flightgear_server = 'localhost'  
flightgear_server_port = 5555  
fg = FlightGear(flightgear_server, flightgear_server_port)

The wrapper makes use of python’s magic methods __setitem__ and __getitem__ to make it easy for us to read or manipulate the property tree.

For example, getting the current altitude of the airplane is as easy as

print fg['/position[0]/altitude-ft']

and setting the altitude is as simple as

fg['/position[0]/altitude-ft'] = 345.2

But the important thing here is, knowing the path to the data you want in the FlightGear property tree. Most of the commonly used properties are available over at Aircraft properties reference – FlightGear Wiki.

Now that we have basic interface between python and FlightGear in place, the next step would be to setup a link between the user interface (a small web app) and the python bridge. We would be using WebSockets for that so as to have a Real-time and an always on link to the bridge which would enable us to in turn communicate with FlightGear in realtime.

We need a WebSocket server in place. So, I used the SimpleWebSocketServer.pyclass from github.com/dpallot/simple-websocket-server.

A websocket server can be created by,

from SimpleWebSocketServer import SimpleWebSocketServer, WebSocket

hostname = 'localhost'  
websocket_server_port = 8888

class SocketHandler(WebSocket):

    def handleMessage(self):
        # print the message when received 
        print self.data

    def handleConnected(self):
        print self.address, 'connected'

    def handleClose(self):
        print self.address, 'closed'

server = SimpleWebSocketServer(hostname, websocket_server_port, SocketHandler)  
server.serveforever()
  • handleMessage is called whenever a client sends a message to the server
  • handleConnected is called when a new client connects to the server
  • handleClose is called when a client disconnects from the server

A message can be sent to the clients by using the sendMessage method from within the SocketHandler.

class SocketHandler(WebSocket):

    def handleMessage(self):
        # send a hello whenever a message is received  
        print self.data
        self.sendMessage('Hello')

    def handleConnected(self):
        print self.address, 'connected'

    def handleClose(self):
        print self.address, 'closed'

We now have a WebSocket server in place. Now the web app can easily talk to this server using javascript websockets API. Which would be continued in upcoming blog articles.

Unit Testing

There are many stories about unit testing. Developers sometimes say that they don’t write tests because they write a good quality code. Does it make sense, if no one is infallible?.

At studies only a  few teachers talk about unit testing, but they only show basic examples of unit testing. They require to write a few tests to finish final project, but nobody really  teaches us the importance of unit testing.

I have also always wondered what benefits can it bring. As time is a really important factor in our work it often happens that we simply resign of this part of process development to get “more time” rather than spend time on writing stupid tests. But now I know that it is a vicious circle.

Customers requierments does not help us. They put a high pressure to see visible results not a few statistics about coverage status. None of them cares about some strange numbers. So, as I mentioned above, we usually focuses on building new features and get riid of tests. It may seem to save time, but it doesn’t.

In reality tests save us a lot of time because we can identify and fix bugs very quickly. If a bug ocurrs because someone’s change we don’t have to spend long hours trying to figure out wgat is going out. That’s why we need tests.  

It is especially visible in huge open source projects. FOSSASIA organization has about 200 contributors. In OpenEvent project we have about 20 active developers, who generate many lines of code every single day. Many of them change over and over again as well as interfere  with each other.

Let me provide you with a simple example. In our team we have about 7 pull requests per day. As I mentioned above we want to make our code high quality and free of bugs, but without testing identifying if pull request causes a bug is very difficult task. But fortunately this boring job makes Travis CI for us. It is a great tool which uses our tests and runs them on every PR  to check if bugs occur. It helps us to quickly notice bugs and maintain our project very well.

What is unit testing?

Unit testing is a software development method in which the smallest testable parts of an application are tested

Why do we need writing unit tests?

Let me point all arguments why unit testing is really important while developing a project.

  • To prove that our code works properly

If developer adds another condition, test checks if method returns correct results. You simply don’t need to wonder if something is wrong with you code.

  • To reduce amount of bugs

It let you to know what inputs params’ function should get and what results should be returned. You simply don’t  write unused code

  • To save development time

Developers don’t waste time on checking every code’s change if his code works correctly

  • Unit tests help to understand software design
  • To provide quick feedback about method which you are testing
  • To help document a code

How to write unit test in Python

In my work I write use tests in Python. I am going to share my sample code  with you now

  • Import module unittest
  • Choose function to test
  • Write unit test

Example OpenEvent test in Python

class TestPagesUrls(OpenEventTestCase):

   def setUp(self):

       self.app = Setup.create_app()

   def test_if_urls_exist(self):

       """Test all urls via GET method"""

       with app.test_request_context():

           for rule in app.url_map.iter_rules():

               if excluded_paths(rule):

                   status_code = self.app.get(request.url[:-1] + str(rule).replace('//', '/'),        follow_redirects=True).status_code

                   self.assertTrue(status_code in [200, 302, 401])

 

I want to check if all views exist but it required a lot of time. That’s why I wonder I how to avoid writing similar tests. Finally, based  on our list of routes I am able to write test which checks code’s status  on every page.

If some of them response returns status_code different than 200, 302 or 401, test fails.This results means that somethings is wrong. Simple, isn’t it ?  Try to test it manually…. This one short test cover about 40 use cases…

This example shows an incredible value of unit tests! If developer makes a bug in response he receives an error that something is wrong with a view. Travis CI allows to reject all  wrong pull requests and merge only these which fulfill our quality requirements.   

Fixing  error is one part but finding a bug is even harder task. But an ability to detect bug on early stage of process development reduces cost of software.

 

Features and Controls of Pocket Science Lab

Prerequisite reading:

PSLab Code Repository and Installation

PSLab is equipped with array of useful control and measurement tools. This tiny but powerful Pocket Science Lab enables you to perform various experiments and study a wide range of phenomena.

Some of the important applications of PSLab include a 4-channel oscilloscope, sine/triangle/square waveform generators, a frequency counter, a logic analyser and also several programmable current and voltage sources.

Add-on boards, both wired as well as wireless(NRF+MCU), enable measurement of physical parameters ranging from acceleration and angular velocity, to luminous intensity and Passive Infra-red. (Work under progress…)

As a reference for digital instruments a 12-MHz Crystal is chosen and a 3.3V voltage regulator is chosen for the analogue instruments. The device is then calibrated against professional instruments in order to squeeze out maximum performance.

Python based communication library and experiment specific PyQt4 based GUI’s make PSLab a must have tool for programmers, hobbyists, science and engineering teachers and also students.

PSLab is interfaced and powered by USB port of the computer. For connecting external signals it has several input/output terminals as shown in the figure.

pslabdesign
New panel design for PSLab

psl2

Feature list for the acquisition and control :

  • The most important feature of PSLab is a 4-channel oscilloscope which can monitor analog inputs at maximum of 2 million samples per second. Includes the usual controls such as triggering, and gain selection. Uses Python-Scipy for curve fitting.
oscilloscope
PSLab Oscilloscope

 

 

Waveform Generators

  • W1 : 5Hz – 5KHz arbitrary waveform generator. Manual amplitude control up to +/-3Volts
  • W2 : 5Hz – 5KHz arbitrary waveform generator. Amplitude of +/-3Volts. Attenuable via software
  • PWM : There are four phase correlated PWM outputs with maximum frequency 32MHz, 15nano second duty cycle, and phase difference control.

Measurement Functions

  • Frequency counter tested up to 16 MHz.
  • Capacitance Measurement. pF to uF range
  • PSLab has several 12-bit Analog inputs (function as voltmeters) with programmable gains, and maximum ranges varying from +/-5mV to +/-16V.

Voltage and Current Sources

  • 12-bit Constant Current source. Maximum current 3.3mA [subject to load resistance].
  • PSLab has three 12-bit Programmable voltage sources/ +/-3.3V,+/-5V,0-3V . (PV1, PV2, PV3)
controls
Main Control Panel

Other useful tools

  • 4MHz, 4-channel Logic analyzer with 15nS resolution.Voltage and Current Sources
  • SPI,I2C,UART outputs that can be configured and controlled entirely through Python functions. (Work in progress…)
  • On-board 2.4GHz transceiver for wireless data acquisition. (Work in progress..)
  • Graphical Interfaces for Oscilloscope, Logic Analyser, streaming data, wireless acquisition, and several experiments developed that use a common framework which drastically reduces code required to incorporate control and plotting widgets.
  • PSLab also has space for an ESP-12 module for WiFi access with access point / station mode.

Screen-shots of GUI apps.

advanced-controls
Advanced Controls with Oscilloscope
wirelesssensordataloger
Wireless Sensors ( Work in progress…)
logicanalyzer
Logic Analyzer

With all these features PSLab is taking a good shape and I see it as a potential tool that can change the way we teach and learn science. 🙂 🙂