socketio

Read more about the article Using Flask SocketIO Library in the Apk Generator of the Open Event Android App

Using Flask SocketIO Library in the Apk Generator of the Open Event Android App

Recently Flask SocketIO library was used in the apk generator of the Open Event Android App as it gave access to the low latency bi-directional communications between the client and the server side. The client side of the apk generator was written in Javascript which helped us to use a SocketIO official client library to establish a permanent connection to the server.

The main purpose of using the library was to display logs to the user when the app generation process goes on. This gives the user an additional help to check what is the mistake in the file uploaded by them in case an error occurs. Here the library established a connection between the server and the client so that during the app generation process the server would send real time logs to the client so that they can be viewed by the user displayed in the frontend.

To use this library we first need to download it using pip command:

pip install flask-socketio

This library depends on the asynchronous services which can be selected amongst the following listed below:

  1. Eventlet
  2. Gevent
  3. Flask development server based on Werkzeug

Amongst the three listed, eventlet is the best performant option as it has support for long polling and WebSocket transports.

The next thing was importing this library in the flask application i.e the apk generator of the app as follows:

from flask_socketio import SocketIO

current_app = create_app()
socketio = SocketIO(current_app)

if __name__ == '__main__':
    socketio.run(current_app)

The main use of the above statement (socket.run(current_app)) is that with this the startup of the web server is encapsulated. When the application is run in debug mode it is preferred to use the Werkzeug development server. However in production mode the use of eventlet web server or gevent web server is recommended.

We wanted to show the status messages currently shown to the user in the form of logs. For this firstly the generator.py file was looked upon which had the status messages and these were sent to the client side of the generator by establishing a connection between them using this library. The code written on the server side to send messages to the client side was as follows:

def handle_message(message):
    if message not None:
        namespace = “/” + self.identifier;
        send(message, namespace = namespace)

As we can see from the above code the message had to be sent to that particular namespace. The main idea of using namespace was that if there were more than one users using the generator at the same time, it would mean the send() method would send the messages to all the clients connected which would lead to the mixing up of status messages and hence it was important to use namespace so that a message was sent to that particular client.

Once the server sent the messages to the client we needed to add functionality to the our client side to receive the messages from them and also update the logs area with suitable content received. For that first the socket connection was established once the generate button was clicked which generated the identifier for us which had to be used as the namespace for that process.

socket = io.connect(location.protocol + "//" + location.host + "/" + identifier, {reconnection: false});

This piece of code established the connection of the socket and helps the client side to receive and send messages from and to the server end.

Next thing was receiving messages from the server. For this the following code snippet was added:

socket.on('message', function(message) {
    $buildLog.append(message);
});

This piece of code receives the messages from the server for unnamed events. Once the connection is established and the messages are received, the logs are updated with each message being appended so as to show the user the real time information about their build status of their app.

This was how the idea of logs being shown in the apk generator was incorporated by making the required changes in the server and client side code using the Flask SocketIO library.

Related Links:

Continue ReadingUsing Flask SocketIO Library in the Apk Generator of the Open Event Android App
Read more about the article Displaying selective logs in Yaydoc with downloadable detailed logs

Displaying selective logs in Yaydoc with downloadable detailed logs

Yaydoc, our automatic documentation generation and deployment project, at the crux of it consists of bash scripts. These bash scripts perform various actions, including documentation generation, deployment to Github and deployment to Heroku.

Since the inception of the User Interface of the Web UI, we have been spitting out the output of the bash script directly to the console output block without any filter. We understand that the output contains a lot of jargon that provides no essential information to the user. Hence, to improve the user experience of our platform, we decided to separate the Detailed logs and show only basic logs outlining the flow of the processes.

To implement this, we append all the logs to a unique file, sending only basic logs to stdout and stderr. Since we attend to store logs and display them in the console block of Web UI simultaneously we use the tee command, piping it with echo commands.

function print_log {
  if [ -n “$LOGFILE” ]; then
    echo -e $1 | tee -a ${LOGFILE}
  else
    echo -e $1
  fi
}

${LOGFILE}  is a unique file that has the same name as the preview directory and the compressed repository. After storing the logs in the file, the contents of the file are outputted using the cat command and is then shown to the user in a modal which is after the Detailed Logs button is clicked.

$(“#btnLogs”).click(function () {
  socket.emit(‘retrieve-detailed-logs’, data);
  ....
});

socket.on(‘retrieve-detailed-logs’, function (data) {
  var process = spawn(‘cat’, [‘temp/’+data.email+’/’+data.uniqueId+’.txt’]);
  process.stdout.setEncoding(‘utf-8’);
  process.stdout.on(‘data’, function (data)) {
    socket.emit(‘file-content’, data);
  }
});

To keep the indentation of the logs, we display the content inside the HTML pre tags. Along with displaying the detailed logs, we also let our two additional functionalities. These are ‘Copy to Clipboard’ and ‘Download as text file’. The ‘copy to clipboard’ functionality is achieved using the clipboard.js jQuery library while the `Download` functionality is achieved using res.download(file) function of ExpressJS response.

socket.on(‘file-content’, function (data) {
  new Clipboard(‘#copy-button’);
  $(‘#detailed-logs’).html(data);
  $(‘#detailed-logs-modal’).modal(‘show’);
});

Resources:

  1. https://clipboardjs.com – A modern approach to copy text to clipboard
  2. https://socket.io/ – The fastest and most reliable real-time engine
Continue ReadingDisplaying selective logs in Yaydoc with downloadable detailed logs

File upload progress in a Node app using Socket.io

If you look at the webapp generator, you’ll see that there is an option to upload a zip file containing event data. We wanted to give visual cue to the user when he is uploading to see how much file has uploaded.

We are uploading the file, and giving the generate start command via socket.io events instead of POST requests here.

To observe file upload progress on socket (when sending file using a Buffer), there is an awesome node module available called socketio-upload-progress.

In our webapp you can see we implemented it on the frontend here in the form.js and here in the backend in app.js

Basically on the backend you should add the socketio-file-upload module as a middleware to express

var siofu = require("socketio-file-upload");
var app = express()
    .use(siofu.router)
    .listen(8000);

After a socket is opened, set up the upload directory and start listening for uploads

io.on("connection", function(socket){
    var uploader = new siofu();
    uploader.dir = "/path/to/save/uploads";
    uploader.listen(socket);
});

On the frontend, we’ll listen for an input change on an file input type element whose id is siofu_upload

var socket = io.connect();
var uploader = new SocketIOFileUpload(socket);
uploader.listenOnInput(document.getElementById("siofu_input"));

One thing to note here is that, if you observe percentage of upload on frontend, it’ll give you false values. The correct values of how much data is actually transferred can be found in the backend. So observe progress in backend, and send percentage to frontend using the same socket.

  uploader.on('progress', function(event) {
    console.log(event.file.bytesLoaded / event.file.size)
    socket.emit('upload.progress', {
      percentage:(event.file.bytesLoaded / event.file.size) * 100
    })
});

 

Continue ReadingFile upload progress in a Node app using Socket.io