Sending Data between components of SUSI MagicMirror Module

SUSI MagicMirror module is a module to add SUSI assistant right on your MagicMirror. The software for MagicMirror constitutes of an Electron app to which modules can be added easily. Since there are many modules, there might be functionalities that need interaction between various modules by transfer of information. MagicMirror also provides a node_helper script that facilitates a module to perform some background tasks. Therefore, a mechanism to transfer information from node_helper to various components of module is also needed.

MagicMirror provides an inbuilt module notification system that can be used to send notification across the modules and a socket notification system to send information between node_helper and various components of the system.

Our codebase for SUSI MagicMirror is divided mainly into two parts. A Main module that handles all the process of hotword detection, speech recognition, calling SUSI API and saving audio after Text to Speech and a Renderer module which performs the task of managing the display of content on the Mirror Screen and playing back the file obtained by Speech Synthesis. Plainly put, Main module mainly handles the backend logic of the application and the Renderer handles the frontend. Main and Renderer module work on different layers of the application and to facilitate communication between them, we need to make a mechanism. A schematic of flow that is needed to be maintained can be highlighted as:

As you can see in the above diagram, we need to transfer a lot of information between the components. We display animation and text based on the current state of recognition in the  module, thus we need to transfer this information frequently. This task is accomplished by utilizing the inbuilt socket notification system in the MagicMirror. For every event like when system enters into listening , busy or recognized speech state, we need to pass message to renderer. To achieve this, we made a rendererSend function to send notification to renderer.

const rendererSend =  (event: NotificationType , payload: any) => {
   this.sendSocketNotification(event, payload);
}

This function takes an event and a payload as arguments. Event tells which event occurred and payload is any data that we wish to send. This method in turn calls the method provided by MagicMirror module to send socket notifications within the module.

When certain events occur like when system enters busy state or listening state, we trigger the rendererSend call to send a socket notification to the module. The rendererSend method is supplied in the State Machine Components available to every state. The task of sending notifications can be done using the code snippet as follows:

// system enters busy state
this.components.rendererSend("busy", {});
// send speech recognition hypothesis text to renderer
this.components.rendererSend("recognized", {text: recognizedText});
// send susi api output json to renderer to display interactive results while Speech Output is performed
this.components.rendererSend("speak", {data: susiResponse});

The socket notification sent via the above method is received in SUSI Module via a callback called socketNotificationReceived . We need to define this callback with implementation while registering module to MagicMirror. So, we register the MMM-SUSI-AI module by adding the definition for socketNotificationReceived method.

Module.register("MMM-SUSI-AI", {
//other function definitions
***
   // define socketNotificationReceived function
   socketNotificationReceived: function (notification, payload) {
       susiMirror.receivedNotification(notification, payload);
   },
***
});

In this way, we send all the notification received to susiMirror object in the renderer module by calling the receivedNotification method of susiMirror object

We can now receive all the notifications in the SusiMirror and update UI. To handle notifications, we define receivedNotification method as follows:

public receivedNotification(type: NotificationType, payload: any): void {

   this.visualizer.setMode(type);
   switch (type) {
       case "idle":
            // handle idle state
           break;
       case "listening":
           // handle listening state
           break;
       case "busy":
           // handle busy state
         break;
       case "recognized":
           // handle recognized state. This notification also contains a payload about the hypothesis text           
           break;
       case "speak":
           // handle speaking state. We need to play back audio file and display text on screen for SUSI Output. Notification Payload contains SUSI Response
           break;
   }
}

In this way, we utilize the Socket Notification System provided by the MagicMirror Electron Application to send data across the components of Magic Mirror module for SUSI AI.

Resources

How Meilix Generator sends Email Notifications with SendGrid

We wanted to notify the users once the build was ready for download. To solve this we attempted making an email server on Meilix Generator but that can send email when it starts but it would take around 20 minutes to get the build ready so we thought of checking the deploy link status and send email whenever the link status was available (200) but the problem with this method was that the link can be pre available if ISO is rebuilt for same event.

Then, we attempted sending mail by Travis CI but the problem in that was closed SMTP ports (they have a strict policy about that) then we thought that Travis CI can trigger the Sendgrid which can send email to the user with the help of API.

We will use this code so that once the deployment of ISO by Travis CI is done it can execute the email script which requests Sendgrid to send email to the user.

after_deploy:
  - ./mail.py

 

We can create code using code generation service of Sendgrid we are going to choose python as it is easier to manipulate strings in python and we are going to use email as an environment variable.

After generation of python 3 code from the sendgrid website we are going to edit the message and email and hide the API key as an environment variable and create an authorization string to be used there too.

The URL will be generated by the below script as the body of url remains same only two things will change the TRAVIS_TAG which is event name and date.

date = datetime.datetime.now().strftime('%Y%m%d')
url="https://github.com/xeon-zolt/meilix/releases/download/"+os.environ["TRAVIS_TAG"]+"/meilix-zesty-"+date+"-i386.iso"

 

We can use this to hide the api key and use it as an environment variable because if the api key is visible in logs anyone can use it to exploit it and use it for spamming purpose.

authorization = "Bearer " + os.environ["mail_api_key"]
headers = {
    'authorization': authorization,

 

The main thing left to edit in the script is the message which is in the payload and is a string type so we are going to use the email received by Meilix generator as an environment variable and concatenate it with the payload string the message sent is in the value which is in the HTML format and we add the generated URL in similar way we added email variable to string.

payload = "{\"personalizations\":[{\"to\":[{\"email\":\"" + os.environ["email"] + "\"}],\"subject\":\"Your ISO is Ready\"}],\"from\":{\"email\":\"xeon.harsh@gmail.com\",\"name\":\"Meilix Generator\"},\"reply_to\":{\"email\":\"xeon.harsh@gmail.com\",\"name\":\"Meilix Generator\"},\"subject\":\"Your ISO is ready\",\"content\":[{\"type\":\"text/html\",\"value\":\"<html><p>Hi,<br>Your ISO is ready<br>URL : "+url+"<br><br>Thank You,<br>Meilix Generator Team</p></html>\"}]}"

 

The sent email looks like this

References

Mark Notifications Read on Click

Screenshot from 2016-08-01 07:31:22

Notification has become a really important way of informing users about the various activities related to them in web apps. There are different types of notification such as web app notification, email notification, desktop notification, push notification, etc. We are going to primarily talk about web app notification and mainly about how to mark them as read.

Create Notification

Creating a notification is plain and simple. You have a json or an object which stores the notification message corresponding to a particular activity. Whenever that activity occurs in the backend, you call the send notification module, which adds the information to the database and shows it in the notification page. As simple as that.

Screenshot from 2016-08-01 07:48:08

Marking Notification as Read

The main functioning of this is plain and simple as well. You have a URL, which on getting a request from the user, marks the notification as read in the database. That’s it.

Screenshot from 2016-08-01 07:48:17

We know how to do this using a button or a link. But the question here is how to mark a notification as read on clicking any part of the notification?? The obvious answer is, well, put the entire notification inside an anchor tag and you are done, right? Well, it would work in many cases. But what if the design structure is such that this doesn’t work somehow. Somehow enclosing the notification inside a particular anchor tag doesn’t solve the purpose. What do we do then?

Identify Whether Inside a DIV

The main problem here actually is how to identify whether the click is inside the enclosing div or somewhere else. Once we solve this problem, we can send an ajax request to the mark read URL and our job is done.

Screenshot from 2016-08-01 07:52:58

So, to identify that a click is indeed inside a div, we use the event.target property of the event clicked.┬áThe target event property returns the element that triggered the event. So we check whether event.target has the “notification” class in our case. If it does not have the “notification” class we check in all it’s parent nodes. We get the parent nodes using the “parent()” function and check whether any of that has notification. If either of the 2 occurs, we consider that the click is inside the div. And thus mark the notification as read.

Screenshot from 2016-08-01 07:51:09

So, once this is done, we mark the notification as read in the backend and our job is done…