Reducing the YouTube response time by 90%

In this blog post, we are going to cover how the audio from Youtube is being used in SUSI Smart Speaker and how we reduced the response time from ~40 seconds to ~4 seconds for an average music video length.

First Approach

Earlier, we were using MPV player’s inbuilt feature to fetch the YouTube music. However, MPV player was a bulky option and the music server had to be started every time before initiating a music video.

video_process = subprocess.Popen([‘mpv’, ‘–no-video’, ‘https://www.youtube.com/watch?v=’ + video_url[4:], ‘–really-quiet’]) # nosec #pylint-disable type: ignore requests.get(‘http://localhost:7070/song/’ + video_url) self.video_process = video_process stopAction.run() stopAction.detector.terminate()

Making it Efficient

To reduce the response time, we created a custom Music Server based on Flask,python-vlc and python-pafy which accepts requests from the main client and instructs the System to play the music with just 90% more efficiency.

app = Flask(__name__)

Instance = vlc.Instance(‘–no-video’)

player = Instance.media_player_new()

url = @app.route(‘/song’, methods=[‘GET’])

def youtube():

    vid = request.args.get(‘vid’)

    url = ‘https://www.youtube.com/watch?v=’ + vid

    video = pafy.new(url)
    streams = video.audiostreams 

    best = streams[3]

    playurl = best.url

    Media = Instance.media_new(playurl)

    Media.get_mrl()

    player.set_media(Media)

    player.play()

    display_message = {“song”:“started”}

    resp = jsonify(display_message)

    resp.status_code = 200

    return resp

However, shifting to this Server removed the ability to process multiple queries and hence we were unable to pause/play/stop the music until it completed the time duration. We wanted to retain the ability to have ‘play/pause/stop’ actions without implementing multiprocessing or multithreading as it would’ve required extensive testing to successfully implement them without creating deadlocks and would’ve been overkill for a simple feature.

Bringing Back the Lost Functionalities

The first Step we took was to remove the vlc-python module and implement a way to obtain an URL that we use in another asynchronous music player.

@app.route(‘/song’, methods=[‘GET’])
def youtube():

    vid = request.args.get(‘vid’)

    streams = video.audiostreams

    best = streams[3]

    playurl = best.url 

    display_message = {“song”: “started”, “url”: playurl}

    resp = jsonify(display_message)

    resp.status_code = 200

    return resp

The next issue was to actually find a way to run the Music Player asynchronously. We used the `subprocess. Popen` method and cvlc to play the songs asynchronously.

try:

    x = requests.get(‘http://localhost:7070/song?vid=’ + video_url[4:])

    data = x.json()

    url = data[‘url’]

    video_process = subprocess.Popen([‘cvlc’, ‘http’ + url[5:], ‘–no-video’])

    self.video_process = video_process

except Exception as e:

    logger.error(e);

And this is how we were able to increase the efficiency of the music player while maintaining the functionalities.

References

Continue ReadingReducing the YouTube response time by 90%

Modifying Finite State Architecture On SUSI Linux to Process Multiple Queries

During the initial stages of SUSI Linux: As the code base grew, it was getting very difficult to manage code, so we opted to implement a Finite State Architecture in our repo. But, as there were new features implemented in the Repo, we realized that we couldn’t process more than one query at a time which restricted a lot of features. eg. The smart speaker was converted to a simple Bluetooth speaker since no response regarding playing/pausing were accepted.

To solve this issue, we made a slight modification in the architecture.

Brief About SUSI States

SUSI is working as a Finite State Machine and is present in 3 states namely IDLE state, Recognising state and Busy state. The State Machine executes in the following order.

  1. IDLE State:
    When the SUSI state Machine is in this State, SUSI is searching for the hotword “SUSI”, waiting to trigger the complete Machine.
  2. Recognizing State

In this State , the State Machine has started the STT client. After recognition, SUSI sends the query to the Server awaiting the response

  1. Busy State

After the response has been received, the TTS client is triggered and the answer is given out by SUSI

Adding a Second Hotword Recognition Class

Now, to allow SUSI to process the second query, The State machine must be triggered while SUSI is giving out the first response and to trigger the State Machine, we must have hotword recognition while SUSI is speaking the answer to the previous query. Hence, a hotword recognition engine is now initiated every time the State Machine enters the busy state.

We will be using Snowboy as Hotword Detection Engine.

 

import os
TOP_DIR = os.path.dirname(os.path.abspath(__file__))
RESOURCE_FILE = os.path.join(TOP_DIR, “susi.pmdl”)
class StopDetector():
   “””This implements the Stop Detection with Snowboy Hotword Detection Engine.”””
    def __init__(self, detection) -> None:
       super().__init__()
       self.detector = snowboydecoder.HotwordDetector(
           RESOURCE_FILE, sensitivity=0.6)
       self.detection = detection
    def run(self):
       “”” Implementation of run abstract method in HotwordDetector. This method is called when thread is
started for the first time. We start the Snowboy detection and declare detected callback as
       detection_callback method declared ina parent class.
       “””
       self.detector.start(detected_callback=self.detection)

 

Now, this class takes the Callback function as a parameter which is passed when the transition to busy state takes place from the recognition state.

 

Modifying the State Machine Architecture

After declaring a second hotword recognition engine , we must modify how the transitions take place between the States of the SUSI State Machine.

Hence the callback that will be triggered is passed from the busy state.

 

def detection(self):
       “””This callback is fired when a Hotword Detector detects a hotword.
       :return: None
       “””
       if hasattr(self, ‘video_process’):
           self.video_process.send_signal(signal.SIGSTOP)
           lights.wakeup()
           subprocess.Popen([‘play’, str(self.components.config[‘detection_bell_sound’])])
           lights.off()
           self.transition(self.allowedStateTransitions.get(‘recognizing’))
           self.video_process.send_signal(signal.SIGCONT)
       if hasattr(self, ‘audio_process’):
           self.audio_process.send_signal(signal.SIGSTOP)  
           lights.wakeup()
           subprocess.Popen([‘play’, str(self.components.config[‘detection_bell_sound’])])
           lights.wakeup()
           self.transition(self.allowedStateTransitions.get(‘recognizing’))
           self.audio_process.send_signal(signal.SIGCONT)

 

As soon as the hotword is detected ,the state machine makes transitions to the Recognition State while pausing the current Music and resumes the Music after the second query has been completed.

 

This is how SUSI processes multiple queries simultaneously while still maintaining finite state archi.

 

Additional Resources

Tags

gsoc, gsoc’18, finite_state_machine, susi_linux, multiple_query, susi.ai, susi

 

Continue ReadingModifying Finite State Architecture On SUSI Linux to Process Multiple Queries

Creating a Factory Reset Daemon for SUSI.AI Smart Speaker

In our constantly evolving SUSI.AI Smart Speaker project, we require regular updates for our devices. And imagine a scenario that during a crucial update, there is a crash or an internet disconnection which stops the SUSI.AI Linux program from booting up. We’ll require a reset method for that. So, we have added a button in SUSI smart speaker that works as a factory reset switch. This daemon was accomplished by using python scripting, bash scripting, and Raspbian’s systemd rules.

Approach followed

We have created a python script that detects the button presses on GPIO port 17. The script is run as soon as the Raspberry Pi is booted using the systemd rules and checks for the device inputs. And if the button press is for more than 7 seconds, the factory_reset.sh script is run which deletes all the contents of the repo and clones it again.

 

#! /bin/bash
# To be executed using a physical button

SCRIPT_PATH=$(realpath $0)
DIR_PATH=$(dirname $SCRIPT_PATH)

cd $DIR_PATH/../..
pwd
mv susi_linux/ susi_temp
git clone https://github.com/fossasia/susi_linux #while testing change to personal repo
pwd
ls
cd susi_linux

rm -rf ../susi_temp

./install.sh

 

Detecting the Button Press

We have Used the library RPi.GPIO to detect button click on raspberry Pi.

while True:
       if GPIO.input(17) == 1:
           pass
       elif GPIO.input(17) == 0 :
           start = time.time()
          while GPIO.input(17) == 0 :
               print(“on”)
               time.sleep(0.1)
          end = time.time()
           total = end – start
           if total >= 7 :
              subprocess.call([‘bash’,‘factory_reset.sh’])
          else :
               mixer = alsaaudio.Mixer()
               value = mixer.getvolume()[0]
              if value != 0:
                  mixer.setvolume(0)
               else:
                   mixer.setvolume(50)
           print(total)
           time.sleep(0.1)

 

If the button press is greater than 7 seconds, factory reset process will start and if the press is less than 7 seconds, the button will function as mute button

 

Auto Booting The program

 

For the script to autorun everytime the raspberry pi started. We create systemd file which will allow the program to start as soon as the device has started

 

[Unit]
Description=SUSI Linux Factory Daemon
After=multi-user.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/pi/SUSI.AI/susi_linux/factory_reset/factory_reset.py

[Install]
WantedBy=multi-user.target

 

This runs the factory reset script to boot up as soon as the Raspberry Pi starts

References

Tags

susi, factory_daemon, factory_reset, gsoc, gsoc’18,susi_linux , fossasia

Continue ReadingCreating a Factory Reset Daemon for SUSI.AI Smart Speaker

Creating a Custom Raspbian Image containing SUSI.AI Linux Libraries

Installing Raspbian and SUSI Linux on your Raspberry Pi can be a long process and if your raspberry Pi crashes due to some bug, you have to repeat the process again and again.

It wastes a lot of valuable time. So, we will discuss a method in which we will have to install the SUSI Linux repo only once and can use it again in case of any issues.

First, we’ll go through the requirements for f installing SUSI Linux on our hardware

Hardware Requirements

1> Raspberry Pi

2> Micro SD card (16GB or greater)

3> USB Mic

4> USB Mouse and USB Keyboard

5> HDMI Monitor

6> ReSpeaker Pi Hat 2 Mic Array(optional)

7> 3.5 mm Jack Headphones / Speaker

 

Step 1: Preparing SD for Installation

1> To format your SD card

  • You can use softwares like SDCardformater to do so.

2> To install Raspbian

  • Download raspbian official build from here
  • Now mount the Image using software like Etcher or win32diskimager

 

Step 2: Installing SUSI Linux on your Pi

1> Navigate to the folder `/home/pi` and make a folder called SUSI.AI

 

cd /home/pi
mkdir SUSI.AI
cd SUSI.AI

 

2> Clone the SUSI Linux repo from here and navigate in the repo

 

git clone http://github.com/fossasia/susi_linux

cd susi_linux/

 

3> Run the installation script by using the command `./install.sh`

 

./install.sh

 

4> Run the configuration script by using the following command

`python3 config_generator.py <stt> <tts> <hotword> <wake>`

 

5> Run SUSI linux with the following command

`python3 -m main`

‘log’

If it plays a bell after you say ‘SUSI’ , it means that your software has been successfully installed

Step 3: Creating the image

1> Now that you have successfully installed SUSI Linux on your raspberry Pi , we will make a backup of the current stage of the system and use it for future references

 

2>Turn off the raspberry Pi , and remove the SD card from the Pi and insert it in your system.

 

3> To create the custom Image , use something like win32 imager and follow the steps below

  • In the text box , create a custom where you want your image to exist
  • Click on read button
  • And voila

 

References

https://www.raspberrypi.org/downloads/raspbian/

Tags

Fossasia, gsoc’18, SUSI.AI , susi_linux , gsoc, SUSI HW, installation

Continue ReadingCreating a Custom Raspbian Image containing SUSI.AI Linux Libraries

Connecting to a Raspberry Pi through a SSH connection Wirelessly

The tech stack of the SUSI.AI smart speaker project is mainly Python/Bash scripts. Every smart speaker has an essential feature that allows the user’s mobile device to connect and give instructions to the speaker wirelessly. To make this connection possible, we are trying to implement this using an SSH connection.

Why SSH?

SSH(a.k.a Secure Shell) is a cryptographic connection which allows secure transfer of data even over an unsecured connection.SSH connection even allows TCP as well as X11 forwarding which are an added bonus.

Step 1: Initial Setup

  • Both the raspberry Pi with raspbian installed and the mobile device should be on a same wireless network
  • One should have an SSH viewer like JuiceSSH(Android) and iTerminal(IOS) installed on their mobile devices
  • Now we must enable SSH on our raspberry Pi

Step 2: Enabling SSH on Raspberry PI

  • To enable SSH on your Pi , follow the steps mentioned below:
Menu > Preferences > Raspberry Pi Configuration.

Choose the interfaces tab and enable SSH

Step 3:Setting Up the client

 

  • Login to your raspberry pi as the root user (pi by default)
  • Type the following command to know the broadcasting ip address
pi@raspberrypi:hostname -I

 

  • Now , open the client on your mobile device and add the configurations

By default the username of the system is ‘pi’ and the password is ‘raspberry’

Step 4: Changing the default SSH password

Since the default password of every RaspberryPi is the same. So , the pi can be accessed by any device that has access to the local network which is not a secure way of accessing the device

  • In the SSH window type ‘passwd’
  • Type the current password
  • Type the new password
  • Re-enter the new password

Now you will be able to login to your raspberry through an SSH connection

Resources


Tags

Fossasia, gsoc, gsoc’18, susi, susi.ai, hardware,susi_linux

Continue ReadingConnecting to a Raspberry Pi through a SSH connection Wirelessly