Handling Planned Actions for the SUSI Smart Speaker

Handling Planned Actions for the SUSI Smart Speaker 

Planned action is the latest feature added to the SUSI Smart Speaker, The user now has the option to set timed actions such as-  settings alarms, countdown timer etc. So the user now can say “SUSI, Set an alarm in one minute” and after one minute the smart speaker will notify you.

The following flowchart represents the workflow for the working of a planned action:

Planned Action Response

The SUSI Server accepts the planned action query and sends a multi-action response which looks like this: 

“actions”: [
      {
        “language”: “en”,
        “type”: “answer”,
        “expression”: “alarm set for in 1 minute”
      },
      {
        “expression”: “ALARM”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 60003,
        “plan_date”: “2019-08-19T22:28:44.283Z”
      }
    ]


Here we can see that we have two actions in the server response. The first action is of the type “answer” and is executed by the SUSI Linux client immediately, the other response has the `plan_date` and `plan_delay` keys which tells the BUSY State of the SUSI Linux client that this is a planned action and is then sent to the scheduler.

Parsing Planned Action Response From The Server 

The SUSI python wrapper is responsible for parsing the response from the server and making it understandable to the SUSI Linux client. In SUSI Python we have classes which represent the different types of actions possible. The SUSI Python takes all the actions sent by the server and parses them into objects of different action types. To enable handling planned actions we add two more attributes to the base action class – `planned_date` and `planned_delay`.


class BaseAction:
    def __init__(self, plan_delay = None, plan_date = None):
        self.plan_delay = plan_delay
        self.plan_date = plan_date

class AnswerAction(BaseAction):
    def __init__(self, expression, plan_delay = None, plan_date = None):
        super().__init__(plan_delay,plan_date)
        self.expression = expression


Here we can see, All the action types which can be planned actions call the base class’ constructor to set the value for planned_delay and planned_date attributes. 

The next step is to parse the different action type object and generate the final result

def generate_result(response):
    result = dict()
    actions = response.answer.actions
    data = response.answer.data
    result[“planned_actions”] = []
    for action in actions:
        data = dict()
        if isinstance(action, AnswerAction):
            if action.plan_delay != None and action.plan_date != None:
                data[‘answer’] = action.expression
                data[‘plan_delay’] = action.plan_delay
                data[‘plan_date’] = action.plan_date
            else:
                result[‘answer’] = action.expression
        if data != {}:
            result[“planned_actions”].append(data)

Here if the action object has a non none value for the planned attributes, the action object’s values are added to a planned actions list.

Listening to Planned Actions in the SUSI Linux Client

In the busy state, we see if the payload coming from the IDLE state is a query or a planned response coming from the scheduler. If the payload is a query, the query is sent to the server, otherwise the payload is executed directly

if isinstance(payload, str):
logger.debug(“Sending payload to susi server: %s”, payload)
reply = self.components.susi.ask(payload)
else :
logger.debug(“Executing planned action response”, payload)
reply = payload


If the payload was a query and the server replies with a planned action response, then 

The server response is sent to the scheduler.

if ‘planned_actions’ in reply.keys():
    for plan in reply[‘planned_actions’]:        self.components.action_schduler.add_event(int(plan[‘plan_delay’])/1000,plan)


The scheduler then schedules the event and send the payload to the IDLE state with the required delay. To trigger planned actions we implemented an event based observer using RxPy. The listener resides in the idle state of the SUSI State Machine. 

        if self.components.action_schduler is not None:
            self.components.action_schduler.subject.subscribe(
                on_next=lambda x: self.transition_busy(x))


The observer in the IDLE state on receiving an event sends the payload to the busy state where it is processed. This is done by the transition_busy method which uses the allowedStateTransitions method.

    def transition_busy(self,reply):
        self.transition(self.allowedStateTransitions.get(
            ‘busy’), payload=reply)

Resources

Understanding the SUSI State Machine – https://blog.fossasia.org/implementing-susi-linux-app-as-a-finite-state-machine/

Audio Structure of SUSI Smart Speaker – https://blog.fossasia.org/audio-structure-of-susi-smart-speaker/

Reactive Python Documentation – https://rxpy.readthedocs.io/en/latest/

Tags

SUSI Smart Speaker, SUSI.AI, FOSSASIA, GSoC19

Continue Reading

Integrating the SUSI.AI Web Client with the Smart Speaker

In the previous versions of the firmware for the smart speaker, we had a separate flask server for serving the configuration page and another flask server for serving the control page. This puts forward inconsistencies in the frontend. To make the frontend and the user experience the same across all platforms, the SUSI.AI Web Client is now integrated into the smart speaker.

Now whenever the device is in Access Point mode (Setup mode), and the user accesses the web client running on the smart speaker, the device configuration page is shown.

If the device is not in access point mode, the normal Web Client is shown with a green bar on top saying “You are currently accessing the local version of your SUSI.AI running on your smart device. Configure now”
Clicking on the Configure now link redirects the user to the control page, where the user can change the settings for their smart speaker and control media playback.


To integrate both the control web page and the configure web page in the web client we needed to combine the flask server for the control and the configure page. This is done by adding all the endpoints of the configure server to the sound server and then renaming the sound server to control server – Merge Servers

Serving the WebClient through the control server


The next step is to add the Web Client static code to the control server and make it available on the root URL of the server.
To do this, first, we have to clone the web client during the installation and add the required symbolic links for the server.

echo "Downloading: Susi.AI webclient"
if [ ! -d "susi.ai" ]
then
    git clone --depth 1 -b gh-pages https://github.com/fossasia/susi.ai.git
    ln -s $PWD/susi.ai/static/* $PWD/susi_installer/raspi/soundserver/static/
    ln -s $PWD/susi.ai/index.html $PWD/susi_installer/raspi/soundserver/templates/
else
    echo "WARNING: susi.ai directory already present, not cloning it!" >&2
fi


The next step is to add the route for the Web Client in the control flask server.


@app.route('/')
def index():
    return render_template('index.html')


Here the index.html file is of the Web Client which we linked to the templates folder in the previous step.


Connecting the web client to the locally running SUSI Server


The Web Client by default connects to the SUSI Server at https://api.susi.ai but however the smart speaker has its own server running. To connect to the local server we do the following

– Before building the local version from the source of web client we need to set the process.env.REACT_APP_LOCAL_ENV environment variable to true.

– While building the javascript function check() checks if the above-mentioned environment variable is set to true and if yes, sets the API_URL to http://<host_IP_address>:4000

function check() {
  if (process && process.env.REACT_APP_LOCAL_ENV === 'true') {
    return 'http://' + window.location.hostname + ':4000';
  }
  return 'https://api.susi.ai';
}

const urls = {
  API_URL: check(),
  .  .//rest URLs
};

Resources

Javascript window.location – https://developer.mozilla.org/en-US/docs/Web/API/Window/location

Symlinks Linux – https://www.shellhacks.com/symlink-create-symbolic-link-linux/
SUSI.AI Web Client documentation – https://github.com/fossasia/susi.ai/blob/master/README.md

Tags

SUSI Smart Speaker, SUSI.AI, FOSSASIA, GSoC19

Continue Reading

Registering The SUSI Smart Speaker With your SUSI.AI account

When the SUSI Smart Speaker is set up for the first time it needs to be configured. After successful configuration, the smart speaker is registered with the associated account so that the user can see their smart speaker device information from the settings of their susi.ai account. There are two ways to configure  the smart speaker:

  • Through the android app
  • Through the Web Configuration Page

Both these processes are shown in detail here – https://github.com/fossasia/susi_installer/blob/development/docs/configure_guide.md


After the configuration setup is done, the Smart Speaker reboots and connects to your WiFi and registers the device with the given account using the login information provided during the setup.

 

Figure: Device Details are shown in the susi.ai account settings after successful configuration.

Working

The Auth Endpoint

Whenever the speaker is configured via the android app or manually via the web interface it uses various endpoints (access-point-server). For storing login information /auth endpoint is used. The /auth endpoint writes the login details to config.json file in /home/pi/SUSI.AI/config.json

The ss-susi-register service is then enabled i.e. the service will run in the next startup which will register the device online after the device is connected to the WiFi.

@app.route(‘/auth’, methods=[‘GET’])
def login():
    auth = request.args.get(‘auth’)
    email = request.args.get(’email’)
    password = request.args.get(‘password’)
    subprocess.call([‘sudo’, ‘-u’, ‘pi’, susiconfig, ‘set’, “susi.mode=”+auth, “susi.user=”+email, “susi.pass=”+password])
    display_message = {“authentication”:”successful”, “auth”: auth, “email”: email, “password”: password}
    if auth == ‘authenticated’ and email != “”:
        os.system(‘sudo systemctl enable ss-susi-register.service’)
    resp = jsonify(display_message)
    resp.status_code = 200
    return resp # pylint-enable


The SYSTEMD Registration Service

ss-susi-register.service – https://github.com/fossasia/susi_installer/blob/development/raspi/systemd/ss-susi-register.service

This is the service which registers the device on bootup after the configuration phase. The service waits for the network services to run such that the registration script is run only after when it is connected to a network. This service uses register.py to register the device online.

[Unit]
Description=Register the smart speaker online
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
WorkingDirectory=/home/pi/SUSI.AI
ExecStart=/usr/bin/python3 susi_installer/raspi/access_point/register.py

[Install]
WantedBy=multi-user.target


The Registration Script 

Register.py – https://github.com/fossasia/susi_installer/blob/development/raspi/access_point/register.py

This script is responsible for the following tasks

  • Get configuration information from config.json
config = json_config.connect(‘/home/pi/SUSI.AI/config.json’)
user = config[‘login_credentials’][’email’]
password = config[‘login_credentials’][‘password’]
room = config[‘room_name’]
  • Use the login information from config.json to get the authorization token for the respective account.
def get_token(login,password):
    url = ‘http://api.susi.ai/aaa/login.json?type=access-token’
    PARAMS = {
        ‘login’:login,
        ‘password’:password,
    }
    r1 = requests.get(url, params=PARAMS).json()
    return r1[‘access_token’]
  • Use the authorization token and other information from config.json and register the smart speaker online.
def device_register(access_token,room):
    g = geocoder.ip(‘me’)
    mac=’:’.join(re.findall(‘..’, ‘%012x’ % uuid.getnode()))
    url=’https://api.susi.ai/aaa/addNewDevice.json?&name=SmartSpeaker’
    PARAMS = {
        ‘room’:room,
        ‘latitude’:str(g.lat),
        ‘longitude’:str(g.lng),
        ‘macid’:mac,
        ‘access_token’:access_token
    }
    r1 = requests.get(url, params=PARAMS).json()
    return r1

  • If the registration fails put back the smart speaker in the access point(configuration) mode and reset the account information in config.json
try:
        access_token=get_token(user,password)
        out=device_register(access_token,room)
        logger.debug(str(out))
        break
    except:
        if i != 2:
            time.sleep(5)
            logger.warning(“Failed to register the device, retrying.”)
        else:
            logger.warning(“Resetting the device to hotspot mode”)
            config[‘usage_mode’]=”anonymous”
            config[‘login_credentials’][’email’]=””
            config[‘login_credentials’][‘password’]=””
            subprocess.Popen([‘sudo’,’bash’, ‘susi_installer/raspi/access_point/wap.sh’])

  • Disable the systemd service
    The script should run only once i.e. only after the configuration process, so the ss-susi-register.service needs to be disabled.
os.system(‘sudo systemctl disable ss-susi-register.service’)

Resources

Creating a Linux service with systemd – https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6

Running shell commands in python – https://cmdlinetips.com/2014/03/how-to-run-a-shell-command-from-python-and-get-the-output/

Tags

SUSI Smart Speaker, SUSI.AI, FOSSASIA, GSoC19

Continue Reading

Play Your Favourite Music Files on your SUSI Smart Speaker

The SUSI smart speaker supports playing local music from any USB device connected to the smart speaker. To play your favourite music directly from files, just put them in a thumb drive and plug it into any one of the four USB ports on the smart speaker. SUSI can either play all songs from the USB device or songs from a specific artist, genre or album.

Working

The first thing that needs to be done is to automount the thumb drive in the smart speaker, for this the usbmount package is used. Further, after the mount is done local skills are created which are then used by the SUSI server to interpret voice commands related to offline music playback.

Breakdown

The code which enables the above functionality is : 

  1. Add dependency – https://github.com/fossasia/susi_installer/blob/1a2950b0eb1f88d4ecbd5b3c348d9b67ac2f4705/install.sh#L287
# usbmount is needed to automount usb drives on susibian(raspbian lite)
if [ $targetSystem = raspi ] ; then
DEBDEPS=”$DEBDEPS hostapd dnsmasq usbmount”
fi


USB mount is added to the dependency list if the installer is running on a Raspberry

  1. Enable offline skills – https://github.com/fossasia/susi_installer/blob/1a2950b0eb1f88d4ecbd5b3c348d9b67ac2f4705/install.sh#L881
mkdir -p $WORKDIR/susi_server_data/generic_skills/media_discovery
touch $WORKDIR/susi_server_data/generic_skills/media_discovery/custom_skill.txt
mkdir -p $WORKDIR/susi_server_data/settings
echo “local.mode = true” > $WORKDIR/susi_server_data/settings/customized_config.properties

Enable the server to work with offline skills stored on the device. The new skills related to offline music playback are stored in /susi_server_data/generic_skills/media_discovery in a file named custom_skill.txt.

  1. Creating on the fly skills – https://github.com/fossasia/susi_installer/blob/1a2950b0eb1f88d4ecbd5b3c348d9b67ac2f4705/install.sh#L758
echo “Preparing USB automount”
# systemd-udevd creates its own filesystem namespace, so mount is done, but it is not visible in the principal namespace.
sudo mkdir /etc/systemd/system/systemd-udevd.service.d/
echo -e “[Service]\nPrivateMounts=no” | sudo tee /etc/systemd/system/systemd-udevd.service.d/udev-service-override.conf

# readonly mount for external USB drives
sudo sed -i -e ‘/^MOUNTOPTIONS/ s/sync/ro/’ /etc/usbmount/usbmount.conf
sudo cp $INSTALLERDIR/raspi/media_daemon/01_create_skill /etc/usbmount/mount.d/
sudo cp $INSTALLERDIR/raspi/media_daemon/01_remove_auto_skill /etc/usbmount/umount.d/


First an override rule is added, which changes `PrivateMounts` rule’s value to `no` in /lib/systemd/system/systemd-udevd.service.  PrivateMounts if set to yes, the processes of this unit will be run in their own private file system (mount) namespace with all mount propagation from the processes towards the host’s main file system namespace turned off. This means any file system mount points established or removed by the unit’s processes will be private to them and not be visible to the host. To learn more about mount namespaces read –
http://man7.org/linux/man-pages/man7/mount_namespaces.7.html 

Next, whenever a device is mounted the 01_create_skill file is executed which contains the following instruction:

python3 /home/pi/SUSI.AI/susi_installer/raspi/media_daemon/auto_skills.py “$UM_MOUNTPOINT”

This calls the auto_skills.py file with the mount point of the storage device.

The auto_skills.py file is used to generate audio skills for the USB drive. It scans for all the files in the USB thumb drive and creates relevant skills.

Whenever the thumb drive is removed it calls out the 01_remove_auto_skill script which has the following instruction –


echo -n > /home/pi/SUSI.AI/susi_server_data/generic_skills/media_discovery/custom_skill.txt


This cleans out the custom_skills.txt file i.e. all the offline skills that were created for music playback are removed and the server no longer responds those skills.

USAGE

Play all music on the USB device

Usage: SUSI, Play Audio

This will play all audio from the USB device connected to the speaker. SUSI Smart speaker currently supports the following audio formats:

  • MP3
  • FLAC
  • OGG
  • WAV

Play All Songs From an Artist

Usage : SUSI, play <artist_name> from USB

Example : SUSI, play Linkin Park from USB

This will play and queue all songs from the given artist if found on the USB device.

Play a Specific Music Genre

Usage : SUSI, play <Genre> from USB

Example : SUSI, play Hard Rock from USB

This will play and queue songs from the USB device that matches the given genre.

Play an Album

Usage : SUSI, play <album_name> from USB

Example : SUSI, play Hybrid Thoery from USB

This will play and queue songs from a specific Album Name.

Note: The above three skills depend on the metadata of the file. The file should have relevant metadata for these skills to work.

Playback Control

Usage : SUSI, <control_keyword>

Example : SUSI, pause or SUSI, resume

Available Music Playback Control keywords

  • Pause : Pause the currently playing music
  • Resume : Resume the currently playing music if paused
  • Restart : Restart the currently playing Music
  • Next : Go to the next song in the current playlist
  • Previous : Plays the previous song in the current playlist
  • Shuffle : Shuffles all songs in the current playlist and play again

Note : Playlist is made for offline Music skills such as play audio or play album from USB.

Tags

SUSI Smart Speaker, SUSI.AI, FOSSASIA, GSoC19

Resources

Continue Reading

Control Your Susi Smart Speaker

The SUSI Smart Speaker is an AI assistant device which runs SUS.AI. To learn to set up your own smart speaker, head up to SUSI Installer. One of the new features of the smart speaker is the ability to control it via a webpage, the smarts speaker now allows the user to control various playback features such play/pause music directly via their mobile phones or laptops which are in the same network. The web page is served via the sound server running locally on the Raspberry Pi. The soundserver provides various methods of the vlcplayer as endpoints. The webpage uses these endpoints to control the smart speaker. Also, an external application such as an android/ios app can use these endpoints(or the webpage) to control the music playback on the device.

Making the Front-end

The front end is served via the flask server on ‘ / ’ endpoint and on the port 7070. Currently, the Front End contains the volume control slider and various buttons to control the audio playback of the device. The responses are sent to the server via javascript. Bootstrap is used for the CSS framework and Fontawesome is used for various icon support. Since the smart speaker should be able to run offline, CDN links for Bootstrap and Fontawesome are not used and the required files are served via the flask server on /static. 

Adding required frameworks:

    <link href=”{{ url_for(‘static’, filename=’bootstrap.min.css’) }}” rel=”stylesheet”>
    <script type=”text/javascript” src=”{{
      url_for(‘static’,filename=’fontawesome.min.js’)
    }}”></script>

Web Page front-end

<div class=”form-signin”>
      <img class=”mb-4″ src=”{{ url_for(‘static’,       filename=’SUSI.AI_Icon_2017a.svg’) }}” alt=”” width=”256″       height=”256″>      {SUSI.AI Icon}
      <h1 class=”h3 mb-3 font-weight-normal”>Smart Speaker Control</h1>
        <div class=”form-group”>
          <fieldset class=”the-fieldset”>
              <legend class=”text-left w-auto”>Volume Control</legend>
              <span class=”font-weight-bold”>0</span>
              <i class=”fas fa-volume-down”></i>
              <input id=”vol-control” class=”slider” type=”range” min=”0″                  max=”100″ value=”100″ step=”1″ oninput=”SetVolume(this.value)”               onchange=”SetVolume(this.value)”></input>              {Volume Control Slider}
              <i class=”fas fa-volume-up”></i>
              <span class=”font-weight-bold”>100</span>
          </fieldset>
          <fieldset class=”the-fieldset”>
              <legend class=”text-left w-auto”>Playback Control</legend>
          <button onclick=”control(‘pause’)”                   class=”btn btn-outline-primary m-2″>
          Pause
          <i class=”fas fa-pause”></i>
          </button>                  {pause control button}
            {similar to the pause button other required buttons are added}
          </fieldset>
          <button onclick=”window.location.href = ‘/set_password’;”            class=”btn btn-warning m-2″>
            Set or Change Password
            <i class=”fa fa-key”></i>
          </button>
        </div>
    </div>

Sending Response to Server

Since this is a control webpage, on sending of a response, the webpage should not reload. To accomplish this all the buttons point to a javascript function which then sends out an HTTP POST request to the server. For this purpose XMLHttpRequest Object is used. The XMLHttpRequest object is used to exchange data with a web server behind the scenes. 


Here the SetVolume function is used to send a request to the /volume endpoint which is used to control the volume of the device. The control function is used to send a post request to audio control endpoints such as /pause /stop /shuffle etc.

      function control(action){
        console.log(action)
        var http = new XMLHttpRequest();
        var url = ‘/’+action;
        http.open(‘POST’, url, true);
        http.send();
      }
      function SetVolume(val){
        console.log(val)
        var http = new XMLHttpRequest();
        var url = ‘/volume?val=’+val;
        http.open(‘POST’, url, true);
        http.send();
      }

Features

The endpoints on the server provide the different audio control features via the vlc player. The endpoints used are listed below –

Play

The play functionality currently is only used directly via the busy state. It currently supports playback via youtube URL or MRLs. 

To play using an MRL(Media Resource Locator) the request URL should have an argument called MRL with the needed MRL value. This also supports multiple semicolon ‘ ; ‘ separated MRLs in a single request. 

Example Request URL: http://127.0.0.1:7070/play?mrl=/home/user/Desktop/song1.mp3;/home/user/Desktop/song2.mp3

@app.route(‘/play’, methods=[‘POST’, ‘PUT’])
def play_route():
    if ‘ytb’ in request.args:
        vlcplayer.playytb(request.args.get(‘ytb’))
        return do_return(‘Ok’, 200)
    elif ‘mrl’ in request.args:
        vlcplayer.play(request.args.get(‘mrl’))
        return do_return(‘Ok’, 200)
    else:
        return do_return(‘Unknown play mode’, 400)

Stop, Next, Previous, Pause and Resume

Stop, next and previous use the inbuilt methods of the MediaListPlayer class and are implemented in the same way. The request type must be POST and the request URL doesn’t require any arguments.

@app.route(‘/stop’, methods=[‘POST’, ‘PUT’])
def stop_route():
    vlcplayer.stop()
    return do_return(‘Ok’, 200)

Pause and Resume are also implemented in the same way but both of these use the same method pause of the MediaListPlayer class as that method acts as a toggle.

Shuffle

The shuffle endpoint shuffles the currently playing song list. It uses the shuffle method of the random library to shuffle the list containing MRLs of all the songs and then initiates a new MediaListPlayer object for playback. The Request URL doesn’t need any arguments.

    def shuffle(self):
        if self.is_playing():
            self.list_player.stop()
            random.shuffle(self.mrl)
            media_list = self.instance.media_list_new(self.mrl)
            self.list_player.set_media_list(media_list)
            self.list_player.play()
            self.softvolume(100, self.player)

Restart

The restart endpoint is used to restart the currently playing audio. It does so by going back in the playlist and playing the current audio again. The Request URL doesn’t need any arguments.

@app.route(‘/restart’, methods=[‘POST’, ‘PUT’])
def restart_route():
    vlcplayer.restart()
    return do_return(‘Ok’, 200)

Volume

The Volume endpoint is used to set the volume of the device, The volume control slider uses this endpoint. A single argument val is needed in the URL of the POST request. Val can have a value ranging from 0 to 100, where 0 means mute and 100 means full volume.

@app.route(‘/volume’, methods=[‘POST’, ‘PUT’])
def volume_route():
    try:
        vlcplayer.volume(request.args.get(‘val’))
        return do_return(‘Ok’, 200)
    except Exception as e:
        logger.error(e)
        return do_return(‘Volume adjustment error’ + e, 400)

Resources

Javascript XML HTTP Requests – https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Using_XMLHttpRequest

Flask HTML Templates –
https://pythonhow.com/html-templates-in-flask/

Tags

SUSI Smart Speaker, SUSI.AI, FOSSASIA, GSoC19

Continue Reading

Audio Structure of SUSI Smart Speaker

Previously whenever a sound had to be played via the smart speaker, the subprocess python module was used to call the CVLC process and play audio via it. This puts forward a number of challenges while implementing various music features such as queuing songs, shuffling songs or handle the volume of the music. Thus, the audio structure was remade in the SUSI Smart Speaker project. The audio playing structure resides mainly in the susi_installer and the susi_linux repository. The above flow chart describes how audio is handled now in the project.

Location of related files:

Busy State: susi_linux/main/states/busy_state.py

Player: susi_linux/main/player

Sound Server: susi_installer/raspi/soundserver

VLC Player: susi_installer/pythonmods/vlcplayer

HW mixer: susi_installer/pythonmods/hwmixer

In the new structure instead of using CVLC and a youtube URL server, we use Python VLC with a sound server. The new structure as given in the flowchart is illustrated step by step below-


 Busy State

The busy state class sends and accepts responses by the server through the susi_python wrapper. Here is the code which is responsible for handling play audio from a USB thumb drive or youtube.

if ‘identifier’ in reply.keys():
    url = reply[‘identifier’]
    if url[:3] == ‘ytd’:
        player.playytb(url[4:])
    else:
        player.play(url)
    self.transition(self.allowedStateTransitions.get(‘idle’))

For example “SUSI, play audio” will have a response like : 

{‘identifier’: ‘file:///media/usb0/example.mp3’, ‘answer’: ‘Playing audio from your usb device’}


For the above response the play method of the player class will be called.

Player

The Player class checks if the soundserver is running and sends the request to it. If the soundserver is not running then the VLC Player class is used directly.

Play method:  

    def play(self, mrl, mode = None):
        self._executeArg(‘play’, ‘mrl’, mrl, mode)
    def _executeArg(self, method, key, arg, mode = None):
        if (mode == ‘server’) or ((mode is None) and (self.mode == ‘server’)):
            send_request(method + ‘?’ + key + ‘=’ + arg)
        else:
            getattr(vlcplayer, method)(arg)

Sound Server

The soundserver provides various methods of the VLC Player as endpoints. These endpoints are then used by the busy state directly or via the remote access webpage or an external application. An external application such as an android/ios app can use these endpoints to control the music playback on the device.

@app.route(‘/play’, methods=[‘POST’, ‘PUT’])
def play_route():
    if ‘ytb’ in request.args:
        vlcplayer.playytb(request.args.get(‘ytb’))
        return do_return(‘Ok’, 200)
    elif ‘mrl’ in request.args:
        vlcplayer.play(request.args.get(‘mrl’))
        return do_return(‘Ok’, 200)
    else:
        return do_return(‘Unknown play mode’, 400)

VLC Player

The VLC Player class is actually responsible for playing and handling the music. This uses python VLC module for handling audio playback and various other functionalities. We use the Media List Player class found in the VLC player module to play music, using Media list player over media player gives us the advantage of queuing the files and essentially making a playlist.

For more info on MediaListPlayer class visit – https://www.olivieraubert.net/vlc/python-ctypes/doc/vlc.MediaListPlayer-class.html

class VlcPlayer():

    def __init__(self):
        self.saved_softvolume = -1
        self.saved_hardvolume = -1
        self.instance = vlc.Instance(“–no-video”)
        self.player = self.instance.media_player_new()
        self.sayplayer = self.instance.media_player_new()
        self.list_player =  self.instance.media_list_player_new()
        self.list_player.set_media_player(self.player)

The play method in VLC player

    def play(self, mrl_string):
        self.mrl = mrl_string.split(“;”)
        media_list = self.instance.media_list_new(self.mrl)
        self.list_player.set_media_list(media_list)
        self.list_player.play()
        self.softvolume(100, self.player)

The play method receives a single MRL or multiple MRLs. If multiple MRLs are sent, they are separated via a semicolon ‘;’. The list player method of VLC Player class takes a list of MRLs as an input so if the received string has more than one MRLs it is broken down into a list of MRLs via the python’s inbuilt split method, which is then added to the list player.

At last play method of the mediaListPlayer class is used to play the music/audio.

Resources

Python VLC library – https://pypi.org/project/python-vlc/

VLC python bindings – https://wiki.videolan.org/Python_bindings

Tags

SUSI Smart Speaker, SUSI.AI, FOSSASIA, GSoC19

Continue Reading

Setting up your own Smart Speaker Assistant as Fast as Possible

If you’re worried about privacy concerns when using smart assistants or just want to build your own one with complete freedom then this guide will help you. SUSI.AI provides Artificial Intelligence for Smart Speakers, Personal Assistants, Robots, Help Desks and Chatbots. SUSI.AI is a completely free and open source software.

In this guide, we will be building our own smart speaker assistant which will talk to the user just like Alexa or Google home. The keyword will be “SUSI”.

Things you will need

  • Raspberry Pi
  • ReSpeaker 2-mics Hat / USB mic / USB sound card
  • SD card
  • speaker
  • 3.5mm Aux cable/ JST PH2.0 connector

Step 1 Download and flash the image

  1. Download the latest Susibian image from https://github.com/fossasia/susi_installer/releases. The downloaded image will look something like susibian-xxxxxxxxxxxx.img.xz
  2. Insert the SD card in your PC
  3. If you’re on Linux, open up a terminal window and go to your downloads folder(or the place where the image is downloaded) and type the following commands

Extract the image 

unxz susibian-<timestamp>.img.xz 

Example:

unxz susibian-201905170311.img.xz

Write the image to SD-card 

Find the Disk Name of your SD card device by typing lsblk into a terminal window.

Here, for me, SD card disk name name is sdc.

sudo dd if=<path_to_downloaded_image_file> of=/dev/<disk_name> bs=4M status=progress

Replace <path_to_downloaded_image_file> to path to susibian image.

Replace <disk_name> with the disk name found in the before mentioned step.

NOTE: In the example command above, sdc is the device name for my scenario only, please check your device name before executing the command as it can result in loss of data!

Or

Use balena etcher :

  1. Select the image
  2. Select SD card device
  3. Flash!

Step 2 Setting up the hardware

  1. Insert the microSD card into your raspberry pi
  2. Attach ReSpeaker Hat to your Raspberry Pi
  3. Connect your speaker through the 3.5mm jack on ReSpeaker Hat or connect your JST PH2.0 speakers using the onboard JST connector
  4. Plug in the power supply to your raspberry pi. 
  1. If you’re using a USB sound card make sure to follow – Audio Debugging

Step 3 Setting up the speaker

  1. Download SUSI-AI android app. https://github.com/fossasia/susi_android/releases/download/1.1/susiai_20180826.apk
  2. Create an account or log in.
  3. In the app, go to settings -> Devices -> click here to move to device setup screen
  4. Wait for your device to show up and then click on setup. 
susi.ai smart speaker
  1. Chose the wifi network you want your speaker to connect to and enter the password.
  2. Add credentials.

OR

  1. Connect your computer or mobile phone to the SUSI.AI hotspot using the password “password”.
  2. Open http://10.0.0.1:5000 which will show you the set-up page as visible below: 
  1. Put in your Wifi credentials. For an open network set an empty password. The device should connect automatically to any open network, leave SSID and password empty.
  2. Click on “Reboot Smart Speaker”

Testing

  1. Wait for reboot of the speaker, SUSI will say “SUSI has started” as soon it is ready.
  2. After the setup, three LEDs on top of ReSpeaker Hat should light up.
  3. Say “SUSI” and wait for a bell sound.
  4. Speak your query!!
  5. Enjoy your new personal assistant

Tags

SUSI Smart Speaker, SUSI.AI, FOSSASIA, GSoC19

Resources

Continue Reading
Close Menu