Bot Development

Read more about the article Maintaining Extension State in SUSI.AI Chrome Bot Using Chrome Storage API

Maintaining Extension State in SUSI.AI Chrome Bot Using Chrome Storage API

SUSI Chrome Bot is a browser action chrome extension which is used to communicate with SUSI AI.The browser action extension in chrome is like any other web app that you visit. It will store all your preferences like theme settings and speech synthesis settings and data till you are interacting with it, but once you close it, it forgets all of your data unless you are saving it in some database or you are using cookies for the session. We want to be able to save the chats and other preferences like theme settings the user makes when interacting with SUSI AI through Susi Chrome Bot. In this blog, we’ll explore Chrome’s chrome.storage API for storing data.

What options do we have for storing data offline?

IndexedDB: IndexedDB is a low-level API for client-side storage of data. IndexedDB allows us to store a large amount of data and works like RDBMS but IndexedDB is javascript based Object-oriented database.

localStorage API: localStorage allows us to store data in key/value pairs which is much more effective than storing data in cookies. localStorage data persists even if the user closes and reopens the browser.

Chrome.storage: Chrome provides us with chrome.storage. It provides the same storage capabilities as localStorage API with some advantages.

For susi_chromebot we will use chrome.storage because of the following advantages it has over the localstorage API:

  1. User data can be automatically synced with Chrome sync if the user is logged in.
  2. The extension’s content scripts can directly access user data without the need for a background page.
  3. A user’s extension settings can be persisted even when using incognito mode.
  4. It’s asynchronous so bulk read and write operations are faster than the serial and blocking localStorage API.
  5. User data can be stored as objects whereas the localStorage API stores data in strings.

Integrating chrome.storage to susi_chromebot for storing chat data

To use chrome.storage we first need to declare the necessary permission in the extension’s manifest file. Add “storage” in the permissions key inside the manifest file.

"permissions": [
         "storage"
       ]

 

We want to store the chat user has made with SUSI. We will use a Javascript object to store the chat data.

var storageObj = {
senderClass: "",
content: ""
};

The storageObj object has two keys namely senderClass and content. The senderClass key represents the sender of the message(user or susi) whereas the content key holds the actual content of the message.

We will use chrome.storage.get and chrome.storage.set methods to store and retrieve data.

var susimessage = newDiv.innerHTML;
storageObj.content = susimessage;
storageObj.senderClass = "susinewmessage";
chrome.storage.sync.get("message",(items) => {
if(items.message){
storageArr = items.message;
}
storageArr.push(storageObj);
chrome.storage.sync.set({"message":storageArr},() => {
console.log("saved");
});
});

 

In the above code snippet, susimessage contains the actual message content sent by the SUSI server. We then set the correct properties of the storageObj object that we declared earlier. Now we can use chrome.storage.set to save the storageObj object but that would overwrite the current data that we have inside chrome’s StorageArea. To prevent the old message data from getting overwritten, we’ll first get all the message content in our storage using chrome.storage.sync.get. Notice how we are passing the “message” string as the first perimeter to the function. This is done because we only want our message content which was saved in the StorageArea. If we pass null instead, it will return all the content inside storageArea. Once we have our messages (which will be an array of objects that we store as storageObj), we will store that into a new array storageArr. We will then push our new storageObj that contains the message and the sender into the array. Finally, we use chrome.storage.sync.set to save the message content in chrome’s StorageArea which can later be retrieved using the “message” key.

storageArr.push(storageObj);
chrome.storage.sync.set({"message":storageArr},() => {
console.log("saved");
});

We use the same procedure to save messages sent by the user.

Note: chrome.storage is not very large, so we need to be careful about what we store or we may run out of storage space. Also, we should not store confidential data in storage since the storage area is not encrypted.

Resources:

Tags:

  • FOSSASIA, codeheat, Chrome extensions, Javascript, Chrome Storage, Chrome Sync, Susi Chrome Bot, SUSI AI, Bot Development
Continue ReadingMaintaining Extension State in SUSI.AI Chrome Bot Using Chrome Storage API
Read more about the article SUSI.AI Chrome Bot and Web Speech: Integrating Speech Synthesis and Recognition

SUSI.AI Chrome Bot and Web Speech: Integrating Speech Synthesis and Recognition

Susi Chrome Bot is a Chrome extension which is used to communicate with Susi AI. The advantage of having chrome extensions is that they are very accessible for the user to perform certain tasks which sometimes needs the user to move to another tab/site.

In this blog post, we will be going through the process of integrating the web speech API to SUSI Chromebot.

Web Speech API

Web Speech API enables web apps to be able to use voice data. The Web Speech API has two components:

Speech Recognition:  Speech recognition gives web apps the ability to recognize voice data from an audio source. Speech recognition provides the speech-to-text service.

Speech Synthesis: Speech synthesis provides the text-to-speech services for the web apps.

Integrating speech synthesis and speech recognition in SUSI Chromebot

Chrome provides the webkitSpeechRecognition() interface which we will use for our speech recognition tasks.

var recognition = new webkitSpeechRecognition();

 

Now, we have a speech recognition instance recognition. Let us define necessary checks for error detection and resetting the recognizer.

var recognizing;

function reset() {
recognizing = false;
}

recognition.onerror = function(e){
console.log(e.error);
};

recognition.onend = function(){
reset();
};

 

We now define the toggleStartStop() function that will check if recognition is already being performed in which case it will stop recognition and reset the recognizer, otherwise, it will start recognition.

function toggleStartStop() {
    if (recognizing) {
      recognition.stop();
      reset();
    } else {
      recognition.start();
      recognizing = true;
    }
}

 

We can then attach an event listener to a mic button which calls the toggleStartStop() function to start or stop our speech recognition.

mic.addEventListener("click", function () {
    toggleStartStop();
});

 

Finally, when the speech recognizer has some results it calls the onresult event handler. We’ll use this event handler to catch the results returned.

recognition.onresult = function (event) {
    for (var i = event.resultIndex; i < event.results.length; ++i) {
      if (event.results[i].isFinal) {
        textarea.value = event.results[i][0].transcript;
        submitForm();
      }
    }
};

 

The above code snipped tests for the results produced by the speech recognizer and if it’s the final result then it sets textarea value with the result of speech recognition and then we submit that to the backend.

One problem that we might face is the extension not being able to access the microphone. This can be resolved by asking for microphone access from an external tab/window/iframe. For SUSI Chromebot this is being done using an external tab. Pressing on the settings icon makes a new tab which then asks for microphone access from the user. This needs to be done only once, so that does not cause a lot of trouble.

setting.addEventListener("click", function () {
chrome.tabs.create({
url: chrome.runtime.getURL("options.html")
});
});navigator.webkitGetUserMedia({
audio: true
}, function(stream) {
stream.stop();
}, function () {
console.log('no access');
});

 

In contrast to speech recognition, speech synthesis is very easy to implement.

function speakOutput(msg){
    var voiceMsg = new SpeechSynthesisUtterance(msg);
    window.speechSynthesis.speak(voiceMsg);
}

 

This function takes a message as input, declares a new SpeechSynthesisUtterance instance and then calls the speak method to convert the text message to voice.

There are many properties and attributes that come with this speech recognition and synthesis interface. This blog post only introduces the very basics.

Resources

 

Continue ReadingSUSI.AI Chrome Bot and Web Speech: Integrating Speech Synthesis and Recognition