API’s in SUSI.AI BotBuilder

In this blog, I’ll explain the different API’s involved in the SUSI.AI Bot Builder and its working. Now if you are wondering how the SUSI.AI bot builder works or how drafts are saved, then this is the perfect blog. I’ll be explaining the different API’s grouped by different API endpoints.

API Implementation

fetchChatBots

export function fetchChatBots() {
 const url = `${API_URL}/${CMS_API_PREFIX}/getSkillList.json`;
 return ajax.get(url, {
   private: 1,
 });
}

This API is used to fetch all your saved chatbots, which are displayed on the BotBuilder Page. The API endpoint is getSkillList.json. Same endpoint is used when a user creates a skill, the difference is query parameter private is passed which then returns your chatbots. Now if you are wondering why we have same endpoint for skills and chatbots, the simple plain reason for this is chatbots are your private skills.

fetchBotDetails

export function fetchBotDetails(payload) {
 const { model, group, language, skill } = payload;
 const url = `${API_URL}/${CMS_API_PREFIX}/getSkill.json`;
 return ajax.get(url, {
   model,
   group,
   language,
   skill,
   private: 1,
 });
}

This API is used to fetch details of bot/skill respectively from the API endpoint getSkill.json. Group name, language, skill name, private and model are passed as query parameters.

fetchBotImages

export function fetchBotImages(payload) {
 const { name: skill, language, group } = payload;
 const url = `${API_URL}/${CMS_API_PREFIX}/getSkill.json`;
 return ajax.get(url, {
   group,
   language,
   skill,
   private: 1,
 });
}

This API is used to fetch skill and bot images from the API endpoint getSkill.json. Group name, language, skill name and private are passed as query parameters.

uploadBotImage

export function uploadBotImage(payload) {
 const url = `${API_URL}/${CMS_API_PREFIX}/uploadImage.json`;
 return ajax.post(url, payload, {
   headers: { 'Content-Type': 'multipart/form-data' },
   isTokenRequired: false,
 });
}

This API is used to upload the Bot image to the API endpoint uploadImage.json.The Content-Type entity header is used to indicate the media type of the resource. multipart/form-data means no characters will be encoded. This is used when a form requires a binary data like the contents of a file or image to be uploaded.

deleteChatBot

export function deleteChatBot(payload) {
 const { group, language, skill } = payload;
 const url = `${API_URL}/${CMS_API_PREFIX}/deleteSkill.json`;
 return ajax.get(url, {
   private: 1,
   group,
   language,
   skill,
 });
}

This API is used to delete Skill and Bot from the API endpoint deleteSkill.json.

storeDraft

export function storeDraft(payload) {
 const { object } = payload;
 const url = `${API_URL}/${CMS_API_PREFIX}/storeDraft.json`;
 return ajax.get(url, { object });
}

This API is used to store draft Bot to the API endpoint storeDraft.json. The object passed as parameter has the properties given by the user such as skill name,group etc., while saving the draft.

readDraft

export function readDraft(payload) {
 const url = `${API_URL}/${CMS_API_PREFIX}/readDraft.json`;
 return ajax.get(url, { ...payload });
}

This API is used to fetch draft from the API endpoint readDraft.json. This API is called on the BotBuilder Page where all the saved drafts are shown.

deleteDraft

export function deleteDraft(payload) {
 const { id } = payload;
 const url = `${API_URL}/${CMS_API_PREFIX}/deleteDraft.json`;
 return ajax.get(url, { id });
}

This API is used to delete the saved Draft from the API endpoint deleteDraft.json. It only needs one query parameter i.e. the draft ID.

In conclusion, the above API’s are the backbone of the SUSI.AI Bot Builder. API endpoints in server ensure the user has the same experience across the clients. Do checkout implementation of different API endpoints in server here.

Resources

Continue ReadingAPI’s in SUSI.AI BotBuilder

Implementing a Chat Bubble in SUSI.AI

SUSI.AI now has a chat bubble on the bottom right of every page to assist you. Chat Bubble allows you to connect with susi.ai on just a click. You can now directly play test examples of skill on chatbot. It can also be viewed on full screen mode.

Redux Code

Redux Action associated with chat bubble is handleChatBubble. Redux state chatBubble can have 3 states:

  • minimised – chat bubble is not visible. This set state is set when the chat is viewed in full screen mode.
  • bubble – only the chat bubble is visible. This state is set when close icon is clicked and on toggle.
  • full – the chat bubble along with message section is visible. This state is set when minimize icon is clicked on full screen mode and on toggle.
const defaultState = {
chatBubble: 'bubble',
};
export default handleActions(
 {
  [actionTypes.HANDLE_CHAT_BUBBLE](state, { payload }) {
     return {
       ...state,
       chatBubble: payload.chatBubble,
     };
   },
 },
 defaultState,
);

Speech box for skill example

The user can click on the speech box for skill example and immediately view the answer for the skill on the chatbot. When a speech bubble is clicked a query parameter testExample is added to the URL. The value of this query parameter is resolved and answered by Message Composer. To be able to generate an answer bubble again and again for the same query, we have a reducer state testSkillExampleKey which is updated when the user clicks on the speech box. This is passed as key parameter to messageSection.

Chat Bubble Code

The functions involved in the working of chatBubble code are:

  • openFullScreen – This function is called when the full screen icon is clicked in the tab and laptop view and also when chat bubble is clicked in mobile view. This opens up a full screen dialog with message section. It dispatches handleChatBubble action which sets the chatBubble reducer state as minimised.
  • closeFullScreen – This function is called when the exit full screen icon is clicked. It dispatches a handleChatBubble action which sets the chatBubble reducer state as full.
  • toggleChat –  This function is called when the user clicks on the chat bubble. It dispatches handleChatBubble action which toggles the chatBubble reducer state between full and bubble.
  • handleClose – This function is called when the user clicks on the close icon. It dispatches handleChatBubble action which sets the chatBubble reducer state to bubble.
openFullScreen = () => {
   const { actions } = this.props;
   actions.handleChatBubble({
     chatBubble: 'minimised',
   });
   actions.openModal({
     modalType: 'chatBubble',
     fullScreenChat: true,
   });
 };

 closeFullScreen = () => {
   const { actions } = this.props;
   actions.handleChatBubble({
     chatBubble: 'full',
   });
   actions.closeModal();
 };

 toggleChat = () => {
   const { actions, chatBubble } = this.props;
   actions.handleChatBubble({
     chatBubble: chatBubble === 'bubble' ? 'full' : 'bubble',
   });
 };

 handleClose = () => {
   const { actions } = this.props;
   actions.handleChatBubble({ chatBubble: 'bubble' });
   actions.closeModal();
 };

Message Section Code (Reduced)

The message section comprises of three parts the actionBar, messageSection and the message Composer.

Action Bar

The actionBar consists of the action buttons – search, full screen, exit full screen and close button. Clicking on the search button expands and opens up a search bar. On clicking the full screen icon openFullScreen function is called which open up the chat dialog. On clicking the exit icon the handleClose function is called, which set chatBubble reducer state to bubble. On full screen view, clicking on the exit full screen icon calls the closeFullScreen functions which sets the reducer state chatBubble to full.

   const actionBar = (
     <ActionBar fullScreenChat={fullScreenChat}>
       {fullScreenChat !== undefined ? (
         <FullScreenExit onClick={this.closeFullScreen} width={width} />
       ) : (
         <FullScreen onClick={this.openFullScreen} width={width} />
       )}
       <Close onClick={fullScreenChat ? this.handleClose : this.toggleChat}/>
     </ActionBar>
   );

Message Section

The message section has two parts MessageList and Message Composer. Message List is where the messages are viewed and the Message composer allows you to interact with the bot through text and speech. ScrollBar is imported from the npm library react-custom-scrollbars. When the scroll bar is moved it sets the state of showScrollTop and showScrollBottom in the chat. messageListItems consists of all the messages between the user and the bot.


   const messageSection = (
     <MessageSectionContainer showChatBubble={showChatBubble} height={height}>
       {loadingHistory ? (
         <CircularLoader height={38} />
       ) : (
         <Fragment>
           {fullScreenChat ? null : actionBar}
           <MessageList
             ref={c => {
               this.messageList = c;
             }}
             pane={pane}
             messageBackgroundImage={messageBackgroundImage}
             showChatBubble={showChatBubble}
             height={height}>
             <Scrollbars
               renderThumbHorizontal={this.renderThumb}
               renderThumbVertical={this.renderThumb}
               ref={ref => {
                 this.scrollarea = ref;
               }}
               onScroll={this.onScroll}
               autoHide={false}>
               {messageListItems}
               {!search && loadingReply && this.getLoadingGIF()}
             </Scrollbars>
           </MessageList>
           {showScrollTop && (
             <ScrollTopFab
               size="small"
               backgroundcolor={body}
               color={theme === 'light' ? 'inherit' : 'secondary'}
               onClick={this.scrollToTop}
             >
               <NavigateUp />
             </ScrollTopFab>
           )}
           {showScrollBottom && (
             <ScrollBottomContainer>
               <ScrollBottomFab
                 size="small"
                 backgroundcolor={body}
                 color={theme === 'light' ? 'inherit' : 'secondary'}
                 onClick={this.scrollToBottom}>
                 <NavigateDown />
               </ScrollBottomFab>
             </ScrollBottomContainer>
           )}
         </Fragment>
       )}
       <MessageComposeContainer
         backgroundColor={composer}
         theme={theme}
         showChatBubble={showChatBubble}>
         <MessageComposer
           focus={!search}
           dream={dream}
           speechOutput={speechOutput}
           speechOutputAlways={speechOutputAlways}
           micColor={button}
           textarea={textarea}
           exitSearch={this.exitSearch}
           showChatBubble={showChatBubble}
         />
       </MessageComposeContainer>
     </MessageSectionContainer>
   );

 const Chat = (
     <ChatBubbleContainer className="chatbubble" height={height} width={width}>
       {chatBubble === 'full' ? messageSection : null}
       {chatBubble !== 'minimised' ? (
         <SUSILauncherContainer>
           <SUSILauncherWrapper
             onClick={width < 500 ? this.openFullscreen : this.toggleChat}>
             <SUSILauncherButton data-tip="Toggle Launcher" />
           </SUSILauncherWrapper>
         </SUSILauncherContainer>
       ) : null}
     </ChatBubbleContainer>
   );

Resources

Continue ReadingImplementing a Chat Bubble in SUSI.AI

List SUSI.AI Devices in Admin Panel

In this blog I’ll be explaining about the Devices Tab in SUSI.AI Admin Panel. Admins can now view the connected devices of the users with view, edit and delete actions. Also the admins can directly view the location of the device on the map by clicking on the device location of that user.

Implementation

List Devices

Admin device Tab

Devices tab displays device name, macId, room, email Id, date added, last active, last login IP and location of the device. loadDevices function is called on componentDidMount which calls the fetchDevices API which fetches the list of devices from /aaa/getDeviceList.json endpoint. List of all devices is stored in devices array. Each device in the array is an object with the above properties. Clicking on the device location opens a popup displaying the device location on the map.

loadDevices = () => {
   fetchDevices()
     .then(payload => {
       const { devices } = payload;
       let devicesArray = [];
       devices.forEach(device => {
         const email = device.name;
         const devices = device.devices;
         const macIdArray = Object.keys(devices);
         const lastLoginIP =
           device.lastLoginIP !== undefined ? device.lastLoginIP : '-';
         const lastActive =
           device.lastActive !== undefined
             ? new Date(device.lastActive).toDateString()
             : '-';
         macIdArray.forEach(macId => {
           const device = devices[macId];
           let deviceName = device.name !== undefined ? device.name : '-';
           deviceName =
             deviceName.length > 20
               ? deviceName.substr(0, 20) + '...'
               : deviceName;
           let location = 'Location not given';
           if (device.geolocation) {
             location = (
               
                 {device.geolocation.latitude},{device.geolocation.longitude}
               
             );
           }
           const dateAdded =
             device.deviceAddTime !== undefined
               ? new Date(device.deviceAddTime).toDateString()
               : '-';
 
           const deviceObj = {
             deviceName,
             macId,
             email,
             room: device.room,
             location,
             latitude:
               device.geolocation !== undefined
                 ? device.geolocation.latitude
                 : '-',
             longitude:
               device.geolocation !== undefined
                 ? device.geolocation.longitude
                 : '-',
             dateAdded,
             lastActive,
             lastLoginIP,
           };
           devicesArray.push(deviceObj);
         });
       });
       this.setState({
         loadingDevices: false,
         devices: devicesArray,
       });
     })
     .catch(error => {
       console.log(error);
     });
 };

View Device

User Device Page

View action redirects to users /mydevices?email<email>&macid=<macid>. This allows admin to have full control of the My devices section of the user. Admin can change device details and delete device. Also admin can see all the devices of the user from the ALL tab. To edit a device click on edit icon in the table, update the details and click on check icon. To delete a device click on the delete device which then asks for confirmation of device name and on confirmation deletes the device.

Edit Device

Edit Device Dialog

Edit actions opens up a dialog modal which allows the admin to update the device name and room. Clicking on the edit button calls the modifyUserDevices API which takes in email Id, macId, device name and room name as parameters. This calls the API endpoint /aaa/modifyUserDevices.json.

 handleChange = event => {
   this.setState({ [event.target.name]: event.target.value });
 };

 render() {
   const { macId, email, handleConfirm, handleClose } = this.props;
   const { room, deviceName } = this.state;
   return (
     <React.Fragment>
       <DialogTitle>Edit Device Details for {macId}</DialogTitle>
       <DialogContent>
         <OutlinedTextField
           value={deviceName}
           label="Device Name"
           name="deviceName"
           variant="outlined"
           onChange={this.handleChange}
           style={{ marginRight: '20px' }}
         />
         <OutlinedTextField
           value={room}
           label="Room"
           name="room"
           variant="outlined"
           onChange={this.handleChange}
         />
       </DialogContent>
       <DialogActions>
         <Button
           key={1}
           color="primary"
           onClick={() => handleConfirm(email, macId, room, deviceName)}>
           Change
         </Button>
         <Button key={2} color="primary" onClick={handleClose}>
           Cancel
         </Button>
       </DialogActions>
     </React.Fragment>
   );
 }

Delete Device

Delete Device Dialog

Delete action opens up a confirm delete dialog modal. To delete a device enter the device name and click on delete. This calls the confirmDelete function which calls the removeUserDevice API which takes in email Id and macId as parameters. This API hits the endpoint /aaa/removeUserDevices.json.

confirmDelete = () => {
   const { actions } = this.props;
   const { macId, email } = this.state;
   removeUserDevice({ macId, email })
     .then(payload => {
       actions.openSnackBar({
         snackBarMessage: payload.message,
         snackBarDuration: 2000,
       });
       actions.closeModal();
       this.setState({
         loadingDevices: true,
       });
       this.loadDevices();
     })
     .catch(error => {
       actions.openSnackBar({
         snackBarMessage: `Unable to delete device with macID ${macId}. Please try again.`,
         snackBarDuration: 2000,
       });
     });
 };

To conclude, admin can now view all the connected SUSI.AI devices along with the user details and location. They can also access users My Devices tab in Dashboard and update and delete devices.

Resources

Continue ReadingList SUSI.AI Devices in Admin Panel

Displaying Private Skills and Drafts on SUSI.AI

The ListPrivateSkillService and ListPrivateDraftSkillService endpoint was implemented on SUSI.AI Server for SUSI.AI Admins to view the bots and drafts created by users respectively. This allows admins to monitor the bots and drafts created by users, and delete the ones which violate the guidelines. Also admins can see the sites where the bot is being used.

The endpoint of both ListPrivateSkillService and ListPrivateDraftSkillService is of GET type. Both of them have a compulsory access_token parameter but ListPrivateSkillService has an extra optional search parameter.

  • access_token(necessary): It is the access_token of the logged in user. It means this endpoint cannot be accessed in anonymous mode. 
  • search: It fetches a bot with the searched name.

The minimum user role is set to OPERATOR.

API Development

ListPrivateSkillService

For creating a list, we need to access each property of botDetailsObject, in the following manner:

Key → Group  → Language → Bot Name  → BotList

The below code iterates over the uuid of all the users having a bot, then over different groupNames,languageNames, and finally over the botNames. If search parameter is passed then it searches for the bot_name in the language object. Each botDetails object consists of bot name, language, group and key i.e uuid of the user which is then added to the botList array.

       JsonTray chatbot = DAO.chatbot;
       JSONObject botDetailsObject = chatbot.toJSON();
       JSONObject keysObject = new JSONObject();
       JSONObject groupObject = new JSONObject();
       JSONObject languageObject = new JSONObject();
       List botList = new ArrayList();
       JSONObject result = new JSONObject();

       Iterator Key = botDetailsObject.keys();
       List keysList = new ArrayList();

       while (Key.hasNext()) {
           String key = (String) Key.next();
           keysList.add(key);
       }

       for (String key_name : keysList) {
           keysObject = botDetailsObject.getJSONObject(key_name);
           Iterator groupNames = keysObject.keys();
           List groupnameKeysList = new ArrayList();

           while (groupNames.hasNext()) {
               String key = (String) groupNames.next();
               groupnameKeysList.add(key);
           }

           for (String group_name : groupnameKeysList) {
               groupObject = keysObject.getJSONObject(group_name);
               Iterator languageNames = groupObject.keys();
               List languagenamesKeysList = new ArrayList();

               while (languageNames.hasNext()) {
                   String key = (String) languageNames.next();
                   languagenamesKeysList.add(key);
               }

               for (String language_name : languagenamesKeysList) {
                   languageObject = groupObject.getJSONObject(language_name);

If search parameter is passed, then search for a bot with the given name and add the bot to the botList if it exists. It will return all bots which have bot name as the searched name.

                   if (call.get("search", null) != null) {
                       String bot_name = call.get("search", null);
                       if(languageObject.has(bot_name)){
                           JSONObject botDetails = languageObject.getJSONObject(bot_name);
                           botDetails.put("name", bot_name);
                           botDetails.put("language", language_name);
                           botDetails.put("group", group_name);
                           botDetails.put("key", key_name);
                           botList.add(botDetails);
                       }
                   }

If search parameter is not passed, then it will return all the bots created by the users.

                    else {
                       Iterator botNames = languageObject.keys();
                       List botnamesKeysList = new ArrayList();

                       while (botNames.hasNext()) {
                           String key = (String) botNames.next();
                           botnamesKeysList.add(key);
                       }

                       for (String bot_name : botnamesKeysList) {
                           JSONObject botDetails = languageObject.getJSONObject(bot_name);
                           botDetails.put("name", bot_name);
                           botDetails.put("language", language_name);
                           botDetails.put("group", group_name);
                           botDetails.put("key", key_name);
                           botList.add(botDetails);
                       }
                   }
               }
           }
       }

List of all bots, botList is return as server response.

ListPrivateDraftSkillService

For creating a list we need to iterate over each user and check whether the user has a draft bot. We get all the authorized clients from DAO.getAuthorizedClients(). We then iterate over each client and get their identity and authorization. We get the drafts of the client from DAO.readDrafts(userAuthorization.getIdentity()). We then iterate over each draft and add it to the drafts object. Each draft object consists of date created,date modified, object which contains draft bot information such as name,language,etc provided by the user while saving the draft, email Id and uuid of the user.

       JSONObject result = new JSONObject();
       List draftBotList = new ArrayList();
       Collection authorized = DAO.getAuthorizedClients();

       for (Client client : authorized) {
         String email = client.toString().substring(6);
         JSONObject json = client.toJSON();
         ClientIdentity identity = new ClientIdentity(ClientIdentity.Type.email, client.getName());
         Authorization userAuthorization = DAO.getAuthorization(identity);
         Map map = DAO.readDrafts(userAuthorization.getIdentity());
         JSONObject drafts = new JSONObject();

         for (Map.Entry entry: map.entrySet()) {
           JSONObject val = new JSONObject();
           val.put("object", entry.getValue().getObject());
           val.put("created", DateParser.iso8601Format.format(entry.getValue().getCreated()));
           val.put("modified", DateParser.iso8601Format.format(entry.getValue().getModified()));
           drafts.put(entry.getKey(), val);
         }
         Iterator keys = drafts.keySet().iterator();
         while(keys.hasNext()) {
           String key = (String)keys.next();
           if (drafts.get(key) instanceof JSONObject) {
             JSONObject draft = new JSONObject(drafts.get(key).toString());
             draft.put("id", key);
             draft.put("email", email);
             draftBotList.add(draft);
           }
         }
       }
       result.put("draftBots", draftBotList);

List of all drafts, draftBotList is returned as server response.

In conclusion, the admins can now see the bots and drafts created by the user and monitor where they are being used.

Resources

Continue ReadingDisplaying Private Skills and Drafts on SUSI.AI

CRUD operations on Config Keys in Admin Panel of SUSI.AI

SUSI.AI Admin Panel now allows the Admin to create, read, update and delete config keys present in system settings. Config keys are API keys which are used to link the application to third party services like Google Maps, Google ReCaptcha, Google Analytics, Matomo, etc. The API key is a unique identifier that is used to authenticate requests associated with the project for usage and billing purposes.

CRUD Operations

Create Config Key

To create a config key click on “Add Config Key” Button, a dialog opens up which has two field Key Name and Key Value. this.props.actions.openModal opens up the shared Dialog Modal. On clicking on “Create”, the createApiKey is called which takes in the two parameters.

handleCreate = () => {
   this.props.actions.openModal({
     modalType: 'createSystemSettings',
     type: 'Create',
     handleConfirm: this.confirmUpdate,
     keyName: this.state.keyName,
     keyValue: this.state.keyValue,
     handleClose: this.props.actions.closeModal,
   });
 };
 handleSave = () => {
   const { keyName, keyValue } = this.state;
   const { handleConfirm } = this.props;
   createApiKey({ keyName, keyValue })
     .then(() => handleConfirm())
     .catch(error => {
       console.log(error);
     });
 }; 

Read Config Key

API endpoint fetchApiKeys is called on componentDidMount and when Config Key is created, updated or deleted.

 fetchApiKeys = () => {
   fetchApiKeys()
     .then(payload => {
       let apiKeys = [];
       let i = 1;
       let keys = Object.keys(payload.keys);
       keys.forEach(j => {
         const apiKey = {
           serialNum: i,
           keyName: j,
           value: payload.keys[j],
         };
         ++i;
         apiKeys.push(apiKey);
       });
       this.setState({
         apiKeys: apiKeys,
         loading: false,
       });
     })
     .catch(error => {
       console.log(error);
     });
 };

Update Config Key

To Update a config key click on edit from the actions column, Update Config Key dialog opens up which allows you to edit the key value. On clicking on update, the createApiKey API is called.

 handleUpdate = row => {
   this.props.actions.openModal({
     modalType: 'updateSystemSettings',
     type: 'Update',
     keyName: row.keyName,
     keyValue: row.value,
     handleConfirm: this.confirmUpdate,
     handleClose: this.props.actions.closeModal,
   });
 };

Delete Config Key

To delete a config key click on delete from actions column, delete config key confirmation dialog opens up. On clicking on Delete, the deleteApiKey is called which takes in key name as parameter.

 handleDelete = row => {
   this.setState({ keyName: row.keyName });
   this.props.actions.openModal({
     modalType: 'deleteSystemSettings',
     keyName: row.keyName,
     handleConfirm: this.confirmDelete,
     handleClose: this.props.actions.closeModal,
   });
 };
 confirmDelete = () => {
   const { keyName } = this.state;
   deleteApiKey({ keyName })
     .then(this.fetchApiKeys)
     .catch(error => {
       console.log(error);
     });
   this.props.actions.closeModal();
 };

In conclusion, CRUD operations of Config Keys help admins to manage third party services. With these operations the admin can manage the API keys of various services without having to look for them in the backend.

Resources

Continue ReadingCRUD operations on Config Keys in Admin Panel of SUSI.AI

Neurolab data transfer – Establishing serial communication between Arduino and Android

In the development process of the Neurolab Android, we needed an Arduino-Android connection for transfer of data from datasets which included String and float data type values. In this blog post, I will show you how to establish a serial communication channel between

Android and Arduino through USB cable through which we can transmit data bidirectionally.

Requirements

Hardware:

  1. Android Phone
  2. Arduino (theoretically from any type, but I’ll be using Arduino Uno)
  3. USB 2.0 Cable Type A/B (for Arduino)
  4. OTG Cable (On The Go)
  5. Normal USB Cable (for transferring the data from Android Studio to your phone)

Software:

  1. Android Studio
  2. Arduino IDE

Wiring and Setup

Wiring must be established in the following way:

                                                                                 Figure: Android-Arduino Setup

Working on the Android Side

We would be using the UsbSerial Library by felHR85 for establishing serial

communication.

1. Adding the dependency:

a) Add the following line of code to your app level build.gradle file.

implementation "com.github.felHR85:UsbSerial:$rootProject.usbSerialLibraryVersion"

Note: The ‘usbSerialLibraryVersion’ may change from time to time. Please keep your project with the latest library version. Updates can be found here.

b) Add jitpack to your project.build.gradle file.

allprojects {
   repositories {
       jcenter()
       maven { url "https://jitpack.io" }
   }
}

2. Working with Arduino:

We need to program the Arduino to send and receive data. We achieve that with the help of Arduino IDE as mentioned above. Verify, compile and upload a sketch to the Arduino for sending and receiving data. If you are a complete beginner in Arduino programming, there are example sketches for this same purpose. Load an example sketch from under the communication segment and choose the serial communication sketch. Here, we will be working with a simple sketch for the Arduino such that it simply echoes whatever it receives on the serial port. Here is sketch code:

// the setup routine runs once when you press reset:
void setup() {
  // initialize serial communication at 9600 bits per second:
  Serial.begin(9600);
}

// the loop routine runs over and over again forever:
void loop() {
  char incomingByte;
   // If there is a data stored in the serial receive buffer, read it and print it to the serial port as human-readable ASCII text.
  if(Serial.available()){  
    incomingByte = Serial.read();
    Serial.print(incomingByte);  
  }
}

Feel free to compile and upload it to your own Arduino.

2. Working with Android:

  1. Firstly, we need an USBManager instance initialized with the system service – ‘USB_SERVICE’. This needs to be done in an Activity (preferably main) so that this instance can be passed to the Communication Handler class, which we are going to create next.
  2. Now, we will be working with a class for handling the serial communications with our Android device and the Arduino board. We would pass the USBManager instance to this class wherein work will be done with that to find the USBDevice and USBDeviceConnection

Starting with, we need to search for any attached Arduino devices to the Android device. We create a method for this and use the ‘getDevicesList’ callback to achieve this in the following way:

public void searchForArduinoDevice(Context context) {
        HashMap usbDevices = usbManager.getDeviceList();

        if (!usbDevices.isEmpty()) {
            boolean keep = true;
            for (Object object : usbDevices.entrySet()) {
                Map.Entry<String, UsbDevice> entry = (Map.Entry<String, UsbDevice>) object;
                device = entry.getValue();

                int deviceVID = device.getVendorId();
                if (deviceVID == ARDUINO_DEVICE_ID) { //Arduino Vendor ID = 0x2341
                    PendingIntent pi = PendingIntent.getBroadcast(context, 0,
                            new Intent(ACTION_USB_PERMISSION), 0);
                    usbManager.requestPermission(device, pi);
                    keep = false;
                } else {
                    connection = null;
                    device = null;
                }
                if (!keep)
                    break;
            }
        }
    }

c. Now, in the Activity where we will be testing or intend to work with the serial connection, we check for the usb permission in a broadcast receiver which is registered in the onCreate method of the activity along with an Intent Filter. The Intent filter has the usp permission as an action added to it.

usbCommunicationHandler = USBCommunicationHandler.getInstance(this, NeuroLab.getUsbManager());

        deviceConnector = new DeviceConnector(NeuroLab.getUsbManager());

        IntentFilter intentFilter = new IntentFilter();
 
        intentFilter.addAction(ACTION_USB_PERMISSION);
        registerReceiver(broadcastReceiver, intentFilter);

d. In the onReceive callback of the broadcast receiver, if the usb permission is granted, we initialize the serial connection with a baud rate for our Arduino device. In this initialization method, we get the connection and serial port of the connected Arduino to the Android device with which we can work. The method is implemented in the following way:

public boolean initializeSerialConnection(int baudRate) {
        connection = usbManager.openDevice(device);
        serialPort = UsbSerialDevice.createUsbSerialDevice(device, connection);
        if (serialPort != null) {
            if (serialPort.open()) { 
                serialPort.setBaudRate(baudRate);
                serialPort.setDataBits(UsbSerialInterface.DATA_BITS_8);
                serialPort.setStopBits(UsbSerialInterface.STOP_BITS_1);
                serialPort.setParity(UsbSerialInterface.PARITY_NONE);
                serialPort.setFlowControl(UsbSerialInterface.FLOW_CONTROL_OFF);
            } else {
                Log.d("SERIAL", "PORT NOT OPEN");
                return false;
            }
        } else {
            Log.d("SERIAL", "PORT IS NULL");
            return false;
        }
        setSerialPort(serialPort);
        return true;
    }

e. Now, with this ‘serialPort’ we can read data from the Arduino in the UsbReadCallback callback from the USBSerialInterface. The data is read in the form of array of bytes. This read data can be used to carry out the various functionalities we want to achieve.

With the above steps we can establish a serial connection between Android and Arduino, transmit data from Arduino to Android device for processing.

The whole working source code of the Neurolab Android project can be found here:

https://github.com/fossasia/neurolab-android/

Thanks for taking the time to read this blog. Hope it was able to make some good contributions to your knowledge base.

References

  1. https://github.com/felHR85/UsbSerial
  2. https://www.arduino.cc/reference/en/language/functions/communication/serial/

Tags: FOSSASIA, Neurolab, GSOC19, Open-source, Arduino, Serial terminal

Continue ReadingNeurolab data transfer – Establishing serial communication between Arduino and Android

How to transfer data in files from different locations to app directory

In Android apps, we might have seen various instances in-app where we can import, export or save files. Now, where does the files go to or come from? There needs to be a specific folder or directory (maybe root) for a particular app which inturn contains the necessary files, the user has worked with from the app. In this blog, we will be learning about the backend of how to create directories and store files in them dynamically with proper content.

Context

I have been working with FOSSASIA on the project Neurolab-Android. In the app, we have various program modes, which has the option to save/record data. The saved data gets logged in a new file in the Neurolab directory/folder. Feel free to go ahead and explore the feature.

The Save/Record feature in Neurolab can be found in the app bar or as an option in the drop down menu present in the app bar itself in any program mode. The feature becomes functional, once the data is imported with the import data feature which is also present in the app bar.

                                              Figure: Demonstration of features in Neurolab

Tutorial

Now, starting off, there are apps out there wherein users can save files from different segments in the app and those saved files can be used by the app itself at other times as they belong to that app itself specifically.

First off, we need to make sure we have a directory for our app, wherein the files will get stored. If the directory or folder is not present, it needs to be created.

File directory = new File(
                Environment.getExternalStorageDirectory().getAbsolutePath() +
                        File.separator + DIRECTORY_NAME);
        if (!directory.exists()) {
            try {
                directory.mkdir();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

Now, once the directory is present, we can go ahead to keep the saved files in it. Also we will be implementing category-wise storage of the files. Simply put, we will be storing csv files in the CSV folder, xls files in the Excel folder, etc. In the below code example, we see the use case of CSV ‘category’.

private void categoryWise() {File categoryDirectory = new File(
                Environment.getExternalStorageDirectory().getAbsolutePath() +
                        File.separator + DIRECTORY_NAME + File.separator + category);
        if (!categoryDirectory.exists()) {
            try {
                categoryDirectory.mkdir();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }}

For saving a file in our app, we need to get (import) the file into our app. Once the file is imported, we can get the path of the file with the help of its URI. Let’s assign the path to a variable named ‘importedFilePath’. Then we place the imported file in the required category directory within our parent app directory depending and deciding upon the extension of the imported file.

File importedFile = new File(importedFilePath);
        FilePathUtil.setupPath();
        Date currentTime = Calendar.getInstance().getTime();
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String fileName = sdf.format(currentTime);
        File dst = new File(Environment.getExternalStorageDirectory().getAbsolutePath() +
                File.separator + DIRECTORY_NAME + File.separator + categoryDirectory + File.separator + fileName + ".$extension");
        if (!dst.exists()) {
            try {                categoryWise()
                transfer(importedFile, dst);
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

Now, we have the ‘importedFile’ and the destination file path (dst) where the file needs to be stored for our app. The ‘extension’ can be of any type you feel the file should be. Here, for the fileName we are using the current date and time together.

Then, we can come to the function ‘transfer’ which has been called above.

private static void transfer(File src, File dst) throws IOException {
        InputStream in = new FileInputStream(src);
        try {
            OutputStream out = new FileOutputStream(dst);
            try {
                // Transfer bytes from in to out
                byte[] buf = new byte[1024];
                int len;
                while ((len = in.read(buf)) > 0) {
                    out.write(buf, 0, len);
                }
            } finally {
                out.close();
            }
        } finally {
            in.close();
        }
    }

In the ‘transfer’ function, we initialize an input stream with the source file path and the output stream with the destination file path. We read the content in the form of  a certain chunk of bytes from the source file and write to the output stream (destination file).

Finally, we close the output and input streams simultaneously.

Thus, we have our code ready to be bound by UI actions/buttons. Once, the user interacts with the action in your app, the imported file will get saved in the specific directory of your app.

That’s it. Hope this blog enhanced your Android development and Java skillset. 

Resources:

  1. Author – Google Android Developers, Article – Data and File storage, Website – https://developer.android.com/guide/topics/data/data-storage
  2. Author – Rakshi and Thomas, Article – How to make a copy of file in android, Source – Stack overflow, Website – https://stackoverflow.com/questions/9292954/how-to-make-a-copy-of-a-file-in-android

Tags: FOSSASIA. Neurolab, GSOC19, Open-source, File-storage

Continue ReadingHow to transfer data in files from different locations to app directory

Dialog Component in SUSI.AI

Dialog Component in SUSI.AI is rendered in App.js to remove code redundancy. Redux is integrated in the Dialog component which allows us to open/close the dialog from any component by altering the modal states. This implementation allows us to get rid of the need of having dialog component in different components.

Redux Code

There are two actions and reducers which control the dialog component. Default state of isModalOpen is false and modalType is an empty string. To open a dialog modal the action openModal is dispatched, which sets isModalOpen to true and the modalType. To close a dialog modal the action closeModal is dispatched, which sets isModalOpen to default state i.e. false.

import { handleActions } from 'redux-actions';
import actionTypes from '../actionTypes';

const defaultState = {
 modalProps: {
   isModalOpen: false,
   modalType: '',
 },
};

export default handleActions(
 {
   [actionTypes.UI_OPEN_MODAL](state, { payload }) {
     return {
       ...state,
       modalProps: {
         isModalOpen: true,
         ...payload,
       },
     };
   },
   [actionTypes.UI_CLOSE_MODAL](state) {
     return {
       ...state,
       modalProps: defaultState.modalProps,
     };
   },
 }
 defaultState,
);

Shared Dialog Component

Dialog Modal can be opened from any component by dispatching an action. 

To open a Dialog Modal: this.props.actions.openModal({modalType: [modal name]});

To close a Dialog Modal: this.props.actions.closeModal();

Shared Dialog Component has a DialogData object which contains objects with two main properties : Dialog component and Dialog size. Other props can also be passed along with these two properties such as fullScreen. Dialog Content of different Dialogs are present in their respective folders. Each Dialog Content has a Title, Content and Actions.Different Dialog types present are:

  1. Confirm Delete with Input: This dialog modal is used when a user deletes account, device and skill. 
  2. Confirm Dialog: This dialog modal is used where confirmation is required from the user/admin such as on changing skill status, on password reset,etc.
  3. Share Dialog: This dialog modal opens up when the share icon is clicked in the chat.
  4. Standard Action Dialog: This dialog modal opens up on restore skill, delete feedback, system settings and bot.
  5. Tour Dialog: This dialog modal opens up SUSI.AI tour.

To add a new Dialog to DialogSection, the steps are:

  1. Import the Dialog Content Component
  2. Add the Dialog Component to DialogData object in the following manner:
const DialogData = {
[dialog componet name]: { Component : [imported dialog component name], size : [size of the Dialog Component]},
}

Code (Reduced)

const DialogData = {
  login: { Component: Login, size: 'sm' },
}
const DialogSection = props => {
 const {
   actions,
   modalProps: { isModalOpen, modalType, ...otherProps },
   visited,
 } = props;

 const getDialog = () => {
   if (isModalOpen) {
     return DialogData[modalType];
   }
   return DialogData.noComponent;
 };

 const { size, Component, fullScreen = false } = getDialog();

return (
   <Dialog
      maxWidth={size}
      fullWidth={true}
      open={isModalOpen || !visited}
      onClose={isModalOpen ? actions.closeModal : actions.setVisited}
      fullScreen={fullScreen}
    >
     <DialogContainer>
       {Component ? <Component {...otherProps} /> : null}
     </DialogContainer>
  </Dialog>
)
};

In conclusion, having a shared dialog component reduces redundant code and allows to have a similar Dialog UI across the repo. Also having one component managing all the dialogs removes the possibility of  two dialogs being fired up at once.

Resources

Continue ReadingDialog Component in SUSI.AI

My Devices in SUSI.AI

In this blog I’ll be explaining how to view, edit and delete connected devices from SUSI.AI webclient. To connect a device open up the SUSI.AI android app, and fill the details accordingly. Device can also be connected by logging in to your raspberry pi. Once the devices is connected you can edit, delete and access specific features for the device from the web client.

My Devices

All the connected devices can be viewed in My Devices tab in the Dashboard. In this tab all the devices connected to your account are listed in a table along with their locations on the map. Each device table row has three action buttons – view, edit and delete. Clicking on the view button takes to device specific page. Clicking on the edit button makes the fields name and room editable in table row. Clicking on the delete button opens a confirm with input dialog. Device can be deleted by entering the device name and clicking on delete.

To fetch all the device getUserDevices action is dispatched on component mounting which sets the reducer state devices in settings reducer. initialiseDevices function is called after all the devices are fetched from the server. This function creates an array of objects of devices with name, room, macId, latitude, longitude and location.

 componentDidMount() {
   const { accessToken, actions } = this.props;
   if (accessToken) {
     actions
       .getUserDevices()
       .then(({ payload }) => {
         this.initialiseDevices();
         this.setState({
           loading: false,
           emptyText: 'You do not have any devices connected yet!',
         });
       })
       .catch(error => {
         this.setState({
           loading: false,
           emptyText: 'Some error occurred while fetching the devices!',
         });
         console.log(error);
       });
   }
   document.title =
     'My Devices - SUSI.AI - Open Source Artificial Intelligence for Personal  Assistants, Robots, Help Desks and Chatbots';
 }
 initialiseDevices = () => {
   const { devices } = this.props;
 
   if (devices) {
     let devicesData = [];
     let deviceIds = Object.keys(devices);
     let invalidLocationDevices = 0;
 
     deviceIds.forEach(eachDevice => {
       const {
         name,
         room,
         geolocation: { latitude, longitude },
       } = devices[eachDevice];
 
       let deviceObj = {
         macId: eachDevice,
         deviceName: name,
         room,
         latitude,
         longitude,
         location: `${latitude}, ${longitude}`,
       };
 
       if (
         deviceObj.latitude === 'Latitude not available.' ||
         deviceObj.longitude === 'Longitude not available.'
       ) {
         deviceObj.location = 'Not found';
         invalidLocationDevices++;
       } else {
         deviceObj.latitude = parseFloat(latitude);
         deviceObj.longitude = parseFloat(longitude);
       }
       devicesData.push(deviceObj);
     });
 
     this.setState({
       devicesData,
       invalidLocationDevices,
     });
   }
 };

Device Page

Clicking on the view icon button in my devices redirects to mydevices/:macId. This page consists of device information in tabular format, local configuration settings and location of the device on the map. User can edit and delete the device from actions present in table. Local configuration settings can be accessed only if the user is logged in the local server.

Edit Device

To edit a device click on the edit icon button in the actions column of the table. The name and room field become editable.On changing the values handleChange function is called which updates the devicesData state. Clicking on the tick icon saves the new details by calling the onDeviceSave function. This function class the addUserDevice api which takes in the new device details.

startEditing = rowIndex => {
   this.setState({ editIdx: rowIndex });
 };
 
 handleChange = (e, fieldName, rowIndex) => {
   const value = e.target.value;
   let data = this.state.devicesData;
   this.setState({
     devicesData: data.map((row, index) =>
       index === rowIndex ? { ...row, [fieldName]: value } : row,
     ),
   });
 };

 handleDeviceSave = rowIndex => {
   this.setState({
     editIdx: -1,
   });
   const deviceData = this.state.devicesData[rowIndex];
 
   addUserDevice({ ...deviceData })
     .then(payload => {})
     .catch(error => {
       console.log(error);
     });
 };

Delete Device

To delete a device click on the delete icon button under the actions column in the table. Clicking on the delete device button opens up the confirm with input dialog modal. Type in the name of the device and click on delete. Clicking on delete calls the handeRemoveDevice function which calls the removeUserDevice api which takes in the macId. On deleting the device user is redirected to the My Devices in Dashboard.

 handleRemoveConfirmation = () => {
   this.props.actions.openModal({
     modalType: 'deleteDevice',
     name: this.state.devicesData[0].deviceName,
     handleConfirm: this.handleRemoveDevice,
     handleClose: this.props.actions.closeModal,
   });
 };
 handleRemoveDevice = () => {
   const macId = this.macId;
   removeUserDevice({ macId })
     .then(payload => {
       this.props.actions.closeModal();
       window.location.replace('/mydevices);
     })
     .catch(error => {
       console.log(error);
     });
 };

In conclusion, My Devices tab in dashboard helps you manage the devices connected with your account along with specific device configuration. Now the users can edit, view and delete their connected devices.

Resources

Continue ReadingMy Devices in SUSI.AI

How to fix undetected Arduino boards in Android

In the development process of the Neurolab Android app, we needed an Arduino-Android connection. This blog explains how to  establish the connection and getting the Arduino board detected in my Android device

Context-connecting the board and getting it detected

Arduino boards are primarily programmed from the Desktop using the Arduino IDE, but they are not limited to the former. Android devices can be used to program the circuit boards using an application named Arduinodroid.

Arduino is basically a platform for building various types of electronic projects and the best part about it is that, it is open-sourced. Arduino, the company has got two products The physical programmable circuit board (often referred to as a microcontroller). 

Examples of Arduino circuit boards – UNO, UNO CH340G, Mega, etc. Find more here.

Connecting the board and getting it detected

Arduino boards are primarily programmed from the Desktop using the Arduino IDE, but they are not limited to the former. Android devices can be used to program the circuit boards using an application named Arduinodroid.

In this blog, we are going to use Arduinodroid app for establishing a connection between the Arduino board and the Android device, getting the board detected in the Android phone and uploading a sketch to it.

Materials/Gadgets required:-

  1. Arduino board (UNO preferably)
  2. Arduino-USB Cable
  3. OTG Cable
  4. Android device

Now, one of the most frequent issues, while establishing a connection and getting the Arduino board detected with the Android device, is the error message of: “No Arduino boards detected” in the Arduinodroid app. There can be a few core reasons for this –

  1. Your Android mobile device isn’t USB-OTG supported – Probably because it is an old model or it might be a company/brand-specific issue.
  2. Disabled OTG Mode – Be sure to enable USB-OTG mode (only if your device has one) from the Developer options in your Android device settings.

Even after trying and making sure of these above points, if you still continue to get an error while uploading a sketch from the Arduinodroid app like this:

                                                            Figure 1: The Error Message

Follow the steps below carefully and simultaneously one after the other:

  1. Look for any external module attached to your Arduino board using jumper wires. If so, remove those connections completely and press the reset button on the Arduino circuit board. The attached modules can be one of the following: Micro SD Card module, Bluetooth module, etc.
  2. Remove pin connections, if any from the TX and RX pin-slots in the Arduino board. These pre-attached pins can cause unnecessary signal transfers which can hinder and make the actual port of Arduino board busy.
  3. Before connecting the Arduino to the Android device, go to the drop down menu in the app at the top-right corner -> Settings -> Board Type -> Arduino -> UNO
  4. Now, you need to code a sketch and make it ready for compile and upload to the circuit board. We will use a basic example sketch for this case. Feel free to try out your own custom coded Arduino sketches. Go to the drop-down menu -> Sketch -> Examples -> Basics -> AnalogReadSignal
  5. Don’t compile the sketch yet because we haven’t connected any Arduino circuit board to our Android device. So first, connect the Arduino circuit board to the Android device through the OTG cable connected to the Arduino-USB cable.
  6. You should see some LEDs lit up on the circuit board (indicates power is flowing to the board). Go ahead to compile the sketch. Click the ‘lightning’ icon on the top in the toolbar of the app. You should see the code/sketch getting compiled. Once done you should see a toast message saying “Compilation finished”. This signifies that your code/sketch has been verified by the compiler.

                                              Figure 2: Successful Compilation of sketch

This process is inevitable and there is hardly any issue while compiling a sketch.

       7. Upload the sketch: Click on the upload icon from the toolbar in the app. Upload             should start once you get a pop-up dialog like this:

                                           Figure 3: Arduino board detected successfully

Once you click Okay, the upload shall start and if your code is correct and matches the particular Arduino circuit board, you shall get a successful upload, which was not the case earlier for the error : “no Arduino boards found” on clicking the upload button.

So, that’s it then. Hope this blog adds value to your development skills and you can continue working bug free with your Android-Arduino connections.

Resources:

  1. Author – Nick Gamon, Article – Have I bricked my Arduino uno problems with uploading to board, Date – Nov’16 2016, Website – https://arduino.stackexchange.com/questions/13292/have-i-bricked-my-arduino-uno-problems-with-uploading-to-board
  2. Author – Arduino Products, Article – Arduino boards, Website – https://www.arduino.cc/en/Main/Boards

3. Author – Anton Smirnov, App name – ArduinoDroid, Website – https://play.google.com/store/apps/details?id=name.antonsmirnov.android.arduinodroid2&hl=en_IN

Tags: FOSSASIA, Neurolab, GSOC19, Open-source, Arduino, Serial terminal

Continue ReadingHow to fix undetected Arduino boards in Android