rss

Read more about the article Fetching Images for RSS Responses in SUSI Web Chat

Fetching Images for RSS Responses in SUSI Web Chat

Initially, SUSI Web Chat rendered RSS action type responses like this: The response from the server initially only contained Title Description Link We needed to improvise the web search & RSS results display and also add images for the results. The web search & RSS results are now rendered as : How was this implemented? SUSI AI uses Yacy to fetchRSSs feeds. Firstly the server using the console process to return the RSS feeds from Yacy needs to be configured to return images too. "yacy":{ "example":"http://127.0.0.1:4000/susi/console.json?q=%22SELECT%20title,%20link%20FROM%20yacy%20WHERE%20query=%27java%27;%22", "url":"http://yacy.searchlab.eu/solr/select?wt=yjson&q=", "test":"java", "parser":"json", "path":"$.channels[0].items", "license":"" } In a console process, we provide the URL needed to fetch data from, the query parameter needed to be passed to the URL and the path to look for the answer in the API response. url = <url>   - the URL to the remote JSON service which will be used to retrieve information. It must contain a $query$ string. test = <parameter> - the parameter that will replace the $query$ string inside the given URL. It is required to test the service. Here the URL used is : http://yacy.searchlab.eu/solr/select?wt=yjson&q=QUERY To include images in RSS action responses, we need to parse the images also from the Yacy response. For this, we need to add `image` in the selection rule while calling the console process "process":[ { "type":"console", "expression":"SELECT title,description,link FROM yacy WHERE query='$1$';" } ] Now the response from the server for RSS action type will also include `image` along with title, description, and link. An example response for the query `Google` : { "title": "Terms of Service | Google Analytics \u2013 Google", "description": "Read Google Analytics terms of service.", "link": "http://www.google.com/analytics/terms/", "image": "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_116x41dp.png", } However, the results at times, do not contain images because there are none stored in the index. This may happen if the result comes from p2p transmission within Yacy where no images are transmitted. So in cases where images are not returned by the server, we use the link preview service to preview the link and fetch the image. The endpoint for previewing the link is : BASE_URL+'/susi/linkPreview.json?url=URL' On the client side, we first search the response for data objects with images in API actions. And the amongst the remaining data objects in answers[0].data, we preview the link to fetch image keeping a check on the count. This needs to be performed for processing the history cognitions too.To preview the remaining links in a loop, we cannot make ajax calls directly in a loop. To handle this, nested ajax calls are made using the function previewURLForImage() where we loop through the remaining links and on the success we decrement the count and call previewURLForImage() on the next link and on error we try previewURLForImage() on the next link without decrementing the count. success: function (rssResponse) { if(rssResponse.accepted){ respData.image = rssResponse.image; respData.descriptionShort = rssResponse.descriptionShort; receivedMessage.rssResults.push(respData); } if(receivedMessage.rssResults.length === count || j === remainingDataIndices.length - 1){ let message = ChatMessageUtils.getSUSIMessageData(receivedMessage, currentThreadID); ChatAppDispatcher.dispatch({ type: ActionTypes.CREATE_SUSI_MESSAGE, message }); } else{ j+=1; previewURLForImage(receivedMessage,currentThreadID, BASE_URL,data,count,remainingDataIndices,j); } }, And we store…

Continue ReadingFetching Images for RSS Responses in SUSI Web Chat
Read more about the article Displaying Blog Posts on SUSI AI Web Chat’s Blog Page and Share Posts

Displaying Blog Posts on SUSI AI Web Chat’s Blog Page and Share Posts

FOSSASIA is maintaining a superior blog and it contains blog posts about projects and programs run by FOSSASIA. While we were implementing SUSI Web Chat Application we got a requirement to implement a blog page. Speciality of this blog page is it is not a separate blog page, it fetches blog posts and other related data by filtering the FOSSASIA’s main blog. In this blog post I’ll discuss how we fetched and managed those data on front-end and how we made the appearance same as the FOSSASIA main blog. First we get blog posts as a JSON. For that we used rss2json API. we can get the RSS feed as a JSON by sending our RSS feed URL to the rss2json API. Check the rss2json API documentation here. It produces all posts as items array. Next we store this array of responses in our application as a state. This response contains blog post titles featured images’ details and post content and other metadata such as tags, author name and published date. We had few requirements to fulfill. First one is to show full content of the blogpost in a new blog page. We can take the full content from response like this, this.state.posts.slice(this.state.startPage, this.state.startPage + 10).map((posts, i) => { let content = posts.content; }) We can use “cintent” variable to show content but it contains the featured image. We have to skip that image. For that, let htmlContent = content.replace(/<img.*?>/, ''); Now we have to render this string value as HTML. For that we have to install “test-to-html” package using below command. npm install html-to-text --save Now we can convert text into html like this htmlContent = renderHTML(htmlContent); We used this HTML content inside the “CardText” tag. <CardText> {htmlContent} </CardText> At the bottom of the post we needed to show author name, tags list and categories list. Since tags and categories come in one array, we have to separate them. First we defined an array which contains all the categories in Fossasia blog. Then we compared that array with the categories we got like this. const allCategories = ['FOSSASIA','GSoC','SUSI.AI'] Compare two arrays, posts.categories.map((cat) => { let k = 0; for (k = 0; k < allCategories.length; k++) { if (cat === allCategories[k]) { category.push(cat); } } }); we defined this “arrDiff” simple function to get the difference of two arrays. var tags=arrDiff(category,posts.categories) Make the list of categories let fCategory=category.map((cat) => <span key={cat} ><a className="tagname" href={'https://blog.fossasia.org/category/' + cat.replace(/\s+/g, '-').toLowerCase()} rel="noopener noreferrer">{cat}</a></span> ); We can use above step to make tags list. Then after used it in the “CardActions” <span className='categoryContainer'> <i className="fa fa-folder-open-o tagIcon"></i> {fCategory} </span>   According to the final requirement we needed to add social media share buttons for Facebook and Twitter. If we need to make a twitter share button we have to follow up this method. But we can use “react-share” npm package to make these kind of share buttons. This is how we made Facebook and Twitter share buttons. First of all we have to install…

Continue ReadingDisplaying Blog Posts on SUSI AI Web Chat’s Blog Page and Share Posts
Read more about the article How RSS Action Type is Implemented in SUSI Android

How RSS Action Type is Implemented in SUSI Android

Important skills of SUSI.AI are to display web search queries, a map of any location and provide a list of relevant information of a topic. RSS action type is similar to websearch action type but when the web search is to be performed on the client side, it is denoted by websearch action type and when the web search is performed by the server itself, it is denoted by rss action type. In this blog, I will show you how rss action type is implemented in SUSI Android. In case of RSS action type server searches the internet and using RSS feeds, returns an array of objects containing : Title Description Link {   "title": "dog-doh: Definitions Index",   "description": "dog-doh: Definitions Index. dog dog and pony show dog biscuit dog collar dog days ...",   "link": "http://websters.yourdictionary.com/index/dog-doh/", } title: Title related to user query description: Description of user query. link: If user want to know more information then user can use link to find more information. How rss action type is parsed in SUSI Android SUSI  reply in json format. It should be parsed properly to show it in android app. We used retrofit library developed by square to parse json data. Retrofit library parse data according to model class. We code model class according to expected json reply. For example, each susi response contains answer jsonarray. There are two jsonarray data and action inside answer jsonarray. We made a different model class for each jsonarray. First model class is SusiResponse. We used this model class to parse ‘answers’ jsonarray. @SerializedName("answers") private List<Answer> answers = new ArrayList<>(); Here we used List<>  because ‘answers’ jsonarray contain a list of jsonarray and jsonobject. Answer is second model class. We used it to parse two important jsonarray ‘data’ and ‘action’. The ‘action’ attribute has information about action type. public class Answer {    @SerializedName("data")    private RealmList<Datum> data = new RealmList<>(); } Here also we used the list because data jsonarray also contains a list of jsonobject but instead of simple list we used RealmList<> because after parsing we save data using realm. ‘data’ jsonarray contain multiple jsonobject and each jsonobject contain three important information ‘title’, ‘description’ and ‘link’. Datum class is the main model class which is used to save and retrieve ‘title’, ‘description’ and ‘link’. setTitle, setLink and setDescription method of Datum class are used to save ‘title’, ‘description’ and ‘link’ and getTitle, getDescription and getLink method are use to retrieve ‘title’, ‘description’ and ‘link’. How rss action type data is retrieved and saved As already mentioned we used retrofit to retrieve data and realm to save data. susiResponse is response we received from SUSI server. We used susiResponse to retrieve a list of Datum class type data. List<Datum> datumList = susiResponse.getAnswers().get(0).getData(); We then loop through datumList and from each element we extract ‘title’, ‘description’ and ‘link’ using getTitle(), getDescription() and getLink() method respectively. Datum class is model class for both retrofit and realm. realmDatum is instance of Datum class and datumRealmList is an instance of…

Continue ReadingHow RSS Action Type is Implemented in SUSI Android
Read more about the article Custom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

Custom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

The SUSI Server is an AI powered server which is capable of responding to intelligent answers based on user’s queries. The queries to the susi server are obtained either as a websearch using the application or as an RSS feed. Two of the actions are websearch and RSS. These actions as the name suggests respond to queries based on search results from the web which are rendered in the clients. In order to use use these action types and display them in the SUSI iOS client, we need to first parse the actions looking for these action types and then creating a custom UI for them to display them. To start with, we need to make send the query to the server to receive an intelligent response from the server. This response is parsed into different action types supported by the server and saved into relevant objects. Here, we check the action types by looping through the answers array containing the actions and based on that, we save the data for that action. if type == ActionType.rss.rawValue {    message.actionType = ActionType.rss.rawValue    message.rssData = RSSAction(data: data, actionObject: action) } else if type == ActionType.websearch.rawValue {    message.actionType = ActionType.websearch.rawValue    message.message = action[Client.ChatKeys.Query] as? String ?? "" } Here, we parsed the data response from the server and looked for the rss and websearch action type followed by which we saved the data we received from the server for each of the action types in their own objects. Next, when a message object is created, we insert it into the dataSource item by appending it and use the `insertItems(at: [IndexPath])` method of collection view to insert them into the views at a particular index. Before adding them, we need to create a Custom UI for them. This UI will consist of a Collection View which is scrollable in the horizontal direction inside a CollectionView Cell. To start with this, we create a new class called `WebsearchCollectionView` which will be a `UIView` consisting of a `UICollectionView`. We start by adding a collection view into the UIView inside the `init` method by overriding it. Declare a collection view using flow layout and scroll direction set to `horizontal`. Also, hide the scroll indicators and assign the delegate and datasource to `self`. Now to populate this collection view, we need to specify the number of items that will show up. For this, we make use of the `message` variable declared. We use the `websearchData` in case of websearch action and `rssData` otherwise. Now to specify the number of cells, we use the below method which returns the number of rss or websearch action objects and defaults to 0 such cells. func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {     if let rssData = message?.rssData {         return rssData.count     } else if let webData = message?.websearchData {         return webData.count     }     return 0 } We display the title, description and image for each object for which we need to create a UI for the cells. Let’s start by creating…

Continue ReadingCustom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching
Read more about the article Showing RSS and Table Type Replies in SUSI Messenger Bots

Showing RSS and Table Type Replies in SUSI Messenger Bots

All the messengers have a “plain text” reply support. To show web search (RSS) or table type replies, either: We need a “list type” (as in Facebook messenger) or “table type” reply support built in the messenger itself.                                                               or We need to convert the RSS or table type reply by SUSI API to plain text, so that we can send it, due to the “plain text” reply support available in almost every messenger. The second point is the most favorable approach, as that way, replying with RSS or table type results is dependent only on the text support feature in the messenger. This way RSS or table type replies can be shown in messengers like REST API Gitter (which do not provide any other reply type support except text). In SUSI web app, the UI of the web search and table type results: As the SUSI web app is made in React js, it provided the app with necessary features to show the results this way. The messengers may not be having these required features. So the task is we need to convert the RSS or table type replies by SUSI API to plain text to send it to the messenger. Let’s work it out. Converting RSS results to text: First get familiar with the SUSI API reply to query “why” by visiting this url - http://api.susi.ai/susi/chat.json?q=why. The JSON object returned will be constituting an array of objects as the value of the data key like: "data": [ { "title": "Why is Oracle male?", "description": "Why is Oracle male?. http dba oracle com why is male htm. Oracle Why is masculine?. ", "link": "http://dba-oracle.com/t_why_is_oracle_male.htm" } ] If we notice carefully each object has 3 main keys namely “title”, “description” and “link”. So extracting these 3 properties from each object and binding them together into 1 string is the task we need to do. So we traverse each object (i.e. rss result) in the data array and we keep on appending the values of “title”, “description” and “link” key values to the ans variable. At the end we send this resultant string to the messenger bot as a reply.   Suppose we have the returned JSON object, in the data variable. // storing the number of rss results var metaCnt = data.answers[0].metadata.count; for(var i=0;i<metaCnt;i++){ ans += ('Title : '); ans += data.answers[0].data[i].title+', '; ans += ('Link : '); ans += data.answers[0].data[i].link+', '; ans += '\n\n'; } // send the message in ans variable to the messenger Converting table type replies to text: First get familiar with the SUSI API reply to query “why” by visiting this url - http://api.susi.ai/susi/chat.json?q=universities+in+australia. The JSON object returned will be constituting an array of objects representing universities as the value of the data key in this form: { "alpha_two_code": "AU", "name": "Australian Correspondence Schools", "country":…

Continue ReadingShowing RSS and Table Type Replies in SUSI Messenger Bots
Read more about the article How SUSI WebChat Implements RSS Action Type

How SUSI WebChat Implements RSS Action Type

SUSI.AI now has a new action type called RSS. As the name suggests, SUSI is now capable of searching the internet to answer user queries. This web search can be performed either on the client side or the server side. When the web search is to be performed on the client side, it is denoted by websearch action type. When the web search is performed by the server itself, it is denoted by rss action type. The server searches the internet and using RSS feeds, returns an array of objects containing : Title Description Link Count Each object is displayed as a result tile and all the results are rendered as swipeable tiles. Lets visit SUSI WebChat and try it out. Query : Google Response: API response SUSI WebChat uses the same code abstraction to render websearch and rss results as both are results of websearch, only difference being where the search is being performed i.e client side or server side. How does the client know that it is a rss action type response? "actions": [ { "type": "answer", "expression": "I found this on the web:" }, { "type": "rss", "title": "title", "description": "description", "link": "link", "count": 3 } ], The actions attribute in the JSON API response has information about the action type and the keys to be parsed for title, link and description. The type attribute tells the action type is rss. The title attribute tells that title for each result is under the key - title for each object in answers[0].data. Similarly keys to be parsed for description and link are description and link respectively. The count attribute tells the client how many results to display. We then loop through the objects in answers,data[0] and from each object we extract title, description and link. let rssKeys = Object.assign({}, data.answers[0].actions[index]); delete rssKeys.type; let count = -1; if(rssKeys.hasOwnProperty('count')){ count = rssKeys.count; delete rssKeys.count; } let rssTiles = getRSSTiles(rssKeys,data.answers[0].data,count); We use the count attribute and the length of answers[0].data to fix the number of results to be displayed. // Fetch RSS data export function getRSSTiles(rssKeys,rssData,count){ let parseKeys = Object.keys(rssKeys); let rssTiles = []; let tilesLimit = rssData.length; if(count > -1){ tilesLimit = Math.min(count,rssData.length); } for(var i=0; i<tilesLimit; i++){ let respData = rssData[i]; let tileData = {}; parseKeys.forEach((rssKey,j)=>{ tileData[rssKey] = respData[rssKeys[rssKey]]; }); rssTiles.push(tileData); } return rssTiles; } We now have our list of objects with the information parsed from the response.We then pass this list to our renderTiles function where each object in the rssTiles array returned from getRSSTiles function is converted into a Paper tile with the title and description and the entire tile is hyperlinked to the given link using Material UI Paper Component and few CSS attributes. // Draw Tiles for Websearch RSS data export function drawTiles(tilesData){ let resultTiles = tilesData.map((tile,i) => { return( <div key={i}> <MuiThemeProvider> <Paper zDepth={0} className='tile'> <a rel='noopener noreferrer' href={tile.link} target='_blank' className='tile-anchor'> {tile.icon && (<div className='tile-img-container'> <img src={tile.icon} className='tile-img' alt=''/> </div> )} <div className='tile-text'> <p className='tile-title'> <strong> {processText(tile.title,'websearch-rss')} </strong> </p> {processText(tile.description,'websearch-rss')} </div> </a> </Paper>…

Continue ReadingHow SUSI WebChat Implements RSS Action Type