iframe – blog.fossasia.org

Adding Support for Playing Youtube Videos in SUSI iOS App

Post author:Jogendra Kumar
Post published:August 21, 2018
Post category:API FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

SUSI supports very exciting features in chat screen, from simple answer type to complex map, RSS, table etc type responses. Even user can ask SUSI for the image of anything and SUSI response with the image in the chat screen. What if we can play the youtube video from SUSI, we ask SUSI for playing videos and it can play youtube videos, isn’t it be exciting? Yes, SUSI can play youtube videos too. All the SUSI clients (iOS, Android, and Web) support playing youtube videos in chat.

Google provides a Youtube iFrame Player API that can be used to play videos inside the app only instead of passing an intent and playing the videos in the youtube app. iFrame API provide support for playing youtube videos in iOS applications.

In this post, we will see how playing youtube video features implemented in SUSI iOS.

Getting response from server side –

When we ask SUSI for playing any video, in response, we get youtube Video ID in video_play action type. SUSI iOS make use of Video ID to play youtube video. In response below, you can see that we are getting answer action type and in the expression of answer action type, we get the title of the video.

actions:
[
{
type: "answer",
expression: "Playing Kygo - Firestone (Official Video) ft. Conrad Sewell"
},
{
identifier: "9Sc-ir2UwGU",
identifier_type: "youtube",
type: "video_play"
}
]

Integrating youtube player in the app –

We have a VideoPlayerView that handle all the iFrame API methods and player events with help of YTPlayer HTML file.

When SUSI respond with video_play action, the first step is to register the YouTubePlayerCell and present the cell in collectionView of chat screen.

Registering the Cell –

register(_:forCellWithReuseIdentifier:) method registers a class for use in creating new collection view cells.

collectionView?.register(YouTubePlayerCell.self, forCellWithReuseIdentifier: ControllerConstants.youtubePlayerCell)

Presenting the YouTubePlayerCell –

Here we are presenting the cell in chat screen using cellForItemAt method of UICollectionView.

if message.actionType == ActionType.video_play.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.youtubePlayerCell, for: indexPath) as? YouTubePlayerCell {
cell.message = message
cell.delegate = self
return cell
}
}

Setting size for cell –

Using sizeForItemAt method of UICollectionView to set the size.

if message.actionType == ActionType.video_play.rawValue {
return CGSize(width: view.frame.width, height: 158)
}

In YouTubePlayerCell, we are displaying the thumbnail of youtube video using UIImageView. Following method is using to get the thumbnail of particular video by using Video ID –

Getting thumbnail image from URL
Setting image to imageView

func downloadThumbnail() {
if let videoID = message?.videoData?.identifier {
let thumbnailURLString = "https://img.youtube.com/vi/\(videoID)/default.jpg"
let thumbnailURL = URL(string: thumbnailURLString)
thumbnailView.kf.setImage(with: thumbnailURL, placeholder: ControllerConstants.Images.placeholder, options: nil, progressBlock: nil, completionHandler: nil)
}
}

We are adding a play button in the center of thumbnail view so that when the user clicks play button, we can present player.

On clicking the Play button, we are presenting the PlayerViewController, which hold all the player setups, by overFullScreen type of modalPresentationStyle.

@objc func playVideo() {
if let videoID = message?.videoData?.identifier {
let playerVC = PlayerViewController(videoID: videoID)
playerVC.modalPresentationStyle = .overFullScreen
delegate?.loadNewScreen(controller: playerVC)
}
}

The methods above present the youtube player with giving Video ID. We are using YouTubePlayerDelegate method to autoplay the video.

func playerReady(_ videoPlayer: YouTubePlayerView) {
videoPlayer.play()
}

The player can be dismissed by tapping on the light black background.

Final Output –

Resources –

Adding Support for Playing Audio in SUSI iOS App

Post author:Jogendra Kumar
Post published:August 21, 2018
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

SUSI.AI supports various actions like the answer, map, table, video play and many more. You can play youtube videos in the chat screen. It also supports for playing audio in the chat screen. In this post, we will see that how playing audio feature implemented in SUSI iOS.

Getting audio action from server side –

In the chat screen, when we ask SUSI to play audio, we get the audio source from the server side. For example, if we ask SUSI “open the pod bay door”, we get the following action object:

"actions": [
{
"type": "audio_play",
"identifier_type": "youtube",
"identifier": "7qnd-hdmgfk"
},
{
"language": "en",
"type": "answer",
"expression": "I'm sorry, Dave. I'm afraid I can't do that."
}
]

In the above action object, we can see that we get two actions, audio_play and answer. In audio_play action, we are getting an identifier type which tells us about the source of audio. Identifier type can be youtube or local or any other source. When the identifier is youtube, we play audio from youtube stream. In identifier, we get the audio file path. In case of youtube identifier type, we get youtube video ID and play from youtube stream. In answer action type, we get the expression which we display in chat screen after thumbnail.

Implementing Audio Support in App –

We use Google’s youtube Iframe API to stream audio from youtube videos. We have a VideoPlayerView that handle all the iFrame API methods and player events with help of YTPlayer HTML file.

Presenting the YouTubePlayerCell –

If the action type is audio_play, we are presenting the cell in chat screen using cellForItemAt method of UICollectionView.

if message.actionType == ActionType.audio_play.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.youtubePlayerCell, for: indexPath) as? YouTubePlayerCell {
cell.message = message
cell.delegate = self
return cell
}
}

Setting size for cell –

Using sizeForItemAt method of UICollectionView to set the size.

if message.actionType == ActionType.audio_play.rawValue {
return CGSize(width: view.frame.width, height: 158)
}

In YouTubePlayerCell, we fetch thumbnail and display in the cell with a play button. On clicking the play button, we open the player and stream music.

Final Output –

Resources –

How does live preview work on SUSI AI chatbot wizard

Post author:DNS-404
Post published:July 15, 2018
Post category:FOSSASIA
Post comments:0 Comments

Live preview is an important feature for the chatbot building process in SUSI.AI botbuilder. It tells the bot creator how the bot looks and behaves at any time.

We use iframe for creating a live preview.

What is iframe?

Frames allow browser window to be split into separate segments. All these segments can be used for displaying a different webpage within the browser window.

is an HTML tag. iframe stands for “inline frame”. It places another HTML document in a frame. The content of the element is used as an alternative text to be displayed if the browser does not support inline frames.

This was first introduced by Microsoft Internet Explorer in 1997, standardized in HTML 4.0 transitional and allowed in HTML5.

Syntax:

<iframe src=”URL”></iframe>

Some attributes:

Height
Width
Name
Id

Code used for preview of chatbot screen:

<iframe title="botPreview" name="frame-1" id="frame-1" src={locationBot} height="600px" style={styles.iframe}></iframe>

Code used for preview of chatbot avatar:

<iframe title="botAvatarPreview" name="frame-2" id="frame-2" src={locationAvatar} height="100px" style={styles.iframe}></iframe>

Preview bot fetches the current theme status and skill status from SUSI server and then applies it to the chatbot. Now, this details will be unique to the user logged in. Hence, we need to somehow identify the person logged in. This is done using access token stored in the cookies which is accessed through cookies.get(‘loggedIn’).

How to pass access token to iframe?

This is the major issue faced when creating preview using iframe because even though the HTML window in iframe is rendered on the same parent page, we can not access the elements of HTML page inside iframe. This is because iframe loads dynamically. There are few solutions for this problem.

One of them is to create a script in iframe that would run when iframe is loaded. Now this script can access the elements of parent window (the one containing iframe) using window.parent.document.getElementById(‘#target’)

The best solution is however to pass the access token in the URL of the source of iframe. Then this can be accessed by HTML page in frame using location.href .

Code sample to pass access token in url:

src = '/BotPreview.html?access='+cookies.get('loggedIn')

Now, we have to split this URL and get the access token. This can be easily done using the following script.

var url = location.href.split(‘?’)[1].split(‘=’);
var access_token = url[1];

Here in this example, the url variable firstly splits the URL at ‘?’. This creates an array with two elements.

[‘url.com/BotPreview.html’, ‘access=abcdef123’]

Then we access the element on index 1, i.e. ‘access=abcdef123’ and split it at the ‘=’. This further creates an array of two elements.

[‘access’, ‘abcdef123’]

Now, we can access the access token which is at index 1 in this array.

References

Wikipedia article on Frames: https://en.wikipedia.org/wiki/HTML_element#Frames