Youtube – blog.fossasia.org

Adding Audio Streaming from Youtube in SUSI Linux

Post author:stealthanthrax
Post published:August 24, 2018
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

In this blog post we will describe how the youtube streaming works in the

SUSI smart speaker and how audio is streamed directly from youtube videos.

To achieve this process, we have used an amazing Open-Source project called MPV music Player along with python libraries like Subprocess.

1.Processing a Query to the server

Firstly , the user asks the smart speaker to play the youtube audio by simply adding a ‘play’ word before his/her favorite song. eg. I’ll say ‘play despacito’ and then the command is recognized and a query is sent to the server which sends the following response as a JSON object.

“actions”: [
     {
       “type”: “answer”,
       “expression”: “Playing Luis Fonsi – Despacito ft. Daddy Yankee”
     },
     {
       “identifier”: “kJQP7kiw5Fk”,
       “identifier_type”: “youtube”,
       “type”: “video_play”
     }]

2.Parsing the response

Then the speaker parses the response in the following way.

The Speaker traverses through all the actions returned in the response and checks for all the “identifier” by assigning a custom class to it.

class VideoAction(BaseAction):
   def __init__(self, identifier , identifier_type):
       super().__init__()
       self.identifier = identifier
       self.identifier_type = identifier_type

Now we check whether the query is the type of a custom class VideoAction and then the client processes the query as the response.

      elif isinstance(action, VideoAction):
          result[‘identifier’] = action.identifier
           audio_url = result[‘identifier’]

3.Implementing the Actions

Now that we have identified that the response contains a Video Action, we can finally implement a way to play the audio from the URL.
We use a music player called MPV Music Player and the library Subprocess to make it run asynchronously.

if ‘identifier’ in reply.keys():
   classifier = reply[‘identifier’]
   if classifier[:3] == ‘ytd’:
       video_url = reply[‘identifier’]
       video_pid = subprocess.Popen(‘mpv –no-video https://www.youtube.com/watch?v={} –really-quiet &’.format(video_url[4:]), shell=True) # nosec #pylint-disable type: ignore
       self.video_pid = video_pid.pid

This is how audio is streamed from youtube videos in SUSI Smart Speaker.

Resources

Adding Support for Playing Youtube Videos in SUSI iOS App

Post author:Jogendra Kumar
Post published:August 21, 2018
Post category:API FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

SUSI supports very exciting features in chat screen, from simple answer type to complex map, RSS, table etc type responses. Even user can ask SUSI for the image of anything and SUSI response with the image in the chat screen. What if we can play the youtube video from SUSI, we ask SUSI for playing videos and it can play youtube videos, isn’t it be exciting? Yes, SUSI can play youtube videos too. All the SUSI clients (iOS, Android, and Web) support playing youtube videos in chat.

Google provides a Youtube iFrame Player API that can be used to play videos inside the app only instead of passing an intent and playing the videos in the youtube app. iFrame API provide support for playing youtube videos in iOS applications.

In this post, we will see how playing youtube video features implemented in SUSI iOS.

Getting response from server side –

When we ask SUSI for playing any video, in response, we get youtube Video ID in video_play action type. SUSI iOS make use of Video ID to play youtube video. In response below, you can see that we are getting answer action type and in the expression of answer action type, we get the title of the video.

actions:
[
{
type: "answer",
expression: "Playing Kygo - Firestone (Official Video) ft. Conrad Sewell"
},
{
identifier: "9Sc-ir2UwGU",
identifier_type: "youtube",
type: "video_play"
}
]

Integrating youtube player in the app –

We have a VideoPlayerView that handle all the iFrame API methods and player events with help of YTPlayer HTML file.

When SUSI respond with video_play action, the first step is to register the YouTubePlayerCell and present the cell in collectionView of chat screen.

Registering the Cell –

register(_:forCellWithReuseIdentifier:) method registers a class for use in creating new collection view cells.

collectionView?.register(YouTubePlayerCell.self, forCellWithReuseIdentifier: ControllerConstants.youtubePlayerCell)

Presenting the YouTubePlayerCell –

Here we are presenting the cell in chat screen using cellForItemAt method of UICollectionView.

if message.actionType == ActionType.video_play.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.youtubePlayerCell, for: indexPath) as? YouTubePlayerCell {
cell.message = message
cell.delegate = self
return cell
}
}

Setting size for cell –

Using sizeForItemAt method of UICollectionView to set the size.

if message.actionType == ActionType.video_play.rawValue {
return CGSize(width: view.frame.width, height: 158)
}

In YouTubePlayerCell, we are displaying the thumbnail of youtube video using UIImageView. Following method is using to get the thumbnail of particular video by using Video ID –

Getting thumbnail image from URL
Setting image to imageView

func downloadThumbnail() {
if let videoID = message?.videoData?.identifier {
let thumbnailURLString = "https://img.youtube.com/vi/\(videoID)/default.jpg"
let thumbnailURL = URL(string: thumbnailURLString)
thumbnailView.kf.setImage(with: thumbnailURL, placeholder: ControllerConstants.Images.placeholder, options: nil, progressBlock: nil, completionHandler: nil)
}
}

We are adding a play button in the center of thumbnail view so that when the user clicks play button, we can present player.

On clicking the Play button, we are presenting the PlayerViewController, which hold all the player setups, by overFullScreen type of modalPresentationStyle.

@objc func playVideo() {
if let videoID = message?.videoData?.identifier {
let playerVC = PlayerViewController(videoID: videoID)
playerVC.modalPresentationStyle = .overFullScreen
delegate?.loadNewScreen(controller: playerVC)
}
}

The methods above present the youtube player with giving Video ID. We are using YouTubePlayerDelegate method to autoplay the video.

func playerReady(_ videoPlayer: YouTubePlayerView) {
videoPlayer.play()
}

The player can be dismissed by tapping on the light black background.

Final Output –

Resources –

Play Youtube Videos in SUSI.AI Android app

Post author:batbrain7
Post published:August 20, 2018
Post category:FOSSASIA
Post comments:0 Comments

SUSI.AI Android app has many response functionalities ranging from giving simple ANSWER type responses to complex TABLE and MAP type responses. Although, even after all these useful response types there were some missing action types all related to media. SUSI.AI app was not capable of playing any kind of music or video.So, to do this the largest source of videos in the world was thought and the functionality to play youtube videos in the app was added.

Since, the app now has two build flavors corresponding to the FDroid version and PlayStore version respectively it had to be considered that while adding the feature to play youtube videos any proprietary software was not included with the FDroid version. Google provides a youtube API that can be used to play videos inside the app only instead of passing an intent and playing the videos in the youtube app.

Steps to integrate youtube api in SUSI.AI

The first step was to create an environment variable that stores the API key, so that each developer that wants to test this functionality can just create an API key from the google API console and put it in the build.gradle file in the line below

def YOUTUBE_API_KEY = System.getenv(‘YOUTUBE_API_KEY’) ?: ‘”YOUR_API_KEY”‘

In the build.gradle file the buildConfigfield parameter names API_KEY was created so that it can used whenever we need the API_KEY in the code. The buildConfigfield was declared for both the release and debug build types as :

release {
minifyEnabled false
proguardFiles getDefaultProguardFile(‘proguard-android.txt’), ‘proguard-rules.pro’
buildConfigField “String”, ‘YOUTUBE_API_KEY’, YOUTUBE_API_KEY
}

debug {
buildConfigField “String”, ‘YOUTUBE_API_KEY’, YOUTUBE_API_KEY
}

The second step is to catch the audio and video action type in the ParseSusiResponseHelper.kt file which was done by adding two constants “video_play” and “audio_play” in the Constant class. These actions were easily caught in the app as :

Constant.VIDEOPLAY -> try {
  identifier = susiResponse.answers[0].actions[i].identifier
} catch (e: Exception) {
  Timber.e(e)
}

Constant.AUDIOPLAY -> try {
  identifier = susiResponse.answers[0].actions[i].identifier
} catch (e: Exception) {
  Timber.e(e)
}

The third step involves making a ViewHolder class for the audio and video play actions. A simple layout was made for this viewholder which can display the thumbnail of the video and has a play button on the thumbnail, on clicking which plays the youtube video. Also, in the ChatFeedRecyclerAdapter we need to specify the action as Audio play and video play and then load the specific viewholder for the youtube videos whenever the response from the server for this particular action is fetched. The YoutubeViewHolder.java file describes the class that displays the thumbnail of the youtube video whenever a response arrives.

public void setPlayerView(ChatMessage model) {
  this.model = model;

  if (model != null) {
      try {
          videoId = model.getIdentifier();
          String img_url = “http://img.youtube.com/vi/” + videoId + “/0.jpg”;

          Picasso.with(itemView.getContext()).load(img_url).
                  placeholder(R.drawable.ic_play_circle_filled_white_24dp)
                  .into(playerView);
      } catch (Exception e) {
          Timber.e(e);
      }

  }

The above method shows how the thumbnail is set for a particular youtube video.

The fourth step is to pass the response from the server in the ChatPresenter to play the video asked by the user. This is achieved by calling a function declared in the IChatView so that the youtube video will be played after fetching the response from the server :

if (psh.actionType == Constant.VIDEOPLAY || psh.actionType == Constant.AUDIOPLAY) {
  // Play youtube video
  identifier = psh.identifier
  chatView?.playVideo(identifier)
}

The fifth step is a bit complicated, as we know the app contains two flavors, one for the FDroid version that contains all the Open Source libraries and software and the second is the PlayStore version which may keep the proprietary software. Since, the youtube api is nothing but a piece of proprietary software we cannot include its implementation in the FDroid version and so different methods were devised to play the youtube videos in both versions. Since, we use flavors and only want that Youtube API is compiled and included in the playstore version the youtube dependency was updated as :

playStoreImplementation files(‘libs/YouTubeAndroidPlayerApi.jar’)

This ensures that the library is only included in the playStore version. But on this step another problem is encountered, as now the code in both the versions will be different an encapsulation method was devised which enabled the app to keep separate code in both flavors.

To keep different code for both variants, in the src folder directories named fdroid and playStore were created and package name was added and then finally a java folder was created in each directory in which the separate code was kept for each flavor. An interface in the main directory was created which was used as a means to provide encapsulation so that it was implemented differently in each of the flavors to provide different functionality. The interface was created as :

interface IYoutubeVid {
fun playYoutubeVid(videoId: String)
}

An object of the interface was made initialised in the ChatActivity and separate classes

named YoutubeVid.kt was made in each flavor which implemented the interface IYoutubeVid. So, now what happened was that depending on the build variant the YoutubeVid class of the particular flavor was called and the video would play according the sources that are available in that flavor.

In ChatActivity the following implementation was followed :

Declaration :

lateinit var youtubeVid: IYoutubeVid

Initialisation :

youtubeVid = YoutubeVid(this)

Call to the overridden function :

override fun playVideo(videoId: String) {
Timber.d(videoId)
youtubeVid.playYoutubeVid(videoId)
}

Final Output

References

Youtube Android player API : https://developers.google.com/youtube/android/player/
Building apps with product flavors – Tomoaki Imai https://medium.com/@tomoima525/building-multiple-regions-apps-with-product-flavors-good-and-bad-fe33e3c31060
Kotlin style guide : https://android.github.io/kotlin-guides/style.html

Implementing YouTube Search API with WebClient

Post author:PrP-11
Post published:August 8, 2018
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

SUSI.AI is an assistant which enables us to create a lot of skills. These skills offer various functionalities which performs different actions. Therefore SUSI has implemented various action types. Some of these are:

Answer
Table
Maps
Rss
audio_play
video_play

When a user answers a query the server sends response in form of actions type. The client then scans the response JSON object and checks the type of action and performs the desired operations. This action types are verified by

1.Processing a Query to the server

3.Implementing the Actions

Resources

Tags