Moving from standard TV to the cloud, with streaming

  • Articles
  • Streaming

Introduction

What is streaming?

Streaming or media streaming is a form of transferring data so it can be treated as a continuous stream. This method is preferred to downloading because the client can display the video before the entire file has been received.

The client must also be able to store data in a buffer in case it is received quicker than required.

What are the types of streaming?

While the past video streaming technologies relied on RTP with RTSP, today’s technologies are almost exclusively based on HTTP.

We can totally use the default HTTP streaming technology without anything else!
However, there is SmoothStreaming. This is a feature of Internet Information Services (IIS) Media Services, which is a HTTP-based platform. SmoothStreaming clients get minimal buffering, because this technology adapts the quality of the video stream in real-time with regards to the client’s bandwidth and CPU usage. This technology is known as adaptive streaming.

 

Choosing the platform and a framework

Picking the platform

Streaming can be implemented on any mobile/tablet platform, such as: iOS, Android, and WinRT (Windows Runtime). Due to the fact that the project we had was already implemented on iOS and Android, we decided to extend it to WinRT.

Magine TV Streaming

 

Selecting a framework that handles encrypted content

After choosing the WinRT platform, the logical choice is to subscribe to Microsoft technologies/products. The Microsoft content protection technology we used is PlayReady.

 

Licensing management

Handling DRM

Every DRM server has its own custom licensing system. Therefore, in order to handle this, custom authentication must be provided. The following example illustrates a custom authentication request.

In the above example, a SToken and UserId are needed for the authentication process. These are encoded in a JSON format, which is in turn encoded in a Base64 format to be sent to the DRM server in order to obtain the license to play the respective content.

The five pillars of PlayReady

In order to use PlayReady in a WinRT project, one must first install PlayReady and add it as a reference. Afterwards, the following five classes must be used:

 

Implementing the player

PlayerFramework

The first logical step after knowing how to handle DRM is obviously to use this in a player. One could choose to use Microsoft’s MediaPlayer in WinRT. In spite of this being a good idea, the MediaPlayer is rather rudimentary.

When using it, we discover that we have to do a lot of grunt work in order to allow it to use adaptive streaming, DRM, and even closed captioning.

The next course of action would be to seek a player designed to work with these technologies.
The search should be very short, as one would discover that Microsoft has already developed and is continuing development for the PlayerFramework.

So what is this PlayerFramework? It’s actually what we were looking for, and even more. Besides the fact that it supports all of the above listed requirements and is in continuous development, it’s open source! Yes, you’ve read that right.

Configuring the player

Assuming that we’ve created a blank page in our application called “PlayerPage”, we should add the player inside the page using XAML. The whole XAML should look like this:

Some of these settings could be eluded of course. If one does not wish to use TTML captions, one might skip this particular plugin.

In order to proceed with what you need to write in the code-behind, you will also require a Playback.cs class, responsible for whatever happens with the PlayerFramework Media Player (which includes setting the Source property in order to play streams). Therefore, one must make the class “Playback.cs”, which contains the following code:

Now that we have the Playback.cs class, we may proceed with writing the following code in the PlayerPage.xaml.cs:

The manifest

As we’ve seen above, we can now play a stream protected by DRM. But what if we want to change between streams (if there is more than one video/audio/caption stream)?
When the manifest from the stream was ready, we saved it in a field called “_manifest” in order to use it for these scenarios.

Video stream

The video stream can only be one. It can have more than one tracks (which contain different qualities or bitrates). It can be obtained by creating a property like illustrated below:

Audio stream

Now that we have a property with the current video stream, we may obtain the different audio streams present inside of it using the following code (making a property in the process):

Subtitle stream

We can obtain the subtitle (caption) stream from the same video stream, by making another property, like so:

All of these streams can be extracted from the manifest object for further examination and processing.

Changing between streams while playing

What if we want to change the current audio stream from English to French? How do we do that?
How can we change the current caption stream?
These are the questions that one needs to ask herself/himself when attempting to switch between streams while the player is active. There are 2 ways of making this work:

Through the manifest

One can use the VideoStreams and CaptionStreams properties explained above in order to change them from the manifest object itself.
The way to do this is simple, following these steps:

1. In the adaptiveSourceManager_AdaptiveSourceOpenedEvent  event, one must also write the following:

2. Once attached to the AdaptiveSourceStatusUpdatedEvent, everything is set. This event will fire each time the manifest on the client is updated. Effectively, one can use the method ‘SelectStreamsAsync’ to set the corresponding audio/caption streams. We will not go into this however, since this has a higher time-complexity than the method we will present below.

Through the MediaPlayer’s properties

The MediaPlayer has two properties we can use: SelectedAudioStream and SelectedCaption. In order to obtain values for these two properties, we can use two other properties from the MediaPlayer: AvailableAudioStreams and AvailableCaptions.

For instance, if we want to set the audio stream and the caption stream to the first ones, we will write the following code:

These two assignments will set the current audio/caption streams accordingly.
Why are these two more effective than the above method?
Mainly it’s because the method presented above in “Through the manifest” has a higher time-complexity. It is a lower level operation granted, but it is handled much more gracefully by PlayerFramework when setting the properties SelectedAudioStream and SelectedCaption.

Challenges and issues

The main challenge we faced was the fast-forward bug that appeared whenever we switched between audio/caption streams. This was solved after hours of work switching between the methods from “Through the manifest” and “Through the MediaPlayer’s properties”, and finding the property RealTimePlayback present on the MediaPlayer which needed to be set to ‘true’.

Another challenge we faced was obviously when we were dealing with the DRM server. Since every DRM server is custom, license acquisition requests could return a status code 500, 504, 200, etc. which needed to be figured out thoroughly, and treated accordingly. This was achievable only because the DRM server API team and our development team communicated extremely efficient.

 

Closing thoughts

Even if creating a player that actually works and handles multiple streams might seem like a difficult task, because of the prerequisites (the five pillars, Playback.cs, treating DRM), with logical thinking and thoroughness one can get the job done in an extremely fashionable manner.

Despite the fact that this project was much more than what we described here (player customization, subtitle/audio that are remembered for different streams), if one follows the right steps mentioned here, she/he can get a good start on creating a player that relies on streams with DRM protection and adaptive streaming.

Also worth mentioning, are the quirks such as the one found in PlayerPage.xaml.cs which cannot be deduced so easily and require a fair amount of research:
s_propertySet["{A5CE1DE8-1D00-427B-ACEF-FB9A3C93DE2D}"] = s_adaptiveSourceManager;
In conclusion, we hope that these guidelines will help anyone who wishes to create apps which use players to stream data.

References:

  1. https://en.wikipedia.org/wiki/Streaming_media
  2. //www.microsoft.com/silverlight/smoothstreaming/

Author:
Ștefan-Gabriel Gavrilaș

Co-Author:
Ionuț-Cătălin Popovici