An Apple patent (number 20110202674) for dynamic adjustments of video streams has appeared at the US Patent & Trademark Office. It relates to streaming video from a server to a client.
Per the patent, using both short term and long term thinning mechanisms, the server reduces its data transfer rate when bandwidth is reduced and increases its data transfer rate when bandwidth is available. In the short term mechanism, the server will dynamically adjust the transmission rate of the video or the quality of the video based on parameters returned by the client. These parameters include currently-buffered data-size, buffered-data duration, and packet loss rate.
In the long term mechanism, the server will adjust the current maximum quality level allowed depending upon the round trip times of data packets transmitted. This invention decreases client rebuffering and packet loss and increases the efficiency of available bandwidth. The inventors are George P. Su and John K. Murata.
Here’s Apple’s background and summary of the invention: “The use of multimedia content on computing devices has increased substantially. Multimedia content is sent from a central server as streaming video and audio files. Streaming of video from a server to a client requires a server, a client and a network. The server transmits the content, the client receives and plays the content and the network transfers the data from the server to the client.
“The server sends out data in small chunks, called data packets, through the network to a client. The client receives the data packets which are then combined together to form the video program and played.
“Different protocols are employed to send data through a network including TCP, Reliable UDP and UDP Streaming. TCP or transport control protocol creates connections between servers and clients and guarantees reliable and in-order delivery of data from the server to the client. Reliable UDP, or reliable user datagram protocol, incorporates quality of service enhancements that improve the ability of real time streaming of data and provides pseudo-guarantees of delivery.
“Reliable UDP retransmits data that has not been received by the client until it detects that the data is late for real time streaming of content. Finally, in UDP streaming, there is no guarantee that the information sent by the server is received by the client. Rather, the data is sent by the server to the client without any retransmission. However, the client will report back to the client any packet loss that has occurred.
“The client can be a variety of devices such as a desktop or laptop computer. The client can also be a mobile device, an evolving category that currently includes but is not limited to cellular telephones, personal data assistants (PDAs) and mobile gaming consoles. For mobile devices, protocols have been developed to regulate the sending and receiving of data, one of which is 3GPP (3rd Generation Partnership Project).
“When the content is streamed, the bandwidth reaching that client may fluctuate. These fluctuations are the result of natural conditions affecting the transmission of data and from congestion in the data network itself. The result of these fluctuations may be interruption of the streaming video or underutilization of available bandwidth.
“Content providers have resolved these difficulties in a number of ways. One prior approach is to make several versions of the same content with different qualities of video. For example, a content provider provides two versions of a program, one lower resolution for lower bandwidth, and one higher resolution for higher bandwidth capabilities.
“The provider initially detects what the user is capable of receiving and sends her the appropriate version, an action called stream switching. Difficulties arise if the network bandwidth changes midstream. The provider may not be able to make adjustments to the stream dynamically.
“In another prior approach, the content provider produces two independent versions of a program that is not based on resolution or bandwidths. For instance, a video program is made with one version showing the program from the point of view of the hero and the other from the point of view of the villain. The user indicates which version she wishes to view and the server sends that version to the client.
“The provider may also wish to provide more versions of a program to stream which allows for more variation of bandwidths. Different cellular telephone models exhibit different reception rates and different telephone carriers provide different transmission rates. To provide the highest quality content to the user, content providers would create a version of their programming for each carrier and telephone model. However, this results in greater expenses to create and store additional versions of the same program.
“Another prior approach is to compress the streamed multimedia content to the user as it is being transmitted. The available bandwidth can be detected and the content is compressed to a necessary size in order to ensure smooth delivery. However, this entails more computing resources each time the number of users increases. To compress and stream a program to five users would not be difficult, but the computing resources required to compress and stream to ten thousand individual users presents a challenge.
“Another prior approach in which to decrease the amount of data being sent by the server is to selectively remove frames from the video program. In this method, the program is pre-created and stored on the server. Video programs are essentially still frames being shown at a rapid rate. Because the frame rate is high, the viewer is tricked into believing there is actual motion.
“The server analyzes the video program and based on an algorithm, makes decisions that indicate which frames are more important than others. The frames which are deemed important are the ‘key frames.’ When the bandwidth of the network decreases and the data transmission rate must be lowered, the server selectively removes non-key frames from the video, but the audio portion of the program remains intact.
“This process is called ‘thinning.’ The client receives the video program with the frames removed. The video may appear jerky because of the missing frames but the user is able to view the program primarily intact. Based on quality of service considerations, the audio portion of the program is not interrupted so that the viewing experience is improved.
“Based on the foregoing, it is clearly desirable to provide a mechanism that allows content providers to dynamically change the quality and transmission rate of multimedia content based on fluctuating bandwidth. It is further desirable to provide a mechanism for allowing content providers to present their offerings in various bandwidths without creating multiple versions of the same program.
“It is also desirable to provide a mechanism for adjusting the quality of a program and making it available to many users without the use of data compression. It is also desirable to provide a mechanism for dynamically adjusting the amount of data transmitted to make the most efficient use of available bandwidth.
“The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.”
— Dennis Sellers