This work focuses on improving streaming video applications through the use of multiple description (MD) coding. A multiple description coder segments a stream into two or more separately decodable streams and transmits these independently over the network. The quality of the received video improves with each received description, but the loss of any one of these streams does not cause complete failure. Thus video playback can continue, at a slight reduction in quality, without waiting for rebuffering or retransmission.

Of course, this gain in robustness comes with a cost. The total bit rate necessary for this MD system to achieve a given distortion will in general be higher than the corresponding rate for a single stream encoder to achieve the same distortion. It is a tradeoff between coding efficiency and robustness. However, in the type of application under consideration, it is not so much a question of whether it is useful to give up some amount of efficiency for an increase in reliability as it is a question of finding the most effective way to achieve this tradeoff.

Many approaches have been previously suggested for multiple description coding. Some of the many contributions include, multiple description quantization, correlating transforms, spatial segmentation, transform domain segmentation, and temporal segmentation. For an in depth review of MD coding see the overview by Goyal.

This particular work considers using an adaptive combination of a number of these methods by taking advantage of the fact that the encoder has access to the original source. The encoder can measure the performance of each approach during encoding and can choose which method to use in an optimal rate-distortion sense.

VIDEO

Multiple description vs. standard single description coding - 224 kB - <AVI>

Compares the distortion caused in each method due to the loss of three frames near the beginning of the sequence. The standard encoding method is shown on the left while MD coding is shown on the right.

RELATED PUBLICATIONS

B. Heng, J. Apostolopoulos, and J. Lim, “Rate-Distortion Optimized MD Mode Selection for Adaptive Multiple Description Video Coding,” to appear in the special issue on “Video Analysis and Coding for Robust Transmission” in the EURASIP Journal on Applied Signal Processing, 1st Quarter 2006.

B. Heng, J. Apostolopoulos, and J. Lim, “End-to-End Rate-Distortion Optimized Mode Selection for Multiple Description Video Coding,” Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, March 2005. <PDF>

B. Heng and J. Lim, "Multiple description video coding through adaptive segmentation,"
49th Annual Meeting of the SPIE, International Symposium on Optical Science and Technology, Denver, August 2004. <PDF>

B. Heng, "Multiple Description Video Coding Through Adaptive Segmentation".
Ph.D. Thesis Proposal, Massachusetts Institute of Technology, June 2004. <PDF>

When the first US television standard was introduced in 1941, interlaced scanning, or interlacing, was used as a compromise between video quality and transmission bandwidth. An interlaced video sequence appears to have the same spatial and temporal resolution as a progressive sequence and yet it only occupies half the bandwidth due to vertical-temporal sub-sampling. This sampling method takes advantage of the human visual system, which tends to be more sensitive to details in stationary regions of a video sequence than in moving regions. Prior to the introduction of the U.S. High Definition Television (HDTV) standard in 1995, interlaced scanning had been adopted in most video standards. For instance, in 1941, the National Television Systems Committee (NTSC) introduced an interlaced-based television standard that was used in the United States exclusively until the introduction of HDTV. As a result, interlacing is still widely used in video systems and is found throughout the video chain, from studio cameras to home television sets.

While interlacing does succeed in reducing transmission bandwidth, it also introduces a number of high frequency artifacts that can be distracting to the human eye. In addition, there are a number of applications where interlaced scanning is unacceptable. For instance, freeze frame display of a television image requires a whole frame to be displayed at once. It is also becoming more popular to view video on a progressively scanned computer monitor or high definition television set. These applications all require interlaced to progressive conversion.

One simple conversion method is to combine the even and odd fields to form a new frame (field repetition). In stationary regions this provides perfect reconstruction of the original video. However, in regions containing significant motion, this results in severe blurring. The desire to eliminate this type of interlacing artifacts provides motivation for developing high quality algorithms for interlaced to progressive conversion, also known as deinterlacing.

This work considers the use of motion adaptive deinterlacing. Motion adaptive deinterlacing methods are designed to take advantage of the varying information that is present in different regions of a video sequence. In stationary regions, there exists significant correlation between the current field and the adjacent fields. Temporal methods such as field repetition are designed to benefit from this fact. In moving regions, this temporal correlation does not exist, and more information can be found in adjacent pixels from the current field. Spatial methods take advantage of this fact. Motion adaptive deinterlacing attempts to combine the benefits of both temporal and spatial methods by segmenting each image into moving and stationary regions using sophisticated motion detection algorithms. Stationary regions are then reconstructed with a temporal method while moving regions are reconstructed with a spatial method.

Motion adaptive deinterlacing can be used to effectively combine the benefits of both spatial and temporal processing with good results. The resulting images are deinterlaced with the method that works best for each region depending on the presence or absence of motion. When compared to other deinterlacing algorithms, motion adaptive methods have been found to have excellent performance independent of the amount of motion present or the frame rate used.

VIDEO

Motion Adaptive Deinterlacing - 1.22MB - <AVI>

This sequence compares the results of deinterlacing using three different approaches. The top video demonstrates spatial deinterlacing and the middle demonstrates temporal deinterlacing. Motion adaptive deinterlacing combines the strengths of both spatial and temporal deinterlacing as is shown at the bottom.

RELATED PUBLICATIONS

B. Heng, "Application of Deinterlacing for the Improvement of Surveillance Video".
MS Thesis, Massachusetts Institute of Technology, June 2001. <PDF>