Grand Challenge on the Use of Image Restoration for Video Coding Efficiency Improvement – ICIP 2017

Aims and Scope

Image restoration schemes have traditionally been used only in a blind scenario, where the objective is to improve the quality of an image after it has suffered various degradations in capture, processing and transmission ­ at the time when the source is no longer available. Usually these schemes are fairly complex and as such, they are either applied only to images or to video frames in offline processing scenarios.

However, image restoration schemes can also be used for compression, because any information that can be restored can be saved, thereby aiding compression. In this use case, one can expect the source to be known at the time of compression, and the encoder can send additional side ­information to the decoder with the compressed bit­stream to specify explicitly how the decoder is supposed to restore. While on one hand, this makes the restoration task substantially easier in principle, for the specific case of video, keeping the computational constraints low enough still remains very challenging.

One type of restoration that all video codecs already incorporate today is deblocking. This challenge will focus on additional restoration steps after deblocking. Recent research work has demonstrated great progress and potential in applying spatial and/or temporal restoration techniques in improving the overall coding efficiency in video compression. Both in-­loop and out-­of-­loop filtering algorithms have been proposed. In both cases, side information may be obtained at the encoder and transmitted together with the core bitstream to the decoder, to facilitate in-­loop or out­-of-­loop restoration after decoding and deblocking, in order to remove coding artifacts, or restore certain information that has been removed by the lossy video compression.

Challenge Requirements

In order to keep the barrier to entry low and encourage researchers in image restoration to get involved in compression, for the purpose of this challenge we would like to focus only on side ­information driven restoration that is conducted out of the loop, and may be independent of the specific codec used.

Participants will be provided a compressed video test set for development and testing their algorithms. The test set will consist of a variety of source video clips containing variable spatial and temporal content characteristics, HEVC and VP9 compressed bitstreams at a variety of bit­rates for each sequence, and the corresponding decoded sequences. The rate­ distortion points will be provided (or could be computed) and an associate independent decoder will be included as well. Participants will be required to design a restoration scheme and a side­ information layer driving the proposed scheme, such that when applied to decoded frames with side­ information, it will serve to reduce their distortions to the source. BDRATE will be computed comparing the new rate­distortion points for each test sequence considering the side information and improved fidelity, to the original. To keep things simple, the metric used for BDRATE computation will be based on PSNR, computed in a universally accepted manner.

Participants will be required to submit a paper describing their algorithm and the obtained results over the test set through the IEEE ICIP 2017 paper submission system online at . In addition, code should be submitted as a zip file as supplementary material under paper submission.

Evaluation Criteria

  1. The organizing committee provides a download location for all testing materials, and the contributors are expected to conduct their own proposed out­loop pre­/post processing algorithms;

  2. Sampled source video clips will be provided, with resolution varying from CIF (352x288) to VGA (640x480), characterized by a large variety of texture and motion complexities. Each source video is in 4:2:0 YUV color format, containing 150 frames.

  3. Each video clip will be compressed at 6­8 different rate­distortion points, where average Y­PSNR (Peak Signal­To­Noise Ratio) in dB will be used as the objective distortion metric. The codec used will be a combination of HEVC and VP9. [Note that there is already some restoration that is conducted in these codecs, for ex. deblocking, SAO, etc., but we are interested in additional restoration beyond these].

  4. All the encoded bitstreams will be provided and sample decoders may be provided as well, so that the contributors may use the decoder to decode the bitstreams to obtain the decoded video at their own effort. This allows contributors to simulate the rate­distortion points at their own. The decoded frames from the encoded bitstreams will be provided for convenience.

  5. The contributor may extract / create side information from the source and compressed videos and encode the side information in their own designated format. The encoded side information that is transmitted to the decoder may be associated with the encoded bitstream, or it may be independent of the codec;

  6. The contributors provides the software to leverage the decoded side information and apply them to the decoded video frames to obtain new restored video clips;

  7. The evaluation committee will use the BDRATE scheme to reevaluate the new rate­distortion performance facilitated by the side information created by the contributors, and the winner will go to the contributors whose algorithm and software produce the highest BDRATE reduction;

  8. The contributors are requested to submit software that can be executed in Linux. The source code is to be written in C/C++. In the event they do not want to make the source code available, they can provide an executable that runs on Linux. The running time of the software should be reasonable. The side information extraction and creation at the encoder side and application of the side information to the decoded videos together should not exceed 6 hours per video clip at every rate­distortion point;

  9. Contributors are also expected to provide the evaluation committee with sufficient rights to allow usage of the provided source­code or executable.

Paper Submission Instructions

The papers should be formatted according to the IEEE ICIP 2017 Paper Kit available under paper submission on , and should be submitted online at by the deadline May 17, 2017. Papers are not mandatory, but once submitted, they will be reviewed and considered in a fast track. The deadline for the zipped software, together with all the supporting documents is July 1, 2017, and please have them submitted through email to either Zoe Liu or Debargha Mukherjee . `

Software / Code Submission Instructions

Code should be submitted as a zip file as a supplementary material under paper submission. Your code zipped folder should contain executable file of the algorithm on the Linux platform along with a README file containing the complete instructions on how to run the code.

Materials / Tools

  1. Two sets of source videos are provided: derflr (containing CIF - 352x288 and SIF - 352x240 video clips) and midres (containing 480p - 854x480 video clips). We select 8 clips in total, 5 for derflr and 3 for midres. Each source video clip is in 4:2:0 YUV color format and contains 60 frames;

  2. All the video clips have been encoded and decoded by two codecs: VP9 and HEVC (x.265 as the encoder). For every video clip, there are 6 target bitrates that have been chosen, which is listed in this spreadsheet [xlsx] [pdf] . Hence 6 pairs of Rate-Distortion (RD) points are collected for every video clip, also listed in the same spreadsheet. For simplicity, we collect the file size of each encoded bitstream as the rate value, and use the conventional PSNR value as the distortion metric.

  3. The VP9 encoder and VP9 decoder , together with the HEVC/x265 encoder are provided. One could simply use the ffmpeg to decode the HEVC bitstreams. For VP9, we use the following commend lines:

    VP9 Encoder: vpxenc -o \$DstFile.webm \$SrcFile.y4m --codec=vp9 --good --cpu-used=0 --threads=0 --profile=0 --lag-in-frames=25 --min-q=0 --max-q=63 --auto-alt-ref=1 --passes=2 --kf-max-dist=150 --kf-min-dist=0 --drop-frame=0 --static-thresh=0 --bias-pct=50 --minsection-pct=0 --maxsection-pct=2000 --arnr-maxframes=7 --arnr-strength=5 --sharpness=0 --undershoot-pct=100 --overshoot-pct=100 --frame-parallel=0 --tile-columns=0 --test-decode=warn -v --psnr --target-bitrate=\$Bitrate --limit=\$Frames

    VP9 Decoder: vpxdec --progress --codec=vp9 --i420 -o \$DstFile.yuv \$DstFile.webm ,


    HEVC/x265 Encoder: x265 --preset veryslow --keyint 150 --merange 256 --tune psnr --frame-threads 1 --frames \$Frames --bitrate \$Bitrate \$SrcFile.y4m \$DstFile.bin .

    All the encoded bitstreams and the corresponding decoded videos are also provided, stored in the corresponding "derflr" and "midres" folders.

  4. The tools of calculating the BDRARTE and BDPSNR are provided, both in matlab code.


Zoe Liu & Debargha Mukherjee

Google, Inc. 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA

Proposed Awards and Sponsorship

One or two (if tied) winners will be selected, tightly according to the Evaluation Criteria. The total award is US $2,000 including the taxable amount, which will be fully sponsored by Google Inc. The organizers reserve the complete right in the final judgement and decision.