Porting the RealVideo 8/9/10 decoder

Porting the RealVideo 8/9/10 decoder

Last updated: $Date: 2005/07/12 19:48:29 $

  1. What is RealVideo 10? Do I need a new decoder?
  2. How do I get the source code for RealVideo?
  3. How do I check out and build only the video codecs?
  4. Where is the source code located?
  5. What is the ANSI C decoder?
  6. Where is and how do I build the ANSI C decoder?
  7. So what is this I hear about the two different bitstream formats, 'Raw' and 'RealVideo'?
  8. Why don't my test files work with the command line decoder?
  9. How do I create more test files than the ones found in the TCK?
  10. When are B frames used in RealVideo?
  11. Is skipping B frames acceptable if they are still encountered in the bitstream?
  12. Why are there no bitstreams in the TCK with B frames?
  13. I see interlaced being mentioned in the source code. Do I need to support this?
  14. What is RPR?
  15. When my port is complete, how do I release it to the Helix community?



What is RealVideo 10? Do I need a new decoder?
No, RealVideo 10 uses the same bitstream format as RealVideo 9, the 30% better compression efficiency is due to encoder side improvements. The RealVideo 8+9 combo decoder is therefore able to decode RealVideo 10 bitstreams, and any documentation and information referring to decoding RealVideo 9, also applies to decoding RealVideo 10.

How do I get the source code for RealVideo?
Check out the Helix Client Restricted Quick Start Guide. This shows you the .buildrc file modifications you need to make to get to the RealVideo source code, and has links to general build system instructions.

How do I check out and build only the video codecs?
Usually the codecs will be built as part of other build targets, like 'splay', but in many cases it is useful to checkout and build only the video codecs. To build only the video codecs, in an empty folder type 'build', then
- Option [0]: choose BIF branch 'helix_restricted'.
- Option [1]: choose Target, then type datatype_rm_video_codec_rv89combo for the RealVideo 8/9/10 combo decoder, or datatype_rm_video_codec_rvg2dec for RealVideo G2. These targets are not listed in the list of targets, but will build just fine.
- Option [2]: choose Profile "helix-client-all-defines"
- Then you can run Build (release or debug), Checkout source etc.
This will build the decoder libraries and command line test decoder in:
datatype-restricted/rm/video/codec/rv89combo, and the decoder DLLs in:
datatype/rm/video/codec/rv89combo

Where is the codec source code located?
When you run the build system, it redirects and checks out the codec source code to datatype-restricted/rm/video/codec/.. (rv89combo or rvg2dec). The actual CVS location is however /rarvcode-video/codec, but you should rarely need to worry about this. The CVS location is needed only if you need to add a new module.

What is the ANSI C decoder?
The ANSI C RealVideo 8/9 decoder (rv89combo_c) is a more or less manually translated version of the C++ decoder. It supports all the same features as the C++ decoder, and has been preferred by several projects, since it does not require a C++ compiler. Note that it does not include any of the optimized code for ARM, StrongARM, XScale, Bulverde, or AltiVec, instructions. These optimized libraries are available only for the C++ version of the decoder. X86-SIMD has been integrated with the ANSI C decoder. Optimizations available in asm / C /or pre-compiled object files for all platforms are leaf functions and use the same functions prototypes. If you follow the code it should be simple to plug in the ARM optimizations into the C code.

Where is and how do I build the ANSI C decoder? The ANSI C decoder is located in the rv89combo_c sub-folder in the rv89combo module. You can build and check out this with the datatype-restricted_rm_video_codec_rv89combo_rv89combo_c build target, in the build system. In order to link the decoder DLLs with the ANSI C decoder libraries, make sure your profile defines HELIX_FEATURE_ANSIC_RV89COMBO, by adding this to the profile in ribosome/build/umakepf: project.AddDefines('HELIX_FEATURE_ANSIC_RV89COMBO'). Then run build for rv89combo as described in question 3 above. Later, you can go to datatype/rm/video/codec/rv89combo and run umake, (n)make manually. To work on the ANSI C source code, and build the ANSI C decoder command line test decoder, go to datatype-restricted/rm/video/codec/rv89combo/rv89combo_c and run umake, (n)make.
So what is this I hear about the two different bitstream formats, 'Raw' and 'RealVideo'?
Typically, our decoder and encoder projects build a DLL and a test executable. The test executable or command line tools work on 'Raw' bitstreams, while the DLLs interface with for instance RealProducer or RealPlayer. RealProducer encodes 'RealVideo' files packaged in RM (RealMedia) that can be played in RealPlayer.

Both Raw and RealVideo formats can refer to either RV8 or RV9, but the Raw format is used with the raw file decoder (the command line decoder) and includes a picture start code to signal the beginning of each frame. In contrast, the RealVideo format is contained in RM files (i.e. files encoded in the RealMedia container file format using Helix Producer or a similar tool). The RealVideo format does not have a PSC for each frame. Instead, RealVideo bitstreams are typically broken into segments for packetization and streaming. Rather than searching for a PSC, the decoder interface for RealVideo bitstreams (AKA the 'frontend') accepts information describing the segment sizes and offsets in order to keep track of where each frame begins and ends.

The raw file decoder bypasses the frontend and looks for a PSC to get the information it needs to decode each frame in the raw bitstream. The raw file decoder is typically used to provide a simple test harness for codec optimization. The codec group also uses the raw format to experiment with coding techniques.

Unfortunately, having the two bitstream formats has led to some confusion on the part of people who are optimizing the codecs. In the case of a recent port of the RV8 decoder, the engineers working on the project didn't realize that the optimized decoder would be driven through the frontend interface when integrated with the helix client. So they stripped out the frontend interface and also removed special code that supports the 'RealVideo' bitstream format when their code coverage tool showed that this code wasn't being exercised. What they ended up with was an optimized raw test file player! So now we're being careful to warn people that the raw bitstream decoder is only for testing optimizations, and you need to be familiar with the decoder frontend to understand how the decoder will interface with the rest of the system in an actual product.

Another question we often get asked is whether there is any way to extract the 'RealVideo' bitstreams from .rm files and feed them to the raw bitstream decoder. We're working on some tools to enable this, but they're not generally available yet.

Why don't my test files work with the command line decoder?
Make sure you know whether the files are 'Raw' or 'RealVideo'. If RealPlayer can play them, they are 'RealVideo' RM files, and the command line decoder can not decode them. If the files are RealVideo 8, also remember to specify the '-8' command line switch to tell the decoder it is RV8. To verify the decoded YUV files (I420 format), you can use yuv2avi.exe, included in the Video Processing Tools package (see below) or snrcomp.exe, included in the TCK to compare with the original.

How do I create more test files than the ones found in the TCK?
To create more Raw test vectors, you can use the rv40enc.exe and rv30enc.exe command line tools found in the rarvcode-video file sharing section in the encoders.zip file. Type rv40enc.exe -h to obtain their usage information. These tools require raw I420 (YUV12) files as their input. You can create these via the tools in the Video Processing Tools package (video_tools.zip) in the same file sharing section. The zip file includes a readme.txt that explains the process.

To create more RealVideo test files, you can use Helix Producer available from the distribution Binary file sharing section. This is a command line tool, see the documentation in its docs folder to get started. Some 3rd party GUI front-ends are available, see this discussion forum thread [external link]. Alternatively, RealProducer Basic or Plus can be used.

When are B frames used in RealVideo?
B frames generally occur in RealVideo streams under the following conditions:

Resolution <= 192x160:
fps <= 15.5 : 0
fps > 15.5 : 0 or 1 (adaptive)

Resolution > 192x160:
fps <= 7.5 : 0
7.5 < fps <= 15.5 : 0 or 1 (adaptive)
fps > 15.5 : 0, 1, 2, or 3 (adaptive)

You can disable B frames in Helix Producer using the custom setting patternAdaptivity set to 0. See this information post [external link] for all the custom codecProperties that can be used for Helix Producer.

Is skipping B frames acceptable if they are still encountered in the bitstream?
That depends on your device, processor, and implementation. In general, B frames are discardable, and can be discarded if your processor does not have enough CPU power to decode full framerate. Whether or not this acceptable if it happens all the time is another question.

Why are there no bitstreams in the TCK with B frames?
This is because strict B frames compliance is not needed. When you need to test bitstreams with B frames, please use the methods described above to generate additional test files.

I see interlaced being mentioned in the source code. Do I need to support this?
We have specified and implemented some basic interlaced functionality in RealVideo 9, but without interlaced support anywhere else in the system, it has not been fully tested and completed. If your port is targeted towards a playback device that could potentially support interlaced, let us know, otherwise, you can ignore interlaced support.

What is RPR?
RPR stands for Reference Picture Resampling, and is a technique where the RealVideo encoder will resample an image before encoding, encode at this smaller size, and indicate in the stream header the size at which it expects the decoder to resample and display the video at. The size changes are automatic in the encoder, and occurs when the encoder detects that the video resolution is so high relative to the bitrate, that it is more efficient and will result in better quality, if the video is encoded at a smaller resolution. Size changes are dynamic, and can occur mid-stream.

RPR will not be used for sizes equal to or below 192x160. For larger framesizes, RPR will not resize to sizes below 192x160. RPR can not be used with the 'Raw' command line format discussed above, it works only with RM files.

You can disable RPR in the encoder with enableRPR custom encoder setting. See this information post [external link] for all the custom codecProperties that can be used for Helix Producer.

When my port is complete, how do I release it to the Helix community?
Your RealVideo port can be checked in to Helix CVS as either a patch to the existing reference code, or as a standalone platform package. Which of these options is used generally depends on what your architecture is and how heavily you have modified the reference code.

Patch releases must contain the absolute minimum of changes to the reference code. Ideally, a patch submission contains only optimized leaf functions, with no changes to the call graphs or data flow. Changes in the data flow of the reference code will be reviewed carefully, and will only be accepted if they are shown to have a positive or at least neutral impact on the performance of all other systems using the reference code. If accepted, a patch release will be merged with the reference code.

Standalone or 'platform' releases are often more appropriate for DSPs, asynchronous multiprocessor systems, and other architectures that diverge from the general purpose processor. Often the data flow must be changed, or hardware specific optimizations such as DMA are adopted. In these cases, the optimized RealVideo port is checked in as a standalone release in the 'platform' subdirectory. For rv89combo and rv89combo_c, this is:
rarvcode-video/codec/rv89combo/platform

Before any release is checked in to Helix CVS, it must pass the TCK verification process. Once that is accomplished, you may submit either a patch release or a standalone release to the mailing list: rarvcode-video-dev@helixcommunity.org.

For a patch release, you would submit your CVS diffs along with any new files you have created. The complete Helix patch procedure is outlined here: https://helixcommunity.org/2004/patch/

For a standalone release, simply package everything that is needed to build your port into a zipfile and provide that. It is helpful to maintain the same directory structure as exists in the reference code, but it is not required.

If you're unsure about whether to proceed with a patch or standalone port, you may consult with engineers on the rarvcode-video-dev mailing list. Generally speaking, a patch release will be reviewed more carefully and may require more concessions on issues such as coding style and readability. There is no guarantee that a patch will be accepted, although you can improve your chances by reading the patch submission guidelines and obtaining guidance from Helix developers. You may always produce a standalone release if your patch is not accepted. The advantage to the patch release is that support for your platform will be easier to maintain by the community, and you'll benefit from bug fixes and new features that are added to the reference code. On the other hand, a standalone release is often the only way to achieve a high level of optimization for certain embedded architectures.

More questions and answers

For more questions and answers, please see this spreadsheet. Examples include how to reduce RealVideo decoder size, more on the differences between the 'Raw' and 'RealVideo' formats

Please visit RARVCode Help Forum to ask questions not answered here.