Entries tagged with wmv - Channel 10

Netflix’s Neil Hunt shares encoding workflow info

benwaggoner — Thu, 20 Nov 2008 01:50:00 GMT

Now, this is a blog post! Neil Hunt, Chief Product Officer for Netflix, has just put up a great blog post talking about their encoding workflow for their video streaming services.

It’s full of awesomeness, but I wanted to excerpt the section describing their 1st gen, 2nd gen, and HD encoding settings and workflow.

First Generation Encoding

Our first set of encodes are based on WMV3 and WMA in ASF with WMDRM10 (Janus). We chose these standards because the Janus components have been widely adopted by our CE partners such as Roku, LG Electronics, Samsung, TiVo, and of course Microsoft Xbox.
We encode most content at 500, 1000, 1600, and 2200kbps VBR, but some titles whose source quality merits it have also been encoded at 3400kbps. The highest bitrate encodes are fit into 720x480 non-square pixels (the usual 1.2 PAR for widescreen content, 0.9 PAR for 4:3), but optimum encoding at lower bitrates is achieved with fewer pixels. Encoded films are normally at 24fps to match the source, while shot-to-video and mixed material is de-interlaced to 30fps (or 25fps for PAL content).

Netflix has been using anamorphic video all along, which I think is an underused feature of Windows Media and other formats. When you’re limited to 720x480 pixels, you want to encode all the pixels you’ve got, without having to synthesize any extra.

Second Generation Encoding

The new Silverlight player (that some users are helping us test as I write) uses VC1 Advanced Profile encoding with PlayReady DRM. A key property is that each GOP header includes frame size and resolution, which allows us to assemble a stream on the fly from different bitrate encodes as your broadband bandwidth fluctuates. (Another key feature is more coverage, including Intel Macs and Firefox users.) We expect to switch completely to the new player later this year.
The VC1 encoders are more efficient than the WMV3 encoders, so we are currently encoding VC1AP at slightly lower birates: 375, 500, 1000, and 1500kbps, all square pixel. At some point we are likely to add a couple more resolutions of non-square pixel encodes capturing the original pixel-aspect-ratio of the source.
We are also re-wrapping the VC1AP encodes in WMDRM10 for CE devices, which will gradually switch to the more efficient encodes in future firmware upgrades.

This is a great example of the improved efficiency of the VC-1 Encoder SDK and tools based on it. Not all Windows Media/VC-1 encoding is equal; the latest tools can offer a very meaningful reduction in bitrate required for a give quality level, improving user experiences and the cost of delivering the content.

The new encodes are backwards compatible to older hardware and software encoders, so even older devices can take advantage of the improvements.

High Definition Encodes

Today we have rights to deliver about 400 streams in HD (720p). More titles will be added over time. We experimented with first-generation WMV3 encodes at 4000kbps and 5500kbps, but settled on second-generation HD encodes with VC1AP at 2600kbps and 3800kbps, which extends their accessibility down to lower home broadband connections. As with SD, encodes of film material are at 24fps, and encodes of shot-to-video material are at 30fps (or 25fps for PAL), rather than the 60fps that would come from a Blu-ray disc - we judged the 60fps content as too expensive of bandwidth for now. In general, these encodes are definitively better than SD, but won't challenge well-executed Blu-ray encodes - that would require a bitrate out of reach for most domestic broadband today. We believe Moore's law will drive home broadband higher and higher enabling full 1080p60 encodes in a few years.

There’s codecs for you – a good 720p experience at 3800 Kbps, which is probably a little below the average 480p bitrate used for MPEG-2 on the DVDs that fly around the nation in those Netflix envelopes.

I’m not that worried about 1080p60 encoding myself. The vast majority of fictional content is shot 24p, including 99% at least of Blu-ray discs. While Blu-ray players may output 60p, the encodes, and hence bitrate requirements, are still 1080p. I find that typical film content in VC-1 wants around 6-8 Mbps for a 1080p24 experience enough better than 720p to be worth the trouble.

I just got my Xbox set up to handle the Netflix streams – now I’ve got to go check out some of those 400 HD titles! Hmm. Pan’s Labyrinth and Heroes.

Anyway, it’s great stuff. I love it when partners can share this kind of detail about what they’re doing.

Tip: ClearType off for screen recordings

benwaggoner — Thu, 27 Mar 2008 05:37:00 GMT

I've been playing around a bunch with screen recordings lately (as you'll see in my next blog post), and I've noticed a pretty common problem: people leaving ClearType on while recording. ClearType is cool stuff, adding improved anti-aliasing on LCD displays. But the way it works is by taking advantage of the little sub-pixel strips of Red, Green, and Blue each LCD pixel is made of. This improves detail and readability. But, as you can see below, when you zoom in you get color fringing on that small text.

Those little color fringes make the video somewhat harder to encode, since they change with the position of the text. And they also look wrong when then video is played back on a non-LCD display, and really wrong on an oddball LCD display which has a different pattern of the colored strips.

ClearType Font Smoothing @ 100% Zoom

ClearType Font Smoothing @ 800% Zoom

Standard Font Smoothing @ 100% Zoom:

Standard Font Smoothing @ 800% Zoom:

When doing any screen captures, be it Camtasia or via a HDMI to SDI bridge like I used for my Expression Encoder 1.0 training, make sure you have ClearType off. In Vista, you do that via the Appearance Settings control panel's Effects button.

If you wind up with source video that has been recorded with ClearType on, turning on Chroma Search can help quality some by enabling the code to pick up on all hue/saturation changes on text as it moves around. I recommend using the "Full True Chroma" mode if using a tool based on the VC-1 Encoder SDK, like Expression Encoder 2.0.

VC-1 for screen recordings (New Expression Encoder encodes up)

benwaggoner — Tue, 15 Jan 2008 16:02:00 GMT

A few months ago I posted links to my Expression Encoder training. However, I didn't encode the original clips, and being a little obsessive I wanted to see wha I could pull off myself with VC-1 for screen shots.

You can see the results of my work here.

Some tips for encoding screen shots:

I-Frame DQuant. While I normally don't recommend DQuant for low bitrates with the VC-1 Encoder SDK, it makes sense to apply it to I-frames for screen recordings, since we don't need very many of them, and with so little motion, that really helps improve the quality of all frames based on the reference frame.
Long GOPs (distance between keyframes) can really help efficiency, since keyframes can take up the majority of bits in the files.
Use B-frames. >1 can pay off a lot in improved efficiency. And more B-frames improve random access when using long GOPs, since B-frames don't need to be decoded when jumping to a particular frame (just the previous I-frame and all P-frames between that and the current frame, plus the following P-frame for a B-Frame). I think I used 3 for these clips.
Chroma Search! A full precision chroma search can pay off for colorful content like screen shots.
In-Loop, but no Overlap filter. The Ovelap filter softens the image some, which is better than getting blocky artifacts at low bitrates with natural images, but looks weird with crisp static content like screen shots. The In-Loop filter is okay since it only kicks on when needed.
2-pass VBR. Since the complexity of screen recordings varies so much, doing an analysis pass as well as letting the codec distribute bits on its own allows for big momentary spikes for quality, while letting the average bitrate stay low.
Quality 90. Using Quality 90, the codec can dip down to QP (Quantization Parameter) 1, letting our reference frames be really crisp, reducing the bits later frames need. A good thing. In the VC-1 Encoder SDK, we can manually set the min and max QP as different parameters, but that feature was added after I did these clips.
Don't be afraid of low bitrates. Using the above, you can need a lot fewer bits/pixel for screen recordings than typical motion content.

Alex Zambelli's VC-1 FAQ

benwaggoner — Mon, 14 Jan 2008 21:09:00 GMT

Alex Zambelli has a VC-1 SDK FAQ up, saving me the trouble of doing one :).

If you have any Q's that haven't been FA'ed up there, let him know.

VC-1 Encoder SDK Professional released - now free

benwaggoner — Fri, 11 Jan 2008 18:43:00 GMT

Way back in September, we announced we were doing a VC-1 Encoder SDK. I'm happy to report that not only are we making it available, but we've decided to release the "Professional" version for free, for incorporation into compression products and tools. It's just a SDK, so it will need to be incorporated into a product to be used. Enterprising developers can find it here.

We've also combined the "Enterprise" and "Broadcast" versions of the SDK into a single library with a single license.

Professional

Professional ("ProSDK") is the baseline version of the SDK. It's a free download with a "clickthough" EULA which will allow its no-cost use in a variety of products.

Performance

The SDK includes a variety of performance improvements, both in algorithms and in use of SIMD instructions like SSE3 and SSSE3. With equivalent settings, you'll get about a 10-15% improvement in speed over the WMP 11/Vista versions of the codec ("FDSK 11") on a recent Intel or AMD processor.

Note that different options are on by default between the two, so it's possible for an encode with FSDK 11 without any registry keys turned on to be faster than an optimized one in the ProSDK, although the quality difference will be bigger.

Quality

Quality is harder to quantify, of course, and can vary quite a bit. Using the standard PSNR (Peak Signal to Noise Ratio) metric, we average about 0.5 dB better with ProSDK compared to an optimized v11 encode using all the registry keys, but it can vary quite a bit with content. The more perceptually-focused SSIM shows improvements on the order of 3-4%. Of course, compared to the stock settings without using registry keys, the improvements will be more dramatic yet.

Note that the DQuant (Differential Quantization) features in the ProSDK are different than in the registry key settings available in the v11 codec. The new DQuant is mainly targeting HD DVD and Blu-ray bitrates, and is less tuned for lower bitrates. For web bitrates, typically you'd want to have DQuant off, or on for I-frames only (making I-frames look better can dramatically improve the quality of lower-motion video, since all other frames in the GOP are based on that I-frame).

I'm just back from vacation and CES, but I'll try to get some sample encodes up demonstrating the quality improvements up next week.

Elementary Streams

The SDK natively creates a VC-1 elementary streams, in file form a ".vc1" (equivalent to a .m2v file in MPEG-2). This makes it possible to directly encode a file that can be muxed into a HD DVD or Blu-ray disc. Thus, the ProSDK will finally enable a wide variety of applications to add support for encoding VC-1 for optical discs. The VC-1 Elementary stream can also be muxed into a MPEG-2 transport stream for IPTV use.

When the SDK is used to make a Windows Media (ASF) file, it is used in conjunction with the existing Format SDK to provide audio encoding and muxing.

Enterprise

The VC-1 Encoder Enterprise SDK ("EntSDK") now incorporates the feature of what we had announced as the "Broadcast" version of the SDK in the same library, although there are different modes for the live scenarios. Enterprise will require a paid, commercial licence from the software company licensing it, in part to offset the greater help required for integrating the more advanced features. Compared to Pro, there are two big areas of improvement in Enterprise: Live and Segment-based encoding.

Live Encoding

The single biggest advantages in the new SDKs are probably in the live area. The basic improvements from Pro are certainly useful, plus the performance improvements make it possible to encode with higher complexity modes, further improving quality. But the new Enterprise features in the live encoding mode go much further.

Lookahead Rate Control

The main reason why 2-pass encoding produces better results than 1-pass encoding is that it enables the codec to "look into the future" in order to make decisions based on how the content is going to change in the future. In the FDSK 11 codec, we introduced a "Lookahead" mode, which enabled the codec to buffer ahead up to 16 frames to make frame mode decisions, like using "BI" (intra-coded B-frames) frames for flash frames, doing better scene detection, and turning off B-frames for fades to activate VC-1's intensity compensation mode. In Enterprise, this is extended to rate control itself in the "Lookahead Rate Control" mode, letting the codec do things like start to lower the data rate a bit to save bits for a coming hard frame, or to know it's safe to spend a lot of frames on an I-frame where following frames are nearly identical.

The net effect of Lookahead Rate Control is that it gives much of the quality improvements from 2-pass encoding available in live encoding, resulting in video that maintains much more consistent quality. In particular, you'll see fewer cases where a sudden change in the video results in a sudden drop in quality.

Dynamic Complexity

One challenge in live encoding, especially at higher resolutions and frame rates, is finding settings that produce optimal quality without dropping frames with the most complex scenes. This often leads to encoding using a less aggressive complexity leaving a lot of CPU headroom most of the time in order to avoid dropping frames in the most intense sections. Additionally, different encoding modes might be more efficient in different sections of the video.

Dynamic complexity address this by dynamically adjusting a number of encoding settings in real time in order to hit a specified level of CPU utilization, switching features on and off to result in the best quality for each section

Segment Reencoding

Professionally encoded VC-1 for HD DVD and Blu-ray took advantage of "segment reencoding", where encoding settings like bitrate and DQuant can be changed per scene. This enables tweaking to get transparency to the source at lower bitrates, and to speed encoding by only using the slowest modes on the most challenging scenes.

Grid encoding

And of course, once you have the ability to encode each segment individually, there's no reason to encode everything on the same machine. The Enterprise SDK includes rich support for grid encoding, where different sections of a long encode can be split up between multiple machines. And now that we have 8- and 16-core machines readily available, we can even split up a single encode across multiple instances on the same system (typically using 4 cores per encoder instance, so a 16-core machine can encode 4 15-minute sections of an hour long movie simultaneously). Unlike other popular grid encoding products, the VC-1 Encoder Enterprise SDK enables full 2-pass VBR encoding support across the nodes, letting the encoding software shift bits between segments for optimal quality and efficiency.

SSE4 support for interlaced compression

There is one performance improvement in the Ent SDK that's not in Pro SDK: SSE4 support for interlaced encoding. The new Intel "Penryn" processors will offer about another 15% improvement from our use of SSE4 instructions when encoding to the Advanced Profile's interlaced mode.

Conclusion

And so begins the new era of VC-1. Many vendors of encoding tools have been working with prerelease versions of the SDKs for a few months now, so anticipate product announcements soon. I'll keep you posted as products ship.

Today, the first shipping SDK products are Inlet's Fathom and Spinnaker.

Flip4Mac 2.2 is out, now with JavaScript Support

benwaggoner — Fri, 11 Jan 2008 06:44:40 GMT

The long-awaited version 2.2 of Flip4Mac has been released. The new version has a variety of new features. Full release notes are here, but the new playback features are:

Added JavaScript support to the browser plugins
Added support for URL Stream Scripting
Improved ASX handling
Improved support for MMS servers and live streams
Improved compatibility with many different web sites
Removed the user enabled "Create Streaming Movies" preference
Added support for manual Deactivation/Reactivation
Fixed potential security issue reported by Drew Yao where an attacker could potentially cause a security issue by deliberately malforming WMV files.

The biggie of course is support for JavaScript, which means many pages that used scripting of the Windows Media OCX component will now work with Flip4Mac. There are a variety of other compatibility impovements for other sites.
As always, provide feedback to Telestream if you find any sites that still aren't compatible with Flip4Mac 2.2.

Hotfix to FSDK 11 for multi-threaded multi-bitrate encoding

benwaggoner — Fri, 11 Jan 2008 00:49:00 GMT

We've just released a hotfix for a recently discovered issue impacting encoding performance when doing muliple bitrates ("Intelligent Streaming"). Essentially, all the encodes happen on a single core, instead of spreading out over up to 4 cores.

http://support.microsoft.com/kb/945170

Recommended for anyone encoding to multiple bitrates. It can improve performance around 4x on a 4-core system.

"DV Show" podcast with David Trescot

benwaggoner — Wed, 17 Oct 2007 22:09:00 GMT

David Trescot of Rhozet and I were guests on last week's episode of Derrick Freeman's "The DV Show." David talked about Carbon (a great compression product, soon to support our VC-1 Encoder SDK), and I answered questions about Windows Media, Silverlight, and the VC-1 Encoder SDK.

The podcast is archived here.

New WMcmd.vbs with improved multithreading and new presets

benwaggoner — Sun, 14 Oct 2007 04:05:00 GMT

Alex Zambelli has updated his invaluable WMCmd.vbs yet again. The two main new features are improved support for running multiple versions at once (great with my new 8-core Barcelona workstation), an explicit QP mode for 1-pass VBR, and refactored presets for different compression levels.

The full details are in the Readme, but here's my summary and elaboration, respectively.

For running multiple versions, the script sets the registry keys, starts the encode, and then reverts them immediately after the encode starts. This should reduce the chance of the wrong keys being set during an encode (not that I've had any problems with that in the last six months or so).

QP is a measure of how compressed the image is, with lower numbers being less compressed. For the explicit QP mode, the script now lets you specify the QP you want, instead of providing a 0-100 range and knowing the magical translation table. Why is this useful? Well, for most readers, you probably don't have an intuitive sense of what quantization parameter you want to use, but it's there for those that do. And better yet, it give you a chance to understand how QP works.

Reading the above, it's clear I need to do a blog post on QP and how to use it. That will reveal the mysteries of the subtle "Quality" control for WMV 1-pass CBR modes.

Lastly, we have preset refactoring, where Alex has cleaned up what combination of settings get applied for different targets for encode time. You can think of these as an extension of the old "Complexity" slider, applying yet more options and getting better results overall. We'll be sharing these recommendations to vendors using our VC-1 Encoder SDK. These new modes are worth some detailed discussion:

fast: Up to 1.5x faster than default with comparable quality.
-v_complexity 2
-v_bframedist 1
-v_lookahead 16
-v_loopfilter 1
-v_overlap 1

Even for the fastest mode, we don't mess with Complexity 1 (the live default in Windows Media Encoder, but very rarely needed even for live encoding on a modern system). And we can use features that help quality a lot without much CPU hit like B-Frames and Lookahead. For any Main or Advanced Profile encode, B-Frames are almost always a big plus. And Lookahead should be used for all 1-pass encodes (there's no downside to having it set for 2-pass encodes; it's ignored).

good: Up to 1.5x slower than default.
-v_complexity 3
-v_bframedist 1
-v_lookahead 16
-v_loopfilter 1

A little higher complexity, and Overlap is off. Overlap causes the image to get softer, so ideally it won't be needed. But for aggressive bitrates, it might be needed with any preset.

better: Up to 2.5x slower than default.
-v_complexity 3
-v_bframedist 1
-v_lookahead 16
-v_loopfilter 1
-v_mslevel 1
-v_msrange 0

We add Integer Chroma Search which can help a lot with animation and motion graphics, and adaptive motion search range, which helps with higher resolutions and higher motion.

best: Up to 4.5x slower than default.
-v_complexity 5
-v_bframedist 1
-v_lookahead 16
-v_loopfilter 1
-v_msrange 0

Complexity jumps from 3 to 5. MSLevel isn't specified because Complexity 5 is a little unique - it has hardcoded amounts of both chroma search and Hadamard motion match that can't otherwise be specified. The nice thing about Complexity 5 is that it can provide some of the quality gains of using registry keys for machines where those can't be set. However, it doesn't set B-frames or Lookahead, so "better" would generally look better and encode faster than a default "Complexity 5"

insane: The slowest and highest quality preset.
-v_complexity 4
-v_bframedist 1
-v_lookahead 16
-v_loopfilter 1
-v_mslevel 2
-v_msrange 0
-v_mmatch 0

And lastly, Insane. Note this goes back down to Complexity 4, which allows us to specify a Full Chroma Search and adaptive SAD/Hadamard Motion Match. This is both better and slower than Complexity 5.

And the above is what I used for most of my encodes, personally.

I'll sometimes use what I think of as "Hyper Insane" which is turning -v_numthreads down to 1, which gives a very slight further improvement. Also, 4x single-thread encodes are faster than a single 4-thread encode on the same hardware. Which is why I wind up using multiple instances so much - For a huge batch of files to encode, I'll be done slightly better running 4 simultaneous single-threaded encodes.

My Expression Encoder Training

benwaggoner — Sun, 09 Sep 2007 00:56:00 GMT

A couple of months ago the Expression Encoder team invited me to record their official training videos, produced by Total Training. They're now online for free.

It was an interesting production process. I've done a number of training titles over the years, including for Class On Demand, and this is the first time I've been able to do it in HD with digital capture. The workflow was simple - HDMI out of my Toshiba G35 laptop into a Blackmagic Design Intensity HDMI capture card. Full 1280x720 60 fps capture, straight into a NLE.

Unfortunately, I didn't get a chance to encode it myself - the cobbler's children don't have shoes and all that. It was encoded with an older version of our codec without all the usual tweaks. But you can certainly get a feel of this very exciting product. I'll try to get the source and redo it with our new VC-1 Encoder SDK - it'd look a lot better.

Here's the individual files (linked to from the Expression training videos page):

Introduction to Expression Encoder
Enhancing Media
Metadata, Markers, & Silverlight
Live Production
Advanced Expression Encoder

I haven't been on camera enough to get used to watching and in particular hearing myself - to my ear I sound like an overcaffinated Jim Henson.

UPDATE: I reencoded the clips myself, details here.

New WMV PowerToy & registry key documentation

benwaggoner — Sat, 08 Sep 2007 18:36:00 GMT

The birth of the VC-1 Encoder SDKs will reduce the need for these over the next few months, but Alex has updated his WMV PowerToy and also revised our documentation about the registry key options.

Here's the new PowerToy. It mainly removes a few options that we determined weren't in the Format SDK 11 implementation, particularly adaptive chroma search and default adaptive deadzone.

And here's the new, hopefully final registry key documentation, reflecting the above and other useful tidbits we've learned.

Silverlight, Expression Encoder 1.0 released

benwaggoner — Thu, 06 Sep 2007 23:38:00 GMT

It's been a long road, but today was the big day for Silverlight to hit the world. Scott Guthrie's Blog has got the best detailed summary of the ton of stuff that got announced today. There are four big things relevant to digital media I want to highlight before I hop on the plane for IBC. I'll be focusing on Silverlight and its scenarios on the blog for the rest of this month.

All drop by the Microsoft booth at IBC if you'd like to say hello and see some demos of both our new codec technologies and Silverlight.

Silverlight 1.0 is out!

The next generation web plugin for digital media is now a released 1.0, so non-early adopters can start installing it and content creators and developers can start authoring to it. Silverlight uses Windows Media for its digital media, so a huge library of compatible content is immediately available. And the current CDN ecosystem for hosting and streaming all works, using Windows Media Services as shipping. Silverlight contains its own implementation of the codecs inside itself, and so doesn't have any dependency on WMP or other OS features, so you'll get an identical experience wherever Silverlight is installed. The set of codecs supported in 1.0 are:

Windows Media Video 7
Windows Media Video 8
Windows Media Video 9
Windows Media Video 10 (progressive only)
Windows Media Audio "Standard" (tested with WMA 9.2 back to WMA 7 files, but the bitstream has been locked-down since WMA 2)
MP3 (in a .mp3 file)

Full details on media features are here.

Silverlight includes robust auto-update features, so support for additional codecs is certainly possible in the future. We'd appreciate any feedback on missing codecs and media features that are keeping anyone from adopting Silverlight today.

Expression Encoder 1.0 is out!

Expression Encoder (formerly Expression Media Encoder) is a new compression tool targeting Silverlight, (although its WMV files will play in all the standard players). Beyond being a great WMV encoder, its real killer feature is easy generation of a Silverlight player experience around the media asset, including subtitles/captioning, thumbnail-based visual navigation, etcetera. It's a deep product, and I'm going to be doing a much more in-depth analysis of it here soon.

One important thing to note about Expression Encoder is that it can be used to build the Silverlight player experience around an already encoded WMV file, so it can be used in conjunction with other compression tools.

Expression Encoder is based on the Windows Media Format SDK 11, so all the registry key tweaks I've been discussing on this blog work perfectly with it.

Silverlight for Linux is announced!

Good details on Miguel de Icaza's web log. In essence, we're partnering with Novell on their Moonlight implementation of Silverlight 1.0 and 1.1 (only 1.1 had been announced before). They'll be providing everything but the media codecs, which will be built by my team and provided via Microsoft.

So, Silverlight today has great reach for Windows and Mac (Intel and PowerPC), with Linux on the way.

Lots of content support!

For any new platform, the chicken-and-egg question is always paramount when discussing adoption - why should users install before there's content, and why should content publishers target it before users have installed it? That we're already compatible with a huge volume of WMV content is a big help here, of course. But nothing beats high-profile content companies targeting the format to get the installs out there (which is also a great validation of the unique value Silverlight provides).

Customers already deploying content include: MLB.com (Major League Baseball), Home Shopping Network, World Wrestling Entertainment, and the "Entertainment Tonight" show.

Silverlight is also now deployed on several Microsoft sites, including the Halo 3 preview site, including the HD version - I worked on the encoding workflow for these assets), Tafiti.com, MSN Extra, and MSN Podium '08. Silverlight will also be used in many other new and updated Microsoft properties.

Hands-on with the Temporary Insanity trailer

benwaggoner — Fri, 03 Aug 2007 07:20:00 GMT

Ancient History

Compression, although an obsession with me since I was 19, didn't appear to be a career option until many years after that. My years at Hampshire College were spent essentially majoring in neuropsychology, minoring in computer science, and spending my evening and weekends helping out my film student buddies. It all seemed hopelessly random to my parents and advisors, but turned out to be the perfect background for what I do now (after all, what's compression but extremely applied neuropsychology?).

After college and a couple of science internships under my belt I decided I didn't want to spend my life writing grant proposals or doing lab work so I started a video production company with my friends, including my recent interviewer Halstead York. The plan was to use emerging technology to be able to produce and post independent films from our own scripts. We thought we had a financing deal lined up back in 1994, and purchase a NLE: (a PowerMac 8100/80 with a Radius VideoVision card, and 4GB SledgeHammer RAID) was purchased for doing video editing. The idea was we could rent it out before and after post in order to cover some of the costs. Then there were two big problems:

The infamous defective BART chip in those early PowerMacs meant it couldn't keep sync for more than a few minutes.
Our financing fell through.

So, there we were, with a script, no money, a bunch of debt, and a NLE that couldn't edit video. However, we found a nice market using shorter clips with looser sync requirements: CD-ROM video! And so we were launched in the heady early days of multimedia. Journeyman Digital was a full service production company for digital media, and we did all the screenwriting, production, and post that we dreamed of, but not for our own projects. But we kept writing screenplays on the side. We got as far as a few meetings with Sony Pictures on one, but like nearly all screenplays, nothing really happened in the end. And while I liked doing the work, when it came down the the fundamental gut check of moving to LA and rolling the dice, I didn't NEED to do it. Instead I got married and soon enough had three little kids, and rather ran out of time for side projects.

Halstead is only recently married and currently kidless, and had time. So he and many members of the old gang dusted off one of our old screenplays, Temporary Insanity and darn if it they didn't actually shoot the whole thing in HD! Halstead just finished up the trailer. Quite an experience seeing jokes I wrote a decade ago there on the screen. And it's amazing to see how it's finally possible to make movies on a hobbyists budget, even with high-end techniques. Check out this post on color correction in the home office.

I didn't have time to work on the production itself (I was busy having that third child get born and joining Microsoft), but I certainly wasn't going to let anyone else compress the trailers (now available for download)!

The project

And so, after all that ramble, we're back to talking about hands-on compression.

Halstead had a pretty typical 2x2 matrix for encoding: two formats at two data rates each:

Formats

MPEG-4 compatible with QuickTime/AppleTV/iPod
Windows Media compatible with Windows Media Player/Flip4Mac/Xbox/Zune/Silverlight

Data rates

3 Mbps for a 720p30 HD version compatible with Xbox360/AppleTV
300 Kbps for a low data rate download, which would also be portable media player compatible (iPod for .mp4, Zune for .wmv)

Workflow

The source was provided as a 730p30 .AVI file using the CineForm Aspect HD codec. It was video-only - audio was provided in a separate .wav file.

HD WMV encoding was easy - I was able to use the source as is. And the current WMCmd.vbs supports specifying a separate .wav file as source for the audio track.

HD .MOV was harder. I wanted to use QuickTime's H.264 encoder to output, since it uses a complexity-constrained mode that is well tuned for computer playback via QuickTime, on both Intel and PPC (and there's a lot of G4 PowerBooks out there among Indie film fans). While it won't offer the same compression efficiency as a highly-tuned H.264 encoder from another encoder, it'll also playback well on more machines.

However, QuickTime, even QuickTime for Windows, can't read AVI files using the standard DirectShow API! Now that we've added support for the QuickTime API in Expression Media Encoder, it's only fair for Apple to support DirectShow now :). So, I used Rhozet Carbon to encode my .avi and .wav source files into a single Photo-JPEG compressed .MOV file that QuickTime could then read (believe it or not, there's no lossless Y'CbCr 4:2:0 encoder in QuickTime for Windows). I wound up doing that compression on my G5, so I could do it in parallel with the WMV encoding on my Windows box.

For the mobile versions, I used VirtualDub to make me a nice 320x180 version of the .AVI and Carbon again to make a 320x180 JPEG .mov.

As an alternative (and what I would have done if this was going to be a high-volume process and not just a one-off) would be to using Carbon to encode all four outputs from the single source. Also, using the "multipass" mode with Carbon and other tools other than QuickTime Player Pro itself results in very, very slow rendering time, since it reruns preprocessing for the entire clip for each pass, although only a small part of the file might be adjusted per pass. So in a high-volume workflow, probably only the 1-pass mode would have been used.

Windows Media Settings

WMV HD @ 3 Mbps:

cscript "C:\Program Files\Windows Media Components\Encoder\WMCmd.vbs" -input "G:\Temp Insanity\Trailer 1 timed v5 720.avi" -output "Trailer 1 720p 3M 192.wmv" -a_input "G:\Temp Insanity\Trailer 1.wav" -a_codec WMASTD -a_mode 4 -a_setting 128_48_2 -v_codec WVC1 -v_mode 4 -v_keydist 5 -v_bitrate 2870000 -v_peakbitrate 6000000 -v_peakbuffer 4000 -v_performance 80 -v_bframedist 1 -v_dquantoption 2 -v_loopfilter 1 -v_mmatch 0 -v_mslevel 4 -v_msrange 0 -v_percopt 2

Pretty standard stuff, with the same basic settings as my previous encodes. A few items of note:

Not excessive vertical motion and HD, so I didn't bother constraining the number of threads.
Since the source was just stereo, I used WMA instead of WMA Pro, in order to preserve Silverlight 1.0 compatibility.
Note the use of the -a_input flag to specify a different audio source.

WMV mobile @ 300 Kbps:

cscript "C:\Program Files\Windows Media Components\Encoder\WMCmd.vbs" -input "Trailer 1 timed v5 320x180.avi" -output "Trailer 1 280 Zune.wmv" -v_codec WMV9 -v_mode 4 -v_keydist 10 -v_bitrate 235000 -v_peakbitrate 600000 -v_peakbuffer 4000 -v_performance 80 -v_bframedist 1 -v_loopfilter 1 -v_overlap 1 -v_mmatch 0 -v_mslevel 2 -v_msrange 0 -v_percopt 2 -v_numthreads 1 -a_codec WMASTD -a_mode 4 -a_setting 48_44_2 -a_peakbitrate 160000

Pretty much identical to the Zune encoding settings I posted last week, except with lower data rates to hit the 300 Kbps total.

The audio was pretty simple, so 48 Kbps was enough when using VBR mode (again VBR audio is a very underused and very useful feature for downloadable files).
the data rate was so low, I went to the max and used -mslevel 2 (full floating point chroma search) and -v_numthreads 1 (single-thread encode). Even with those, this encoded much quicker than the HD version, since the frame size was so much smaller.
Main Profile is required by Zune, and thus I can't use DQuant.

QuickTime Settings

QuickTime's advanced settings aren't available via command-line, so I'll include screen shots of my MPEG-4 settings.

I matched the WMV settings as closely as appropriate.

MPEG-4 Main Profile @ 3 Mbps

The "Current" mode passes through the source frame size and frame rate (Note it would have said 1280x720 (Current) above - I had a different source loaded when I took the screen shot).
"Optimize for Download" is the equivalent of our 2-pass VBR modes. However it lacks the ability to specify a peak buffer rate or duration.
QuickTime specifies keyframe rate in terms of total frames between keyframes, not total seconds.
The "Better" mode for audio encoding quality is optimal for 16-bit sources. The "Best" mode only improves >16-bit sources
The Multi-pass mode improves quality, but can make encoding time very unpredictable. The WMV versions encoded quite a bit faster on a similar era machine (Dual 3.4 GHz "NetBurst" Xeon versus dual 2.0 GHz G5). My main compression box, a quad AMD, was busy doing some other work.
QuickTime lacks a true 2-pass VBR audio mode. For MPEG-4 exports, I only get 1-pass CBR. With a QuickTime export, I could have gotten a 1-pass VBR encode, but only in a MP3 style "range" encode, where the final file size could vary substantially. For soundtracks in downloadable files, this makes WMA a more efficient codec.
Main Profile is compatible with AppleTV, and uses B-frames. The "Extended" profile is theoretically for streaming, but it's been grayed out in QuickTime since H.264 support launched in QuickTime 7.0, and I've never seen a H.264 Extended Profile stream in the wild.

The mobile encode was the same, except with the lower video and audio data rate, and its use of the Baseline profie, required for iPod compatibility.

Differences

So, how did the two encodes come out?

For the most part, they both looked and sounded good (or at least accurate - the audio mix will be improved in a later version). The biggest difference was in flatter areas, especially with shadows. That's where the VC-1 Differential Quantization and Perceptual Optimization come in, plus the ability to use different block sizes(4x4, 4x8, 8x4, and 8x8), to better compress the edges and interiors of flat areas. The Baseline and Main Profiles of H.264 are limited to 4x4 blocks only, and H.264 doesn't have an equivalent mechanism to DQuant to compress flat areas of the image less.

Again, another H.264 encoder could have done a better job here, although at the cost of higher decode requirements, by using features like CABAC and multiple reference frames. High Profile, and hence 8x8 blocks, are not compatible with QuickTime's H.264 decoder, nor those in the AppleTV or iPod. The iPod-required Simple Profile doesn't support B-frames or CABAC.

Here's some samples from the available clips that show different levels of banding. Sorry the luma levels don't quite match - it's surprisingly difficult to get exact level screen grabs out of the QuickTime and DirectShow pipelines. If anything, these minimize the banding seeing in the clips when looking at them in QuickTime on a Mac (2.2 to 1.8 gamma correction issue?).

H.264:

VC-1:

H.264:

VC-1:

Encoding for the Zune

benwaggoner — Tue, 17 Jul 2007 01:11:00 GMT

So, we've made a couple of HD clips now - how about on the other end of the size spectrum? Let's talk about compressing for the Zune. As, usual, we'll use the Elephant's Dream source.

Zune.net has some reasonably detalied compression recommendations for targeting the Zune, and the settings are a little different than what we've seen before. But they're really targeted for consumers. Let's target the important constraints and work back for appropriate settings:

Format: Windows Media Video (.wmv)

Video codec: Windows Media Video 9 (Simple, Main, and Advanced Profiles), Windows Media Video 9 Screen, Windows Media Video 9 Image Version 2, Windows Media Video 9 VCM.

Video resolution: up to 320x240 (QVGA) or 320x180 (16:9 QVGA)

Maximum video bit rate: 500 Kbps (recommended for the best balance of optimal battery life and video quality) up to 1.5 Mbps

Video peak bit rate: up to 1.5 Mbps

Complexity or profile: Main profile, VBR

Audio codec: Windows Media Audio (.wma)

Maximum audio bit rate: WMA Standard, CBR, 128 Kbps (recommended), up to 192 Kbps, Stereo, 44.1 kHz

Maximum total bit rate: 1.692 Mbps, 1.5 Mbps for peak video plus 192 Kbps for audio

So, for Elephant's Dream, what does the above tell us? Our technical constraints to be able to sync (going beyond these would force a reencode) are:

WMV with WMV9 Main Profile and WMA "Standard"
Video at 320x180 24p
Audio as 44.1 stereo
Video with a peak rate of 1500 Kbps and audio peak of 192 Kbps

So, let's look at a couple of scenarios - Maximum quality, and maximized compression efficiency

Maximimum Efficiency

Video average 500 Kbps peak 1500 Kbps
Audio average 96 Kbps peak 192 Kbps

Maximized Quality

Video average 1000 Kbps peak 1500 Kbps
Audio 192 Kbps CBR

We could also do CBR video as well, I suppose, but that wouldn't help quality much, and would waste quite a few bits (and joules for playback).

In our previous examples, we used Windows Media Encoder session files and then WMCmd.vbs scripts. This time around, let's take the third portable example for settings files, and use .PRX files. A .PRX file defines the settings for an encode in a fashion supported by most compression tools that support the Format SDK. The "Windows Media Profile Editor" is installed along with Windows Media Encoder.

.PRX for the Maximum Efficiency Scenario:

And for Maximum Quality:

Registry Keys:

And for registry keys (set via WMV9 PowerToy, of course)? Pretty similar to last time:

We're Main Profile now, so DQuant doesn't apply. So unlike the 2 Mbps sample, I'm going to use Adaptive Deadzone to get better quality in the flat areas (before we discovered that DQuant without Perceptual was better). Since encoding at 320x180 is so fast, we can turn Chroma Search up to max. My rule of thumb is at least one thread per 64 pixels high, so we at most would want to use 2-threads. But it'll be a little more efficient to do 1.

Looking at the final results, both look and sound pretty good, with an edge to the higher bitrate, of course. More challenging content would benefit more from the higher rates, of course.

The biggest challenge in the encode is simply going from 1920x1080 down to 320x180 - that's a 36:1 reduction in pixels. The action holds up nicely, but the credits are pretty illegible due to scailng. if I was doing this content for distribution, I would have cropped the left/right more aggressively for the credits, eliminating the black bars/left right in order to keep the center of the image larger. Cropping 240 L/R and 135 T/B would do the trick, albeit eliminating some of the groovy animated graphics in the corners. But for that, I leave the proof as an exercise for the reader...

EDIT EDIT: Sorry about the bad link again - here it is fixed: Direct link to the .zip file

I'm interviewed by Jan Ozer at Streaming Media.com

benwaggoner — Thu, 21 Jun 2007 03:51:00 GMT

I've been helping out Jan Ozer from Streaming Media (and many other publications) explore the new codec and its many settings. As part of that we did this interview about VC-1, Silverlight, and Windows Media. We also talk about our forthcoming VC-1 SDKs.

And here it is.

Elephant's Dream 720p @ 2 Mbps file now posted

benwaggoner — Mon, 18 Jun 2007 20:55:00 GMT

UPDATE: Didn't work. Try this link

Sorry this took so long, but here it is:

The link from this page instead

Click to play, right-click to download.

Comments appreciated.