Entries tagged with expression encoder - Channel 10

Silverlight 2 RC0 is out

benwaggoner — Wed, 01 Oct 2008 15:01:00 GMT

The first public release canditate (RCo) for Silverlight 2 is now available for download by developers. While anyone can install it, the main goal is for developers to make sure they aren't hit by any breaking changes for Silverlight 2 RTM so they can fix any issues. No one will be auto-updated to this release. Users will be auto-updated to Silverlight 2's final release when it's posted.

All Silverlight 1.0 projects should work fine, but there may be some Silverlight 2 Beta projects that require updates.

For the media folks, there's a couple of new things I want to mention.

There's a new scaling algorithm that's a lot faster and much higher quality - a nice compromise when you can get it!
Additional optimizations have been done for the VC-1 video decoder, so playback will be faster. The gains are biggest for content using B-frames.

Since the default Expression Encoder templates have "always on" scaling and its presets use B-frames by default, the above will provide a nice performance boost for existing content. The above are "always on" features - you don't need to update anything to take advantage of them. So expect smoother frame rates on lower-end machines.

Further information:

EDIT: Added ScottGu link

Expression Encoder Service Pack 1 preview

benwaggoner — Wed, 24 Sep 2008 03:19:00 GMT

James Clarke has a blog post up describing some of the new features in the forthcoming Expression Encoder 2 Service Pack 1.

It's got some great new stuff, and we're demoing it here at Streaming Media West this week.

Read the whole post - there's lots of good stuff in there. A couple of my favories are

A new Silverlight 2 base player using .NET. Among other things, this will enable players that display the video at 100% scale by default, improving quality and performance.
New A/B compare modes. Awesome stuff for high-touch encoding and codec tweaking
Some even more VC-1 advanced options for High Codec Nerditry.
The first public release of Microsoft H.264 compression technology.

Anyway, a lot of good stuff for a SP1. I'm looking forward to it.

Low Latency webcasting with Windows Media and Siverlight

benwaggoner — Sun, 22 Jun 2008 05:25:00 GMT

So, Streaming Media is doing a special "Europe edition" of Streaming Media, and I'm doing an article about webcasting for it. I've getting a bunch of questions about how to deliver low-latency live streaming to Silverlight, and so with their permission, I'm excerpting this section on that topic.

EDIT: A correction was made; turning off Fast Cache in Windows Media Services does not have any effect on webcasting latency.

While by default, Windows Media can offer 15-20 second end-to-end delay, it's possible to drive it down to 2-3 seconds with best practices on a good network, and we're looking at what we can do to push it to below even that. The critical thing is to tune the encoder, server, and player latency together.

There's an older but more detailed "Reducing Broadcast Delay" document on this topic over at the Windows Media portal that may be worth perusing.

Low Latency Webcasting

Latency is the measure of how much time goes between when video enters the encoder and leave the user display. Latency is something that doesn’t matter at all in some markets, and matters a lot in others.

The reason we have latency is buffering, and the reason we have buffering is for quality and reliability. By having the server wait several seconds after a video stream is received before sending it out, it’s able to support more peaks and valleys in the data rate, and makes it possible for a dropped packet to be detected and resent before its needed. In the same way, buffering in the player lets it average out data rates and recover dropped packets as well.

Large buffers were extremely important in the modem era, and are still useful in many kinds of networks today. The defaults are good for delivering high quality content over a variety of networks. But when minimizing the latency is important, and reliable networks are available, the end to end delay can be reduced substantially.

Since total end-to-end latency is the sum of the encoder latency, server latency, and player latency, plus how long it takes the packets to travel between each of those, improving latency requires tweaks to the encoder, server, and player in parallel.

Encoder

On the encoder side, reducing buffer size reduces startup latency. WME and Expression Encoder both enable you to go down to one second.. Using Lookahead or Lookahead Rate Control will increase latency beyond the buffer value (typically about another half a second for LRC).

For audio, Windows Media includes the “WMA Low Delay” audio codecs from 192-64 Kbps which provide lower latency than the normal WMA modes. If you’re targeting Silverlight 1.0 or WMP 10 or earlier, you’ll want to use that for low latency. If you can require WMP 11 or Silverlight 2, you can use the lower delay yet WMA 10 Pro codec at 32-96 Kbps.

Server

WMS features like Advanced Fast Start can dramatically reduce latency for on-demand content, but don’t apply to live webcasting as the server has to wait for content to arrive from the encoder it in real time, and so can only play out at real time.

Server buffering can be turned off entirely in WMS for Windows Server 2003 and 2008, yielding a significant drop in latency.

EDIT:

The previous version of this post suggested that turning off Fast Cache would help webcasting latency. On further research, this turns out not to be the case. Disabling Fast Cache will slow down on-demand startup time, but will have no effect one way or the other for live content.

Player

Normally the player isn’t under control of the streamer. By default WMP dynamically picks an optimum buffer size based on its measurements of network and stream performance. However, it’s possible to lower the buffer size in the player’s options. This can help reduce latency when watching streams with a good connection, but could produce pauses in the video when watching video from the general Internet.

Silverlight makes the player buffer size a controllable parameter, so an optimal setting can be applied for the content, and even adjusted on the fly. This is controlled by the BufferingTime parameter in a Silverlight MediaElement.

Announcing Expression Encoder Customer Panel

benwaggoner — Sat, 21 Jun 2008 06:29:00 GMT

The Silverlight and Expression teams are very customer focused, and we like to release products on a fast cadence. We've got a lot of methods of customer outreach, and I'm happy to share a new one, the Expression Encoder Customer Panel.

David Sayed has all the details over on his blog:

Expression Encoder Customer Panel

Now that Expression Encoder 2 has been released, the team is hard at work on the next version of the product.
We want to be as responsive to community feedback as possible, and would like to set up a panel of customers with whom we can have ongoing conversations.

What is the Customer Panel?

The Customer Panel is a way for us to build strong relationships with users of our product. We'll use it as a way to understand your needs, workflows as well as how you use the product. We'll also use it to float future feature ideas and get feedback on them

What is the time requirement?

The Customer Panel is an ongoing project. The overall impact should be minimal and if at any point you no longer want to participate, just tell us. Typically we'll aim to reach out the Panel every couple of months or so. There may some times when we reach out more frequently.

How is this different to a Technology Adoption Program (TAP)?

TAPs are for specific releases and focus on getting you pre-releases of the software to play with. We'll use the same framework for the Customer Panel, but it is longer lived and goes beyond a particular release. This is an opportunity to have lasting impact on product direction.

OK sign me up!

If you want to take part, please send me an email by clicking Here it is

Sample projects

It includes a number of sample projects (quoting from the help file). These are a good jumping off point for various scenarios.

Simple encode

The simple encode sample is a console application that demonstrates how to encode a file from a console application when showing progress. To use it, just run the built application (Simple.exe) at a command prompt.

Asynchronous encoding

The WPFEncode application demonstrates how to encode a file from a UI application when displaying the progress of the encoding process. To use the application, click the Browse button to point to a file that you want to encode and then click the Encode button to encode it. When the file is encoding, a bar displays the encoding progress.

When you press the Encode button, the application creates a thread to perform the encode to make sure that the UI thread is not blocked. The progress events are called from a non-UI thread. Consequently, we use the Windows Presentation Foundation Dispatcher to marshal the updates to the UI thread.

If you try to close the application when the encoding is still occurring, the application prompts you to confirm that you want to stop the process. If you confirm, the application stops the encoding thread but delays closing the application until the thread actually stops.

MediaInfo
The MediaInfo sample is a console application that demonstrates how you can use the MediaItem class to extract media and metadata information from a media file. To use it, just run the built application (MediaInfo.exe) at a command prompt with the full path of a media file as the parameter, as shown in the following example:
Publishing plug-in

Publishing plug-ins enables you to take video and Microsoft Silverlight template assets that you have created as part of an encoding job and then do something with encoded assets. For example, you can upload the media asset by using FTP, WEBDAV, or METAWEBLOG API, publishing the asset to an asset management system or web service, or creating a data CD.

The following code sample demonstrates how to use a publishing plug-in to add the output of an encode job to a Zip file by using the Open Packaging Convention components found in Microsoft .NET 3.0.

A publishing plug-in is required to derive from the PublishPlugin class, which in turn derives from the EncoderPlugin class:

Template plug-in

The template plug-in lets you customize template settings and interact with the job through the IPluginHost interface. The plug-in also lets you add or remove files from the template output when the template is published.

The following code sample demonstrates how to add the default template parameter control to Microsoft Expression Encoder with only several lines of XAML. It also implements a custom script command creation feature that shows how the template plug-in can interact with the current job by adding a script command at the current play head position of the selected MediaItem.

Within Expression Encoder, every template receives a template plug-in. If the template doesn’t specify one, a default one will be assigned. A template plug-in derives from the TemplatePlugin class, which in turn derives from the EncoderPlugin class.

What a difference a half-decade makes! Live VC-1 today and at launch

benwaggoner — Thu, 05 Jun 2008 02:11:00 GMT

A recent conversation over at the Streaming Media Advanced list sparked a rant from me about the importance of comparing implementations of codecs, not just codecs.

To that end, I thought I'd do a demo to show how much improvement there's been in Windows Media since the launch of Windows Media 9 Series back in 2003.

A recent conversation over at the Streaming Media Advanced list sparked a rant from me about the importance of comparing implementations of codecs, not just codecs.

To that end, I thought I'd do a demo to show how much improvement there's been in Windows Media since the launch of Windows Media 9 Series back in 2003.

Below, you'll find two streams, encoded with the same settings but with tools from different eras. The streams are both

Live encoding from a preprocessed file (so that preprocessing differences don't matter)
And yes, it's the "Lady Washington" footage again.
640x360
29.97 fps
600 Kbps video
64 Kbps 44.1 stereo WMA audio
5 second buffer
Keyframe every 5 seconds

The difference is that the first is encoded with the original Windows Media Video 9 codec (ala just Windows Media Player 9 installed, like a stock Windows XP SP2 machine), and the second with the VC-1 Encoder SDK implementation in Expression Encoder 2. Pretty dramatic differences, I hope!

	Live Windows Media 2003	Live Windows Media 2008
Silverlight embedded page	2003 in Silverlight	2008 in Silverlight
Direct link to WMV (right-click to download)	2003 direct WMV link	2008 direct WMV link

...reminding me that I really need to blog how to make a Silverlight dual media player that can play two versions of the same clip in sync.

So, what makes this big difference? There's been a huge amount of work and three major releases (Format SDK 9.5 and 11, and VC-1 Encoder SDK) since then, so I won't give a complete list, but a few of the highlights.

4-way threading instead of 2-way threading, doubling performance on modern machines.
Lots of SSE2 and SSE3 optimizations to improve performance.
"Adaptive Complexity" that dynamically adjusts the complexity of the encoder, to make sure it's always using all available CPU power, without ever dropping frames. This compares to the old default live complexity of 1 (out of a 0-4 range).
Using B-frames (supported in the old decoder, but not used in the original encoder) which improve compression efficiency and enable efficient encoding of flash frames.
Lookahead Rate Control, where the codec buffers a few frames into the future, so it knows when it needs to start saving some bits for an upcoming keyframe, or when it's save to use a lot of bits on a few challenging frames.

A lot of the above are about performance. With offline encoding, better performance just makes for faster encoding. But for live encoding, it helps quality, because it allows the codec to do more math per pixel to find the optimum way to encode that file.

So, how big a difference is this? Below is a graph showing the Quantization Parameter (QP) for the two encodes. QP is a measure of how encoded each frame is, with higher values more highly compressed. In VC-1, the range is 0-31. A good rule of thumb is that QP of much below 8 looks pretty good, and QP of 8 or above...won't. Now, a live SD encode at 600 Kbps is pretty darn aggressive, so there's plenty of spots where both encodes certainly show artifacts. And quality varies a lot throughout the file as the the complexity of the video goes up and down, as this is a Constant Bitrate (CBR) encode. Note the relatively low QPs near the end of the file, where the easy credits scroll comes in.

But still, the modern VC-1 implementation (in red) with all of the above is dramatically better. While the old encoder (in blue) spikes all the way up to the maximum QP of 31, the new one is typically several QP lower, and maxes out at a QP of 20 instead.

That said, man, my Excel-fu sure has declined over the years. Hard to believe I used to teach classes on making good-looking Excel charts back in the early 90's. Anyway, just remember that lower is better, and red is our current stuff.

Videos from the "Secret Compressionist's Ball"

benwaggoner — Thu, 29 May 2008 03:39:00 GMT

I've always called my annual compressionists party something like the "Ben Waggoner Compressionist's Party" but David Sayed from the Expression Encoder team suggests the much better name of the "Secret Compressionist's Ball"

Better yet, he's got a couple of video interviews from the party! You'll see me in the background in my mildly famous purple shirt in a few shots.

First, Tony Houghton from Promoscape, talking about Silverlight.

Second, Bruce Lidl from MainConcept talking about their VC-1 codec implementation they're now licensing out, including support on Mac and Linux.

The clips are embedded at smaller than they're encoded; double-click to see them in their full glory.

Expression Suite 2 and Expression Encoder 2 now shipping!

benwaggoner — Thu, 01 May 2008 22:43:00 GMT

May 1st is a happy day for me! I went over to www.microsoft.com/expression, and was delighted to find that the Expression Studio 2 is now shipping, including the final version of Expression Encoder 2. There's a 30-day trial available for download. Boxed copies of Studio are available today at CompUSA and Office Max, with other retailers coming. There will also be the Microsoft Expression Professional Subscription for $999/year (academic and volume pricing coming soon), which includes Expression Studio along with Visual Studio, Office, Visio, XP, Vista, Virtual PC, and Parallels Desktop (to run all of the above on Mac).

As I've said before, I feel that Expression Encoder 2 is the best compression tool at fulfilling the needs of its market since Terran Interactive's Media Cleaner Pro 3.1 (so long ago I can't even find anything on the web about it!). Here are some of my favorite things about it from my obsessive compression nerd's perspective.

First desktop priced VC-1 Encoder SDK based product ($199).
And the VC-1 implementation gives you great quality by default, and gives you an advanced mode that accesses the useful features, while handling the more esoteric ones behind the scenes. For example, the rule for WMV PowerToy was use Static Motion Vector Cost if using P-frame DQuant, otherwise use Dynamic. So Expression Encoder 2 just has a DQuant setting, and uses the right Motion Vector Cost based on that.
High quality scaling and deinterlacing modes
WMA Pro support, targeting Silverlight 2's support for that. So danceable music at 48 Kbps.
"Smart by default" - you can use the same preset and feed it a PAL 16:9 interlaced file and a NTSC 4:3 progressive file, and it'll do the right thing without having to do a single click: you'd get a 640x360 deinterlaced 25p file and a 640x480 29.97 file.
Which means you don't have a huge combination of settings to deal with all the video formats. Instead, the presets are scenario based; you just need one "1 Mbps streaming" option that'll handle all the video file variants.
Great multi-clip templates, so you can make a video gallery, with each clip having its own optional thumbnail-based chapter navigation (hmm, that deserves a nice demo).
Easy creation of thumbnail-based chapter navigation!
Captioning and metadata support, including ISAN.
Available soon, an integrated publishing module to go straight to Silverlight Streaming.
Support for a wide variety of source formats, including QuickTime, AVI, Windows Media, VOB (DVD), many other flavors of MPEG-2, MPEG-4, and AVISynth.
Automation available via both command-line and a .NET object model.
Simple editing features, like in/out points, preroll/postroll videos, and image and animated overlays.
GPU acceleration for previews
A great A/B compare feature, enabling short sections of the file to be encoded with a variety of compression settings, and then compared (playing in real time! zoomed in!, with an A/B split screen slider!) to both the source and each other.
An unbelievably cool live encoding module, which supports multiple cameras with live switching, streaming from files, including looping, live metadata insertion, and big quality improvements from lookahead rate control and dynamic complexity. I owe you a blog post describing that as well.

So, in summary, I guess I'd say I like it.

I've had a some blog posts highlighting different projects done with Expression Encoder 2 which you can follow along with.

Hands on with high-touch encoding: Streaming Media All-Stars Redo

Encoding screen recordings for Silverlight in VC-1 with Expression Encoder 2

"What Happens in Vegas" - 720p Movie trailer at 2 Mbps via Silverlight Streaming

I've got other information about Expression Encoder and how it fits into Silverlight at

My NAB Presentations

Silverlight Media technologies overview in Expression newsletter

And, of course I'll be covering Expression Encoder 2 as one of the products we cover in my

My June 23-27th class at Stanford

Hands on with high-touch encoding: Streaming Media All-Stars Redo

benwaggoner — Wed, 30 Apr 2008 06:47:00 GMT

Introduction

As Expression Encoder 2 approaches its immenent release, I've been using it for more and more real-world projects. This recent one was particularly chewy fun, and I thought it would make a good tutorial for a high-touch workflow.

As you may remember from a few weeks ago, I was one of the inaugural class of Streaming Media's Streaming Media All-Stars. There was a fun video montage of all of us on baseball cards being announce by ballpark-style narration. Good stuff, but the FLV compression wasn't quite up to my standards for this rare intersection of compression obsession and personal vanity. So I contacted Streaming Media and asked if I could take my own whack at it.

I'll have an expanded version of this post as an article in an upcoming issue of Streaming Media Magazine. If you don't get it, you can sign up for a free subscription.

The Source

One thing I noticed in the original is that the background graphics and a few of the animations were interlaced, as you can see in the last "before" image at the very bottom of the page.

While deinterlacing it may have been possible, the heavyweight motion-adaptive deinterlacers available for technologies like AVISynth can be finicky to configure, and extremely slow. And in the end, nothing beats getting the source fixed in the first place. Compression is the art of getting output that's as close to the original as possible with the bits you have available; often getting access to higher quality sources can provide a much bigger improvement to final quality than all the codec tweaking in the world.

So, I contacted the post house, and they fixed the background interlacing (it was just a matter of properly flagging the source as interlaced in After Effects) and re-rendered it for me as a lossless RGB PNG codec QuickTime .mov file. However, there were two shots that snuck through where one layer was still interlaced. I didn't want to wait for another disc, so I dived into After Effects (in the end, all difficult preprocessing jobs seem to wind up in After Effects). I used the "Reduce Interlace Filter" with a softness of 1 to blend the two fields together. Traditional deinterlace methods messed up the text on the cards too much. However, the softness increase from that filter wound up causing a slight visual discontinuity when it kicked in. So, I broke out the two shots with interlacing into layers, and then used a five-frame cross-dissolve transition from the original progressive frames to the start of the interlaced shot which hid the slight loss of focus (masked in part by the motion). Both interlaced shots ended on a hard cut, so I was able to switch back to the original video without a transition.

I then rendered the new version out from After Effects in 32-bit float (to reduce the risk of introducing banding via an 8-bit to 8-bit conversion) into the Lagarith codec in YV12 mode, which uses the native 8-bit 4:2:0 colorspace of VC-1 and other codecs. This means that Expression Encoder doesn't need to do any color space conversion, making compression slightly faster.

Markers

The other notable issue with the original clip was "keyframe popping"; when an obvious "jump" in the video happens at the keyframe rate of the video. Watch the original FLV, and you'll see it during any of the longer static shots. Since the whole section with the cards is one single long shot over 3 minutes long without any hard cuts, there wasn't a place for natural keyframes (automatically inserted at a hard cut) to go. Thus keyframe transitions would happen while the cards were otherwise static, making even a slight change visible.

I also wanted to show off the Expression Encoder templates a bit by doing thumbnail navigation. In EEv2, I'm able to graphically set markers on particular frames, and set them to be keyframes and/or thumbnails. A thumbnail becomes an image file which, with the supported templates, automatically gets included in the menus for navigation (think a chapter on a DVD). Normally you also want to make the chapter points keyframes, since keyframes support immediate random access, as no other frames need to be decoded before displaying a keyframe.

This was an opportunity to kill two birds with one stone; if I set the markers on the first static frame of every card, it'd be nice high quality image that all the later frames that reference that I-frame can be based on, propagating its quality forward. If I set my keyframe spacing long enough, there wouldn't be any other keyframes in that interval to cause keyframe popping, and so the static card would be very consistent.

So, I set a marker for each person, flagged to be both a thumbnail and a keyframe. The audio doesn't always sync up exactly so that the person's name begins after their card is down, so sometimes the first name is cut off. This would have been easy to fix by just delaying the audio a second.

You can also use non-thumbnail keyframe markers; these become keyframes without showing up in navigation. I stuck a few of those in as well in the intro/outro sections, on the first full frames after the logo gets built. Since the sponsor pays the bills (Ripcode in this case), I always want to make sure that logos remain nice and crisp.

Setting keyframes has been around in compression projects for ages now; I did a lot of this in Premiere 4.0 for Cinepak encodes in the pre-Media Cleaner days, since Cinepak was prone to keyframe popping issues. Modern codecs like VC-1 do a much better job of finding good natural keyframes, and also to reduce popping issues. The Silverlight version would have looked a lot better than the Flash even if I hadn't set them, but they did get a further boost in quality. But don't think this is something you should be doing in every case; this clip is unusual in having minutes without cuts with a mix of static and moving elements, at an extremely low bitrate.

Encoding Settings

Now, what encoding settings do we want to use?

Video

Frame Rate: Source. We want to capture all the motion in the source perfectly (29.97 frames per second in this case).
Keyframe Interval: 15 seconds. The longest gap between markers in the source is a hair left than 15 seconds, so this will prevent keyframe popping between cards.
Profile: VC-1 Advanced Profile. So we can use DQuant, as discussed below.
Mode: VBR Peak Constrained. This is a progressive download project, so VBR Peak Constrained gives us optimum quality.
Bitrate (average): 488 Kbps. Matching the original FLV's actual bitrate (400 Kbps was requested, but VP6 overshot by over 20%).
Peak bitrate: 896 Kbps. So video + audio + overhead (9 Kbps in this case) max bitrate is a consumer broadband friendly 1000 Kbps total.
Peak Buffer Size: 15 seconds. So the buffer duration can contain an entire Group of Pictures (a keyframe and frames that reference it).
Width and Height: 640x480. Same as source. The original project had both 320x240 and 640x480, but they used the same data rate, so I'm doing just the 640x480 and Silverlight embed can be set to the desired size.

Audio

Codec: WMA. We're targeting Silverlight 1.0 compatibility, so WMA Pro isn't an option
Mode: VBR. Always better quality for progressive download.
Bitrate: 48 Kbps. Matching the data rate of the FLV source. Also this is minimum bitrate for WMA VBR. I always try to use at least 48 Kbps for WMA progressive for that reason; it's a massive quality jump from 32 Kbps CBR for typical content.
Sample Rate: 44.1 KHz. Same as source. Also, 44.1 is the native audio rendering mode for Silverlight, and so offers the same quality and better performance versus 48 KHz.
Bits per Sample :16. The only option for WMA
Channels: Stereo. VBR audio requires stereo. I'd use mono if I needed to do 32 Kbps, since there isn't stereo separation important to the experience here.
Audio peak bitrate: 96 Kbps. Again, so total peak comes out as 1000 Kbps. The audio isn't that difficult or variable, so higher likely wouldn't sound any different.
Audio peak buffer size: 1.5 Kbps. The default is almost always fine.

Advanced Codec Settings

Video Complexity: Best (5). It's a short clip at a reasonable frame size. Complexity 3 probably would have been just as good, but the encode only takes about 12 minutes at 5, so I didn't bother doing anything less than the max (I love my new 8-core workstation!).

Perceptual Optimizations

Adaptive Dead Zone: Conservative. The normal default. It softens out edges that might ring or get too blocky, but not by too much. I tried both Off and Aggressive, and Conservative definitely looked the best, as usual.
DQuant: I-Frames Only. There aren't many I-frames (mainly the few dozen we set manually, and perhaps a few more natural ones), but they contain the important visual data of the faces on the cards, so we want them to be as high quality as possible. DQuant spends too many bits on smooth parts of the image to use on every frame, but upping the bitrates on a few dozen I-frames won't hurt quality much, and improves the quality of the static parts of the card we wind up staring at for those many seconds.

Filters

In-Loop: On. Always on unless using Simple Profile; it helps reduce artifacts and improve quality, particularly at these aggressive bitrates
Overlap: On. Further hides artifacts, which are a challenge with motion graphics at such a low bitrate
Denoise: Off. The source doesn't have a hint of noise. If there were a lot of textures, Denoise can help to soften them some for easier encoding, but there's not much texture either.
Noise Edge Removal: Off. This is really only useful for noisy edges of analog captures, and even then we're better off cropping. It obviously doesn't apply here.

Group of Pictures

B-Frame Number: 2. Normally we use 1 for film and video sources, but for this kind of motion graphics, 2 is more efficient. This gives us an IBBP pattern, so each B-frame is adjacent to a P-frame. 3 B-frame is less efficient in this case, probably since with the IBBBP pattern the middle B-frame is two frames away from a reference frame (only I and P frames can be reference frames), and the P-frames are too far apart and so require more bits to store the change over four frames instead of three since the previous I or P frame. Using 2 also gives us better random access than 1, since worse-case random access time is based on the maximum number of P-frames between I-frames. With 15 seconds between keyframes at 30 fps, that gives us 450 frames per GOP maximum. With 2 B-frames, that'll gives us 149 P-frames per GOP, the same (and thus the same random access) as if we had a 5 second GOP without P-frames (the old Windows Media Encoder default).
Scene Change Detection: On. This will give us natural keyframes where need them. The codec seems to do a good job of putting them in the right place. I've never changed this in EE.
Adaptive GOP: On. Always have this on.
Closed GOP: Off. This is required to be on for CBR encodes in EE, but slightly reduces quality with VBR encoding. In particular, it can increase keyframe popping, since an Open GOP pattern starts with B-frames before the first keyframe/I-frame, you get BBIBBPBBP..., with the B-frames able to reference the last P-frame of the previous GOP. This helps smooth over changes between GOPs, since you have the leading B-frame(s) to spread the change over.

Motion Estimation

Chroma Search: Full True Chroma. Motion Graphics is a canonical time we want chroma search. The encode is so fast, there's no reason to not go for the full meal deal and do Full True Chroma.
Match Method: SAD. For this kind of content with very simple, flat areas, the Sum of Absolute Differences Motion Match is actually both higher quality and faster than either Hadamard or my normal video/film default of Adaptive.
Search Range: Adaptive. The smallest range works for most of the frames, but there's some very fast motion when the cards zoom in which need the bigger range. Adaptive it is.

Output

The Output pane has some of my favorite usability features of Expression Encoder, letting us apply rich templates and automatic publishing.

First, the Template. I picked the "Clean" template, which has a nice subtle overlay control, and a popup navigation via the thumbnails we made above when you mouse over the top of window. It also supports going full screen with a double-click. One thing I like about Clean is that the video fills the frame exactly, without having to account for the control bar or other elements. So I can embed at exactly 640x480 for a 640x480 clip.

The publish mode (I've got the optional Silverlight Streaming publishing plugin installed) lets me automatically or manually upload the final project to our Silverlight Streaming service. This is a great way to test or deliver Silverlight projects. You can sign up for a free account with 10 GB of storage and 5 TB/month of bandwidth.

Before/After

So, how much did all this help? Here's a couple of the more pronounced before/after shots. All the below are inserted as 100% scale PNG, so there's no scaling or further compression to complicate the comparison. Note that the FLV came out darker for some reason. I'm not sure what the cause of that was; the VC-1 brightness matches the source. Perhaps something to do with the Mac/Windows gamma difference on the platform the FLV was encoded on? This actually makes VC-1's job relatively harder, since the motion graphics are easier to see.

And you can see the actual clips in action here:

Before: FLV VP6 in Flash

After: WMV VC-1 in Silverlight

Detail improvements

I grabbed a frame right after the transition that really shows the detail difference between VP6 and VC-1 here; it's especially striking in the texture of the shirt. The VP6 gets sharper after a keyframe pop, but this is how it starts. VC-1 quality in the card is maintained perfectly throughout.

FLV VP6:

WMV VC-1:

Deinterlacing improvements

In this frame (man, do I look like a stiff!), you can see the effect of my blend deinterlace to hide the fields. Notice the ringing artifacts in the original frame. Encoding fields as progressive is extremely challenging for codecs, since you have high motion 1-pixel high horizontal lines, combing high frequency and high detail. I normally don't like doing a blend, since those double-images are also hard to encode, but it was only for a very short duration in this clip, and the deinterlacing filters I had handy had a lot of trouble preserving the text perfectly.

FLV VP6:

WMV VC-1:

An exhaused me talks to James Clarke about Expression Encoder 2

benwaggoner — Wed, 09 Apr 2008 21:23:00 GMT

Oops, forgot to post this video clip of a VERY exhaused me at Mix talking to Expression Encoder's James Clarke about 2.0. The clip itself is actually quarter HD (960x540), so double-click on it to take it full screen.

Could be a good case study for the captioning features in Silverlight...

Tutorial: Encoding screen recordings for Silverlight in VC-1 with Expression Encoder 2

benwaggoner — Thu, 27 Mar 2008 23:55:00 GMT

One of the best parts of my job at Microsoft is when I can put aside the video strategy stuff and do some real-world hands-on video compression encoding for a project. My friends on the IIS team asked me to encode their new tutorials for Silverlight playback, and I thought it was a great project to illustrate the screen encoding tips I talked about a few weeks ago.

As mentioned a few weeks back, Silverlight 1.0 and 2 only support the Windows Media Video 7, 8, and 9 (aka VC-1) for video codecs. We don't support the older Windows Media Video 7 and 9 Screen codecs. This is a fine thing from my perspective; it makes the install size of Silverlight smaller, and we can get better results with our current VC-1 implementation than we can out of the screen codecs. This is because a modern OS like Vista's Aero Glass or Mac OS X 10.5 using a lot of gradients and transparencies that older screen codecs don't handle efficiently, but matches much more closely the kind of video image that VC-1 is designed for.

So, using the beta of Expression Encoder 2, which incorporates the new VC-1 Encoder SDK, let me show a real world project delivering in VC-1 for screen captures.

Goal

The job was to provide a series of source clips demonstrating common tasks in the new IIS 7. Previous screen recordings the team had done used the Windows Media Video 9 Screen and Windows Media Audio 9 Voice codecs with a total bitrate of 500 Kbps for 1024x768, 5 frames per second. There were apparent artifacts in both video and audio, although the content itself was comprehensible. I wanted to reduce the total bitrate to 400 Kbps, while tripling the frame rate to 15 fps and largely eliminating apparent video or audio issues.

Additionally, I also wanted to make files with specs to stream off Silverlight Streaming, which recommends a max peak bitrate of 1400 Kbps. So the total of my peak of video and audio needed to be no more than 1400.

Source

The source had been recorded in Techsmith's Camtasia Studio product, which captures screen activity live to an .AVI file using their lossless video codec. Camtasia does a great job of this kind of screen recording; something like the HDMI to HD SDI I used for my previous Expression Encoder 1.0 training would have been serious overkill for this low-motion lower resolution content, and forced an extra color conversion step.

The tech spec for all the files was:

Video: 1024x768 15 fps
Audio 44.1 KHz 16-bit stereo

Encoding Settings

Video Settings

Frame Rate: Source. VC-1 is extremely efficient, so we can increase the frame rate from the typical 5 to the full 15 that were originally captured
Key frame interval: 20. This is an unusually high setting, but critical to keeping our bitrate down. Since screen recordings often have long sequences without any dramatic changes in the video, it's pretty common for the B and P frames to be tiny, and I-frames to make up the majority of the total bandwidth. So if you wind up with too frequent I-frames, they wind up spending a ton of bits repeating the same static parts of the frame leaving the codec unable to spend those bits on other parts of the image. The normal drawback of long gaps between I-frames is slow random access. However, random access is really a matter of how many P-frames there are between I-frames (as B-frames can be skipped during decoding since no frame references them). Thus, increasing the number of B-frames between P-frames improves random access. Since we'll be using 4 B-frames as you'll see below, only 1 out of 5 frames between I-frames is a P-frame, giving us a max of 60 P-frames between I-frames (15 fps, of which 3 can be P-frames, over 20 seconds between I-frames). So, we'll have about the same random access performance as if we'd encoded at 30 fps with the standard 1 B-frame and a max 4-second keyframe interval (30 fps, of which 15 can be P-frames, over 4 seconds between I-frames)
Profile: VC-1 Advanced Profile, so we can use the I-frame DQuant feature below. For Silverlight 1 (which is progressive-scan only) the lack of I-frame DQuant is the only disadvantage to Main Profile compared to Advanced Profile.
Mode: VBR peak constrained, so we can specify both an average bitrate (to control file size) and a peak (to make sure it fits within the Silverlight Streaming 1400 Kbps maximum). VBR peak constrained is always a 2-pass encoding process, which we also want in order for the codec to be able to do optimal bitrate distribution over this file with highly variable complexity
Bitrate (Average): 350 Kbps, leaving us with 50 Kbps to use on audio.
Peak Bitrate: 1300 Kbps, leaving another 100 for audio peak.
Buffer Size: 5 I stuck with the default, which is fine for VBR at this bitrate. Bigger would give the codec a little more flexibility to move bits around, but could make playback of the web a little more touchy on slower speed connections.
Width and Height: 1024x768, matching the source.

Audio Settings

Codec: WMA. While Silverlight 2 adds support for Windows Media Audio 10 Professional, it isn't supported in Silverlight 1.0, which we wanted to use for this demo. We'll stick with good old WMA for maximum backwards compatibility.
Mode: VBR. Again, so the codec will distribute bits optimally throughout the piece, savings bits from pauses and spending them on harder bits of content
Bitrate: 48 Kbps. This is the lowest supported bitrate for WMA in VBR mode. I could go lower with CBR, but there's often some high-frequency artifacts in WMA CBR @ 32 Kbps and below for voice I find annoying, so I'd rather have overkill with VBR @ 48 Kbps.
Sample Rate: 44.1 KHz. Silverlight's internal sound engine runs at 44.1, so I recommend encoding audio to that to avoid an unneeded sample rate conversion. In this case, it also matches the source.
Bits per sample: 16, the only option with WMA. I'd use it anyway, as it matches the source.
Channels: Stereo, the only option with VBR WMA. WMA will intelligently encode the audio only once when it's identical in both channels, so it's safe to encode a mainly mono mix like this as stereo without a risk of inefficiency. The source in this case is nominally stereo, but is a mono mix.
Audio Peak Bitrate: 96, to add to the 1300 for video and to keep us under the 1400 Kbps max for Silverlight Streaming. That's plenty for voice content.
Audio Peak Buffer Size: 1.5. This default is nearly always fine.

Advanced Codec Settings

Video Complexity: Normal (3). The default is just fine for simple motion like in screen recordings. Higher values are mainly useful with lots of differing motion in fine details, like with film grain or particle effects. I probably could have gotten away with lower without much drop in quality for this content.

Perceptual Options

Adaptive Deadzone: Off. This is good for preserving some coarse texture like film grain, but we don't have any textures we want to preserve here - it's pretty much flat areas, gradients, and fine details like font edges.
DQuant: I-Frames Only. DQuant is short for Differential Quantization, where the codec is able to vary the degree of compression (quantization) per macroblock (16x16 block of pixels) in the frame. The DQuant implementation in the VC-1 Encoder SDK used in Expression Encoder 2 looks for areas of smoother texture and then compresses them less. This implementation is much more aggressive than the one that shipped with Format SDK 11, and isn't appropriate for most low-bitrate encoding. But for screen captures, using it's just for I-frames (which are only 1 our of 60, as we determined above) can improve the quality of the I-frames without taking too many bits away from the other frames. And by establishing a very clean reference frame, the following frames based on the I-frame, or based on a frame based on the I-frame, start with a near-perfect copy of the screen image to start from. This reduces the common effect in older codecs where the image can be soft or blocky after a scene change, with the quality improving over the next few frames even though the original image didn't have that change.

Filters

In-Loop: On. The In-Loop deblocking filter softens areas where a compression artifact would otherwise be visible, and then predicts future frames on that improved version. This always helps quality at Silverlight bitrates, and I recommend it always be on as long as a low-powered device like a cellphone isn't being targeted; it does slightly increase CPU requirements for playback.
Overlap: On. The Overlap filter further softens potential artifacts. Since Silverlight doesn't have the postprocessing modes of Windows Media Player, the overlap filter is good to have on at typical Silverlight bitates. It's more of a brute,force filter than the In-Loop Filter, and can soften the image a bit at high bitrates.
Denoise: Off. Source isn't noisy.
Noise Edge Removal: Off. No noisy edges

Group of Pictures

B-Frame Number: 4. We get two things out of using this instead of the normal 1 with screen recordings. First, it helps improve compression efficiency, given the very simple motion in screen recordings. A B-frame can be based on the previous and/or next I- and P-frame, but not another B-frame. With content like film or video with some random noise in them, too many B-frames hurt quality since a B-frame can be so temporally separate from its reference frames. But a Camtasia screen-record is pixel-perfect, without any random noise. So we actually get an improvement in efficiency. Also, the greater number of B-frames lets us push up the interval between keyframes without hurting latency (as mentioned above), further improving efficiency. Going from a keyframe every 5 and 1 B-frame to a keyframe ever 20 and 4 B-frame, I was able to get better quality at 350 Kbps than I was getting at 600 Kbps before.
Scene Change Detection: Always have this on. It will automatically insert an I-frame at cuts, improving compression efficiency and random access.
Adaptive GOP: On: Always have this on. It tells the codec not to insert I-frames at regular intervals as defined by "Keyframe every" but just treat that as a maximum distance between GOPs. This helps efficiency quite a bit.
Closed GOP: No. Always have this off. Closed GOP makes editing easier (which we're not going to do) but hurts efficiency slightly.

Motion Estimation

Chroma Search: Full True Chroma. Not normally needed with screen captures, but helpful in this case as the recordings were done with ClearType on. See the previous blog post about ClearType why that's a potential problem.
Motion Method: SAD. The Sum of Absolute Differences is quite a bit faster than the alternate Hadamard or Adaptive modes, and perfectly good for screen recordings without any noise.
Search Range: Adaptive. Sometimes those dialog boxes can go pretty fast. And with 4 B-frames, each P-frame has to go back a 1/3rd of a second to the previous P or I-frame for reference. An adaptive motion search range makes sure it'll find the match if it's there.

The Results

And here's the final files, embedded in Silverlight up at IIS.net. Remember to double-click on the video windwo to go full screen and enjoy their full glory. Beyond being a compression demo, they're pretty darn useful demos of common IIS7 activities. There will be a few more files uploaded in the next few weeks, and I'll update this post to include those.

Installing Necessary IIS7 Components on Windows Vista

Install only the components you need for your Web applications by leveraging IIS7’s modular architecture. This tutorial will cover installing the modules necessary for serving ASP and ASP.NET pages from IIS7 in Windows Vista.

Serving New Content

More flexible deployment options let you decide exactly how you want your Web content served by IIS7. This tutorial will cover creating your first Web site, Web application and Virtual Directory through the new IIS Manager graphical-user-interface.

Editing Configuration Files

Strongly typed schema written in clear-text XML makes IIS7 configuration files simple to read and edit. This tutorial covers reading and setting configuration in ApplicationHost.config at the server level and Web.config files at the site and application level.

Troubleshooting Unexpected Issues

Prescriptive detailed errors, automatic failure tracing and more exposed runtime information make IIS7 the simplest and quickest Web server to troubleshoot. This tutorial will cover debugging site and application failures with the advanced diagnostic features in IIS7.

Setting Up FastCGI for PHP

Improved performance and greater reliability for PHP applications is ensured by the new FastCGI component for IIS7 and previous versions. This tutorial will cover installing PHP 5.2.1 and the new FastCGI component to IIS7 in Windows Vista.

Delegating Configuration to web.config Files

Distributed, file-based configuration is a powerful new feature of IIS7 that enables delegated management of Web application settings at a very granular level. This tutorial will cover the structure of IIS and ASP.NET configuration, unlocking IIS configuration for delegation, creating and setting configuration in Web.config files and using location tags.

Using ASP.NET Forms Authentication

HTTP request processing is more integrated in IIS7 allowing ASP.NET features like Forms Authentication to process requests for non-ASP.NET content like ASP, PHP or media files. This tutorial will cover configuring authentication settings in Web.config, adding users and roles to membership, and configuring authentication for all content types in Integrated Pipeline Mode.

Configuring SSL in IIS Manager

Enabling powerful SSL security to protect your Web applications is simpler to setup with IIS Manager and easier to deploy with self-signed certificates in IIS7. This tutorial will cover adding self signed certificates, creating certificates with a Certificate Authority and setting up HTTPS bindings.

Extending Web server Functionality in .NET

Building Web server add-ons and extensions is simpler and less time-consuming because IIS7 supports .NET extensibility through the IHTTPModule and IHTTPHandler interfaces that ASP.NET developers already know and use today. This tutorial will cover building a .NET module starting with the Managed Module Kit, implementing the IHTTPModule interface, attaching EventHandlers to pipeline events and configuring IIS7 to use the module in the request pipeline.

Improving Performance with Native Output Caching

Dramatically reduce Web application response time by leveraging native HttpCacheModule in IIS7 that stores all application outputs in Kernel mode cache. This tutorial will cover enabling and configuring user-mode and kernel-mode caching by creating new output caching rules in config and through the IIS Manager GUI.

Tip: ClearType off for screen recordings

benwaggoner — Thu, 27 Mar 2008 05:37:00 GMT

I've been playing around a bunch with screen recordings lately (as you'll see in my next blog post), and I've noticed a pretty common problem: people leaving ClearType on while recording. ClearType is cool stuff, adding improved anti-aliasing on LCD displays. But the way it works is by taking advantage of the little sub-pixel strips of Red, Green, and Blue each LCD pixel is made of. This improves detail and readability. But, as you can see below, when you zoom in you get color fringing on that small text.

Those little color fringes make the video somewhat harder to encode, since they change with the position of the text. And they also look wrong when then video is played back on a non-LCD display, and really wrong on an oddball LCD display which has a different pattern of the colored strips.

ClearType Font Smoothing @ 100% Zoom

ClearType Font Smoothing @ 800% Zoom

Standard Font Smoothing @ 100% Zoom:

Standard Font Smoothing @ 800% Zoom:

When doing any screen captures, be it Camtasia or via a HDMI to SDI bridge like I used for my Expression Encoder 1.0 training, make sure you have ClearType off. In Vista, you do that via the Appearance Settings control panel's Effects button.

If you wind up with source video that has been recorded with ClearType on, turning on Chroma Search can help quality some by enabling the code to pick up on all hue/saturation changes on text as it moves around. I recommend using the "Full True Chroma" mode if using a tool based on the VC-1 Encoder SDK, like Expression Encoder 2.0.

My "Encoding Video for Microsoft Silverlight" presentation from MIX is up

benwaggoner — Thu, 06 Mar 2008 16:10:00 GMT

And here it is. You can watch in Silverlight, or download in WMV or for iPod or Zune.

VC-1 for screen recordings (New Expression Encoder encodes up)

benwaggoner — Tue, 15 Jan 2008 16:02:00 GMT

A few months ago I posted links to my Expression Encoder training. However, I didn't encode the original clips, and being a little obsessive I wanted to see wha I could pull off myself with VC-1 for screen shots.

You can see the results of my work here.

Some tips for encoding screen shots:

I-Frame DQuant. While I normally don't recommend DQuant for low bitrates with the VC-1 Encoder SDK, it makes sense to apply it to I-frames for screen recordings, since we don't need very many of them, and with so little motion, that really helps improve the quality of all frames based on the reference frame.
Long GOPs (distance between keyframes) can really help efficiency, since keyframes can take up the majority of bits in the files.
Use B-frames. >1 can pay off a lot in improved efficiency. And more B-frames improve random access when using long GOPs, since B-frames don't need to be decoded when jumping to a particular frame (just the previous I-frame and all P-frames between that and the current frame, plus the following P-frame for a B-Frame). I think I used 3 for these clips.
Chroma Search! A full precision chroma search can pay off for colorful content like screen shots.
In-Loop, but no Overlap filter. The Ovelap filter softens the image some, which is better than getting blocky artifacts at low bitrates with natural images, but looks weird with crisp static content like screen shots. The In-Loop filter is okay since it only kicks on when needed.
2-pass VBR. Since the complexity of screen recordings varies so much, doing an analysis pass as well as letting the codec distribute bits on its own allows for big momentary spikes for quality, while letting the average bitrate stay low.
Quality 90. Using Quality 90, the codec can dip down to QP (Quantization Parameter) 1, letting our reference frames be really crisp, reducing the bits later frames need. A good thing. In the VC-1 Encoder SDK, we can manually set the min and max QP as different parameters, but that feature was added after I did these clips.
Don't be afraid of low bitrates. Using the above, you can need a lot fewer bits/pixel for screen recordings than typical motion content.

Compression for Silverlight Webcast posted

benwaggoner — Tue, 11 Dec 2007 06:44:00 GMT

The "Compression with Silverlight" Live Meeting I did with Michael Scherotter has now been posted.

Check it out here at the Synergist blog.

My Expression Encoder Training

benwaggoner — Sun, 09 Sep 2007 00:56:00 GMT

A couple of months ago the Expression Encoder team invited me to record their official training videos, produced by Total Training. They're now online for free.

It was an interesting production process. I've done a number of training titles over the years, including for Class On Demand, and this is the first time I've been able to do it in HD with digital capture. The workflow was simple - HDMI out of my Toshiba G35 laptop into a Blackmagic Design Intensity HDMI capture card. Full 1280x720 60 fps capture, straight into a NLE.

Unfortunately, I didn't get a chance to encode it myself - the cobbler's children don't have shoes and all that. It was encoded with an older version of our codec without all the usual tweaks. But you can certainly get a feel of this very exciting product. I'll try to get the source and redo it with our new VC-1 Encoder SDK - it'd look a lot better.

Here's the individual files (linked to from the Expression training videos page):

Introduction to Expression Encoder
Enhancing Media
Metadata, Markers, & Silverlight
Live Production
Advanced Expression Encoder

I haven't been on camera enough to get used to watching and in particular hearing myself - to my ear I sound like an overcaffinated Jim Henson.

UPDATE: I reencoded the clips myself, details here.

Silverlight, Expression Encoder 1.0 released

benwaggoner — Thu, 06 Sep 2007 23:38:00 GMT

It's been a long road, but today was the big day for Silverlight to hit the world. Scott Guthrie's Blog has got the best detailed summary of the ton of stuff that got announced today. There are four big things relevant to digital media I want to highlight before I hop on the plane for IBC. I'll be focusing on Silverlight and its scenarios on the blog for the rest of this month.

All drop by the Microsoft booth at IBC if you'd like to say hello and see some demos of both our new codec technologies and Silverlight.

Silverlight 1.0 is out!

The next generation web plugin for digital media is now a released 1.0, so non-early adopters can start installing it and content creators and developers can start authoring to it. Silverlight uses Windows Media for its digital media, so a huge library of compatible content is immediately available. And the current CDN ecosystem for hosting and streaming all works, using Windows Media Services as shipping. Silverlight contains its own implementation of the codecs inside itself, and so doesn't have any dependency on WMP or other OS features, so you'll get an identical experience wherever Silverlight is installed. The set of codecs supported in 1.0 are:

Windows Media Video 7
Windows Media Video 8
Windows Media Video 9
Windows Media Video 10 (progressive only)
Windows Media Audio "Standard" (tested with WMA 9.2 back to WMA 7 files, but the bitstream has been locked-down since WMA 2)
MP3 (in a .mp3 file)

Full details on media features are here.

Silverlight includes robust auto-update features, so support for additional codecs is certainly possible in the future. We'd appreciate any feedback on missing codecs and media features that are keeping anyone from adopting Silverlight today.

Expression Encoder 1.0 is out!

Expression Encoder (formerly Expression Media Encoder) is a new compression tool targeting Silverlight, (although its WMV files will play in all the standard players). Beyond being a great WMV encoder, its real killer feature is easy generation of a Silverlight player experience around the media asset, including subtitles/captioning, thumbnail-based visual navigation, etcetera. It's a deep product, and I'm going to be doing a much more in-depth analysis of it here soon.

One important thing to note about Expression Encoder is that it can be used to build the Silverlight player experience around an already encoded WMV file, so it can be used in conjunction with other compression tools.

Expression Encoder is based on the Windows Media Format SDK 11, so all the registry key tweaks I've been discussing on this blog work perfectly with it.

Silverlight for Linux is announced!

Good details on Miguel de Icaza's web log. In essence, we're partnering with Novell on their Moonlight implementation of Silverlight 1.0 and 1.1 (only 1.1 had been announced before). They'll be providing everything but the media codecs, which will be built by my team and provided via Microsoft.

So, Silverlight today has great reach for Windows and Mac (Intel and PowerPC), with Linux on the way.

Lots of content support!

For any new platform, the chicken-and-egg question is always paramount when discussing adoption - why should users install before there's content, and why should content publishers target it before users have installed it? That we're already compatible with a huge volume of WMV content is a big help here, of course. But nothing beats high-profile content companies targeting the format to get the installs out there (which is also a great validation of the unique value Silverlight provides).

Customers already deploying content include: MLB.com (Major League Baseball), Home Shopping Network, World Wrestling Entertainment, and the "Entertainment Tonight" show.

Silverlight is also now deployed on several Microsoft sites, including the Halo 3 preview site, including the HD version - I worked on the encoding workflow for these assets), Tafiti.com, MSN Extra, and MSN Podium '08. Silverlight will also be used in many other new and updated Microsoft properties.