Archive for January, 2007

How not to store data

This blog has been spending a lot of time discussing storage, and will continue to do so. This is, after all, perhaps the largest new workflow challenge with HD, 2K and 4K cameras, for those moving up from the SD world. I’ve already discussed archiving (offline storage) a bit, and I’m planning some posts on online storage. But first, I think a word on how not to store data might be in order.

First off, just to be clear, since there’s some terminology confusion… in the production world, “online” storage doesn’t mean Internet or network storage. It means specifically what we could call “active” storage, almost always hard drives. The drives you actually edit off off, as contrasted with, for instance, tape or DVD backups sitting in your closet, which are “offline” storage.

Onward.

Digital production involves working with large amounts of data. The IT industry has developed a wealth of techniques for managing data in ways that are robust, affordable, and convenient.

Unfortunately, many smaller digital production facilities don’t really have a serious IT guy on staff. Their data management schemes tend to resemble what one would expect if a technique that works well for hobbyists — sticking your footage on external Firewire drives — is scaled up for the entire business. I’ve seen facilities with a dozen or more external drives kicking around, with data haphazardly distributed around on them, and with no set of practices in place to make sure that there are always at least two copies of everything.

You really don’t want to do this. It’s expensive. It wastes a lot of time. And when you’re dealing with RED footage (estimate about one hour per 100 GB with REDCODE RAW 4K), it’s going to get really impractical, because you’ll need multiple external drives per feature.

Far worse, from the perspective of someone with IT experience, who knows how to keep data safe… it’s really just downright scary.

There are far better options. I’ll be posting about some of them this week.

Market segmentation, IT & skepticism

Early on, there was huge skepticism about what Red was doing. This came largely from people who had experience with traditional film and video acquisition, and they were skeptical with good reason. The RED ONE looks like a huge leap in technology, coming out of nowhere.

The crucial point many of these skeptics failed (and often still do fail) to understand is that it isn’t coming out of nowhere. It’s just not coming out of the traditional film/video acquisition industry.

Red is being built around technologies — and, perhaps more importantly, adopting a mindset — from a much more competitive and fast-moving industry, where features are driven by technology rather than by market segmentation schemes. I’m speaking, of course, of the commodity information technology market — the market where, for instance, hard drives have gotten 250 times larger (per dollar) in the last 10 years.

Now, of course, digital technology has been invading the traditional film/video acquisition industry for more than a decade. You can’t, as far as I’m aware, even buy an analog video camera these days, and several major motion pictures have been shot digitally. But the crucial difference is that while the existing vendors in this industry have co-opted some IT technology, they haven’t adopted the mindset I mention above — of building products around what the technology makes possible.

If one conceives of the motion picture camera market as a continuum, with low-end consumer camcorders at one end and film cameras at the other, for a very long time, products have been clustered mostly at the extremes, without much in the middle. This makes sense, historically, because for a long time shooting at high quality could only be made so cheap (due to the unavoidable cost of film stock), while low-cost electronic acquisition could only be made so good (due to technological limitations). There really wasn’t much of a continuum at all; there were two distinct markets with very different budget ranges and requirements.

High-quality digital acquisition has removed the material conditions which required this. For some years we’ve been seeing consumer camcorders slowly creep up-market, and increasingly find use in filmmaking, but for the most part the traditional camera vendors still don’t really seem to understand what has happened. They’re still mostly using new technologies to build cameras for the same old market segments — consumer camcorders at one end, and expensive systems designed for deep pocket customers like broadcasters and major productions at the other, with little in between.

Red isn’t thinking like this; they’re starting with what’s possible and figuring out how to make the best camera at the best price.

I’m not approaching Red from the traditional film/video industry — I’m approaching it from the IT industry. My reaction when I first heard about Red was not “They can’t do that!” but something more like “It’s about time someone did that!”

Archiving #2: tape vs. drives

Having basically ruled out optical storage, we’re left with two major archiving options: data tape and hard drives.

The leading high-end data tape format is LTO-3. Drives are available from many vendors, and you can expect to pay upwards of $3500. Tapes are 400 GB, and cost around $55 if you shop around, though you have to buy tapes in 20 packs (over $1000) to get prices like that.

This makes LTO-3 media a lot cheaper than hard drive space; it’s about $0.14/GB, or $0.23 per minute to store REDCODE RAW 4K footage. In contrast, you can get a 500 GB hard drive for around $200, which gives you a price of $0.40/GB or $0.66/minute for that Red footage.

But, of course, you have to take into account that big up-front cost for the tape drive. When does that pay itself off? Let’s do some math.

Assume $480 for a 1 TB hard drive (Hitachi is shipping these in Q1 in an external case (or a bay of a multi-bay external case, much more on these in a later post), and $4000 + $60/tape on the tape side (a price you can get without buying a thousand bucks worth of tape at once).

At these rates, the tape drive pays for itself when you’ve got 12 TB worth of data to store. If you’re storing 24 TB of data, tape is down to $0.32/GB, while hard drive storage is still $0.48/GB. For 40 TB of storage, tape is down to $0.25/GB.

So, which should to pick? Well, tape is a bit of a hassle vs. hard drive storage (much more on this aspect of archiving in a latter post). And 12 TB is a good bit of storage, even for 4K (well, compressed with REDCODE, anyway). It’s enough storage for ~120 hours of footage. If you’re making a documentary or a reality TV program you’ll probably need more storage than that fairly quickly, but that’s enough storage for all the footage comprising ten 100 minute narrative features shot at a 7:1 shooting ratio. It’s probably going to take a while to shoot ten movies, by which time hard drives will probably be more competitive, since hard drive prices tend to drop faster than tape prices.

Based on these numbers, this one is going to be a tough call for a lot of people.

Archiving #1: optical formats

The RED ONE is clearly going to generate a very large amount of data, even using REDCODE compression. At the 28 MB/s rate quoted for 4K, a minute of footage will be about 1.65 GB.

How do you deal with all of this data? This will be the first in a series of posts discussing archiving; it will address optical storage options. Future posts will address tape and hard drive storage options. The subject of online (working) storage will also be addressed in the future, in another series of posts.

For people with busy schedules, here’s the executive summary. How do you store hundreds of gigabytes of footage cheaply and conveniently on optical discs today? You don’t.

Standard single-layer and dual-layer DVDs can be ruled out immediately. A double layer DVD would only hold a bit over five minutes of footage, which is not remotely practical. But just for the record, storage costs would be about $0.35/minute to store REDCODE 4K footage on double layer DVDs.

Blu-ray and HD DVD have higher capacities. Maybe they’re more plausible? Not at the moment. We can probably write off HD DVD for the same reason we wrote of standard DVDs. A 15 GB single-layer HD DVD disc (the only sort your can presently burn) will only hold about nine minutes of footage, and the drives and media are far more expensive, even per gigabyte. A 15 GB HD DVD disc costs around $18. That works out to about $1.98/minute. Yikes.

Blu-ray is slightly more plausible. Single layer Blu-ray discs hold 25 GB. That’s 15 minutes of footage. Better — people seem to manage numbers in this range with film reels — but still not exactly ideal. A burner will set you back around $900, which is a lot… but a 25 GB blank disc costs around $20, for a per-minute cost of about $1.32/minute. This beats out HD-DVD, but it’s still quite pricey.

All media prices assume you’re buying in quantity.

These optical drives also all share another major problem — speed. Even 2x Blu-ray burners — the fastest of the formats — only burn at about 9 MB/second, which means recording a minute of footage takes three minutes. Read performance is similarly slow, making it impossible to play footage back directly from the disc, thus eliminating one of the major advantages optical storage would otherwise have over data tape.

So, while high capacity optical media might seem like the wave of the future, it isn’t practical at this point. Even if all these prices fall by 50% by the time Red starts shipping cameras, is still won’t be very practical.

In the next post in this series, we’ll turn to that stalwart of high-capacity data storage, still going strong in the 21st century: tape.

Changing the economics of feature production?

Can a cheap camera change the economics of feature production? I’ve thought a lot about this, and the answer I’ve come up with is “sometimes”.

If you’re making a $100M movie with tons of visual effects work, A-list actors, etc. then even if your camera and your film stock and processing costs were free you’d probably only save a couple percent, at most, on your budget. This means that, obviously, an inexpensive digital cinema camera won’t enable you to make a $100M movie for very much less than $100M. Or a even $5M movie for much less than $5M. If digital is chosen for films in this price range, it will be because it’s better for a particular production, not because it’s cheaper.

It’s more interesting to approach this from the other side. What are the people making $3000 movies with $20000 worth of equipment doing right now? They’re not spending anything on crew payroll, because they’re just making movies with their film school friends. They’re not spending anything on cast, because they’re getting underemployed actors, who just want to act, to work for free. They’re probably spending next to nothing on things like wardrobe and locations, because they’re taking interesting stuff their cast/crew have free access to (the rich uncle’s yacht, etc.) and writing them into the script. They’re choosing their subject matter so they don’t need stunts, explosions, elaborate visual effects, or other stuff that just fundamentally can’t be done for cheap. They’re doing post-production on the desktop, even if that means they can’t twiddle knobs to tweak color in real-time as the footage plays.

If they do all of this well enough (and many of them won’t, of course, but some will), you can’t tell they’ve done it on the cheap. But there is still one place where you can tell, immediately, if a production is an ultra-low-budget affair… right now, there’s just no way to do acquisition on the cheap and have quality that rivals 35mm on a big screen. Unlike with all these other things, it doesn’t matter how talented you are, or how much time you’re willing to invest, or how carefully you tailor your script to fit within your limitations. There’s just no way to shoot a 35mm-quality feature without spending tens of thousands of dollars (probably upwards of $100K) on camera rental, film stock, processing, telecine, lab costs, etc. or renting one of the existing digital cinema cameras at a daily rate that’s probably higher than the production’s entire budget. (And ending up with digital footage you probably can’t practically use with your desktop editing system anyway.)

And this is where inexpensive digital cinema cameras make a huge difference. If RED delivers, 6 months from now there will be a way to get that quality level, with maybe a $25-40K camera package you can buy outright, and few additional costs on a per-production basis. Now, obviously this approach won’t let you tell every story. You can’t make Star Wars or Lord of the Rings for $3000 just because you can shoot 4K for cheap. But there are a lot of interesting stories it will let you tell.

This isn’t going to radically change Hollywood, mind you… but bringing the cost of 35mm-quality acquisition down from $100K per feature to $25-40K as a one-time expense and maybe $1500 worth of hard drives per feature is certainly going to have a major impact on a particular segment of the market. And it will matter to major distributors, because they’ll start seeing ultra-low-budget stuff on their radar that they might want to consider buying because of the picture quality, rather than in spite of it.