By Michael Reichmann and Juergen Specht
May, 2005
Some BackgroundFor most of the past 10 years (effectively, the entire geological age of digital SLR cameras) photographers have been almost oblivious to a lurking danger. This threat – and it is no exaggeration to call it that – can best be understood by analogy.
Imagine that you were a photographer working with film, some time during the 20th century. Many of us don't have to make too great an effort to imagine this, because we were.
You shot your transparency or negative film, had them processed (or processed them yourself), made your prints, and then filed the negatives safely away in acid free storage boxes, so that the next time you or your clients needed a print the negatives were be safely available.
And, sure enough, whenever needed – even ten or twenty years later, we'd go back to our negatives, put them in the enlarger, and make a new print. And often, because over the intervening time our darkroom skills had advanced, or maybe because we had a new enlarger or we were using an improved paper or chemistry, our new prints turned out to be superior to what we had been able to produce before.
Now, imagine the following scenario. We retrieve our files, find the negative or slide that we want to reprint, and then discover that it has become opaque. The image is gone or otherwise inaccessible. We still have the piece of film that originally went though the camera, but the image itself cannot be accessed!
Good Lord – what could have happened? Well, imagine if the answer was that the company that made your original roll of film had manufactured it so that the film only fit into one type of enlarger, and that those enlargers aren't being made anymore. Or that the chemical properties of the dyes used to make that roll of color film were such that they would only interact to form an image with matching dyes in a printing paper from that same company; but – sorry, that company was sold a few years ago and the new owners decided to stop making that type of paper.
Totally unacceptable of course. But really, this is a pretty far-fetched scenario – isn't it?
No. Actually it isn't, because this is exactly the situation that we now face with our digital camera's RAW files. Let's see if we can understand what's going on and why the current situation has come to a head.
_________________
What is a RAW File?Here is a
detailed article explaining what a RAW file is. Simply put, it's the output of the camera's sensor with minimal processing. This means it contains all of the data about the image captured. Regard it as a digital negative. A negative though that has not yet been processed. The image is therefore latent; there, but undeveloped. This offers huge advantages to the digital photographer, because it allows us to re-visit our unprocessed files at any time in the future, and reprocess them again, as we find appropriate.
Developing the RAW FileYou've pressed the shutter and taken the photograph. It might be of aunt Millie eating a piece of chocolate cake at the July 1st BBQ, or possibly one of the alien mothership landing on the White House lawn. In any event, it's precious to you, aunt Millie, or maybe to world posterity. Only time will tell.
But it's your photograph. You took it, and it belongs to you. But first it needs to be developed (to use that easily understood word).
You have two basic choices. You can have the camera process it, or your can do it yourself later on the computer. If you choose that the camera does it, you set the camera to output a JPG file. This means that in a fraction of a second the camera will process the image, permanently setting the linearity, matrix processing, white point, color balance, color space, sharpening, contrast, brightness, and saturation, and then will save the file to the camera's memory card in an 8 bit compressed format – ie: a JPG file.
If you've shot a JPG file, and if this is what you want and need, that's fine. You've got a fully baked image. One that might accept a bit of correction later in Photoshop, but which is essentially complete in itself. Once the camera creates the JPG file it throws away the sensor's data, and its ready to take the next shot.
But, many photographers aren't happy with in-camera JPGs. Just as serious photographers and working professionals always liked to develop their own film, or at least have a pro-lab do so rather than simply trusting it to the drug store lab, so to do photographers now want to do their own processing of their camera's files.
So, they set their cameras to shoot in RAW mode. This means that the camera isn't processing the sensor output (other than to provide an LCD review image), but instead saves the image data to the camera's memory in what is called a RAW file. This file has not had any linearization, matrix processing, contrast, brightness or saturation processing. It has not been sharpened or lossily compressed, and it is still in 16 bit format, with no color space clipping. (All cameras do some image processing, because image sensors are analogue devices and the output needs to be turned into a digital file).
But, what you get is a virgin image (if you will). One that is ours to turn into a photograph that best reflects our memory or vision of what we wanted to record.
Once you transfer the RAW files from your camera's memory card to the computer the RAW file needs to be turned into something usable. It needs to have everything done to it that the camera does when it produces a JPG. But now, you are in control. You can set all of the parameters yourself, and can do so in 16 bit mode and a large color space, to avoid clipping colors and compressing tonalities. You can set your own sharpening, your own white balance, and more. The file is yours, and you can do with it as you wish, both technically and artistically.
But can you?
You can if you have a copy of the manufacturer's proprietary software for decoding the file. And herein lies the problem. What happens if you've lost your software disk? What happens if you change computers and can't find the CD any longer? What happens if the Megaflex Company (the makers of your camera) goes out of business, and no longer has a copy of the software on their web site for you to download?
What happens when your new Quantum Cube based computer no longer can read CDs, or DVDs, or its operating system can't deal with something as old and arcane as Windows XP or Mac OSX?
Far fetched you say? We'll, how many of you have a stack of 3.5" floppies somewhere in your desk drawer, and when was the last time you had a floppy disk drive attached to your computer? Still do – you say? OK. How about 5.25" floppies? Bet you don't have one of these sitting around anymore, except maybe moldering away in the basement somewhere.
How about being able to play those Sony Betamax tapes, or the 8 track cartridges you used to love so much? Can your current computer still understand Visicalc files from your Osbourne computer? I thought not.
What are the implications then for RAW files? Are the above examples relevant to the current situation?
YES.
You as a photographer have two issues to deal with. One – is the preservation of your RAW files in a physical format that will be readable into the future. There are many strategies for this that basically involve making multiple copies of your files on different media, and keeping those media safe from physical damage. If these files are copied to new media and new media types on a regular basis then you have a strategy for physical preservation.
The second issue is that of being able to actually make use of the data that's in the RAW file, and this is where we run into trouble.
_________________
The Tower of BabbleThe digital camera industry is very young; less than 10 years old. Because of this each camera maker has developed their own RAW formats. And, as their ability to improve the way in which information from the camera's sensor is recorded improves, these manufacturers have changed their formats continuously, usually with each new camera model.
How many RAW formats are there? Believe it or not, as of mid-2005 there are more than 100 different RAW formats in existence.
This alone is cause for concern. Companies come and go, and even those that persist often orphan old formats. There are in fact RAW formats in existence for which no manufacturer's software is currently available. And that's after less than 10 years.
Then there's the ugly matter of encryption. This was brought to the fore when in early 2005 Nikon released its flagship camera, the D2x, with encrypted white balance data.
Now, Nikon isn't the first, or the only company to encrypt their RAW files. But because they did so in such an intransigent way it has acted as a tipping point for the entire industry. Nikon owners are furious with the company, and quite a backlash has developed.
But let's apportion blame fairly. Other companies have done the same. In late 2003, for example, Sony encrypted the RAW files for its then new F828 digicam. It took only 3 days until the encryption was broken, but that's beside the point.
Companies like Canon and others are not free of guilt. Most camera makers use tricks to hide their data. Fortunately there are some very smart programmers out there, and to our knowledge none of these software tricks remain more than a trivial annoyance to anyone that really wants to crack them.
But, in the US at least, with the Millennium Copyright Act, the game has changed. Simply put, anyone that cracks encrypted or otherwise protected intellectual property is subject to criminal prosecution. And, since it possibly can be argued that the code (not content) in an encrypted or ÒprotectedÓ RAW file belongs to its creator (meaning the camera maker), any company or programmer breaking that code needs have liability concerns.
Similarly the proposed European Software Patents Protocol could mean that RAW formats will be patent protected and therefore not freely usable, even for interoperability purposes. If this comes to pass, European photographers could lose free access rights to their own photographs.
Now we all know that there are quite a few companies that write generic RAW converters. Many of these are small cottage industries and are highly unlikely to be prosecuted by the likes of the large camera makers. But there are larger companies, such as Adobe and Phase One, which are much more tempting targets for lawsuits, and who have litigation and liability lawyers who try and look after their best interests.
This is what has happened in early 2005. Both Adobe with Camera RAW 3.1 and Phase One with Capture One 3.7 have refused to decode Nikon's new white balance encryption on D2x and D50 camera files. Not that they couldn't (individual programmers did it days after it was released), but because it exposes these corporations to liability.
This is an unacceptable situation. We can't fault Adobe and Phase One. They're simply looking after their own best corporate interests. But who then is looking after our best interests? Apparently no one, and certainly not the camera makers.
We write camera makers (plural) because while Nikon has put itself in the bull's-eye of consumer scorn, almost all camera makers are guilty of promoting a seemingly never-ending succession of proprietary RAW formats.
This has to stop!
_________________The Solution?There are two solutions – the adoption by the camera industry of...
A: Public documentation of RAW formats; past, present and future
or, more likely...
B: Adoption of a universal RAW format
Proposal A simply requires that camera makers recognize that they are doing their customers a disservice by hobbling RAW files with various layers of proprietary processes and encryption methodologies. Once I, as the photographer, have pressed the shutter, the image file belongs to me; not Canon, not Nikon, not Pentax, or anyone else. Me! And by putting up roadblocks to my unfettered access to these files, they are acting against my needs and my interests.
Proposal B is a request that camera makers adopt a universal RAW file format. Adobe has put forward the
DNG format as an open standard. Is this the one to adopt? We really have no axe to grind, one way or the other. If DNG is seen as a suitable standard, that's great. A large number of independent software makers have already embraced it, and Leica has adopted it as their native standard for their new Digital Module R for their R8/R9 camera bodies.
But if there's a better way, please propose it and get the rest of the camera industry to adopt it.
Listen to us. We don't care who wins the corporate battles. The only winner we care about are ourselves and our fellow photographers. And, in this matter, if you're not with us, you're against us.
The OpenRAW Working Group With this as background, the
OpenRAW Working Group was founded.
The goal of OpenRAW is to encourage image preservation and give creative choice on how images are processed to the creators of the images – us – photographers.
This group is an amalgam of photographers and other interested people founded to create awareness of these issues, suggest a solution, and provide coordination for these efforts.
Some photographers might ask "Why should I care about OpenRAW? Somebody will always figure out the file formats." OpenRAW replies:
Third party processing alternatives only very slightly increase the probability that today's images can be processed at some future date. Third party developers are often small, and more likely than camera manufacturers to disappear. Image processing software must be continually updated to work on current computing systems. Just because Brand X can do it today, doesn't mean that Brand X will be usable 50 years hence.
Third party software producers, just like camera manufacturers, aren't publishing the file formats they have deciphered. They each hold that knowledge to themselves for competitive advantage.
In important and related issues, no one currently is producing software that can recover data from damaged RAW files. Files can be damaged by media deterioration or by software bugs. Only by knowing what the parts of the file are, and what those parts mean, is there a hope of recovering useful data from damaged files.
Third party software producers don't always decipher all parts of RAW files, only the parts they need to make their software work.
Having open RAW documentation speeds up the development of new methods of conversion. Having formats closed inhibits growth and maturing of digital photography. One of the most visible results is that we still do not have decent camera profiles, because camera profiling should be done in RAW mode.
The more camera companies "hide the data," the more time and resources are needed by third party developers just to decipher it – let alone add features to their programs. And, at some point, it becomes no longer economically feasible to do so.
The OpenRAW Working Group believes that open file formats encourage innovation. OpenRAW is not asking manufacturers to disclose the trade secrets of their sensor construction or their firmware, only the structure of the RAW data recorded when the photographer presses the shutter button, because:
1. There is no guarantee that manufacturers won't change their camera software support policies and abandon older cameras from current software.
2. There is no guarantee that manufacturers will exist in the future. (Consider Contax and their groundbreaking full frame Digital N camera of just a few years ago – now completely orphaned. Could your camera suffer a similar fate one day?)
3. There is no guarantee that a software program will execute correctly in the future. So even if we keep old hardware and software around, there is no guarantee that it will work properly on future computer systems.
4. There is no guarantee that support will be added to new conversion software for older and discontinued models.
The bottom line is that if only the manufacturers' software can fully edit RAW images, photographers and consumers of their images are captives of the manufacturers' software strategies as they change with time.
You should also consider that as RAW processing software improves (and it is improving every year), photographers with orphaned proprietary RAW files will be left behind. For example, Canon's latest DPP software does not support the Canon D30 camera, which is just a few years old, and which is still in use by some photographers.
Finally, consider the problems of digital asset management and the cataloging of files. Many pros as well as amateurs have literally Gigabytes of RAW files. The various asset management programs can't hope to keep up with the ever increasing number of proprietary formats. And as time passes and these programs are enhanced, what are the chances that they will still be able to read your older RAW files? If Canon can no longer do it with one of their own formats from just a few years ago, how can we expect third parties to do any better?
_________________Act Now!The time has come for us as digital photographers, and purchasers of equipment from the major manufacturers, to speak out. Historically, when people with a common cause, a just cause, speak out, their combined voices are heard.
Below is an open letter to all of the major digital camera makers. If you agree with its basic premise, we urge you copy it off the screen and paste it into an email to the camera maker of your choice, or to each of them for that matter. Just copy the text of the letter, click on a manufacturer's e-mail link, and past the text into your mail program. If you prefer you can print it out and mail or fax it instead.
This petition is being posted on a large number of web sites; sites with a combined readership of more than one million photographers from every country in the world.
Add your voice today.Dear Sirs,
I am writing to add my name to the list of photographers from around the world that are requesting that your company, as well as the other major digital camera makers, adopt a policy of open documentation of RAW formats, past, present and future.
I am also requesting that your company adopt a universal RAW format. The DNG format has been put forward as such a possible standard, but we are willing to accept any truly open standard as the industry may agree upon.
I support the position on this taken by the OpenRAW Working Group (
http://www.openRAW.org/).
Please add my voice to those that are against proprietary and encrypted RAW file formats. I urge you to act swiftly to support your customers so as to ensure our continued loyalty to your company's brands and products.
Sincerely,
(Your name)
Please copy the above text and then click on the link below to see a page of e-mail addresses for most of the major digital camera makers
__________________________________Camera Maker's E-mail Addresses__________________________________
An InvitationThis is an invitation to all web site authors and publishers, as well as camera clubs and those in the print industry, to freely copy and republish the contents of this page. We only request that you leave the text unchanged, though you may of course add whatever additional commentary you wish. You may also translate this text into the language of your choice.
The more voices that are raised on this issue, the sooner that the camera makers will hear us and act upon our needs.
For further information on this initiative, please contact either
Michael Reichmann at The Luminous Landscape or
Juergen Specht at OpenRAW.
Versions of this page in
MS Word as well as
PDF format may be downloaded.
