Speculations about DNG


Future DNG standardisation?

(First version 12 June 2006; this version 21 August 2006)

Summary

This speculation is in 2 parts: ISO will treat as relatively high priority a standard for archiving raw image data from digital cameras; and ISO will base this on DNG.

I believe: ISO won't consider the revision of the existing TIFF/EP standard to be such high-priority. ISO won't identify any alternative to DNG as the basis for archiving raw image data. ISO will develop a standard, (which I'll call "DNG/A" for convenience), as a set of constraints and conformance-level-descriptions on Adobe's DNG specification.

Background

ISO's TC 42 (Technical Committee 42) is responsible for ISO's "Photography" standards. One of their outputs was TIFF/EP, which was based on Adobe's TIFF 6.0. It is the ISO standard for an image data format for "electronic photography". It includes the ability to hold the output from a sensor using a CFA, (colour filter array), commonly used in digital cameras. So it is, inter alia, the standard raw file format, although somewhat out-of-date. TIFF 6.0 and/or TIFF/EP are used to various degrees by some major camera manufacturers, including Canon and Nikon, as the basis for their raw file formats, including CR2 and NEF.

By analogy, ISO's standard "document file format for long-term preservation" is "PDF/A", ISO 19005-1. It isn't a complete specification. Instead, the document is a set of constraints and conformance-level-descriptions on Adobe's PDF Version 1.4 specification. It identifies a suitable set of constraints on a demonstrably-successful comprehensive working specification in order to make it suitable for ISO's specific scope. This method avoids unnecessary "tinkering" with the original version, because all changes are explicit and have to be justified, and makes it easier for products to support both versions, with clear options.

Why develop an archival standard?

The issue of "long term preservation of digital images" has become a hot-topic over the last year or two. As digital cameras overtake film cameras, and as raw shooting becomes popular among professional and other leading photographers, many influential organisations are investigating how to archive raw images so that they can still be used in decades to come. The US Library of Congress, ASMP, I3A, OpenRAW, PADI, and UPDIG are all putting resources into this topic. (I3A is the Secretariat for TC 42. The US Library of Congress, and the US National Archives and Records Administration, are leading participants in TC 42).

Why not just bring TIFF/EP up-to-date? While there is an obvious need to bring the standard up-to-date eventually, (or drop it), there doesn't appear to be a high-priority need. It doesn't appear to be used in any laws or regulations. (ISO standards don't automatically force companies to conform to them). TIFF/EP is simply used when companies believe it is useful to them. And they have alternatives, (such as DNG), if they just want a specification to work to, as some companies have shown.

But wouldn't a revised TIFF/EP be a good archival standard? An archival standard has to be designed for the purpose, because there are specialist requirements. TIFF/EP wasn't. A revised TIFF/EP could serve a dual role as a working raw format and an archival standard. But only one of those roles is high-priority. In effect, DNG is TIFF/EP brought up to date and made fit for purpose, so achieving that dual role would take extra effort and time. The world urgently needs a standard for archiving raw image data from digital cameras.

Does it have to be a standard? Initially, probably not. An ISO/TS (Technical Specification) or an ISO/PAS (Public Available Specification) would offer a credible direction. But some countries may need laws for this purpose, for example for public records or forensic evidence, and a formal standard would eventually be useful.

Why base an archival standard on DNG?

The short answer is "because it's there"! The longer answer discusses why that is important:

ISO isn't an independent innovative body for creating specifications. It typically doesn't have independent experts - its experts belong to companies and organisations already active in its fields of activity. For example, some of the leading participants in TC 42 come from Adobe, Canon, Kodak, Foveon, Fuji, Microsoft, Nikon, Olympus, and Sony. It typically doesn't innovate when it doesn't have to - for example, when it developed TIFF/EP, it based it on Adobe's TIFF 6.0. Specifications that have pedigree, political acceptability, and are known to be implementable, are "good enough" for ISO.

Like NEF, CR2, PEF, and some other raw file formats, DNG is based on TIFF 6.0 and/or ISO's TIFF/EP. Unlike formats from camera manufacturers, DNG is designed to cater for raw image data from the cameras of many manufacturers, and is designed to be an archival format. It has extra capability too, so the archival variant, (DNG/A, say), might be a slim specification that identified a set of constraints on the full specification, rather like the relationship between ISO's PDF/A and Adobe's PDF 1.4. ISO may object to fields such as DNGPrivateData, or want to reduce some variability, or identify conformance levels, etc. (I believe they should concentrate on which standard contents must be present, rather than try to restrict optional content, but that isn't really part of this speculation).

Does DNG have pedigree? Yes - it is based on the TIFF 6.0 that ISO already used for TIFF/EP, and upon TIFF/EP itself. It includes XMP, Adobe's eXtensible Metadata Platform, as does PDF/A, ISO's standard "document file format for long-term preservation". Its colour matrices use CIE XYZ coordinates.

Does DNG have political acceptability? (This doesn't just mean whether countries will accept it, but also whether other members of TC 42 and other TCs accept it). Yes, with two provisos:

  1. Nikon's position is unpredictable. It is unlikely that Nikon would propose its NEF format in place of DNG, because it would have to publish the specification and relinquish various rights. But it may simply object to the concept of an archival raw file format. (I suspect Nikon would adopt a more constructive attitude).
  2. Microsoft doesn't have its own raw file format to offer, but it may claim that WDP (Windows Media Photo) was adequate for the purpose, and offer that instead. (WDP is not even proven in its own right, and certainly not as an archival format!)
    Microsoft's Director of Professional Photographer Community Tim Grey has offered an alternative view: "Operating System support for different raw data formats can be achieved through the use of CODECs to read data. These CODECs would be developed by camera vendors and software providers using Microsoft's development guidelines".

If TC 42 collectively wants an archival raw file format, neither of those views will be sustained. Having CODECs written to Microsoft's development guidelines is hardly an acceptable long-term archival strategy!

Is DNG known to be implementable? Yes! It can currently cater for more than 120 cameras from more than 20 manufacturers, and is supported by more than 140 products from more than 120 sources, some of them writing DNG files. It has a published specification, a freely-available (optional) SDK, and global royalty-free licenses to use these.

References

TIFF 6.0: Originally "Tag [or Tagged] Image File Format", revision 6.0. (Commonly known as TIFF or TIF). Owned by Adobe.

TIFF/EP: ISO standard 12234-2, "Photography - Electronic still picture imaging - Removable memory - Part 2: Image data format - TIFF/EP".


Polarisation into "NEF" and "DNG-friendly"?

(First version, 15 June 2006; this version, 21 August 2006)

Summary

This speculation is that, over the next year or two, the world of raw shooting will polarise into a "committed to NEF" camp, and a "migrating towards DNG" camp. These will split "people shooting raw" about 20% : 80%.

(This speculation isn't about what happens after that next year or two).

Background

DNG offers various benefits. Its key inhibitor is that it is not supported by some raw-handling products. The most important category of such products are products supplied by camera manufacturers. Camera manufacturers differ in their acceptance of DNG. Some appear willing to support it in some way. As they do, many photographers will cease to be inhibited in their personal evaluation of their benefits and ability of adopting a DNG-based workflow.

What is happening is that there is a "new breed" of comprehensive image processors, supplied by major companies. (Apple's Aperture, Adobe's Lightroom, Nikon's/Nik's NX, and perhaps some by smaller companies). They offer non-destructive metadata editing, rather than pixel-rendering as in familiar photo-editors. These image processors are not just raw converters - all of the above can handle TIFFs and JPEGs, and they provide similar features whether they are handling raws or those. In other words, "raw conversion", the demosaicing task, is a relatively small component of the product, and for many file formats isn't needed.

Nikon's NX can store the results of editing TIFFs and JPEGs in NEF format. Lightroom can store the results of editing TIFFs and JPEGs in Linear DNG format. In other words, NEF and DNG themselves are not just raw file formats. Products that started as real raw converters are going to be competing with much more comprehensive products. They may not be able to become such image processors, with demosaicing being "just" an optional component, in time. If they think their strengths are with demosaicing, the other features may appear to be a diversion, but in fact they are more important.

Now: "Commitment to various raw formats"

Why "~ 58%"?
The 2006 RAW Survey - Chapter 3
About 27% always use products that don't accept DNG. A guesstimated 31% sometimes use products that don't accept DNG.

This treats people as committed to various raw formats when they want to use products that don't support relevant alternative raw formats. If someone wants to use a particular product from a camera manufacturer, and it only supports XYZ, then they are committed to XYZ. (It is irrelevant that some other people think some other product is at least as good - what matters is what individual photographers believe).

With few exceptions, products from camera manufacturers don't support DNG. I guesstimate that about 58% of raw-shooters use, (at least some of the time), a product that doesn't support DNG, often a product from a camera manufacturer. So they are committed to a native raw file format, at least partially. Only perhaps 42% of photographers could currently accept being fully committed to DNG. Obviously, they don't all yet use DNG. But they could do so, and may in future become committed to it.

In the diagram below, "3FR" refers to the Hasselblad-Imacon 39 MP raw file format. Hasselblad-Imacon photographers aren't "committed", because Hasselblad-Imacon provide software that both converts this format to DNGs and reads the resultant DNGs. This is an example of what is likely to happen with other camera manufacturers - removal of the inhibition about adopting a full DNG-based workflow.

Now - many people committed to native raw formats because of their software:

(Some photographers are committed to DNG already. These include those who use cameras that output DNG as their native raw format, and those that rely on features in DNG, such as the exploitation of XMP metadata within DNG).

Future: "Commitment to NEF"

Why "< 20%"?
The 2006 RAW Survey - Chapter 1
The 2006 RAW Survey - Chapter 3
34% use a Nikon camera. Perhaps 41% use software from Nikon. In a year or two, some other software still won't support DNG.

This speculates that, within a year or two, all significant camera manufacturers except Nikon will either use DNG as their native raw file format, or else provide their own interworking between their native raw file formats and DNG. (Nikon could surprise me, and provide such interworking - but I'm guessing it won't).

Obviously the most significant camera manufacturer is Canon. If they build upon CR2 in the way that Nikon are building upon NEF, this speculation won't happen. But there are hints that, while Canon intends to continue with CR2 as their out-of-camera raw file format, they may provide interworking with DNG, and they consider DNG to be a good archival format:

Canon Consumer Imaging Group Director Chuck Westfall:
"Canon intended to keep its RAW data recording methods proprietary" ...
"Adobe's DNG file format has excellent features for archival storage" ...
" Canon might consider the possibility of adding DNG support in future versions of RAW image conversion software".

Pentax have also offered a glimpse of this future:

"The K100D's standard software package consists of the PENTAX PHOTO Laboratory 3 ... and the PENTAX PHOTO Browser 3 that offers faster data processing speed and better operability than the previous version, and can convert RAW images to the DNG format". In fact, PENTAX PHOTO Browser 3 with the ability to convert to DNG is available for all Pentax dSLRs.

If Canon go this way, what will other camera manufacturers do? Build their future on NEF and join the NX camp? Go their own way? Or make it easy for their users to exploit DNG and the products that support it? I speculate that, if Canon go this way, so eventually will the other non-Nikon camera manufacturers. They would rather be aligned with a company that doesn't make digital cameras, and in fact already supports their raw file formats, than with the 2nd largest camera manufacturer!

Future - few people committed to native raw formats because of their software:

References