Powered by OPF
OPF Q&A Static Archive
Ask on GitHub
133 questions · 234 answers · 193 comments · 172 tags · 118 contributors · archived August 2026
Archived copy. This is a static snapshot of the OPF community Q&A (qanda.digipres.org), preserved from its Question2Answer database. It is read-only and nothing here has been edited since capture in August 2026. To ask new questions, use the OPF community on GitHub.

What is a file format?

5
votes

Not to get too theoretical, but I think it is important to develop a common understanding of key concepts in the digital preservation field. 

File formats are often discussed but there are varying ideas put forward to explain what they are/what the term means. This variance can cause confusion and make it difficult to agree on common processes and tools to use for digital preservation purposes. 

Some sub-questions to help the discussion:

Is a file format a standard for use in formatting files?

Is a file format a common way of structuring the internal structure (format) of files?

Is a file format whatever The National Archive's DROID tool identifies?

Can the spread sheet files created by Microsoft Excel 2007 SP2 with an ODS extension be considered Open Document Spread sheet formatted files?

Can the (significantly different) spread sheet files created by OpenOffice 3.x with the ODS extension be considered Open Document Spread sheet formatted files?

Do we need multiple terms to refer to the various different types of concepts that we currently refer to as "file formats"

asked by euanc on 21 May 2014 · 2778 views

7 answers

7
votes

For what it's worth, we say:

"A file format is a method of storing digital information in a computer file, allowing its later use by computer systems or people. There are thousands of different file formats for different kinds of digital content and there may be several different versions of the "same" file format. 

A file format is often confused with the software most commonly used to create or use it. 
For example, we talk about "Microsoft Word" files, or "Acrobat PDF" files. Despite these 
naming conventions, in principle a file format is not bound to any particular software – 
even if this is sometimes the case in practice. 
 
A file format is like a language which is only spoken by certain pieces of software. In 
general, the greater the number of languages, and the fewer speakers there are for 
each, the harder it is to maintain the ability to read and understand them over long 
periods of time."
 
http://www.nationalarchives.gov.uk/documents/information-management/file-formats-for-transfer.pdf - section 2.
 
 
 

 

answered by dclipsham on 6 Jun 2014
  • This is the definition I most adhere to and it makes me think: Using this definition a set of format trees could be developed. Signatures could be associated with points in the trees, e.g. "word 97-2003" could be a format and "word 97-2003 created by Office 2003 SP1" could be a subformat of that format, each with their own signature. Those that only cared about files being of "word 97-2003" format could make use of the high level signature and those needing to understand the potential differences introduced by the creating application could use that "lower level" signature.
    There could also be an identifier (though probably not a signature) for the intended interaction environment. Files could be tagged with that ID (though probably not automatically) and that could be used for risk analysis (e.g. don't have the environment, your content is at risk).
    — euanc, 6 Jun 2014
  • Right now (personally, and my views at this point may not necessarily represent The National Archives' past, current, or future policy), I think rendering intent and canonical prescription will not always necessarily marry up (for examples, see any web page created and optimised for e.g IE5 or Netscape 3 etc.), and my personally favoured approach is to make the bitstream in its original form wholly available and to allow the end user to choose how they interpret it, through whatever means they deem appropriate.

    I also think where possible, heritage institutions should make available rendering options, but I'm conscious of costs and relative value.

    Lately I've been thinking about video games (okay, I'm usually thinking about video games), but specifically about how devs would tailor their sound, colour schemes etc to suit a range of hardware environments. Even back in the day, everybody knew that SVGA/SoundBlaster was king, but a person with EGA/Tandy, or even onboard or possibly no sound could possibly still experience the 'intellectual entity' of the video game. The experience will certainly have been different, but which version was 'true'?' Is the canonical version in this case the developers 'ideal rendering,' or are all renderings fundamentally acceptable, and should we provide them all? Can we even possibly hope to?
    — dclipsham, 7 Jun 2014
  • Thanks for the detailed comment David. I have a fair bit to share regarding your questions and concerns. This whole discussion is really fascinating to me and does help to explain the difficulty in answering the question of this page.

    I have discussed some answers to your last three questions here: http://www.openplanetsfoundation.org/comment/371#comment-371 . To summarise, I believe we should endeavour to provide at least one "representative" interaction environment for use in interacting with the content being preserved. Ideally we could provide multiple representative interaction environments but that may never be fiscally practical.
    Providing a representative rendering environment would likely be the best we can do in the absence of knowledge of the intended interaction environment.

    In the case of websites, I've discussed those many times in the past and can assure you that in the case of such content it is really quite straight forward to recreate the original interaction environments for those http://digitalcontinuity.org/post/22115350229/accessing-authentic-archived-websites-well-aaaww

    Lastly, addressing the second paragraph of your comment I really believe that the cost of alternative strategies is likely to be greater than that related to maintaining interaction environments. I discussed that further here: http://digitalcontinuity.org/post/47460207305/reply-to-dshrs-comment
    — euanc, 10 Jun 2014
3
votes

The FADGI glossary (http://www.digitizationguidelines.gov/term.php?term=fileformat) has this to say:

Set of structural conventions that define a wrapper, formatted data, and embedded metadata, and that can be followed to represent images, audiovisual waveforms, texts, etc., in a digital object. The wrapper component on its own is often colloquially called a file format. The formatted data may consist of one or more encoded binary bitstreams for such entities as images or waveforms, and/or textually-encoded data, often marked up with XML or HTML, for texts. The embedded metadata may be skeletal or extensive.

This definition has been tailored to fit the planning activities carried out by the FADGI File Format Subgroup. Meanwhile, in the digital library community, the broad concepts underlying the FADGI definition are often subsumed under the generic term format, although this usage does not generally require that all three elements (wrapper, bitstream, and metadata) be present at the same time. Here are two definitions for format from authoritative bodies in the field:

  • A set of syntactic and semantic rules for mapping between an information model and a serialized bit stream. Many formats can be grouped into loose categories, or families, sharing a general set of encoding rules that are further restricted or extended for the specific format or profile. A format version is considered a profile. (Combined definition from the United Digital Formats Registry (UDFR), slide 7 in the Unified Digital Formats Registry Stakeholder Meeting PowerPoint Slides; and JHOVE2, JHOVE2 glossary.)
  • The internal structure and encoding of a digital object, which allows it to be processed, or to be rendered in human-accessible form. A digital object may be a file, or a bitstream embedded within a file. (From the U.K. National Archives Digital Preservation Technical Paper Automatic Format Identification Using PRONOM and DROID.)

Additional definitions of format have been offered by the InterPARES 2 Project and the Library of Congress Sustainability of Digital Formats Planning Web site.

answered by kmur on 25 Jun 2014
1
votes
I would suggest that there is no good answer to this question, as much as we all would like there to be one. A "file format" can be defined by many things and can have a range of purposes. Even a new and (supposedly) clearly defined file format can be adapted, extended, appropriated or twisted over time as new software implementations add proprietary functionality, interpret ambiguous definitions in different ways or simply fail to support all aspects of a format. As a consequence it's impossible to come up with a concise definition. At best a file format is a somewhat fuzzy entity.
 
A file format might be defined by a file format specification, a reference implementation for an application that renders it or indeed any other software that saves out data in a particular way. In practice it can often be a somewhat unclear combination of all three.
 
A file format might be designed to capture and store data in a carefully ordered and organised manner. Or it might simply capture the state of a software application at a particular time (eg. early MS Word formats).
 
Sheila Morrissey provides an excellent example of the challenges in understanding, defining, or indeed modelling, what a format actually is in: "The Network is the Format: PDF and the Long-term Use of Digital Content". This quote gives a hint at some of the complexity wrapped up in just this one example (but the whole paper is well worth a read):
"Older versions of the PDF specification included an appendix called “Implementation Notes”, which describes at least some of the deviations from the specification for which Acrobat reader attempts to compensate. These notes do not comprise a part of the ISO PDF 32000-1:2008 document. Further, these notes, while helpful, beg the question as to what we are to consider authoritative with respect to PDF format instances: the specification, or the behavior of the Acrobat reader application."
 
answered by prwheatley on 10 Jun 2014
  • Paul, you're making me feel like my question wasn't clear. Your reply seems to be addressing "what are instances of file formats?", not "What is a file format?". Does that make sense?
    — euanc, 10 Jun 2014
  • I didn't want to make you feel that *:-), but I'm not sure I agree with you about my answer. I tried to consider what defines a file format, and what the purpose of a file format is. These seem like important considerations to consider in the context of your question, and this follows on very closely from your ODS example.

    I guess what I'm trying to say in the answer is that we tend to use the term "file format" (which sounds like a precise, specifiable, verifiable entity) to describe something that in practice is actually a loose collection of concepts, documentation and realisations in software (which is very imprecise, virtually impossible to specify, difficult to verify and impossible to agree on).
    — prwheatley, 10 Jun 2014
  • Ok, but I'll now politely refer you to http://qanda.digipres.org/77/should-discussion-about-questions-comments-answer-fields

    And instead ask if you could attempt to define as many of those versions of "file format" as possible?

    e.g. a file format (v1) is: xxxxxxx
    a file format (v2) is: xxxxxxx
    a file format (v3) is: xxxxxx

    Thanks!
    — euanc, 10 Jun 2014
  • To be fair to you though you did answer the question, you said that there is no answer to it. Which *is* an answer.... ah either way, specific answers (even if there are many) would be useful
    — euanc, 10 Jun 2014
  • Exactly. My answer isn't a "discussion". Show me a specific answer and I'll show you an example that breaks it. What did you mean by this bit? 'And instead ask if you could attempt to define as many of those versions of "file format" as possible?'
    — prwheatley, 10 Jun 2014
  • Ok, lets go down this rabbit hole, I don't believe the list of definitions is quite as extensive as you suggest. How about we start your process with the definitions given so far:

    Definition 1: "A file format is a method of storing digital information in a computer file, allowing its later use by computer systems or people"

    Definition 2: " A file format is a predefined method for capturing and storing data in a carefully ordered and organised manner"

    Definition 3: "A file format is a predefined method of capturing the state of a software application at a particular time"

    Definition 4: "A file format is a predefined method for encoding information into a form which can only be processed and rendered comprehensible by very specific combinations of hardware and software"

    Definition 5: "A file format is a file extension"

    Definition 6: "A file format is an option available in an application when choosing to "save as" a file"

    Definition 7: "A file format is an option available in an application when choosing to "open as" a file"

    Definition 8: "A(n open as) file format is the set of intentions about how the file might be accessed"

    Definition 9: "A file format is a way of structuring the content in a file that limits the number of environments that can be used to interact with it without altering the content that was intended to be made accessible via that interaction."

    Are there any of these definitions that you can't break with examples?

    Should we relabel some of these and continue to use them but not refer to them as "file formats" but rather some variant of "file formats"?

    My basic concern and reason for wanting to clarify this is that "file format" is a fundamental concept in digital preservation. The term is used interchangeably to mean different things and that is the very problem it would be worthwhile to overcome, without a common understanding of it the use of the term is useless as it can't convey intended meaning.

    For example, if someone tells me they have file for me to preserve, and I ask what file format it has, then without a common understanding of what it means they could respond with many different answers, most of which would be completely unhelpful for me. If instead I ask them what internal formatting standard does it conform to, or what environment should be used to interact with it, then I will get some useful information. Which is to say, if we both agreed that it meant one of the definitions corresponding to the last two questions then I could save time and energy in conversing with them and, most importantly, avoid confusion.
    — euanc, 10 Jun 2014
  • I think we need a qanda.meta to do this, else it's going to get very messy. Very quickly, the fuzzier the better. So (1) is the hardest to break. I reckon I could have a pretty good pop at the rest. Of course (1) doesn't really say anything useful in preservation terms, but I guess that's my point.

    Incidentally I see exactly what you're after from that very clear example, but in practice you probably don't want to be precise about it. You just want a quick impression of what delights you might find in the file. To say it's a "doc" or "I saved it from Word" is probably enough.

    Isn't the only precise definition of what a file is (in file format terms), actually the version of the software that created it? Anything else is just an observation of a file's characteristics that might loosely classify it in some way that we generally associate with the term "file format".
    — prwheatley, 10 Jun 2014
  • So, a file format:

    Is a digital container for content - made to a specification (the container, not the content) - which allows human beings to view the content using a computer.

    Sort of (1), (2) and (sort of) (9) above.

    No?
    — crouchmi, 11 Jun 2014
  • @crouchmi if you think all formats have a specification, you're in for a nasty surprise! :-)
    — anjackson, 11 Jun 2014
  • I can't tell how flippant you're being here, Mr Jackson, so I'll plow on:

    1) Chaos and entropy are extant in this crazy mundane world we live in, but just as 'database' *implies* structured data -- 'format' *implies* specifications. They may not be documented, heck - they may not be known outside of the skull of the mad scientist who is using them, but you encapsulate content in a file format to re-use the content in a consistent, predictable way. SHARE and REUSE. No?

    2) And I guess that its not just human beings who access content contained in file formats - computers are potential users as well. My bad.
    — crouchmi, 11 Jun 2014
  • Ah, it seems I misinterpreted 'specification' to mean some kind of documentation or definition independent of the specific software/user that reads or writes the bitstreams. Of course, something somewhere 'specifies' the content of a bitstream, but I don't tend to think of that as 'a specification' because it can be so weak/ambiguous. But you are right, it can still be considered a specification.
    — anjackson, 11 Jun 2014
0
votes
From my experience, the many people distinguish formats based on extensions. This means that people will distinguish a TSV from a CSV, but not an Excel ODS from an OpenOffice ODS or a PDF 1.3 from a PDF/A3.

I don't think that's an exact or even good use of the term, but it's the most understandable use of it for most people. If the same content is somehow different when placed in a PDF 1.3 and a PDF/A3, why do both of them carry the same .pdf at the end? Given that, maybe we should start speaking of file format versions for anything beneath the level of 3-letter abstraction.

Edit: As suggested in the comments, extensions have many failings, such as a text file created on a Unix system that could be Javascript code, JSON data, or just text. MIME type does a better job of differentiating between these cases. It too isn't perfect, but I think a definition needs a practical output for it to be useful. Otherwise, we are stuck in the same epistomological problems.
answered by nkrabben on 22 May 2014
  • I like the approach, taking the most simple definition as the base and defining other terms "format version"(?) for referring to related, more complex, concepts. However there were and are many programs that create files without extensions. So incorporating extensions into the definition seems like it might limit it too much.
    — euanc, 23 May 2014
  • I agree that extensions are too limiting. There was a time on certain platforms where extensions were unnecessary. Maybe an approached based on MIME types would be more appropriate?
    — thorsted, 4 Jun 2014
  • Since extensions are usually just three characters, they're often unintentionally used for unrelated formats. In cases like these, we'd need to refer to them as something like "the .xyz format from ABC corporation." I think extension is just one facet, even at the level of popular perception.
    — gmcgath, 19 Jun 2014
0
votes

This is how Microsoft defines the word format in their application for the application/msword mime type:

From any microsoft word application select "Save As..." from the "File" menu.  Enter a filename, make sure that "Normal" is specified for the file type, and click "Save".

If you wanted multiple terms I'd suggest there are the:

saved as file format (which is the file's lineage: the application that created a file, its version, and the set of options selected by the user when saving the file). This is how MS chose to describe the msword file format in the quote above.

open as file format (the file's interface: the set of intentions about how the file might be accessed e.g. I'm declaring that it is a PDF, that can be opened by PDF applications compliant with v 1.7+, and that it conforms with a PDF accessibility standard so it can be faithfully rendered in a screen reader)

In practice, I think it is the open as file format that we really care about & I think Pronom does a pretty good job of mapping that space. Of course, because applications inevitably make mistakes when declaring open as file formats, it can often be very helpful to know the saved as file format too.

answered by richardlehane on 6 Jun 2014
  • I really like the saved as and openable as options but I also think we need something for the set consisting of the combination of saved as and open with. “open with” would be the configured software environments that the object was intended to be opened with in order to be interacted with.
    Each object/file/set of files may have one or more “open with” environments. But an important distinction between “open with” and “open as” might be that some possible “open as” options may not be appropriate for the file for whatever reason (e.g. they mispresent the content or distort it in some meaningful way) which would be why the “open with” option would be useful.
    — euanc, 6 Jun 2014
-1
votes
These are great questions to ask and I think a common understanding of these key concepts is important.

I have found myself asking these questions when working on a collections of born digital material. I use tools like DROID, JHOVE, FITS to determine if the files are well formed and vaild and are what they say they are. But I find myself forcing a few files to conform to the DROID standard just to get them through the system.

DROID is an ongoing project and needs to be updated frequently to stay ahead of changing file structures.

I think the best approach is really based on your digital preservation strategy, will forcing all files to conform to a standard make them easier to migrate in the future or are you more interested in preserving the original file?
answered by thorsted on 4 Jun 2014
  • Valuable discussion but I'm going to vote this down as it doesn't actually provide an answer to the question. (Actually that is some feedback about this site, it could do with a better method for managing discussion vs answering the questions).
    — euanc, 4 Jun 2014
  • Can we agree on a definition that is preservation strategy agnostic?
    — euanc, 4 Jun 2014
  • If a seemingly valid file isn't being identified by DROID we'd be extremely interested to understand why. Please raise such issues via pronom@nationalarchives.gsi.gov.uk so we may investigate further. I certainly would not wish to encourage anybody to alter their files to fit with what DROID wants to see.

    See here for an example of where such feedback helps improve the accuracy of DROID/PRONOM: https://groups.google.com/forum/#!topic/pronom/pIgV6iK2uss
    — dclipsham, 6 Jun 2014
  • A question for you David, how would/should DROID deal with the ODS issue outlined in the description of the question? I.e. Microsoft and the opensource developers differ in their interpretation of the standard, can files created by both sets of applications be both "valid"? I actually really like the registration tree approach outlined here: http://en.wikipedia.org/wiki/Internet_media_type . Could identification within a similar tree like structure be a possibility for a future version of DROID?
    — euanc, 6 Jun 2014
  • BTW, this scares the hell out of me: "I find myself forcing a few files to conform to the DROID standard just to get them through the system."

    Fudging with originals just because DROID is unhappy with them is very troubling to me, unless you are responsible for creating the files. DROID and JHOVE etc. will absolutely always lag behind actual format usage, that's unavoidable.
    — anjackson, 6 Jun 2014
  • The folks at pronom have been responsive to our issues and have corrected signatures to properly identify a valid file. It is a wonderful tool, but things change so fast I wouldn't rely on one tool. When Adobe changed their XMP structure within PDF's in Acrobat Pro 11, it took a few months to get the new signature files released and even longer to get them in our preservation system.
    — thorsted, 6 Jun 2014
  • I think it's important to understand what DROID is doing in order to answer this question.

    DROID is scanning the bitstream of the file, comparing it against a list of signature sequences, and reporting back when it finds a match. Nothing less, nothing more.

    A suitably skilled individual can easily create a 'file' that contains a matching byte sequence and nothing else, and DROID will report a match (See Ross Spencer's Skeleton Suite work for a tool that does exactly that - https://github.com/ross-spencer).

    So where has the sequence come from? Simply our internal research. We are a very small team who work on PRONOM as part of our wider roles. We are not world-experts in every (or even any) particular file format , but we have enough experience of developing file format signatures to understand what we are looking for. The National Archive's (current) use-case for PRONOM/DROID is to understand what types of formats are entering our collection, primarily so we can drive our own workflows and preservation strategies. We do this for our own business purposes and make our research freely available. I'm pleased and proud that other institutions and individuals (generally) seem to trust our data output.

    What we are not, and have never asserted ourselves to be, is arbiters about what constitutes a canonical format. DROID simply has no notion of validation, and users who want validation should seek other appropriate tools for that, which they themselves judge to be fit for purpose.

    As for registration trees, I do like this too, however it kind of falls down (currently) at the opening sentence. "All media types should be registered using the IANA registration procedures." I agree, they should be. But they're not and I'm not sure it's realistic that they ever will be. For our part, we (currently) tend to add a media type to PRONOM only when either the IANA register, or the vendor themselves have asserted a particular media type.
    — dclipsham, 6 Jun 2014
  • DROID/PRONOM have never attempted to assert anything about the validity of a file format, rather the byte stream matches particular identifying characteristics which are common to files of a certain format. If both variants of ODS match the defined siganture, they'll be identified as ODS.
    — DavidUnderdown, 6 Jun 2014
  • Forgive me if I sounded like I was being negative toward DROID and the team. Thank you for the clarification. I think it is wise for everyone to know what the purpose of tools like DROID are for. I was hoping to answer the question stated in in this thread "Is a file format whatever The National Archive's DROID tool identifies?"
    — thorsted, 6 Jun 2014
  • Not at all - my latest response was to Euan's question about QDS! Non-threading comments boards are always prone to this :)
    — dclipsham, 6 Jun 2014
  • I'll just add that DROID and PRONOM are excellent, particularly very impressive given the size of the team!
    — euanc, 6 Jun 2014
-1
votes

Apologies for answering my own question, but one answer can be found on the Pronom website:

 

"File formats encode information into a form which can only be processed and rendered comprehensible by very specific combinations of hardware and software."

 

It does occur to me that many signatures in Pronom match with more than one of the things that match that definition. 

answered by euanc on 10 Jun 2014