File format identification

File Format Identification is the process of figuring out the format of a sequence of bytes. Operating systems typically do this by file extension or by embedded MIME information. Forensic applications need to identify file types by content. It is one of the categories of file analysis.

Tools

Data Sets

If you are working in the field of file format identification, please consider reporting the results of your algorithm with one of these publicly available data sets:

NPS govdocs1m - a corpus of 1 million files that can be redistributed without concern of copyright or PII. Download from the Digital Coprora website
The NPS Disk Corpus - a corpus of realistic disk images that contain no PII. Information is at: https://digitalcorpora.org/?s=nps

Bibliography

Current research papers on the file format identification problem. Most of these papers concern themselves with identifying the file format of a few file sectors, rather than an entire file. Please note that this bibliography is in chronological order!

2003

Content Based File Type Detection Algorithms, by M. McDaniel and M.H. Heydari, 36th Annual Hawaii International Conference on System Sciences (HICSS'03) - Track 9, 2003.

2005

Fileprints: identifying file types by n-gram analysis, LiWei-Jen, Wang Ke, Stolfo SJ, Herzog B.., Proceeding of the 2005 IEEE workshop on information assurance, 2005. (PDF)
D.J. Hickok, D.R.Lesniak, M.C. Rowe, File Type Detection Technology, 2005 Midwest Instruction and Computing Symposium.(PDF)

2006

K. Martin, N. Shahmehri, File type identification of data fragments by their binary structure, Proceedings of the IEEE workshop on information assurance, pp.140-147, 2006. (https://ieeexplore.ieee.org/document/1652088)

2007

Oscar: File Type Identification of Binary Data in Disk Clusters and RAM Pages, by M. Karresand, N. Shahmehri, May 2006.
Using Artificial Neural Networks for Forensic File Type Identification, by R.M. Harris, May 2007.

2008

M.C. Amirani, M. Toorani, and A. Beheshti, A New Approach to Content-based File Type Detection, Proceedings of the 13th IEEE Symposium on Computers and Communications (ISCC'08), pp.1103-1108, July 2008. (PDF)

2009

V. Roussev, and S. Garfinkel, "File Classification Fragment-The Case for Specialized Approaches," Systematic Approaches to Digital Forensics Engineering (IEEE/SADFE 2009), Oakland, California. (PDF)
I. Ahmed, K.-S. Lhee, H. Shin and M. Hong, On Improving the Accuracy and Performance of Content-based File Type Identification, Proceedings of the 14th Australasian Conference on Information Security and Privacy (ACISP 2009), pp.44-59, LNCS (Springer), Brisbane, Australia, July 2009.

2010

Fast File-type Identification, by I. Ahmed, K.-s. Lhee, H. Shin and M. Hong, March 2010.

2011

Fast Content-Based File Type Identification, by I. Ahmed, K.-S. Lhee, H.-J. Shin, M.-P. Hong, February, 2011
Tika in Action, by C. Mattmann and J. Zitting, December, 2011.