Magic bytes: how to read a hex dump and identify any file

The first 16 bytes of a file usually say what it is, if you know how to read them. A hands-on introduction to hex dumps, file signatures and the handful of patterns worth memorising.

detectionhow-tolow-level

Every tool that identifies files by content, from the Unix file command to antivirus engines to our detector, starts in the same place: the first bytes. Most binary formats announce themselves there with a fixed sequence called a magic number or file signature. Learning to read a hex dump is the fastest way to stop trusting filenames, and it takes about ten minutes.

What a hex dump is

A hex dump shows a file’s bytes as numbers. Each byte is a value from 0 to 255, written in hexadecimal as two characters from 00 to FF. A dump typically shows sixteen bytes per line, with the offset on the left and, on the right, the same bytes as text where they happen to be printable.

00000000  25 50 44 46 2d 31 2e 37  0a 25 e2 e3 cf d3 0a 31  |%PDF-1.7.%.....1|
00000010  20 30 20 6f 62 6a 0a 3c  3c 2f 54 79 70 65 2f 43  | 0 obj.<</Type/C|

Reading that: at offset 0 the bytes are 25 50 44 46, which in ASCII are %, P, D, F. This is a PDF, and the next bytes say version 1.7. You did not need the filename.

Getting a dump on any system

# macOS and Linux
xxd -l 32 file            # first 32 bytes
hexdump -C file | head    # same, different formatting
od -A x -t x1z -N 32 file # the POSIX way

# Windows PowerShell
Format-Hex -Path file -Count 32

Or drop the file on the detector, which shows the opening bytes under the verdict, without uploading the file.

The signatures worth knowing by heart

A dozen patterns cover most of what crosses a normal desk.

Bytes ASCII Format
25 50 44 46 %PDF PDF
50 4B 03 04 PK.. ZIP, and DOCX, XLSX, PPTX, JAR, EPUB, APK
89 50 4E 47 0D 0A 1A 0A .PNG.... PNG
FF D8 FF JPEG
47 49 46 38 GIF8 GIF
1F 8B gzip, including .tar.gz
4D 5A MZ Windows executable (EXE, DLL)
7F 45 4C 46 .ELF Linux executable
D0 CF 11 E0 A1 B1 1A E1 Old Office (DOC, XLS, PPT), MSI
53 51 4C 69 74 65 SQLite SQLite database
52 49 46 46 .. .. .. .. 57 41 56 45 RIFF....WAVE WAV (AVI for AVI, WEBP for WebP)
00 00 00 .. 66 74 79 70 ....ftyp MP4, MOV, HEIC, AVIF (the brand after ftyp says which)

The full table, with offsets and notes, is at /signature/.

Three things that trip people up

Not every signature is at offset 0. TAR’s ustar is at byte 257. ISO images put CD001 at byte 32769. DMG’s signature is a trailer in the last 512 bytes. MP4’s ftyp is at byte 4, after a size field. If the first bytes look like nothing, check the known offsets before concluding the file is unknown.

The same signature can mean several formats. PK opens a plain ZIP and also every modern Office document, Java archive, Android app and EPUB, because they are all ZIP files underneath. Telling them apart means looking at the file names inside the archive: word/document.xml is Word, META-INF/MANIFEST.MF is a JAR, classes.dex is an APK. The same goes for the old Office signature D0 CF 11 E0, shared by DOC, XLS, PPT, MSI and Outlook messages.

Text formats have no signature. JSON, CSV, HTML, source code, shell scripts and configuration files are just text. A hex dump shows readable characters and nothing else. Identifying them means recognising structure ({ and quotes for JSON, <!DOCTYPE for HTML, #!/ for a script), which is where a signature table stops helping and a content-based detector earns its place.

Reading a suspicious file

The practical payoff is spotting a disguise. A file called invoice.pdf whose dump starts 4D 5A is a Windows program, whatever the icon says. A photo.jpg that begins 3C 21 44 4F (<!DO) is an HTML page. A report.docx beginning 25 50 44 46 is a PDF that somebody renamed, probably harmlessly, but it will not open in Word.

The rule is simple: the name is a claim, the bytes are the evidence. When they disagree, believe the bytes, then ask why they disagree.

A worked example

Here is the opening of a file that arrived as scan.pdf:

00000000  4d 5a 90 00 03 00 00 00  04 00 00 00 ff ff 00 00  |MZ..............|
00000010  b8 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 80 00 00 00  |................|
00000040  0e 1f ba 0e 00 b4 09 cd  21 b8 01 4c cd 21 54 68  |........!..L.!Th|
00000050  69 73 20 70 72 6f 67 72  61 6d 20 63 61 6e 6e 6f  |is program canno|

MZ at offset 0, the value 80 00 00 00 at offset 0x3C pointing to the PE header, and the DOS stub text “This program cannot be run in DOS mode” starting at 0x4E. Not a scan, not a PDF: a Windows executable with a PDF icon. The executable safety guide covers what to do next, and the short answer is: nothing, except delete it.

Where to go from here

The signature reference lists the magic bytes for every format in our catalogue, with the offset and the common variants. The per-format guides explain what should come after the signature. And the detector does the whole thing for you, in the browser, when you would rather not read hex before coffee.

← All posts