Converting Hex Data to Binary for Image and PDF Processing

A government agency needed a reliable solution to convert hexadecimal-encoded data back into binary format for further processing. The data, originating from a legacy document storage system, contained mixed content, including images and PDFs, which required proper identification and extraction.

Challenges Faced

• The client had large volumes of hex-encoded data representing documents and images but lacked a streamlined way to decode and process it.
• Some files were corrupted or incomplete, requiring additional validation steps.
• The system needed to distinguish between images and PDFs automatically to ensure proper handling.
• Manual conversion was not feasible due to the scale of data and processing speed required.

Our Approach & Process

To automate the conversion and processing, we developed a custom Python-based solution that:

• Decoded the hex-encoded data back into binary format using native Python bytes.fromhex().
• Identified file types through custom magic number detection, analyzing byte patterns for PNG, JPEG, and TIFF formats.
• Processed images using PIL (Python Imaging Library) for opening, handling, and saving images, utilizing BytesIO for in-memory operations.
• Reconstructed PDFs using PyMuPDF (fitz module) for PDF validation, handling document resources, and ensuring integrity.
• Logged errors and validation results, providing the client with a clear audit trail of processed files.

Technologies used

• Python & bytes.fromhex() for hex decoding.
• Custom magic number detection for accurate file type identification.
• Pillow (PIL) for image processing with in-memory handling via BytesIO.
• PyMuPDF (fitz) for PDF validation and reconstruction.