How to Archive RAW Photos?

As a photographer, preserving your images in their original RAW format may be important. This lets you edit and utilise new image editing software on your original photographs anytime.

We will guide you step by step and tackle one problem at a time.

Be Ready for the Future

As newer image formats and displays emerge, there is an increasing shift towards High Dynamic Range (HDR) imaging. Outdated file formats and legacy editing software may present challenges; however, RAW files contain all the necessary information to ensure future compatibility and preserve image quality.

🤔

Please do not confuse HDR photos with photos edited with HDR effect. True HDR has a vast spectrum of colours and brightnesses that a standard SDR display cannot show. The HDR effect is often done by including information through multiple images in the clipped SDR spectrum, resulting in a picture with all information in the highlights and shadows. However, The HDR effect is not a natural solution. An actual HDR image will always be superior and appear more life-like.

💻

Most HDR displays are from Apple, and other full HDR-ready displays are already on the market, yet they are still very expensive. Samsung is not too far behind with its AMOLED displays. We can see the future shifting towards HDR screens, and this will greatly impact photographers. How will our images look on smartphone screens and tablets? They are the first devices to utilise HDR screens and new HDR image formats.

Canon's HEIF files employ the response curve used in Dolby Vision and HDR10 standards, facilitating cross-HDR compatibility but lacking backwards compatibility with Standard Dynamic Range (SDR) systems. Sony's HEIF files utilise Panasonic's HLG curve, potentially offering the best of both worlds. Notably, the HEIF standard allows for multiple image files within a single container, presenting a potential solution for delivering both HDR and SDR versions without compromise. While there is currently no established workflow for producing HDR images, it is prudent to be mindful that this capability may become available in the future. As HDR displays gain prominence over printed photographs, photographers targeting HDR exhibitions need not concern themselves with producing restrictive SDR versions.

Ok we get it! We should preserve our RAW images and wait for what is to come in the future!

If you are interested in reading more about efficient image formats check out our post here, we cover this topic in our website guide.

Creating a RAW Photo Archive

While cloud archiving is an option, we lean towards offline storage solutions, which are likely to be more cost-effective and efficient in the long run.

Approaches:

Build yourself a cost-effective computer with many internal SSD slots. Make sure that every SSD supports SMART function, this feature will provide you with information about the health of the SSD. SMART can only be used with internally installed SSDs.
Get yourself a RAID1 ready SSD bay. The downside is that you need to buy two identical SSDs which further increases the cost. But true RAID1 is probably the most secure way to archive your files, for us paranoids. It is highly unlikely that both SSDs will fail at the same time, and the remaining SSD will have the full information of your files.
Get yourself a SSD case with an internal SMART readout, this is often a small OLED display with the SSD health information. There are not much on the market currently but the technology is coming.

💾

Be warned that modern NVME SSDs that promise high speed will also generate lots amount of heat that usually cannot be handled in a small case or in a pc rack without strong ventilation. We recommend investing in slower SSDs; they are cost-effective, and as long they have a NAND flash, they will probably last about as long as the more expensive, speedier alternatives. Look at SATA SSDs; they are larger and a lot slower, generate less heat and besides speed, they come with no further downsides.

🙃

The biggest factor for the lifetime of an SSD is its maximum capacity. Once a section is damaged, the SSD will automatically replace it with healthy sectors. Every time this happens, the SSD health will be degraded in your SMART readout. Hence, larger SSDs will last a lot longer than smaller-capacity SSDs. Once an SSD fails to load a file, it is usually too late, and the file cannot be recovered. The general rule is to compare the capacity of the SSD that is used with the remaining health. Also, Keep at least 20% capacity-free to be on the safe side! Once you reach 80% capacity, we recommend buying a larger drive.

The File Size Problem

One significant drawback of the raw format is its large file size, which can consume gigabytes per photo session. While yes AVIF format might be the answer to reduce the file size to a minimum, we also limit our possibilities in the future. So if changing the image format is not an option, we have to look for other ways to reduce the file size.

File Compression

Efficient compression methods are predominantly optimised for text-based content, and without access to a supercomputer for CMIX compression, few alternative options genuinely yield significant improvements.

We recommend looking towards ZPAQ file compression, which is a high-efficiency multi-purpose compression with superior results.

Compressing your images can have a lot of capacity savings, and these days, using a hardware-intensive compression like ZPAQ with modern hardware is not that much of a problem.

📷

We achieved around 20-30% file size reduction through compressing our Sony RAW files in ZPAQ format. We usually archive a folder of RAW images per photo session, not per single file and not everything in one archive.

Additional Benefits of ZPAQ

ZPAQ is lacking the repair functionality as seen in WinRAR and FreeArc, but the overall compression is very robust and forgiving. Even with damaged sectors the archive will always open and provide functionality for the healthy sectors.

🥸

Damaging a compressed archive is harder than you think. Have you ever had this issue in your life? Also, the repair functionality from some other formats is very limited and can only repair a very light degree of damage.

Encryption: ZPAQ offers encryption with a password. This can be very useful if you store your files on a cloud or share your drive frequently with others. We do recommend to encrypt your important work when you might not be the only one to have access to it.

How to use ZPAQ

On mac or linux: Use homebrew!

How to install homebrew:

How to encode a new ZPAQ archive:

Open the terminal and run ls for an overview of your current folder. Run cd to enter your desired folder. Once you have the target location of the files you want to compress, run the following command:

zpaq a nameofarchive.zpaq target -key yourpassword -method 5

nameofarchive: write the name you want to give the new archive
target: what folder or files you want to compress
yourpassword: include your password (optional) otherwise, remove -key

📂

Make sure to use -method 5 (higher compression than 4,3,2,1), as this will determine the file size in the end. Also, remember that files without the -key password will not be encrypted.

⏰

ZPAQ has symmetrical encode and decode behaviour. Given the same hardware, decoding an archive will take exactly as long as it took to encode it.

Decode a ZPAQ archive:

zpaq x nameofarchive.zpaq -key yourpassword

Type zpaq in your terminal for the full command list.

On windows: Use chocolatey

How to install chocolatey:

The commands should be very similar like our homebrew guide above.

How Secure is ZPAQ?

ZPAQ is considered very secure as an archive. The additional amount of hardware that is needed to decode makes it also hard for crawlers to look into your files. ZPAQ does not allow single file extraction (only file structure view); the whole archive must always be extracted first. When you additionally apply a password, then the archive is fully encrypted, and it is impossible to look into the files of the archive without providing the correct password. Cloud services are never really private unless you create your own cloud, and this approach is a good way to protect your privacy.

Read the full story

Already have an account? Sign in