Native and fast lossless compression within ERDAS IMAGINE

While ERDAS IMAGINE supports lossless compression through several different means (MrSID, JPEG2000, Packbits, etc.), this discussion focuses on IMAGINE's native run length encoding (RLE) compression. Typical RLE algorithms compress adjacent duplicate pixel values and are a common lossless compression. The implementation of RLE in ERDAS IMAGINE uses a dynamic range run length compression (DR-RLE) rather than the typical RLE. DR-RLE not only compresses adjacent duplicate pixel values but also compresses unused pixels values within the data’s defined data range. This has been the case since the earliest days of .img for both athematic (continuous) and thematic data.

What do we mean when we say it compresses the dynamic range of the data? An example provides a good description. Many modern data types are collected as 11 or 12-bit data, yet we must store it as 16-bit data. If we have an 11-bit dataset, all possible pixel values between value 2048 (unsigned 11-bit +1) and value 65,535 (unsigned 16-bit max) are not used by the dataset, but must take up disk space. When we store these 11-bit data in the .img data format using a DR-RLE compression, all unused pixel values are compressed to a small fraction of its initial 16-bit size.

Elevation data can also benefit from the .img lossless compress because few DEMs need values anywhere near the 16-bit maximum value of 65,535. Thematic data also can benefit. Often have 100 or less categories (classes) and thus must be stored 8-bit. When encoded using DR-RLE, unused values are compressed.

One gottcha discovered by Donn Rodekohr at Auburn Univ., floating-point data stored in the .img DR-RLE should not be processed with intensive math functions in the modeler. Using these floating-point data in complicated models are shown to have some degradation in processing speed. We will address this issue in future versions of IMAGINE. So, keep your DR-RLE data storage limited to integer data for the time-being.

Below are a few size comparisons. From my previous GeoTIFF post (http://field-guide.blogspot.com/2008/05/what-is-wrong-with-my-geotiff.html) you will recall that Packbits is TIFF’s RLE compression. Packbits compresssion is not a DR-RLE compression.

DR-RLE comparison using a 512 x 512 single band .img file:
271KB 8-bit uncompressed (with pyramid layers)
271KB 8-bit RLE compressed (with pyramid layers)
527KB rescaled to 16-bit uncompressed (with pyramid layers)
272KB rescaled to 16-bit RLE compressed (with pyramid layers)

Packbits comparison using a 512 x 512 single band .tif file:
258KB 8-bit uncompressed (without pyramid layers)
258KB 8-bit packbits compressed (without pyramid layers)
514KB rescaled to 16-bit uncompressed (without pyramid layers)
514KB rescaled to 16-bit packbits compressed (without pyramid layers)

But what about the speed? When ERDAS implemented DR-RLE back in 1993, we focused on it being efficient. This effort, and because DR-RLE is so very simple to compress and decompress we see some speed gains from DR-RLE compressed img data over uncompressed .img data. This is the case for today’s processors as well. Having an efficient access mechanism, small data footprint stored in an ultra-simple compression & de-compression format helps processing speeds to disk or to screen.

When an .img file is >2.1GB, IMAGINE rolls the data over in an .ige file and the .img file stores metadata, spatial indices and so forth. The DR-RLE compression is not supported within .ige files. The original design of the .ige is for the simple and fast access of very large uncompressed data files. The .ige design does not allow for any file compression whatsoever. (See: http://field-guide.blogspot.com/2010/04/when-does-img-image-roll-over-to-ige.html)

For files >2.1GB which can stand a little value loss, we suggest our own ECW format or LizardTech’s MrSID format. We have added a ECW read capability to IMAGINE Essentials 9.2 and an encoding capability with IMAGINE Professional 9.2. While IMAGINE supports MrSID Generation 2 and Generation 3 (MG2 & MG3 respectively), we must use MG3 for MrSID files >2.1GB and lossless compression. ECW does not support lossless compression.

Are there any side effects (good or bad) when pulling these .img DR-RLE data into ArcGIS?” Not that I have seen in 15 years. In-fact, ESRI and ERDAS both use the same DR-RLE algorithm. ESRI uses it in GRID, ERDAS in .img. As well, ERDAS provides (writes) many of the raster data objects (RDO) in ESRI products, and .img support is one of them (RLE being part of .img support).

Any positive side effects in ArcGIS? Where ESRI uses the ERDAS / ESRI RDO within their product lines, the user will see the same performance improvements in ArcGIS. In other words, other than the issue with floating-point data, rock-n-roll in ArcGIS.

How do you take advantage of the .img dynamic range run length encoding in ERDAS IMAGINE?
Set “RLE” as your “Default Compression” in the "IMAGINE Image Files (Native)" preference category.
Set “Default” as your “Data Compression” in the "Spatial Modeler” preference category. This will cover Import, Save As, Subset (which is currently a modeler function), Mosaic Tool, MosaicPro, Spatial Modeler and all IMAGINE for files less than 2.1GB. We are planning to make RLE 'on' in the preferences in 9.3, if I hear a demand growing from the user community. Comments?

IMAGINE now supports encoding and decoding DR-RLE, JPEG, LZW, TIFF Packbits, MrSID, JPEG2000, ECW, et al. What other compressions do you folks wish to see within the ERDAS IMAGINE suite?