ERDAS ECW/JP2 SDK GDAL Driver Updated

The ERDAS ECW/JP2 SDK team in Perth Australia has done some work to update the ECW drivers in GDAL. These have been submitted to the GDAL community, and should make their way into a future version of GDAL.

There are multiple changes, but what I want to raise in this post, the performance of GDAL driver was improved in two main areas. While these bottlenecks were not in the ERDAS Desktop and Server product lines, we wanted to ensure our GDAL users had as similar a decoding performance as possible. 

1.  GDAL 1.9 and earlier versions have a performance bottleneck when a single band is decoded line by line. This is not the preferred method of reading ECW data; nevertheless it is done by some folks. 
2. GDAL 1.9 and earlier versions did not use the SDK’s implementation of supersampling, and GDAL’s own implementation was slower. Switching to supersampling using the SDK implementation halved time of the execution. 

While working on this small project, we did notice other areas where we can improve ECW support in the GDAL drivers going forward.

The Perth Team in Australia will improve the GDAL drivers more so when we upgrade GDAL to support the v5 of the SDK and the new ECW v3 file format early in the upcoming new year.

As has been policy, developer tools for decoding ECW and JP2 data using the SDK on desktop continue to be free.

See: http://trac.osgeo.org/gdal/ticket/4822

Q/A From Today's ECW/JP2 SDK Webinar

Below are the questions and answers that we covered in today's ERDAS ECW/JP2 SDK Webinar.

Q1: Compression is always followed by the loss of detail information, right? How does ECW handle that? Which compression rates are useful?

A1: ECW does not support numerically lossless compression at this time. However, if you use a target compression of 4 to 6:1 on RGB data, you should have a visually lossless image at a 2x zoom. See my blog for more information: http://field-guide.blogspot.com/2010/07/understanding-target-compression-ratio.html


Q2: Would you recommend ECW over JP2?

A2: I would recommend ECW over JP2 for all applications where an 8-bit lossy compression works. In the GIS community, a visually lossless setting is usually fine. We did a test when I was at Georgia Tech to see what compression was needed for the majority of users using a statewide dataset. The results are documented on my blog, linked above.

But remember, the ERDAS ECW/JP2 SDK supports JP2 very solidly! Any product using the SDK will have fast JP2 support and even faster ECW support.


Q3: It seems ECW cannot be used on ArcGIS Server and image server, any comment?

A3: ERDAS will release a new product, ECW for ArcGIS Server, spring 2011 to address this very issue. Look for it!


Q4: For aerial photos and orthophotos, which is better for delivering products without losing resolution and data, ECW or JP2?

A4: Pixel resolution is never lost in compression. The first thing you’ll notice in compression is very minor loss of contrast. As you compress further, a little more contrast is lost, and in heavier compression artifacts appear around road lines, edges and other linear features on aerial imagery. The question to ask is, “Do I need visually lossless or numerical lossless compression?” In most GIS applications, visually lossless compression works very well. In some (but not all) remote sensing applications, numerically lossless compression is needed.

The reason data vendors deliver numerically lossless data is that the data provider does not know if you are a GIS user doing imagery analysis or an image expert doing detailed remote sensing. They deliver one version to all, numerically lossless JP2. No problem, the ECW/JP2 SDK supports this!


Q5: Does a standard/report exist for the compression rates for several products (vector map production, base ortho maps, etc.)?

A5: I wish I knew of one, but I do not. This is an area for cartographers and remote sensing researchers to look for research grants. We need a standard.


Q6: How much faster can ERDAS ECW/JP2 SDK make image compression?

A6: I tested and documented this on my blog. Some SDK customers say their applications using v4.2 ECW encode 7x faster than it did in v3.3. Our research indicates most applications will experience greater than 2 x improvements over v3.3. But, JP2 is faster in v4.2 as well. Please see the following link for some tests I ran: http://field-guide.blogspot.com/2010/04/single-core-vs-multi-core-image.html


Q7: What can I provide a customer if he cannot read the ECW format - can I offer him a simple/free ECW reader/decoder?

A7: Many GIS applications can read ECW. ERDAS has a free viewer, ER Viewer that can be used as well. See: http://www.erdas.com/products/ERDASERMapper/ERDASERViewer/Details.aspx


Q8: At times, ECW cannot be opened directly in ArcGIS 10. Any suggestions about how to do that?

A8: There are two possibilities. One, ArcGIS 10 uses a version of GDAL that uses an older version of the ERDAS ECW/JP2 SDK . GDAL has now addressed that and supports v4.2; ESRI will pick up the ball and do the rest.

Two, some organizations who have licenses for the ERDAS ECW/JP2 SDK have added things to the ECW file that are not in compliance with the file format. This will cause problems. You might want to contact the organization that created the data to make sure they do not add any ‘goodies’ to the file format.


Q9: Are there any plans to use GP-GPU for compression?

A9: Yes. I cannot give specifics at this time, but I can say ‘yes.’


Q10: Can I lose accurate pixel value when I compress ECW data?

A10: At the present time, ECW has a visually lossless capability, not a numerically lossless capability.


Q11: Do you have any plans to support the updating of parts of an existing ECW file?

A11: Yes, this is under discussion. I will assume your question indicates an interest and will put your name in the request column. Thanks!


Q12: What would you use for storing floating point data, e.g. DEM data?

A12: Up until now, geospatial people have been using some sort of DR-RLE compression, such as that found in ERDAS’s IMG file format. Going forward, as the floating point JPEG2000 standard takes hold we will see more people use JP2 for lossless floating point data.

The question to ask is whether the maximum 2.5x reduction in file size on encoding worth the slowdown in decoding?


Q13: Which format is useful for signed16-bit data, ECW or JP2?

A13: ECW does not support signed data right now. Therefore, JP2 is the solution at this time.


Q14: Spanish users are having issues with plug-ins when using the Spanish version of the ERDAS ECW/JP2 SDK. Are you going to solve this soon?

A14: The ECW/JP2 SDK should not have a problem with this. If there is a problem with the free plug-in, it is in the plug-in code, not the ECW/JP2 SDK. We are working with Autodesk Germany on the AutoCAD 2011/2011 Plug-in to solve language support issues right now.


Q15: Paul, can you say anything publicly about when the AutoCAD 2010 plug-in will appear?

A15: We are waiting on Autodesk to help with a few installer questions. After that, we are ready to go.


Q16: Does ECW support stereo imagery data?

A16: Not inside the ECW file format. There are some companies that have modified the ECW file format (breaking the file for others), to support stereo parameters inside the file format. We have asked them to work with us to get it formally supported in the formal standard.

Until that time, the stereo parameters can be stored in the external .aux file.


Q17: Why is it necessary to have a plug-in to work with ECW in ERDAS software?

A17: When older versions of ERDAS software were made, ERDAS had not purchased ER Mapper and ECW support was incomplete. ERDAS purchased ER Mapper in May 2007, so that is when native ECW support became standard.


Q18: For unsigned 8-bit, unsigned 16 bit mss data, which format is more useable, ECW or JP2?

A18: ECW is an unsigned 8-bit data format. It can store panchromatic, RGB and multispectral data. 16-bit data should be stored in JP2 at this time.


Q19: Can the ECW format support hyperspectral data?

A19: ECW is a lossy, unsigned 8-bit data format that supports 65,535 bands. Give ECW a try!


Q20: We need to be able to make the background pixels of imagery transparent in ArcMap. ECW doesn't support this, but JP2 does. Is there any way to get the background pixels in ECW files to "disappear" in ArcMap (i.e. set the bg values to 255 or 0)?

A20: ECW added support for transparency in ECW/JP2 SDK v4.1, released in the fall of 2010. We had beta versions available in the summer, but this was too late for ESRI to get it supported in ArcGIS Desktop 10. GDAL is now updated, and the ball is in ESRI’s court. Let them know you want it!

In the ERDAS IMAGINE 2011 Exporter and MosaicPro, we enable you to define whatever value you want to be transparent; you can even define a region using a polygon.

Using ERDAS IMAGINE 2011’s “Create ECW Transparency” feature, you can also add a transparency layer to an existing ECW file via DNs or polygons without recompressing.


Q21: ECW or JP2 - where can I use each format and I how do I know?

A21: Both file formats are broadly accepted in the geospatial industry. If you wish to buy a product that does not support ECW/JP2, tell that product’s manager to give me a call. Remember, the SDK to provide desktop/client ECW, JP2, ECWP reading is free to the software developer.


Q22: Will you post the webinar.

A22: As usual, I forgot to record the webinar! I will re-record and post it later this week.

ERDAS ECW/JP2 SDK Product Page Link: http://www.erdas.com/products/ERDASECWJPEG2000SDK/Details.aspx

YouTube Video: http://www.youtube.com/watch?v=y6rjEgOkhoc&feature=plcp

ERDAS ECW/JP2 SDK, Version 4.2 Available Now!

ERDAS Just released the following Press Release:

Norcross, GA — ERDAS announces the release of a new version of the ERDAS ECW/JP2 SDK, which allows software developers to enable rapid and efficient file handling in the applications they create by including robust support for selected wavelet compression formats and protocols.

The ERDAS ECW/JP2 SDK provides support for the visually-lossless ECW imagery format and the ECWP protocol. ERDAS’ Enhanced Compression Wavelet (ECW) technology compresses bulky imagery files into manageable sizes, enabling faster processing and transfer of imagery and reducing the hardware investment required for the geospatial workflow. Using minimal memory, ECW can compress massive files to smaller sizes faster than other formats, while still preserving the quality of the images. ECW files can then be reopened quickly, in many cases faster than uncompressed imagery. Additionally, multi-resolution level of detail is built into the file, eliminating the need to generate or distribute pyramids or overviews. Enhanced Compression Wavelet Protocol (ECWP) allows streaming of ECW images, enabling rapid delivery of large quantities of imagery over the internet to thousands of users with standard server hardware of just eight cores or less. The SDK also provides support for the numerically-lossless JPEG2000, an ISO-certified compressed image format commonly used for geospatial imagery.

“Massive files are a fact of life in the geospatial industry, and as sensors evolve to collect more information and at higher resolutions, typical file sizes are steadily increasing. While any compression format can make a file somewhat smaller to save money on storage space, the highest hardware costs of dealing with large files result from the need to acquire extra memory and more expensive server technology. Organizations also experience a relentless drain on profit from the productivity reduction caused by a slow workflow,” said Mark Sheridan, Development Manager for the ECW SDK and Image Web Server, ERDAS. “Using ECW and ECWP, applications can compress, decompress and transfer files extremely fast with very little memory and standard server equipment, imparting a comprehensive cost reduction that other formats just cannot offer.”

New for Version 4.2, the ERDAS ECW/JP2 SDK is being offered as three different editions, with tool sets targeted specifically for either desktop or server applications. As in previous versions, the SDK is also available as a free, read-only edition that allows developers to sample the speed and power of ECW and ECWP before upgrading to one of the full editions. All editions of the ERDAS ECW/JP2 SDK are compatible with Microsoft Visual Studio 2005, 2008, and 2010.

ERDAS ECW/JP2 SDK Desktop Read-Only is a free product that provides software developers with tools to add read support for ECW, ECWP and JPEG2000 in their desktop applications.

ERDAS ECW/JP2 SDK Desktop Read-Write provides software developers with tools to add read support for ECW, ECWP and JPEG2000 in their desktop applications. It also provides the tools to compress images into ECW and JPEG2000 data formats.

The newly released ERDAS ECW/JP2 SDK Server Read-Write provides software developers with tools to enable server applications to read ECW, ECWP, and JPEG2000 data, serve ECW and JPEG2000 data, and compress images into the ECW and JPEG2000 data formats.

To learn more about the ERDAS ECW/JP2 SDK, please visit the official product page or attend a free webinar on March 22nd.
For more information about ERDAS or its products and services, please call +1 770 776 3400, toll free +1 877 GO ERDAS, or visit www.erdas.com.

ERDAS ECW JPEG2000 SDK v4.1 and v4.2 (another GDAL Update)

With the recent release of the ERDAS 2011 software portfolio, the ERDAS ECW JPEG2000 SDK team has been able to focus with Frank Warmerdam on the latest ECW SDK GDAL work. I have recently spoken with Frank on the subject. The joint goal is to update to the latest ECW SDK and expand GDALs capabilities in using the ECW SDK, and allowing use of some ECW SDK tools not available to GDAL customers heretofore. I will post more details, when the work is almost complete to discuss all that is now included in GDALs support of the ECW SDK.

But I will say this, if you plan to access the ECW SDK via GDAL, you will be happy. Also, you may want to consider contracting to Frank to help with your implementation. ERDAS is working closely with him, and Frank can help you put rock-solid ECW and JPEG2000 support (using the ECW SDK) in your products.

Do remember, you must get a product license to use the ECW SDK with GDAL. Here is the product page: http://www.erdas.com/products/ERDASECWJPEG2000SDK/Details.aspx

See info on the latest ERDAS ECW/JP2 SDK.

ERDAS ECW JPEG2000 SDK v.4.1 (GDAL Update)

I recently got an update from the ECW team in Perth concerning the ERDAS ECW JPEG2000 SDK v4.1 support in GDAL.

ERDAS is currently compiling the ERDAS ECW JPEG2000 v4.1 SDK against GDAL 1.7.2, in both 32- and 64-bit Windows variants. It is known to compile against 1.5 as well, so using the 1.7.2 driver with 1.6 should probably work (with a bit of fidgeting around). There remain some outstanding issues to address (like opacity channel support), but once the ERDAS 2011 products are released later this quarter, ERDAS will finalize our GDAL driver and give to the work Frank Warmerdam and the GDAL team for incorporation into the main GDAL source.

ECW Support in ArcGIS 10 Desktop

With the release of ArcGIS 10, ESRI’s desktop products support ECW image files natively through GDAL. The GDAL move was part of ESRI's switch from the ERDAS Raster Data Objects (RDO) that ESRI paid ERDAS to use. It was very low cost, as ESRI has historically been a business partner, but ESRI wanted to go in another direction. (The ESRI community can expect a bumpy raster ride for a few releases.)

The GDAL version used in ArcGIS 10 uses an older version of the ERDAS ECW JPEG2000 SDK (ECW SDK), so opacity is not supported and the speed is a little slower than what will be available using the latest version of the ECW SDK, v4.1.

ERDAS is working with GDAL to upgrade GDAL to the latest version of the ECW SDK, scheduled to be complete before Feb 1, 2011. Once the GDAL work is complete, software companies using GDAL (such as ESRI) need only download the Read Only ECW SDK and build in their desktop application as defined by GDAL and according to the ECW SDK EULA for free ECW, ECWP and JPEG2000 desktop read capability.

GDAL is a SDK and the ECW SDK EULA states a SDK cannot deliver the ECW SDK with their SDK. The solution is for GDAL to build against the ECW SDK, but not deliver the ECW SDK. Then the companies using GDAL can simply download the Read Only ECW SDK from ERDAS and put GDAL and the ECW SDK in their desktop software application.

The beauty of the way the Read Only and Full ECW SDKs were put together, once the developer has done the work for ECW, the work to implement ECWP and JPEG2000 may take as few as 30 more minutes’ development time.

If a company wishes their software application to write ECW and JPEG2000 data, they can use the same GDAL approach noted above; and with a purchase of the Full ECW SDK from ERDAS, they add writing ECW and JPEG2000 data.

For ECW support in ArcGIS Server see: http://field-guide.blogspot.com/2011/04/news-release-erdas-releases-new-product.html

For streaming ECW and JP2 data via ECWP in ArcGIS desktop see: http://field-guide.blogspot.com/2011/05/esri-adding-for-ecwp-in-arcgis.html


Why ECW?
Many in the geospatial community correctly think of ECW (Enhanced Compression Wavelet) as a data format that saves disk space. A more powerful aspect of ECW, because of its fundamental mathematical breakthrough enabling fast compression, the same breakthrough provides very fast decompression as well; while only using a small amount of RAM. Whether accessing ECW on a desktop, over a LAN, or over the internet, ECW is very fast. In many cases, ECW is much faster than accessing the imagery in an uncompressed format.

What is ECWP?
Geospatial images can be hundreds or thousands of gigabytes in size. Traditional mechanisms for serving image data over the internet are inadequate when high-speed performance is required. ECWP (Enhanced Compression Wavelet Protocol) leverages the benefits of a the mathematically efficient ECW format and allows the server to stream these large geospatial images to a user's application, rather than sending a regular image ver HTTP, while only using a tiny amount of RAM. JPEG2000 can be streamed with ECWP as well, with higher performance than JPIP.

Why JPEG2000?
JPEG2000 is an ISO-certified wavelet compression image format that is commonly used for geospatial imagery. The format is defined by the Joint Photographic Experts Group. Because JPEG2000 was designed by a committee to do a great many things, it will never be as fast as ECW. Because JPEG2000 compression and decompression technology is similar to ECW, it was simple to add this capability to the ERDAS ECW JPEG2000 SDK, even supporting JPEG2000 streaming within ECWP. But, because JPEG2000 was created by a committee to a great many things, JPEG2000 is unlikely to ever be as fast as ECW.

ECW on Wikipedia

ERDAS Releases New Version of ERDAS ECW JPEG2000 SDK

As many of you know, I am also the Product Manager for the ERDAS ECW JPEG2000 SDK. I am proud of the work the SDK development team in Perth has done. Components of this released version of the SDK are being used in all ERDAS 2010 v10.1 products. There is a discussion forum for the SDK on ERDAS Communities.

The following was released to the press earlier today:

Norcross, GA — ERDAS announces the release of a new version of the ERDAS ECW JPEG2000 SDK, which enables software developers to include robust support for selected wavelet compression formats and protocols in the desktop applications they create.

The ERDAS ECW JPEG2000 SDK provides read and write support for the visually-lossless ECW and numerically-lossless JPEG2000 imagery formats, and read support for the ECWP protocol. Enhanced Compression Wavelet (ECW) is a compressed format designed specifically for geospatial imagery, which can quickly compress and decompress huge images using minimal memory. Enhanced Compression Wavelet Protocol ECWP) allows streaming of ECW images, enabling rapid delivery of terabyte-sized images over the internet using inexpensive server technology. JPEG2000 is an ISO-certified compressed image format that is commonly used for geospatial imagery.

Originally used in the ER Mapper products, this SDK has been implemented in all ERDAS software. ERDAS IMAGINE, LPS and ERDAS APOLLO use the SDK to provide support for these wavelet compression and transmission technologies. For over ten years, this industry-proven SDK has produced stable, high-quality commercial applications.

“The ERDAS ECW JPEG2000 SDK allows any software developer to quickly implement industry-proven support of the ECW and JPEG2000 formats in their desktop applications, easily increasing the value and appeal of those applications to end users. On top of that, we’ve included changes in the latest release that result in remarkable performance increases, allowing your customers to decode ECW and JPEG2000 images and stream via ECWP faster than ever before,” said Mark Sheridan, Director of Engineering at ERDAS and one of the original developers of ER Mapper and ECW technology.

Developers can use the newest SDK to implement read and write support for ECW and JPEG200 images with opacity bands, allowing images to overlay other imagery cleanly without showing compression artifacts around the edges. Also, the SDK can now store compressed blocks delivered over an ECWP stream in the client cache, so previously viewed data can be opened faster. This also reduces the server’s bandwidth requirement, since less data is transferred to the client.

The C++ API provided in the SDK includes new functions for file decoding, rendering and accessing lower-level JPEG2000 features, such as the number of quality layers to include during decoding. Additionally, the API allows developers to retrieve and set options related to opacity, fussy or resilient modes in the decoder, the configuration of J2I index files, and caching parameters.

ERDAS distributes a free version of the SDK that enables read support for ECW, JPEG2000 and ECWP on its product web site.

For more information about ERDAS or its products and services, please call +1 770 776 3400, toll free +1 877 GO ERDAS, or visit www.erdas.com.

Understanding the Target Compression Ratio when using the ERDAS ECW JPEG2000 SDK

It is important to note that when using the ERDAS ECW JPEG2000 SDK, the ECW and JPEG2000 target compression ratio makes no guarantees about the actual output size that will be achieved. Images with certain features (for example, air photos showing large regions of a similar color, like background values, oceans or forests) are easier to compress than others (completely random images).

When compressing images there is a tradeoff between the degree of compression achieved and the quality of the resulting image. The highest rates of compression can only be achieved by discarding some less important data from the image, known as lossy decompression. In the ERDAS ECW JPEG2000 SDK, the target compression ratio is an abstract figure representing your choice in this tradeoff. It approximates the likely ratio of input file size to output file size, given certain parameters for compression.

It is often unwise to try and force each image into a compressed file of the same size. This is because the resulting compressed images will have widely varying quality levels which may reduce their usefulness and any subsequent processing will have decreased value when combining different image tiles of different quality compression together.

Below is a table taken from the ERDAS IMAGINE 2010 Help (to be released later in 2011). This table was created from in-house testing and customer feedback.

Imagery	Target Compression Ratio
Visually Lossless RGB Image	Crisp Image Interpretation (2x zoom)	4:1
Near Visually Lossless RGB Image	Clear Image Interpretation (2x zoom)	6:1
RGB Image	High Quality Printed Maps and typical GIS applications	15:1 to 25:1
RGB Image	Internet or Email Distribution	15:1 to 40:1
Visually Lossless Panchromatic Image	Crisp Image Interpretation (2x zoom)	2:1
Near Visually Lossless Panchromatic Image	Clear Image Interpretation (2x zoom)	3:1
Panchromatic Image	High Quality Printed Maps and typical GIS applications	10:1 to 15:1
Panchromatic Image	Internet or Email Distribution	15:1 to 30:1
Visually Lossless Multispectral Image	Crisp Image Interpretation (2x zoom)	3:1
Near Visually Lossless Multispectral Image	Clear Image Interpretation (2x zoom)	4:1
Numerically Lossless RGB, Panchromatic, and Multispectral (JPEG 2000 Only)	Imagery with perfect numeric reconstruction	1:1

Single core vs. multi-core image processing (vol 2)

As discussed in Single core vs. multi-core image processing (vol 1), I want to present the times for ECW and JPEG2000 compression in ERDAS IMAGINE 2010 v10.1. I tested using the Export to ECW and Export to JPEG2000 commands, running in batch mode.

As you may recall, with the release of ERDAS IMAGINE 2010 this past December 2009, each IMAGINE Advantage license provides the customer the capability to batch process up to four simultaneous processes. If you have more IMAGINE Advantage licenses brokered using a floating license manager, each floating IMAGINE Advantage license can add four more simultaneous processes. In this experiment, I tested with one IMAGINE Advantage for 1 – 4 processes and another floating IMAGINE Advantage license for 5 – 8 processes.

With ERDAS IMAGINE 2010 v10.1 the latest version of the ECW JPEG2000 Codec SDK, v4.1 is used. This new version of the SDK has focused in JPEG2000 improvements. This new version of the ECW JPEG2000 Codec SDK will be made available to the market soon.

As discussed in vol 1, thanks to Dell, Inc. for providing the test system and SAM Inc for inspiring the software performance upgrades. The test system is configured as follows:

Dell Precision T7500
Processor: Dual Quad Core, 2.13GHz with 4MB Cache
System RAM: 4GB 1066MHz
Internal Controller: C4 SATA Non-RAID
Internal Disks: Four 1.5TB 7200 RPM SATA with 3.0Gbps, 16MB Data Burst Cache

External Storage: 15 Bay SAS/SATA Array
Array Controller: PERC 6/E SAS RAID, 256MB Memory
Array Disks: Ten 750GB 7200 RPM 3.0Gbps (note above, array can hold 15 disks)
Configuration: RAID 0

Operating System: Microsoft Windows Server 2008 x64, R2

I ran tests using the 2095 uncompressed strip GeoTIFFs of 5000 x 5000 x 3-bands x 8-bit data (each file is 73,282MB). The target compression ratio for both ECW and JPEG2000 was 15:1.

When processing one image at a time, there was a lot of unused processing capacity. For example, when compressing one ECW file at a time 700MB of total RAM was used; 25% of all eight cores were used; and 20% of the disk bandwidth was used.

By using this extra capacity with multiple simultaneous compressions we more effectively use our total available computing power. When compressing eight ECW files simultaneously 1.3GB of total RAM was used; 75% of all eight cores were used; and 90% of the disk bandwidth was used.

While the ERDAS IMAGINE batch engine does not limit its multi-core processing to the number of cores on the system, I stopped at eight because the disk capacity was filling up. I am convinced I could have pushed the number of simultaneous processes to 10 (or more) if I had added five more disks to fill the array container (but these additional five disks were not available).

ERDAS ER Mapper users will have slightly faster times when compressing one image, But, when running a batch of many images and more especially when running simultaneous processes, ERDAS IMAGINE will be significantly faster. In the future, we will incorporate into ERDAS IMAGINE what makes ERDAS ER Mapper faster when processing one image. Below are the times:

# Processes	ECW Time	JPEG2000 Time
1	2:51:15	5:46:44
2	1:29:27	2:44:44
3	0:43:48	1:58:24
4	0:41:19	1:47:03
5	0:41:37	1:06:55
6	0:35:49	0:55:03
7	0:33:01	0:45:51
8	0:31:53	0:43:14

Looking at these numbers, we can see that a new day is rising for the remote sensing and GIS user community. Think for a moment, one ECW image was being completed every 0.9 seconds when running eight processes, and one JPEG2000 image was being completed every 1.2 seconds. This means you complete a large compression task in less time than it takes to each you to go eat lunch.

If you need a reason for your boss to spend money on an updated system, in the long term a fast processing configuration like the one I have tested is likely the least expensive productivity enhancement they can buy. All they need to do is determine how fast solutions are needed. Is compressing 2095 images over lunch fast enough?

Volume 1 Post Reference: http://field-guide.blogspot.com/2010/04/single-core-versus-multi-core-image.html

Single core vs multi-core image processing (vol 1)

For many years the remote sensing and photogrammetry community have demanded more performance in processing of image data. (I still remember how excited I was when upgrading from an IBM XP with a 10MB hard disk to a Compaq 386 with a 300MB hard disk.) While, some functions continue to be limited by raw processing performance, most applications can receive massive benefit from a higher performance disk configuration. The next few post will discuss this topic.

Thanks to Dell, Inc. for the test system they provided. This is a cost-effective, yet solid performance server configuration. The test system is as follows:

Dell Precision T7500
Processor: Dual Quad Core, 2.13GHz with 4MB Cache
System RAM: 4GB 1066MHz
Internal Controller: C4 SATA Non-RAID
Internal Disks: Four 1.5TB 7200 RPM SATA with 3.0Gbps, 16MB Data Burst Cache

External Storage: 15 Bay SAS/SATA Array
Array Controller: PERC 6/E SAS RAID, 256MB Memory
Array Disks: Ten 750GB 7200 RPM 3.0Gbps
Configuration: RAID 0

Operating System: Microsoft Windows Server 2008 x64, R2

Since some people will read this blog months after these posts, I will not post prices. Notwithstanding my reluctance to mention prices, I will say that the performance boost will pay for itself. I suggest you look for the price this system on Dell’s website.

This effort actually began a few years ago over the Christmas Holidays when Chuck Patterson (SAM Inc. IT), Dell, Haiyan Qu (ERDAS Customer Support) and I got together to understand the performance of Dell’s EqualLogic Storage Array Network (SAN) device running ERDAS IMAGINE processes. That discussion was in the middle of ERDAS’ development of multi-core batch processing (former working name BatchX). With the completion of the multi-core batch processing effort at ERDAS, Dell and I restarted the testing with standard Disk Arrays. We will expand to SANs over the next few months.

As a benchmark to the speed of the system, I ran tests using the standard copy command in a .bat command file to determine the read-write speed of the configuration. Here are the speeds recorded on the system using 2095 uncompressed GeoTIFFs of 5000 x 5000 x 3-bands x 8-bit data (each file is 73,282MB):

0:14:26 Disk Array to Disk Array
0:23:01 Internal Disk to Disk Array
0:27:20 Disk Array to Internal Disk
0:40:01 Internal Disk, different disks
0:53:56 Internal Disk, same disks

As expected, the RAID 0 Disk Array really boosts performance. To be noted, RAID 0 is not redundant, and therefore long-term data storage on this configuration is not a good idea. But, processing data (especially writing) on a RAID 0 gives you powerful a performance boost.

In the next few posts, I will present file creation speeds of JPEG2000 and ECW in ERDAS IMAGINE 2010.1’s implementation of the ECW JPEG2000 Codec SDK v4.1 using this configuration. I hope you will find posts these useful.

Volume 2 post: http://field-guide.blogspot.com/2010/04/single-core-vs-multi-core-image.html

JPEG2000 Performance

When the former ER Mapper, Ltd (purchased in May 2007 by ERDAS) began work in 2003 to add support for JPEG2000 (JP2) to the ECW SDK, ER Mapper noticed JP2 specifications were very broad. As a result, JP2’s speed would not approach ECW speeds for decades, if ever. Think of it this way, JP2 is a Sports Utility Vehicle (SUV or 4-wheel drive) and ECW is a Formula 1 Ferrari.

The geospatial industry deals with dozens to thousands of images at a time and speed is critical. While geospatial users need the Formula 1 Ferrari, they must drive on public roads (lowest common IT standards), and thus must use the SUV not the Formula 1.

How can SUV owners get more speed from their JP2 beast? By using the ERDAS ECW/JP2 SDK’s J2I file. This ‘index’ file to the JP2 file allows customers to more rapidly access JP2 data in a viewing technology that uses the ECW SDK to read ECW and JP2 files. The J2I file will significantly improve your performance by fine tuning your existing engine.

The improvement in speed depends on the encoding method. Some encoding methods mandated by less geospatially sophisticated IT departments are like pulling a loaded trailer behind the SUV. An engine tuning helps, but slow starts cannot be avoided.

You may ask, how do I create these on JP2 J2I files in ERDAS IMAGINE 2010? For one image at a time, just touch the file with any ERDAS IMAGINE function, and like the aux file, it is automatically created.

For batch creation, use the Edit Image Metadata (aka Image Command Tool). Select one image needed, select ‘Batch’, the select all images needed in the batch tool.

You may be asking, why is there no check-box function for creation the J2I file on Edit Image Metadata? All you need to do is touch the file with any ERDAS IMAGINE function and it is created. Thus only touching the file in Batch Mode is needed.

And with the IMAGINE Advantage 2010 batch mode, you can touch up to four images at a time with each IMAGINE Advantage floating license you have available.

Back to our cars (image file formats); what I want from my GIS and remote sensing data format and software is a street version Ferrari (has a roof, stereo, leather seats, electric windows, air-conditioning, and of course GPS).


Ref: What are J2I files? and Why compress? 

See info on the latest ERDAS ECW/JP2 SDK.