As you may recall, with the release of ERDAS IMAGINE 2010 this past December 2009, each IMAGINE Advantage license provides the customer the capability to batch process up to four simultaneous processes. If you have more IMAGINE Advantage licenses brokered using a floating license manager, each floating IMAGINE Advantage license can add four more simultaneous processes. In this experiment, I tested with one IMAGINE Advantage for 1 – 4 processes and another floating IMAGINE Advantage license for 5 – 8 processes.
With ERDAS IMAGINE 2010 v10.1 the latest version of the ECW JPEG2000 Codec SDK, v4.1 is used. This new version of the SDK has focused in JPEG2000 improvements. This new version of the ECW JPEG2000 Codec SDK will be made available to the market soon.
As discussed in vol 1, thanks to Dell, Inc. for providing the test system and SAM Inc for inspiring the software performance upgrades. The test system is configured as follows:
Dell Precision T7500
Processor: Dual Quad Core, 2.13GHz with 4MB Cache
System RAM: 4GB 1066MHz
Internal Controller: C4 SATA Non-RAID
Internal Disks: Four 1.5TB 7200 RPM SATA with 3.0Gbps, 16MB Data Burst Cache
External Storage: 15 Bay SAS/SATA Array
Array Controller: PERC 6/E SAS RAID, 256MB Memory
Array Disks: Ten 750GB 7200 RPM 3.0Gbps (note above, array can hold 15 disks)
Configuration: RAID 0
Operating System: Microsoft Windows Server 2008 x64, R2
I ran tests using the 2095 uncompressed strip GeoTIFFs of 5000 x 5000 x 3-bands x 8-bit data (each file is 73,282MB). The target compression ratio for both ECW and JPEG2000 was 15:1.
When processing one image at a time, there was a lot of unused processing capacity. For example, when compressing one ECW file at a time 700MB of total RAM was used; 25% of all eight cores were used; and 20% of the disk bandwidth was used.
By using this extra capacity with multiple simultaneous compressions we more effectively use our total available computing power. When compressing eight ECW files simultaneously 1.3GB of total RAM was used; 75% of all eight cores were used; and 90% of the disk bandwidth was used.
While the ERDAS IMAGINE batch engine does not limit its multi-core processing to the number of cores on the system, I stopped at eight because the disk capacity was filling up. I am convinced I could have pushed the number of simultaneous processes to 10 (or more) if I had added five more disks to fill the array container (but these additional five disks were not available).
ERDAS ER Mapper users will have slightly faster times when compressing one image, But, when running a batch of many images and more especially when running simultaneous processes, ERDAS IMAGINE will be significantly faster. In the future, we will incorporate into ERDAS IMAGINE what makes ERDAS ER Mapper faster when processing one image. Below are the times:
| # Processes | ECW Time | JPEG2000 Time |
1 | 2:51:15 | 5:46:44 |
2 | 1:29:27 | 2:44:44 |
3 | 0:43:48 | 1:58:24 |
4 | 0:41:19 | 1:47:03 |
5 | 0:41:37 | 1:06:55 |
6 | 0:35:49 | 0:55:03 |
7 | 0:33:01 | 0:45:51 |
8 | 0:31:53 | 0:43:14 |
Looking at these numbers, we can see that a new day is rising for the remote sensing and GIS user community. Think for a moment, one ECW image was being completed every 0.9 seconds when running eight processes, and one JPEG2000 image was being completed every 1.2 seconds. This means you complete a large compression task in less time than it takes to each you to go eat lunch.
If you need a reason for your boss to spend money on an updated system, in the long term a fast processing configuration like the one I have tested is likely the least expensive productivity enhancement they can buy. All they need to do is determine how fast solutions are needed. Is compressing 2095 images over lunch fast enough?
Volume 1 Post Reference: http://field-guide.blogspot.com/2010/04/single-core-versus-multi-core-image.html
6 comments:
So, how does this differ from doing the same thing with a bash script on linux with gdal_translate?
http://linfiniti.com/2009/12/batch-convert-a-directory-of-tiffs-to-ecw/
There are several differences, but I will discuss but one. The batch wizard within ERDAS IMAGINE is designed to provide the power of scripting to users who have do not have the skills to write scripts, or there is no significant value in them spending the time to learn how to writ scripts. While ERDAS maintains a solid suite of tools for the scientist, ERDAS has always strived to create a simple approach to bring the benefits of remote sensing to the larger mapping market.
In my opinion, Bourne-again shell (BASH) scripting is for 5% of the mapping market; ERDAS’ COTS solutions strives to help the remaining 95%
Bash is not as widespread, but the same could be done in python on multiple OS platforms with or without os.system calls.
On the other hand, having seen the current "terror" the command line/scripting holds for otherwise competent gis users, I can see why the gui batch tool would be useful.
You mentioned that 4.1 of the ECW SDK will be released soon. Are you able to disclose the various licensing options at this time? We currently use an ancient version from well before the ERDAS acquisition of ER Mapper (something like 2.4) and that had a 500MB limit of compression for free users, and commercial users had a one-off fee with no additional royalties.
Giga, I cannot discuss the details on my Blog, as it is not released to the public, yet. I can say there will be a very fast Desktop Read Only for ECW and JPEG2000 SDK, which will be free. Please look to the ERDAS.com site for many more details this month.
Post a Comment