The first ERDAS Manual

Brad Skelton, as a hobby over the years, has been the ERDAS archivist. I enjoy visiting his office and looking at the more than 30 years of ERDAS historical ‘stuff.’ One bit of ‘stuff’ Brad has archived carries a lot of significance, the very first ERDAS Manual. I am embarking on an effort to scan and OCR the document into a searchable PDF that looks exactly like the original document. Two beautiful and multi-talented ERDAS ladies, Candy Billips and Jenn Gazdziak, will help me with the effort. (Truth be told, I could not do it without their help.)

The first ERDAS Manual was written by Mark Finlay, Larry Brantley and Brad Skelton. Andrea Gernazian (Bruce Rado's, one of the three ERDAS founders, wife) proof-read the manual and performed word processing duties along with Melissa Ergle using Wordstar on a Cromemco Z80 microcomputer. The entire document was printed with great effort on a Brother daisy wheel printer (which had the tendency to jam every few pages, or smear printer dust over the pages). Andrea coined the phrase “The Oh, Brother! Printer”.

I quoted from another page of the first ERDAS Manual in the blog post, A Brief History of ERDAS IMAGINE

Below is the introduction page (page 2-1) from the first manual. Note the spelling error ’digitial,’ which everyone missed in 1983.

INTRODUCTION Rev 7.1 / 1 June 1983
A. ERDAS 2300/2400 SYSTEM

I. Introduction
The ERDAS 2300/2400 system is a complete turn key stand alone image processing system for performing image analysis of remotely sensed data, such as LANDSAT or SPOT satellite data, video digitized image data such as aerial photographs, or gridded polygon data such as digitized soils or topography maps. The ERDAS software package that comes with the 2400 system provides many standard image processing functions , a complete and integrated geographic information system (GIS), polygon capture software for use with a table digitizer, and many data base management utilities, all run in a menu-driven and highly interactive environment. The ERDAS 2400 system may be configured to work with several different image processors, ranging from the low cost Digital Graphics CAT system to the more powerful, higher resolution displays such as the Raster Technologies and Gould (DeAnza) image processors. Through the GIS software, many different types of data may be combined in one analysis to provide solutions to problems that cannot be solved using only one type of data. The ERDAS mapping package can produce true scale hard copy map output on the graphics line printer provided as standard equipment. The amount of data that can be processed and stored on the system is limited only by the amount of disk space that is purchased with the system, and more disk space can always be added later if the need arises. As little as 20 megabytes of storage can be used for small scale image processing requirements, or as much as 1000 megabytes can be purchased. A standard 9-track tape drive is also provided with the system to allow for easy access to digital image data from a large number of sources. A number of different CPUs, ranging from the PDP 11/23+, and MICRO-VAX for small scale operations, up to the VAX 11/780, for large scale image processing, are available to satisfy the most demanding applications. An optional table digitizer may also be purchased to allow the system to capture data directly from maps of almost any scale. Multi-user capability is also provided with the ERDAS system. One or many users may perform data processing simultaneously, and in fact, each individual user can run and manage several concurrently running programs so that tasks such as loading a tape can be performed in the background, while other, more interactive tasks, can be done at the console. Data files may be shared by several users or protected from unauthorized access by the file protection system of the RSXll-M or VMS operating system. The ERDAS system is designed to provide a powerful software tool for managing, displaying, and processing digitial image and geographic data, without requiring sophisticated knowledge of computers or programming.

Copyright 1983, 1984 by ERDAS, Inc. All rights reserved

Utah State University Image Standardization Tools

Years ago I ran into a tool built by the Remote Sensing/GIS lab and Utah State University. I recently stumbled onto the tool once again and immediately thought of sharing the tool with you folks.

The website states, “This website provides three tools that create ERDAS Imagine TM spatial models (.gmd format). Each tool creates a different .gmd file providing a slightly different approach to standardization. These tools are designed for Landsat 5 TM and Landsat 7 ETM+ scenes and require, as input, the header file (*.h1) accompanying NLAPS formatted imagery from the USGS at Eros Data Center.”

It offers,

• DN-to-Reflectance Conversion
• COST Atmospheric Correction
• COST without Tau (Dark Object Subtraction) Atmospheric Correction

Check it out and have fun...

Utah State University Image Standardization Tools

Condolences to Dr. Robert Moses' family, friends, and colleagues at PCI Geomatics

I send out my condolences to Dr. Robert Moses' family, friends, and colleagues at PCI Geomatics. Dr. Moses passed away Monday, May 3rd at his home in Chelsea, Quebec, Canada.

PCI Press Release: http://www.pcigeomatics.com/pressnews/Dr_Robert_Moses.pdf

Single core vs. multi-core image processing (vol 2)

As discussed in Single core vs. multi-core image processing (vol 1), I want to present the times for ECW and JPEG2000 compression in ERDAS IMAGINE 2010 v10.1. I tested using the Export to ECW and Export to JPEG2000 commands, running in batch mode.

As you may recall, with the release of ERDAS IMAGINE 2010 this past December 2009, each IMAGINE Advantage license provides the customer the capability to batch process up to four simultaneous processes. If you have more IMAGINE Advantage licenses brokered using a floating license manager, each floating IMAGINE Advantage license can add four more simultaneous processes. In this experiment, I tested with one IMAGINE Advantage for 1 – 4 processes and another floating IMAGINE Advantage license for 5 – 8 processes.

With ERDAS IMAGINE 2010 v10.1 the latest version of the ECW JPEG2000 Codec SDK, v4.1 is used. This new version of the SDK has focused in JPEG2000 improvements. This new version of the ECW JPEG2000 Codec SDK will be made available to the market soon.

As discussed in vol 1, thanks to Dell, Inc. for providing the test system and SAM Inc for inspiring the software performance upgrades. The test system is configured as follows:

Dell Precision T7500
Processor: Dual Quad Core, 2.13GHz with 4MB Cache
System RAM: 4GB 1066MHz
Internal Controller: C4 SATA Non-RAID
Internal Disks: Four 1.5TB 7200 RPM SATA with 3.0Gbps, 16MB Data Burst Cache

External Storage: 15 Bay SAS/SATA Array
Array Controller: PERC 6/E SAS RAID, 256MB Memory
Array Disks: Ten 750GB 7200 RPM 3.0Gbps (note above, array can hold 15 disks)
Configuration: RAID 0

Operating System: Microsoft Windows Server 2008 x64, R2

I ran tests using the 2095 uncompressed strip GeoTIFFs of 5000 x 5000 x 3-bands x 8-bit data (each file is 73,282MB). The target compression ratio for both ECW and JPEG2000 was 15:1.

When processing one image at a time, there was a lot of unused processing capacity. For example, when compressing one ECW file at a time 700MB of total RAM was used; 25% of all eight cores were used; and 20% of the disk bandwidth was used.

By using this extra capacity with multiple simultaneous compressions we more effectively use our total available computing power. When compressing eight ECW files simultaneously 1.3GB of total RAM was used; 75% of all eight cores were used; and 90% of the disk bandwidth was used.

While the ERDAS IMAGINE batch engine does not limit its multi-core processing to the number of cores on the system, I stopped at eight because the disk capacity was filling up. I am convinced I could have pushed the number of simultaneous processes to 10 (or more) if I had added five more disks to fill the array container (but these additional five disks were not available).

ERDAS ER Mapper users will have slightly faster times when compressing one image, But, when running a batch of many images and more especially when running simultaneous processes, ERDAS IMAGINE will be significantly faster. In the future, we will incorporate into ERDAS IMAGINE what makes ERDAS ER Mapper faster when processing one image. Below are the times:

# Processes	ECW Time	JPEG2000 Time
1	2:51:15	5:46:44
2	1:29:27	2:44:44
3	0:43:48	1:58:24
4	0:41:19	1:47:03
5	0:41:37	1:06:55
6	0:35:49	0:55:03
7	0:33:01	0:45:51
8	0:31:53	0:43:14

Looking at these numbers, we can see that a new day is rising for the remote sensing and GIS user community. Think for a moment, one ECW image was being completed every 0.9 seconds when running eight processes, and one JPEG2000 image was being completed every 1.2 seconds. This means you complete a large compression task in less time than it takes to each you to go eat lunch.

If you need a reason for your boss to spend money on an updated system, in the long term a fast processing configuration like the one I have tested is likely the least expensive productivity enhancement they can buy. All they need to do is determine how fast solutions are needed. Is compressing 2095 images over lunch fast enough?

Volume 1 Post Reference: http://field-guide.blogspot.com/2010/04/single-core-versus-multi-core-image.html

Single core vs multi-core image processing (vol 1)

For many years the remote sensing and photogrammetry community have demanded more performance in processing of image data. (I still remember how excited I was when upgrading from an IBM XP with a 10MB hard disk to a Compaq 386 with a 300MB hard disk.) While, some functions continue to be limited by raw processing performance, most applications can receive massive benefit from a higher performance disk configuration. The next few post will discuss this topic.

Thanks to Dell, Inc. for the test system they provided. This is a cost-effective, yet solid performance server configuration. The test system is as follows:

Dell Precision T7500
Processor: Dual Quad Core, 2.13GHz with 4MB Cache
System RAM: 4GB 1066MHz
Internal Controller: C4 SATA Non-RAID
Internal Disks: Four 1.5TB 7200 RPM SATA with 3.0Gbps, 16MB Data Burst Cache

External Storage: 15 Bay SAS/SATA Array
Array Controller: PERC 6/E SAS RAID, 256MB Memory
Array Disks: Ten 750GB 7200 RPM 3.0Gbps
Configuration: RAID 0

Operating System: Microsoft Windows Server 2008 x64, R2

Since some people will read this blog months after these posts, I will not post prices. Notwithstanding my reluctance to mention prices, I will say that the performance boost will pay for itself. I suggest you look for the price this system on Dell’s website.

This effort actually began a few years ago over the Christmas Holidays when Chuck Patterson (SAM Inc. IT), Dell, Haiyan Qu (ERDAS Customer Support) and I got together to understand the performance of Dell’s EqualLogic Storage Array Network (SAN) device running ERDAS IMAGINE processes. That discussion was in the middle of ERDAS’ development of multi-core batch processing (former working name BatchX). With the completion of the multi-core batch processing effort at ERDAS, Dell and I restarted the testing with standard Disk Arrays. We will expand to SANs over the next few months.

As a benchmark to the speed of the system, I ran tests using the standard copy command in a .bat command file to determine the read-write speed of the configuration. Here are the speeds recorded on the system using 2095 uncompressed GeoTIFFs of 5000 x 5000 x 3-bands x 8-bit data (each file is 73,282MB):

0:14:26 Disk Array to Disk Array
0:23:01 Internal Disk to Disk Array
0:27:20 Disk Array to Internal Disk
0:40:01 Internal Disk, different disks
0:53:56 Internal Disk, same disks

As expected, the RAID 0 Disk Array really boosts performance. To be noted, RAID 0 is not redundant, and therefore long-term data storage on this configuration is not a good idea. But, processing data (especially writing) on a RAID 0 gives you powerful a performance boost.

In the next few posts, I will present file creation speeds of JPEG2000 and ECW in ERDAS IMAGINE 2010.1’s implementation of the ECW JPEG2000 Codec SDK v4.1 using this configuration. I hope you will find posts these useful.

Volume 2 post: http://field-guide.blogspot.com/2010/04/single-core-vs-multi-core-image.html

When does a .img image roll-over to a .ige image?

The HFA .img format was designed by ERDAS in the early 1990’s. At that time, Microsoft’s operating system file size limit on disk of 2,147,483,648 bytes (2.1GB) was considered huge. Today, 2.1GB is not considered huge.

In 1999, before Microsoft upgraded their OS to support a true 32-bit file size on disk (2^32 - 1) or 4.2GB), ERDAS released a work-around (tricking the OS) using the .ige spill-over image files (an uncompressible 64-bit offset data file). ERDAS has plans (but no time-table at present) to move to a 64-bit offset .img file, eliminating the need for the .ige files (and allowing DR-RLE compresssion of .img files >2.1GB).

Until the time when a 64-bit offset .img file with DR-RLE support is supported, the mapping community must deal with the question, "when will this file spill-over to a .ige file?" To help the ERDAS IMAGINE community with this challenge, I want to take a moment to explain when spill-over to the .ige file occurs. This will allow you to plan for the transition from a .img to a .ige file when creating new files.

One major thing to understand, when determining if an image needs a .ige file ERDAS counts pixels, not bytes. This is done because when using the DR-RLE compression ERDAS cannot know prior to writing out the image the size of the final compressed image. Therefore, counting pixels rather than bytes is a safe method.

Below are helpful calculations for understanding the transition zone…

Nevertheless, these calculations do not account for all the elements contributing to the final file size. The number of data blocks (determined by the block size) and metadata such as stats, histogram, projection, and attributes (thematic data) will also play into the final file size of a .img file. Therefore, you are not likely to store the full pixel numbers noted below before the rollover.

The formula for calculating the total number of pixels is:
Total bytes allowed / bytes per pixel = total # pixels
For 1-bit data: 2^31 / 0.125 = 17,179,869,184 pixels (17.180 gigapixels)
For 2-bit data: 2^31 / 0.25 = 8,589,934,592 pixels (8.590 gigapixels)
For 4-bit data: 2^31 / 0.5 = 4,294,967,296 pixels (4.294 gigapixels)
For 8-bit data: 2^31 / 1 = 2,147,483,648 pixels (2.147 gigapixels)
For 16-bit data: 2^31 / 2 = 1,073,741,824 pixels (1.073 gigapixels)

Take the square root to get an approximate square array size for the 1-band panchromatic or thematic (color palette) image:
Sqrt of 17,179,869,184 gives a 131,072 X 131,072 x 1-band x 1-bit image
Sqrt of 8,589,934,592 gives a 92,681 x 92,681 x 1-band x 2-bit image
Sqrt of 4,294,967,296 gives a 65,536 x 65,536 x 1-band x 4-bit image
Sqrt of 2,147,483,648 gives a 46,341 x 46,341 x 1-band x 8-bit image
Sqrt of 1,073,741,824 gives a 32,768 x 32,768 x 1-band x 16-bit image

For a 3-band image, the formula is (less than 8-bit not used for multi-layer files):
(Total bytes allowed / # bands) / bytes per pixel = pixels in each band
For 8-bit data: (2^31 / 3) / 1 = 715,827,883 pixels
For 16-bit data: (2^31 / 3) / 2 = 357,913,941 pixels

Take the square root to get an approximate array size for the 3-band image:
Sqrt of 715,827,883 gives a 26,755 x 26,755 x 3-band x 8-bit image
Sqrt of 357,913,941 gives a 18,919 x 18,919 x 3-band x 16-bit image

For a 4-band images:
For 8-bit data: sqrt ((2^31 / 4) / 1) gives a 23,170 x 23,170 x 4-band x 8-bit image
For 16-bit data: sqrt ((2^31 / 4) / 2) gives a 16,384 x 16,384 x 3-band x 16-bit image

Related Blog Post, Fast and Lossless Compression

Tried out ERDAS Communities?

The ERDAS Communities forums are starting to pick up speed. There are quite a few long-time ERDAS (power) users on the forums. Some of these people are former ERDAS employees as well. You will find current ERDAS employees on-line from time-to-time (as work load permits).

Check out ERDAS Communities soon.

It can be found at: http://community.erdas.com/

Trip Report from Perth Australia and Tokyo Japan ERDAS Rocks 2010

Sorry for the late posting this year; we in product management are very much in demand supporting sales on the recent 2010 release and working overtime planning 2010.1, 2011 and beyond. Far more than ‘all is well’ is the mantra early in 2010. Indeed, ERDAS Rocks!

Over the last two weeks, I have been in Perth Australia and Tokyo Japan for the ERDAS Rocks 2010 Tour. The customer and prospect reception was tremendous. I discussed the history of ERDAS IMAGINE and ERDAS ER Mapper as well as the current efforts in fusing of ER Mapper, Image Compressor, and IMAGINE technologies. I also discussed the future of the technology fusing efforts, which will include LPS and new technologies we have yet to openly discuss.

Chris Tweedie discussed the current stage of ERDAS APOLLO and the fusing of ERDAS IWS technology into ERDAS APOLLO. During his demo in both Perth and Tokyo, the audience was amazed at the speed ERDAS APOLLO served imagery and vectors. He did such a good job a PO for ERDAS APOLLO was waiting on us when we arrived in the Perth ERDAS office the following morning.

In Tokyo, the audience had thought web services were supposed to be slow, that slow was the nature web services. Were they every pleasantly surprised; their questions and comments showed a new interest in web services. And to think, what we have in development is faster than what we demoed live in Perth and Tokyo!

To my Chinese friends, Happy New Year! The year of the Tiger is off to a roaring start for ERDAS.

Two Poems to Usher in 2010

Pardon me as I post something non-geospatial to end 2009 and begin 2010. To me, these poems are insightful on why it is important check ourselves every so often to make sure we have our priorities in order. Happy New Year.

THE WAYS
But to every man there openeth,
A way, and way, and away,
A high soul climbs the highway,
And the low soul gropes the low,
And in between on the misty flats,
The rest drift to and fro.

But to every man there openeth,
A high way and a low,
And every mind decideth,
The way his soul shall go.
-John Oxenham

THE WINDS OF FATE
One ship sails East,
And another West,
By the self-same winds that blow,
'Tis the set of the sails
And not the gales,
That tells the way we go.

Like the winds of the sea
Are the waves of time,
As we journey along through life,
'Tis the set of the soul,
That determines the goal,
And not the calm or the strife.
-Ella Wheeler Wilcox

What is LPS eATE?

LPS eATE is an add-on module to LPS for automatically extracting terrain from stereo optical imagery. The difference between LPS eATE and LPS ATE is the precision and accuracy of LPS eATE is much higher than LPS ATE, requiring less manual editing after the automatic processing is complete.

LPS eATE is a scalable tool, scaling from parallel processing using multiple cores on a single system, to distributed processing using multiple cores on multiple systems on your network.

Because of the processing power and the new algorithms a highly dense point cloud (matching on almost every point in the stereo coverage), can be output.

LPS eATE is being released soon to customers as a technology preview. If you have collected elevation points using optical imagery in the past, you will want to test-drive LPS eATE. (And ERDAS would like your feedback on your test drive.)

ERDAS APOLLO Benchmarks with Open Source Image Servers

Chris Tweedie, and ERDAS employee did some image server benchmarking on his own time. The results indicate ERDAS APOLLO is as strong in the image server world as its name indicates.

Chris' tests are an extension to the initial OGC Web Map Service raster benchmarks conducted at FOSS4g 2009 in Sydney. Unfortunately ERDAS were unable to participate. Chris took in upon himself to not only run the same suite of tests but expand and test on a completely different platform.

Have a heavy image server load to lift, you owe it to yourself to have a look. http://blog.webmapper.com.au/image-server-benchmark/

ERDAS Announces Joel Campbell as new President

ERDAS, Inc. just released this to the press:

ERDAS Announces Joel Campbell as new President

Norcross, GA — ERDAS Inc. is pleased to announce that industry veteran Joel Campbell will join the management team as the new President in January 2010, reporting to the ERDAS Board of Directors.

Mr. Campbell is well known and highly regarded throughout the geospatial industry, where he has been a featured speaker, lecturer and trainer for many geospatial organizations around the world. He has over 20 years of experience in the geospatial industry, in a variety of senior roles including sales, business development and product management. His previous employers include GeoEye, Definiens, EarthData and ESRI, along with operating his own consulting firm.

Most recently, Campbell was the Senior Director of Product Management for GeoEye, where he helped manage the company’s expansion into new commercial markets and supported the launch of products from the recently launched GeoEye-1 earth imaging satellite. During more than a decade with ESRI, he held chief leadership and management positions in the U.S. sales operation. These included Director of U.S. Sales, supervising regional offices and several hundred staff members, as well as expanding the company’s presence in the Washington D.C. area.

“Joel brings invaluable experience to ERDAS.” said Juergen Dold, President, Geospatial Solutions Division, Hexagon “We are pleased that he will be joining the management team, where he can apply his diverse experience across the geospatial value chain from capture to delivery.”

“ERDAS offers a compelling suite of products to the geospatial community, providing the world’s most respected desktop remote sensing and photogrammetry solutions, complimented with innovative enterprise solutions,” said Campbell. “I am thrilled to join the management team as we continue to focus on serving the full spectrum of our customers’ needs.”

-----
My comment: Joel met with the ERDAS employees last week for a question and answer session. My fellow ERDAS employees (and I) are excited about Joel becoming an Erdasian. Welcome aboard Joel!