JPEG2000 Performance

When the former ER Mapper, Ltd (purchased in May 2007 by ERDAS) added support for JPEG2000 they noticed JPEG2000 specifications were very broad. As a result, JPEG2000’s speed would not approach ECW speeds for a decades, if ever. Think of it this way, JPEG2000 is a Sports Utility Vehicle (SUV or 4-wheel drive) and ECW is a Formula 1 Ferrari.

The geospatial industry deals with dozens to thousands of images at a time and speed is critical. While geospatial users need the Formula 1 Ferrari, they must drive on public roads (lowest common IT standards), and thus must use the SUV not the Formula 1.

How can SUV owners get more speed from their beast? By using ECW SDK’s J2I file. This ‘index’ file to the JPEG2000 file allows the customer to more rapidly access JPEG2000 data in a viewing technology that uses the ECW SDK to read ECW and JPEG2000 files. The J2I file will significantly improve your performance by fine tuning your existing engine.

The improvement depends on the encoding method. Some encoding methods mandated by less sophisticated IT departments are like pulling a loaded trailer behind the SUV. An engine tuning helps, but slow starts cannot be avoided.

You may ask, how do I create these on JPEG2000 J2I files in ERDAS IMAGINE 2010? For one images at a time, just touch the file with any ERDAS IMAGINE function, and like the aux file, it is automatically created.

For batch creation, use the Edit Image Metadata (aka Image Command Tool). Select one image needed, select ‘Batch’, the select all images needed in the batch tool.

You may be asking, why is there no check-box function for creation the I2G file on Edit Image Metadata? All you need to do is touch the file with any ERDAS IMAGINE function and it is created. Thus only touching the file in Batch Mode is needed.

And with IMAGINE Advantage 2010 batch mode, you can touch up to four at a time with each IMAGINE Advantage floating license you have available.

Back to our cars (file formats); I want a street version Ferrari (has a roof, stereo, leather seats, electric windows, air-conditioning, and of course GPS).


Ref: What are J2I files? and Why compress?

GEOINT 2009 Interview with GISCafe

Here is an interview with me at GEOINT 2009 found in GISCafe.

http://www10.giscafe.com/video/display_media.php?category_id=10197&link_id_display=29522

Transparency with MrSID background data

For many years ERDAS IMAGINE has not provided correct transparency creation with the IMAGINE MrSID Encoders. This was addressed in 9.3 and continues on through the 9.3 Service Packs and into ERDAS IMAGINE 2010. Yet, it is not obvious how to do this.

The user must set the NoData (NullData) in ImageInfo (LayerInfo), then use the IMAGINE MrSID Encoders. Voilà you have a true transparent background data.

ERDAS will make the correct transparency creation ability more prominent and flexible in a future release.

Here is one big thing to remember; ERDAS uses 0 as the transparent value while ESRI uses 255. ERDAS as well as other remote sensing and photogrammetric packages have a black background page while ESRI has a white background page. The more careful image providers now create 8-bit data with a range from 1 to 254 rather than 0 to 255. All 0 and 255 values will be outside of the actual image footprint. This approach allows their customers to be flexible with whatever package they they are using at the moment.

MrSID Performance on MosaicPro and more....

ERDAS has researched, researched, tuned and tuned.... now we are processing 3982 MrSID DOQQs (all the state of Georgia, USA) into one seamless mosaic. The memory usage does not exceed 1GB at any point. We did not get an exact time, and file size.... but it took about 3 days and produced a >700GB output file.

3982 is NOT the maximum number, just the largest number we have tested. The steady memory usage indicates the maximum number is much, much higher than 3982.

One thing to note..... There is a bug in Microsoft's XP x32 operating system affecting very large file creation. Our conversations with Microsoft have lead to the conclusion the bug was introduced between XP SP1 and XP SP2. The bug keeps the operating system from supporting very large files. The maximum file size you can create depends on your system configuration. The largest we have heard of is 70GB. Most systems cannot exceed 35GB.

We have tried rolling back the XP operating systems back to SP1, or back to the original Windows XP, but that has not been successful. Once Windows XP SP2 is loaded, something is not being unloaded. This is a puzzle for both ERDAS and Microsoft.

What is the solution for large file creation in ERDAS IMAGINE 9.3.x? Use Windows XP x64 or Windows Vista. ERDAS will expand operating system support for ERDAS IMAGINE 2010, the versions to be announced later.

** A post-post clarification, if I may. I receive a few emails concerning the XP x32 file size limit. This limit is not related to any specific file format. Rather, it is limited to the operating system not allowing the creation of a single large file of any type. Again, there are some variations, but if you need large files of any type you must move from Windows XP x32. For all its bad press, Vista x32 does not have this problem.

More on Consolidated Mosaic

Well, it has become well known by now that MosaicPro will be included as a native part of IMAGINE Advantage and LPS Core in ERDAS IMAGINE 2010, due Fall 2009. The new MosaicPro had an internal working name of Consolidated Mosaic. MosaicTool, MosaicPro, lessons from ER Mapper and more were used to create the new MosaicPro (http://field-guide.blogspot.com/2009/06/consolidated-mosaic.html). Moreover, we have optimized MosaicPro to be much more RAM and CPU scalable, and that it touches each pixel the fewest times possible.

Yet, let me yell this from the house top.... all ERDAS IMAGINE and LPS customers with current Software Maintenance (SWM) will be receiving this 'new' MosaicPro in version 2010. It is a real gem. You will be impressed.

Here are some early tests:
Dual-core 2 GHz CPU, 4GB RAM, standard disks, Vista Business

Load 3840 images:
9.3.2 16:40 minutes; 463 MB RAM
2010 3:05 minutes; 98 MB RAM

Load 4896 images:
9.3.2 35:06 minutes; 669 MB RAM
2010 4:16 minutes; 113 MB RAM

Cutline Generation on 4896 images:
9.3.2 45:47 minutes; 1,606 MB RAM
2010 01:41 minutes; 125 MB RAM

Mosaicking 127 images:
9.3.2 41:41 minutes
2010 29:11 minutes

Dual Quad-core 2.328 GHz CPU, 16GB RAM, average speed disk array, XP-64
Mosaicking 3982 images:
9.3.2 Not possible
2010 >700GB output file in under 2.5 days

Mosaicking 1147 images:
9.3.2 Not possible
2010 >2.6TB output file in under 5.5 days

All processes used a maximum of just over 800MB of RAM. We researched using more RAM with MosaicPro and found it does not help. This is because we have tuned it so thoroughly that the bottleneck is now the operating system’s thread handling, and hard disk I/O. So, it is time to test fast disks.

But, when using the new added ECW / JPEG2000 direct-write from MosaicPro, MosaicPro will need more RAM. We are working on that as well. We expect we can protect the speed while shrinking down the memory requirement significantly. We expect this will occur in version 2010.1 (Sprilg 2010) or version 2011 (Fall 2010).

Now, we are ready to take MosaicPro to 64-bit. As you can see, when ERDAS goes to 64-bit, we will really take advantage of what 64-bit can really do. Many companies have used a 64-bit port to mask their sloppy coding by accessing more RAM. Not at ERDAS! We know there are a lot of issues surrounding true speed improvement and large file handling. We want our loyal customers to have it all! (http://field-guide.blogspot.com/2009/02/benefits-of-64-bit-architecture-in.html)

Personally, I enjoy looking at ERDAS IMAGINE as a CPU race-car entry. Maybe we can get Intel to test MosaicPro as it does ECW in its performance testing matrix?

Return to Singapore

After 16 years and 11 months, I am returning to Singapore. In September 1992 I traveled to Singapore with Karen Willoughby, Bruce Rado and Jack Dangermond; followed up by Adelaide, Australia. The trip was to attend the 1992 South Asia and OZRI User Group Meetings (UGM). In those days, ERDAS and ESRI held joint international UGMs as most of our distributors sold both companies products and the companies were considered business partners.

At the UGMs I presented a pre–release version of ERDAS IMAGINE 8.0.2. The products I presented were the recently released IMAGINE Digital Ortho and the soon to be released WYSIWYG Map Composer, IMAGINE Vector Module, and the graphic flow chart model builder enhancement to Spatial Modeler script language, Model Maker.

During the same meetings, Jack presented ESRI’s soon to be released product, ArcView. This was ESRI’s move to a graphical user interface. While ERDAS IMAGINE had already made the jump in 1991, ESRI was making the jump in 1992.

When we had some quiet time, Jack demoed ArcView to me and answered my questions. I was impressed at the simplicity of the product. I recognized it a difficult task to take niche technology and create a product to reach out to a wider, non-geospatial trained community. As I discussed ArcView with him, I saw that the product could make that transition.

After we finished looking and discussing ArcView, Jack said he wanted to see IMAGINE Model Maker. Of course, I ran the models that I had planned to run and knew the models would run. Then, as I did to Jack, he asked me to do things that were not planned. He wanted a specific model built from the beginning. I cannot remember the models, but I built and ran them… no problems (ArcView had crashed all over the place on Jack when I asked for specific things).

Almost 17 years later, ArcView has grown up to ArcGIS. For the release of ArcGIS 9.0 in 2004, ESRI copied the Model Maker idea to create Model Builder. ArcView targeted a horizontal expansion of the market and Model Maker targeted a deeper penetration in the existing market. Both product ideas are powerful and have succeeded.

Where will geo-processing be in another 16 years and 11 months? As for ERDAS IMAGINE, imagine that ERDAS IMAGINE 2010 is just the beginning.

IMAGINE Your Future (The Map of the Future is an Intelligent Image)

Here is a corporate video I believe was created from interviews made during the 1991 ERDAS User Group Meeting at SwissHotel; and completed during the spring of 1992. I am in this video at about 3:48. Thanks are due to Rob Luxeder (in the video at 6:08) for finding this jewel.

http://www.youtube.com/watch?v=F2L0dJBHZBw

People I see in the video are: Doug Stowe, Rudolph Richter, Bob Parrott, Kass Green, Richard Lacey, Gail MacAulay, Bill Newland, Roger Hoffer, Oliver Weatherbee, John Althausen (both then students at South Carolina), Bruce Rado, Lawrie Jordan, Brad Skelton, Jeff Dooley, Steve Sperry, Cheryl Brantley, Andy Zusmanis, Andy Bury, Xinghe Yang, Stan Quinn, Donn Rodekohr, Kurt Schwoppe, Barrie Collins, Rob Luxeder, Mike Schlemmer, Fred Woods, Andrea G., Hongyue Lin, Bill Sharp, Lynn Davis and Paul Beaty.

Who can you find?

See the Brief History of ERDAS IMAGINE in this blog here: http://field-guide.blogspot.com/2009/04/brief-history-of-erdas-imagine.html

Reprojecting Compressed Images

Have you ever had a very large compressed image you needed to reproject? The challenge is, you do not want to reproject to a new uncompressed version of the image file and then re-compress.

Have you used Calibration in ERDAS IMAGINE?

Let’s say you have a 2GB 20:1 MrSID compressed color infrared (CIR) image you need to deliver to an ArcGIS user in your organization. The last thing you want to do is resample and re-compress, right? That takes too much disk space and time.

Display the image in the Viewer, select Raster > Geometric Correct > Reprojection. On the dialog that appears, select the new projection and then select the ‘ruler’ on the Geo Correction Tools dialog to calibrate the image.

This process will create a .aux for you MrSID file storing the mathematical model to reproject the image on-the-fly.

Deliver both the MrSID image and the .aux file to your ArcGIS friend. That as fast and simple.

Notes:

1. Starting ERDAS IMAGINE 9.2+, the IMAGINE MrSID encoders write full projection information to the MrSID header. WKT strings are written to MG2, and WKT strings and GeoTIFF tags to MG3 data.
2. With this change, the need for .sdw files and .aux files for map and projection data are no longer needed, but are available.

ERDAS IMAGINE 2010 will have a Shoebox, what is that?

In the new Ribbon Interface for ERDAS IMAGINE 2010 we will add a new feature named, "Shoebox." The idea is to provide the customer a easy to use tool where a list of the data they are likely to use in a project are readily available. The Shoebox will not load the data in a database, but rather create an XML list of the locations of the data.

Hammad Kahn outlines some of the basic concepts of the Shoebox in, “Kicking Around With the Shoebox,” found at labs.erdas.com. Some of you have commented and asked questions below the article.

We may seem a little vague at times when discussing things on labs.erdas.com because it may not be the right time to discuss the more strategic features and future of new tool. Please do not feel we are offended when we side-step a question. In fact, we will use the specific question to help refine the new tool and from time-to-time contact you in private for clarification.

Anyway, please check out Kicking Around With the Shoebox at: http://labs.erdas.com/blog_view.aspx?q=6098