Merging DRGs Manually

By Bill Allen

Note: I'm told this page doesn't work quite right in Netscape 4.x, in that some text may disappear behind illustrations. For the moment, you can retrieve that text with View/Page Source.

Who needs this article? It should interest anyone who wants to merge USGS 1:24000-scale topographic maps, also known as digital raster graphics (DRGs) and as GeoTIFFs (TIFF6 files with embedded geographic information). Specifically, it can be useful to those who...
...aren't using MicroDEM at all, probably because of not having an MS-Windows machine, or...
...want to work only in Photoshop but don't have the system and patience for working with 96Mb PSD and 1.6Gb temp files.

As explained on the "MicroDEM for Artists" page, there are some issues about getting large images out of that program at full resolution (a new workaround was added 23 Feb. 2001). This may not be any problem if you only need a small area (subset before zooming in), or if your end use is to drape the results over a mesh with a certain amount of blurring, anyway. In fact, you may only need to read on if you, too, are a fanatic about retaining full image resolution with enormous, meglomaniac topo merges!

To any Photoshop jockey it would seem natural that the best way to hand-merge four topos would be through using Photoshop layers and channels. However, layers can't use indexed color, and four 5500x6800 RGBA images as layers in a PSD large enough to hold them will challenge most systems. If you are a power user gifted with great patience and intrigued with how well your machine will handle 1.6Gb temp files, have at it! Meanwhile, below is a technique that always keeps the files in indexed color, and runs much more quickly.

Note to new MicroDEM users: To merge DRGs in MicroDEM, use File/Data Manipulation/Merge/Imagery. Cancel from the file open dialog after you have added the last DRG to the merge. After a lot of busyness, you will be given a report but not shown the new merged image. To see that, use File/Open Scanned Map, and look for your file, probably in the C:\mapdata\SATS or C:\mapdata\images folder. It is actually in two files--proprietary .idx and .bn1 formats--that take up about 119Mb for four DRGs with margins left off (say 'Yes' when asked, Clip to quad margins (USGS DRG only)?). The merge is still at full resolution, but, to use it outside MicroDEM, you now have to save it out in a format best suited to its destination (BMP, BMP with World File, or GeoTIFF). Subset first if you don't need the full file, then zoom to the view you want, or to the best resolution view you can get. The BMP will be a huge uncompressed RGB file, while the GeoTIFF, like the original DRGs, uses indexed color and PackBits compression. (A GeoTIFF, if manipulated and saved from an image editor to get better LZW compression or to convert to RGB, will probably lose its georeferencing in the save. A manipulated BMP, however, won't lose its world file. Just change its .bwf file extension to .jwf or .tfw if saving the topo to JPEG or TIFF, respectively.)

Getting Started

Preparing Photoshop

Among various image editors, I have and know only Photoshop, so that's what I used to develop this DRG merging technique, and the only way I can explain it. Here goes...

First, right off, make sure that you have got your memory settings correct for handling huge files. Use the fastest, most open, and least fragmented scratch disk(s) available. Follow the suggestions in the Photoshop Help file under "Improving Performance" for getting the best RAM/physical memory settings. Close any applications you don't absolutely need. If you've been running for awhile, reboot to clear out the RAM and caching. If tight on drive space, clear out unneeded files, including brower caches, trash, dead temp files, etc., etc. You will find that there is more speed and less likelihood of crashing Photoshop if your scratch disks have been defragmented recently, but any drive big enough for this kind of work will take forever to defrag, so now may not be the best time to start it.

You will need a half-gig of combined scratch disk space for merging four DRGs using this technique (using RGB layers requires more than three times as much). In versions of Photoshop with image caching, screen redraw may or may not go faster with settings higher than 1, but higher settings require more HD space. Finally, think of Edit/Purge/All as a special friend to visit often.

If you had video acceleration turned down or off, such as for getting OpenGL to work right, such as ATI accelerators with some applications including MicroDEM, you probably will want to turn it back on full for merging work.

Once you get into the process, arrange your screen so that Photoshop has to do the smallest possible and fewest redraws. Try not to cover the working window with any other windows. Work as close in, and in as small a window, as is practical. Remember that Photoshop pops up dialogs where you last closed them, so close them not over your working window. If you are close to maxing out temp file size, keep an eye on it, Edit/Purge/All frequently (it takes time but can cut the temp file size by as much as half), and, most importantly, bail out before maxing out

Set your units preference to pixels. In the View menu, you will want rulers and guides visible, and guide snap on.

Loading & Quick-Masking the First DRG

Which DRG should you start with in Photoshop? As you have probably noticed by now, all but a few DRGs and corresponding files (DEMs, DOQs, DLGs, etc.) are rotated slightly to the left or right. Presuming that you are merging four DRGs that share a common corner, as we will be here, then, if the rotation is to the left, start with the top right DRG, otherwise with the top left DRG.

Why that DRG first? You will be creating a working file about four times the size of the first file, and it may save a canvas-expanding step if the first file has the most northerly point of the four, and if you then work clockwise. Each of the four will determine one extent of the final image, so which you choose to start with is arbitrary.

Warning: To follow this technique, do NOT crop or rotate these DRGs before finishing the entire merge.

The DRGs used here correspond to the DEMs used in the 10/30m merging tip in 3D Artist issue #43 and on the "MicroDEM for Artists" page: Copper Creek Benches, Lost Spring, Mount Holmes, and Ticaboo Mesa. You can follow along with the same DRGs, or with topos from your own region of interest.

Although DRGs are USGS files, and though the USGS as well as others sell them, they are not available for free download from the USGS itself. Utah DRGs can be downloaded free as .tar.gz files, or purchased on CD, from the Garfield County, Utah DRG listings at commercial vendor GIS Depot, or you can download the uncompressed .tif's free along with their companion .tfw files from the Water Rights Division of the Utah Department of Natural Resources. This Water Rights site also offers decollared versions, which could save some steps here, but this is an unusual service, and the steps themselves are part of the whole point of this tutorial, so "with collars" is the correct choice. (Please be courteous and don't tie up either of these wonderful resources with multiple simultaneous downloads.)

Tip: For more about working with .tar.gz (TAR'd and GZipped) files, see the "MicroDEM for Artists" page.

In this area of UTM zone 12, DRGs rotate slightly to the left, so I start by opening the upper right quad, named "Mount Holmes." Go to Image/Mode/Color Table, count three cells from the right in the top row of the table, click on it, and change it from black/null to a highly visible color such as RGB 255,0,255 (magenta) not already in the 13 other colors used by DRGs. This color will act as your alpha channel for the rest of this process.

Before closing the Color Table, save the modified palette with a name like "drg+1.act". Then close the table with OK and immediately execute a File/Save As to a new file name, such as "o37110g5a.tif" ("a" for alpha channel). It's your choice for byte order (my own "rule of thumb" is to use whichever is appropriate for the system I'm on--PC for PC and Mac for Mac), but enable LZW compression. (Remember, we are staying in indexed color, so don't go to RGB at any point.)

Double-click on the Magic Wand Tool and, under the Magic Wand Options dialog, set Tolerance to 1 and disable Anti-aliased. Now click the tool in an open white area of the collar (the wide margin around the actual map). Most of the collar should become immediately selected, less some pretty big areas, but leave those alone for the moment (and don't select similar).

Click on the Tool palette's "Edit in Quick Mask Mode" icon, or the Channels' palette "Save selection as channel" icon. In the Channels list, if the new channel ("Quick Mask" or "Alpha 1") isn't active, make it so. If the selection is still active, then deselect all. Finally, if the mask color (e.g., 50% red or whatever you change it to, such as magenta) is mainly over the map rather than over the collar, then double-click on the channel and change its options to Color Indicates: Selected Areas.

Don't worry about any imperfections in the collar area, not even the big ones. Instead, look for problems only within the map area. Any place where there is mask color within the map area needs to be deleted. In this first topo, there is only one small spot like this, but some may come up with large swashes of masking across the map. Delete these problems to white. Also check all four corners, as you will need every valid pixel there for a good end result.

What's needed next is to use the Magic Wand (same settings) to select the inside of the mask--the white part of the mask that represents the map. Versions 5.5 and 6 may let you do this, but Photoshop 5.0 (what I have currently), won't let you use this tool this way. So, with the mask active, I select all and copy, do a File/New. Accept the defaults, paste, and Layer/Flatten Image. Any mask intrusions into the map area will immediately stand out, so get those.

Now use the Magic Wand to select just the inside of the map area, Select/Inverse, and delete the collar area all to black. Don't do any other selection editing yet.

Time out: With the map area selected (before inversing), this is a good opportunity to note how big the map area is in pixels, as shown by the Info palette. Rounding up, you'll probably get something like 4550x5720.

Select all and copy. Back in the original image, also select all, make sure you are in the mask channel, and paste. Bingo! There went most of your selection cleanup work. Deselect all, save your working file, and close the scratch window (without ever having saved it).

With the mask channel active and the Index channel visible, zoom in to "fat pixel" view (around 500%, so individual pixels are distinguishable and can be easily selected individually). Now edit the little masking problems by scrolling methodically along the mask's interior border. With just a little practice this can go quite quickly. It will go fastest, and the results may prove useful later, if you don't worry about fringe pixels along the border, and if you leave in non-grid map information that overruns the border. It's the grid elements outside the map, mainly lines and numbers, that we want to make sure get added to the mask.

Tip: If you find that one of the topo's own colors looks confusingly like the color in the 50%-red mask, double-click on the channel and, in Mask Options, click in the color square and choose a new color. Magenta (RGB 255,0,255) at 50% works well in most cases.

Delete to black to add to the mask, and delete to white to subtract from it. (The Rectangular Marquee Tool works best for corralling fat pixels, working in broad rather than exact selections.) Don't be too fussy with the little fringe single-pixels along the outside edge of the border, but do deal with any unmasked grid ticks, numbers, and border "slivers" that are outside the map area, and do be fussy with any mask intrusions through the map border into the map's interior, including small intrusions at the map corners. All topos need work, some more than others. Remember, however, that the map is rotated, so don't fight a natural stairstepping along the map borders (which theoretically should be black but aren't always).

This all can go very fast with just a little practice. Flip the Index channel off and on for quick reference, and, when finished with one tour of the map's border border with the Index channel on, take another quick tour with it off to catch any remaining big problem spots (any changes, however, are usually best done with the map in view). Just remember that a few individual pixels here and there among millions probably aren't going to matter much. Don't spend more than a few minutes on this whole stage. Just get the big stuff.

Creating a Super DRG

With the mask channel active, click on the Channels dialog's "Load channel as selection" icon. Now, with the Index channel active, click on the background color in the Toolbar and set the color to be the same as your 14th index color, which here is RGB 255,0,255 (with indexed color, you cannot set to any color not already in the color table). Hit Delete, drag the mask channel to the Channels dialog's trash can, and save this file for the last time as a master working copy for itself.

Musing: Saving each of the individual masked quads is only for backup. You should still have the original DRG with all the marginal info, so why not reduce file space by deleting compression-defeating data and removing an entire channel before saving this working file? However, don't try to save yet more file space by cropping away the empty collar, as that would lose the 1:1 masking relationship between it and the original DRG. Remember also that any GeoTIFF saved from Photoshop will be an ordinary TIFF, not a GeoTIFF (having lost its georeferencing header and PackBits compression in the process), although having an accompanying .tfw may be good enough. You really do want to leave original USGS DRGs untouched.

File/Save As to a new file, which here I'm calling "MtEllsworthDRG4.tif". This now is what we will use for holding all four DRGs.

Since this is the northernmost of the four DRGs, we can crop this file right to its uppermost map pixel. Zoom in to fat pixels on the upper right corner and drag a guide down from the ruler to the edge above the last pixel. Zoom all the way out and use the Crop Tool to select the entire image, then use Cmd/Ctrl+ [Mac/Win commands] and the scroll bars to zoom in again to fat pixels in the upper right corner and adjust the crop downward to snap to the guide line.

We can't yet know exactly what the other extents will be for the final file, but adding the map area height you noted earlier (5720) to this file's canvas height will be a good guess for leaving enough room to paste the bottom-right map. Do the canvas extension now, after making sure your background color is set to RGB 255,0,255. Save and close the file. If you haven't been freeing up memory as you went along, make sure now to do a Edit/Purge/All.

Open the next DRG, the lower right file, which for me is Ticaboo Mesa. Here is a review of the steps to take, with a few steps cut out or short-cutted from the handling the first file:

1. Go to Image/Mode/Color Table and load your custom palette (drg+1.act) if it isn't already present, and OK out.
2. File/Save As (the DRG) with a new name.
3. Magic-Wand select the collar and create a mask. Set the mask to respresent the (collar) selection area.
4. Make sure that any selection area within the map's borders is deleted to white.
5. Select the whole mask and copy it to a grayscale scratch file and flatten.
6. Select the inside map area, inverse the selection, and delete to black.
7. Select all and copy, move to the DRG mask channel, select all, and paste. Lose the scratch grayscale image.
8. Inspect the mask/map border to clean up big problems.
9. Load the mask as a selection. Now, you can either just copy the inverse selection and save/close this file, or you can do like with the first file and delete the collar to the 14th color, trash the mask, copy the map selection, and save the file.

Once you have the second map placed to your satisfaction, and fixed in place by deselection, you can set the master image's righthand extent. Zoom to fat pixels, place a guide, and crop the entire image to it. Now add the horizontal pixel count you noted earlier to the image's width. Save, close all open files, and Edit/Purge/All.

Repeat the above same steps for the bottom-left DRG (Lost Spring), except for finding the image's bottom extent and cropping accordingly. Then repeat again for the upper-right DRG (Copper Creek Benches) and crop to the image's left-hand extent.

If you computer seems to be really struggling by the third DRG addition, you may want to do a reboot to force clearing out RAM and caches.

Note: As you may have guessed, pasting into the master image's selection is arbitrary. You could just as well paste on top of the master image, and you may want or need to occasionally to do that, but pasting into should generally work slightly better, most especially for maneuvering the fourth map into the merge under the first three.

All that's left is a little cleanup work along the common former borders that now evenly divide the new image. You can scan manually for the very visible 14th color, where there are small gaps in those borders, and repair them by hand, or you can fix them all together in this fashion: Click on the 14th color at one of the edges with the Magic Wand and do a Select/Similar. Now, holding down the Opt/Alt (Cmd/Ctrl in older Photoshop versions) key, deselect the sliver areas along the image's borders. Set your background color to whichever you prefer of the first 13 colors (black is probably best, since all the problems should be where there is supposed to be a grid line), delete, deselect, and save.

If you followed closely with these particular DRGs, and if you cropped down to the first non-mask pixel at each edge of the image, then your final pixel count should be within a few pixels of 9081x11416 in a 13Mb TIFF.

You now have completed the basic four-DRG merge. In fact, if your destination program doesn't use georeferencing, you are finished and you can skip the rest of this page. Note, by the way, that all you have to do to merge this super DRG into a super-super DRG, if your system has that kind of power, is to select the 14th color, inverse the selection, and copy. There's no more masking work needed.

Georeferencing a Super DRG

USGS DRGs, DEMs, etc. in 1:24000 scale are georeferenced by their top-left point, the 0,0 pixel or DEM point. (If an image is two pixels wide and high, its bottom-right pixel has coordinates 1,1.) To create a .tfw world file for our newly merged "super DRG," we need to start with the upper-left pixel of the upper-left DRG, which in this case is Copper Creek Benches. MicroDEM, the USGS DLGViewer, and others can give you that info, but the best way, if you have it, is to use the .tfw that came with that DRG. Open it in any word processor, and save it out as a text file with the exact name you have given the merged image plus the .tfw extension (or .bfw or .jfw if you convert to BMP or JPEG). For my new image, the new world file is named MtEllsworthDRG4.tfw.

The original world file o37110g6.tfw looks like this in NotePad:

No TFW? The same information is embedded in a GeoTIFF header. For instance, using MicroDEM's Analyze/File metadata/GeoTIFF on this file gets the same scale but rounded-up coordinates for the 0,0 pixel. (You don't have to have the DRG open in MD to do that.) You can do the same with the USGS DLG Viewer by opening a DRG and using Tools/Control Center: Metadata/GeoTIFF Tags. (DLG Viewer is a lightweight GIS utility for Windows only, available free from mcmcweb.er.usgs.gov/viewers/dlg_view.html.)

Note that nothing is given here about the coordinate of the top-left pixel of the map itself, only what the coordinates are as if the map extends all the way to the the upper-leftmost (probably white) pixel on the DRG's collar. From what we have here, however, we can calculate the coordinates for the 0,0 pixel of our new super DRG. This only requires looking at fat pixels in Photoshop and doing some elementary math, like this:

Pick out a single pixel that is easily distinguished in the upper right corner of both the new super DRG and the original upper-left DRG. Do a rectangular selection from and including that pixel to the upper-left corner in both images. Use Photoshop's Info dialog to get the exact pixel coordinates. By pointing at the pixel you want (it has to be fat to be sure), you can get its coordinates, which should be exactly one less in each dimension than the measurement width and height of the selection.

The original corner pixel shown right figure above is 329 pixels to the right of the 0,0 pixel, so multiply 329 x 2.4384 = 802.2336, and add that to 521185.644507 (UTM east-west coordinates increase to the east), to get a new easting coordinate of 521987.878107. Since, as it is shown above at left, this pixel is also in the leftmost pixel column (column 0) of the new super DRG, we now have one UTM coordinate calculated.

Note: Decimal UTM values may be insignificant (both MicroDEM and DLG Viewer seem to round them up), but you probably wouldn't be reading this if you weren't up for dealing with picky detail.

Because the common reference pixel we are using is offset vertically in both images, we have to determine the correct offset for row 0 pixels in the new image. Substract the absolute Y location of the pixel in the super DRG (it's at 0,35) from the same pixel's absolute location in the original DRG (it's at 329,336), but add 1 first, so you aren't subtracting the pixel from itself. This is (336+1) - 35 = 302. That is the offset of the 0 row in the new image from the 0 row in the original, so multiply 302 x 2.4384 = 736.3968, and, since this is a move southward, subtract that from 4192590.863260, which gets 4191854.466460.

So my MtEllsworthDRG4.tfw text file looks like...

That's all. One way to test how well this worked is to open an uncompressed copy of your new TIFF in MicroDEM, which will prompt you for the UTM zone and the datum (in this case, zone 12 and NAS-C--not NAS-R!). If MD doesn't seem to change from its default datum of WGS84, proceed anyway and use a word processor like Notepad to check the .tDw file that MD creates. This file should show that MD recognized your choice, even though the dialog display wasn't updated. If not, change the .tDw manually to "NAS-C" and MD will get it right the next time without going through a dialog.

Notes: 1. MicroDEM doesn't read LZW-compressed files, and Photoshop doesn't provide the option of saving with the inferior but legally-unentangled PackBits compression method preferred throughout the GIS community.
2. You will have to tell MD the UTM zone. Refer to the metadata in the original DRG, or to a map of UTM zones.

Page history: This page was first publicly posted on 19 Jan. 2001.

This page still has some work planned for it, from refining current text and images to showing how to do geo-accurate subsetting in Photoshop, so come on back!