![PullSDTS home [page banner]](pullsdts.gif) By Bill Allen This page is home to PullSDTS, a cross-platform[*], open-source utility scripted with Python 2+ for listing and extracting the contents of TAR and "tarball" (.tar.gz, _tar.gz, .tgz) archive files, and especially for inspecting SDTS transfers and doing some specific conversions from DEMs to triangles and grayscale heightfields, all working in UTM NAD27. |
Python Notes Tkinter Notes File Formats SDTS Notes Glossary Index |
What's new: PullSDTS version 2.0 is now in beta with considerable improvements over version 1.x as well as some new capabilities (SDTS and GeoTIFF inspection, SDTS DEM viewing, DEM export as RAW heightfield, and .tif.gz handling), with more yet to come (DEM export as OBJ or RIB mesh, 3D viewing).
> Beta 3 isn't ready yet for public release, but you can see some test RIB output here.
> Beta 2c of 6 Dec. adds the ability to flexibly export any SDTS DEM as a RAW 8- or 16-bit .raw heightfield file that can be loaded into Photoshop, Leveller, etc. This has been tested with 10 and 30m, Level 1 and 2, new and old SDTS DEMs, and in feet or whole or decimal meters. Beta 2c1 of 8 Dec. fixed a minor GUI problem. Beta 2c2 and 2c3 of 27-28 Dec. deals with a small change in Python since version 2.0 that caused PullSDTS to crash under Python 2.2, as reported by Rob Lowrie running on Linux (thanks!). Beta 2c4 of of 5 Jan. fixes a problem Cecilia Ziemer found with sea-level DEMs (thanks!), and also fixes some range-entry problems.
> Beta 1b of 24 Sept. added DEM grayscale viewing (with some fixes posted on 25 and 28 Sept. and 3 Oct.).
> Beta 0i was the first public release of version 2.0 on 10 Sept. 2001.
Contents
|
 |
Pull2 is an all-new program that's built upon modules instead of a version 1's single script, with newly written or much revised code to continue original functions (tarball inspection and extraction) while adding many more. This beta has version 2's new functions for inspecting GeoTIFFs and SDTS transfers and for SDTS DEM grayscale viewing, and now has the first of its planned conversion/export features--RAW heightfield files as 8- or 16-bit. The beta has been released early because of Pull2's major improvements over version 1 in handling gzip and TAR files, and because its expanding capabilities are useful and working now.
The freeware Python 2.x language interpreter must be installed for PullSDTS to work. Python is available from www.python.org as a free download for Win95+/NT+ (5.54Mb) and PowerMac OS9 (6.5Mb binary, 8.5Mb binhex), as RPMs for RedHat Linux 6.1, and as source code for other types of Linux and Unix. (Python comes with Linux, but make sure you have version 2+.)
Code-a-phobia alert! Non-programmers can immediately put PullSDTS to work without knowing anything about, or doing anything with, programming. Python itself must be installed (usually painless except for the big download), but otherwise it can be completely ignored. Then follow the simple PullSDTS installation instructions and treat the pullsdts.py file like any other program: click on it to run it, or shortcut it to your desktop, or add it to your start menu.
The "small print": PullSDTS is provided as is without warranty for fitness for use for any purpose, so use it at your own risk. In particular, there is NO rigorous checking that data has been extracted without error, so this data should NOT be used for mission-critical GIS work, such as search and rescue. PullSDTS is free beta software placed into the public domain with no use or copyright restrictions (see the full license statement). This Web page and its presentation, however, are copyright with all rights reserved (see page bottom for copyright statement), and may not be copied or mirrored, but please do link.
Installation: Create a folder named "DEMpy," such as C:\DEMpy. Drop into it all the files found in the pullsdts.zip download, or grabbed individually from the modules list. (You can use other folder names and any writable drive, but C:\DEMpy is recommended.) Remember that the Python 2.0+ interpreter must installed. If you don't have Python, get it from www.python.org, and let it install itself where and how it wants.
You now can boot PullSDTS like any other program. The best way on Mac and Windows is to shortcut the pullsdts.py script to your desktop.
Getting started: There is nothing complex about using PullSDTS. The program starts with a window (see below) and simple buttons that are visible and active according to the current context. A top row of buttons are for main actions, such as selecting a file or exiting the program. A bottom row of buttons appear as needed for subsidiary actions such as getting an SDTS report. All main buttons have Alt/Opt keyboard shortcuts and tips (click on Tips to turn off tips). There are no menus.
Click [Select File], to get a file selector that will look for these file formats: .tar, .tar.gz,
TARs & tarballs: (Figure 1) If you want to extract a TAR or tarball's contents, click on [Extract All] (you can't extract files individually, yet). Unlike with version 1, CRC32 and size checking are done on gzip'd files, and header checksum checking is done on TAR'd files. Failures are reported but shouldn't usually keep files from being extracted anyway. Note that, however, as with version 1, folder structures are not extracted, and duplicate file names are assigned underscore "_" name extensions to avoid overwriting.
Note: It is not a good idea to place any files inside Pull's own folder (C:\DEMpy or whatever you named it) other than those specified under Modules.
When a TAR or tarball's contents list appears, you will see file names, file sizes in bytes, modification date/time, and order within the TAR file. You will also see a row of buttons above the contents report that you can use to sort the list on those attributes as well as file type.
SDTS files: If an SDTS identity (*iden.ddf) file is selected, PullSDTS will immediately provide an SDTS report. If such a file is found within a TAR or tarball, Pull2 will offer you the opportunity to inspect the USGS SDTS transfer if it is a DEM or DLG (Figure 1). Clicking on the [SDTS Report] button ([SDTS Report/View] if this is a DEM file) quickly gets the report, like seen in Figure 2.
Where to get USGS SDTS DEMs: They were once available free from a USGS government site, but that distribution function was transferred in mid-2001 to GIS Data Depot which, as a commercial site, offers both free files and, for a small fee, broader and faster access. Most of the DEMs used for demonstration in these pages can be found by quad name on the "24K DEM" pages for Garfield, Kane, and San Juan Counties in southeastern Utah, which can be located starting from www.gisdatadepot.com/catalog/US/61068/sublist.html. If you are refused downloads (due to some limit on the number of simultaneous users), or if the download comes in at less than 3Kbps, try again during off-peak hours, or sign up for faster paid access. For free access, a steady 3.59-3.61Kbps seems to be as good as it gets on a fast, solid dial-up connection, though you might get bursts around 3.8 during quieter times, such as Sunday 3am. The site is designed to frustrate robotized and mass downloads, so you have to grab files one by one.
Figures 1 (left) & 2![Fig. 1. Viewing the contents of a TAR or tarball file. Note that you can [Extract All] or, since this is a USGS SDTS file, get a report on the SDTS transfer.](fig01.gif)
![Fig. 2. Viewing a report on an SDTS transfer. Note that a [View DEM] button has appeared since this is a DEM file.](fig02.gif)
|
GeoTIFFs & other TIFFs: Any .tif or .tif.gz file selected will bring up a report about image dimensions, color type, compression method, and image description. If it is a USGS GeoTIFF, the description will include some GIS information (Figure 3). If the file is a .tif.gz, you will see a button at bottom right to extract the TIFF, should you want to. Similarly, if the TIFF isn't gzipped, a button will appear that allows you to do that. Gzipping a USGS GeoTIFF can cut its size in half.
Warning: Although we have found no problems with PullSDTS .tif.gz files, and none have been reported, this is still beta software and so it must be recommended that you back up any file you gzip with Pull2 before deleting the original file, or at least first test Pull's .tif.gz file with another ungzip program.
Where to get GeoTIFFs: USGS DRG topographic maps, moreso than DEMs, can be found all over the Web, especially on sites specializing in one U.S. state, such as provided by a state agency or institution of higher learning. The coverage varies from state to state, and many states have no such coverage. Search at Google with the related keywords, where Google's priortized results should point you to the most popular destination for what you are seeking. An alternative, often slower but sometimes the only free alternative, is to go to the DRG pages for your state at GIS Data Depot, as explained in "where to get USGS SDTS files."
If Pull detects an SDTS DEM in a tarball, the [SDTS Report] button will instead say [SDTS Report/Load] (Figure 1), and clicking on that gets you the regular report plus a button to [Load DEM] (Figure 2). When you click on that, the DEM loading and previewing feature kicks in. It loads the DEM data, converts it to 254 grays plus two colors reserved for voids (blue) and missing data (red), and saves it all out to a temporary file, then loads that into the view window (Figure 4).
![Fig. 4. The DEM view window. Note the [GIF Save] button.](fig04.gif)
The DEM preview window docks and undocks politely with the main window, and docking can be turned on/off with a check box at the main window's upper right. For 10m data, the image is subsampled down to 33% to match the size of 30m DEM views (there is no zoom in or out). The [Save GIF] button lets you save this view out as a GIF file. There is a [Viewing Gamma] slider that doesn't, however, affect the saved image. The docking choice and gamma setting are preserved. View images are left in the temp folder during a session, so DEMs can be revisited with less delay, but are cleaned out when PullSDTS is closed.
The DEM view creation process chugs along nicely on a 650Mhz Pentium 3 when working with 30m DEMs, but is a bit slow with 10m data. While only a relatively few SDTS DEMs have yet been tested, this feature has been tried and found to work with new and old 10m and 30m DEMs, including old 30m Level 1, and including elevation data in feet or whole or decimal meters.
Mac screen shot by Cecilia Ziemer
There is one major addition to PullSDTS 2.0 with the beta 2c version: the ability to save out RAW 8- and 16-bit files in a variety of ways. When the tarball, TAR, or SDTS transfer set selected is a DEM, you will see a [SDTS Report/Load] button, which, when clicked brings up the SDTS report, and then hit [Load DEM]. This gets the 8-bit preview window with a new [Save RAW] button. Clicking on that brings up a dialog with a variety of choices. If you have [Tips] enabled, some of these choices are explained by moving the cursor over the [?] boxes to the right of some sections in the dialog. This brings up a tip window just like when moving the mouse over a button, except that, on the Mac, the tip window stays up twice as long to allow reading of longer tips (on PC it simply stays up until the mouse leaves the [?] box).
The user can work in a choice of feet, decimeters, or whole meters. This Units setting relates to calculations and output rather than to whatever is used in the original file(s). Feet or decimeters will work best (giving better "granularity"), but this only matters when working with exact-unit or equalize-to-range output, explained below.
Byte order can be Mac or PC. It will initially default to your system, but it works just as well either way for Photoshop (just remember which it is for filling in the Photoshop RAW dialog). It may have to be set to "PC" for some destination applications, such as perhaps Leveller.
The default is to not put a 128-byte header into the RAW file, in which case the .raw file name Pull offers will include the pixel dimensions and whether the file is in PC or Mac format.
If a header is used, the offered file name won't have those details. The details can be found in the header, which can be read by dropping the RAW file into a word processor (but don't save it!) or by using the type command in a Windows MS-DOS window. The first header character signals the format "<" for PC and ">" for Mac. A short comment can be included in the header. Be sure to tell the importing program (Photoshop) that there is a 128-byte header or the RAW file won't load correctly.
Grays can be saved to the RAW file as exact elevation values (the result will look dark gray in Photoshop with little or no visible variation without doing an Equalize), or equalized in Pull from the file's own elevation extents to a spread of more than 65,500 values (the default, for best visual results), or equalized according to a user-defined extents range. (Out of 65,535 values possible in 16-bit grayscale, 0-10 and 65,521-65,535 are reserved for fill and void areas.)
The last and first grays choices both can be tiled in Photoshop, but the middle choice can't because the equalization is only relative to that one file itself, just like in Photoshop. If you wanted to merge four DEM RAW grayscales, you can find their common minimum and maximum elevations and use their common greatest extents for all four as being relative to each other, or you could stick with the actual original values, which are relative to Mean Sea Level.
For zero exaggeration, you would use Exact as the choice. For maximum exaggeration, use Equalize to Self. Equalize to Range usually gives less than maximum exaggeration but also can be creatively applied. An example of that is specifying a range specific to one geographic feature, such as a mountain, that will be the focus in the RAW file's destination application.
The [ConUS] button lets the user set the equalization range to the Pull's default maximum elevation extents for the continental United States. These extents are relative to almost all SDTS DEMs available to potential users, but using them may give less than optimum results if your project doesn't happen to include both Mt. Whitney and Death Valley.
If the RAW save operation is completed, key settings are preserved and saved to the pullsdts.ini file, so a whole project once underway can be pursued without having to set the RAW save dialog for each quad.
One can't do much with a 16-bit grayscale file in Photoshop 5.0. Even the Magic Wand tool doesn't work on it. An alternative is to work in 8-bit, for which Pull tries to get the best equivalents to the 16-bit values.
See a RAW file export project walkthrough (103Kb download).
That's it for now for PullSDTS features and how to use them. There are more features to come, most importantly DEM export as mesh to RenderMan RIB, with OBJ export planned, too.
Try as one might to standardize, some things are just going to be different across platforms.
Windows notes:
Change the key script's name to "pullsdts.pyw" to suppress Python's MS-DOS debug window. Because Pull2 is in early beta, it is recommended to run it as .py, not .pyw. (When experiencing problems and reporting a bug, please run the script as .py. See "Bug Reports.)
Mac notes:
1) There appears to be no way to suppress the Python interpreter window when running PullSDTS.
2) PullSDTS is unable to assign original date/times to files extracted on Mac under OS9 (due to what appears to be a problem with Mac Python's time.utime() function). However, you can see date/times in the contents list window when inspecting a TAR file, and can copy that list to the clipboard.
3) You may see a leading colon in the file name field in save-as file dialogs, such as ":1168IDEN.DDF". This error (in Mac Python Tkinter) has no effect, so just ignore it and proceed.
4) If PullSDTS stops with a memory error when loading 10m DEMs on a Mac G4 under either OS 9.0.4 or 9.1, increase Memory Allocation in the Python Interpreter's Preferred box to 16010.
Thanks to Cecilia Ziemer for testing PullSDTS on PowerMac, and for providing Mac screen shots used here.
Linux notes:
Nothing to report. Let me know if you find something that doesn't work, or doesn't seem to work like it should.
Thanks to Rob Lowrie for reporting a bug, now fixed, when running PullSDTS under Python 2.2 on Linux.
![Fig. 6. A button tip, which can be turned off with the [Tips] check box.](fig06.gif)
Button tips: As you sweep the mouse over buttons in PullSDTS 2, you will see a small window appear to one corner of the button with brief information about that button's use. On Windows and Linux, this tip will stay up until the mouse leaves the button, while on Mac it stays up only briefly. You can turn this feature off by clicking on the [Tips] checkbox in the top right of PullSDTS's main window (Figure 1 or 2).
Preferences: PullSDTS 2 will eventually get a preferences dialog, but you already can make some changes by editing the pullsdts.ini inits file, and also by changing some basic settings in the main pullsdts.py module. For instance, in the .ini file you can change the font size used in the report window, and you can specify a different temp folder. And you can change pullsdts.py so that SDTS reports will accumulate automatically into one long report on the system clipboard without ever hitting Pull's [Copy] button.
After you have run PullSDTS at least once, and after you have closed it, you can open the pullsdts.ini file in any plain-text wordprocessor such as NotePad or SimpleText. Carefully change listsize = 8 to 9 or whatever number to change the report window font size, and change temp = C:\Python\DEMpy\tmp to whatever drive and folder you would like, using the folder naming convention for your particular operating system.
The pullsdts.ini file will look something like this:
[fonts] listsize = 8 boldsize = 8 textsize = 9 [main] endian = little xloc = 614 gamma = 1.61 dock = 1 tips = 1 yloc = 192 [RAW] unit = ft rangemin = 14494 fillvalue = 1 rangemax = -282 distribution = range bitcount = 8 [folders] inlast = C:\DemTest temp = C:\DEMpy\tmp outlast = C:\DEMpy
Don't mess with any other .ini settings, and, if your changes appear to cause an error, simply delete the .ini and let PullSDTS create a new one the next time it runs. NOTE: Any changes you make to the .ini while PullSDTS is running will be lost when it quits. Make changes after quitting PullSDTS.
Turning off inits: If for some reason you don't want PullSDTS to use an inits file, you can turn that feature off by opening pullsdts.py and searching for usingINI = 1 near the top of the script. Change that to usingINI = 0, and Pull2 will boot to its default settings every time.
Running from another Tkinter application: If you run PullSDTS from an interactive programming environment (IDE) that itself is Tkinter-based, such as IDLE on Windows or IDE on Mac, be sure in pullsdts.py to change usingIDLE = 0 to usingIDLE = 1 to avoid multiple hazards, especially hanging the Tkinter mainloop command. (And also be sure to change it back to usingIDLE = 0 to run standalone.)
Stacking & automating copying: Whenever Pull2 reports something, you can hit the [Copy] button to put that report on the clipboard, replacing the clipboard's previous contents. If you would like the report copies to accumulate (no clearing), change copyadd = 0 in pullsdts.py to =1. It you would like reports all to be automatically copied to the clipboard, change the adjacent copyauto = 0 statement to =1. Thus, with both set to true, =1, you can inspect one file after another and, when done, paste one long report about all of them into your word processor without ever hitting [Copy].
If you encounter problems, first make sure that you have all seven files present as noted in the Module List. Then check this page (bookmark it now) to see if there are newer versions of those scripts that may have fixed the problem you are experiencing. Next, delete the pullsdts.ini file while PullSDTS isn't running, which may lose problem initialization settings.
Report bugs to Bill Allen. Please tell the version numbers of the operating system, Python, and PullSDTS, but do NOT send attachments. Windows users who encounter problems should use the script ".py" file name extension so that they can try to see and copy any reports from the Python MS-DOS debug window. If that window closes prematurely, and if you are comfortable with running scripts from within a Python IDE (integrated development environment--a script editor), then please try doing that. Remember, that, if your IDE is itself Tkinter-based, to change the usingIDLE constant in pullsdts.py to =1, but change it back to =0 when booting it normally.
While PullSDTS 2 is written to be cross-platform, it has only been tested on Windows ME and PowerMac OS9. Since PullSDTS hasn't been used yet on a truly wide range of TAR files, you are beta testing. If you receive an error message that says:
This is not a recognized TAR file: magic = [something]
Come back to get updates with any bug fixes and with coming new abilities. This being an avocation rather than vocation for the author, there is no time table, and development at times may go quiet, but all this work is very much well underway.
Being in early beta, PullSDTS 2 doesn't yet have most its main target function implemented: converting and exporting DEM data in 3D formats. The next step in Pull's progress will be mesh export to RIB entity and OBJ files, and maybe RAW triangle export if anyone needs it. No other 3D formats are planned for support, though it should be easy enough for others to add format support.
These files must ALL be in the same folder as the main pullsdts.py script. (DEMpy, mentioned here, is a future program that builds upon the PullSDTS code base. Except for pullsdts.py, these modules are designed to be shared by both programs.)
| File | Dated | Purpose & notes | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Files that MUST be in the folder:
Other files you may find in the same folder (leave them): pullsdts.ini - an inits file created & maintained by PullSDTS *.pyc - "compiled" modules (e.g., guinter.pyc) created by Python to boot a little faster tmp or pull_tmp folder - used by PullSDTS to hold its temporary files | |||||||||||||||||||||||
Q: Where does one get the SDTS DEM and GeoTIFF DRG files that PullSDTS works with?
A: See "where to get USGS SDTS files" and "where to get GeoTIFFs" above.
Q: Does (or will) PullSDTS open .zip, .z, or non-TAR .gz files?
A: No, except that it does inspect and report on TIFFs in .tif.gz files, and both opens and stores .tif.gz.
Anyone able to write Python scripts, and who wants to work with other types of special gzip files (perhaps .bmp.gz for example), could readily adapt to their own needs the gzOpen function and gzClass in the DEMpy gztar.py module here.
Q: Does (or will) PullSDTS save .tar or .gz files?
A: No, although it can save and open .tif.gz files, and will soon have other specialty gzip file handling for its own purposes, such as .rib.gz.
Q: In its role as a "tarball" utility, is PullSDTS intended to open only USGS SDTS .tar.gz files?
A: No. Although PullSDTS's primary mission is to work with SDTS DEM transfers as .tar.gz, .tar, and extracted DDF files, it is also evolving to deal with the greater variety of TAR and tarball (.tar.gz and .tgz) files that might be thrown at it. However, "Pull" does not handle complex TAR files, and does not extract folder structures.
Q: Do you have a mailing list?
A: No. Bookmark this page or What's New to watch for updates.
Q: Will PullSDTS be expanded to convert to coordinate systems and datums other than UTM NAD27, and to other DEM/DTED file types?
A: No. That is all well outside the planned scope of the PullSDTS utility and (future) DEMpy application, which are intended only for 3D terrain art projects. For greater functionality and for mission-critical GIS work, Windows users should obtain the indispensable freeware MicroDEM GIS utility. If you don't have Windows 98+/NT+ but really need what MicroDEM does, why not take the money you will save on it (easily worth hundreds of dollars) down to your local mall and buy a cheap Windows machine? (96Mb+ RAM recommended.) The new (Sept. 2001) $2,495 Visual Nature Studio for Windows (www.3dnature.com) also sounds like it will do most of what DEMpy is planned to do, and a great deal more, all at GIS standards with a full mix of GIS data types and datums, and the reported ability to merge huge sets of GIS data.
PullSDTS is as much a useful small utility as it is a demonstration of 1) cross-platform Python/Tkinter programming, and 2) the ease with which any program can work transparently with TAR and gzip files without forcing users to deal with unnecessary trouble and mess (such as comes with opening USGS SDTS DEM transfers of 18 or more DDF files with names that are neither unique nor informative).
Elsewhere here, you will find work-in-progress explanations about SDTS transfers and their DDF files, and about Python and Tkinter programming. The discussion is aimed at inexperienced programmers, as an encouragement for more people to get into writing their own utilities, but includes topics of advanced interest, such as extensive details on parsing DDF files.
This is all part of development work toward a personal project to manage and manipulate huge swaths of digital 3D real estate as SDTS UTM NAD27 DEMs organized in database form. As part of this, PullSDTS will be expanded over time as a testbed for user interface, data inspection, and output features for the larger program, called "DEMpy."
PullSDTS Home Page News:
05 Jan. 2002: Beta 2c4 posted with several small but important fixes in how elevation ranges are handled in 2D RAW export.
27 & 28 Dec. 2001: Beta 2c2 and 2c3 posted with a small fix in two modules to be compatible with Python 2.2.
18 Dec. 2001: New page posted to provide a walkthrough of using Pull's RAW heightfield export feature, 103Kb download.
8 Dec. 2001: Beta 2c1 posted with a minor GUI fix.
6 Dec. 2001: Beta 2c posted with new ability to export SDTS DEMs as 8- or 16-bit RAW files.
3 Oct. 2001: Beta 1b3 posted with fix for an error that prevented the DEMpy module from being used outside its own folder.
28 Sept. 2001: Beta 1b2 posted with fix for a bug in working with some tarball names.
25 Sept. 2001: Beta 1b1 posted with some cosmetic improvements for Mac users.
24 Sept. 2001: Beta 1b posted with an all-new ability to load SDTS DEM data, show a grayscale view, and save that view as a GIF.
16 Sept. 2001: Beta 0j posted with some internal improvements.
10 Sept. 2001: First PullSDTS 2.0 version posted, as beta 0i.
8 April 2001: Version 1.5 posted with many improvements, and this page significantly revised.
10 March 2001: Version 1.4 posted with new compatibility for PowerMac OS9 and other improvements.
1 March 2001: Version 1.3--the first with a graphical user interface (GUI), and this page both first posted, all as an outgrowth of work first posted on 6 Feb. 2001 with the earliest PullSDTS version.
See also "What's New on the Project Site."