Some GIS Spatial data sources

Across the years, I've been occasionally in need for geospatial data, whether a World Map, or a Satellite backdrop to a small map for a city, a free satellite image or shapefile was very important.
Luckily, many data sources exist in the Internet that have been published by many Space and satellite-owning agencies across the world, and many derived data that can enhance any map.

Some of my more important sources:
Global Land Cover Facility (GLCF)
http://glcf.umiacs.umd.edu/
A great source for many satellite imagery products and datasets, it can be accessed through ftp and provide wealth of file formats and resolutions, must check it.

NASA LandSat GeoCover data set
https://zulu.ssc.nasa.gov/mrsid/mrsid.pl
It provides access to LandSat Mosaics that has been created for most of the globe in 2 time eras (1990 and 2000), it delivers MrSID compressed mosaics with 15m pixel size.

StatSilk List of downloadable free shapefiles
http://www.statsilk.com/maps/download-free-shapefile-maps
Provides a list of downloadable shapefiles that contains many datasets.

Global Administrative Areas
http://www.gadm.org/
Provides shapefiles for administrative boundaries for the countries.

Blue Marble Next Generation
http://earthobservatory.nasa.gov/Features/BlueMarble/
(Direct download) ftp://neo.sci.gsfc.nasa.gov/bluemarble/bmng/bmng_arcview/
The Blue Marble is a gorgeous dataset, it provides 500m imagery for the whole world that is color processed and provides a great basemap backdrop for World Maps.

True Marble (by Unearthed Outdoors)
http://www.unearthedoutdoors.net/global_data/true_marble/download
Another global dataset that is created from LandSat imagery, this time the data has been color and band processed to provide a more natural color and appearance of the imagery, but the full resolution 15m imagery are available for purchase only, in the link you can download the the 250m images.

Global Land Cover 2000 (by the EU JRC)
http://bioval.jrc.ec.europa.eu/products/glc2000/glc2000.php
The Global Land Cover objective is to become a harmonized land cover database over the whole globe, it provides Land Cover maps for the whole world and can be utilized in many analysis functions (mostly on the national level).

Symbolizing Trees Realistically with ArcMap

A simple guide about how to symbolize trees in a campus map in ArcMap, it creates very visually-appealing detailed gardens/landscape/campus maps.
http://www.esri.com/news/arcwatch/0911/tip.html

Introduction to Graphs and some of their algorithms

The Graph Theory and the study of Graphs one of the more advanced topics that I wanted to study for so long, it has applications in the domain of GIS network analysis and routing, so having a basic understanding of the algorithms of Graphs analysis is vital if you are trying to understand what's going on while the Hourglass is revolving.

Here are some articles about Graphs and some of their algorithms that can be a simple intro the Graph theory:

Algorithm of the Week: Shortest Path in a Directed Acyclic Graph
Algorithm of the Week: Bellman-Ford Shortest Path in a Graph
Algorithm of the Week: Dijkstra Shortest Path in a Graph
Algorithm of the Week: Topological Sort of a Graph
Algorithm of the Week: Shortest Path in a Graph
Algorithm of the Week: Graph Best-First Search
Algorithm of the Week: Graph Depth-First Search
Algorithm of the Week: Graph Breadth First Search
Algorithm of the Week: Graphs and their Representation

Visualizing SQL Server 2008 Spatial Data

An interesting article on how to visualize the SQL Server 2008 new spatial data types (geometry and geography) through the Reporting Services Map Control and the ESRI MapIt product.

MSDN Magazine - Visualizing Spatial Data

uDig version 1.2

A project that I've been watching closely and I expect it to become a major player in the GIS FOSS field.

uDig is a programming framework built on the Eclipse Rich Client Platform (RCP) and GeoTools, the Open Source Java GIS Toolkit.

Some time in August they released version 1.2 of uDig, a great release.

ArcGIS 10 Transforms the Way People Use GIS

ESRI has just released ArcGIS 10 ... the biggest large scale upgrade in the ArcGIS line since 2002 when it released the first ArcGIS Desktop product.

"ArcGIS is a complete system that enables you to solve your most challenging issues," said Damian Spangrud, ArcGIS product manager. "This release provides greater ease of use, more powerful analysis tools, additional Web GIS applications, and new mobile platforms. It also promotes collaboration via an online system for using, sharing, and organizing geographic content."

Read more at http://www.esri.com/news/releases/10_2qtr/arcgis10-download.html

Planetizen City ties

If only they include GPS and update their textures automatically, it would be a GIS dream come true :D

The new trend of CIM

In the last few years we have seen the trend of BIM (Building Information Modeling) where a building is represented using real-world entities instead of a bunch of lines.
This trend is now making a shift in the way architects present their ideas and interact with their models (using software like Autodesk Revit, ArchiCAD and Vectorworks instead of AutoCAD and Bentley Microstation).

BIM would be a great answer for Architects.

But more recently a new term was coined, CIM or the City Information model, which aims to transform the urban planners way of handling their plans just like architects.
Some little known applications has started to explore the concept of modeling the whole urban environment and how much information can we extract from such model, in essence how the model would reveal relationships previously little known to the planner or very complex to map such relationships.

CityCAD 
CityCAD from the British Holistic City Software aims to become the city planners main tool (where they previously relied on GIS systems, AutoCAD drawings or even paper maps and census tables).

See CityCAD Software: Stop Masterplanning! (from The Pop-Up City) and Visualizing and Analyzing Plans with CityCAD (from Planetizen).
CityCAD as a software is more into the space and green planning paradigm, and is currently less into the social, socioeconomic and demographic aspects of the city, but quite frankly these are very complex subjects that need much more research on how to model them and how to turn such issues into solvable questions that a model (and the dataset that we feed into that model) can answer.

CityEngine 
CityEngine from Procedural is more into visualizing the 3D city than assisting in the city planning tasks, but a city-optimized visualization tool is very beneficial to urban planners and would bring the city planners some part of the technology that they always wanted, that to visualize the city's current state to better understand it visually, and to visualize the post-plan state that represents their work (to analyze the plan visually).

See New York City In 2259 (from The Pop-Up City).
Some very irrelative point I like about CityEngine is that it uses the Eclipse RCP Platform (which I like but have nothing to do with the software itself).

Other attempts
Maybe one of the older attempts is the The Open Planning Project, it tries to use the Open Source model to create a community of contributers to create an open solution for city planning.

But this list is very incomplete and doesn't touch on the positive aspects of using a systematic way on handling city planning tasks verses the negative impacts using a limited mindset system to solve a very open-ended problem, and the subject needs many more posts other than this one.

[First Image from Wikimedia Commons]

Depreciation plans for ArcGIS 9.3.1 and 9.4 (2)

In the last post I published a list depreciated features in ArcGIS 9.3.1 and 9.4, a shorter list is here:

• ArcGIS Desktop will not support Windows 2000, Visual Basic 6, Visio, Visual Studio 2005 in 9.4 and later will not support Visual Basic for Applications in 9.5
• ArcGIS Engine and ArcReader will not support Sun Solaris in 9.5 (keeping only Windows and Linux)
• ArcInfo Workstation will not support IBM AIX in 9.4 and later will not support Sun Solaris in 9.5, which will be the last release of the product
• ArcSDE will not support Informix 32-bit, SQL Server 2000, PostgreSQL on Red Hat 4, HP-UX on PA-RISC (keeping HP-UX on Itanium), Oracle 9i, Server-side geocoding in 9.4 and later will be the only component in ArcGIS to support Sun Solaris in 9.5
• ArcIMS will not support ArcMap Server, ArcIMS Route Server, IBM AIX, HP-UX in 9.4 and will later not support ArcIMS Metadata Server and Sun Solaris (keeping only Microsoft Windows and Linux) in 9.5, which will be the last release of the product
• ArcGIS Server will not support Enterprise Application Developer Framework (EADF) for Java in 9.4

Depreciation plans for ArcGIS 9.3.1 and 9.4

From the James Fee GIS Blog, a great source to GIS & ESRI info and news, any serious GIS user should check it out.

ESRI released a roadmap of its technologies and platforms to abandon in the next releases of ESRI ArcGIS platform, it includes:

The ArcGIS 9.3.1 Platform (all of the ArcGIS products) is the last version to support:

• Microsoft Windows 2000
• Microsoft Visual Studio 2005 (it is very simple to move from VS2005 to VS2008 and to VS2010)
• Internet Explorer 6 (while it is still used ... it should be phased out of support lifecycles)

ArcGIS Desktop 9.3.1 is the last version to support:

• Crystal Report Wizard, a new reporting engine is used that should be more robust and have better features and performance.
• The Survey Editor Component and Survey Dataset within the Survey Analyst (I don't know what's left from the Survey Analyst)
• CASE tools and the Import Schema Wizard (that means goodbye for Microsoft Visio)
• Microsoft Visual Basic 6 (time to leave such ancient platform)
• Sun Solaris 9 for ArcReader and ArcGIS Engine (This keeps only Solaris 10/Linux/Microsoft Windows Platform)

ArcInfo Workstation 9.3.1 is the last version to support:

• IBM AIX (the only remaining platforms are Microsoft Windows and Sun Solaris)

ArcIMS 9.3.1 is the last version to support:

• ArcMap Server Extension to ArcIMS
• ArcIMS Route server extension
• IBM AIX
• HP-UX (the only remaining platforms are Microsoft Windows, Linux and Sun Solaris)

ArcSDE 9.3.1 (part of ArcGIS Server) is the last version to support:

• 32-bit Informix on the AIX and Solaris platforms
• Microsoft SQL Server 2000
• HP-UX on the PA-RISC processor (support for ArcSDE technology on HP-UX on the Itanium processor is continued)
• PostgreSQL on Red Hat 4 (support for newer version of Red Hat Linux is continued)
• Oracle 9i
• Server-side geocoding in ArcSDE (ArcGIS Server geocode services have replaced the server-side geocoding functionality in ArcSDE)

ArcGIS Server 9.3.1 is the last version to support:

• Enterprise Application Developer Framework (EADF) for ArcGIS Server Enterprise Java release (EADF provided out-of-the box Enterprise Java Beans (EJBs) that were ready to use and deploy across supported J2EE 1.4 application servers. Users can still take advantage of EJBs by following patterns highlighting the ArcGIS Java Web Services Toolkit (AgsJWS) and EJB3)
• Sun Solaris 9 (with the exception of the ArcSDE technology component)

ArcGIS Mobile 9.3.1 is the last version to support:

• Windows CE .NET platform (including Windows CE .NET 4.2 and 5.0)

The ArcGIS 9.4 Platform (all of the ArcGIS products) is the last version to support:

• Sun Solaris (with the exception of the ArcSDE technology component)

ArcGIS Desktop 9.4 will be the last version to support:

• Microsoft Visual Basic for Applications (VBA) (as Microsoft no longer promotes or updates VBA)

ArcInfo Workstation 9.4:

• ArcInfo Workstation will be decoupled from ArcGIS Desktop 9.4
• Windows 7 as the last expected Windows platform upgrade
• Support for connecting to ArcSDE

ArcIMS 9.4 is the last version to support:

• ArcIMS Metadata Server (replacing it with search technology in the ArcGIS Geoportal Extension 9.3.1, and in ArcGIS Server 9.4)

For the next major release after ArcGIS 9.4 (maybe 9.5 or 10.0):

• The last planned release for ArcInfo Workstation on Windows (which will be the last supported platforms)
• The last planned release for ArcIMS on Windows and Linux (which will be the last supported platforms)

A Brief History of ERDAS IMAGINE

If you ever wonders how an application you use evolves through the years and it happens that you are an ERDAS Imagine user, you'll greatly enjoy this article about the history of the ERDAS Remote Sensing platform.

It is a very powerful application and it gained much of its functionality through the years, and you can see how it managed to become one of the field's more powerful players.

PS: The post picture is from the same blog and it shows a late eighties button for ERDAS.

New Global DEM released from METI and NASA

The Ministry of Economy, Trade, and Industry (METI) of Japan and the United States National Aeronautics and Space Administration (NASA) at 29th of June jointly released Version 1 of the ASTER (Advanced Spaceborne Thermal Emission and Reflection Radiometer) Global Digital Elevation Model (GDEM).

Some information about the ASTER instrument:

• Building Agency: METI of Japan
• Carrier Spacecraft: USA NASA’s Terra
• Launch date: December 1999
• Stereoscopic capability: Along-track (no need to pass multiple times over the same area to collect stereo image data)
• Bands available for DEM generation: VNIR (near infrared spectral band)
• DEM base-to-height ratio: 0.6
• Spatial resolution: 15m in the horizontal plane
• Single ASTER VNIR scene size: About 60 km-by-60 km ground area (4,100 samples by 4,200 lines)

Some information about the ASTER GDEM:

• Land surface coverage: between 83°N and 83°S
• Grid cell size: 1 arc-second (30 m)
• Tiles Format: GeoTIFF
• Number of tiles: 22,600
• Single tile size: 1°-by-1°
• Coordinate System Type: Geographic (Lat/Long coordinates)
• Earth Model: WGS84/EGM96 geoid
• Collection Date: 2000-2009
• Vertical accuracy estimate: 20 meters (at 95 % confidence)
• Horizontal accuracy estimate: 30 meters (at 95 % confidence)
• Area of missing data: Areas with constant cloud coverage

This mission is remarkable because it involved the automated processing of the entire 1.5-million-scene ASTER archive, it took approximately one year to complete production of the beta version of the ASTER GDEM using a fully automated approach.

The process included:

1. Stereo-correlation to produce 1,264,118 individual scene-based ASTER DEMs
2. Cloud masking to remove cloudy pixels
3. Stacking all cloud-screened DEMs
4. Removing residual bad values and outliers
5. Averaging selected data to create final pixel values
6. Correcting residual anomalies
7. Partitioning the data into 1°-by-1° tiles.

Previously, the most complete Global DEM publicly available was the SRTM (Shuttle Radar Topography Mission), it has an 80% land coverage (between 60°N and 57°S). The ASTER GDEM expands coverage to 99%; also the SRTM data has a grid cell size of 3 arc-seconds (90 m) outside of the USA borders, despite being captured at 1 arc-second, it was later averaged before being publicly released.

I hope that NASA would release the full resolution 1 arc-second data of the SRTM DEM globally to augment the ASTER GDEM data and allow for a unified DEM which benefits from the ability of the Radar to collect data in the places constantly covered by clouds.

I currently don't know the shoreline data used to outline the DEM data, but I hope they release it as well or use a publicly available one, such as the NGA Prototype Global Shoreline (PGS) data.

A good technical page about the new ASTER GDEM can be found here.

METI and NASA marks version 1 of the ASTER GDEM as “experimental” or “research grade”, but the size and accuracy of the data makes such an initial release a very good step toward a highly enhanced and accurate Global DEM very near.

Download sources:

Earth Remote Sensing Data Analysis Center (ERSDAC) of Japan

NASA’s Land Processes Distributed Active Archive Center (LP DAAC)

I will post shortly about my experience with the download of the new GDEM and its estimated size and usage characteristics.

Sources and News Coverage:

NASA’s Land Processes Distributed Active Archive Center

The Map Room, Astronomy Magazine, Universe Today, BBC News

uDig 1.1.0 released

uDig 1.1.0 is released according to uDig and How 2 Map blogs, I find this application very interesting, as it holds a great potential in becoming the #1 GIS desktop application in the Open Source arena, because I think it is the most user friendly interface and bacause of its programability.

Holes in ArcGIS Polygons from AutoCAD

It remained a mystery to me how Polygons can be exported from CAD to ArcGIS with holes (or donuts, if you prefer to call it so) until recently.

I read in the GIS CAD Interoperability blog that when ArcGIS encouters a block, it considers all the objects inside it to create a multipart object (that is multipart polygon, multipart polyline or multipoint) which is exactly how ArcGIS stores its polygons with holes. 

That means that to create a polygon with hole in AutoCAD you only need to select the polygons creating the polygon and its holes and make a block, this will tell the ArcGIS when importing CAD data that it is a single object with multipart geometry.

Some old Findings

I always find some things that may seem silly to people, yet very useful to other people, I'll keep posting about such things until I find that really interesting thing that no-one can find silly, and keep it to myself.
Some of these silly things:

• If anybody tries to convince you that the GIS is another platform you can learn and lightly use it then later forget about it, be warned, it is a really dangerous plot to turn normal people into geography-loving map-making freaks that speak all of the time about the lack and inaccuracy of the publicly available geo-data and about how accurate the data you personally collect and process compared to your government's data.
• I'm a big loser to stay in the Microsoft developers camp for so long (VB 2/3/4/5/6, VB.NET, then C#), I've heard about Java in as early as 96/97, but never tried it before 2 months ago, and it is a great language to learn and a great community to join
• Apple got some of the greatest designers in the tech industry, I just wish I have enough money to be a full-time mac user (or that they become more affordable in foreign countries).
• Adobe is the graphics designer best friend, Thank God Microsoft is failing in its pathetic attempts to take over the graphics market.
• People using AutoCAD should be using AutoCAD 'til the end of their lives, its the best punishment for learning such a bad program from the beginning, it is one of the worst large applications ever (every industry has better solutions for its users, especially in the Architecture, Landscape and GIS industries), I have to switch over to anything else.