Managing And Integrating Heterogeneous Geospatial Data Sources

Combining Diverse Geospatial Data

Integrating varied geospatial data sources presents challenges due to differences in formats, standards, coordinate systems, data models, accuracy, and precision. Successful integration requires transforming datasets to match coordinate reference systems, conflating data to a common schema, leveraging interoperability standards, utilizing metadata, and employing customized tools and technologies. When thoughtfully combined, heterogeneous geospatial data sources can be fused into unified mapping and analysis platforms that support critical decisions and provide new geospatial insights.

The Challenges of Varied Formats and Standards

Geospatial data is captured, structured, and stored in many ways, resulting in a fragmented landscape of datasets with inconsistent formatting, schema, projections, precision, accuracy, and meaning. Satellite imagery, GPS traces, building footprints, road networks, parcel boundaries, address points, terrain models, and survey measurements differ widely in structure and semantic content. These dissimilarities pose difficulties when aggregating diverse data sources into an integrated geospatial framework. Variances must be reconciled through conflation, harmonization, and translation processes before analysis can occur across aggregated data layers. A lack of standardized practices, schemas, and specifications has led to this heterogeneity within the geospatial domain.

Key Considerations for Integration

Several technical factors should be evaluated when developing workflows to combine disparate geospatial data sources:

• Coordinate reference systems and projections
• Data formats such as vectors, rasters, points, surfaces
• Schemas, models, semantics, and feature representations
• Resolution, accuracy, precision variances
• Metadata completeness and quality
• Interoperability support and standards usage
• Processing capabilities to transform, conflate, and fuse sources

Assessing these aspects across datasets allows the identification of compatibility gaps that must be reconciled through ETL and integration methods before analysis can occur. Understanding data lineage, meaning, spatial references, constraints, and quality is also critical.

Matching Coordinate Reference Systems

A key prerequisite when merging geospatial data is aligning different coordinate reference systems (CRS) and map projections utilized across sources. GPS, CAD, BIM, GIS, surveying, and remote sensing platforms use various earth-mapping datums, ellipsoids, and grids to register positions. For example, longitude and latitude coordinates pinned to the WGS84 geodetic datum must be transformed to a universal transverse Mercator projection before integrating with a national mapping agency’s road dataset. Coordinate transformations rescale points, lines, and polygons from one CRS to another while minimizing distortion. Assigning all data layers to a unified projection ensures alignment for overlay analysis.

Handling Different Data Models and Schemas

Geospatial data models represent real-world entities like roads, buildings, terrain, networks, and boundaries using points, vectors, rasters, surfaces, tensors, and other structural forms. Sources adopt different feature representations, attributes, and relationships to encode spatial data based on intended usage. For example, a 3D building model uses geometries, materials, and coordinates suited for visualization while a city planning agency encodes buildings with address, ownership, zoning, taxation attributes for cadastre purposes. Conflating these disparate models into a unified schema requires reconciling structural differences while preserving meaning and relationships. Complex transformation workflows that map attributes and handle differently modeled features are needed to harmonize diverse data models for aggregation.

Dealing with Varying Accuracy and Precision

Geospatial data exhibits variability in accuracy and precision depending on collection methods, sensors, processing techniques, and tolerances. For example, consumer-grade GPS locations, lidar scans, and satellite imagery have meter or sub-meter accuracy while survey-grade GNSS coordinates offer centimeter-level precision. Roads traced from aerial photos likely have higher positional uncertainty versus official government road networks surveyed with RTK GPS. Similarly, terrain models built from sparse elevation point samples are less accurate than those derived photogrammetrically from stereoscopic imagery. Propagating and tracking uncertainty measures is necessary when fusing data layers with divergent precision. This avoids improper usage of inaccurate data and allows positional variances to be quantified after integration.

Transformation and Conflation Techniques

Specialized computational techniques help reconcile structural differences when fusing geospatial data sources:

• Coordinate transformation – Projects geospatial coordinates between datum, ellipsoid and grid systems while minimizing distortion
• Format conversion – Changes how geospatial features are encoded from vectors to rasters, point clouds to tensor grids, surfaces to digital elevation models etc.
• Edge matching – Aligns adjacent but unmatched dataset boundaries by trimming, extending, resampling or morphing edge coordinates
• Schema mapping – Relates elements between differently structured data models and translates attribute names and semantics
• Georeferencing – Assigns real-world coordinate system to imagery or other geospatial data lacking spatial reference
• Generalization – Alters resolution and level of detail to reconcile precision and accuracy variance between source datasets

Applying these techniques facilitates fusion of geospatial data layers with dissimilar structure, formatting, references, precision, and models. This enables integrated analysis, modeling, and visualization despite originating from heterogeneous sources.

Example Workflow for Integrating Road Network Data

Consider a transportation agency needing to build an authoritative road network dataset for a region by combining their existing GIS roads layer with a commercial streets dataset purchased for newer sub-divisions. This requires reconciling differences:

1. Evaluate metadata, schemas, structure, semantics, accuracy, CRS details
2. Identify schema, attribute field mismatches between road layers
3. Develop data model to preserve details from both sources
4. Convert data formats as needed to enable conflation
5. Design ETL process, schema mappings to translate into fused model
6. Standardize street naming, subtype classification across sources
7. Generalize, resample layers to match highest precision source
8. Configure coordinate transformation pipeline
9. Align edge coordinates at dataset boundaries programmatically
10. Merge aligned road vectors into unified geodatabase for QA/QC checks

Similar workflows apply when aggregating other multi-sourced geospatial data like building footprints, parcel boundaries, utility networks etc.

Leveraging Interoperability Standards and Specifications

Given the heterogeneity challenges outlined, the geospatial industry has defined various standards and specifications to enhance interoperability between datasets, software platforms, web services, and data models. Adopting these common conventions simplifies translation, transformation tasks while conveying critical metadata on coordinate systems, accuracy, formatting, structure and other attributes. This eases workflows for multi-source data integration compared to ad-hoc approaches.

OGC Standards for Interoperability

The Open Geospatial Consortium publishes various specifications covering geospatial data formalization, web services, and encodings. Important ones include:

• GML & KML – XML grammar to express vector/raster geospatial data and associated metadata
• GeoTIFF – Raster imagery encoding standard with georeferencing tags
• WFS – Web service API to transfer, query geospatial vector data
• WMS – Web service API to access, render geospatial map images
• CSW – Discovery, search mechanism across metadata catalogs
• SOS – Web service API to query real-time sensor, telemetry data

Using OGC standards where possible optimizes interoperability and eases integrating multi-sourced heterogenous geospatial data.

GeoJSON as a Common Exchange Format

GeoJSON is a JSON-based open specification for encoding geospatial vector data with location coordinates along with metadata. GeoJSON representations are stateless and straightforward to translate to/from other OGC formats like GML, KML. This simplicity has fueled adoption by web mapping libraries like OpenLayers and Leaflet. Using GeoJSON as an intermediate data exchange format smooths workflows for ETL between geospatial data sources.

The Role of Metadata for Context and Meaning

Detailed metadata is crucial when dealing with heterogeneous geospatial data to avoid usage errors and establish proper analytical context. Metadata conveys lineage, positional accuracy, collection methods, processing history, ownership, temporal extents and other critical information that provides framing narrative. For example, geospatial analysis misusing low-resolution crowd-sourced data instead of survey-grade measurements risks generating misleading findings. Capturing rich metadata stories including uncertainty gauges, precision estimates, and intended fitness-for-purpose guides proper data integration strategies.

Best Practices for Organization and Documentation

Metadata practices to improve multi-source data integration outcomes:

• Classify accuracy, metrics, constraints for each dataset
• Log processing techniques applied during conflation, ETL
• Pin metadata directly to datasets for persistence, transfer
• Map semantics between elements across schemas
• Store metadata in easily queried repositories to facilitate discovery
• Structure metadata in standards-based formats like ISO 19115
• Automate metadata generation where possible

Thoughtful organization and documentation of metadata aids geospatial analysts in properly wielding heterogeneous data assets.

Tools and Technologies for Fusion

Specialized tools and platforms help streamline workflows required to transform, integrate, and analyze heterogeneous geospatial data sources by automating complex processing chains. Choosing solutions that align to industry interoperability standards best positions organizations for multi-source data usage.

GIS Software Capabilities

Most mature geographic information systems software offer strong native support for ingesting popular geospatial data formats, structuring data per standard models, exporting per open specifications, and visually styling integrated display of layers with some allowing customization using embedded scripting languages. Commercial platforms like Esri ArcGIS, Hexagon Geospatial, and open solutions QGIS provide ETL, coordinate transformation, visualization, analytics, and metadata management capabilities that ease some multi-source data integration challenges.

Open Source Options for Customization

When aiming to tailor automated workflows for specialized translation, conflation and fusion purposes, open source geospatial tools allow custom orchestration. Platforms like GDAL, PostGIS, GeoServer, GeoTools, OpenLayers powered by Python, Java, and JavaScript ecosystem libraries permit configurable data manipulation pipelines. Open standards access also encourages interoperation with other systems.

Cloud Platforms for Scalable Data Integration

Cloud infrastructure delivers vast compute scalability for processor-intensive geospatial data operations required when integrating large heterogeneous datasets including coordinate reprojection, format transformations, edge matching, spatial joins, generalizations, interpolation, and conflations. Cloud virtual machines can be dynamically allocated to parallelize geo data translation workflows that would take prohibitive timeframes on desktop hardware. Cloud services like AWS, Microsoft Azure, Google Cloud offer managed geospatial platforms to handle large-volume multi-source data fusion.

Setting up a Geospatial Data Hub

Organizations can realize substantial benefits by establishing internal geospatial hubs hosting integrated cached versions of curated external data alongside internally managed business datasets. This enables locating relevant geospatial information assets through one discovery portal while budgeting for only required external data. Key aspects for deploying successful multi-tenant geospatial hubs include scalable infrastructure, configurable integration workflows aligned to standards, collaboration features, and governance protocols covering security, access control, and data maintenance.

Managing Access and Updates

Centralized geospatial hubs must provide capabilities to add new conformed datasets and retire obsolete ones per configurable policies while granting tiered access to employee teams. Automating permissions assignment, entitlement reviews, license management, and user access auditing is required, along with protocols for handling update cycles from external content providers and version control for transformed derivatives.

Conclusion and Future Outlook

Integrating relevant geospatial data from varied sources provides organizations deeper location-based insight for improved decision making but requires solving nuanced technical challenges around harmonizing heterogeneity prevalent today. As more entities adopt geospatial capabilities, standards adoption will increase, easing some aspects of multi-source data aggregation, but varied collection means ensures some conflation workload persists. Cloud infrastructure availability promises to boost compute scalability for organizations needing to process increasing geospatial and temporal volumes at global breadth. While heterogeneity poses perennial integration challenges, conflation techniques leveraging common standards and cloud infrastructure will enable geospatial data relevance, timeliness and richness to continually improve.