Improving Spatial Data Quality And Integrity In Gis

Ensuring Accurate Location Data

The geographic location of spatial features in a GIS is critical to enable accurate analysis and visualization. However, location data can easily become inaccurate due to issues with data collection, transformation errors, or lack of quality control. Implementing methods to continually assess, improve, and maintain the accuracy of coordinate data is essential.

Assessing Current Data Accuracy

Performing a quantitative assessment of the overall positional accuracy of a dataset establishes a benchmark to improve upon. This involves statistical analysis by comparing a sample of coordinates against an independent, highly-accurate reference dataset for the same locations. The difference in coordinates reflects the error magnitude, with summary statistics like mean, standard deviation, and root mean square providing insight into accuracy across the entire dataset. Code for coordinate accuracy assessment may include statistical analysis functions from spatial libraries.

Implementing Quality Assurance Procedures

A key way to improve and maintain coordinate accuracy is through quality assurance procedures that systematically check new data prior to integration. This includes both automatic and manual checks, such as running automated topology rules to catch digitizing errors, or manually inspecting samples of new feature coordinates pre and post-integration. Well-defined coordinate accuracy standards, documentation for all data and changes, and training for those editing data are important to support quality assurance.

Performing Coordinate Transformations

Transforming coordinates from one spatial reference system to another, such as between projected and geographic coordinates, can introduce positional error. Execution of coordinate transformations should utilize precise transformation parameters and methods suitable to the data extent. Assessing coordinate accuracy pre and post-transformation provides insight into the magnitude of error introduced. Code for coordinate transformations requires setting the appropriate input and output projections, datums, and parameters.

Example Code for Coordinate Conversion

// Projection parameters
const inProjection = '+proj=utm +zone=10 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs ';
const outProjection = '+proj=longlat +ellps=GRS80 +datum=WGS84 +no_defs ';

// Coordinate Transformation
coordinates.forEach(coord => {
  [longitude, latitude] = proj4(inProjection, outProjection, [coord[0], coord[1]]);
  
  // Calculate accuracy statistics
  sumOfSquaredError += (longitude - longRef) ** 2 + (latitude - latRef) ** 2; 
});

// Report accuracy statistics
const rmsError = Math.sqrt(sumOfSquaredError / n);

Detecting and Fixing Topological Inconsistencies

Features in close proximity should have defined topological relationships, with no gaps or overlaps between boundaries. Errors in digitizing, editing, or transformation can disrupt topology, requiring both automated and manual intervention to fix.

Common Topological Errors

Common topological inconsistencies include undershoots and overshoots where features do not meet cleanly, overlap between features that should not intersect, and gaps between features that should be contiguous. Failure to maintain topology makes spatial analysis invalid. Code must systematically flag topological errors for review and correction.

Automated Topology Checking

GIS provides automated topology rule checking to systematically scan spatial data and highlight errors for correction. Rules enforce expectations like lines intersecting at nodes, polygons fully enclosing and not overlapping, and alignment of coincident boundaries. Reviewing and resolving flagged topology issues improves integrity. Code for running topology checks involves iterating through features and evaluating spatial relationships.

Manual Validation Techniques

In addition to automated checking, manual visual inspection by GIS analysts comparing spatial data against reference layers can reveal subtle topology errors not caught programmatically. Interactively editing features to snap intersections and reshape boundaries to meet cleanly maintains integrity. Code may assist manual review by scripting zoom/pan to flagged locations.

Example Code for Topology Cleaning

// Set topology rules
const rules = {
  mustNotOverlap: true,
  mustNotHaveGaps: true,  
  mustSnapAtEndpoints: true
};

// Run automated topology check
features.forEach(feature => {
  topologyViolations = getViolatingRules(feature, features, rules); 
  
  // Try automated snap
  if ('mustSnapAtEndpoints' in topologyViolations)
    feature = snapToNearestEndpoint(feature, features);

  // Flag other violations for manual review
  else logTopologyError(feature, topologyViolations);  
});

// Manually correct errors
flaggedFeatures.forEach(feature => { 
  edited = manuallyEditFeature(feature);
  
  if (getViolatingRules(edited, features, rules).length == 0)
    commitEdit(edited); // Error resolved
  else retryManualEdit(edited); // Further fixes needed 
});

Managing Attribute Accuracy

In addition to location, feature attributes in GIS provide meaningful descriptive information tied to spatial objects. However, attributes can easily contain errors or uncertainties which affects analysis. Thus applying methods to improve attribution quality and flag uncertain values is key.

Techniques for Finding Attribute Errors

Strategies for systematically finding attribution errors involve both automated rules and manual review. Rules may check for blank, null, or duplicate attributes that should be unique, outlier values that exceed expected ranges, and improper data formats. Manual inspection by domain experts aids in detecting incorrect or imprecise classifications and descriptive values. Code for attribute error checking functions evaluates values against expected parameters.

Correcting Mislabeled Features

Finding and properly re-labeling misattributed features maintains information quality. For points or polygons labeled with the wrong class, automated re-classification functions can suggest more likely labels based on proximate features and pre-defined model logic. Manual override allows further adjustments. Code for automated attribution should use probabilistic methods like random forests to suggest corrected labels.

Handling Uncertainty in Attributes

For features with attributes not directly measured or observed but estimated via models, it is vital to quantify and store uncertainty metrics. This may involve probability distributions, standard deviations, or confidence intervals indicating the precision and accuracy for each estimated attribute value. Communicating uncertainty aids proper analytic use. Code should persist not just values but uncertainty bounds in the attribute schema.

Example Code for Improving Attribution

// Check for blank attributes
features.forEach(feature => {
  blankAttrs = feature.attributes.filter(attr => attr.value == null);
  
  if (blankAttrs.length > 0)
    flagFeature(feature, 'blank_attribute'); 
});

// Correct mislabeled features
mislabeledFeatures.forEach(feature => {
  likelyLabel = classify(feature, trainingData);
  
  feature.label = likelyLabel;
  commitEdit(feature);
});

// Store uncertainty estimates 
estimatedFeatures.forEach(feature => {
  feature.attributes.push({
    value: feature.estimatedValue,
    standardDeviation: feature.valueUncertainty 
  });
});

Updating Data to Reflect Real-World Changes

Spatial data layers represent a snapshot of real-world configurations and phenomena at the time of capture. Keeping the data current via systematic updates as the underlying geography changes over time is crucial for validity.

Identifying Outdated Spatial Data

Strategies for finding stale data requiring update include comparing recent high-resolution base imagery to vector layers and noting discrepancies, checks against authoritative datasets that may reveal omitted changes, and field observation of areas with higher change rates. Code to flag potentially outdated areas can use image processing and change detection algorithms.

Procedures for Systematically Updating

To prevent data from becoming progressively more outdated, agencies should follow standardized workflows for systematic data updates on a regular cycle, integrating new captures for areas likely to see change. Code should orchestrate batch geoprocessing workflows that handle acquiring new source data, transforming geometry and attributes, enforcing topology, and reconciling against existing layers.

Integrating New Data Sources

Incorporating new captures from field devices, sensors, crowdsourcing, or commercialproviders into existing layers requires properly translating geometry and attributes between schemas. Code must mediate differences in structure, semantics, coordinate systems, metadata models, formats, and topologies so combined data meets integrity rules. This requires flexible schema mapping and transformation scripts.

Example Code for Merging Updated Data

// Handle geometry changes
newFeatures.forEach(feature => {
  mapToTemplateSchema(feature); // Translate structure
  reproject(feature); // Convert CRS 
  snapToReferenceLayer(feature, refLayer); // Align
});

// Handle attribute changes
updatedFeatureAttributes.forEach(attributes => {
  translateAttributes(attributes, attributeMaps); // Convert semantics  
  qaAttributes(attributes); // Validate
});

// Merge with existing data
updatedLayer = union(existingLayer, newFeatures);
cleanTopology(updatedLayer);
updatedLayer = symmetricalDifference(updatedLayer, obsoleteFeatures);
commitMergedLayer(updatedLayer);

Ongoing Monitoring and Maintenance

Continual monitoring of spatial data quality issues along with proper documentation and archiving enables tracking integrity over time and tracing the lineage of changes.

Implementing Data Integrity Checks

Scheduling routine automated rule-based integrity checks on critical GIS layers with emails alerting administrators to problems allows keeping on top of errors before they propagate further. Custom script tools can evaluate layers against organizational quality standards for geometry, topology, attribution, semantics, formats etc. and log issues.

Tracking Data Lineage and Edits

Full documentation of data provenance and editing history provides accountability. GIS metadata should capture source datasets, editing processes and methods, attribution schema changes, and authorship details. Code can automatically append timestamped log entries for entity creations, edits, deletions, and merges.

Archiving Historical Data Snapshots

Saving periodic archived versions of spatial data layers provides a historical record of changes that aids in determinig when and how integrity issues arose. Code can assist by scripting export of layer copies with meaningful version numbers at regular intervals for storage.

Example Code for Data Auditing

// Schedule weekly check  
cron.schedule('0 0 * * 0', runQualityCheck);  

// Check functions
function runQualityCheck() {
  checkSpatialAccuracy(datasets);
  checkAttributes(datasets);  
  checkTopologies(datasets);
  
  // Log any errors
  if (errors.length > 0) 
    emailAdmins(errorReport);
}

// Archive layers
datasets.forEach(dataset => {  
  archivedLayer = dataset.copy(`v${getNewVersionNumber()}`); 
  export(archivedLayer, archiveDirectory); 
});