Using The Qgis Processing Framework Without The Gui

ByThe GISTips Team

The QGIS processing framework provides access to hundreds of geospatial analysis tools and algorithms. These tools can be accessed through the graphical user interface (GUI) of the QGIS desktop application. However, the processing framework also offers a scripting and command line interface for advanced users and developers.

This article explores techniques for utilizing the processing framework without relying on the standard GUI. Instead, we will leverage the underlying Python console and APIs. This allows for advanced scripting and automation scenarios beyond what is possible with the graphical modeler.

Accessing the Processing Framework from the Python Console

The QGIS Python console provides direct access to the processing framework and algorithms. From the console, we can import algorithm providers, execute tools, and pass parameters.

Importing processing algorithm providers

Processing algorithms in QGIS are grouped into Python plugin packages called providers. To access algorithms in the console, we first need to import their parent provider:

import processing
from qgis import processing
from qgis.analysis import QgsNativeAlgorithms

This imports the Native QGIS algorithms, which contains over 600 tools for vector, raster, and database operations.

Calling processing algorithms

With the algorithm provider imported, we can now execute specific tools. Each algorithm is invoked as a Python function:

processing.run("qgis:buffer") 
processing.run("gdal:rastercalculator")

We specify the provider name and algorithm id. Parameters are passed with keyword arguments or dictionaries.

Passing parameters and handling outputs

Algorithm parameters are configured by named arguments or parameter dictionaries:

params = {
  'INPUT': point_layer,
  'DISTANCE': 100, 
  'SEGMENTS': 10  
}
result = processing.run("native:buffer", params)
buffered_layer = result['OUTPUT']

Outputs from processing algorithms are stored in a dictionary returned by run(). We can access individual outputs by name.

Processing Algorithms for Geoprocessing Tasks

With access to the processing framework, we can now leverage the hundreds of geospatial analysis and geoprocessing algorithms from QGIS and installed plugins.

Vector analysis and geoprocessing

For manipulation and analysis of vector data, key algorithms include:

  • Buffer, clip, intersect, dissolve polygons
  • Spatial joins based on topology or location
  • Aggregate points into hexbins or heatmaps
  • Overlay analysis with intersection, union, or difference
# Intersect hotels and floodplain polygons
processing.run("native:intersection", {
  'INPUT': hotels_layer,
  'OVERLAY': floodplain_layer, 
  'OUTPUT': 'memory:' 
})

# Spatial join counties to climate data
processing.run("qgis:joinattributestable", {  
  'INPUT': counties_layer,
  'FIELD': 'ISO',
  'INPUT_2': climate_data, 
  'FIELDS_TO_COPY': ['temp', 'rainfall'],
  'METHOD': 1,  
  'PREFIX': 'climate_',  
  'OUTPUT': 'memory:'  
})

Raster analysis and geoprocessing

For working with and analyzing raster datasets, key types of algorithms include:

  • Map algebra for arithmetic combinations of rasters
  • Hydrology algorithms for watersheds, flow direction, accumulation
  • Interpolation from vector points to create elevation rasters
  • Band math for calculations on raster pixel values
# Slope calculation from DEM  
slope = processing.run("gdal:slope", {'INPUT': dem_raster})
 
# Flow direction with common outlet point 
dir = processing.run("native:handledirection", {
  'INPUT': dem_raster,
  'OUTPUT': 'memory:',
  'USE_FD8': True, 
  'START_X': 217833, 
  'START_Y': 644633
}) 

# Raster band math
ndvi = processing.run("native:rastercalculator", {
    'FORMULA': '(NIR - RED) / (NIR + RED)',
    'RED': red_band,
    'NIR': nir_band,
    'OUTPUT': 'memory:' 
})

Using the graphical modeler

The QGIS graphical modeler provides a visual programming environment for building workflows from processing algorithms. Models created in the GUI can also be accessed from the Python console.

This allows them to be integrated into scripts and larger automation tasks.

# Get graphical model as Python function
deforestation_model = QgsProcessingModelAlgorithm.fromFile("deforestation_estimates.model3")
 
# Execute model with inputs
results = deforestation_model({ 
  'start_year': 2000,   
  'end_year': 2050
})
 
loss_estimates = results['deforestation']

Processing Framework Under the Hood

Understanding key aspects of how the QGIS processing framework is designed and implemented helps with effectively leveraging algorithms.

How algorithms are implemented

Processing algorithms consist of Python classes that extend QgsProcessingAlgorithm. The key components include:

  • initAlgorithm() – Define parameters and outputs
  • prepareAlgorithm() – Create subsidiary resources
  • processAlgorithm() – Implement core algorithm logic
  • helpString(), displayName() – Documentation

When executed, these components facilitate passing parameters and data between QGIS and the algorithm code.

Extending and developing new algorithms

The processing framework can be extended by writing new algorithms that inherit from QgsProcessingAlgorithm. These custom algorithms integrate seamlessly with existing tools.

from qgis.core import QgsProcessingAlgorithm
from qgis.core import QgsProcessingParameterNumber
from qgis.core import QgsProcessingOutputNumber

class DoubleInputNumber(QgsProcessingAlgorithm):
 
    INPUT_NUMBER = 'INPUT_NUMBER'
    OUTPUT_NUMBER = 'OUTPUT_NUMBER'
 
    def initAlgorithm(self, config):
 
        self.addParameter(
            QgsProcessingParameterNumber(
                self.INPUT_NUMBER, 
                'Input number'
            ) 
        )
 
        self.addOutput(
            QgsProcessingOutputNumber(
                self.OUTPUT_NUMBER, 
                'Doubled number'
            )
        )
        
    def processAlgorithm(self, parameters, context, feedback):
     
        input_number = self.parameterAsDouble(
            parameters,
            self.INPUT_NUMBER,
            context
        )
       
        doubled = input_number * 2
        
        return {self.OUTPUT_NUMBER: doubled}

Automating Workflows with Batch Processing and Scripts

Automating repetitive tasks allows efficient processing of multiple datasets. Scripting also facilitates re-running the same workflow on updated inputs.

Batch processing multiple datasets

The processing framework includes batch processing capabilities for iterating over many inputs:

# Get list of shapefile paths
shapefiles = listdir('./data') 

# Run district area computation on all files 
for filepath in shapefiles:
     district_areas = processing.run("qgis:calculatevectorareas", {
        'INPUT': filepath,
        'OUTPUT': 'memory:'  
     })

Scripting workflows for automation

We can write Python scripts that execute a sequence of algorithms. This allows automating spatial modeling and geoprocessing workflows.

# Script to process new satellite imagery 

# Load input image
image = QgsRasterLayer('/data/new_imagery.tif', 'main')  

# Pan sharpen
pan_sharpened = processing.run("gdal:pansharp", {
  'SPECTRAL_BANDS': image,
  'PANCHROMATIC_BAND': pan_band 
}) 

# Cloud masking 
masked = processing.run("s2cloudless:s2cloudless", {
   'INPUT': pan_sharpened['OUTPUT']   
})

# NDVI 
ndvi = processing.run("native:rastercalculator", {
  'FORMULA': '(B8 - B4) / (B8 + B4)', 
  'INPUT': masked['OUTPUT'],
  'OUTPUT': 'memory:'
}) 

# Classify     
classes = processing.run("native:reclassify", {
  'INPUT_RASTER': ndvi['OUTPUT'],  
  'RASTER_BAND': 1,
  'TABLE': [0,0.2,1,0.4,2,1,3],
  'NO_DATA': -9999,
  'OUTPUT': 'memory:'   
})

# Save final output
QgsRasterLayer(classes['OUTPUT']).save('/data/classified.tif')

Optimizations and Best Practices

There are variety of methods for improving performance when running processing algorithms, especially when handling large datasets.

Speeding up processing tasks

  • Use memory layers instead of writing intermediates to disk
  • Clip large inputs to the analysis extent required
  • Sample raster data to lower resolution if possible
  • Use simplified inputs and merge output later if feasible

Handling large data volumes

  • Split processing by tiles for very large coverage
  • Use lower precision data types when possible
  • Encoder raster outputs like GeoTIFF for compression
  • Drop fields and attributes not required for analysis

Example Codes for Common Use Cases

The following examples demonstrate applying processing algorithms for several common GIS analysis tasks.

Geocoding addresses

table = QgsVectorLayer('/data/addresses.csv', '', 'delimitedtext') 

params = {
  'INPUT': table,
  'STREET_FIELD': 'street',
  'BUILDING_FIELD': 'number', 
  'CITY_FIELD': 'city',
  'OUTPUT': 'memory:' 
} 

result = processing.run("native:geocode", params)
geocoded = result['OUTPUT']

Spatial joins

# Join polygons to nearest points
processing.run("native:joinbynearest", {
   'INPUT': polygons,
   'INPUT_2': points, 
   'FIELDS_TO_COPY': ['data1', 'data2'],
   'PREFIX': 'pt_', 
   'NEAREST_POINTS': 1,
   'OUTPUT': 'memory:' 
})

# Points in polygon join
processing.run("qgis:joinattributesbylocation", {
  'INPUT': points,
  'JOIN': polygons,
  'PREDICATE': [0], # Intersects   
  'JOIN_FIELDS': ['value1', 'value2'],
  'METHOD': 1,
  'PREFIX': 'poly_',
  'OUTPUT': 'memory:'  
})

Raster clipping 

# Clip raster to vector extents
processing.run("gdal:cliprasterbymasklayer", {
    'INPUT': input_raster,
    'MASK': clip_vector,
    'ALPHA_BAND': False,
    'CROP_TO_CUTLINE': True,
    'KEEP_RESOLUTION': True,
    'EXTRA': '',
    'OUTPUT': 'memory:'
})

# Clip by bounding box  
processing.run("native:cliprasterbyextent", {
   'INPUT': input_raster,
   'EXTENT': [xmin, xmax, ymin, ymax], 
   'NODATA': -9999,
   'OUTPUT': 'memory:' 
})

Leave a Reply

Your email address will not be published. Required fields are marked *