Common Pitfalls And Solutions For Using Pyqgis In Windows Standalone Scripts

Troubleshooting QGIS Python Issues on Windows

Locating the Cause

When issues arise with PyQGIS scripts on Windows, the first step is identifying the root cause. Check any error messages output to the console or logs for clues. Error messages indicating file paths, access issues, or version conflicts can help narrow things down. If a particular script triggers problems, isolate parts of the code to locate the problem area.

Next, verify factors like the Python version, QGIS version, involved packages, and environment settings. For Python scripts, a mismatch between versions and packages often causes trouble. Confirm QGIS links to the right Python binary intended for the scripts. The Python environment visible to QGIS for plugins differs from the system Python install. On Windows, multiple isolated Python environments can lead to confusion.

Common Windows-Specific Problems

Windows poses unique challenges for running PyQGIS scripts standalone outside of the QGIS Python environment. Special characters in Windows file paths can cause encoding errors. Restricted user account permissions may block access to key files and packages. And dependency conflicts crop up between QGIS, Python, and libraries.

Table of Contents

Encoding Errors from File Paths with Special Characters

File paths on Windows frequently include special characters like spaces or unicode characters. By default, Python uses the ASCII encoding, which cannot represent these characters. Attempting to access these paths then triggers decoding errors containing strings like:

'utf-8' codec can't decode byte 0x90 in position 4: invalid start byte

The solution involves using raw strings that handle the encoding automatically or normalizing the paths to plain ASCII characters.

Permission Errors Accessing Files or Packages

Unlike Linux, Windows restricts access to key directories like Program Files and the root C drive by default. As a result, Python scripts fail when trying to install packages globally or write logs and config data. The errors resemble:

[Errno 13] Permission denied

To avoid permission issues, use virtual environments, run as admin, modify user permissions, or install packages in user directories instead.

Compatibility Issues Between QGIS, Python, and Libraries

On Windows, QGIS relies on a specific version of Python along with particular library dependencies. Standalone scripts can introduce conflicts, especially if outdated versions remain after upgrading QGIS itself. Tricky to debug, mismatches manifest through import or attribute errors like:

ModuleNotFoundError: No module named 'qgis'
AttributeError: 'NoneType' object has no attribute 'addProvider'

Isolation via virtual environments addresses compatibility problems by allowing for separate packages. Pointing QGIS to custom Python environments also helps avoid conflicts.

Fixing Encoding and Path Issues

Using Raw Strings for Windows File Paths

Unlike normal strings, raw strings passed into Python contain the exact value without encoding. Prefix path strings with r to automatically handle special characters on Windows:

my_path = r'C:\My Folder\data\sample.geojson'

Raw strings support escaped characters and retain backslashes and unicode values without conversion or errors.

Normalizing Special Characters in Strings

An alternative fixes paths by replacing spaces and unicode characters with ASCII equivalents like underscores and percentages:

import os
path = os.path.normpath(r'C:\data\sample file.txt')
print(path)
# C:\data\sample_file.txt

The os.path.normpath standardized paths to use / instead of \\.

Setting the Correct Locale and Encodings

For scripts handling file input/output, set the Python locale and encoding values to avoid issues with special characters:

import locale
locale.setlocale(locale.LC_ALL, 'en_US.utf8')

Explicitly open files using the correct encoding as well:

 datafile = open('data.csv', encoding='utf-8')

Resolving Permission Errors

Running Scripts with Admin Privileges

Scripts that require elevated permissions can use the “Run as administrator” option:

Right click on the .py file
Choose Properties > Advanced
Check “Run as administrator”
Click Apply

This prompts Windows to approve admin access when launching the script.

Granting User Account Permissions to Access Files

Update user account permissions to provide standard users access to protected directories like Program Files:

Right click the folder and select Properties
Switch to the Security tab and choose Edit
Click Add and enter the user account name
Check the box for Full Control permission

This permits the user account to read, write, and execute files in the folder.

Installing Packages in User Directories Instead of System

For handling packages, avoid global installs requiring root access. Instead, use per-user installs pointing to UserProfile:

python -m pip install --user pytz

This places packages in AppData\Roaming avoiding permission errors.

Managing QGIS and Python Environments

Creating Isolated Environments with Conda or Virtualenv

Virtual environments like Conda and Virtualenv encapsulate all dependencies for Python scripts:

# Make an environment called qgis-scripts
conda create -n qgis-scripts

# Activate environment 
conda activate qgis-scripts  

# Install desired packages
conda install gdal numpy pandas

With dependencies isolated, version conflicts no longer block scripts from running.

Installing Compatible Version Combinations

Determine the QGIS Python version then install matching packages:

# Find QGIS Python version
import sys
print(sys.version)

# Install packages for that version
py -3.7 -m pip install numpy===1.15.2

Use virtual environments and version pinning to achieve compatibility between components.

Pointing QGIS to Custom Python Environments

If QGIS links against the wrong Python version, Update the path to use the desired binary:

Find the custom python.exe path ie C:\Python37\python.exe
Open QGIS > Settings > System > Environment > PYTHONPATH
Add the full path and python.exe to the variable

Now QGIS leverages the standalone Python install rather than any system default.

Debugging Techniques

Adding Print Statements and Logging

Include print statements to output variable values and application flow:

data = getData()
print(f'data shape is: {data.shape}')

filtered = processData(data) 
print(f'processed shape is: {filtered.shape}')

Logging captures runtime output without cluttering up the results:

import logging
logging.basicConfig(filename='logs.txt', level=logging.DEBUG)

logging.info('Script start') 
...
logging.info('Data loaded successfully')

Using Debuggers and IDEs like PyCharm

Debuggers step through code and inspect variables at any breakpoint:

import pdb
pdb.set_trace()

data = getInput()
data.head() # <- Debugger stops here

FULL IDEs like PyCharm provide built-in debugging UIs without any code changes:

Set breakpoints where issues may occur
Step through code line by line
View data values and exceptions in realtime

Analyzing Stack Traces and Fixing Code Issues

For crashes, look at the exception output and stack trace to pinpoint failure points:

File "script.py", line 15, in  
   analyze(data_frame)
File "analyze.py", line 10, in analyze
   return data.mean() 
AttributeError: 'NoneType' object has no attribute 'mean'

This reveals issues down the chain like the wrong variable name or failure to return a value. Tweak code accordingly to address bugs.

Example Scripts

Walking through practical scripts illustrates solutions for common PyQGIS Windows issues:

Sample Script Using Raw File Paths

Handling Windows paths with spaces:

import osgeo.gdal as gdal

# raw string avoids decoding errors 
shp_path = r'C:\GIS Data\Demo\buildings.shp'

# Open the datasource
data_source = gdal.OpenEx(shp_path)

# Print raster info
print(data_source)

Privilege Escalation to Access Protected Folders

Run with admin rights to install packages globally:

# Require admin privileges
if not "ADMIN" in os.environ.keys():
    exit("Restart script as admin!")

# Now able to access protected locations 
PROGRAM_FILES = os.environ['ProgramFiles'] 

# Install package system-wide
subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'pandas'])

Using Virtualenv to Avoid QGIS Conflicts

Isolate PyQGIS dependencies:

# Create virtual environment
python -m venv qgis-scripts
qgis-scripts\Scripts\activate

# Install PyQGIS packages
python -m pip install PyQGIS

# Set environment variable to customize plugin paths
os.environ['QGIS_PLUGINPATH'] = r'C:\gisplugins'

# Now able to import QGIS libraries
import qgis.core