Improvement Service
Idox’s Uniform supports creation and automation of Spatial Information Service’s National Data set

Background:

Developing a consistent and accurate national geospatial dataset

As part of the Improvement Service, the Spatial Information Service (SIS) was set up in late 2015 to develop the One Scotland Gazetteer (OSG) – the definitive register of all Scottish addresses and the Spatial Hub – a portal that enables spatial data created and maintained by all 32 Scottish local authorities (LAs) and two National Parks to be readily accessed as a consistent national dataset.

Challenge:

Collating and merging disparate datasets into one central Spatial Hub

For the Spatial Hub to work effectively, all 32 LAs must provide their data to the SIS in a standardised and automated way – reducing the amount of processing required to create a national dataset. This includes data across planning and building standards.

Statutory consultees such as Marine Scotland, Forestry & Land Scotland, SEPA, and NatureScot all operate regionally or nationally so require access to data that doesn’t stop at local authority boundaries. And while they could go to 34 separate planning authority public access sites, not only is this time-consuming, but data is not available to download, and every LA publishes its information in slightly different ways, creating data inconsistencies. A national data set was therefore required to standardise and simplify the process.

Idox has been supporting the Scottish Government in this initiative for several years. Given that all LAs in Scotland use Idox’s solutions for planning and building standards, Idox was integral to streamlining the processes to populate SIS’s central portal.

One of the key challenges SIS faced was the timeliness of uploading gazetteer data to SIS’s One Scotland Gazetteer (OSG) portal. With KPIs from AddressBase and Geoplace to meet, the initial process was for data to be manually exported from Idox’s Uniform system and uploaded into the portal every few weeks. But with such variation between LAs in terms of resource and digital maturity, there was often a lag of up to 10 weeks leading to a lack of consistency in data uploads. Another challenge of this project was the stakeholder management of 32 different LAs. Due to the way the gazetteer works, there are several IT specifications that need configuring at the council end, such as IP whitelisting, opening ports and firewall settings.

Fortunately, with many LAs using a hosted version of Uniform, these settings can be quickly and easily adjusted. But for those LAs early in their cloud journey, changes can be more complex.

Solution:

Automating the extraction of data from Uniform into a central SIS portal

To facilitate the upload of more timely planning and building information from LAs, SIS tried a different approach. This included scraping textual and spatial data from the Idox Public Access systems used by councils and merging it into a single data set. While this was an effective method, SIS and Idox recognised that further efficiencies could be gained by automating the process of merging and centralising the data into the portal.

The next phase of moving this amalgamation of data onto a more robust and streamlined footing was to leverage the Idox Digital Platform and Cloud Connector Framework (CCF) used by LAs and the Scottish Government as part of the eDevelopment process.

This included populating Idox’s Uniform casework management system with online submissions from the Scottish Government’s internally developed online portal. Instead of using the CCF to populate the data into Uniform, it would be used to extract data from Uniform and transferred to a central portal hosted by SIS.

Will Hensman, Geospatial Information Project Manager at Improvement Service comments, “As Idox is so ingrained as a software supplier to the Scottish government for planning, building standards and the gazetteer, it made economic sense to partner with Idox to develop this new process, rather than develop our own system that would need to integrate with it anyway.”

Creating the task for the gazetteer project was fairly straightforward to complete as specifications and formats are already prescribed by industry standards. The digital transfer format dictates the fields and values so they can be exported in the correct form from Uniform using the CCF at the click of a button.

For planning and building standards, there is no standard specification, meaning there were multiple fields that could potentially be exported which fuelled some nervousness among LAs about making their data more widely available. After discussions around the parameters, goals and benefits of the project, as well as which fields, dates and how many fields should be included, LAs were at ease and motivated to take the project forward.

The task also enables users to define what codes to exclude. Given that Uniform records data across a myriad of LA functions in addition to planning and building standards, such as environmental health, estates management, land charges, trading standards, licensing and private sector housing processes, it was essential to narrow down the precise data required for SIS’s central dataset.

Outcomes:

Automated processes delivering centralised, up-to-date and accurate geospatial data

SIS started the task deployment in 2022 and is currently live with 29 out of 32 LAs with full, automated, weekly data extracts to its centralised hub. Will comments, “Every Friday we replace the entire database with a complete data transfer of all the planning, and building standards information we need from Uniform, ensuring that the data is centralised, up-to-date and accurate.”

The gazetteer task is run daily and replaces the entire One Scotland Gazetteer (OSG) database with the most up to date data available from local authorities.

SIS was also impressed with the speed of task completion, Will adds, “We did a lot of work testing how long the data transfer would take each week, the load on the servers, how it might interact with other CCF tasks that LAs have in place. The task runs incredibly quickly, even our largest LA in terms of number of records takes less than seven minutes to complete, which is impressive.”

SIS has now made the data live and available to LAs to review and check planning data, coupled with a PowerBI dashboard that highlights any issues in the integrity and quality of data. The dashboard enables LAs to see how many applications have missing information such as UPRNs and use that added insight to update their data accordingly.

Currently, 40 of the Spatial Hub datasets are available openly under an Open Government Licence (OGL), with the more valuable datasets available for free to PSGA members across Great Britain and as commercial products to all other organisations

Will comments, “The business benefits for SIS and LAs have been significant. Operationally by saving time through reducing manual processes with automation, and reputationally as we can provide almost real-time data to GeoPlace and Address Base for the gazetteer, ensuring accurate data is supplied to users including the emergency services.”

Another benefit has been automating uploads to the Tell Me Scotland system. This is a public information notices portal where information such as planning, roads and licensing notices issued by LAs are made available to the public. Previously, LAs would have to manually upload every notice, but this is now automated via the CCF data exported from Uniform, saving LAs even more time and effort and empowering teams to focus on other activities.

Further down the line, this project will also support the next phase of the Scottish Government’s digital planning programme. One of its aims will be to reduce invalid planning applications by front-loading the application process with more high-quality data, incorporating spatial hub and CCF data. Working closely with Idox, SIS plans to develop the CCF task further to be smarter and collect much more information, as well as move from a weekly to daily extraction to keep up with such regular changes in the world of planning.