Running Nominatim and Rendering in a Single Database

This tutorial shows how to set up a custom import style which creates a rendering database next to the geocoding database.

Since version 4.2.0 Nominatim allows to use osm2pgsql’s flex output to describe the table layout for the initial import. The Lua-based style is a lot more flexible than the simple json import configuration offered by Nominatim right now. One of the things you can do with it, is to create additional tables next to the ones needed by Nominatim. This paves the road to a much requested feature: running geocoding and rendering maps from the very same database.

This tutorial uses an experimental feature of Nominatim. You will need the latest release to make it work. Please use with care and be aware that details of the process may change in the future.

Choosing the import styles

In order to have a combined database, you need to have a flex style configuration for all the separate parts. For this tutorial we will use the import-extratags.lua style for the Nominatim database and the compatible.lua style from the flex example of osm2pgsql for the rendering database. The latter just stands in as an example layout. You will, of course, need to choose the style that provides the rendering database for your particular map layout.

Two osm2pgsql flex styles can only be used together when they

• define different table names for their output tables
• have different names for all global Lua variables they use.

As a rule, you should try to avoid using global variables in your Lua style files as much as possible. Database schemas are a great way to avoid clashes with table names.

When choosing the style, make sure it is really meant to work with the flex output. Styles like osm-carto still use the pgsql output with a Lua tagtransform script. This cannot be used with the flex style. Have a look at the experimental carto flex branch if you want to experiment with carto rendering.

Preparing the configuration

Nominatim has all the mechanisms for import and updates already built in, so the easiest way to get to a combined database is to configure it to set up the rendering tables in addition to its own tables.

First of all, create a project directory for Nominatim. We use this to collect all the necessary scripts and settings.

mkdir project-flex
cd project-flex

Now copy all necessary Lua style files into the project directory. This will make it simpler for Lua to find the styles later. Assuming you have the Nominatim source code in a directory $NOMINATIM_SRC, then run:

cp $NOMINATIM_SRC/settings/flex-base.lua .
cp $NOMINATIM_SRC/settings/import-extratags.lua .
cp $NOMINATIM_SRC/osm2pgsql/flex-config/compatible.lua .

Next you need a script that combines the two separate style files. Create a file flex-combined.lua in your project directory with the following content:

-- Make sure lua finds out scripts
package.path = '?.lua;' .. package.path

-- Create a table that saves all the handlers we need.

handlers = {node = {}, way = {}, relation = {}}

local function save_handlers()
    if osm2pgsql.process_node ~= nil then
        table.insert(handlers.node, osm2pgsql.process_node)
    end
    if osm2pgsql.process_way ~= nil then
        table.insert(handlers.way, osm2pgsql.process_way)
    end
    if osm2pgsql.process_relation ~= nil then
        table.insert(handlers.relation, osm2pgsql.process_relation)
    end
    osm2pgsql.process_node = nil
    osm2pgsql.process_way = nil
    osm2pgsql.process_relation = nil
end

-- load nominatim style
local nominatim = require('import-extratags')
save_handlers()

--- load mapnik style
local mapnik = require('compatible')
save_handlers()

-- calling both

local function run_all_handlers(object, hlist)
    local saved_tags = object.tags
    for i, handler in pairs(hlist) do
        -- Handlers tend to modify the tag list. Hand in a copy.
        if i == #hlist then
            object.tags = saved_tags
        else
            object.tags = {}
            for tk, tv in pairs(saved_tags) do
                object.tags[tk] = tv
            end
        end
        handler(object)
    end
end

function osm2pgsql.process_node(object)
    run_all_handlers(object, handlers.node)
end

function osm2pgsql.process_way(object)
    run_all_handlers(object, handlers.way)
end

function osm2pgsql.process_relation(object)
    run_all_handlers(object, handlers.relation)
end

The script loads each style file in turn and saves their processing functions in a local variable. Then it overwrites the processing functions with its own version which simply calls the functions it has previously saved. That’s it!

To tell Nominatim about your custom style file, you need to set the appropriate configuration variable:

echo NOMINATIM_IMPORT_STYLE=flex-combined.lua >> .env

Nominatim has an optimisation where it does not propagate changes in OSM data to dependent objects. For example, when a node is moved around, then the geometry of a way that this node is part of usually changes. Such geometry changes are rarely relevant for geocoding but they are important to update when making maps. Therefore, disable this optimisation:

echo NOMINATIM_UPDATE_FORWARD_DEPENDENCIES=yes >> .env

If you want to make further changes to the Nominatim configuration like changing the name of the database or using a flatnode file, have a look at the configuration settings and adapt your .env file accordingly.

You may also want to download additional data into your project directory at this point.

Importing and updating

You can now run the import as described in the Import guide:

nominatim import --osm-file <data file> 2>&1 | tee setup.log

Once it is finished, your database will contain the usual tables of a Nominatim import as well as tables planet_osm_point, planet_osm_line, planet_osm_polygon and planet_osm_roads for rendering.

You can also keep your data up to date by simply following Nominatim’s update guide.

Updates with tile expiry

If you want to use the expiry mechanism of osm2pgsql to update your tiles, then it is not possible anymore to use Nominatim’s builtin update function. Instead you need to do updates manually with osm2pgsql and then run the postprocessing. In this section we show how to do this with osm2pgsql-replication. You find the osm2pgsql-replication script bundled with the Nominatim source code under osm2pgsql/scripts/osm2pgsql-replication. Copy the script into your project directory.

You should already have a combined database imported with Nominatim as shown above. Next you need to initialise the replication process:

./osm2pgsql-replication init -d nominatim

If your database is named differently, adapt the name after the ‘-d’ parameter. Next you need a script that runs all the post-processing steps. Create a script named postprocess-update.sh in your project directory with the following content:

#!/bin/sh

# For Nominatim run the indexing.
nominatim index

# Then run tile expiry.
# This is the example from https://switch2osm.org. Adapt this command to
# whatever you need to do.
render_expired --map=s2o --min-zoom=13 --max-zoom=20 -s /run/renderd/renderd.sock < /var/cache/renderd/dirty_tiles.txt
rm /var/cache/renderd/dirty_tiles.txt

Make the script executable:

chmod 755 postprocess-update.sh

Now you can run a batch of updates with a command like this:

./osm2pgsql-replication update -d nominatim --post-processing ./postprocess-update.sh --osm2pgsql-cmd /usr/local/lib/nominatim/osm2pgsql -- -s -a -O flex -S flex-combined.lua -e 12 -o /var/cache/renderd/dirty_tiles.txt

The --osm2pgsql-cmd parameter tells osm2pgsql-replication where to find the osm2pgsql binary. This should be the binary bundled with Nominatim. If you have installed Nominatim somewhere other than the default /usr/local you need to adopt the path. The parameters after -- are the parameters with which osm2pgsql is called. Choosing the flex output and the style is of course mandatory. You might want to add other parameters here. For example, if you have configured a flatnode file, you must add -F <location of your file> to the parameter list.