Import Styles

Configuring the Import

In the very first step of a Nominatim import, OSM data is loaded into the database. Nominatim uses osm2pgsql for this task. It comes with a flex style specifically tailored to filter and convert OSM data into Nominatim's internal data representation.

There are a number of default configurations for the flex style which result in geocoding databases of different detail. The Import section explains these default configurations in detail.

You can also create your own custom style. Put the style file into your project directory and then set NOMINATIM_IMPORT_STYLE to the name of the file. It is always recommended to start with one of the standard styles and customize those. You find the standard styles under the name import-<stylename>.lua in the standard Nominatim configuration path (usually /etc/nominatim or /usr/local/etc/nominatim).

The remainder of the page describes how the flex style works and how to customize it.

The `flex-base.lua` module

The core of Nominatim's flex import configuration is the flex-base module. It defines the table layout used by Nominatim and provides standard implementations for the import callbacks that make it easy to customize how OSM tags are used by Nominatim.

Every custom style should include this module to make sure that the correct tables are created. Thus start your custom style as follows:

local flex = require('flex-base')

The following sections explain how the module can be customized.

Changing the recognized tags

If you just want to change which OSM tags are recognized during import, then there are a number of convenience functions to set the tag lists used during the processing.

Warning

There are no built-in defaults for the tag lists, so all the functions need to be called from your style script to fully process the data. Make sure you start from one of the default style and only modify the data you are interested in. You can also derive your style from an existing style by importing the appropriate module, e.g. local flex = require('import-street').

Many of the following functions take key match lists. These lists can contain three kinds of strings to match against tag keys: A string that ends in an asterisk * is a prefix match and accordingly matches against any key that starts with the given string (minus the *). A suffix match can be defined similarly with a string that starts with a *. Any other string is matched exactly against tag keys.

`set_main_tags()` - principal tags

If a principal or main tag is found on an OSM object, then the object is included in Nominatim's search index. A single object may also have multiple main tags. In that case, the object will be included multiple times in the index, once for each main tag.

The flex script distinguishes between four types of main tags:

always: a main tag that is used unconditionally
named: consider this main tag only, if the object has a proper name (a reference is not enough, see below).
named_with_key: consider this main tag only, when the object has a proper name with a domain prefix. For example, if the main tag is bridge=yes, then it will only be added as an extra row, if there is a tag bridge:name[:XXX] for the same object. If this property is set, all other names that are not domain-specific are ignored.
fallback: use this main tag only, if there is no other main tag. Fallback always implied named, i.e. fallbacks are only tried for named objects.

The set_main_tags() function takes exactly one table parameter which defines the keys and key/value combinations to include and the kind of main tag. Each lua table key defines an OSM tag key. The value may be a string defining the kind of main key as described above. Then the tag will be considered a main tag for any possible value. To further restrict which values are acceptable, give a table with the permitted values and their kind of main tag. If the table contains a simple value without key, then this is used as default for values that are not listed.

Example

local flex = require('import-full')

flex.set_main_tags{
    boundary = {administrative = 'named'},
    highway = {'always', street_lamp = 'named'},
    landuse = 'fallback'
}

In this example an object with a boundary tag will only be included when it has a value of administrative. Objects with highway tags are always included. However when the value is street_lamp then the object must have a name, too. With any other value, the object is included independently of the name. Finally, if a landuse tag is present then it will be used independely of the concrete value if neither boundary nor highway tags were found and the object is named.

`set_prefilters()` - ignoring tags

Pre-filtering of tags allows to ignore them for any further processing. Thus pre-filtering takes precedence over any other tag processing. This is useful when some specific key/value combinations need to be excluded from processing. When tags are filtered, they may either be deleted completely or moved to extratags. Extra tags are saved with the object and returned to the user when requested, but are not used otherwise.

set_prefilters() takes a table with four optional fields:

delete_keys is a key match list for tags that should be deleted
delete_tags contains a table of tag keys pointing to a list of tag values. Tags with matching key/value pairs are deleted.
extra_keys is a key match list for tags which should be saved into extratags
extra_tags contains a table of tag keys pointing to a list of tag values. Tags with matching key/value pairs are moved to extratags.

Key list may contain three kinds of strings: A string that ends in an asterisk * is a prefix match and accordingly matches against any key that starts with the given string (minus the *). A suffix match can be defined similarly with a string that starts with a *. Any other string is matched exactly against tag keys.

Example

local flex = require('import-full')

flex.set_prefilters{
    delete_keys = {'source', 'source:*'},
    extra_tags = {amenity = {'yes', 'no'}}
}
flex.set_main_tags{
    amenity = 'always'
}

In this example any tags source and tags that begin with source: are deleted before any other processing is done. Getting rid of frequent tags this way can speed up the import.

Tags with amenity=yes or amenity=no are moved to extratags. Later all tags with an amenity key are made a main tag. This effectively means that Nominatim will use all amenity tags except for those with value yes and no.

`set_name_tags()` - defining names

The flex script distinguishes between two kinds of names:

main: the primary names make an object fully searchable. Main tags of type named will only cause the object to be included when such a primary name is present. Primary names are usually those found in the name tag and its variants.
extra: extra names are still added to the search index but they are alone not sufficient to make an object named.

set_name_tags() takes a table with two optional fields main and extra. They take key match lists for main and extra names respectively.

Example

local flex = require('flex-base')

flex.set_main_tags{highway = {traffic_light = 'named'}}
flex.set_name_tags{main = {'name', 'name:*'},
                   extra = {'ref'}
                  }

This example creates a search index over traffic lights but will only include those that have a common name and not those which just have some reference ID from the city.

`set_address_tags()` - defining address parts

Address tags will be used to build up the address of an object.

set_address_tags() takes a table with arbitrary fields pointing to key match lists. To fields have a special meaning:

main: defines the tags that make a full address object out of the OSM object. This is usually the housenumber or variants thereof. If a main address tag appears, then the object will always be included, if necessary with a fallback of place=house. If the key has a prefix of addr: or is_in: this will be stripped.
extra: defines all supplementary tags for addresses, tags like addr:street, addr:city etc. If the key has a prefix of addr: or is_in: this will be stripped.

All other fields will be handled as summary fields. If a key matches the key match list, then its value will be added to the address tags with the name of the field as key. If multiple tags match, then an arbitrary one wins.

Country tags are handled slightly special. Only tags with a two-letter code are accepted, all other values are discarded.

Example

local flex = require('import-full')

flex.set_address_tags{
    main = {'addr:housenumber'},
    extra = {'addr:*'},
    postcode = {'postal_code', 'postcode', 'addr:postcode'},
    country = {'country-code', 'ISO3166-1'}
}

In this example all tags which begin with addr: will be saved in the address tag list. If one of the tags is addr:housenumber, the object will fall back to be entered as a place=house in the database unless there is another interested main tag to be found.

Tags with keys country-code and ISO3166-1 are saved with their value under country in the address tag list. The same thing happens to postcodes, they will always be saved under the key postcode thus normalizing the multitude of keys that are used in the OSM database.

`set_unused_handling()` - processing remaining tags

This function defines what to do with tags that remain after all tags have been classified using the functions above. There are two ways in which the function can be used:

set_unused_handling(delete_keys = ..., delete_tags = ...) deletes all keys that match the descriptions in the parameters and moves all remaining tags into the extratags list. set_unused_handling(extra_keys = ..., extra_tags = ...) moves all tags matching the parameters into the extratags list and then deletes the remaining tags. For the format of the parameters see the description in set_prefilters() above.

Example

local flex = require('import-full')

flex.set_address_tags{
    main = {'addr:housenumber'},
    extra = {'addr:*', 'tiger:county'}
}
flex.set_unused_handling{delete_keys = {'tiger:*'}}

In this example all remaining tags except those beginning with tiger: are moved to the extratags list. Note that it is not possible to already delete the tiger tags with set_prefilters() because that would remove tiger:county before the address tags are processed.

Customizing osm2pgsql callbacks

osm2pgsql expects the flex style to implement three callbacks, one process function per OSM type. If you want to implement special handling for certain OSM types, you can override the default implementations provided by the flex-base module.

Changing the relation types to be handled

The default scripts only allows relations of type multipolygon, boundary and waterway. To add other types relations, set RELATION_TYPES for the type to the kind of geometry that should be created. The following kinds of geometries can be used:

relation_as_multipolygon creates a (Multi)Polygon from the ways in the relation. If the ways do not form a valid area, then the object is silently discarded.
relation_as_multiline creates a (Multi)LineString from the ways in the relations. Ways are combined as much as possible without any regards to their order in the relation.

Example

local flex = require('import-full')

flex.RELATION_TYPES['site'] = flex.relation_as_multipolygon

With this line relations of type=site will be included in the index according to main tags found. This only works when the site relation resolves to a valid area. Nodes in the site relation are not part of the geometry.

Adding additional logic to processing functions

The default processing functions are also exported by the flex-base module as process_node, process_way and process_relation. These can be used to implement your own processing functions with some additional processing logic.

Example

local flex = require('import-full')

function osm2pgsql.process_relation(object)
    if object.tags.boundary ~= 'administrative' or object.tags.admin_level ~= '2' then
      flex.process_relation(object)
    end
end

This example discards all country-level boundaries and uses standard handling for everything else. This can be useful if you want to use your own custom country boundaries.

Customizing the main processing function

The main processing function of the flex style can be found in the function process_tags. This function is called for all OSM object kinds and is responsible for filtering the tags and writing out the rows into Postgresql.

Example

local flex = require('import-full')

local original_process_tags = flex.process_tags

function flex.process_tags(o)
    if o.object.tags.highway ~= nil and o.object.tags.access == 'no' then
        return
    end

    original_process_tags(o)
end

This example shows the most simple customization of the process_tags function. It simply adds some additional processing before running the original code. To do that, first save the original function and then overwrite process_tags from the module. In this example all highways which are not accessible by anyone will be ignored.

The `Place` class

The process_tags function receives a Lua object of Place type which comes with some handy functions to collect the data necessary for geocoding and writing it into the place table. Always use this object to fill the table.

The Place class has some attributes which you may access read-only:

object is the original OSM object data handed in by osm2pgsql
admin_level is the content of the admin_level tag, parsed into an integer and normalized to a value between 0 and 15
has_name is a boolean indicating if the object has a full name
names is a table with the collected list of name tags
address is a table with the collected list of address tags
extratags is a table with the collected list of additional tags to save

There are a number of functions to fill these fields. All functions expect a table parameter with fields as indicated in the description. Many of these functions expect match functions which are described in detail further below.

delete{match=...} removes all tags that match the match function given in match.
grab_extratags{match=...} moves all tags that match the match function given in match into extratags. Returns the number of tags moved.
clean{delete=..., extra=...} deletes all tags that match delete and moves the ones that match extra into extratags
grab_address_parts{groups=...} moves matching tags into the address table. groups must be a group match function. Tags of the group main and extra are added to the address table as is but with addr: and is_in: prefixes removed from the tag key. All other groups are added with the group name as key and the value from the tag. Multiple values of the same group overwrite each other. The function returns the number of tags saved from the main group.
grab_main_parts{groups=...} moves matching tags into the name table. groups must be a group match function. If a tags of the group main is present, the object will be marked as having a name. Tags of group house produce a fallback to place=house. This fallback is return by the function if present.

There are two functions to write a row into the place table. Both functions expect the main tag (key and value) for the row and then use the collected information from the name, address, extratags etc. fields to complete the row. They also have a boolean parameter save_extra_mains which defines how any unprocessed tags are handled: when True, the tags will be saved as extratags, when False, they will be simply discarded.

write_row(key, value, save_extra_mains) creates a new table row from the current state of the Place object.
write_place(key, value, mtype, save_extra_mains) creates a new row conditionally. When value is nil, the function will attempt to look up the value in the object tags. If value is still nil or mtype is nil, the row is ignored. An mtype of always will then always write out the row, a mtype of named only, when the object has a full name. When mtype is named_with_key, the function checks for a domain name, i.e. a name tag prefixed with the name of the main key. Only if at least one is found, the row will be written. The names are replaced with the domain names found.

Match functions

The Place functions usually expect either a match function or a group match function to find the tags to apply their function to.

The match function is a Lua function which takes two parameters, key and value, and returns a boolean to indicate that a tag matches. The flex-base module has a convenience function tag_match() to create such a function. It takes a table with two optional fields: keys takes a key match list (see above), tags takes a table with keys that point to a list of possible values, thus defining key/value matches.

The group match function is a Lua function which also takes two parameters, key and value, and returns a string indicating to which group or type they belong to. The tag_group() can be used to create such a function. It expects a table where the group names are the keys and the values are a key match list.

Using the gazetteer output of osm2pgsql

Nominatim still allows you to configure the gazetteer output to remain backwards compatible with older imports. It will be automatically used when the style file name ends in .style. For documentation of the old import style, please refer to the documentation of older releases of Nominatim. Do not use the gazetteer output for new imports. There is no guarantee that new versions of Nominatim are fully compatible with the gazetteer output.

Changing the Style of Existing Databases

There is normally no issue changing the style of a database that is already imported and now kept up-to-date with change files. Just be aware that any change in the style applies to updates only. If you want to change the data that is already in the database, then a reimport is necessary.

Import Styles

Configuring the Import

The flex-base.lua module

Changing the recognized tags

set_main_tags() - principal tags

set_prefilters() - ignoring tags

set_name_tags() - defining names

set_address_tags() - defining address parts

set_unused_handling() - processing remaining tags

Customizing osm2pgsql callbacks

Changing the relation types to be handled

Adding additional logic to processing functions

Customizing the main processing function

The Place class

Match functions

Using the gazetteer output of osm2pgsql

Changing the Style of Existing Databases

The `flex-base.lua` module

`set_main_tags()` - principal tags

`set_prefilters()` - ignoring tags

`set_name_tags()` - defining names

`set_address_tags()` - defining address parts

`set_unused_handling()` - processing remaining tags

The `Place` class