Tokenizers🔗

The tokenizer module in Nominatim is responsible for analysing the names given to OSM objects and the terms of an incoming query in order to make sure, they can be matched appropriately.

Nominatim offers different tokenizer modules, which behave differently and have different configuration options. This sections describes the tokenizers and how they can be configured.

Important

The use of a tokenizer is tied to a database installation. You need to choose and configure the tokenizer before starting the initial import. Once the import is done, you cannot switch to another tokenizer anymore. Reconfiguring the chosen tokenizer is very limited as well. See the comments in each tokenizer section.

Legacy tokenizer🔗

Danger

The Legacy tokenizer is deprecated and will be removed in Nominatim 5.0. If you still use a database with the legacy tokenizer, you must reimport it using the ICU tokenizer below.

The legacy tokenizer implements the analysis algorithms of older Nominatim versions. It uses a special Postgresql module to normalize names and queries. This tokenizer is automatically installed and used when upgrading an older database. It should not be used for new installations anymore.

Compiling the PostgreSQL module🔗

The tokeinzer needs a special C module for PostgreSQL which is not compiled by default. If you need the legacy tokenizer, compile Nominatim as follows:

mkdir build
cd build
cmake -DBUILD_MODULE=on
make

Enabling the tokenizer🔗

To enable the tokenizer add the following line to your project configuration:

NOMINATIM_TOKENIZER=legacy

The Postgresql module for the tokenizer is available in the module directory and also installed with the remainder of the software under lib/nominatim/module/nominatim.so. You can specify a custom location for the module with

NOMINATIM_DATABASE_MODULE_PATH=<path to directory where nominatim.so resides>

This is in particular useful when the database runs on a different server. See Advanced installations for details.

There are no other configuration options for the legacy tokenizer. All normalization functions are hard-coded.

ICU tokenizer🔗

The ICU tokenizer uses the ICU library to normalize names and queries. It also offers configurable decomposition and abbreviation handling. This tokenizer is currently the default.

To enable the tokenizer add the following line to your project configuration:

NOMINATIM_TOKENIZER=icu

How it works🔗

On import the tokenizer processes names in the following three stages:

During the Sanitizer step incoming names are cleaned up and converted to full names. This step can be used to regularize spelling, split multi-name tags into their parts and tag names with additional attributes. See the Sanitizers section below for available cleaning routines.
The Normalization part removes all information from the full names that are not relevant for search.
The Token analysis step takes the normalized full names and creates all transliterated variants under which the name should be searchable. See the Token analysis section below for more information.

During query time, only normalization and transliteration are relevant. An incoming query is first split into name chunks (this usually means splitting the string at the commas) and the each part is normalised and transliterated. The result is used to look up places in the search index.

Configuration🔗

The ICU tokenizer is configured using a YAML file which can be configured using NOMINATIM_TOKENIZER_CONFIG. The configuration is read on import and then saved as part of the internal database status. Later changes to the variable have no effect.

Here is an example configuration file:

normalization:
    - ":: lower ()"
    - "ß > 'ss'" # German szet is unambiguously equal to double ss
transliteration:
    - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml
    - ":: Ascii ()"
sanitizers:
    - step: split-name-list
token-analysis:
    - analyzer: generic
      variants:
          - !include icu-rules/variants-ca.yaml
          - words:
              - road -> rd
              - bridge -> bdge,br,brdg,bri,brg
      mutations:
          - pattern: 'ä'
            replacements: ['ä', 'ae']

The configuration file contains four sections: normalization, transliteration, sanitizers and token-analysis.

Normalization and Transliteration🔗

The normalization and transliteration sections each define a set of ICU rules that are applied to the names.

The normalization rules are applied after sanitation. They should remove any information that is not relevant for search at all. Usual rules to be applied here are: lower-casing, removing of special characters, cleanup of spaces.

The transliteration rules are applied at the end of the tokenization process to transfer the name into an ASCII representation. Transliteration can be useful to allow for further fuzzy matching, especially between different scripts.

Each section must contain a list of ICU transformation rules. The rules are applied in the order in which they appear in the file. You can also include additional rules from external yaml file using the !include tag. The included file must contain a valid YAML list of ICU rules and may again include other files.

Warning

The ICU rule syntax contains special characters that conflict with the YAML syntax. You should therefore always enclose the ICU rules in double-quotes.

Sanitizers🔗

The sanitizers section defines an ordered list of functions that are applied to the name and address tags before they are further processed by the tokenizer. They allows to clean up the tagging and bring it to a standardized form more suitable for building the search index.

Hint

Sanitizers only have an effect on how the search index is built. They do not change the information about each place that is saved in the database. In particular, they have no influence on how the results are displayed. The returned results always show the original information as stored in the OpenStreetMap database.

Each entry contains information of a sanitizer to be applied. It has a mandatory parameter step which gives the name of the sanitizer. Depending on the type, it may have additional parameters to configure its operation.

The order of the list matters. The sanitizers are applied exactly in the order that is configured. Each sanitizer works on the results of the previous one.

The following is a list of sanitizers that are shipped with Nominatim.

split-name-list🔗

Sanitizer that splits lists of names into their components.

PARAMETER	DESCRIPTION
`delimiters`	Define the set of characters to be used for splitting the list. (default: ',;')

strip-brace-terms🔗

This sanitizer creates additional name variants for names that have addendums in brackets (e.g. "Halle (Saale)"). The additional variant contains only the main name part with the bracket part removed.

tag-analyzer-by-language🔗

This sanitizer sets the analyzer property depending on the language of the tag. The language is taken from the suffix of the name. If a name already has an analyzer tagged, then this is kept.

Arguments:

filter-kind: Restrict the names the sanitizer should be applied to
             the given tags. The parameter expects a list of
             regular expressions which are matched against 'kind'.
             Note that a match against the full string is expected.
whitelist: Restrict the set of languages that should be tagged.
           Expects a list of acceptable suffixes. When unset,
           all 2- and 3-letter lower-case codes are accepted.
use-defaults:  Configure what happens when the name has no suffix.
               When set to 'all', a variant is created for
               each of the default languages in the country
               the feature is in. When set to 'mono', a variant is
               only created, when exactly one language is spoken
               in the country. The default is to do nothing with
               the default languages of a country.
mode: Define how the variants are created and may be 'replace' or
      'append'. When set to 'append' the original name (without
      any analyzer tagged) is retained. (default: replace)

clean-housenumbers🔗

Sanitizer that preprocesses address tags for house numbers. The sanitizer allows to

define which tags are to be considered house numbers (see 'filter-kind')
split house number lists into individual numbers (see 'delimiters')

PARAMETER	DESCRIPTION
`delimiters`	Define the set of characters to be used for splitting a list of house numbers into parts. (default: ',;')
`filter-kind`	Define the address tags that are considered to be a house number. Either takes a single string or a list of strings, where each string is a regular expression. An address item is considered a house number if the 'kind' fully matches any of the given regular expressions. (default: 'housenumber')
`convert-to-name`	Define house numbers that should be treated as a name instead of a house number. Either takes a single string or a list of strings, where each string is a regular expression that must match the full house number value.

clean-postcodes🔗

Sanitizer that filters postcodes by their officially allowed pattern.

PARAMETER	DESCRIPTION
`convert-to-address`	If set to 'yes' (the default), then postcodes that do not conform with their country-specific pattern are converted to an address component. That means that the postcode does not take part when computing the postcode centroids of a country but is still searchable. When set to 'no', non-conforming postcodes are not searchable either.
`default-pattern`	Pattern to use, when there is none available for the country in question. Warning: will not be used for objects that have no country assigned. These are always assumed to have no postcode.

clean-tiger-tags🔗

Sanitizer that preprocesses tags from the TIGER import.

It makes the following changes:

remove state reference from tiger:county

delete-tags🔗

Sanitizer which prevents certain tags from getting into the search index. It remove tags which matches all properties given below.

PARAMETER	DESCRIPTION
`type`	Define which type of tags should be considered for removal. There are two types of tags 'name' and 'address' tags. Takes a string 'name' or 'address'. (default: 'name')
`filter-kind`	Define which 'kind' of tags should be removed. Takes a string or list of strings where each string is a regular expression. A tag is considered to be a candidate for removal if its 'kind' property fully matches any of the given regular expressions. Note that by default all 'kind' of tags are considered.
`suffix`	Define the 'suffix' property of the tags which should be removed. Takes a string or list of strings where each string is a regular expression. A tag is considered to be a candidate for removal if its 'suffix' property fully matches any of the given regular expressions. Note that by default tags with any suffix value are considered including those which don't have a suffix at all.
`name`	Define the 'name' property corresponding to the 'kind' property of the tag. Takes a string or list of strings where each string is a regular expression. A tag is considered to be a candidate for removal if its name fully matches any of the given regular expressions. Note that by default tags with any 'name' are considered.
`country_code`	Define the country code of places whose tags should be considered for removed. Takes a string or list of strings where each string is a two-letter lower-case country code. Note that by default tags of places with any country code are considered including those which don't have a country code at all.
`rank_address`	Define the address rank of places whose tags should be considered for removal. Takes a string or list of strings where each string is a number or range of number or the form -. Note that default is '0-30', which means that tags of all places are considered. See https://nominatim.org/release-docs/latest/customize/Ranking/#address-rank to learn more about address rank.

tag-japanese🔗

This sanitizer maps OSM data to Japanese block addresses. It replaces blocknumber and housenumber with housenumber, and quarter and neighbourhood with place.

Token Analysis🔗

Token analyzers take a full name and transform it into one or more normalized form that are then saved in the search index. In its simplest form, the analyzer only applies the transliteration rules. More complex analyzers create additional spelling variants of a name. This is useful to handle decomposition and abbreviation.

The ICU tokenizer may use different analyzers for different names. To select the analyzer to be used, the name must be tagged with the analyzer attribute by a sanitizer (see for example the tag-analyzer-by-language sanitizer).

The token-analysis section contains the list of configured analyzers. Each analyzer must have an id parameter that uniquely identifies the analyzer. The only exception is the default analyzer that is used when no special analyzer was selected. There are analysers with special ids:

'@housenumber'. If an analyzer with that name is present, it is used for normalization of house numbers.
'@potcode'. If an analyzer with that name is present, it is used for normalization of postcodes.

Different analyzer implementations may exist. To select the implementation, the analyzer parameter must be set. The different implementations are described in the following.

Generic token analyzer🔗

The generic analyzer generic is able to create variants from a list of given abbreviation and decomposition replacements and introduce spelling variations.

Variants🔗

The optional 'variants' section defines lists of replacements which create alternative spellings of a name. To create the variants, a name is scanned from left to right and the longest matching replacement is applied until the end of the string is reached.

The variants section must contain a list of replacement groups. Each group defines a set of properties that describes where the replacements are applicable. In addition, the word section defines the list of replacements to be made. The basic replacement description is of the form:

<source>[,<source>[...]] => <target>[,<target>[...]]

The left side contains one or more source terms to be replaced. The right side lists one or more replacements. Each source is replaced with each replacement term.

Tip

The source and target terms are internally normalized using the normalization rules given in the configuration. This ensures that the strings match as expected. In fact, it is better to use unnormalized words in the configuration because then it is possible to change the rules for normalization later without having to adapt the variant rules.

Decomposition🔗

In its standard form, only full words match against the source. There is a special notation to match the prefix and suffix of a word:

- ~strasse => str  # matches "strasse" as full word and in suffix position
- hinter~ => hntr  # matches "hinter" as full word and in prefix position

There is no facility to match a string in the middle of the word. The suffix and prefix notation automatically trigger the decomposition mode: two variants are created for each replacement, one with the replacement attached to the word and one separate. So in above example, the tokenization of "hauptstrasse" will create the variants "hauptstr" and "haupt str". Similarly, the name "rote strasse" triggers the variants "rote str" and "rotestr". By having decomposition work both ways, it is sufficient to create the variants at index time. The variant rules are not applied at query time.

To avoid automatic decomposition, use the '|' notation:

- ~strasse |=> str

simply changes "hauptstrasse" to "hauptstr" and "rote strasse" to "rote str".

Initial and final terms🔗

It is also possible to restrict replacements to the beginning and end of a name:

- ^south => s  # matches only at the beginning of the name
- road$ => rd  # matches only at the end of the name

So the first example would trigger a replacement for "south 45th street" but not for "the south beach restaurant".

Replacements vs. variants🔗

The replacement syntax source => target works as a pure replacement. It changes the name instead of creating a variant. To create an additional version, you'd have to write source => source,target. As this is a frequent case, there is a shortcut notation for it:

<source>[,<source>[...]] -> <target>[,<target>[...]]

The simple arrow causes an additional variant to be added. Note that decomposition has an effect here on the source as well. So a rule

- "~strasse -> str"

means that for a word like hauptstrasse four variants are created: hauptstrasse, haupt strasse, hauptstr and haupt str.

Mutations🔗

The 'mutation' section in the configuration describes an additional set of replacements to be applied after the variants have been computed.

Each mutation is described by two parameters: pattern and replacements. The pattern must contain a single regular expression to search for in the variant name. The regular expressions need to follow the syntax for Python regular expressions. Capturing groups are not permitted. replacements must contain a list of strings that the pattern should be replaced with. Each occurrence of the pattern is replaced with all given replacements. Be mindful of combinatorial explosion of variants.

Modes🔗

The generic analyser supports a special mode variant-only. When configured then it consumes the input token and emits only variants (if any exist). Enable the mode by adding:

  mode: variant-only

to the analyser configuration.

Housenumber token analyzer🔗

The analyzer housenumbers is purpose-made to analyze house numbers. It creates variants with optional spaces between numbers and letters. Thus, house numbers of the form '3 a', '3A', '3-A' etc. are all considered equivalent.

The analyzer cannot be customized.

Postcode token analyzer🔗

The analyzer postcodes is pupose-made to analyze postcodes. It supports a 'lookup' variant of the token, which produces variants with optional spaces. Use together with the clean-postcodes sanitizer.

The analyzer cannot be customized.

Reconfiguration🔗

Changing the configuration after the import is currently not possible, although this feature may be added at a later time.