Writing custom sanitizer modules🔗

Sanitizers are used for preprocessing name and address information from the OpenStreetMap input data for the search index. Read more about them in the Customizing sanitizers section.

This section explains how to write your own sanitizer step function.

Using custom sanitizer modules🔗

To use a custom made sanitizer step, simply refer to the sanitizer module in the step property. There are two ways to include external modules: through a library or from the project directory.

To include a module from a library, use the absolute import path as name and make sure the library can be found in your PYTHONPATH.

Example

You have your sanitizer steps in a Python package my_sanitizer and want to refer to the step implemented in module translate_street.py.

- step: my_sanitizer.translate_street
  config: some other config info for the step

To use a custom module without creating a library, you can put the module somewhere in your project directory and then use the relative path to the file. Include the whole name of the file including the .py ending.

Example

You have put your module translate_street.py directly into the project directory.

- step: translate_street.py
  config: some other config info for the step

Basic sanitizer module setup🔗

A sanitizer module must export a single factory function create with the following signature:

def create(config: SanitizerConfig) -> Callable[[ProcessInfo], None]

The function receives the custom configuration for the sanitizer and must return a callable (function or class) that transforms the name and address terms of a place. When a place is processed, then a ProcessInfo object is created from the information that was queried from the database. This object is sequentially handed to each configured sanitizer, so that each sanitizer receives the result of processing from the previous sanitizer. After the last sanitizer is finished, the resulting name and address lists are forwarded to the token analysis module.

Sanitizer functions are instantiated once and then called for each place that is imported or updated. They don't need to be thread-safe. If multi-threading is used, each thread creates their own instance of the function.

Sanitizer configuration🔗

The SanitizerConfig class is a read-only dictionary with configuration options for the sanitizer. In addition to the usual dictionary functions, the class provides accessors to standard sanitizer options that are used by many of the sanitizers.

`get_bool(param, default=None)` 🔗

Extract a configuration parameter as a boolean.

Parameters:

Name	Type	Description	Default
`param`	`str`	Name of the configuration parameter. The parameter must contain one of the yaml boolean values or an UsageError will be raised.	required
`default`	`Optional[bool]`	Value to return, when the parameter is missing. When set to `None`, the parameter must be defined.	`None`

Returns:

Type	Description
`bool`	Boolean value of the given parameter.

`get_choice(param, opt_cls, default=None)` 🔗

Return an integer representing a value from a fixed list of choices.

The choices must be given in form of a class containing class constants of type int. The given value in the configuration must be a string which, after having been cnverted to upper case and hyphens having been replaced with underscores, corresponds to one of the class constants. Only values starting with an ASCII letter and containing ASCII letters and numbers, hyphen and underscore are accepted.

Parameters:

Name	Type	Description	Default
`param`	`str`	The parameter containing the choice variable.	required
`opt_cls`	`type[Any]`	The class describing valid choices.	required
`default`	`Optional[int]`	Default value to use when none is supplied in the configuration. Must be an integer representing one of the choices. When left None (the default), then the parameter is required and leaving it out will throw an error.	`None`

Returns:

Type	Description
`int`	The integer with the selected choice.

`get_delimiter(default=',;')` 🔗

Return the 'delimiters' parameter in the configuration as a compiled regular expression that can be used to split strings on these delimiters.

Parameters:

Name	Type	Description	Default
`default`	`str`	Delimiters to be used when 'delimiters' parameter is not explicitly configured.	`',;'`

Returns:

Type	Description
`Pattern[str]`	A regular expression pattern which can be used to split a string. The regular expression makes sure that the resulting names are stripped and that repeated delimiters are ignored. It may still create empty fields on occasion. The code needs to filter those.

`get_filter(param, default='PASS_ALL')` 🔗

Returns a filter function for the given parameter of the sanitizer configuration.

The value provided for the parameter in sanitizer configuration should be a string or list of strings, where each string is a regular expression. These regular expressions will later be used by the filter function to filter strings.

Parameters:

Name	Type	Description	Default
`param`	`str`	The parameter for which the filter function will be created.	required
`default`	`Union[str, Sequence[str]]`	Defines the behaviour of filter function if parameter is missing in the sanitizer configuration. Takes a string(PASS_ALL or FAIL_ALL) or a list of strings. Any other value of string or an empty list is not allowed, and will raise a ValueError. If the value is PASS_ALL, the filter function will let all strings to pass, if the value is FAIL_ALL, filter function will let no strings to pass. If value provided is a list of strings each string is treated as a regular expression. In this case these regular expressions will be used by the filter function. By default allow filter function to let all strings pass.	`'PASS_ALL'`

Returns:

Type	Description
`Callable[[str], bool]`	A filter function that takes a target string as the argument and returns True if it fully matches any of the regular expressions otherwise returns False.

`get_pattern(param, default=None, flags=0)` 🔗

Extract a configuration parameter as a regular expression and return a compiled regex pattern. Use 'flags' to initiate the pattern with different compile flags.

If 'default' is not given, then the parameter must exist or a UsageError is thrown.

`get_string_list(param, default=tuple())` 🔗

Extract a configuration parameter as a string list.

Parameters:

Name	Type	Description	Default
`param`	`str`	Name of the configuration parameter.	required
`default`	`Sequence[str]`	Takes a tuple or list of strings which will be returned if the parameter is missing in the sanitizer configuration. Note that if this default parameter is not provided then an empty list is returned.	`tuple()`

Returns:

Type	Description
`Sequence[str]`	If the parameter value is a simple string, it is returned as a one-item list. If the parameter value does not exist, the given default is returned. If the parameter value is a list, it is checked to contain only strings before being returned.

The main filter function of the sanitizer🔗

The filter function receives a single object of type ProcessInfo which has with three members:

place: PlaceInfo: read-only information about the place being processed. See PlaceInfo below.
names: List[PlaceName]: The current list of names for the place.
address: List[PlaceName]: The current list of address names for the place.

While the place member is provided for information only, the names and address lists are meant to be manipulated by the sanitizer. It may add and remove entries, change information within a single entry (for example by adding extra attributes) or completely replace the list with a different one.

PlaceInfo - information about the place🔗

This data class contains all information the tokenizer can access about a place.

`address` `property` 🔗

A dictionary with the address elements of the place. They key usually corresponds to the suffix part of the key of an OSM 'addr:' or 'isin:' tag. There are also some special keys like country or country_code which merge OSM keys that contain the same information. See Import Styles for details.

The property may be None if the place has no address information.

`centroid` `property` 🔗

A center point of the place in WGS84. May be None when the geometry of the place is unknown.

`country_code` `property` 🔗

The country code of the country the place is in. Guaranteed to be a two-letter lower-case string. If the place is not inside any country, the property is set to None.

`name` `property` 🔗

A dictionary with the names of the place. Keys and values represent the full key and value of the corresponding OSM tag. Which tags are saved as names is determined by the import style. The property may be None if the place has no names.

`rank_address` `property` 🔗

The rank address before any rank correction is applied.

`searchable_address` `property` 🔗

List of address terms that should be used to build the search index.

`searchable_names` `property` 🔗

List of place names that should be indexed for searching.

`is_a(key, value)` 🔗

Set to True when the place's primary tag corresponds to the given key and value.

`is_country()` 🔗

Set to True when the place is a valid country boundary.

`is_street()` 🔗

Return True when the place is a street object.

PlaceName - extended naming information🔗

Each name and address part of a place is encapsulated in an object of this class. It saves not only the name proper but also describes the kind of name with two properties:

kind describes the name of the OSM key used without any suffixes (i.e. the part after the colon removed)
suffix contains the suffix of the OSM tag, if any. The suffix is the part of the key after the first colon.

In addition to that, a name may have arbitrary additional attributes. How attributes are used, depends on the sanitizers and token analysers.

The default ICU token analyser currently understands the following additional properties:

analyzer determines which token analysis module will be used to finalize the treatment of names.
partial marks a name that is not a full name in itself but may rarely appear in conjunction with the other names. Can be used to allow prefixes and suffixes to be recognised but not found on their own. Matches against such a partial match will always be lower ranked than full name matches.

`clone(name=None, kind=None, suffix=None, attr=None)` 🔗

Create a deep copy of the place name, optionally with the given parameters replaced. In the attribute list only the given keys are updated. The list is not replaced completely. In particular, the function cannot to be used to remove an attribute from a place name.

`get_attr(key, default=None)` 🔗

Return the given property or the value of 'default' if it is not set.

`has_attr(key)` 🔗

Check if the given attribute is set.

`set_attr(key, value)` 🔗

Add the given property to the name. If the property was already set, then the value is overwritten.

Example: Filter for US street prefixes🔗

The following sanitizer removes the directional prefixes from street names in the US:

Example

import re

def _filter_function(obj):
    if obj.place.country_code == 'us' \
       and obj.place.rank_address >= 26 and obj.place.rank_address <= 27:
        for name in obj.names:
            name.name = re.sub(r'^(north|south|west|east) ',
                               '',
                               name.name,
                               flags=re.IGNORECASE)

def create(config):
    return _filter_function

This is the most simple form of a sanitizer module. If defines a single filter function and implements the required create() function by returning the filter.

The filter function first checks if the object is interesting for the sanitizer. Namely it checks if the place is in the US (through country_code) and it the place is a street (a rank_address of 26 or 27). If the conditions are met, then it goes through all available names and removes any leading directional prefix using a simple regular expression.

Save the source code in a file in your project directory, for example as us_streets.py. Then you can use the sanitizer in your icu_tokenizer.yaml:

...
sanitizers:
    - step: us_streets.py
...

Warning

This example is just a simplified show case on how to create a sanitizer. It is not really meant for real-world use: while the sanitizer would correctly transform West 5th Street into 5th Street. it would also shorten a simple North Street to Street.

For more sanitizer examples, have a look at the sanitizers provided by Nominatim. They can be found in the directory src/nominatim_db/tokenizer/sanitizers.

Writing custom sanitizer modules🔗

Using custom sanitizer modules🔗

Basic sanitizer module setup🔗

Sanitizer configuration🔗

get_bool(param, default=None) 🔗

get_choice(param, opt_cls, default=None) 🔗

get_delimiter(default=',;') 🔗

get_filter(param, default='PASS_ALL') 🔗

get_pattern(param, default=None, flags=0) 🔗

get_string_list(param, default=tuple()) 🔗

The main filter function of the sanitizer🔗

PlaceInfo - information about the place🔗

address property 🔗

centroid property 🔗

country_code property 🔗

name property 🔗

rank_address property 🔗

searchable_address property 🔗

searchable_names property 🔗

is_a(key, value) 🔗

is_country() 🔗

is_street() 🔗

PlaceName - extended naming information🔗

clone(name=None, kind=None, suffix=None, attr=None) 🔗

get_attr(key, default=None) 🔗

has_attr(key) 🔗

set_attr(key, value) 🔗

Example: Filter for US street prefixes🔗

`get_bool(param, default=None)` 🔗

`get_choice(param, opt_cls, default=None)` 🔗

`get_delimiter(default=',;')` 🔗

`get_filter(param, default='PASS_ALL')` 🔗

`get_pattern(param, default=None, flags=0)` 🔗

`get_string_list(param, default=tuple())` 🔗

`address` `property` 🔗

`centroid` `property` 🔗

`country_code` `property` 🔗

`name` `property` 🔗

`rank_address` `property` 🔗

`searchable_address` `property` 🔗

`searchable_names` `property` 🔗

`is_a(key, value)` 🔗

`is_country()` 🔗

`is_street()` 🔗

`clone(name=None, kind=None, suffix=None, attr=None)` 🔗

`get_attr(key, default=None)` 🔗

`has_attr(key)` 🔗

`set_attr(key, value)` 🔗