Leave No Bit Unturned

APACHE, GEOINFORMATICS, NOMINATIM, OPENSTREETMAP, POSTGIS,

POSTGRESQL, UBUNTU

OPENSTREETMAP
NOMINATIM SERVER FOR
GEOCODING
2015-03-19 | KORTSI | 31 COMMENTS

    
5.00 avg. rating (98% score) - 9 votes
Here’s how to install the OpenStreetMap Nominatim service on
your own server. It can be used to geocode and reverse geocode
addresses and map coordinates. You will also get a web interface
which loads map tiles from openstreetmap.org while doing
geocoding requests using your own server.

I was faced with thousands of geocoding requests per hour, and with
such a high volume it is not wise to burden the global OSM servers.
Even with aggressive caching it would have potentially been too
much.

I will use Ubuntu 14.04 LTS as the platform. Just a basic install with
ssh server. We will install Apache to serve http requests. Make sure
you have enough disk space and RAM to hold the data and serve it
ef ciently. I used the Finland extract, which was about a 200 MB
download. The resulting database was 26 GB after importing,
indexing and adding Wikipedia data. The Wikipedia data probably
actually took more disk space than the OSM data. My server has 4
GB RAM, which seems to be enough for this small data set.

Table of Contents [hide]

1 Sofware requirements
2 Kernel tuning
3 PostgreSQL tuning
 4 Dedicated user
5 Database users
6 Download and Compile Nominatim
6.1 Clone the Repository from Github
6.2 Compile Nominatim
6.3 Con gure Local Settings
7 Download Data
8 Import Data
9 Database Production Settings
10 Create the Web Site
10.1 Apache con guration
10.2 Install the Nominatim Web Site
11 Enable OSM Updates
12 Wrap Up
12.1 Useful links

Sofware requirements
PostgreSQL with PostGIS extension:

1 sudo apt-get install \

2   postgresql postgis postgresql-contrib \
3   postgresql-server-dev-9.3 postgresql-9.3-postgis-2.1 \
4   postgresql-doc-9.3 postgis-doc

Apache with PHP5:

1 apt-get install \
2   apache2 php5 php-pear php5-pgsql php5-json php-db

Git and various tools:

1 apt-get install \
2   wget git autoconf-archive build-essential \
3   automake gcc proj-bin

Tool for handling OpenStreetMap data:

1 apt-get install osmosis

Needed libraries:

1 apt-get install \
2   libxml2-dev libgeos-dev libpq-dev libbz2-dev libtool \
3   automake libproj-dev libboost-dev libboost-system-dev \
4   libboost-filesystem-dev libboost-thread-dev \
5   libgeos-c1 libgeos++-dev lua5.2 liblua5.2-dev \
6   libprotobuf-c0-dev protobuf-c-compiler
Kernel tuning
We must increase kernel shared memory limits. Also we’ll reduce
swappiness and set the kernel memory overcommit to be a bit more
conservative. Make sure you have anough swap space, more than
physical memory is a good idea.

These have been estimated using 4 GB RAM.

1 sysctl -w kernel.shmmax=4404019200
2 sysctl -w kernel.shmall=1075200
3 sysctl vm.overcommit_memory=2
4 sysctl vm.swappiness=10
5  
6 echo "kernel.shmmax=4404019200" >> /etc/sysctl.conf
7 echo "kernel.shmall=1075200" >> /etc/sysctl.conf
8 echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
9 echo "vm.swappiness=10" >> /etc/sysctl.conf

PostgreSQL tuning
The following directives were set in
/etc/postgresql/9.3/main/postgresql.conf:

1 shared_buffers                = 2048MB
2 work_mem                      = 50MB
3 maintenance_work_mem          = 1024M
4 fsync                         = off    # dangerous!
5 synchronous_commit            = off
6 full_page_writes              = off    # dangerous!
7 checkpoint_segments           = 100
8 checkpoint_timeout            = 10min
9 checkpoint_completion_target  = 0.9
10 effective_cache_size          = 2048M

The buffer and memory values depend on your available RAM so

you should set sane values accordingly.  These are good for 4 GB
RAM.

There are two truly dangerous settings here. Setting fsync off will
cause PostgreSQL to report successful commits before the disk has
con rmed the data has been written successfully. Setting
full_page_writes off exposes the database to the possibility of
partially updated data in case of power failure. These options are
used here only for the duration of the initial import, because setting
them so makes the import much faster. Fsync and full page writes
must be turned back on after the import to ensure database
consistency.
The synchronous_commit also jeopardizes the durability of
committed transactions by reporting successful commits to clients
asynchronously, meaning that the transaction may not have been
fully written to disk. It does not compromize  consistency as the two
previous directives discussed. It just means that in the case of power
failure, some recent transactions that were already reported to the
client as having been committed, may in fact be aborted and rolled
back. The database will still be in a consistent state. We’ll leave it off
because it speeds up the queries, and we don’t really care about it in
this database. We can always rebuild it from scratch. But if you have
other important databases in the same database cluster, I would
recommend turning it back on as well, since the setting is cluster-
wide.

The maintenance_work_mem will also be reduced to a lower value

later, after the import.

Restart PostgreSQL to apply changes:

1 pg_ctlcluster 9.3 main restart

Dedicated user
We’ll install the software under this user’s home directory (user id
“nominatim”).

1 useradd -c 'OpenStreetMap Nominatim' -m -s /bin/bash nominatim

Database users
We will also make “nominatim” a database user, as well as the
already created “www-data” (created by the apache2 package).

Nominatim will be a superuser and www-data will be a regular one.

This must be done as a database administrator. The “postgres” user

by default is one (root is not). Change user:

1 su - postgres

Create database users:

1 createuser -sdRe nominatim
2 createuser -SDRe www-data
3 exit

Download and Compile Nominatim

Su to user nominatim:

1 su - nominatim

Set some environment variables. We’ll use these later. These are the
download locations for les and updates. Lines 3 and 4 below are for
Finland. Customize to your needs.

Also set the BASE_URL to point to your server and install directory.
The last forward slash seems to be important.

1 WIKIPEDIA_ARTICLES="http://www.nominatim.org/data/wikipedia_artic
2 WIKIPEDIA_REDIRECTS="http://www.nominatim.org/data/wikipedia_redi
3 OSM_LATEST="http://download.geofabrik.de/europe/finland-latest.os
4 OSM_UPDATES="http://download.geofabrik.de/europe/finland-updates"
5 BASE_URL="http://maps.example.org/nominatim/"

You can browse available areas at download.geofabrik.de.

Clone the Repository from Github

We will use the release 2.3 branch:

1 git clone --recursive \

2   https://github.com/twain47/Nominatim.git \
3   --branch release_2.3
4  
5 cd Nominatim

The –recursive option is needed to clone everything, because the

repository contains submodules.

Compile Nominatim
Simply:

1 ./autogen.sh
2 ./configure
3 make

If there were no errors, we are good. In case of missing libraries,

check that you installed all requirements earlier.

Con gure Local Settings

Some required settings should go to ~/Nominatim/settings/local.php:

1 cat >settings/local.php <<EOF

2 <?php
3 // Paths
4 @define('CONST_Postgresql_Version', '9.3');
5 @define('CONST_Postgis_Version', '2.1');
6 @define('CONST_Website_BaseURL', '$BASE_URL');
7  
8 // Update process
9 @define('CONST_Replication_Url', '$OSM_UPDATES');
10 @define('CONST_Replication_MaxInterval', '86400');
11 @define('CONST_Replication_Update_Interval', '86400');
12 @define('CONST_Replication_Recheck_Interval', '900');
13 ?>
14 EOF

The last four de nitions are only required if you are going to do
incremental updates.

Download Data
OpenStreetMap data (about 200 MB for Finland):

1 wget -O data/latest.osm.pbf $OSM_LATEST

The OSM data is all that is really required. The rest below are
optional.

Wikipedia data (about 1.4 GB):

1 wget -O data/wikipedia_article.sql.bin $WIKIPEDIA_ARTICLES

2 wget -O data/wikipedia_redirect.sql.bin $WIKIPEDIA_REDIRECTS

Special phrases for country codes and names (very small):

1 ./utils/specialphrases.php --countries >data/specialphrases_count

Special search phrases (a few megabytes):

1 ./utils/specialphrases.php --wiki-import >data/specialphrases.sql

Next we’ll import all this stuff into the database.

Import Data
The utils/setup.php will create a new database called “nominatim”
and import the given .pbf le into it. This will take a long time
depending on your PostgreSQL settings, available memory, disk
speed and size of dataset. The full planet can take days to import on
modern hardware. My small dataset took a bit over two hours.

1 ./utils/setup.php \
2   --osm-file data/latest.osm.pbf \
3   --all --osm2pgsql-cache 1024 2>&1 \
4   | tee setup.log

The messages will be saved into setup.log in case you need to look at
them later.

If you had the Wikipedia data downloaded, the setup should have
imported that automatically and told you about it. You can import
the special phrases data if you downloaded it earlier using these
commands:

1 psql -d nominatim -f data/specialphrases_countries.sql

2 psql -d nominatim -f data/specialphrases.sql

Database Production Settings

Now that the import is done, it is time to con gure the database to
settings that are suitable for production use. The following changes
were made in /etc/postgresql/9.3/main/postgresql.conf:

1 maintenance_work_mem          = 128M
2 fsync                         = on
3 full_page_writes              = on

You should also set the synchronous_commit directive to on if you

have other databases running on this same database cluster. See the
PostgreSQL Tuning section earlier in this post.

Apply changes:

1 pg_ctlcluster 9.3 main restart

Create the Web Site

The following commands have to be run as root.

Create a directory to install the site into and set permissions:

1 mkdir /var/www/html/nominatim
2 chown nominatim:www-data /var/www/html/nominatim
3 chmod 755 /var/www/html/nominatim
4 chmod g+s /var/www/html/nominatim

Ask bots to keep out:

1 cat >/var/www/html/nominatim/robots.txt <<'EOF'

2 User-agent: *
3 Disallow: /
4 EOF

Apache con guration

Edit the default site con guration le /etc/apache2/sites-
enabled/000-default.conf and make it look like something like below:

1 <VirtualHost *:80>
2   ServerName maps.example.org
3   ServerAdmin [email protected]
4   DocumentRoot /var/www/html
5  
6   ErrorLog ${APACHE_LOG_DIR}/error.log
7   CustomLog ${APACHE_LOG_DIR}/access.log combined
8  
9   <Directory "/var/www/html/nominatim">
10     Options FollowSymLinks MultiViews
11     AddType text/html .php
12   </Directory>
13 </VirtualHost>

Apply changes by restarting web server:

1 apache2ctl restart

Install the Nominatim Web Site

The installation should be done as the nominatim user:

1 su - nominatim
2 cd Nominatim

Run the setup.php with the option to create the web site:

1 ./utils/setup.php --create-website /var/www/html/nominatim

At this point, the site is ready and you can point your browser to the
base URL and try it out. It should look something like this:
Enable OSM Updates
We will con gure a cron job to have the database updated
periodically using diffs from Geofabrik.de. As the nominatim user:

1 rm settings/configuration.txt
2 ./utils/setup.php --osmosis-init

Enable hierarchical updates in the database (these were off during

import to speed things up):

1 ./utils/setup.php --create-functions --enable-diff-updates

Run once to get up to date:

1 ./utils/update.php --import-osmosis --no-npi

There seems to be a time check in the import function that prevents

another run immediately. Instead it will wait until the interval
con gured in the “Con gure Local Settings” section earlier in this
post has passed (default is 1 day).

We will add the command to crontab to be executed every monday

at 03:00. Run:

1 crontab -e

Add a new line to the crontab le looking like this:

 1 00 03 * * mon /home/nominatim/Nominatim/utils/update.php --import

A log le will be created as /home/nominatim/nominatim-update.log.

Wrap Up
That’s it! You can get JSON or XML data out of the database with
http requests. A couple of examples:

Forward geocoding:

http://maps.example.org/nominatim/search.php?
format=jsonv2&q=mannerheimintie, helsinki

Reverse geocoding:

http://maps.example.org/nominatim/reverse.php?
format=jsonv2&lat=60.1805&lon=24.9434

Check out the Nominatim documentation for details.

This post was adapted from my notes of such an installation process.

Let me know in the comments section if I introduced any mistakes.
Other comments are also welcome!

Useful links
http://www.openstreetmap.org
http://nominatim.openstreetmap.org/
http://wiki.openstreetmap.org/wiki/Nominatim
http://wiki.openstreetmap.org/wiki/Nominatim/Installation
https://github.com/cyclestreets/nominatim-install

SHARE:

     More

LIKE THIS:

Like
Be the first to like this.
31 THOUGHTS ON “OPENSTREETMAP NOMINATIM SERVER
FOR GEOCODING”

Luis Ernesto Mendoza Sánchez

2015-03-23 AT 08:19

Hi, you used 32 or 64 bit version?

Thanks!

kortsi
2015-03-26 AT 14:03

64-bit Ubuntu

Gustavo Ferreira
2015-04-08 AT 02:12

Hello Kortsi, great article.

When I try to access the reverse.php get this: “{” error “:” Unable
to geocode “}”. I followed all the steps of your article, which can
be?

Thanks for your attention.

 kortsi
2015-04-08 AT 09:40

That may mean there’s something wrong with your query. Did
you give the “lat” and “lon” parameters? Those are required.

Marco Antonio Siqueiros

2015-06-30 AT 21:02

i have an erro in import data to postgres

Using projection SRS 4326 (Latlong)

NOTICE: table “place” does not exist, skipping
NOTICE: type “keyvalue” does not exist, skipping
NOTICE: type “wordscore” does not exist, skipping
NOTICE: type “stringlanguagetype” does not exist, skipping
NOTICE: type “keyvaluetype” does not exist, skipping
NOTICE: function get_connected_ways(pg_catalog.int4[]) does
not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Out of memory for dense node cache, reduce –cache size
Error occurred, cleaning up
ERROR: Error executing external command:
/home/nominatim/Nominatim/osm2pgsql/osm2pgsql -lsc -O
gazetteer –hstore -C 1024 -P 5432 -d nominatim
/home/nominatim/Nominatim/data/latest.osm.pbf
Error executing external command:
/home/nominatim/Nominatim/osm2pgsql/osm2pgsql -lsc -O
gazetteer –hstore -C 1024 -P 5432 -d nominatim
/home/nominatim/Nominatim/data/latest.osm.pbf

Marco Antonio Siqueiros

2015-06-30 AT 21:46

my web site it in “visual”

imagen of website
http://i60.tinypic.com/2mrv413.png

error when i want to shearch “restaurant in helsinki”

http://i60.tinypic.com/1zmju5l.png

 kortsi
2015-06-30 AT 22:23

I think you are running out of RAM because it says “Out of

memory for dense node cache, reduce –cache size”. That’s
why the whole import fails.

You should try with a lower –osm2pgsql-cache size. Or

increase swap size for the machine.

Marco Antonio Siqueiros

2015-07-01 AT 02:13
i put my ubuntu server in a computer with a cpu of 3 core with 4
ram

I solved my problem, it was the cache size, I put 1024

then i have the problem
——
Out of memory for dense node cache, reduce –cache size
Error occurred, cleaning up
——
i saw in a external post that i was the cache size, and i put 512
and it’s work….

this is the post

https://github.com/twain47/Nominatim/issues/105

——
Yes, you do because osm2pgsql reports it is out of memory. It is
currently trying to use 4GB and that seems to be too much. If
you have 8GB of memory, try with 2048 and if that does not
work either with 1024. Also make sure you have a few dozen GB
of swap enabled.
——

in URL: http://localhost/nominatim/
i got map, but in top it showing error

Data: DB Error: no such table

———————
You don’t need to change anything in any les, just run
./utils/setup.php –osm- le –all –osm2pgsql-cache 1024 for the
import as described in the installation instructions.
——————-

the problem was solved

http://i58.tinypic.com/2e3n4ed.png

Sarah
2015-08-13 AT 10:28

Setting postgresql’s shared_buffers to 2GB when you have only

4GB RAM is not the best of ideas because this memory will be
permanently reserved by the database server. Postgres
recommends to use around 25% of memory but something like
256MB is more to the point with so little RAM because there
are other applications running next to postgres during import
(notably osm2pgsql which hogs lots of memory).

Also, don’t forbid to overcommit memory unless you have really

a lot of RAM. osm2pgsql relies on being able to overcommit.

Al
2015-09-11 AT 08:29

Hello Kortsi, very interesting article.

As I read it to install our Nominatim DB, I give some feedback on

our experience.
The server used is vm with 4*3.1 Ghz, 15 Go Ram and high
speed harddrives of 400 Go.

We imported only some country (fr, uk, it, be, de, sp, it). To
process them we merged all les in a single one of 9 Go. It took 8
hours.
The import ran for about 36 hours and consumed 180 Go of
disk.

Best Regards
Al

gustavoDiniz
2015-09-28 AT 02:23

kortsi hello , good night!

when I try to use reverse geocoding , Info Some of addressparts

not appear at the moment only appears this for example :

Return , BrazilRetornoBrazilbr

You know what might be causing the lack of data ?

Sebass
2015-10-03 AT 01:02

Hi,

I am at the importing phase and when I type “./utils/setup.php –

osm- le data/latest.osm.pbf –all –osm2pgsql-cache 1024 2>&1
| tee setup.log” I get the error “the path supplied to –osm- le
does not exist”

Sebass
2015-10-03 AT 01:11

Worked it out I had downloaded osn le to root

“./utils/setup.php –osm- le latest.osm.pbf –all –osm2pgsql-
cache 1024 2>&1 | tee setup.log” xed it

Freddy Taborda
2015-10-14 AT 05:50

Muchas gracias por la excelente explicacion de como usar esta

herramienta, voy a trabajar esta con guracion para OpenGTS.
Saludos

ricarda
2016-02-17 AT 08:49

sorry – got a big problem. as a newbie i dont know much about

apache

after
./utils/setup.php –create-website /var/www/html/nominatim

i got that error:

WARNING: Unable to access the website at

http://maps.example.org/nominatim/
You may want to update settings/local.php with
@de ne(‘CONST_Website_BaseURL’, ‘http://[HOST]/[PATH]/’);
Setup nished.
 ricarda
2016-02-17 AT 20:29

it words now. this error is solved

Tavito
2016-05-20 AT 15:55

How did you solved it? Thanks in advance!

Neha
2017-03-02 AT 10:49

i am getting the same error, how did you solved it ?

Interspock
2018-08-18 AT 01:17

Same here… can anyone help me?

Andy
2016-04-10 AT 12:48

What more can I say but it works!

The most complete and accurate set up instructions I have

found!

Sambhav
2016-05-06 AT 14:02

Where can u nd the error log after installing the Nominatim

server ?

lieubv
2016-06-17 AT 10:17

Hi Kortsi.
This a best guide i ever read about Nominatim installation.
Even this time some changes need to be done that different
from what you said.
Anyway, i am enjoying it, that’s great.
Thank you so much.

[email protected]

David
2016-06-22 AT 10:25

Excellent instructions – these are the rst ones that actually

have worked rst time. I have imported a small sample country
to ensure that the platform is working as expected.

I will not be using the updating functionality. How do I now

import another country? What is the command required?

Thanks!!

Andy
2016-10-19 AT 15:12

If I wanted to use say jtiledownloader to export tiles please

could you tell me how?

Thanks

Andy.

Andy W
2017-05-12 AT 16:30

Databases are different, you have to do a full install of a

tileserver as well.

jbx
2017-03-05 AT 12:53

The –create-website option was removed last year and won’t

work.
https://github.com/twain47/Nominatim/commit/85c72cdccdb6
adcad7a78643c569935e88a73182
 Neha Sharma
2017-03-08 AT 08:40

Thank you!! It solved my problem. Now nominatim is working.

Neha Sharma
2017-03-08 AT 08:41

Thank You !! It solved my problem, now nominatim is working.

Andy W
2017-05-12 AT 16:33

I am looking at servicing 100 reverse geocoding requests per

second.

Does anybody know what sort of machine could do this, or

would multiple servers be required?

I have this running on a Virtualbox VM with 4 cores and 4GB

RAM but can only do about 10 per second.

Thanks

Andy.

Pim
2017-10-20 AT 09:12

Hi guys,

We have been trying to setup a server for quite some time now
and are getting database errors every time. After the import
into Postgres (the last step) it’s an error every singl time.

Is there a way/place where I can get an indexed DB dump which

I can import and skip the import process??

Hope you can help (i will be watching this threat)

 kortsi
2017-10-20 AT 13:08

I haven’t seen any DB dumps around. One issue might be size.

I did a full planet import about a year ago and the resulting
database took nearly a terabyte of disk space.

What kind of an error are you getting?