Providing Windows binaries/packages about osm2pgsql HOT 155 CLOSED

/cc @onepremise - based on your build system so you have a binary yet for osm2pgsql that runs?

from osm2pgsql.

/cc @emirhartato, member of @hotosm who I know would love to get access to a working build to use in a workshop by the last week in April

from osm2pgsql.

Hi Dane,

Yes I have a cygwin build which is working. I'm also getting together a
MinGW build which works with the new MinGW64 environment I've built. Let me
know which you're interested in so I can make it available.

Thanks,

Jason

On Mon, Apr 8, 2013 at 8:48 PM, Dane Springmeyer
[email protected]:

/cc @emirhartato https://github.com/emirhartato, member of @hotosmhttps://github.com/hotosmwho I know would love to get access to a working build to use in a workshop
by the last week in April

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16088128
.

from osm2pgsql.

Jason, both would be great for testing, but really whatever you have.

from osm2pgsql.

Hello, sorry for late reply.

This the error log for different build of osm2pgsql on Windows:
https://gist.github.com/emirhartato/5360318
https://gist.github.com/emirhartato/5360305

Build 0.69 that I got from http://tile.openstreetmap.org/osm2pgsql.zip doesn't shows any error. But the polygon sometimes doesn't work and polyline for road sometimes get messy. Here's the screenshot:

I'm running on MacOSX with 0.70, seems to work very well. But this build is not available for Windows.

from osm2pgsql.

Version 0.69 is pretty old. From what I can gather it is from sometime in 2010. That is presumably still pre 64 bit ID, so it won't work with the current planet at all, which is why it desperately needs replacing. People have compiled much newer working versions of osm2pgsql on windows though. They were just never put on tile.osm.org.

The errors shown in your logs look like configuration errors with the database parameters not correctly set. It might be possible that the defaults have changed in newer versions which is why you might need to set db parameters which you didn't use to need to do.

from osm2pgsql.

@apmon can you explain how to set database parameters? Hopefully if that can help solve the problem. There were 6 laptops tested and all of them have same error log. I also did a fresh install of PostgreSQL and PostGIS and create a new database, but still have the same error log.

from osm2pgsql.

Not sure if it helps, but here are my notes on the postgres configuration:
Tune Parameters for postgresql.conf

• autovacuum = off
• checkpoint_segments = 20
• shared_buffers = 512MB # min 128kB
• work_mem = 256MB # min 64kB
• maintenance_work_mem = 512MB # min 1MB
• synchronous_commit = off

Import example:

osm2pgsql.exe -v -c -d osm -U osm -W -H localhost -P 5432 -S default.style
-s -C 1600 --hstore --number-processes=4 -r pbf
....\data\us-midwest-northeast-pacific-south-west.osm.pbf

I also have a recent build of osm2psql which works. I have some mapnik
debugging to get through on mingw64, but I'll make it all available to our
build server here:

https://vanguard.houghtonassociates.com

The next build will have a package you can download under the Artifacts tab:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-27/artifact

Is there a specific time-frame you need this?

On Thu, Apr 11, 2013 at 2:18 AM, emirhartato [email protected]:

@apmon https://github.com/apmon can you explain how to set database
parameters? Hopefully if that can help solve the problem. There were 6
laptops tested and all of them have same error log. I also did a fresh
install of PostgreSQL and PostGIS and create a new database, but still have
the same error log.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16218534
.

from osm2pgsql.

Hello...

Well, there will be a workshop in 6th May. My team will teach osm2pgsql and
we will go through OSM Tool GUI in QGIS rather than running it through
command prompt since our target participants are not an IT expert or geek.

Our team also already have Advanced OpenStreetMap documentation written by
Jeff Haack and it's available in English and Bahasa Indonesia and there's a
chapter that provide step-by-step tutorial about setting up PostGIS and
osm2pgsql. We will published it soon on learosm.org site but we can't
published it until this issue is solved.

But if that's the only way to work, it's still possible to change the
tutorial on documentation but it would be nuce if we have a working
osm2pgsql with OSM Tool GUI in QGIS for Windows.
On Apr 11, 2013 10:55 PM, "Jason Huntley" [email protected] wrote:

Not sure if it helps, but here are my notes on the postgres configuration:
Tune Parameters for postgresql.conf

• autovacuum = off
• checkpoint_segments = 20
• shared_buffers = 512MB # min 128kB
• work_mem = 256MB # min 64kB
• maintenance_work_mem = 512MB # min 1MB
• synchronous_commit = off

Import example:

osm2pgsql.exe -v -c -d osm -U osm -W -H localhost -P 5432 -S default.style
-s -C 1600 --hstore --number-processes=4 -r pbf
....\data\us-midwest-northeast-pacific-south-west.osm.pbf

I also have a recent build of osm2psql which works. I have some mapnik
debugging to get through on mingw64, but I'll make it all available to our
build server here:

https://vanguard.houghtonassociates.com

The next build will have a package you can download under the Artifacts
tab:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-27/artifact

Is there a specific time-frame you need this?

On Thu, Apr 11, 2013 at 2:18 AM, emirhartato [email protected]:

@apmon https://github.com/apmon can you explain how to set database
parameters? Hopefully if that can help solve the problem. There were 6
laptops tested and all of them have same error log. I also did a fresh
install of PostgreSQL and PostGIS and create a new database, but still
have
the same error log.

—
Reply to this email directly or view it on GitHub<
https://github.com/openstreetmap/osm2pgsql/issues/17#issuecomment-16218534>

.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16243529
.

from osm2pgsql.

My guess would be the important parameters to resolve your issues are the following:

"-U osm -W -H localhost -P 5432"

-W forces a password promt where you can specify the password for the user osm. However, you have to make sure that postgresql is set up for password authentication and that you have set the appropriate password for the osm user in postgresql.

QGis should not be affected by any of this, as it simply uses an existing postgis db, no matter how you created it.

I have not heard of OSM Tool GUI before, so I don't know what it does. (And google doesn't seem to give me anything useful either) But assuming it is a wrapper around osm2pgsql to make sure you don't have to work with the command line, then you would simply have to extend OSM Tool GUI to set the appropriate parameters of osm2pgsql.

Oh, reading your message again, I see OSM Tool GUI is something / a plugin of QGis. But the same should still apply, that the integration module needs to be adapted to set the correct parameters when calling osm2pgsql. Just that it then does effect QGis... ;-)

You probably want to test it on the command line first. Make sure it works, and then contact the developers of the tool gui.

from osm2pgsql.

@apmon - Yes, I wrote "OSM Tool GUI" a while back: it is a python wrapper that works with the 0.69 (old) osm2pgsql (https://bitbucket.org/springmeyer/hot_tools/src/2321b0d0adae/osm_tools/command_dock2.py). Getting the password prompt working was extremely hard. Also when I've tried to compile osm2pgsql the getopt code has been the most difficult to port. So I presume that something changed in the behavior of the password prompt at some point, breaking the plugin.

@emirhartato - can you focus on testing osm2pgsql from the command line ONLY. Do not test with the QGIS plugin. We want to first make sure that an updated osm2pgsql build for windows works from the command line. After that I can take a look at any updates needed for the plugin to adapt it.

@emirhartato - my sense is you need to provide the -W flag when testing to that a password prompt comes up.

from osm2pgsql.

Well, that makes the task of fixing the "OSM Tool GUI" somewhat easier, if we know who to contact and know they are still interested in fixining things... ;-)

If password prompt is causing issues, it shouldn't be too difficult to try and add a different way to supply the password to osm2pgsql. One option would be to just add it as another parameter. However, the disadvantage is that the command line arguments are listed in the process outputs like top (not sure if that is the case in windows), so the password would then be publicly visible to everyone on the computer.

But yes, we first need to know that osm2pgsql itself works before fixing the wrapper around it.

@onepremise Nice to see those efforts. Having any form of working osm2pgsql on windows is obviously a good start, but the more "native" the compile is, the better. If I understand it correctly mingw is more "native" than cygwin, so that would be preferable. If we can get it to build from a clean source in git and can then add a windows build server to the ci.osm.org, then perhaps we can prevent it from constantly breaking and make windows a properly supported platform of osm2pgsql.

from osm2pgsql.

+1 to an insecure and easy way to pass a password for windows users. I configure my linux/osx postgres installs to not need a password, but the postgres windows installers require a password based on how the installers work. So a backdoor here for development would be really handy.

from osm2pgsql.

Nice to see those efforts. Having any form of working osm2pgsql on windows is obviously a good start, but the more "native" the compile is, the better. If I understand it correctly mingw is more "native" than cygwin, so that would be preferable.

Correct, mingw64 is definitely preferred, and I will providing the mingw64 build as soon as i tie up some lose ends with a Mapnik issue I'm working on in mingw 64 bit.

I'm very close to providing a stable build with MinGW-AD64S:

https://github.com/onepremise/MinGW-AD64S

Until then, I have the cygwin version, with 64bit IDs enabled, available here:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-35/artifact

Another build will follow up with the latest osm2pgsql updates.

from osm2pgsql.

Here's a build with the latest changes rebased:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

from osm2pgsql.

Here are the steps I followed to get @onepremise's build to work for me on Win7 https://gist.github.com/jlar/5375480

from osm2pgsql.

Sweet, thanks @jlar! @emirhartato - could you try following those instructions ^^ and let us know how things work for you?

from osm2pgsql.

Sweet! Will test it out tonight.
On Apr 13, 2013 5:32 AM, "Dane Springmeyer" [email protected]
wrote:

Sweet, thanks @jlar https://github.com/jlar! @emirhartatohttps://github.com/emirhartato- could you try following those instructions ^^ and let us know how things
work for you?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16321077
.

from osm2pgsql.

Ok, so I tested @onepremise build. It's still get me this error:

osm2pgsql SVN version 0.81.0 (64bit id space)
Password:
Error: Connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

from osm2pgsql.

@emirhartato - that error indicates usually means that postgres is not running. Can you confirm that you have postgres installed and it is running? Make sure to you can connect to postgres/postgis via pgadminIII and/or QGIS.

from osm2pgsql.

@emirhartato can you include your osm2pgsql command-code? Perhaps try specifying -H localhost (if applicable)?

from osm2pgsql.

Hello,

Thank you guys for your valuable efforts.
But it seems that the build from @onepremise has some problems.
More exactly, on italy-latest.osm.pbf I caught an unhandled exception which was somewhere in the processing of relations (17410).

I have managed to find an id of OSM object. This is a relation
http://www.openstreetmap.org/browse/relation/957103

Moreover, I would like to mention that @onepremise's Cygwin build works much slower then the one provided by Dominik Perpeet.
Here is my benchmarks for cygwin and Dominik's (http://customdebug.com/osm/osm2pgsql.zip) builds for italy-latest.osm.* files.

--------------------------- Cygwin ---------------------------
.bz2
Node 94.1k/s Way 3.97k/s Relation 29.66/s

.pbf
Node 205.8k/s Way 5.01k/s Relation 10.82/s

--------------------------- Dominik ---------------------------
.bz2
Node 136.4k/s Way 12.03k/s Relation 133.92/s

.pbf
Node 213.2k/s Way 14.91k/s Relation 130.9/s

from osm2pgsql.

@Rungee - thanks for the feedback. I think at this point we are aiming at something that just works, performance concerns, while important, can come later. I wonder if the unhandled exception can be explained by the different binary or by a difference in the osm2pgsql version being packaged? Do you have the ability to contact Dominik to see what git revision his build is from?

/cc @apmon - what do you think about pushing the output of git describe into the reported osm2pgsql version? Note for git describe to work an annotated tag is needed git tag -a. It looks like your v0.82.0 tag is annotated nicely, but wanted to confirm this.

from osm2pgsql.

@jlar @onepremise
actually it works, but for some place for some reason, polygons didn't show up and polyline looks really weird.

Error message for version 0.8:

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P 5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm
osm2pgsql SVN version 0.80.0 (32bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Out of memory for node cache dense index, try using "--cache-strategy sparse" in
stead
Error occurred, cleaning up

Error message for 0.81 from https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P 5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm
osm2pgsql SVN version 0.81.0 (64bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for sparse node cache
Out of memory for sparse node cache, reduce --cache size
Error occurred, cleaning up

from osm2pgsql.

err? that's strange. You're missing some tables in your database. Did you
deploy pgSnapshot Schema for OSM? Did it complete successfully? Also, might
want to reduce your memory usage or at least provide your mem usage via
comandline, -C 1600 is what i used. Hope that helps!

On Thu, Apr 18, 2013 at 5:19 AM, emirhartato [email protected]:

@jlar https://github.com/jlar @onepremisehttps://github.com/onepremise

actually it works, but for some place for some reason, polygons didn't
show up and polyline looks really weird.

[image: F5EF59E9-94CA-4C6D-9A25-E7747B2F48FB]https://f.cloud.github.com/assets/1563226/395674/93f6c656-a808-11e2-9633-688fd907b163.png

Error message for version 0.8:

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P
5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm
osm2pgsql SVN version 0.80.0 (32bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this
warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Out of memory for node cache dense index, try using "--cache-strategy
sparse" in
stead
Error occurred, cleaning up

Error message for 0.81 from
https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P
5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm

osm2pgsql SVN version 0.81.0 (64bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this
warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for sparse node cache
Out of memory for sparse node cache, reduce --cache size
Error occurred, cleaning up

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16566100
.

from osm2pgsql.

@onepremise. Notice that his visual screenshot is for really old osm2pgsql while the recent build is running out of memory and never finishing. @apmon can you advise if this seems odd? @emirhartato can you tell us how much memory your machine has and how large this specific osm extract is?

from osm2pgsql.

Osm2pgsql uses virtual memory fairly wasteful and kind of assumes that you have 64 bit address space and an efficient memory management. So there are a number of places, where it just allocates something like 2GB of memory with a good chance of never writing to that memory and thus never needing that amount of physical memory. However, depending on your over commit settings of your OS and how well your OS handles those situations, this might not work well.

As the error message sais, you should try the parameter "--cache-strategy sparse" which is less wastefull of virtual memory and thus might work (It roughly limits itself to what is specified in the -C parameter).

from osm2pgsql.

okay, update..
so it's work pretty well https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

I'm adding -H localhost and -W on my command line to get it work

Thanks everyone

from osm2pgsql.

@emirhartato - thanks for the update. What about the -C flag and/or the --cache-strategy sparse flag? It seems like you will need to specify those to avoid over memory usage.

@emirhartato - also, what are the next steps for you? are you good to go for your workshop, or do you need the QGIS plugin updated to also pass custom flags?

from osm2pgsql.

@springmeyer well, I have 4 GB RAM and all participants for the workshop should have 4 GB RAM minimum on their system (we put system pre-requisite on the invitation) and we won't import huge osm file so I think it won't be a problem. But I will put on a note on our tutorial that if it doesn't work because of memory issue.

next step... I think it's easy enough, I assume our participants are advance enough. they had beginner OSM-QGIS-InaSAFE and OSM-QGIS-InaSAFE intermediate before so it shouldn't be a problem ;)

from osm2pgsql.

Just FYI, there's also been new builds since then, we are up to 43:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-43

Navigate up a link and you can see all the builds :)

On Fri, May 3, 2013 at 9:34 PM, emirhartato [email protected]:

okay, update..
so it's work pretty well
https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

I'm adding -W on my command line to get it work

Thanks everyone

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-17425911
.

from osm2pgsql.

okay, thanks for the details guys. Hoping your workshop goes well @emirhartato. thanks @onepremise for making this possible!

from osm2pgsql.

Hey @springmeyer and @onepremise , this is great. I was in the same boat, I am used to linux and OS X and I was trying to support users on windows and the build that is easily available by looking around is the old build 0.69 and we had all sort of issues. Just found this build and tried it (this version: https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-60/artifact) Here is our experience:

• trying default (copy paste from the QGIS gui) we got the error:

  Error: Connection to database failed: could not connect to server: No such file or directory
  Is the server running locally and accepting
  connections on Unix domain socket "/tmp/.s.PGSQL.5432"

• then we added -H localhost and we got:

Using built-in tag processing pipeline
Allocating memory for sparse node cache
Out of memory for sparse node cache, reduce --cache size
Error occurred, cleaning up

• so we tried --cache-strategy sparse but we got the following error:

Using built-in tag processing pipeline
Allocating memory for sparse node cache
Out of memory for sparse node cache, reduce --cache size
Error occurred, cleaning up

- finally it all worked with -H localhost --cache 400
A bit slow but the area was small and it worked well.

I had checked a couple month ago and I don't think I had seen this option on the wiki but now it is documented there which is great but I think that maybe this build should be more recommended and documented.

Thanks for the work !

from osm2pgsql.

Hi All,

I am need for 64bit windows executable for osm2psql... Any idea if it already available?

from osm2pgsql.

Hi All,

I've been getting emails on this subject so I figured I would share. I now have a working 'true' 64bit build of osm2psql for windows. I'm in the process of checking in the remainder source and making it available on our build server here:

https://vanguard.houghtonassociates.com/browse/OSM

It should be out there by the end of next week. Yes, PBF is supported :)

from osm2pgsql.

Hi All,

I've been getting emails on this subject so I figured I would share. I now
have a working 'true' 64bit build of osm2psql for windows. I'm in the
process of checking in the remainder source and making it available on our
build server here:

https://vanguard.houghtonassociates.com/browse/OSM

It should be out there by the end of next week. Yes, PBF is supported :)

Thanks,

Jason

On Wed, Sep 25, 2013 at 1:51 PM, halo123 [email protected] wrote:

Hi All,

I am need for 64bit windows executable for osm2psql... Any idea if it
already available?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-25109382
.

from osm2pgsql.

I have made some changes to compile the thing under Visual Studio 2013 + cmake (did not managed to find compatible protobuf-c for PBF support, however).
Maybe it can be useful: https://github.com/alex85k/osm2pgsql
test build: https://dl.dropboxusercontent.com/u/63393258/osm2pgsql_win_unstable.zip

Is there some "official" working windows build now?

from osm2pgsql.

Thanks @alex85k for providing Visual Studio build.
Tested with 64 bit ProgreSQL 9.3 and PostGIS 2.1 successfully.

Please keep us informed about your builds, or if you could manage upload section, even better as we could use it for reference.

Cheers

from osm2pgsql.

We tried also Cygwin version from one of the posts above linked in OSM wiki, and ArcGIS OSM plugin, and let me say this:

Cygwin version simply doesn't work here, unlike Alex's build with same command line:

C:\>osm2pgsql.exe my.osm -d osm -U postgres -P 5432 -W xxx --hstore

      0 [main] osm2pgsql 9480 find_fast_cwd: WARNING: Couldn't compute FAST_CWD pointer.  Please report this problem to
the public mailing list cygwin@cygwin.com
osm2pgsql SVN version 0.81.0 (64bit id space)

Error: Connection to database failed: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

OTOH, postgis data already generated by Alex's build show some issues in both QGIS and ArcMap, like missing lines, polygons, etc...

But ArcGIS plugin provides perfect results!
I assume it's not trivial, but can't you guys sneak into how they've made it (assuming *nix build behaves the same)?

from osm2pgsql.

Did you try the most recent cygwin build?

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-89

We will also have a new Mingw64 bit build out this week.

On Tue, Jun 17, 2014 at 9:13 PM, klonuo [email protected] wrote:

I tried also cygwin version and ArcGIS OSM plugin, and let me say this:

Cygwin version simply doesn't work here, unlike alex's build with same
command line:

C:>osm2pgsql.exe my.osm -d osm -U postgres -P 5432 -W xxx --hstore

  0 [main] osm2pgsql 9480 find_fast_cwd: WARNING: Couldn't compute FAST_CWD pointer.  Please report this problem to

the public mailing list [email protected]

osm2pgsql SVN version 0.81.0 (64
bit id space)

Error: Connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

OTOH, postgis data already generated by alex's build show some issues in
both QGIS and ArcMap, like missing lines, polygons, etc...

But ArcGIS plugin https://github.com/Esri/arcgis-osm-editor provides
perfect results!
I know it's not trivial, but can't you guys sneak into how they've made it
(assuming *nix build behaves the same)?

—
Reply to this email directly or view it on GitHub
#17 (comment)
.

from osm2pgsql.

I tried it now. I managed to make it transfer data to postgis by explicitly setting host -H (although it's localhost) and then setting PGPASS variable as if I omit -W from command line - program exits, but if I set -W then the program fails to finish data transfer as it complains it can't make table xxx (which is my password). So I set PGPASS variable with my password and completed transfer successfully.

Database is exactly the same as with Alex's build. It's my fault I complained about missing features, but the problem is this - ArcGIS plugin provides GDB layer with separate polygons, while if I load data from created postgis database I don't get separate polygons, but all are merged is large polygon blocks:

link: http://i.imgur.com/4NgP8kB.png

I don't know why is this.

from osm2pgsql.

OTOH, postgis data already generated by Alex's build show some issues in both QGIS and ArcMap, like missing lines, polygons, etc...

I figured out why this happened. Data is missing when I use this utility on multiple files, which I assumed is valid as in this quote (PostGIS in Action):

Appending versus overwriting with osm2pgsql
By default, OSM will overwrite the tables and create them fresh. If you’re appending
multiple OSM files at different points in time, you’ll want to use the --append switch
to switch to append mode. Note that you can process multiple files at once by sepa-
rating the filenames with spaces, for example, file1.osm file2.osm ....

Plus program doesn't complain when I feed it with multiple files and screen log shows like everything is fine.

If I use this tool with --append and transfer files one by one or if I use it just on single osm for testing, then I get all data transferred, but again it doesn't look right as polygons are merged in large groups unlike example from ArcGIS osm plugin.

from osm2pgsql.

I can confirm that the Cygwin build runs fine under Windows. I have just one concern: Did anyone test the parameter "--number-processes" with a value greater than 1?
In my environment this is not working due to error message about fork.

from osm2pgsql.

Yes, both versions run fine. Visual Studio build is several times faster, while it does not support binary input.

Let me just finish my crash lending here by saying that ArcGIS plugin behaves as described because ESRI does not support geometry collections. That's where my noise come from...

from osm2pgsql.

Some statistics for area "Bremen" (http://download.geofabrik.de/europe/germany/bremen.html):

So, the msvc build is the fastest.

@alex85k:
How did you generate the config.h on windows (cmake repo)?

from osm2pgsql.

How is pbf? both bz2 and raw OSM XML are much slower than pbf, and that's where the speed matters

from osm2pgsql.

Oh, it iseems I forgot to add config.h.in when committing. Sorry, will try to find it and update my repository. By the way, cmake system is ''much'' simpler than autotools variant.

PBF support needs protobuf-c, that can be built with configure (MSYS or cygwin) but hard to build with Visual Studio (may need a lot of patching and build system replacement). I'll look if needed.

from osm2pgsql.

Here is the updated build: https://dl.dropboxusercontent.com/u/63393258/osm2pgsql.zip
(for latest version, not tested!)

The sources are updated: https://github.com/alex85k/osm2pgsql/tree/msvc-july2014
config.h.in is there too :)

from osm2pgsql.

By the way, cmake system is ''much'' simpler than autotools variant

All build systems suck. Autotool's big advantage is that it's so common, and that someone has encountered most problems you're likely to find.

from osm2pgsql.

I have tried to add PBF support alex85k@a4e3da8,

Binary is created with slightly modified protobuf-c and geos-3.4.2 to allow VS2013 Express compilation, no compiler flags changing was done, so the build builds on Unix system with cmake may be buggy.

Here is the binary build:
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql.zip
Can someone check if it is really working?

from osm2pgsql.

@alex85k
Still missing a "config.h.in" in a branch (july or march...) ;-)

from osm2pgsql.

/facepalm/ Yes, I forgot it again. It is in ignored files list :)
Re-pushed again. (git push --force)

from osm2pgsql.

Here is the binary build:
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql.zip
Can someone check if it is really working?

@alex85k I tried it. In comparison to previous version this one has ~10 new libs build with mingw.

C:\PostgreSQL\9.3\bin\osm2pgsql>osm2pgsql.exe test.osm.pbf -d osm_test -U postgres -P 5432
osm2pgsql SVN version 0.70-win-cmake (64bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE:  table "planet_osm_point" does not exist, skipping
NOTICE:  table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE:  table "planet_osm_line" does not exist, skipping
NOTICE:  table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE:  table "planet_osm_polygon" does not exist, skipping
NOTICE:  table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE:  table "planet_osm_roads" does not exist, skipping
NOTICE:  table "planet_osm_roads_tmp" does not exist, skipping
Using built-in tag processing pipeline
Allocating memory for sparse node cache
Node-cache: cache=800MB, maxblocks=102400*zd, allocation method=8192
Mid: Ram, scale=100
ERROR: PBF support has not been compiled into this version of osm2pgsql, please either compile it with pbf support or use one of the other input formats

Do I miss something?

from osm2pgsql.

I have missed something :) The folowing line, actually: ADD_DEFINITIONS(-DBUILD_READER_PBF)
Now it should work. (repushed again)
https://github.com/alex85k/osm2pgsql/tree/msvc-july2014
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql.zip
(also removed unneeded libraries)

Considering automatic builds - we have created this system for OSRM recenty using AppVeyor.com :
Project-OSRM/osrm-backend#998
if the code will be ready, adding auto-builds will be simple (for now, at least protobuf-c patches are "unofficial")

from osm2pgsql.

@alex85k Unfortunately still no go - it crashes while reading the file.

This is the output for the exception:

The thread 0x176c has exited with code 0 (0x0).
Unhandled exception at 0x00007FF9C75F7D0B (ntdll.dll) in osm2pgsql.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.

from osm2pgsql.

Crashed too on my pc, but worked OK when I added style option. Will build a debug version tomorrow to see the location of the problem.

from osm2pgsql.

Published the compatible cmake-based code of protobuf-c:
protobuf-c/protobuf-c#155

from osm2pgsql.

The debug binary build is ready, all sources published.
https://dl.dropboxusercontent.com/u/63393258/osm2pgsqldebug.zip

However, it gives the errors related to PROJ library, can debug but do not know how to fix them...

from osm2pgsql.

Tried it.
Don't know from your last line if crash is to be expected, so in case if not here's what I got:

Debug Assertion Failed!

Program: C:\PostgreSQL\9.3\bin\osm2pgsqldebug\osm2pgsql.exe
File: f:\dd\vctools\crt\crtw32\misc\dbgheap.c
Line: 1322

Expression: _CrtIsValidHeapPointer(pUserData)

For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.

(Press Retry to debug the application)

from osm2pgsql.

This is some memory error at the end of first pj_* call (proj4 library), the reason may be anywhere before it...

from osm2pgsql.

Line 70 of https://github.com/alex85k/osm2pgsql/blob/msvc-july2014/reprojection.c is failing. Maybe some data or confuguration is needed.

from osm2pgsql.

@alex85k maybe there is a simple type-o. Line 70 states "+proj=longlat...", but shouldn't it be "+proj=latlong...."

from osm2pgsql.

@alex85k maybe there is a simple type-o. Line 70 states "+proj=longlat...", but shouldn't it be "+proj=latlong...."

Just tried it. The same runtime error, unfortunately...

from osm2pgsql.

I had some time to setup a msvc build, so my experience is as follows.

@alex85k
Did you already integrate this protobuf-c patch? #129
I did but have nevertheless a heap error after freeing some characters while processing processOsmDataDenseNodes (see https://github.com/openstreetmap/osm2pgsql/blob/master/text-tree.c#L82)

I have also an error reading in a larger file (germany-latest.osm.bz2). osm2pgsql crashes after processing nodes, ways and relations. For me, it looks like a memory leak and is not related to proj4...

from osm2pgsql.

Did you already integrate this protobuf-c patch? #129

I did not, but I had copied my protobuf-c.h to include/protobuf-c folder and also renamed lib_protoc.lib to protoc.lib before calling build.bat. Protobuf from was used from
https://gist.github.com/alex85k/8637217#file-build_other-bat
protobuf-c from
https://github.com/alex85k/protobuf-c
wingetopt from
https://github.com/alex85k/wingetopt

The source of my heap corruption errors is Debug+Release runtime used in the same program (some static libraries or different build flags). Trying to fix the build procedure now.
http://stackoverflow.com/questions/8740500/heap-corruption-with-strdup
https://connect.microsoft.com/VisualStudio/feedback/details/333868/programs-using-a-static-library-and-calling-strdup-crash
the corruption at https://github.com/openstreetmap/osm2pgsql/blob/master/text-tree.c#L82 seems to have the same reason (but with different libraries statically linked to runtime)

from osm2pgsql.

My bug was fixed by replacing

OPTFLAGS=   /nologo /Zi /MD /Fdproj.pdb

with

OPTFLAGS=   /nologo /Zi /MDd /Fdproj.pdb

in nmake.opt of Proj4 sources. But currently I can not reproduce the procedure second time...

from osm2pgsql.

To fix the strdup-related probems you need to build 4 libraries not only in debug mode, but with debugging MSVCRTD runtime (NODEFAULTLIB warning should vanish).
I had to add /MDd (or change from MD) flags inside configs and makefiles of bzip2, zlib, libxml2 and proj4. cmake-based projects seem to do this automatically.

from osm2pgsql.

Here are the basic libraries sources with fixed configs and batch file:
https://dl.dropboxusercontent.com/u/63393258/debug_basic_libs_src.7z

The "secondary" problem is solved. Now the real PBF reading error is to be fixed (osm.bz2 works but the same fragmet pbf file breaks the converter)

from osm2pgsql.

I usually take a 3rdParty package which includes these most used libraries (zlib, libxml2 etc.), so I don't have control over the linker flags. But all projects that are under my control have correct linker flags (/MDd). I checked your 3rdParties: I haven't linked them statically in my msvc build, so I'm obviously not affected by the linker problems.

I didn't get if you already have patched your protobuf-c? Take a look at this diff from Edwards in the last days:

diff --git a/protobuf-c/protobuf-c.c b/protobuf-c/protobuf-c.c
index c7fb21d..b66ed93 100644
--- a/protobuf-c/protobuf-c.c
+++ b/protobuf-c/protobuf-c.c
@@ -2304,6 +2304,8 @@ parse_required_member(ScannedMember *scanned_member,
            if (bd->data == NULL)
                return FALSE;
            memcpy(bd->data, data + pref_len, len - pref_len);
+       } else {
+           bd->data = NULL;
        }
        bd->len = len - pref_len;
        return TRUE;

from osm2pgsql.

@martin-s Yes, that was it! The bug disappeared after patching protobuf-c. Thank you very much!
I'll update my protobuf-c fork now and publish debug&release builds.

from osm2pgsql.

Very good. Don't forget to align version number with the repository (should be 0.85 :-)

from osm2pgsql.

OK, here is the updated sources:
https://github.com/alex85k/osm2pgsql/tree/msvc-august2014
Windows-compatible protobuf-c:
https://github.com/alex85k/protobuf-c

And, finally, MSVC 2013 debug build with working PBF support for testing:
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql_debug.7z

I'll make the release version in few days (need to recompile all libs) and publish the full build scripts (maybe automating the builds with AppVeyor.com).

from osm2pgsql.

So, here is the release build:
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql.zip

And this is the package that I used to build all related libraries (with some unneeded ones) and osm2pgsql:
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql_builder.zip
(installed PostgreSql, git, svn and Visual Studio 2013 [express] are required, edit settings.bat and call build_all.bat).

from osm2pgsql.

I did not manage to run Python tests (hard to install necessary modules, ActivePython is now asking for money...)

from osm2pgsql.

@alex85k ActivePython should be free, but anyway that's just the same as standard Python distribution from python.org + pywin32 extensions. Maybe you want to use their package manager?
What additional modules are needed for running the tests? Is it only psycopg2?

from osm2pgsql.

I tried again with clean Python 2.7.8. The package psycopg2 was found on
http://www.stickpeople.com/projects/python/win-psycopg/
(pip install was not working, even after installing pip :) )
The line from pwd import getpwnam seems to be not needed but I created fake pwd.py anyway.

Then the tests start, but give many errors (it uses /tmp, sudo, and other Unix-specific things).

After some patching of regression-test.py, 15 of 40 tests pass :) I guess more patching (or PostgreSql tuning) is needed.
P.S. At least one call of osm2pgsql crashed during testing.

from osm2pgsql.

Example test log (commented other test suites):

Running diff-import for basic case
*********************************
.****************************************
Running initial import for Parallel processing
*********************************
Running diff-import for Parallel processing
*********************************
.****************************************
Running initial import for Parallel processing with non 100% node-cache
*********************************
Running diff-import for Parallel processing with non 100% node-cache
*********************************
.E
======================================================================
ERROR: runTest (__main__.BasicSlimTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/regression-test.py", line 432, in setUp
    self.setUpGeneric(self.parameters, full_import_file)
  File "tests/regression-test.py", line 384, in setUpGeneric
    self.assertEqual (proc.returncode, 0, "Execution of osm2pgsql --slim with options: '%s' failed:\n%s\n%s\n" %
(str(parameters), outp, outerr))
AssertionError: Execution of osm2pgsql --slim with options: '['-C0']' failed:
----------------------------------------------------------------------
Ran 8 tests in 110.687s

FAILED (errors=1)
Cleaning up test database

from osm2pgsql.

The line from pwd import getpwnam seems to be not needed but I created fake pwd.py anyway.

Looking at the test file, that line is unused import, which should be commented / deleted

Then the tests start, but give many errors (it uses /tmp, sudo, and other Unix-specific things).

Except calls to osm2pgsql there are two more process calls at lines 543 which can be commented/ignored, and 605 which can be replaced with os.rmdir()

You perhaps did also search/replace on "/tmp/psql-tablespace" to Windows sane name.

I can't see other problems assuming parameters passed to osm2pgsql are valid.

from osm2pgsql.

Thank you! I'll try once more on my home pc.
Last time I have just commented all sudo's and replace ./osm2pgsql with osm2pgsql, /tmp/psql-tablespace with psql-tablespace. Will do it more thoughtfully this time.

from osm2pgsql.

@alex85k I downloaded your latest binary and tried to run the tests with this regression-test.py.
This is my log: test_log.txt

So after 40 tests it reports 8 failures which doesn't seem like failures but more like rounding errors, and 6 errors where 4 are "failed: ERROR: tablespace "tablespacetest" does not exist" (I assume not related to your binary) and the rest are:

AssertionError: Execution of osm2pgsql with options: '['-l']' failed:
Projection code failed to initialise

and

AssertionError: Execution of osm2pgsql --slim with options: '['-C0']' failed:

just like in your report.

from osm2pgsql.

Thank you for regression-test.py!
My current modification of it: https://gist.github.com/alex85k/96d831ee3c80357c1f67 (will change)
(I got access denied when creating files in default temporary Windows folder)

After debugging Projection bug is caught: for googling incorrect variable value C:PROJSHARE gave the
following patch that solved this single problem: http://trac.osgeo.org/proj/ticket/227

from osm2pgsql.

Tablespace errors seem to be fixed by choosing write-accessible temp dir:
https://gist.github.com/alex85k/96d831ee3c80357c1f67#file-regression-test-py-L10

Now the first test suite is working without errors (with PROJ.4 in C:\PROJ) and second suite ghives only one runtime error to be fixed.
There are also 8 failures in suites 3 and 4 (total 9 errors, 8 of which can be rounding errors).

I am debugging the runtime error (called by breaking.bat in updated build suite https://dl.dropboxusercontent.com/u/63393258/osm2pgsql_builder.zip).

from osm2pgsql.

The test log: https://gist.github.com/alex85k/96d831ee3c80357c1f67#file-test-log
(python file is included to osm2pgsql_builder.zip). Fixing the runtime error (memory corruption detected on free(sparese_block)).

Are you sure the remaining failures are rounding errors?

from osm2pgsql.

One more memory error is fixed (introduced with Windows patching):
alex85k@6dbbadd

from osm2pgsql.

@alex85k I redo the test with you suggested edit - instead temp folder use current one and then I added some settings to make cygwin version work with the test suite:

--- a
+++ b
@@ -6,10 +6,11 @@
 # from pwd import getpwnam
 import subprocess

-tmp_table = os.path.join(os.environ['temp'], 'psql-tablespace')
+tmp_table = os.path.join(os.getcwd(), 'psql-tablespace')

 usr, pwd = 'postgres', 'MY_PASSWORD'
-credentials = ['--username', usr]
+os.environ['PGPASS'] = pwd
+credentials = ['--username', usr, '--host', 'localhost']
 conn_str = "user={} password={} ".format(usr, pwd)

 full_import_file="liechtenstein-2013-08-03.osm.pbf"

I did not understood what to do about proj.4, so now I get these results:

1. test_log1.txt (your binary)
2. test_log2.txt (cygwin binary)

Cygwin version fails on same tests and then some more, and it took 3x longer time to finish.

About the failures of type ... expected 24751 but was 24750.0 .... This may be problem in the test itself, as expected value is integer while compared value is float. It looks like rounding error but perhaps test author can comment why it fails here. This doesn't look like it can be Windows specific.

from osm2pgsql.

Thank you for clarification!
I have included your file to the repository fork: https://github.com/alex85k/osm2pgsql/commits/msvc-august2014

Projection code failed to initialize can be solved by unpacking https://dl.dropboxusercontent.com/u/63393258/PROJ.zip to drive C: .
The test log now contains only rounding errors: https://gist.github.com/alex85k/96d831ee3c80357c1f67#file-test-log

I have updated the release version: https://dl.dropboxusercontent.com/u/63393258/osm2pgsql.zip
and debug version https://dl.dropboxusercontent.com/u/63393258/osm2pgsql_debug.7z

from osm2pgsql.

Several times I had strange "can not make directory" testing errors:
https://gist.github.com/alex85k/96d831ee3c80357c1f67#file-test_debug-log
(thought it was bad files from the old terminated runs but no...).

P.S. Sorry, it was my dumb mistake in .py file. Reason still unknown - both debug and release versions.

from osm2pgsql.

Release version bundled with tests: https://dl.dropboxusercontent.com/u/63393258/osm2pgsql_testRelease.zip

from osm2pgsql.

The problem of tablespace tests was solved by manually deleting tablespace from pgadmin (I built debug and release in different directories and often stoppen the testing without cleanup)

from osm2pgsql.

Projection code failed to initialize can be solved by unpacking https://dl.dropboxusercontent.com/u/63393258/PROJ.zip to drive C: .
The test log now contains only rounding errors: https://gist.github.com/alex85k/96d831ee3c80357c1f67#file-test-log

Thanks Alex, after unpacking Proj and rerunning tests on your updated release version, only rounding failures remain, which is great result IMHO: test_log3.txt
Congratulations!

Several times I had strange "can not make directory" testing errors:
https://gist.github.com/alex85k/96d831ee3c80357c1f67#file-test_debug-log
(thought it was bad files from the old terminated runs but no...).

I didn't see anything like it, and have no idea what could be the problem.

from osm2pgsql.

I did my compulsory test with OSM file of Germany (http://download.geofabrik.de/europe/germany-latest.osm.pbf) and failed. So, there's still work to do, to get it working.

osm2pgsql.exe -v -c -d osm -U osm -W -H localhost -P 5432 -S default.style
-s --hstore --number-processes=1 germany-latest.osm.pbf

I succeed in importing the file, when I use MinGW build.

from osm2pgsql.

@martin-s could you please post the command line?
P.S. Thank you) May be memory leak, testing...

from osm2pgsql.

No memory growing detected (100mb pbf)... May be 2billion int problem in filesize?... Do you remember on which phase it crashed? (I started processing the Germany but not sure if can keep it running for 6 hours)

from osm2pgsql.

It crashed right after the print "Processing: Node(xxx) Way(xxx) Relation(xxx)". Maybe here:
https://github.com/openstreetmap/osm2pgsql/blob/52b0468b6c73b68ebbaeaf91a9264367dbacd4b6/parse-pbf.c#L523

The computer on which I test has 24 GB RAM.

from osm2pgsql.

It seems I had the same. Processing: Node(xxx) Way(xxx) Relation(xxx) was working for first hours, then at some moment computer relaxed and went to standy ad did not wake up (common bug for my hardware on WIndows 8). If only we can have faster way to debug... Does not s
maller ~500 mb extracts crash it?
P.S. Trying 687Mb Russia to test...

from osm2pgsql.

By the way, are you running the last release version? Previous one contained at least two memory errors for sure.

from osm2pgsql.

Yes, I took the one (msvc build) which includes the python tests. Mh, some OSM size between 100 MB and the 2.1 GB from Germany. Hard to say which size is suitable to produce the error. I can try the Netherlands (929 MB).

from osm2pgsql.

I have a windows desktop I can test on, but don't like installing stuff (python, etc). It would work for testing large extracts, having 32GB of RAM, and the DB server being an independent machine with 32GB of RAM itself.

from osm2pgsql.

I tried Russia (687 Mb) - no errors, this was not enough... (stopped manually after committing tansaction, because my PC is does not allow me to sleep in the room).

@pnorman: Python is not needed to test large extracts - just the binary build (and, maybe, Visual Studio Express to debug).

from osm2pgsql.

I tried Netherlands (929 MB) - no errors. Next would be the Alps (1.5 GB) or Germany (2.1 GB).

from osm2pgsql.