Help w Spark? about barefoot HOT 6 CLOSED

For Spark, we upload the bfmap file to HDFS. This allows the Spark executors read the map from HDFS, which should be better than having them all connecting to the PostgreSQL database at the same time.

To create a bfmap file from the PostgreSQL database, use this code: https://gist.github.com/jongiddy/67c7ace4e7394e1e5f3bea978ddf74ec (this is set to run inside a Vagrant virtual machine, but changing the hardwired /vagrant paths will make it suitable for other environments).

To read the bfmap file from HDFS, we created a HadoopMapReader: https://gist.github.com/jongiddy/b68be517274a424df84d2bea4cdd6354

And our BroadcastMatcher then looks like this (although I have edited out some application-specific code): https://gist.github.com/jongiddy/286857e09f9881854a725634ca82b515

from barefoot.

@smattheis No worries! To be clear, we never saw a problem caused by the locking. I think I added that while debugging a thread-safety issue that actually occurred in a part of my code.

from barefoot.

@jongiddy @smattheis thanks guys this is awesome

from barefoot.

@jongiddy Thanks, for sharing that! BTW: ... double-checked locking, you could have let me know. ;)

from barefoot.

@jim2 I'm glad it helps. If you any more question, feel free to keep this issue open, or open another one. Once the question/problem regarding spark is resolved for you, please close the issue.

@jongiddy Alright. Anyways, your reference in the comment explains that it's a code smell. I never knew that.

from barefoot.

I'm closing this as it seemed to be resolved.

from barefoot.