tc / elastic-mapreduce-ruby Goto Github PK

The latest version of the emr does not work with the newer versions of ruby
I had to change the colons with when and got it to work to an extent but when I'm trying to create a job flow it is giving me further errors that I am unable to figure out

TRana:elastic-mapreduce-ruby trana$ ./elastic-mapreduce --create --alive --log-uri s3n://my-example-bucket/logs Created job flow j-36U2JMAE73054

Error: undefined method to_str' for #<RuntimeError:0x007fed8c280a88> /Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/amazon/retry_delegator.rb:51:inrescue in method_missing'
/Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/amazon/retry_delegator.rb:42:in method_missing' /Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/client.rb:88:inrun_jobflow'
/Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/commands.rb:1324:in enact' /Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/commands.rb:47:inblock in enact'
/Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/commands.rb:47:in each' /Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/commands.rb:47:inenact'
/Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/commands.rb:2428:in create_and_execute_commands' /Users/trana/elastic-mapreduce-cli/elastic-mapreduce-ruby/elastic-mapreduce-cli.rb:13:in<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:45:in require' /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:45:inrequire'
./elastic-mapreduce:6:in `

When running "elastic-mapreduce --create ...", I get the above error.

Amazon now requires InstanceProfiles when creating new clusters. Any chance of supporting those?

The --tag option is not supported

As it stands you can choose between i386 and x86 m1.small and m1.medium machines. Could you add a parameter to this script to allow a user to select the ARCH upon the creation of a new cluster?

Or you could tell me how to do it, and I could supply a patch.

Thanks!

When trying to launch a cluster using the MapR distribution I'm getting an error that "--supported-product" is not a valid option.

./elastic-mapreduce --list
INFO Exception Retriable invalid response returned from DescribeJobFlows: {"Error"=>{"Details"=>#<NoMethodError: undefined method `random_create' for UUID:Class>, "Code"=>"InternalFailure", "Type"=>"Sender"}} while calling DescribeJobFlows on Amazon::Coral::ElasticMapReduceClient, retrying in 3.0 seconds.

Hi,

I am getting a the following error while trying to run the ./elastic-mapreduce --create

My knowledge on Ruby is low and I am not able to figure out as well . Please throw some light on the same.

The following error is getting thrown.
INFO Exception Retriable invalid response returned from RunJobFlow: {"Error"=>{"Type"=>"Sender", "Code"=>"InternalFailure", "Details"=>#<TypeError: String can't be coerced into Fixnum>}} while calling RunJobFlow on Amazon::Coral::ElasticMapReduceClient, retrying in 3.0 seconds.

It would be valuable to be able to specify and reverse proxy to be used for accessing emr.

I get an error when passing in multiple --cache arguments:

$ ./elastic-mapreduce 
  --create
  --cache <file1>
  --cache <file2>
  ...

Error: Repeating --cache CACHE_FILE is not allowed, previous value was "<file1>"

EMR definitely supports multiple cache files though, and line 781 of commands.rb seems to support it too:

      stream_options = []
      for ca in get_field(:cache, []) do
        stream_options << "-cacheFile" << ca
      end

Is this a bug, or is there another way in which I should pass in multiple cache files? (I already tried a single --cache <file1>,<file2>,<file3> but my job eventually failed saying that it didn't have a valid cache URL)

Hi folks,

When trying to run ./elastic-mapreduce under ruby 1.9, I get the following:

elastic-mapreduce-cli/amazon/coral/httpdestinationhandler.rb:23: warning: else without rescue is useless
<internal:lib/rubygems/custom_require>:29:in `require': elastic-mapreduce-cli/amazon/coral/httpdestinationhandler.rb:19: syntax error, unexpected ':', expectingkeyword_then or ',' or ';' or '\n' (SyntaxError)

Now, I know the specs say that ruby 1.8.7 should be used, but I believe that it is possible to make the script run on ruby 1.9 with minor modifications.

The reason for this error is that the ruby syntax changed from 1.8 to 1.9, see http://stackoverflow.com/questions/9305509/case-expression-different-in-ruby-1-9.

To make it work on 1.9 while keeping compatibility with 1.8, you need to convert all colons in when statements to the 'then' keyword, eg:

        @uri = case endpoint
               when URI then     # previously, ':' instead of 'then'
                   endpoint
               else
                 URI.parse(endpoint)
               end

If the when-statements are not on a single line (as in example above), you can in fact leave the 'then' out as well.

The places in the code to adjust like this are (line numbers approximate):

amazon/coral/httpdestinationhandler.rb:23
amazon/coral/querystringmap.rb:50

I scanned all case-statements in the rest of the code, these are all OK. So I think this was just an oversight.