I agree. I managed to get the application running via grails 1.3.7 on ubuntu 12.04 but I don't even want to post the steps because there were so many hacks. I'm sure there is a better way to just have grails create a deployable war that you can just drop into tomcat but I'm not familiar with grails deployments and configuration. I agree the installation/setup instructions in the README could use some help.

from ice.

I am new to Grails framework, Can you please post details installation steps in README file , I am using Ubuntu12.04LTS with grails-2.1.1.

Please let me know how to run this step
4.Get all libraries listed in ivy.xml

from ice.

i found these two commands very helpful grails war and grails run-app
Alternatively, importing the project into IntelliJ's IDE made it very easy to build/run the app.
Currently, the problem I have is that I don't know how to get the app to find my ice.properties file

from ice.

@ralph-tice Place the ice.properties file in grails-app/conf

from ice.

I will get the build cleaned up today.

Thanks
-Fang

from ice.

Also here is the list of jars I had to cobble together from various sources and place under lib/ to get the app to fire up.

ant-1.7.0.jar
guava-14.0.1.jar
httpcore-4.1.jar
jcommon-1.0.16.jar
joda-time-2.0.jar
org.json-chargebee-1.0.jar
wstx-asl-3.2.9.jar
aws-java-sdk-1.4.4.1.jar
httpclient-4.1.jar
javacsv-2.0.jar
jfreechart-1.0.13.jar
mail-1.4.1.jar
poi-3.7.jar

from ice.

I update the ice project to be built on grails 2.2.1. Now to start running ice, you can following the following steps (copied form readme):

4.1 Pull the project

4.2 Run grails wrapper from the project root directory. This step will pull all necessary jar from maven central.

4.3 Make sure ice.properties is added to directory src/java

4.4 From project root directory, run ./grailsw run-app. Note you may need to add system properties like ./grailsw -Dice.s3AccessKeyId= -Dice.s3SecretKey= run-app.

from ice.

hi fangi,
@fangji :
I attempted to make an installation of the application yesterday 19-JUN-13 but couldnt complete it...Today ie 20-JUN-13 I see that some steps are being removed like
1.No tomcat installation
2.Get all library files in ivy.xml
etc
wat does this exactly mean......It would be great if you could publish a more descriptive steps on installation of ICE either here or on any blog post.....

@rayrod2030 : u got the app running thats great......wish u change ur mind and post the steps :-)

from ice.

@cottonmouth00 Steps 4.1 through 4.4 cover bootstrapping the project in the readme as well as Fang's last comment.

@fangji Thank you for your help and for bumping to grails 2.2.1! I stepped through every step much more easily this time. I can't use grailsw because of these errors:

$ ./grailsw
Error opening zip file or JAR manifest missing : wrapper/springloaded-core-1.1.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument

However, when I invoke grails directly everything appears to work. When I try to hit the application at localhost:8080/ice however I get this stacktrace:

| Server running. Browse to http://localhost:8080/ice
2013-06-20 08:39:41,810 [http-bio-8080-exec-4] INFO  [localhost].[/ice]  - Initializing Spring FrameworkServlet 'gsp'
2013-06-20 08:39:41,811 [http-bio-8080-exec-4] INFO  [localhost].[/ice]  - GSP servlet initialized
| Error 2013-06-20 08:39:42,256 [http-bio-8080-exec-4] ERROR errors.GrailsExceptionResolver  - IOException occurred when processing request: [GET] /ice/
Stream closed. Stacktrace follows:
Message: Stream closed
   Line | Method
->> 187 | doCall  in com.netflix.ice.DashboardController$_closure8
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|    68 | doCall  in com.netflix.ice.DashboardController$_closure1
|   895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run     in     ''
^   680 | run . . in java.lang.Thread
| Error 2013-06-20 08:39:42,328 [http-bio-8080-exec-4] ERROR [/ice].[jsp]  - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,336 [http-bio-8080-exec-4] ERROR errors.GrailsExceptionResolver  - ServletException occurred when processing request: [GET] /ice/
File "/WEB-INF/grails-app/views/error.jsp" not found. Stacktrace follows:
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,367 [http-bio-8080-exec-4] ERROR [/ice].[jsp]  - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,379 [http-bio-8080-exec-4] ERROR [/ice].[grails]  - Servlet.service() for servlet grails threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,382 [http-bio-8080-exec-4] ERROR [/ice].[gsp]  - Servlet.service() for servlet [gsp] in context with path [/ice] threw exception [File "/WEB-INF/grails-app/views/error.jsp" not found] with root cause
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread

I tried to look in the commit history to see what changed but it looks like you might have rebased some commits because I only see one commit now in github.

from ice.

@cottonmouth00
As @ralph-tice mentioned, please follow steps 4.1 through 4.4 cover bootstrapping the project in the readme. Please also make sure you have Grails 2.2.1 installed and have GRAILS_HOME and JAVA_HOME set.

@ralph-tice
Fromt the running grailsw error, it looks like grails wrapper wasn't set up correctly. Did you run grails wrapper and see a wrapper directory was added to your project root?

from ice.

@fangji aha!! I didn't realize grails wrapper was different from grailsw. ./grailsw -Dice.s3AccessKeyId=$AWS_ACCESS_KEY_ID -Dice.s3SecretKey=$AWS_SECRET_KEY run-app works now, but I still get the same runtime exceptions trying to load http://localhost:8080/ice as in my previous comment.

from ice.

mm...

Try http://localhost:8080/ice/dashboard/summary instead.

from ice.

OK, that works. I guess the index route is missing?

from ice.

@fangji Thanks for the README update,

I have followed your readme and installed

1. Grails version: 2.1.1
2. Java - java version "1.6.0_27"
3. Uninstall running tomcat server.

Pull the latest code from repository and ran

grails wrapper

Wrapper installed successfully

Created ice-properties file like this and copied to grails-app/conf

ice.billing_s3bucketname=
ice.billing_s3bucketprefix=ice
ice.processor=true
ice.processor.localDir=/opt/ice/ice_processor
ice.work_s3bucketname=
ice.work_s3bucketprefix=iceworker
ice.s3AccessKeyId=
ice.s3SecretKey=
ice.startmillis=1370044800
ice.account1=
ice.account2=
ice.account3=
ice.reader=true
ice.processor.localDir=/opt/ice/ice_reader
ice.s3AccessKeyId=
ice.s3SecretKey=
ice.startmillis=1370044800
ice.account1=
ice.companyName=

Then I ran this command - ./grailsw -Dice.s3AccessKeyId=$AWS_ACCESS_KEY_ID -Dice.s3SecretKey=$AWS_SECRET_KEY run-app

I am getting this error

| Running Grails application
2013-06-20 21:47:19,401 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]
2013-06-20 21:47:19,519 [main] INFO core.StandardService - Starting service Tomcat
2013-06-20 21:47:19,519 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30
2013-06-20 21:47:19,828 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found
2013-06-20 21:47:20,280 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext
2013-06-20 21:47:27,191 [localhost-startStop-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/root/.grails/ivy-cache/net.sf.ehcache/ehcache-core/jars/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml
2013-06-20 21:47:27,406 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults.
2013-06-20 21:47:27,434 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults.
2013-06-20 21:47:28,981 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@27f33aef: startup date [Thu Jun 20 21:47:26 UTC 2013]; parent: Root WebApplicationContext
2013-06-20 21:47:32,403 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-06-20 21:47:32,651 [localhost-startStop-1] INFO BootStrap - Starting ice...
2013-06-20 21:47:34,883 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up...
2013-06-20 21:47:34,895 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started...
2013-06-20 21:47:34,895 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting...
| Error 2013-06-20 21:47:34,933 [localhost-startStop-1] ERROR BootStrap - Startup failed
Message: null
Line | Method
->> 411 | put in java.util.Hashtable

| 160 | setProperty in java.util.Properties
| 145 | doCall . . . . . . . . . . . . . in BootStrap$_closure1
| 301 | evaluateEnvironmentSpecificBlock in grails.util.Environment
| 294 | executeForEnvironment . . . . . in ''
| 270 | executeForCurrentEnvironment in ''
| 334 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync
| 166 | run in java.util.concurrent.FutureTask
| 1146 | runWorker . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor
| 615 | run in java.util.concurrent.ThreadPoolExecutor$Worker
^ 679 | run . . . . . . . . . . . . . . in java.lang.Thread

Please advise me to solve this problem

from ice.

@Jithinxavi you have a lot empty properties in your ice.properties file. did you remove them?
You have reader enabled, but didn't specify ice.reader.localDir

For the first time, I strongly suggest you only enable processor first.

from ice.

@Jithinxavi
Please note the following 3 things:

1. you don't need to set ice.s3AccessKeyId and ice.s3SecretKey in ice.properties file
2. please don't include the real access key and secret key in the issue comment any more.
3. The error happens at BootStrap$_closure1.doCall(BootStrap.groovy:145):
   properties.setProperty(IceOptions.LOCAL_DIR, prop.getProperty("ice.reader.localDir"));
   that means you do have ice.reader=true in your ice.properties file. please double check.

from ice.

Hi fangji, i have made a Clean installation from start and this I got this ----

@fangji : command not working for me :
./grailsw -Dice.s3AccessKeyId=A11111111111 -Dice.s3SecretKey=A22222222222 run-app

Script 'Dice.s3AccessKeyId=A11111111111111111' not found, did you mean:

1. CreateService
2. Package
3. GenerateController
4. Interactive
5. CreateMultiProjectBuild_

Please make a selection or enter Q to quit: q

And on using the following command i got the server up but when reached on browser it shows error
#./grailsw run-app

Output on Browser :
HTTP Status 404 - /ice/WEB-INF/grails-app/views/error.jsp)

Output on Server Terminal :
| Error 2013-06-21 07:02:34,353 [http-bio-8080-exec-2] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/
Cannot get property 'accountService' on null object. Stacktrace follows:
Message: Cannot get property 'accountService' on null object
Line | Method
->> 80 | doCall in com.netflix.ice.DashboardController$_closure3

| 68 | doCall in com.netflix.ice.DashboardController$_closure1
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run in ''
^ 662 | run . . in java.lang.Thread

Also please take a look at my ice.properites file . i have not enabled reader as you mentioned earilier.

contents ice.properities file
ice.processor=true
ice.reader=false
ice.reservationCapacityPoller=false
ice.reservationPeriod=threeyear
ice.reservationUtilization=HEAVY
ice.ondemandCostAlertThreshold=250
ice.startmillis=1364774400000
ice.companyName=ABC
ice.billing_s3bucketname=billing-test-ice
ice.billing_s3bucketprefix=ice/billing/
ice.work_s3bucketname=work_abcice
ice.work_s3bucketprefix=work_billing/
ice.processor.localDir=/mnt/ice_processor
ice.reader.localDir=/mnt/ice_reader
ice.monthlycachesize=12
ice.account.account1=0123456789

from ice.

@cottonmouth00 You ice.properties file look fine to me.
Did you miss '-' in the command? I tried the following and got the same error:
./grailsw Dice.s3AccessKeyId=A11111111111 -Dice.s3SecretKey=A22222222222 run-app
| Script 'Dice.s3AccessKeyId=A11111111111' not found, did you mean:

1. CreateService
2. Package
3. RefreshDependencies
4. Interactive
5. CreateScript

Please make a selection or enter Q to quit: q

from ice.

@fangji - Thanks fangi i realised that and executed it already and now stuck here ! ... please take a look -

#./grailsw run-app -Dice.s3AccessKeyId=A11111111111 -Dice.s3SecretKey=A22222222222 run-app

Output on Browser :
HTTP Status 404 - /ice/WEB-INF/grails-app/views/error.jsp)

Output on Server Terminal :
| Error 2013-06-21 07:02:34,353 [http-bio-8080-exec-2] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/
Cannot get property 'accountService' on null object. Stacktrace follows:
Message: Cannot get property 'accountService' on null object
Line | Method
->> 80 | doCall in com.netflix.ice.DashboardController$_closure3

| 68 | doCall in com.netflix.ice.DashboardController$_closure1
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run in ''
^ 662 | run . . in java.lang.Thread

from ice.

@cottonmouth00 It looks like you are trying to access the UI with reader disabled.

from ice.

Hi @fangji : ,

yes the reader wasnt enabled and now its been enabled, and got a new error

| Error 2013-06-23 17:57:49,455 [http-bio-8080-exec-1] ERROR errors.GrailsExceptionResolver - IOException occurred when processing request: [GET] /ice/
Stream closed. Stacktrace follows:
Message: Stream closed
Line | Method
->> 187 | doCall in com.netflix.ice.DashboardController$_closure8

| 68 | doCall in com.netflix.ice.DashboardController$_closure1
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run in ''
^ 662 | run . . in java.lang.Thread
| Error 2013-06-23 17:57:49,595 [http-bio-8080-exec-1] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:57:49,620 [http-bio-8080-exec-1] ERROR errors.GrailsExceptionResolver - ServletException occurred when processing request: [GET] /ice/
File "/WEB-INF/grails-app/views/error.jsp" not found. Stacktrace follows:
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:57:49,651 [http-bio-8080-exec-1] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:57:49,690 [http-bio-8080-exec-1] ERROR [/ice].[grails] - Servlet.service() for servlet grails threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:57:49,698 [http-bio-8080-exec-1] ERROR [/ice].[gsp] - Servlet.service() for servlet [gsp] in context with path [/ice] threw exception [File "/WEB-INF/grails-app/views/error.jsp" not found] with root cause
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
2013-06-23 17:57:50,268 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers...
| Error 2013-06-23 17:58:12,412 [http-bio-8080-exec-5] ERROR errors.GrailsExceptionResolver - IOException occurred when processing request: [GET] /ice/
Stream closed. Stacktrace follows:
Message: Stream closed
Line | Method
->> 187 | doCall in com.netflix.ice.DashboardController$_closure8

| 68 | doCall in com.netflix.ice.DashboardController$_closure1
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run in ''
^ 662 | run . . in java.lang.Thread
| Error 2013-06-23 17:58:12,436 [http-bio-8080-exec-5] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:58:12,442 [http-bio-8080-exec-5] ERROR errors.GrailsExceptionResolver - ServletException occurred when processing request: [GET] /ice/
File "/WEB-INF/grails-app/views/error.jsp" not found. Stacktrace follows:
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:58:12,462 [http-bio-8080-exec-5] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:58:12,474 [http-bio-8080-exec-5] ERROR [/ice].[grails] - Servlet.service() for servlet grails threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread
| Error 2013-06-23 17:58:12,484 [http-bio-8080-exec-5] ERROR [/ice].[gsp] - Servlet.service() for servlet [gsp] in context with path [/ice] threw exception [File "/WEB-INF/grails-app/views/error.jsp" not found] with root cause
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in ''
| 334 | service . . . . . . . in ''
| 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run . . . . . . . . . in ''
^ 662 | run in java.lang.Thread

I have installed ICE on Micro EC2 instance. on accessing the link :
http://ec2-107-22-116-234.compute-1.amazonaws.com:8080/ice/

I get this :-
{"status":200,"data":[]}

from ice.

@cottonmouth00 Try this url for now:
http://ec2-107-22-116-234.compute-1.amazonaws.com:8080/ice/dashboard/summary

I will fix the index redirecting in a little later.

from ice.

Hi @fangji - finally i am happy to see something on my browser.
http://ec2-107-22-116-234.compute-1.amazonaws.com:8080/ice/dashboard/detail

My boxes seems to be empty for account, region, product, operation etc. is this due to permission/accessbility issue if so where can i correct this.

Thanks

from ice.

@cottonmouth00 @Jithinxavi To verify ice runs correctly, I suggest you do the following:

1. Make sure you do receive billing files in billing s3 bucket. Make sure the billing files do contain hourly usage line items.
2. Re-run ice from a clean slate by following instruction in instructions 4.a-4.e in readme.
3. Make sure Ice writer finishes correctly. If you see errors, post your logs before log "AWS usage processed.".
4. Make sure you have non-empty output files in your working s3 bucket.

from ice.

Hi @fangji ,

Followed your above instructions and still I am facing the same issues, I could see some files are written in my prefix, billing bucket and worker bucket, but unfortunately there is no files in /mnt/ice_processor & /mnt/ice_reader
both these directory owned by root, ist any permission related issues? Please advise .

| Compiling 58 source files
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
| Running Grails application
2013-06-24 16:40:17,409 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]
2013-06-24 16:40:17,443 [main] INFO core.StandardService - Starting service Tomcat
2013-06-24 16:40:17,443 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30
2013-06-24 16:40:17,623 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found
2013-06-24 16:40:18,003 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext
2013-06-24 16:40:23,644 [localhost-startStop-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/root/.grails/ivy-cache/net.sf.ehcache/ehcache-core/jars/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml
2013-06-24 16:40:23,956 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults.
2013-06-24 16:40:23,985 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults.
2013-06-24 16:40:25,497 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@6781a7dc: startup date [Mon Jun 24 16:40:22 UTC 2013]; parent: Root WebApplicationContext
2013-06-24 16:40:29,440 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-06-24 16:40:29,705 [localhost-startStop-1] INFO BootStrap - Starting ice...
2013-06-24 16:40:31,629 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up...
2013-06-24 16:40:31,643 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started...
2013-06-24 16:40:31,643 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting...
2013-06-24 16:40:31,881 [localhost-startStop-1] INFO basic.BasicManagers - trying to find new tag group and data managers...
2013-06-24 16:40:32,715 [localhost-startStop-1] INFO basic.BasicManagers - poller thread for com.netflix.ice.basic.BasicManagers started...
2013-06-24 16:40:32,715 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - poller starting...
2013-06-24 16:40:32,724 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers...
2013-06-24 16:40:32,716 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - AWS usage processed.
2013-06-24 16:40:47,306 [localhost-startStop-1] INFO util.SessionIdGenerator - Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [14,532] milliseconds.
2013-06-24 16:40:47,354 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring FrameworkServlet 'grails'
2013-06-24 16:40:47,515 [main] INFO http11.Http11Protocol - Starting ProtocolHandler ["http-bio-8080"]
| Server running. Browse to http://localhost:8080/ice
1372092165688
1372092194028
1372092203220
2013-06-24 16:45:32,769 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers...

And How can I run this process in background, if my ssh is closed application is not working , I tried with nohup but no luck.

Thanks,
Jithin Xavier

from ice.

@Jithinxavi from your logs, it looks like BillingFileProcessor did not find any billing files to process at all. Can you confirm what billing files you have in your billing s3 bucket? What values did you set for the following properties?
ice.billing_s3bucketname
ice.billing_s3bucketprefix

from ice.

@fangji , Please find the details below.

ice.billing_s3bucketname=
ice.billing_s3bucketprefix=ice/billing/

Once I ran the grails app its automatically created one directory inside my billing bucket called ice and inside that created couple of files starting with billing(prefix) - Any issues here? or it should be inside the billing bucket itself?

from ice.

@Jithinxavi ice.billing_s3bucketname is the bucket where you detailed billing files are. ice.work_s3bucketname is the bucket Ice writer will use to store output files. It looks like you have the same values for both properties.

from ice.

@fangji , No , Both have different values,

ice.billing_s3bucketname=dsb-billingdata
ice.billing_s3bucketprefix=ice/billing/
ice.work_s3bucketname=dsb-iceworker
ice.work_s3bucketprefix=work_billing/

Inside the dsb-billingdata bucket have all the billing data files(.czv,gz etc) and inside the ice directory having the app generated data(attached screenshot)

And I have one separate bucket for ice write - dsb-iceworker ,inside of this having one folder and have output files.

from ice.

@Jithinxavi the billing files in your billing s3 bucket don't look right. As mentioned in prerequisite in readme, the billing files should look like: -aws-billing-detailed-line-items--.csv.zip or -aws-billing-detailed-line-items-with-resources-and-tags--.csv.zip. Please double check with Amazon that you receive the correct billing files.

from ice.

@Jithinxavi it looks like your ice.billing_s3bucketprefix should be empty since those billing files are directly under the s3 bucket. Make sure you have read permission to the billing s3 bucket and read&write permission to working s3 bucket. But since we didn't see any permission errors in your Ice writer log, you probably have the right permission already.

Please change ice.billing_s3bucketprefix to following and try again:
ice.billing_s3bucketprefix=

from ice.

Thanks @fangji for your valuable notes, It's looks awesome.

I have couple of concerns- 1. how can I run this apps in background, now when my ssh session is interrupting, application as well shutting down. I tried with nohup but no look.
2. In screenshot I can see one option "Breakdown" this tab is missing in my dashboard. How can I configure that?
3. Compare to screenshot in Readme , application style has little different, that is because of "highstock"?

from ice.

@Jithinxavi

1. One option is to create a war file and deploy it to tomcat
2. To enable breakdown feature, you will need to do step 6 in advanced options. Please see readme.
3. The ui style should be the same. In what way is yours different from the screenshots in readme?

from ice.

@fangji -

1. We have created one war file and follow the below steps.
   war file in - /ice/target/ice.war

   ubuntu@Netflix-ICE-Monitoring:~/ice$ sudo ./grailsw run-war
   | Done creating WAR target/ice.war
   | Running Grails application
   Tomcat Server running WAR (output written to: /home/ubuntu/ice/target/tomcat-out.txt)
   | Server running. Browse to http://localhost:8080/ice

ice.properties file I have passed,

ice.s3AccessKeyId=Access Key
ice.s3SecretKey=Secret Key

But unfortunately, I'm getting this problem ,looks like some credentials file issues.

2013-06-25 07:32:52,347 [localhost-startStop-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/home/ubuntu/ice/work/war/WEB-INF/lib/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml
2013-06-25 07:32:52,897 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults.
2013-06-25 07:32:53,025 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults.
2013-06-25 07:32:54,953 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@314382c6: startup date [Tue Jun 25 07:32:46 UTC 2013]; parent: Root WebApplicationContext
2013-06-25 07:32:58,059 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-06-25 07:32:58,513 [localhost-startStop-1] INFO BootStrap - Starting ice...
2013-06-25 07:32:59,230 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up...
2013-06-25 07:32:59,237 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started...
2013-06-25 07:32:59,249 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting...
2013-06-25 07:32:59,329 [localhost-startStop-1] INFO basic.BasicManagers - trying to find new tag group and data managers...
2013-06-25 07:33:00,526 [localhost-startStop-1] ERROR basic.BasicTagGroupManager - cannot poll data
AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 914BBD25162C7BAD, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: hfBpaymvraL7JBTIzpGfcOfaX8FoRdoFtV9BghyM+iob2SCRbb6xY9SpOg757W6H
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773)
at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191)
at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64)
at com.netflix.ice.basic.BasicTagGroupManager.(BasicTagGroupManager.java:54)
at com.netflix.ice.basic.BasicManagers.doWork(BasicManagers.java:111)
at com.netflix.ice.basic.BasicManagers.init(BasicManagers.java:59)
at com.netflix.ice.reader.ReaderConfig.(ReaderConfig.java:98)
at BootStrap$_closure1.doCall(BootStrap.groovy:163)
at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:301)
at grails.util.Environment.executeForEnvironment(Environment.java:294)
at grails.util.Environment.executeForCurrentEnvironment(Environment.java:270)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-06-25 07:33:00,535 [localhost-startStop-1] INFO basic.BasicTagGroupManager - poller thread for tagdb_all started...
2013-06-25 07:33:00,536 [tagdb_all] INFO basic.BasicTagGroupManager - poller starting...
2013-06-25 07:33:00,546 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling
AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: D5A7D61ACF577CAA, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: Ggt46yIaVWaMnhlWmyvfeesp1IC8nN7iZWZV+QaWx8kjLCgk5wKkCM5PnrJz6Mx5
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2950)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
at com.netflix.ice.common.AwsUtils.listAllObjects(AwsUtils.java:102)
at com.netflix.ice.processor.BillingFileProcessor.poll(BillingFileProcessor.java:75)
at com.netflix.ice.common.Poller.doWork(Poller.java:50)
at com.netflix.ice.common.Poller.access$000(Poller.java:28)
at com.netflix.ice.common.Poller$1.run(Poller.java:88)
at java.lang.Thread.run(Thread.java:679)
2013-06-25 07:33:00,552 [tagdb_all] ERROR basic.BasicTagGroupManager - Error polling
AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 61291A1A92D9DC79, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: nIbn8SrGymPPT8oPl+LXuq35x/m287bpuAFR63JVcA7x//Ep3T4alBsaAOv/or6a
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773)
at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191)
at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64)
at com.netflix.ice.common.Poller.doWork(Poller.java:50)
at com.netflix.ice.common.Poller.access$000(Poller.java:28)
at com.netflix.ice.common.Poller$1.run(Poller.java:88)
at java.lang.Thread.run(Thread.java:679)
2013-06-25 07:33:00,603 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_hourly_all started...
2013-06-25 07:33:00,604 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_hourly_all started...
2013-06-25 07:33:00,604 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_daily_all started...
2013-06-25 07:33:00,604 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_daily_all started...
2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_weekly_all started...
2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_weekly_all started...
2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_monthly_all started...
2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_monthly_all started...
2013-06-25 07:33:00,606 [localhost-startStop-1] INFO basic.BasicManagers - poller thread for com.netflix.ice.basic.BasicManagers started...
2013-06-25 07:33:00,603 [cost_hourly_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,606 [usage_monthly_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,605 [cost_monthly_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,605 [usage_weekly_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,605 [cost_weekly_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,604 [usage_daily_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,604 [cost_daily_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,604 [usage_hourly_all] INFO basic.BasicDataManager - poller starting...
2013-06-25 07:33:00,616 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - poller starting...
2013-06-25 07:33:00,617 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers...
2013-06-25 07:33:00,722 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring FrameworkServlet 'grails'
2013-06-25 07:33:00,785 [main] INFO http11.Http11Protocol - Starting ProtocolHandler ["http-bio-8080"]
Server running. Browse to http://localhost:8080/ice
2013-06-25 07:33:56,645 [http-bio-8080-exec-21] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/dashboard/getAccounts
Stacktrace follows:
java.lang.NullPointerException
at com.netflix.ice.basic.BasicTagGroupManager.getMonthMillis(BasicTagGroupManager.java:126)
at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:137)
at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:223)
at com.netflix.ice.DashboardController$_closure2.doCall(DashboardController.groovy:73)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-06-25 07:33:56,871 [http-bio-8080-exec-22] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [POST] /ice/dashboard/getData
Cannot invoke method getStart() on null object. Stacktrace follows:
java.lang.NullPointerException: Cannot invoke method getStart() on null object
at com.netflix.ice.DashboardController.roundInterval(DashboardController.groovy:621)
at com.netflix.ice.DashboardController.doGetData(DashboardController.groovy:397)
at com.netflix.ice.DashboardController$_closure11.doCall(DashboardController.groovy:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-06-25 07:34:31,131 [http-bio-8080-exec-14] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/dashboard/getAccounts
Stacktrace follows:
java.lang.NullPointerException
at com.netflix.ice.basic.BasicTagGroupManager.getMonthMillis(BasicTagGroupManager.java:126)
at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:137)
at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:223)
at com.netflix.ice.DashboardController$_closure2.doCall(DashboardController.groovy:73)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-06-25 07:34:31,136 [http-bio-8080-exec-12] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [POST] /ice/dashboard/getData

Please let me know if anything went wrong.

Thanks,
Jithin Xavier

from ice.

@Jithinxavi You don't need to set properties ice.s3AccessKeyId and ice.s3SecretKey in ice.properties file. They are system properties, you can set them through JAVA_OPTS.

from ice.

@fangji - I am newbie to tomcat and java, Can you please advise me to how to set the JAVA_OPTS. file with credentials?

And any restriction is there to add AWS account in ICE? (I have added 5 account butonly I can see 4)

Thanks,
Jithin Xavier

from ice.

@Jithinxavi If you are using tomcat, you can probably set it as env variable. Here is what I found on google when searching "tomcat how to set JAVA_OPTS": http://tomcat.10.x6.nabble.com/How-to-set-JAVA-OPTS-variable-in-tomcat6-0-16-td2119946.html
For the account issue, I'd open up the billing file and check if the account id is really there.

from ice.

Any chance of having ice use the instance metadata for credentials? (Ie.
Instance has been started with an IBM role)
On 26/06/2013 5:06 AM, "fangji" [email protected] wrote:

@Jithinxavi https://github.com/Jithinxavi You don't need to set
properties ice.s3AccessKeyId and ice.s3SecretKey in ice.properties file.
They are system properties, you can set them through JAVA_OPTS.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4#issuecomment-19999450
.

from ice.

@fangji - 1. Installed and configured tomcat7 with ice.s3AccessKeyId and ice.s3SecretKey in /etc/init.d/tomcat

JAVA_OPTS="ice.s3AccessKeyId=Accesskey ice.s3SecretKey=SecretKey" - correct?

1. Added my ice.war file in /var/lib/tomcat7/webapps and restart tomcat, and application I can access from http://url:8080/ice/dashboard/summary ,but still getting the access key error in tomcat catalina.out

Please have look on my catalina.out

13-06-26 11:39:27,706 [pool-2-thread-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/var/lib/tomcat7/webapps/ice/WEB-INF/lib/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml
2013-06-26 11:39:27,974 [pool-2-thread-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults.
2013-06-26 11:39:28,015 [pool-2-thread-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults.
2013-06-26 11:39:29,737 [pool-2-thread-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@2f473a2e: startup date [Wed Jun 26 11:39:23 UTC 2013]; parent: Root WebApplicationContext
2013-06-26 11:39:33,803 [pool-2-thread-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-06-26 11:39:34,079 [pool-2-thread-1] INFO BootStrap - Starting ice...
2013-06-26 11:39:35,283 [pool-2-thread-1] INFO processor.ProcessorConfig - starting up...
2013-06-26 11:39:35,289 [pool-2-thread-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started...
2013-06-26 11:39:35,293 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting...
2013-06-26 11:39:35,394 [pool-2-thread-1] INFO basic.BasicManagers - trying to find new tag group and data managers...
2013-06-26 11:39:36,420 [pool-2-thread-1] ERROR basic.BasicTagGroupManager - cannot poll data
AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 9E2DF6F4826EC604, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: 6siKtaIQMJetW/v/ojJXu/H0w8IqfYbaZi7Qp51Bm0egEHboPyI9gY5G8otRKnlt
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773)
at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191)
at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64)
at com.netflix.ice.basic.BasicTagGroupManager.(BasicTagGroupManager.java:54)
at com.netflix.ice.basic.BasicManagers.doWork(BasicManagers.java:111)
at com.netflix.ice.basic.BasicManagers.init(BasicManagers.java:59)
at com.netflix.ice.reader.ReaderConfig.(ReaderConfig.java:98)
at BootStrap$_closure1.doCall(BootStrap.groovy:163)
at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:301)
at grails.util.Environment.executeForEnvironment(Environment.java:294)
at grails.util.Environment.executeForCurrentEnvironment(Environment.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-06-26 11:39:36,429 [pool-2-thread-1] INFO basic.BasicTagGroupManager - poller thread for tagdb_all started...
2013-06-26 11:39:36,432 [tagdb_all] INFO basic.BasicTagGroupManager - poller starting...
2013-06-26 11:39:36,469 [tagdb_all] ERROR basic.BasicTagGroupManager - Error polling
AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 50F18CAE514C2A2E, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: xa7BsuK/N/tfU/2u6vJZZqKifyjbP6AKtogVwkgbfMhllk3iKgTNVnx00e9zDe33
2013-06-26 11:39:36,429 [pool-2-thread-1] INFO basic.BasicTagGroupManager - poller thread for tagdb_all started...
2013-06-26 11:39:36,432 [tagdb_all] INFO basic.BasicTagGroupManager - poller starting...
2013-06-26 11:39:36,469 [tagdb_all] ERROR basic.BasicTagGroupManager - Error polling
AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 50F18CAE514C2A2E, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: xa7BsuK/N/tfU/2u6vJZZqKifyjbP6AKtogVwkgbfMhllk3iKgTNVnx00e9zDe33
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773)
at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191)
at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64)
at com.netflix.ice.common.Poller.doWork(Poller.java:50)
at com.netflix.ice.common.Poller.access$000(Poller.java:28)
at com.netflix.ice.common.Poller$1.run(Poller.java:88)
at java.lang.Thread.run(Thread.java:679)
2013-06-26 11:39:36,469 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling
AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: F537BB4073C04D3F, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: StbH8zIxtNvh8C6P2YHM8bs86ukp1XPSq97vgq7LzBgN4Qcp+OzXp7sqoBgd53pY
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2950)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
at com.netflix.ice.common.AwsUtils.listAllObjects(AwsUtils.java:102)
at com.netflix.ice.processor.BillingFileProcessor.poll(BillingFileProcessor.java:75)
at com.netflix.ice.common.Poller.doWork(Poller.java:50)
at com.netflix.ice.common.Poller.access$000(Poller.java:28)
at com.netflix.ice.common.Poller$1.run(Poller.java:88)
at java.lang.Thread.run(Thread.java:679)
2013-06-26 11:39:36,534 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_hourly_all started...
2013-06-26 11:39:36,535 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_hourly_all started...
2013-06-26 11:39:36,535 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_daily_all started...
2013-06-26 11:39:36,536 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_daily_all started...
2013-06-26 11:39:36,536 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_weekly_all started...
2013-06-26 11:39:36,536 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_weekly_all started...
2013-06-26 11:39:36,537 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_monthly_all started...
2013-06-26 11:39:36,537 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_monthly_all started...
2013-06-26 11:39:36,537 [pool-2-thread-1] INFO basic.BasicManagers - poller thread for com.netflix.ice.basic.BasicManagers started...
2013-06-26 11:39:36,535 [cost_hourly_all] INFO basic.BasicDataManager - poller starting...
2013-06-26 11:39:36,537 [usage_monthly_all] INFO basic.BasicDataManager - poller starting...
2013-06-26 11:39:36,537 [cost_monthly_all] INFO basic.BasicDataManager - poller starting...
2013-06-26 11:39:36,537 [usage_weekly_all] INFO basic.BasicDataManager - poller starting...

from ice.

@Jithinxavi JAVA_OPTS should be "-Dice.s3AccessKeyId=Accesskey -Dice.s3SecretKey=SecretKey"
I will check in the code to get credentials from instance metadata today. Hopefully it will make things a little easier for you.

from ice.

Oh! That's great @fangji - I have added the new entry as well but no luck. once you made the metadata change, Please let me know.

from ice.

@fangji will you able to use the token from the instance userdata also?

from ice.

@Jithinxavi @ma78 Now, if access keys are not set through System properties, Bootstrap.groovy will use credentials of instance metadata by using InstanceProfileCredentialsProvider.

from ice.

@fangji Thanks for the update. I have added credentials as userdata in instance, like below.

ubuntu@Netflix-ICE-Monitoring:$ GET http://169.254.169.254/latest/user-data
ice.s3AccessKeyId=AccessKeyId
ice.s3SecretKey=s3SecretKey@Netflix-ICE-Monitoring:$

Is this correct?

from ice.

I have been fighting this all day, not sure why I am hitting a null exception...?
Any insight would be greatly appreciated. I am not a Java developer and fairly new to AWS.

sudo ./grailsw -Dice.s3AccessKeyId=theS3AccessKeyId -Dice.s3SecretKey=theS3SecretKey run-app

| Running Grails application
2013-07-02 11:49:00,603 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]
2013-07-02 11:49:00,625 [main] INFO core.StandardService - Starting service Tomcat
2013-07-02 11:49:00,625 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30
2013-07-02 11:49:00,693 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found
2013-07-02 11:49:00,746 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext
2013-07-02 11:49:02,602 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@326d234c: startup date [Tue Jul 02 11:49:02 PDT 2013]; parent: Root WebApplicationContext
2013-07-02 11:49:03,193 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-07-02 11:49:03,214 [localhost-startStop-1] INFO BootStrap - Starting ice...
| Error java.lang.NullPointerException
| Error at java.util.Hashtable.put(Hashtable.java:394)
| Error at java.util.Properties.setProperty(Properties.java:143)
| Error at java_util_Properties$setProperty.call(Unknown Source)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
| Error at BootStrap$_closure1.doCall(BootStrap.groovy:117)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| Error at java.lang.reflect.Method.invoke(Method.java:597)
| Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1243)
| Error at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
| Error at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:956)
| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
| Error at groovy.lang.Closure.call(Closure.java:412)
| Error at groovy.lang.Closure.call(Closure.java:406)
| Error at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:308)
| Error at grails.util.Environment.executeForEnvironment(Environment.java:301)
| Error at grails.util.Environment.executeForCurrentEnvironment(Environment.java:277)
| Error at org.codehaus.groovy.grails.commons.DefaultGrailsBootstrapClass.callInit(DefaultGrailsBootstrapClass.java:60)
| Error at org.codehaus.groovy.grails.web.context.GrailsConfigUtils.executeGrailsBootstraps(GrailsConfigUtils.java:74)
| Error at org.codehaus.groovy.grails.web.context.GrailsContextLoader.initWebApplicationContext(GrailsContextLoader.java:108)
| Error at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
| Error at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
| Error at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
| Error at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
| Error at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
| Error at java.util.concurrent.FutureTask.run(FutureTask.java:138)
| Error at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
| Error at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
| Error at java.lang.Thread.run(Thread.java:680)
| Error 2013-07-02 11:49:03,282 [localhost-startStop-1] ERROR BootStrap - Startup failed
Message: null
Line | Method
->> 394 | put in java.util.Hashtable

| 143 | setProperty in java.util.Properties
| 117 | doCall . . . . . . . . . . . . . in BootStrap$_closure1
| 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment
| 301 | executeForEnvironment . . . . . in ''
| 277 | executeForCurrentEnvironment in ''
| 303 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync
| 138 | run in java.util.concurrent.FutureTask
| 895 | runTask . . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run in ''
^ 680 | run . . . . . . . . . . . . . . in java.lang.Thread

ice.properties:

ice.processor=true
ice.reader=false
ice.startmillis=1370070000000
ice.billing_s3bucketname=abucket
ice.billing_s3bucketprefix=
ice.work_s3bucketname=work_abucket
ice.work_s3bucketprefix=
ice.processor.localDir=/mnt/ice_processor
ice.account.account1=00123456789

from ice.

@ebuychance It looks like you didn't specify ice.companyName in your ice.properties file. The readme says it's only needed for running reader. But actually it's also required for running processor in the code. I will fix the code in a little later. For now you can try adding ice.companyName to your ice.properties file.

from ice.

Wow that was a fast response, work around and a code fix!
I am very impressed and thankful. Thankful both for the open source project and bug support.

Yep that did the trick, loved seeing this message: "AWS usage processed"

from ice.

I think the README is in much better shape now, thanks for all the awesome effort @fangji and very nice project !

from ice.

@fangji - i am a new bie here , i have been working on Netflix since two weeks and unable to proceed further because of some errors . I am in my local environment and got the ice /netflix .
I have my ice properties file in right location .

the issue is when i run this command :
grailsw -Dice.s3AccessKeyId=ABCDEFADHAHAH -Dice.s3SecretKey=ox737271dhwhhdgqg737hsh run-app

I get the following error 👍

Extracting C:\Users\roin.grails\wrapper\grails-2.2.1-download.zip to C:\Users\r
oin.grails\wrapper\2.2.1
Exception in thread "main" java.util.zip.ZipException: error in opening zip file

    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:215)
    at java.util.zip.ZipFile.<init>(ZipFile.java:145)
    at java.util.zip.ZipFile.<init>(ZipFile.java:159)
    at org.grails.wrapper.GrailsWrapper.extract(GrailsWrapper.java:147)
    at org.grails.wrapper.GrailsWrapper.configureGrailsInstallation(GrailsWr

apper.java:138)
at org.grails.wrapper.GrailsWrapper.main(GrailsWrapper.java:58)

I have already installed the following files in LIB
ant-1.7.0.jar | guava-14.0.1.jar | httpcore-4.1.jar jcommon-1.0.16.jar joda-time-2.0.jar org.json-chargebee-1.0.jar wstx-asl-3.2.9.jar aws-java-sdk-1.4.4.1.jar httpclient-4.1.jar javacsv-2.0.jar jfreechart-1.0.13.jar mail-1.4.1.jar poi-3.7.jar

Please Help.

from ice.