teevity / ice Goto Github PK

Not completely sure what is breaking, and this error message isn't helping ;-)

A nice error message saying "hey, we are trying to (read|write) (bucket name)" would greatly improve debugging (assuming this is the problem).

thanks! Looking forward to getting this working...

ok, back to debugging... thanks!

nickg

| Error AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: FD5A4301E1B43050, AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method., S3 Extended Request ID: xxx/yyy
| Error at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:686)
| Error at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:350)
| Error at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:202)
| Error at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3066)
| Error at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:961)
| Error at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:847)
| Error at com.netflix.ice.common.AwsUtils.download(AwsUtils.java:334)
| Error at com.netflix.ice.common.AwsUtils.download(AwsUtils.java:329)
| Error at com.netflix.ice.common.AwsUtils.downloadFileIfNotExist(AwsUtils.java:317)
| Error at com.netflix.ice.basic.BasicReservationService.init(BasicReservationService.java:96)
| Error at com.netflix.ice.processor.ProcessorConfig.(ProcessorConfig.java:93)
| Error at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
| Error at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
| Error at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
| Error at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
| Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrConstructorNewInstance(ReflectiveInterceptor.java:975)
| Error at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
| Error at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
| Error at BootStrap$_closure1.doCall(BootStrap.groovy:186)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| Error at java.lang.reflect.Method.invoke(Method.java:616)
| Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1243)
| Error at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
| Error at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:956)
| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
| Error at groovy.lang.Closure.call(Closure.java:412)
| Error at groovy.lang.Closure.call(Closure.java:406)
| Error at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:308)
| Error at grails.util.Environment.executeForEnvironment(Environment.java:301)
| Error at grails.util.Environment.executeForCurrentEnvironment(Environment.java:277)
| Error at org.codehaus.groovy.grails.commons.DefaultGrailsBootstrapClass.callInit(DefaultGrailsBootstrapClass.java:60)
| Error at org.codehaus.groovy.grails.web.context.GrailsConfigUtils.executeGrailsBootstraps(GrailsConfigUtils.java:74)
| Error at org.codehaus.groovy.grails.web.context.GrailsContextLoader.initWebApplicationContext(GrailsContextLoader.java:108)
| Error at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
| Error at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
| Error at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
| Error at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
| Error at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
| Error at java.util.concurrent.FutureTask.run(FutureTask.java:166)
| Error at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
| Error at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
| Error at java.lang.Thread.run(Thread.java:679)
| Error 2013-11-05 07:17:52,776 [localhost-startStop-1] ERROR BootStrap - Startup failed
Message: The request signature we calculated does not match the signature you provided. Check your key and signing method.
Line | Method
->> 686 | handleErrorResponse in com.amazonaws.http.AmazonHttpClient

| 350 | executeHelper in ''
| 202 | execute . . . . . . . . . . . . in ''
| 3066 | invoke in com.amazonaws.services.s3.AmazonS3Client
| 961 | getObject . . . . . . . . . . . in ''
| 847 | getObject in ''
| 334 | download . . . . . . . . . . . . in com.netflix.ice.common.AwsUtils
| 329 | download in ''
| 317 | downloadFileIfNotExist . . . . . in ''
| 96 | init in com.netflix.ice.basic.BasicReservationService
| 93 | . . . . . . . . . . . . . in com.netflix.ice.processor.ProcessorConfig
| 186 | doCall in BootStrap$_closure1
| 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment
| 301 | executeForEnvironment in ''
| 277 | executeForCurrentEnvironment . . in ''
| 334 | innerRun in java.util.concurrent.FutureTask$Sync
| 166 | run . . . . . . . . . . . . . . in java.util.concurrent.FutureTask
| 1146 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 679 | run in java.lang.Thread

I'm having an issue where I set ice.s3AccessKeyId and ice.s3SecretKey in the ice.properties file, but on startup it attempts to use an IAM role anyhow.

if I pass those properties on at startup with -D, it works as expected.

| Error 2013-08-28 20:03:25,218 [localhost-startStop-1] ERROR BootStrap  - Startup failed
Message: Unable to load credentials from Amazon EC2 metadata service
    Line | Method
->>   99 | loadCredentials                  in com.amazonaws.auth.InstanceProfileCredentialsProvider
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|     37 | getCredentials                   in     ''
|   2969 | invoke . . . . . . . . . . . . . in com.amazonaws.services.s3.AmazonS3Client
|    911 | getObject                        in     ''
|    801 | getObject . . . . . . . . . . .  in     ''
|    334 | download                         in com.netflix.ice.common.AwsUtils
|    329 | download . . . . . . . . . . . . in     ''
|    317 | downloadFileIfNotExist           in     ''
|     96 | init . . . . . . . . . . . . . . in com.netflix.ice.basic.BasicReservationService
|     89 | <init>                           in com.netflix.ice.processor.ProcessorConfig
|    179 | doCall . . . . . . . . . . . . . in BootStrap$_closure1
|    308 | evaluateEnvironmentSpecificBlock in grails.util.Environment
|    301 | executeForEnvironment . . . . .  in     ''
|    277 | executeForCurrentEnvironment     in     ''
|    334 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync
|    166 | run                              in java.util.concurrent.FutureTask
|   1146 | runWorker . . . . . . . . . . .  in java.util.concurrent.ThreadPoolExecutor
|    615 | run                              in java.util.concurrent.ThreadPoolExecutor$Worker
^    679 | run . . . . . . . . . . . . . .  in java.lang.Thread

Caused by FileNotFoundException: http://169.254.169.254/latest/meta-data/iam/security-credentials/
->>  106 | readResponse                     in com.amazonaws.internal.EC2MetadataClient
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|     88 | readResource                     in     ''
|     53 | getDefaultCredentials . . . . .  in     ''
|     73 | loadCredentials                  in com.amazonaws.auth.InstanceProfileCredentialsProvider
|     37 | getCredentials . . . . . . . . . in     ''
|   2969 | invoke                           in com.amazonaws.services.s3.AmazonS3Client
|    911 | getObject . . . . . . . . . . .  in     ''
|    801 | getObject                        in     ''
|    334 | download . . . . . . . . . . . . in com.netflix.ice.common.AwsUtils
|    329 | download                         in     ''
|    317 | downloadFileIfNotExist . . . . . in     ''
|     96 | init                             in com.netflix.ice.basic.BasicReservationService
|     89 | <init> . . . . . . . . . . . . . in com.netflix.ice.processor.ProcessorConfig
|    179 | doCall                           in BootStrap$_closure1
|    308 | evaluateEnvironmentSpecificBlock in grails.util.Environment
|    301 | executeForEnvironment            in     ''
|    277 | executeForCurrentEnvironment . . in     ''
|    334 | innerRun                         in java.util.concurrent.FutureTask$Sync
|    166 | run . . . . . . . . . . . . . .  in java.util.concurrent.FutureTask
|   1146 | runWorker                        in java.util.concurrent.ThreadPoolExecutor
|    615 | run . . . . . . . . . . . . . .  in java.util.concurrent.ThreadPoolExecutor$Worker
^    679 | run                              in java.lang.Thread

I think there may be a race condition that's causing tagdb_all to be written in a way that the reader can't understand. After struggling for quite awhile I managed to get ice working once, then when I brought it down and up again to run inside a screen session, it broke in the same way that it was breaking before.

The first sign of trouble is that there is an error polling in the BasicGroupTagManager. Then, other subsequent calls to getAccounts and getData fail. See log lines below.

I've confirmed that the tagdb_all file exists and contains data. It's about 150KB.

I'm running a single grails process with the processor, reader, and reservationCapacity set to true. I've pasted in my ice.properties, with some values replaced with x's.

# whether or not to start processor
ice.processor=true

# whether or not to start reader/UI
ice.reader=true

# whether or to start reservation capacity poller
ice.reservationCapacityPoller=true

# reservation period, possible values are oneyear, threeyear
ice.reservationPeriod=oneyear
# reservation utilization, possible values are LIGHT, HEAVY
ice.reservationUtilization=HEAVY

# url prefix, e.g. http://ice.netflix.com/. Will be used in alert emails.
ice.urlPrefix=http://ice.test.com/

# from email address
ice.fromEmail=brandon@xxxxxx

# ec2 ondemand hourly cost threshold to send alert email. The alert email will be sent at most once per day.
ice.ondemandCostAlertThreshold=250

# ec2 ondemand hourly cost alert emails, separated by ","
ice.ondemandCostAlertEmails=brandon@xxxxxx

# change the following properties to your settings
ice.startmillis=1354320000000
ice.companyName=DreamBoxLearning, Inc.
ice.billing_s3bucketname=dreambox-billing-reports
ice.billing_s3bucketprefix=
ice.work_s3bucketname=dreambox-ops
ice.work_s3bucketprefix=work_ice/
ice.processor.localDir=/mnt/ice_processor
ice.reader.localDir=/mnt/ice_reader
ice.monthlycachesize=12

# change the follow account settings
ice.account.dreambox=xxxxxx

| Error 2013-06-20 22:58:46,877 [tagdb_all] ERROR basic.BasicTagGroupManager  - Error polling
Message: null
   Line | Method
->> 106 | hashCode             in com.netflix.ice.common.TagGroup
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   917 | get                  in java.util.concurrent.ConcurrentHashMap
|   119 | getTagGroup . . . .  in com.netflix.ice.common.TagGroup
|   184 | deserialize          in com.netflix.ice.common.TagGroup$Serializer
|   161 | deserializeTagGroups in     ''
|    69 | poll                 in com.netflix.ice.basic.BasicTagGroupManager
|    50 | doWork . . . . . . . in com.netflix.ice.common.Poller
|    28 | access$000           in     ''
|    88 | run . . . . . . . .  in com.netflix.ice.common.Poller$1
^   679 | run                  in java.lang.Thread

| Error 2013-06-20 23:00:09,677 [http-bio-8080-exec-5] ERROR errors.GrailsExceptionResolver  - NullPointerException occurred when processing request: [GET] /ice/dashboard/getAccounts
Stacktrace follows:
Message: null
    Line | Method
->>  126 | getMonthMillis in com.netflix.ice.basic.BasicTagGroupManager
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|    137 | getAccounts    in     ''
|    223 | getAccounts .  in     ''
|     73 | doCall         in com.netflix.ice.DashboardController$_closure2
|   1146 | runWorker . .  in java.util.concurrent.ThreadPoolExecutor
|    615 | run            in java.util.concurrent.ThreadPoolExecutor$Worker
^    679 | run . . . . .  in java.lang.Thread

| Error 2013-06-20 23:00:09,917 [http-bio-8080-exec-7] ERROR errors.GrailsExceptionResolver  - NullPointerException occurred when processing request: [POST] /ice/dashboard/getData
Cannot invoke method getStart() on null object. Stacktrace follows:
Message: Cannot invoke method getStart() on null object
    Line | Method
->>  621 | roundInterval in com.netflix.ice.DashboardController
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|    397 | doGetData     in     ''
|    314 | doCall . . .  in com.netflix.ice.DashboardController$_closure11
|   1146 | runWorker     in java.util.concurrent.ThreadPoolExecutor
|    615 | run . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^    679 | run           in java.lang.Thread

I have been able to get the application up and running on a single node running both the processor and the reader but I can't seem to see any logs for how the processor works or what it's doing (or not doing). I am able to see the reader attempt to read the "s3 work bucket" looking for files to render but obviously not finding any because I don't think my processor is working. Any tips on where to look in the logs for the processor trying to access the billing bucket? I have enabled debugging in log4j root() and still can't seem to see any processor activity. This will be a pretty great tool once I get it ingesting our billing information. Cheers.

It's not really open source if it depends on non-free things; in this case, the dependency on Highcharts. Sexy as it may be, the highstock library is only free for non-commercial use and while it's lovely that they've creative commons licenced it, CC-BY-NC is not something we can redistribute. I'd like to think in 2013 you could find a charting library that doesn't inhibit us packaging.

AfC

I'm running ICE on an EC2 instance, and trying to force the client traffic over SSL. But not only does going to /ice/ redirect you to localhost, but even going to https://$IP/ice/dashboard/summary redirects back to http. I'm not familiar enough with grails or tomcat to get this behavior to stop.

Anyone have any suggestions?

I've installed tomcat, grails, and ivy via brew on MacOSX.
I copied build-template.xml to build.xml
when I run:
GRAILS_HOME=/usr/local ant
I get:

Buildfile: /Users/ralphtice/personal/ice/build.xml
     [echo] /usr/local/bin/grails

war:
     [exec] Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/groovy/grails/cli/support/GrailsStarter
     [exec] Caused by: java.lang.ClassNotFoundException: org.codehaus.groovy.grails.cli.support.GrailsStarter
     [exec]     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
     [exec]     at java.security.AccessController.doPrivileged(Native Method)
     [exec]     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
     [exec]     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
     [exec]     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
     [exec]     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

BUILD FAILED
/Users/ralphtice/personal/ice/build.xml:46:

I invoke ivy and get:

Caused by: java.io.FileNotFoundException: JAR entry org/apache/ivy/core/settings/ivy-configurations.xml not found in /Users/ralphtice/homebrew/Cellar/ivy/2.3.0/libexec/ivy-2.3.0.jar

When I remove this reference in ivy.xml everything breaks:

Exception in thread "main" java.text.ParseException: Problem occurred while parsing ivy file: Cannot add dependency 'javax.mail#mail;1.4.1' to configuration 'compile' of module netflix#ice;[email protected] because this configuration doesn't exist! in file:/Users/ralphtice/personal/ice/ivy.xml

I have been using ICE successfully since June, hosting both processor and reader on a t1.micro that is rebooted by cron every night (!) as I don't know enough about Tomcat to optimize. The build was from ~July 30 (no actual commit can be derived, because my dev laptop crashed since then).

Recently, around a week ago, ICE began to simply render blank screens when I navigated to the site. In looking at the logs, I see this stack trace repeated hundreds of times.

Oct 29, 2013 12:08:19 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.sun.org.apache.xerces.internal.parsers.SAXParser. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1587)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1546)
at org.xml.sax.helpers.NewInstance.newInstance(NewInstance.java:82)
at org.xml.sax.helpers.XMLReaderFactory.loadClass(XMLReaderFactory.java:227)
at org.xml.sax.helpers.XMLReaderFactory.createXMLReader(XMLReaderFactory.java:190)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.(XmlResponsesSaxParser.java:110)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:75)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:72)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:562)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:319)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2950)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)
at com.netflix.ice.basic.BasicManagers.doWork(BasicManagers.java:94)
at com.netflix.ice.basic.BasicManagers.poll(BasicManagers.java:81)
at com.netflix.ice.common.Poller.doWork(Poller.java:50)
at com.netflix.ice.common.Poller.access$000(Poller.java:28)
at com.netflix.ice.common.Poller$1.run(Poller.java:88)
at java.lang.Thread.run(Thread.java:724)

I am in the process now of rebuilding from head and recreating my configuration files, but wanted to see if anyone has seen this exception before or has ideas of what it could mean.

Thanks.

Hi,

There is a prerequisite to acquire the proper license of highstock before using ice. I do not want to invest any money into it, just want to explore the features of ice. Is there any workaround?

Thanks,
Prateek

In our account we have a "billing" tag on most of our resources. I've pulled the latest copy of Ice and added "ice.customTags=billing" to ice.properties but I don't see any references to the tags in the interface anywhere. We're enabled for the beta detailed billing as well.

The Prerequisite instructions imply that only Programmatic Access needs to be turned in AWS Billing Preferences. This is incorrect: Detailed Billing Report or Detailed Billing Report (with resources and tags) must be turned on also. If only Programmatic Access is turned on, the S3 bucket will be filled with files such as: xxxxxxxxxxxx-aws-billing-csv-2013-06.csv, which is not recognizable by ICE.

I am trying to run Ice on a linux server and when accessing the dashboard summary page it is giving me the error message shown below. I have set the companyName value in the file ice.properties (stored in dir src/java) so not sure why I am getting this. Have tried running on Java 6 and Java 7 as well as Ubuntu and Amazon Linux and get the same error message.

| Error 2013-06-25 07:28:01,968 [http-bio-8080-exec-3] ERROR [/ice].[default]  - Servlet.service() for servlet [default] in context with path [/ice] threw exception
Message: Error applying layout : main
    Line | Method
->> 1146 | runWorker in /grails-app/views/layouts/main.gsp
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|    615 | run       in     ''
^    679 | run . . . in     ''

Caused by GroovyPagesException: Error processing GroovyPageView: Error executing tag <sitemesh:wrapTitleTag>: Error evaluating expression ["${ReaderConfig.getInstance().companyName} AWS Usage Dashboard"] on line [9]: Cannot get property 'companyName' on null object
->>  464 | runWorker in /grails-app/views/layouts/main.gsp
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Caused by GrailsTagException: Error executing tag <sitemesh:wrapTitleTag>: Error evaluating expression ["${ReaderConfig.getInstance().companyName} AWS Usage Dashboard"] on line [9]: Cannot get property 'companyName' on null object
->>   11 | doCall    in /grails-app/views/layouts/main.gsp
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Caused by GroovyPagesException: Error evaluating expression ["${ReaderConfig.getInstance().companyName} AWS Usage Dashboard"] on line [9]: Cannot get property 'companyName' on null object
->>    9 | doCall    in /grails-app/views/layouts/main.gsp
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Caused by NullPointerException: Cannot get property 'companyName' on null object
->>   21 | doCall    in opt_ice_grails_app_views_layouts_main_gsp$_run_closure1_closure15_closure24_closure25
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|     23 | doCall    in opt_ice_grails_app_views_layouts_main_gsp$_run_closure1_closure15
|     25 | doCall .  in opt_ice_grails_app_views_layouts_main_gsp$_run_closure1
|     38 | run       in opt_ice_grails_app_views_layouts_main_gsp
|   1146 | runWorker in java.util.concurrent.ThreadPoolExecutor
|    615 | run       in java.util.concurrent.ThreadPoolExecutor$Worker
^    679 | run . . . in java.lang.Thread
2013-06-25 07:28:02,171 [http-bio-8080-exec-3] INFO  [localhost].[/ice]  - Initializing Spring FrameworkServlet 'grails-errorhandler'

Loving this tool. When looking at the graphs it produces all the data is in GMT, which can be confusing when trying to line up events with local time.

Is there (or should there be) a simple way to convert everything over to a local time, so that when you are looking graphs everything appears in local time.
Cheers

When processing a file that contains usage for the cg1.4large instance type I'm getting the error below. I'm using the ice.customTags config property. Without customTags set I don't get this error.

| Error 2013-09-25 15:21:52,288 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor -
Message: null
Line | Method
->> 393 | getLatestHourlyTotalPrice in com.netflix.ice.basic.BasicReservationService

| 207 | process in com.netflix.ice.basic.BasicLineItemProcessor
| 743 | processOneLine . . . . . in com.netflix.ice.processor.BillingFileProcessor
| 709 | processBillingFile in ''
| 670 | processBillingZipFile . . in ''
| 187 | poll in ''
| 50 | doWork . . . . . . . . . in com.netflix.ice.common.Poller
| 28 | access$000 in ''
| 88 | run . . . . . . . . . . . in com.netflix.ice.common.Poller$1
^ 680 | run in java.lang.Thread

Since the ICE code is trying to download reservation_capacity.txt from the work bucket, I assume the processor somehow creates that file up there? Am I correct?

Quicker question : What creates reservation_poller.txt in the working bucket?

    // read from s3 if not exists
    File file = new File(config.localDir, "reservation_capacity.txt");

    if (!file.exists()) {
        logger.info("downloading " + file + "...");
        AwsUtils.downloadFileIfNotExist(config.workS3BucketName, config.workS3BucketPrefix, file);
        logger.info("downloaded " + file);

The list in BasicReservationService is only including US and EU regions, but not SA and AP.

As this list seems autogenerated, can we regenerate it with all prices? I get NullPointerExceptions for instances in South America or Asia. If you post instructions how to regenerate the list I can create a pull request as well.

This isn't an issue with ICE, which is an awesome tool.
This is more about AWS and timeliness of providing data.

It's been 68 hours now since the last bit of data has come in (Jun 30, 2AM). Is anyone else experiencing this?

I posted to the AWS forums. If anyone has more experience with this, can you also post here or there?

https://forums.aws.amazon.com/thread.jspa?messageID=474867

Thanks!

Some additions for AWS support if you could add these?

"xxx","xxxx","xxxx","LineItem","xxx","AWS Support (Business)","0",,,,,,"N","Recurring Fee",,,,,"100.000000",,"100.000000",,,,,,

"xxx","xxx","xxx","LineItem","","AWS Support (Enterprise)","xxx","xxx","xxx","Dollar","","","N","5% of monthly AWS usage from $500K-$1M","2013-08-01 00:00:00","2013-08-31 23:59:59","0.0000000000","0.0500000000","0.0000000000",

"xxx","xxx","xxx","LineItem","","AWS Support (Enterprise)","xxx","xxx","xxx","Dollar","","","N","7% of monthly AWS usage from $150K-$500K","2013-08-01 00:00:00","2013-08-31 23:59:59","0.0000000000","0.0700000000","0.0000000000",

"xxx","xxx","xxxx","LineItem","","AWS Support (Enterprise)","xxx","xxx","xxx","Dollar","","","N","10% of monthly AWS usage for the first $0-$150K","2013-08-01 00:00:00","2013-08-31 23:59:59","xxx","0.1000000000","xxx",

"xxx","xxx","xxx","LineItem","","AWS Support (Enterprise)","xxx","xxx","xxx","Dollar","","","N","3% of monthly AWS usage from $1M+","2013-08-01 00:00:00","2013-08-31 23:59:59","0.0000000000","0.0300000000","0.0000000000",

getResourceGroupLists errors if the product has no resources associated with it. We don't use RDS and so this would crash because TagGroupManager tagGroupManager = getManagers().getTagGroupManager(product); is assumed to return an object.

I have temporarily mitigated it by surrounding the inner-most for-loop with a try/catch to ignore unused AWS products.

Since I don't know if this is anywhere close to an appropriate solution, I'm just reporting the issue and not trying to submit a true fix.

We have been playing with this tool for a couple weeks and really love it! It is a great contribution so thank you for the ongoing support. My company is an AWS reseller and we deal with large companies that typically have many AWS accounts that we consolidate under our reseller payer account. With this situation, we have a unique requirement where we have to "re-blend" groups of linked account. This is basically the process generating the blended rate for a subset of our linked accounts as apposed to all of them which is what AWS delivers in our month end reports. This could leverage the Breakdown feature from a grouping perspective, but I was wondering if you could answer the following:

1. What part of the ICE application is responsible for all calculations? This will tell me where we should focus our efforts to extend ICE with this functionality as well as what type of dev resource would be required (i.e. java or ruby).
2. Do you have any plans to add such functionality? If so, when?

Any thoughts you could provide here would be most helpful. If we are able to add this extension, we would be more than happy to contribute it back to the main project for others to use as well!

Thanks,
David Benson
Principal
DSB Consulting

I'm seeing a situation in the last two to three months where the costs that ice is reporting are drastically higher than the billing summaries we are seeing within the AWS console. This happens to coincide with a large purchase of reserved resources (rds, ec2, elasticache) that we made around that same time that our reporting starting looking way off in ice. Currently we are seeing reported costs in the ballpark of 28% over actual aws reported costs in ice for the last few months.

Is it safe to assume that if we have not properly configured ice to account for our reserved instances that we'll see incorrect (as in much higher numbers) cost rollups within ice?

Also I've verified these numbers are way over for us both with standard billing files and our detailed billing files with tags and resources.

The screenshot below is from the 'Details With Resource Groups' page in the Ice dashboard:

For some reason the values for 'aggregated' and 'cloudfront' are the same. Ignoring 'cloudfront', the aggregated total for this table is actually $19368.70.

This looks like a bug in Ice.

Can we get a license agreement posted?

I hope this helps someone else. I had to make these change to get ice to respond correctly over SSL. Should be the same if you want to use a port other than 8080, minus the https.

Change all references to localhost:8080 to the correct domainname and port in these files:
conf/system/server.xml
grails-app/conf/Config.groovy
work/plugins/tomcat-2.2.1/scripts/Tomcat.groovy

This may be a known issue but I just wanted to make sure..

ICE doesn't seem to pick up my EBS costs?

When I filter for EBS in a given region where I know I have EBS costs I see nothing. Am i missing something here?

Error java.lang.RuntimeException: failed to poll reservation prices ~/ice_processor/reservation_prices.threeyear.HEAVY (No such file or directory)

This is the configuration file. /home/ec2_user/ice_processor and /home/ec2_user/ice_reader both exist, but are empty.

# whether or not to start processor
ice.processor=true

# whether or not to start reader/UI
ice.reader=true

# whether or not to start reservation capacity poller
ice.reservationCapacityPoller=false

# reservation period, possible values are oneyear, threeyear
ice.reservationPeriod=threeyear
# reservation utilization, possible values are LIGHT, MEDIUM, HEAVY
ice.reservationUtilization=HEAVY

# the highstock url; host it somewhere else and change this if you need HTTPS
ice.highstockUrl="http://code.highcharts.com/stock/highstock.js"

# url prefix, e.g. http://ice.netflix.com/. Will be used in alert emails.
ice.urlPrefix=

# from email address
ice.fromEmail=

# ec2 ondemand hourly cost threshold to send alert email. The alert email will be sent at most once per day.
ice.ondemandCostAlertThreshold=250

# ec2 ondemand hourly cost alert emails, separated by ","
ice.ondemandCostAlertEmails=

# modify the following 5 properties according to your billing files configuration. if you have multiple payer accounts, you will need to specify multiple values for each property.
# s3 bucket name where the billing files are. multiple bucket names are delimited by ",". Ice must have read access to billing s3 bucket.
ice.billing_s3bucketname=paolo-billing
# prefix of the billing files. multiple prefixes are delimited by ","
ice.billing_s3bucketprefix=,
# specify your payer account id here if across-accounts IAM role access is used. multiple account ids are delimited by ",". "ice.billing_payerAccountId=,222222222222" means assumed role access is only use
d for the second bucket.
#ice.billing_payerAccountId=,123456789012
# specify the assumed role name here if you use IAM role access to read from billing s3 bucket. multiple role names are delimited by ",". "ice.billing_accessRoleName=,ice" means assumed role access is onl
y used for the second bucket.
#ice.billing_accessRoleName=,ice
# specify external id here if it is used. multiple external ids are delimited by ",". if you don't use external id, you can leave this property unset.
#ice.billing_accessExternalId=

# specify your custom tags here. Multiple tags are delimited by ",". If specified, BasicResourceService will be used to generate resource groups for you.
# PLEASE MAKE SURE you have limited number (e.g. < 100) of unique value combinations from your custom tags, otherwise Ice performance will be greatly affected.
#ice.customTags=tag1,tag2

# start date in millis from when you want to start processing the billing files
ice.startmillis=1364774400000

# you company name. it will be used by UI
ice.companyName=Opower

# s3 bucket name where Ice can store output files. Ice must have read and write access to billing s3 bucket.
ice.work_s3bucketname=paolo-output
# prefix of Ice output files
ice.work_s3bucketprefix=ice/

# local directory for Ice processor. the directory must exist.
ice.processor.localDir=~/ice_processor

# local directory for Ice reader. the directory must exist.
ice.reader.localDir=~/ice_reader

# monthly data cache size for Ice reader.
ice.monthlycachesize=12

# change the follow account settings
# ice.account.account1=123456789011
# ice.account.account2=123456789012
# ice.account.account3=123456789013
# ice.account.account4=123456789014
# ice.account.account5=123456789015
# ice.account.account6=123456789016

# set reservation owner accounts. "ice.owneraccount.account2=account3,account4" means reservations in account2 can be shared by account3 and account4
# if reservation capacity poller is enabled, the poller will try to poll reservation capacity through ec2 API (desribeReservedInstances) for each reservation owner account.
ice.owneraccount.account1=
ice.owneraccount.account2=account3,account4
ice.owneraccount.account5=account6

# if reservation capacity poller needs to use IAM role to access ec2 API, set the assumed role here for each reservation owner account
# ice.owneraccount.account1.role=ice
# ice.owneraccount.account2.role=ice
# ice.owneraccount.account5.role=ice

# if reservation capacity poller needs to use IAM role to access ec2 API and external id is used, set the external id here for each reservation owner account. otherwise you can leave it unset.
# ice.owneraccount.account1.externalId=
# ice.owneraccount.account2.externalId=
# ice.owneraccount.account5.externalId=

Hi,

I've activated the programmatic access on my billing preferences about a week ago, following your instructions.
On my billing preferences I have activated, Programmatic Access, Detailed Billing Report and Detailed Billing Report (with resources and tags).

But so far on my S3 bucket I only get the reports xxxxxxxxx--aws-billing-csv--.csv, not any file following the xxxxxxx-aws-billing-detailed-line-items--.csv.zip naming standard.

Do I need to wait a bit longer for AWS to generate the first file?

Can we have an example xxxxxxx-aws-billing-detailed-line-items--.csv.zip included on the application package to be used for testing and validate if the installation is working?

Thanks

I've got a two node cluster set up with an ice processor and ice reader running with no issues. The only thing that doesn't seem to be working properly is the ice reader displaying new data as it comes in. I can verify that the processor is running every hour and see the processor processing the latest detailed billing file for the month, updating it's local files and then uploading the updated processed files to the s3 work bucket.

I can also verify that the reader runs INFO basic.BasicManagers - trying to find new tag group and data managers... about every 5 minutes. The issue I am seeing is that the reader never seems to pick up and process the new data in the work bucket and when I leave ice running overnight I notice that I don't see the past 10 or 12 hours of data in the UI until I restart the ice reader process in tomcat.

Any ideas what could be causing this?

I think it would be very helpful to create tagged releases of ice and to have those releases go through some CI tool like jenkins or travisCI resulting in a war with all dependencies included. Would be great to then have these war's available from S3. I'm currently working on a chef cookbook that will install the following:

java
tomcat
nginx

Once all that is setup the only thing that's left to do is retrieve ice.war.x.x.x from an S3 bucket or nexus repository and deploy and configure the ice.properties file via chef attributes.

Anyway it's just a thought to make deployment much easier. Preconfigured AMI's might be a good alternative as well.

How can we password protect the ice dashboard?

Hi, followed the instructions and was able to run ICE processor with no apparent issues, I see files appear in the S3 bucket, etc. However, when I try to start the reader (on the same server, after switching roles in ice.properties), I get the below error, and I'm not sure what I could be doing wrong:

./grailsw -Dice.s3AccessKeyId=* -Dice.s3SecretKey=* run-app
| Running Grails application
2013-08-24 10:09:50,172 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]
2013-08-24 10:09:50,208 [main] INFO core.StandardService - Starting service Tomcat
2013-08-24 10:09:50,209 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30
2013-08-24 10:09:50,396 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found
2013-08-24 10:09:50,497 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext
2013-08-24 10:09:57,712 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@33ebee7c: startup date [Sat Aug 24 10:09:56 BST 2013]; parent: Root WebApplicationContext
2013-08-24 10:10:01,345 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-08-24 10:10:01,426 [localhost-startStop-1] INFO BootStrap - Starting ice...
2013-08-24 10:10:03,154 [localhost-startStop-1] INFO basic.BasicManagers - trying to find new tag group and data managers...
| Error com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler)
| Error at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:585)
| Error at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:319)
| Error at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
| Error at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979)
| Error at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2950)
| Error at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
| Error at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)
| Error at com.netflix.ice.basic.BasicManagers.doWork(BasicManagers.java:94)
| Error at com.netflix.ice.basic.BasicManagers.init(BasicManagers.java:59)
| Error at com.netflix.ice.reader.ReaderConfig.(ReaderConfig.java:100)
| Error at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
| Error at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
| Error at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
| Error at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
| Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrConstructorNewInstance(ReflectiveInterceptor.java:975)
| Error at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
| Error at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
| Error at BootStrap$_closure1.doCall(BootStrap.groovy:202)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| Error at java.lang.reflect.Method.invoke(Method.java:606)
| Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1243)
| Error at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
| Error at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)
| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:956)
| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)
| Error at groovy.lang.Closure.call(Closure.java:412)
| Error at groovy.lang.Closure.call(Closure.java:406)
| Error at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:308)
| Error at grails.util.Environment.executeForEnvironment(Environment.java:301)
| Error at grails.util.Environment.executeForCurrentEnvironment(Environment.java:277)
| Error at org.codehaus.groovy.grails.commons.DefaultGrailsBootstrapClass.callInit(DefaultGrailsBootstrapClass.java:60)
| Error at org.codehaus.groovy.grails.web.context.GrailsConfigUtils.executeGrailsBootstraps(GrailsConfigUtils.java:74)
| Error at org.codehaus.groovy.grails.web.context.GrailsContextLoader.initWebApplicationContext(GrailsContextLoader.java:108)
| Error at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
| Error at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
| Error at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
| Error at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
| Error at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
| Error at java.util.concurrent.FutureTask.run(FutureTask.java:166)
| Error at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
| Error at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
| Error at java.lang.Thread.run(Thread.java:724)
| Error Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
| Error at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:148)
| Error at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:277)
| Error at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:75)
| Error at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:72)
| Error at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
| Error at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
| Error at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:562)
| Error ... 51 more
| Error Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.
| Error at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
| Error at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
| Error at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
| Error at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
| Error at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1388)
| Error at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1018)
| Error at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
| Error at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
| Error at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489)
| Error at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
| Error at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
| Error at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
| Error at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
| Error at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:139)
| Error ... 57 more
| Error 2013-08-24 10:10:04,362 [localhost-startStop-1] ERROR BootStrap - Startup failed
Message: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler)
Line | Method
->> 585 | handleResponse in com.amazonaws.http.AmazonHttpClient

| 319 | executeHelper in ''
| 190 | execute . . . . . . . . . . . . in ''
| 2979 | invoke in com.amazonaws.services.s3.AmazonS3Client
| 2950 | invoke . . . . . . . . . . . . . in ''
| 478 | listObjects in ''
| 462 | listObjects . . . . . . . . . . in ''
| 94 | doWork in com.netflix.ice.basic.BasicManagers
| 59 | init . . . . . . . . . . . . . . in ''
| 100 | in com.netflix.ice.reader.ReaderConfig
| 202 | doCall . . . . . . . . . . . . . in BootStrap$_closure1
| 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment
| 301 | executeForEnvironment . . . . . in ''
| 277 | executeForCurrentEnvironment in ''
| 334 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync
| 166 | run in java.util.concurrent.FutureTask
| 1145 | runWorker . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor
| 615 | run in java.util.concurrent.ThreadPoolExecutor$Worker
^ 724 | run . . . . . . . . . . . . . . in java.lang.Thread

Caused by AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
->> 148 | parseXmlInputStream in com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser

| 277 | parseListBucketObjectsResponse in ''
| 75 | unmarshall . . . . . . . . . . . in com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller
| 72 | unmarshall in ''
| 62 | handle . . . . . . . . . . . . . in com.amazonaws.services.s3.internal.S3XmlResponseHandler
| 31 | handle in ''
| 562 | handleResponse . . . . . . . . . in com.amazonaws.http.AmazonHttpClient
| 319 | executeHelper in ''
| 190 | execute . . . . . . . . . . . . in ''
| 2979 | invoke in com.amazonaws.services.s3.AmazonS3Client
| 2950 | invoke . . . . . . . . . . . . . in ''
| 478 | listObjects in ''
| 462 | listObjects . . . . . . . . . . in ''
| 94 | doWork in com.netflix.ice.basic.BasicManagers
| 59 | init . . . . . . . . . . . . . . in ''
| 100 | in com.netflix.ice.reader.ReaderConfig
| 202 | doCall . . . . . . . . . . . . . in BootStrap$_closure1
| 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment
| 301 | executeForEnvironment . . . . . in ''
| 277 | executeForCurrentEnvironment in ''
| 334 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync
| 166 | run in java.util.concurrent.FutureTask
| 1145 | runWorker . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor
| 615 | run in java.util.concurrent.ThreadPoolExecutor$Worker
^ 724 | run . . . . . . . . . . . . . . in java.lang.Thread

Caused by SAXParseException: Premature end of file.
->> 198 | createSAXParseException in com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper

| 177 | fatalError in ''
| 441 | reportError . . . . . . . . . . in com.sun.org.apache.xerces.internal.impl.XMLErrorReporter
| 368 | reportError in ''
| 1388 | reportFatalError . . . . . . . . in com.sun.org.apache.xerces.internal.impl.XMLScanner
| 1018 | next in com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver
| 607 | next . . . . . . . . . . . . . . in com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl
| 116 | next in com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl
| 489 | scanDocument . . . . . . . . . . in com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl
| 835 | parse in com.sun.org.apache.xerces.internal.parsers.XML11Configuration
| 764 | parse . . . . . . . . . . . . . in ''
| 123 | parse in com.sun.org.apache.xerces.internal.parsers.XMLParser
| 1210 | parse . . . . . . . . . . . . . in com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser
| 139 | parseXmlInputStream in com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser
| 277 | parseListBucketObjectsResponse . in ''
| 75 | unmarshall in com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller
| 72 | unmarshall . . . . . . . . . . . in ''
| 62 | handle in com.amazonaws.services.s3.internal.S3XmlResponseHandler
| 31 | handle . . . . . . . . . . . . . in ''
| 562 | handleResponse in com.amazonaws.http.AmazonHttpClient
| 319 | executeHelper . . . . . . . . . in ''
| 190 | execute in ''
| 2979 | invoke . . . . . . . . . . . . . in com.amazonaws.services.s3.AmazonS3Client
| 2950 | invoke in ''
| 478 | listObjects . . . . . . . . . . in ''
| 462 | listObjects in ''
| 94 | doWork . . . . . . . . . . . . . in com.netflix.ice.basic.BasicManagers
| 59 | init in ''
| 100 | . . . . . . . . . . . . . in com.netflix.ice.reader.ReaderConfig
| 202 | doCall in BootStrap$_closure1
| 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment
| 301 | executeForEnvironment in ''
| 277 | executeForCurrentEnvironment . . in ''
| 334 | innerRun in java.util.concurrent.FutureTask$Sync
| 166 | run . . . . . . . . . . . . . . in java.util.concurrent.FutureTask
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 724 | run in java.lang.Thread

on accounts with large data sets (ie. greater than 50 linked accounts) i am finding that erroneous data pops up. example.. in the month of August there are no charges in ap-southeast-2 on the bill yet ICE is telling me i have charges in that region.. which isn't possible!

Hi, we try to use ICE into our AWS account, but we only have 1 account to all our teams, and all the administration be handle by Ops team, but we have Tags into all the EC2 and other services to identify what product by the owner of this Instance, and will loved can filter the global budget into small groups using this tags...

Can be this possible? I think many small/medium teams how share the same account into many Teams can be use this feature.

BWT: Possible the request be similar to this #29 but I'm not sure if the same request.

Thanks in advance!

I understand that in order to use more granular categories using the detailed billing files that we'll have to provide some code to query based on my custom tags but are there any required tags needed for basic reporting to work?

Hi. I am not getting anything in my working bucket although the processor looks to have completed correctly:

| Running Grails application
2013-06-27 00:40:59,590 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]
2013-06-27 00:40:59,646 [main] INFO core.StandardService - Starting service Tomcat
2013-06-27 00:40:59,646 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30
2013-06-27 00:41:00,455 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found
2013-06-27 00:41:00,741 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext
2013-06-27 00:41:20,564 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@346969d3: startup date [Thu Jun 27 00:41:15 UTC 2013]; parent: Root WebApplicationContext
2013-06-27 00:41:29,617 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-06-27 00:41:29,759 [localhost-startStop-1] INFO BootStrap - Starting ice...
2013-06-27 00:41:31,826 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up...
2013-06-27 00:41:31,845 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started...
2013-06-27 00:41:31,872 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting...
2013-06-27 00:41:32,342 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring FrameworkServlet 'grails'
2013-06-27 00:41:32,963 [main] INFO http11.Http11Protocol - Starting ProtocolHandler ["http-bio-8080"]
| Server running. Browse to http://localhost:8080/ice
2013-06-27 00:41:37,415 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - AWS usage processed.

| Error 2013-08-14 15:22:56,025 [http-bio-8080-exec-10] ERROR errors.GrailsExceptionResolver  - IllegalArgumentException occurred when processing request: [POST] /ice/dashboard/getData
Invalid format: "2013-08-14 03PM" is malformed at "PM". Stacktrace follows:
Message: Invalid format: "2013-08-14 03PM" is malformed at "PM"
    Line | Method
->>  866 | parseDateTime in org.joda.time.format.DateTimeFormatter
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|    402 | doGetData     in com.netflix.ice.DashboardController
|    316 | doCall . . .  in com.netflix.ice.DashboardController$_closure11
|   1146 | runWorker     in java.util.concurrent.ThreadPoolExecutor
|    615 | run . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^    679 | run           in java.lang.Thread

After processing data from ice-processor and trying to use ice-reader, I got those errors from ice-reader.

I enabled Programmatic Access and Detailed Billing Report, which generated the necessary aws-billing-detailed-line-items-2013-06.csv.zip file. However, when Ice ran it threw tons of index out of bound errors and crashed.

All of the out of bound errors were for column 19, which looks like costIndex from the initIndexes method (withTags is false). I looked at the actual data file and it seems that the column headers would put costIndex at 17. However, there appear to be cells in the CSV data that include commas (for example, the ItemDescription field for metrics).

I know basically nothing about the Java CSV parser, so I don't know what the correct answer is there. However, since leaving the initIndexes as-is eventually bombs, I decided to see what happens if I rebase the last 3 columns to 17/18/19 (so costIndex = 17).

That removes the out of bounds errors, though I expect there is bad/weird data being generated due to ItemDescription rows that have commas in them - but no results are output since BillingFileProcessor then throws an error from ensureOpen in ZipInputStream (from line 605 in BillingFileProcessor).

So I'm stumped, grails 2.2.1 and I've tried running with both java 1.7 and openjdk.

You mention in the readme that if the bucket with the files to be processed are in a different owned account that you can use cross-account roles to get access.

could you update that to explain how to get ice/grails to call the STS service for the token and how to incorporate that into the ice processor?

How is the performance of processor and worker running on separared free tier m1.micro instances? Is is acceptable? :)

We'd find it useful to convert from $ to £.

We'd also find it useful to obfuscate the figures by introducing a made up currency to convert into.

I have this line in my reservation_capacity.txt

xxxx,ap-southeast-2,xxxx,ap-southeast-2a,xxxxx,31536000,m1.xlarge,Windows with SQL Server Standard (Amazon VPC),2,Heavy Utilization,active

I have two SQL instances running in that zone that match the RI yet ice is showing me that these are Unused. Perhaps it is getting confused with a windows only OS?

(I commented out the number after ap-southeast-2a because I wasn't sure what it referred to).

Other NetflixOSS projects use Cloudbees for building the master branch as well as PRs: For example:
https://netflixoss.ci.cloudbees.com/job/asgard-master/

I think Ice should be built in here as well. Apart from the advantage of knowing when PRs are good, it also makes it easier for others to start using Ice because Cloudbees can keep the latest successful build artifacts for the master. Being able to download the successfully built WAR makes getting started with Ice much easier.

Also, where is the mailing list for Ice? I don't see it listed anywhere like other projects... e.g.
https://groups.google.com/forum/#!forum/eureka_netflix

Thanks.

When I try to build the project, I get:

/home/ubuntu/ice-master/build.xml:22: Cannot find /home/ubuntu/ice-master/${env.WORKSPACE}/Tools/build/properties.xml imported from /home/ubuntu/ice-master/build.xml

If I mark that as optional, the build then fails on:

/home/ubuntu/ice-master/build.xml:35: Cannot find /home/ubuntu/ice-master/${build.files}/webapplication.xml imported from /home/ubuntu/ice-master/build.xml

Marking that optional eventually yields:

Target "module.resolve" does not exist in the project "ice". It is used from target "setup".

Am I overlooking something in setting up the project? Any suggestions where I can find these files?

So I've had quite a harrowing process getting Ice loaded, but it seems like I'm almost there.

Currently I'm receiving the following when a user tries to navigate to http://IPADDRESS:8080/ice/dashboard/summary

| Error 2013-07-19 17:45:47,185 [http-bio-8080-exec-4] ERROR errors.GrailsExceptionResolver  - NoSuchElementException occurred when processing request: [POST] /ice/dashboard/getData
Stacktrace follows:
Message: null
    Line | Method
->> 1115 | nextEntry in java.util.TreeMap$PrivateEntryIterator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   1162 | next      in java.util.TreeMap$ValueIterator
|    627 | doGetData in com.netflix.ice.DashboardController
|    316 | doCall    in com.netflix.ice.DashboardController$_closure11
|   1146 | runWorker in java.util.concurrent.ThreadPoolExecutor
|    615 | run       in java.util.concurrent.ThreadPoolExecutor$Worker
^    679 | run . . . in java.lang.Thread

The data appears to load at the top, with the account numbers, regions, etc. The fields are populated, but on the bar immediately below, the little spinning circle gif is all that shows.

I have tried deleting the contents of the s3 working bucket and both local working directories for the processor and reader, then setting ice.processor=true and ice.reader=false, and running ./grailsw run-app again from the project root directory. Once it says AWS Usage has been processed I Ctrl-C, swap the trues/falses in ice.properties, and rerun ./grailsw run-app. Is that what is intended?

How we can able to enable "breakdown" tab in ice dashboard, I can see in readme about this, but I didn't understand what mentioned there, Can anyone please describe much more about that functionality enable.

https://github.com/Netflix/ice#advanced-options - 6.Resource service (*)

At my company we have a custom provisioning tool very similar to Asgaard that basically provides the portal for all of our engineers to instance nodes in AWS and configure and deploy our software to those nodes. One of the things that this tool does for us is to tag ec2 and rds nodes with several custom tags but for this discussion let's say product and envirionment are the two important tags I'm looking to filter on.

I have implemented a version of the SampleMapDbResourceService replacing some of the logic with our custom logic to parse these tags and concatenate them "{product} - {environment}" to form a resource name. I am seeing results based on these tags for the resource filters but there are some discrepancies which I'm in the process of debugging.

Would you say this is the best way to go about organizing resources when you have custom tags? In SampleMapDbResourceService what purpose does the MapDb object serve? I see that it's getting populated and the base class commit's at the end of a processing run but how is it getting read back to the UI or is it?

When trying to run grailsw on Windows I receive the following error:

Exception: java.lang.NumberFormatException thrown from the UncaughtExceptionHandler in thread "main"

I've run grails clean and removed the .grails folder from the user directory hoping that fixes it. The grails wrapper successfully completes. It just can't run the app. I'm also unable to desploy the war to Tomcat 7.

hello,

I have been using ice for some time and its been working fine. However I have recently moved to a new server. Since that move the billing data is not updating. It keeps on ignoring the files. The configuration is identical between builds but now I get the following logs..

2013-11-06 03:19:14,667 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721828-aws-billing-csv-2013-08.csv
2013-11-06 03:19:14,667 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721828-aws-billing-csv-2013-09.csv
2013-11-06 03:19:14,667 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721828-aws-billing-csv-2013-10.csv
2013-11-06 03:19:14,667 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-billing-csv-2013-11.csv
2013-11-06 03:19:14,670 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-billing-detailed-line-items-with-resources-and-tags-2013-08.csv.zip
2013-11-06 03:19:14,677 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-billing-detailed-line-items-with-resources-and-tags-2013-09.csv.zip
2013-11-06 03:19:14,677 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-billing-detailed-line-items-with-resources-and-tags-2013-10.csv.zip
2013-11-06 03:19:14,677 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-billing-detailed-line-items-with-resources-and-tags-2013-11.csv.zip
2013-11-06 03:19:14,678 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-cost-allocation-2013-08.csv
2013-11-06 03:19:14,678 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-cost-allocation-2013-09.csv
2013-11-06 03:19:14,678 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-cost-allocation-2013-10.csv
2013-11-06 03:19:14,678 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file 528086721821-aws-cost-allocation-2013-11.csv

It is a completely new install so I am unsure why it would be ignoring the files.

Unlike the crashed for unused AWS products, I'm not sure what the reason for this crash is, but at approximately line 432 there is the call ResourceGroup.getResourceGroups(appgroup.data.get(appgroupProduct.toString())) which is throwing a null pointer error for me.

I have mitigated this by initializing resourceGroupsOfProduct to null and then wrapping the above call in a try/catch with a continue. This seems to be working ok, but again, I don't know why this is erroring and I'm sure my "fix" is just a workaround.

I can't filter for Sydney. The Sydney usage is then added to the pool of costs which makes it difficult to get correct information on the other regions.