No output in working bucket about ice HOT 25 CLOSED

@ma78 It looks like BillingFileProcessor couldn't find any billing files to process. Please check your properties ice.billing_s3bucketname and ice.billing_s3bucketprefix have correct values.

from ice.

I'm getting this same issue and I've already double checked my properties. My billing bucket does not have another folder in it so I left prefix blank.

If my billing bucket contains billing files that are not in the two specified formats listed in the prerequisites, will that prevent ice from processing the correct files I have in the bucket? Should the correct files be moved to a folder within the billing bucket?

from ice.

@allisonmin billing bucket containing files in other format will not prevent ice from processing the correct files. Since many people have had this problem, I added more log info in BillingFileProcessor to help us debug it. Please get the latest code and run processor again and search for BillingFileProcessor logs showing how many files were found in billing bucket. You can search for string " in billing bucket".

from ice.

Where do I search for BillingFileProcessor logs?

from ice.

@allisonmin search for "BillingFileProcessor " and " in billing bucket" in your log file.

from ice.

I had the same issue. I was putting '/' for both the billing report bucket and ice work buckets, but nothing would happen. I restarted with those value empty and it immediately started processing the files.

from ice.

I will try the above suggestion to not use '/' as a prefix, but where is the log file?

from ice.

@rsmithHM Do you mean you were adding the '/' at the end of the bucket name or the bucket prefix?

ice.billing_s3bucketname=example-name/
ice.billing_s3bucketprefix=
ice.work_s3bucketname=ice_working/
ice.work_s3bucketprefix=

from ice.

I was doing this:
ice.billing_s3bucketname=example-name
ice.billing_s3bucketprefix=/
ice.work_s3bucketname=ice_working
ice.work_s3bucketprefix=/

which didn't work.

this does:

ice.billing_s3bucketname=example-name
ice.billing_s3bucketprefix=
ice.work_s3bucketname=ice_working
ice.work_s3bucketprefix=

Once I made that change, the next time I started ice, I got some pretty quick feedback on stdout where it was processing my files in ice.blling.

from ice.

it is just ignoring the files in the bucket...

2013-07-01 05:40:31,301 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - found 9 in billing bucket...
2013-07-01 05:40:31,302 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxx-aws-billing-csv-2013-04.csv
2013-07-01 05:40:31,302 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxx-aws-billing-csv-2013-05.csv
2013-07-01 05:40:31,302 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxx-aws-billing-csv-2013-06.csv
2013-07-01 05:40:31,303 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxx-aws-billing-csv-2013-07.csv
2013-07-01 05:40:31,303 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-06.csv.zip
2013-07-01 05:40:31,303 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxx-aws-cost-allocation-2013-04.csv
2013-07-01 05:40:31,304 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxx-aws-cost-allocation-2013-05.csv
2013-07-01 05:40:31,304 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxx-aws-cost-allocation-2013-06.csv
2013-07-01 05:40:31,304 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file aws-programmatic-access-test-object
2013-07-01 05:40:31,305 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - AWS usage processed.

Are there additional logs that are viewable?

from ice.

I'm still not getting any output or BillingFileProcessor logs.

2013-07-01 13:33:05,068 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]
2013-07-01 13:33:05,161 [main] INFO core.StandardService - Starting service Tomcat
2013-07-01 13:33:05,162 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30
2013-07-01 13:33:05,329 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found
2013-07-01 13:33:05,466 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext
2013-07-01 13:33:11,513 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@10b09f58: startup date [Mon Jul 01 13:33:10 UTC 2013]; parent: Root WebApplicationContext
2013-07-01 13:33:14,030 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-07-01 13:33:14,156 [localhost-startStop-1] INFO BootStrap - Starting ice...
2013-07-01 13:33:15,284 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up...
2013-07-01 13:33:15,294 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started...
2013-07-01 13:33:15,294 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting...
2013-07-01 13:33:15,366 [localhost-startStop-1] INFO basic.BasicManagers - trying to find new tag group and data managers...
2013-07-01 13:33:16,974 [localhost-startStop-1] INFO basic.BasicManagers - poller thread for com.netflix.ice.basic.BasicManagers started...
2013-07-01 13:33:16,982 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - poller starting...
2013-07-01 13:33:16,982 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers...
2013-07-01 13:33:17,044 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring FrameworkServlet 'grails'
2013-07-01 13:33:17,158 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - AWS usage processed.
2013-07-01 13:33:17,433 [main] INFO http11.Http11Protocol - Starting ProtocolHandler ["http-bio-8080"]

from ice.

@ma78 You have one detailed billing file with resources and tags in the billing bucket (xxxxxxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-06.csv.zip). This billing file is getting ignored because you didn't subclass ResourceService (see "6. Resource service" in Advanced Options in readme). There are 2 ways to solve this issue. 1. If this is your first time running Ice, I suggest you enable detailed billing file with Amazon. The detailed billing file looks like xxxxxxxxxxxx-aws-billing-detailed-line-items-2013-06.csv.zip. 2. You can subclass ResourceService following the instruction in "6. Resource service" in Advanced Options in readme.

I will also update the readme and clarify this.

@allisonmin Please pull the latest code, run again and post your log. The new code will print out more info to help debugging your problem.

from ice.

I've pulled the latest code and I'm getting the same log as the one I posted before.

from ice.

@allisonmin with the latest code, you should have something like "found 9 in billing bucket..." in the log like shown in @ma78's logs. Please double check your are running the latest code. Also are your billing files located directly under your billing s3 bucket? If so, your ice.billing_s3bucketprefix should be:
ice.billing_s3bucketprefix=

from ice.

I'm getting the same logs as @ma78 now. Thanks.

from ice.

Where can I find the the SampleMapDbResourceService mentioned in "6. Resource Service" ?

from ice.

@allisonmin https://github.com/Netflix/ice/blob/master/src/java/com/netflix/ice/basic/SampleMapDbResourceService.java

from ice.

I'd like to request a simple tickbox somewhere to decide whether the detailed vs detailed-with-tags-and-resources gets used :)

from ice.

@ma78 I change the code so that, if only billing files with tags and resources are available, even if resource service is not present, processor will still process those billing files.

Let me know if it works for you.

from ice.

No dice..

| Server running. Browse to http://localhost:8080/ice
2013-07-02 07:09:56,685 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - found 10 in billing bucket...
2013-07-02 07:09:56,686 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxx-aws-billing-csv-2013-04.csv
2013-07-02 07:09:56,686 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxx-aws-billing-csv-2013-05.csv
2013-07-02 07:09:56,686 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxx-aws-billing-csv-2013-06.csv
2013-07-02 07:09:56,687 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxx-aws-billing-csv-2013-07.csv
2013-07-02 07:09:56,695 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxx-aws-cost-allocation-2013-04.csv
2013-07-02 07:09:56,695 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxx-aws-cost-allocation-2013-05.csv
2013-07-02 07:09:56,695 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxx-aws-cost-allocation-2013-06.csv
2013-07-02 07:09:56,696 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxx-aws-cost-allocation-2013-07.csv
2013-07-02 07:09:56,696 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file aws-programmatic-access-test-object
| Error 2013-07-02 07:09:56,869 [com.netflix.ice.processor.BillingFileProcessor] ERROR common.AwsUtils - failed to find usage_hourly_all_2013-06
2013-07-02 07:09:56,870 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - trying to download xxxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-06.csv.zip...
2013-07-02 07:10:01,979 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - downloaded xxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-06.csv.zip
2013-07-02 07:10:01,980 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - processing xxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-06.csv.zip...
2013-07-02 07:10:12,806 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - reached end of file.
2013-07-02 07:10:12,807 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - done processing xxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-06.csv.zip
| Error 2013-07-02 07:10:13,475 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling

java.lang.NullPointerException
at com.netflix.ice.processor.BillingFileProcessor.poll(BillingFileProcessor.java:173)
at com.netflix.ice.common.Poller.doWork(Poller.java:50)
at com.netflix.ice.common.Poller.access$000(Poller.java:28)
at com.netflix.ice.common.Poller$1.run(Poller.java:88)
at java.lang.Thread.run(Thread.java:724)

It might also be worth noting that I had to change my ice.properties file to not use /mnt but rather /home/me/ice.processor otherwise it wouldn't download the file. I also had to create the folder as well before it would successfully download. The readme doesn't mention this directory has to exist.

from ice.

I'm getting the same polling error.

| Server running. Browse to http://localhost:8080/ice
2013-07-02 12:56:33,901 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - found 10 in billing bucket...
2013-07-02 12:56:33,902 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxxx-aws-billing-csv-2013-05.csv
2013-07-02 12:56:33,902 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxxx-aws-billing-csv-2013-06.csv
2013-07-02 12:56:33,902 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxxx-aws-billing-csv-2013-07.csv
2013-07-02 12:56:33,927 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxxx-aws-cost-allocation-2013-05.csv
2013-07-02 12:56:33,927 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxxx-aws-cost-allocation-2013-06.csv
2013-07-02 12:56:33,927 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file xxxxxxxxxxxx-aws-cost-allocation-2013-07.csv
2013-07-02 12:56:33,927 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - ignoring file aws-programmatic-access-test-object
| Error 2013-07-02 12:56:34,109 [com.netflix.ice.processor.BillingFileProcessor] ERROR common.AwsUtils - failed to find usage_hourly_all_2013-05
2013-07-02 12:56:34,109 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - trying to download xxxxxxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-05.csv.zip...
2013-07-02 12:56:35,832 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - downloaded xxxxxxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-05.csv.zip
2013-07-02 12:56:35,832 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - processing xxxxxxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-05.csv.zip...
2013-07-02 12:56:50,003 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - processed 500000 lines...
2013-07-02 12:57:01,760 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - processed 1000000 lines...
2013-07-02 12:57:04,133 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - reached end of file.
2013-07-02 12:57:04,134 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - done processing xxxxxxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2013-05.csv.zip
| Error 2013-07-02 12:57:06,009 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling
Message: null
Line | Method
->> 173 | poll in com.netflix.ice.processor.BillingFileProcessor

| 50 | doWork in com.netflix.ice.common.Poller
| 28 | access$000 in ''
| 88 | run in com.netflix.ice.common.Poller$1
^ 722 | run . . . in java.lang.Thread

from ice.

@ma78 @allisonmin I over looked one place. Please pull the latest and try again.

from ice.

Thanks @fangji, it works!

from ice.

Well i pulled the new code, and also enabled detailed billing (just plain, without resources-and-tags) on my account and it is now working!

What will ice do if it sees both the detailed and the detailed-with-resources-and-tags file in there? Which will it process?

from ice.

@ma78 if both the detailed and the detailed-with-resources-and-tags files are present, it depends on if you subclassed and used ResourceService, if yes, detailed-with-resources-and-tags file will be used, if not, detailed file will be used.

from ice.