Comments (9)
fangji
commented on June 12, 2024
ReservationCapacityPoller.java is responsible for generating reservation_capacity.txt. If you set ice.reservationCapacityPoller=true in ice.properties, BootStrap.groovy will start a ReservationCapacityPoller instance. You also need to specify reservation owner accounts in ice.properties, please see 2.2. and 2.3 in Advanced Options.
from ice.
rseaman
commented on June 12, 2024
@fangji , @dwinter3 and I are working together on this issue.
Currently, this is what we're seeing.
2013-07-24 21:21:48,715 [localhost-startStop-1] INFO processor.ReservationCapacityPoller - poller thread for com.netflix.ice.processor.ReservationCapacityPoller started...
2013-07-24 21:21:48,716 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - poller starting...
2013-07-24 21:21:48,718 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - downloading /var/ice_processor/reservation_capacity.txt...
2013-07-24 21:21:53,205 [com.netflix.ice.processor.ReservationCapacityPoller] INFO common.AwsUtils - file not found in s3 /var/ice_processor/reservation_capacity.txt
2013-07-24 21:21:53,205 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - downloaded /var/ice_processor/reservation_capacity.txt
2013-07-24 21:21:53,205 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - read 0 reservations.
| Error 2013-07-24 21:22:01,162 [com.netflix.ice.processor.ReservationCapacityPoller] ERROR processor.ReservationCapacityPoller - Error polling
Message: null
Line | Method
->> 595 | updateEc2Reservations in com.netflix.ice.basic.BasicReservationService
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 178 | poll in com.netflix.ice.processor.ReservationCapacityPoller
| 50 | doWork . . . . . . . in com.netflix.ice.common.Poller
| 28 | access$000 in ''
| 88 | run . . . . . . . . . in com.netflix.ice.common.Poller$1
^ 679 | run in java.lang.Thread
Here is our current ice.properties (some redacted info):
# whether or not to start processor
ice.processor=true
# whether or not to start reader/UI
ice.reader=false
# whether or not to start reservation capacity poller
ice.reservationCapacityPoller=true
# reservation period, possible values are oneyear, threeyear
ice.reservationPeriod=oneyear
# reservation utilization, possible values are LIGHT, HEAVY
ice.reservationUtilization=HEAVY
# url prefix, e.g. http://ice.netflix.com/. Will be used in alert emails.
ice.urlPrefix=
# from email address
ice.fromEmail=
# ec2 ondemand hourly cost threshold to send alert email. The alert email will be sent at most once per day.
ice.ondemandCostAlertThreshold=250
# ec2 ondemand hourly cost alert emails, separated by ","
ice.ondemandCostAlertEmails=
# modify the following 5 properties according to your billing files configuration. if you have multiple payer accounts, you will need to specify multiple values for each property.
# s3 bucket name where the billing files are. multiple bucket names are delimited by ",". Ice must have read access to billing s3 bucket.
ice.billing_s3bucketname=s3billingbucketname
# prefix of the billing files. multiple prefixes are delimited by ","
ice.billing_s3bucketprefix=
# specify your payer account id here if across-accounts IAM role access is used. multiple account ids are delimited by ",". "ice.billing_payerAccountId=,222222222222" means assumed role access is only used for the second bucket.
ice.billing_payerAccountId=123123123123
# specify the assumed role name here if you use IAM role access to read from billing s3 bucket. multiple role names are delimited by ",". "ice.billing_accessRoleName=,ice" means assumed role access is only used for the second bucket.
ice.billing_accessRoleName=ICE-ICE-BABY
# specify external id here if it is used. multiple external ids are delimited by ",". if you don't use external id, you can leave this property unset.
ice.billing_accessExternalId=
# start date in millis from when you want to start processing the billing files
ice.startmillis=0
# you company name. it will be used by UI
ice.companyName=ZaaS
# s3 bucket name where Ice can store output files. Ice must have read and write access to billing s3 bucket.
ice.work_s3bucketname=s3workbucketname
# prefix of Ice output files
ice.work_s3bucketprefix=
# local directory for Ice processor. the directory must exist.
ice.processor.localDir=/var/ice_processor
# local directory for Ice reader. the directory must exist.
ice.reader.localDir=/var/ice_reader
# monthly data cache size for Ice reader.
ice.monthlycachesize=12
# change the follow account settings
ice.account.ZaaSProd=123123123123
ice.account.MBA=123123123123
ice.account.ZaaSDev=123123123123
ice.account.Platform=123123123123
ice.account.QA=123123123123
ice.account.Solutions=123123123123
ice.account.ITNew=123123123123
ice.account.ITOld=123123123123
ice.account.Community=123123123123
ice.account.Training=123123123123
# set reservation owner accounts. "ice.owneraccount.account2=account3,account4" means reservations in account2 can be shared by account3 and account4
# if reservation capacity poller is enabled, the poller will try to poll reservation capacity through ec2 API (desribeReservedInstances) for each reservation owner account.
ice.owneraccount.ZaaSProd=ZaaSProd
#ice.owneraccount.account2=
#ice.owneraccount.account5=
# if reservation capacity poller needs to use IAM role to access ec2 API, set the assumed role here for each reservation owner account
ice.owneraccount.ZaaSProd.role=ICE-ICE-BABY
#ice.owneraccount.account2.role=
#ice.owneraccount.account5.role=
# if reservation capacity poller needs to use IAM role to access ec2 API and external id is used, set the external id here for each reservation owner account. otherwise you can leave it unset.
#ice.owneraccount.account1.externalId=
#ice.owneraccount.account2.externalId=
#ice.owneraccount.account5.externalId=
from ice.
rseaman
commented on June 12, 2024
By the way, we're using VPC reservations. Does Ice handle those?
from ice.
fangji
commented on June 12, 2024
@rseaman @dwinter3
Strange thing is that there's no line 595 com.netflix.ice.basic.BasicReservationService... Did you make some changes to that file? If you can post the code around where the exception happened, it will help me debug.
BTW, you should not put the owner account at right side: ice.owneraccount.ZaaSProd=
from ice.
rseaman
commented on June 12, 2024
@fangji Ok, corrected that, I'm still not entirely clear how ice.owneraccount.xxxx= needs to be laid out, is it only 'ice.owneraccount.ZaaSProd=' ?
The reason that higher line number popped up is because we were using an older commit. I have updated to the most current code in this git repo but we're still seeing similar issues. I do not see a reservation_capacity.txt anywhere, the working s3 bucket, /var/ice_processor, /var/ice_reader, etc. I do now see a reservation_prices.oneyear.HEAVY, and it is clearly getting farther.
Here's the result of our current run:
| Running Grails application
2013-07-24 22:23:46,587 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]
2013-07-24 22:23:46,697 [main] INFO core.StandardService - Starting service Tomcat
2013-07-24 22:23:46,697 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30
2013-07-24 22:23:48,684 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found
2013-07-24 22:23:49,930 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext
2013-07-24 22:24:12,893 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@1afcfd10: startup date [Wed Jul 24 22:24:05 UTC 2013]; parent: Root WebApplicationContext
2013-07-24 22:24:21,108 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.
2013-07-24 22:24:21,496 [localhost-startStop-1] INFO BootStrap - Starting ice...
2013-07-24 22:24:23,323 [localhost-startStop-1] INFO BootStrap - Found role: ICE-ICE-BABY
2013-07-24 22:24:26,913 [localhost-startStop-1] INFO basic.BasicReservationService - poller thread for com.netflix.ice.basic.BasicReservationService started...
2013-07-24 22:24:26,918 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up...
2013-07-24 22:24:26,914 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - poller starting...
2013-07-24 22:24:26,920 [localhost-startStop-1] INFO processor.ReservationCapacityPoller - poller thread for com.netflix.ice.processor.ReservationCapacityPoller started...
2013-07-24 22:24:26,920 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - poller starting...
2013-07-24 22:24:26,920 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - downloading /var/ice_processor/reservation_capacity.txt...
2013-07-24 22:24:30,471 [com.netflix.ice.processor.ReservationCapacityPoller] INFO common.AwsUtils - file not found in s3 /var/ice_processor/reservation_capacity.txt
2013-07-24 22:24:30,474 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - downloaded /var/ice_processor/reservation_capacity.txt
2013-07-24 22:24:30,474 [com.netflix.ice.processor.ReservationCapacityPoller] INFO processor.ReservationCapacityPoller - read 0 reservations.
2013-07-24 22:24:36,919 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - start polling reservation prices...
2013-07-24 22:24:37,131 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.small in us-east-1: 169.0 0.014
2013-07-24 22:24:37,131 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.medium in us-east-1: 338.0 0.028
2013-07-24 22:24:37,132 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.large in us-east-1: 676.0 0.056
2013-07-24 22:24:37,135 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.xlarge in us-east-1: 1352.0 0.112
2013-07-24 22:24:37,136 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m3.xlarge in us-east-1: 1489.0 0.123
2013-07-24 22:24:37,136 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m3.2xlarge in us-east-1: 2978.0 0.246
[...]
2013-07-24 22:24:37,371 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.small.windows in sa-east-1: 372.94 0.05
2013-07-24 22:24:37,371 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.medium.windows in sa-east-1: 746.0 0.095
2013-07-24 22:24:37,371 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.large.windows in sa-east-1: 1492.0 0.19
2013-07-24 22:24:37,371 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m1.xlarge.windows in sa-east-1: 2984.0 0.38
2013-07-24 22:24:37,371 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m3.xlarge.windows in sa-east-1: 3282.0 0.402
2013-07-24 22:24:37,372 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m3.2xlarge.windows in sa-east-1: 6564.0 0.805
2013-07-24 22:24:37,372 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for t1.micro.windows in sa-east-1: 84.0 0.013
2013-07-24 22:24:37,372 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m2.xlarge.windows in sa-east-1: 2160.0 0.223
2013-07-24 22:24:37,372 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m2.2xlarge.windows in sa-east-1: 4320.0 0.445
2013-07-24 22:24:37,373 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for m2.4xlarge.windows in sa-east-1: 8640.0 0.89
2013-07-24 22:24:37,373 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for c1.medium.windows in sa-east-1: 747.0 0.12
2013-07-24 22:24:37,373 [com.netflix.ice.basic.BasicReservationService] INFO basic.BasicReservationService - exisitng reservation price for c1.xlarge.windows in sa-east-1: 2984.0 0.48
| Error 2013-07-24 22:24:40,155 [com.netflix.ice.processor.ReservationCapacityPoller] ERROR processor.ReservationCapacityPoller - Error polling
Message: null
Line | Method
->> 379 | updateEc2Reservations in com.netflix.ice.basic.BasicReservationService
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 178 | poll in com.netflix.ice.processor.ReservationCapacityPoller
| 50 | doWork . . . . . . . in com.netflix.ice.common.Poller
| 28 | access$000 in ''
| 88 | run . . . . . . . . . in com.netflix.ice.common.Poller$1
^ 679 | run in java.lang.Thread
Here is our ice.properties (redacting repeated or seemingly irrelevant info):
# whether or not to start processor
ice.processor=true
# whether or not to start reader/UI
ice.reader=true
# whether or not to start reservation capacity poller
ice.reservationCapacityPoller=true
# set reservation owner accounts. "ice.owneraccount.account2=account3,account4" means reservations in account2 can be shared by account3 and account4
# if reservation capacity poller is enabled, the poller will try to poll reservation capacity through ec2 API (desribeReservedInstances) for each reservation owner account.
ice.owneraccount.ZaaSProd=
ice.owneraccount.ZaaSProd.role=ICE-ICE-BABY
from ice.
fangji
commented on June 12, 2024
It looks like you have reserved instance in a zone not recognizable by Ice... In method BasicReservationService.updateEc2Reservations, line 366, can you add some debug code and see what the output is?
logger.info("reserved instance zone=" + reservedInstances.getAvailabilityZone();
from ice.
rseaman
commented on June 12, 2024
@fangji Netflix doesn't use VPCs, correct? We do, our reservations are in VPCs. Could that be the source of this issue?
Thank you for all your help so far, by the way.
I'll try to get some debugging in.
from ice.
fangji
commented on June 12, 2024
we don't have reserved instances in vpc. But it looks like the endpoints for vpc and ec2 are the same and it looks Ice is able to get list of reserved instances, so let's see what the debug info says.
from ice.
rseaman
commented on June 12, 2024
@fangji I pulled the newest code and refreshed our s3 bucket/ice_reader/ice_processor. It's working just fine now! Thanks for all your help with this.
from ice.
Related Issues (20)
- ca-central-1 not shwoing up in UI HOT 3
- What is ice reports Timezone? HOT 2
- Project Status HOT 3
- ERROR basic.BasicReservationService - No Zone for ap-south-1c [doesn't actually exist] HOT 3
- JimRoth fork HOT 4
- Redshift RI's breaking ec2_instance costs
- Since the 12th of July 2019, EC2 reserved instances billing stopped working... HOT 6
- Question about Using AWS S3 in region "cn-northwest-1"(China) HOT 4
- ICE dashborad show wrong price HOT 2
- how to set Reserved Instance's Region? HOT 1
- How to set RI region?
- Can anyone explain what is "reservationUtilization=HEAVY" and FIXED? HOT 1
- Do ice support https? HOT 1
- How to change default listen url: http://localhost:8080/${appName}? HOT 2
- What does Usage mean on ice dashboard? HOT 3
- Any plan migrate from Detailed Billing Reports to Cost and Usage Reports? HOT 3
- ProcessorConfig.getAccountsFromOrganizations error in jimroth/ice fork
- Failed to parse billing file HOT 3
- ERROR BootStrap - Startup failed when running docker-compose up
- load AWS credentials from any provider in the chain HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ice.