Question regarding reserved instances and ice wildly over-reporting costs about ice HOT 9 OPEN

@rayrod2030 Can you help me find out the cause of this? First, can you compare rds, ec2 and elasticache cost between Ice and your standard billing files and let me know the difference? If you see a cost difference can you also find out the usage difference?

Thanks
-Fang

from ice.

@fangji These are the numbers I've analyzed for the last 3 months for EC2 (ec2, ec2_instance, ebs, eip, cloudwatch), RDS and Elasticache in ice and our aws billing as reported in our primary billing account. As you can see we made some reserved purchases in august and that seems to have effected EC2 ice costs the most. Appreciate your help with this.

from ice.

@rayrod2030 Thanks for providing the data.

For the ec2 cost, I assume it includes both ec2 and ec2_instance? Is the difference mainly in ec2_instance?

Does the ec2 aws billing include amortized upfront cost?

If the difference is mainly in ec2_instance, can you confirm the ec2_instance usage hours still match between Ice and the billing?

Thanks
-Fang

from ice.

@fangji For my rollups in my spreadsheet I combined ec2_instance, ec2, ebs, cloudwatch and eip ice resources as those are the resources that AWS combines under the "Amazon Elastic Compute Cloud" line item. I used those combined totals for comparison.

Our AWS billing summaries in our main payment account does show upfront costs for reserved purchases for August and September as well as showing 0.00 cost line items for our reserved instance usage for the last two months. I suspect if I were to calculate the usage using the standard on-demand pricing that it should make up for the large difference in ec2_instance costs in ice.

I can confirm that since august our ec2 costs within ice have been pretty stable and all increases we have seen in ice are in the ec2_instance resource.

If anything our ec2_instance usage is slightly higher in our aws billing summary but not by that much.

So am I basically seeing a discrepancy due to the fact that ice is processing my reserved instance usage at the on-demand price?

from ice.

@rayrod2030 Do you have ice.reservationCapacityPoller=true in ice.properties file? In reservations page on UI, can you see if the number of reserved instances match with what you have?

from ice.

@fangji Currently we are not running reservationCapacityPoller in production as I was not sure if it was needed to report proper reservation pricing or if it enabling it would just give you move visibility into how many of your reserve instance capacity was being used. I'm assuming now that if we don't have this enabled we won't be seeing proper costs based on reserved instance pricing?

from ice.

@rayrod2030 If you are not running reservationCapacityPoller, then Ice should show reserved&ondemand instance usage&cost as it is in billing file.

Did you define reservation owner accounts? Do you have properties like ice.owneraccount.* in your ice.properties file and what values do those properties have?

from ice.

@fangji I believe I do have reservation owner accounts defined. Here is what my ice properties file looks like for our processor. I've edited it to scrub any real account information.

#
## Ice Configuration File
#
## Dynamically generated by Chef
##
## Local modifications will be overwritten by Chef.
##

ice.processor=true
ice.processor.localDir=/var/ice_processor

ice.billing_s3bucketname=billing-stuff
ice.billing_s3bucketprefix=

ice.reader=false
ice.reader.localDir=/var/ice_reader

ice.work_s3bucketname=ice-work-files
ice.work_s3bucketprefix=

ice.startmillis=1367294400000

ice.account.green=111111111111
ice.account.blue=222222222222
ice.account.pink=333333333333
ice.account.red=444444444444
ice.account.yellow=555555555555
ice.account.gray=666666666666
ice.account.purple=777777777777
ice.account.black=888888888888

ice.owneraccount.black=green,blue,pink,red,yellow,gray,purple

ice.reservationCapacityPoller=false

# # reservation period, possible values are oneyear, threeyear
ice.reservationPeriod=threeyear
# # reservation utilization, possible values are LIGHT, HEAVY
ice.reservationUtilization=HEAVY

ice.companyName=Fizzbuzz

ice.customTags=user:Product,user:Environment,user:Type

from ice.

@rayrod2030 If you don't have reservation capacity poller enabled, please remove ice.owneraccount.* properties. Try that and re-process the data.

from ice.