uber-archive / buck-http-cache Goto Github PK

This is a follow up to #4 since the solution proposed there isn't ideal for us. Our keys are generally small and the values are big which matches OFFHEAP_VALUES best so OFFHEAP_TIERED isn't ideal for us. We also use a 2 day retention policy.

Our issue in particular is that we exceed our memory capacity and are being killed by the system.

Ideally, we'd like to see a way to set a max memory usage so that we aren't being killed by the system.

In order to segment which requests to the http cache are coming from different machines (CI vs local machines for example) it would be helpful to be able to pass a tag to the buck-http-cache when making the GET request, perhaps as a query param or similar. I don't believe this is currently possible (no custom logging options appear in this method) but correct me if I'm wrong.

cc @zayhero @kageiit

We're trying to run the buck-http-cache as a cluster with 3 nodes (deployed via Kubernetes). However, it quickly (often within ~1 hour) reaches a bad state where requests repeatedly throw errors like this:

[29/Jun/2018:01:06:40 +0000] "PUT /artifacts/key HTTP/1.1" 500 110 "-" "okhttp/3.6.0" 1
ERROR [2018-06-29 01:06:40,490] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 51ddcabd89d87889
! java.lang.NullPointerException: null
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl$2.call(GridCacheAtomicSequenceImpl.java:514)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl$2.call(GridCacheAtomicSequenceImpl.java:487)
! at org.apache.ignite.internal.processors.cache.GridCacheUtils$21.call(GridCacheUtils.java:1613)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.internalUpdate(GridCacheAtomicSequenceImpl.java:258)
! ... 61 common frames omitted
! Causing: org.apache.ignite.IgniteCheckedException: null
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.internalUpdate(GridCacheAtomicSequenceImpl.java:264)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.incrementAndGet(GridCacheAtomicSequenceImpl.java:178)
! ... 60 common frames omitted
! Causing: org.apache.ignite.IgniteException: null
! at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:946)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.incrementAndGet(GridCacheAtomicSequenceImpl.java:181)
! at com.uber.buckcache.datastore.impl.ignite.IgniteDataStoreProvider.putData(IgniteDataStoreProvider.java:113)
! at com.uber.buckcache.datastore.impl.ignite.IgniteDataStoreProvider.putData(IgniteDataStoreProvider.java:82)
! at com.uber.buckcache.resources.buckcache.BuckCacheResource.addArtifactToCache(BuckCacheResource.java:147)
...

We originally ran Apache Ignite in PARTITIONED mode, and we even tried out REPLICATED mode... but it's still happening. We suspect it's because either Ignite or the cache itself is kind of brittle when there's momentary network disconnections between the nodes in the buck-http-cache cluster.

Has anyone seen these problems before, or have any tips on what to do? Thanks!

and I get this warning in the logs

WARN [2016-10-28 16:48:29,903] org.apache.ignite.internal.processors.cache.GridCacheProcessor: Off heap maximum memory configuration property will be ignored for the cache working in OFFHEAP_VALUES mode (memory usage will be unlimited): metadata. Consider configuring eviction policy or switching to OFFHEAP_TIERED mode or.

Hi,

We have just implement this app in our CI environment and it seems to be working quiet well.

Is there any option to enable persistence storage so that the cache is not lost on a server reboot?

Where does buck cache server store the actual cache?
Also, can I configure the path?

I'm looking at running 2 cache servers (on different ports) on the same machine.
The reason for that is buck's current limitation with support for min sdk compiler option (facebook/buck#1803).
There's no way to set min sdk per target and in our case we have a bunch or libraries that need to be built both for iOS 9 and 10, depending on which app we plan to integrate them into.

This means a cache generated for iOS 10.0 build is not usable when building for 9.0, causing all kinds of errors.

Until buck has that option, we just have 2 different cache directories per min sdk.
So we'd want to have 2 cache servers running for each target sdk, and we need to make sure each server will have its separate storage path.