Sorry for the bare bones issue, I'd be happy to flesh out more but I'm not sure where to start providing data. I also suspect this might be a skein bug, but I'm not sure - happy to point it in their direction if that is the case
When creating a YarnCluster and specifying worker_vcores, the scheduler starts up fine, but no workers instantiate (either via a scale or by specifying initial n_workers). The application logs look like there's a mismatch between what containers are being requested and what skein wants, which points to a problem on there end, but I'm not too sure and figured I'd start here. Any ideas for additional things to look for?
Specifying worker_memory works fine, but will always end up having only 1 core per container regardless of how much memory you give it, so you end up with situations where there's 10 containers with 100gb of ram each only using 1 core. Pains me inside :)
19/01/15 19:04:25 INFO skein.ApplicationMaster: Application specification successfully loaded
19/01/15 19:04:25 INFO skein.ApplicationMaster: Running as user hadoop
19/01/15 19:04:25 INFO client.RMProxy: Connecting to ResourceManager at ip-10-201-102-148.ginkgobioworks.com/10.201.102.148:8030
19/01/15 19:04:25 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
19/01/15 19:04:25 INFO skein.ApplicationMaster: gRPC server started at ip-10-201-102-157.ginkgobioworks.com:42873
19/01/15 19:04:25 INFO skein.ApplicationMaster: WebUI server started at ip-10-201-102-157.ginkgobioworks.com:32841
19/01/15 19:04:25 INFO client.RMProxy: Connecting to ResourceManager at ip-10-201-102-148.ginkgobioworks.com/10.201.102.148:8032
19/01/15 19:04:25 INFO skein.ApplicationMaster: Initializing service 'dask.worker'.
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_0
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_1
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_2
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_3
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_4
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_5
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_6
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_7
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_8
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_9
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_10
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_11
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_12
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_13
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_14
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_15
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_16
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_17
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_18
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_19
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_20
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_21
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_22
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_23
19/01/15 19:04:25 INFO skein.ApplicationMaster: REQUESTED: dask.worker_24
19/01/15 19:04:27 INFO impl.AMRMClientImpl: Received new token for : ip-10-201-102-157.ginkgobioworks.com:8041
19/01/15 19:04:27 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 1, releasing container_1547573448325_0009_01_000002
19/01/15 19:04:28 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 2, releasing container_1547573448325_0009_01_000003
19/01/15 19:04:29 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 3, releasing container_1547573448325_0009_01_000004
19/01/15 19:04:30 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 4, releasing container_1547573448325_0009_01_000005
19/01/15 19:04:31 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 5, releasing container_1547573448325_0009_01_000006
19/01/15 19:04:32 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 6, releasing container_1547573448325_0009_01_000007
19/01/15 19:04:33 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 7, releasing container_1547573448325_0009_01_000008
19/01/15 19:04:34 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 8, releasing container_1547573448325_0009_01_000009
19/01/15 19:04:35 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 9, releasing container_1547573448325_0009_01_000010
19/01/15 19:04:36 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 10, releasing container_1547573448325_0009_01_000011
19/01/15 19:04:37 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 11, releasing container_1547573448325_0009_01_000012
19/01/15 19:04:38 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 12, releasing container_1547573448325_0009_01_000013
19/01/15 19:04:39 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 13, releasing container_1547573448325_0009_01_000014
19/01/15 19:04:40 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 14, releasing container_1547573448325_0009_01_000015
19/01/15 19:04:41 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 15, releasing container_1547573448325_0009_01_000016
19/01/15 19:04:42 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 16, releasing container_1547573448325_0009_01_000017
19/01/15 19:04:43 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 17, releasing container_1547573448325_0009_01_000018
19/01/15 19:04:44 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 18, releasing container_1547573448325_0009_01_000019
19/01/15 19:04:45 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 19, releasing container_1547573448325_0009_01_000020
19/01/15 19:04:46 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 20, releasing container_1547573448325_0009_01_000021
19/01/15 19:04:47 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 21, releasing container_1547573448325_0009_01_000022
19/01/15 19:04:48 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 22, releasing container_1547573448325_0009_01_000023
19/01/15 19:04:49 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 23, releasing container_1547573448325_0009_01_000024
19/01/15 19:04:50 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 24, releasing container_1547573448325_0009_01_000025
19/01/15 19:04:51 WARN skein.ApplicationMaster: No matching service found for resource <memory:2048, vCores:1>, priority 25, releasing container_1547573448325_0009_01_000026