Comments (22)
stiv-yakovenko
commented on July 16, 2024
This is what I see in browser:
Error 500 : /recommend/2c188f15c3b9d956fd28a7ad19e2f98c
java.lang.IllegalStateException: java.lang.ClassCastException
java.lang.IllegalStateException: java.lang.ClassCastException
at com.cloudera.oryx.app.als.PartitionedFeatureVectors.lambda$mapPartitionsParallel$4(PartitionedFeatureVectors.java:139)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:479)
at com.cloudera.oryx.app.als.PartitionedFeatureVectors.mapPartitionsParallel(PartitionedFeatureVectors.java:141)
at com.cloudera.oryx.app.serving.als.model.ALSServingModel.topN(ALSServingModel.java:267)
at com.cloudera.oryx.app.serving.als.Recommend.get(Recommend.java:107)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:491)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:764)
at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1676)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1007)
at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:540)
at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:518)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException
from oryx.
stiv-yakovenko
commented on July 16, 2024
This is larger log fragment:
Sep 18 18:50:11 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:50:11,788 INFO ALSServingModelManager:122 Updating model
Sep 18 18:50:26 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:50:26,906 INFO ALSServingModelManager:129 Model updated: ALSServingModel[features:10, implicit:true, X:(1748759 users), Y:(19606 items, partitions: [0:8258, 1:2140, 2:2762, 3:2743, 4:1382, 5:850, 6:526, 7:945]...), fractionLoaded:0.97303146]
Sep 18 18:50:26 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:50:26,907 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1748759 users), Y:(19606 items, partitions: [0:8258, 1:2140, 2:2761, 3:2743, 4:1382, 5:850, 6:526, 7:946]...), fractionLoaded:0.97303146]
Sep 18 18:52:15 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:52:15,329 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1797437 users), Y:(21038 items, partitions: [0:2046, 1:4044, 2:1688, 3:5384, 4:1394, 5:1932, 6:1374, 7:3176]...), fractionLoaded:0.974506]
Sep 18 18:53:15 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:53:15,329 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1902704 users), Y:(21459 items, partitions: [0:2183, 1:4122, 2:1724, 3:5457, 4:1431, 5:1933, 6:1397, 7:3212]...), fractionLoaded:0.99872214]
Sep 18 18:55:17 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:55:17,436 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2036927 users), Y:(22126 items, partitions: [0:2139, 1:4386, 2:1781, 3:5632, 4:1514, 5:1962, 6:1432, 7:3280]...), fractionLoaded:1.0]
Sep 18 18:57:03 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:57:03,085 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2076436 users), Y:(22718 items, partitions: [0:2151, 1:4633, 2:1831, 3:5793, 4:1549, 5:1971, 6:1464, 7:3326]...), fractionLoaded:1.0]
Sep 18 18:58:33 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 18:58:33,947 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2131008 users), Y:(23286 items, partitions: [0:2185, 1:4814, 2:1880, 3:5958, 4:1594, 5:2007, 6:1487, 7:3361]...), fractionLoaded:1.0]
Sep 18 19:00:21 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:00:21,807 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2175464 users), Y:(23782 items, partitions: [0:2206, 1:4987, 2:1910, 3:6094, 4:1628, 5:2033, 6:1513, 7:3411]...), fractionLoaded:1.0]
Sep 18 19:01:56 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:01:56,122 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2240881 users), Y:(24077 items, partitions: [0:2238, 1:5071, 2:1946, 3:6178, 4:1636, 5:2044, 6:1522, 7:3442]...), fractionLoaded:1.0]
Sep 18 19:02:09 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:02:09,983 INFO ALSServingModelManager:108 Loading new model
Sep 18 19:02:11 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:02:11,428 INFO ALSServingModelManager:122 Updating model
Sep 18 19:02:23 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:02:23,618 INFO ALSServingModelManager:129 Model updated: ALSServingModel[features:10, implicit:true, X:(2003528 users), Y:(22758 items, partitions: [0:1806, 1:4904, 2:1879, 3:5917, 4:1502, 5:1937, 6:1450, 7:3363]...), fractionLoaded:0.9704026]
Sep 18 19:03:34 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:03:34,617 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2003529 users), Y:(23652 items, partitions: [0:4998, 1:4306, 2:4681, 3:3284, 4:1215, 5:2202, 6:1635, 7:1331]...), fractionLoaded:0.97083074]
Sep 18 19:04:34 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:04:34,617 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2076070 users), Y:(24243 items, partitions: [0:5031, 1:4548, 2:4714, 3:3447, 4:1245, 5:2215, 6:1654, 7:1389]...), fractionLoaded:0.9767408]
Sep 18 19:06:52 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:06:52,914 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2190733 users), Y:(24728 items, partitions: [0:5100, 1:4675, 2:4753, 3:3552, 4:1277, 5:2276, 6:1690, 7:1405]...), fractionLoaded:1.0]
Sep 18 19:08:31 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:08:31,097 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2235091 users), Y:(25472 items, partitions: [0:5245, 1:4972, 2:4794, 3:3631, 4:1298, 5:2358, 6:1707, 7:1467]...), fractionLoaded:1.0]
Sep 18 19:10:08 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:10:08,492 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2276908 users), Y:(26076 items, partitions: [0:5349, 1:5245, 2:4839, 3:3702, 4:1316, 5:2416, 6:1716, 7:1493]...), fractionLoaded:1.0]
Sep 18 19:10:36 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:10:36,614 INFO AbstractOryxResource:86 Model loaded fraction: 1.0
Sep 18 19:11:44 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:11:44,178 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2308382 users), Y:(26508 items, partitions: [0:5478, 1:5392, 2:4875, 3:3747, 4:1313, 5:2468, 6:1727, 7:1508]...), fractionLoaded:1.0]
Sep 18 19:13:25 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:13:25,918 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(2317022 users), Y:(26671 items, partitions: [0:5494, 1:5463, 2:4891, 3:3763, 4:1314, 5:2489, 6:1731, 7:1526]...), fractionLoaded:1.0]
Sep 18 19:13:49 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:13:49,107 INFO SolverCache:78 Computing cached solver
Sep 18 19:13:49 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:13:49,139 INFO SolverCache:83 Computed new solver Solver[org.apache.commons.math3.linear.RRQRDecomposition$Solver@2249514e]
Sep 18 19:14:40 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:14:40,464 INFO ALSServingModelManager:108 Loading new model
Sep 18 19:14:41 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:14:41,877 INFO ALSServingModelManager:122 Updating model
Sep 18 19:14:49 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:14:49,846 INFO ALSServingModelManager:129 Model updated: ALSServingModel[features:10, implicit:true, X:(1789789 users), Y:(23217 items, partitions: [0:5032, 1:4562, 2:4613, 3:2932, 4:1002, 5:2332, 6:1514, 7:1230]...), fractionLoaded:0.97462803]
Sep 18 19:14:49 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:14:49,847 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1789789 users), Y:(23217 items, partitions: [0:5032, 1:4562, 2:4613, 3:2931, 4:1002, 5:2332, 6:1515, 7:1230]...), fractionLoaded:0.97462803]
Sep 18 19:16:45 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:16:45,856 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1792025 users), Y:(24146 items, partitions: [0:3094, 1:1993, 2:3554, 3:2421, 4:2725, 5:1433, 6:7558, 7:1368]...), fractionLoaded:0.9750782]
Sep 18 19:17:45 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:17:45,858 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1830456 users), Y:(24497 items, partitions: [0:3107, 1:2237, 2:3547, 3:2451, 4:2718, 5:1472, 6:7564, 7:1401]...), fractionLoaded:0.9938407]
Sep 18 19:20:03 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:20:03,332 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1843671 users), Y:(24757 items, partitions: [0:3158, 1:2199, 2:3577, 3:2457, 4:2744, 5:1488, 6:7715, 7:1419]...), fractionLoaded:1.0]
Sep 18 19:21:43 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:21:43,234 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1851763 users), Y:(24953 items, partitions: [0:3196, 1:2243, 2:3586, 3:2471, 4:2764, 5:1491, 6:7771, 7:1431]...), fractionLoaded:1.0]
Sep 18 19:23:22 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:23:22,764 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1857862 users), Y:(25255 items, partitions: [0:3219, 1:2288, 2:3612, 3:2471, 4:2775, 5:1560, 6:7869, 7:1461]...), fractionLoaded:1.0]
Sep 18 19:25:02 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:25:02,574 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1862290 users), Y:(25399 items, partitions: [0:3229, 1:2319, 2:3620, 3:2465, 4:2790, 5:1578, 6:7920, 7:1478]...), fractionLoaded:1.0]
Sep 18 19:26:43 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:26:43,088 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1867851 users), Y:(25591 items, partitions: [0:3246, 1:2352, 2:3630, 3:2482, 4:2806, 5:1586, 6:7991, 7:1498]...), fractionLoaded:1.0]
Sep 18 19:27:47 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:27:47,135 INFO ALSServingModelManager:108 Loading new model
Sep 18 19:27:52 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:27:52,574 INFO ALSServingModelManager:122 Updating model
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:28:00,136 WARN AbstractServingModelManager:71 Error while processing message; continuing
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.removeIf(MutableLHashParallelKVObjObjMapGO.java:1163)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.retainRecentAndIDs(FeatureVectorsPartition.java:104)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.retainRecentAndIDs(PartitionedFeatureVectors.java:205)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.retainRecentAndItemIDs(ALSServingModel.java:331)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModelManager.consumeKeyMessage(ALSServingModelManager.java:128)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModelManager.consumeKeyMessage(ALSServingModelManager.java:45)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.api.serving.AbstractServingModelManager.consume(AbstractServingModelManager.java:69)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.lambda.serving.ModelManagerListener.lambda$contextInitialized$0(ModelManagerListener.java:141)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.lambda$log$0(LoggingCallable.java:48)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.lambda$asRunnable$1(LoggingCallable.java:66)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:28:00 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:28:00,139 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1617081 users), Y:(22471 items, partitions: [0:2677, 1:1101, 2:3355, 3:1892, 4:2669, 5:1310, 6:7941, 7:1526]...), fractionLoaded:0.92405206]
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:29:46,427 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@46c8a82d
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.TopNConsumer.accept(TopNConsumer.java:30)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.forEach(MutableLHashParallelKVObjObjMapGO.java:128)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.forEach(FeatureVectorsPartition.java:114)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.lambda$topN$5(ALSServingModel.java:270)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.lambda$mapPartitionsParallel$2(PartitionedFeatureVectors.java:105)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.call(LoggingCallable.java:56)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:29:46 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String] with root cause
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.TopNConsumer.accept(TopNConsumer.java:30)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.forEach(MutableLHashParallelKVObjObjMapGO.java:128)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.forEach(FeatureVectorsPartition.java:114)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.lambda$topN$5(ALSServingModel.java:270)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.lambda$mapPartitionsParallel$2(PartitionedFeatureVectors.java:105)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.call(LoggingCallable.java:56)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 18 19:29:46 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:30:03 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:30:03,937 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1621166 users), Y:(23233 items, partitions: [0:3057, 1:2586, 2:3843, 3:6160, 4:381, 5:973, 6:3328, 7:2905]...), fractionLoaded:0.92427194]
Sep 18 19:31:03 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:31:03,939 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1709170 users), Y:(23424 items, partitions: [0:3077, 1:2628, 2:3856, 3:6171, 4:392, 5:987, 6:3392, 7:2921]...), fractionLoaded:0.9722466]
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:31:29,852 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@1a214c66
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.TopNConsumer.accept(TopNConsumer.java:30)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.forEach(MutableLHashParallelKVObjObjMapGO.java:128)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.forEach(FeatureVectorsPartition.java:114)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.lambda$topN$5(ALSServingModel.java:270)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.lambda$mapPartitionsParallel$2(PartitionedFeatureVectors.java:105)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.call(LoggingCallable.java:56)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:31:29 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String] with root cause
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.TopNConsumer.accept(TopNConsumer.java:30)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.forEach(MutableLHashParallelKVObjObjMapGO.java:128)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.forEach(FeatureVectorsPartition.java:114)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.lambda$topN$5(ALSServingModel.java:270)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.lambda$mapPartitionsParallel$2(PartitionedFeatureVectors.java:105)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.call(LoggingCallable.java:56)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 18 19:31:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:31:35,681 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@4c30c35d
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.TopNConsumer.accept(TopNConsumer.java:30)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.forEach(MutableLHashParallelKVObjObjMapGO.java:128)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.forEach(FeatureVectorsPartition.java:114)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.lambda$topN$5(ALSServingModel.java:270)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.lambda$mapPartitionsParallel$2(PartitionedFeatureVectors.java:105)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.call(LoggingCallable.java:56)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:31:35 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String] with root cause
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.TopNConsumer.accept(TopNConsumer.java:30)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.forEach(MutableLHashParallelKVObjObjMapGO.java:128)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.forEach(FeatureVectorsPartition.java:114)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.lambda$topN$5(ALSServingModel.java:270)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.lambda$mapPartitionsParallel$2(PartitionedFeatureVectors.java:105)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.call(LoggingCallable.java:56)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 18 19:31:35 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:32:05 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:32:05,732 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@4d79b330
Sep 18 19:32:05 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:32:05 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:32:05 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:32:05 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:32:05 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:32:31 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:32:31,456 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@77b4a2c7
Sep 18 19:32:31 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:32:31 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:32:31 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:32:31 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:32:31 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:33:23 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:33:23,597 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1768312 users), Y:(23567 items, partitions: [0:3098, 1:2659, 2:3891, 3:6206, 4:392, 5:984, 6:3396, 7:2941]...), fractionLoaded:1.0]
Sep 18 19:35:01 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:35:01,190 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1776786 users), Y:(23839 items, partitions: [0:3131, 1:2713, 2:3925, 3:6273, 4:404, 5:981, 6:3416, 7:2996]...), fractionLoaded:1.0]
Sep 18 19:39:50 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:39:50,949 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@70777ab
Sep 18 19:39:50 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:39:50 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:39:50 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:39:50 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:39:50 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:41:19 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:41:19,826 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@9c0893b
Sep 18 19:41:19 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:41:19 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:41:19 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:41:19 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:41:19 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:41:21 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:41:21,951 INFO ALSServingModelManager:108 Loading new model
Sep 18 19:41:23 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:41:23,312 INFO ALSServingModelManager:122 Updating model
Sep 18 19:41:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:41:29,316 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@1ce01049
Sep 18 19:41:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:41:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:41:29 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:41:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:41:29 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:41:30,872 WARN AbstractServingModelManager:71 Error while processing message; continuing
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.String
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.koloboke.collect.impl.hash.MutableLHashParallelKVObjObjMapGO.removeIf(MutableLHashParallelKVObjObjMapGO.java:1163)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.FeatureVectorsPartition.retainRecentAndIDs(FeatureVectorsPartition.java:104)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.als.PartitionedFeatureVectors.retainRecentAndIDs(PartitionedFeatureVectors.java:205)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModel.retainRecentAndItemIDs(ALSServingModel.java:331)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModelManager.consumeKeyMessage(ALSServingModelManager.java:128)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.app.serving.als.model.ALSServingModelManager.consumeKeyMessage(ALSServingModelManager.java:45)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.api.serving.AbstractServingModelManager.consume(AbstractServingModelManager.java:69)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.lambda.serving.ModelManagerListener.lambda$contextInitialized$0(ModelManagerListener.java:141)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.lambda$log$0(LoggingCallable.java:48)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at com.cloudera.oryx.common.lang.LoggingCallable.lambda$asRunnable$1(LoggingCallable.java:66)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: at java.lang.Thread.run(Thread.java:748)
Sep 18 19:41:30 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:41:30,877 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1742631 users), Y:(22512 items, partitions: [0:2863, 1:2246, 2:3814, 3:6107, 4:282, 5:785, 6:3413, 7:3002]...), fractionLoaded:0.9812401]
Sep 18 19:43:52 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:43:52,022 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1742631 users), Y:(22811 items, partitions: [0:2801, 1:762, 2:3046, 3:2313, 4:5589, 5:2616, 6:3696, 7:1988]...), fractionLoaded:0.9812432]
Sep 18 19:44:52 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:44:52,033 INFO ALSServingModelManager:96 ALSServingModel[features:10, implicit:true, X:(1770277 users), Y:(22811 items, partitions: [0:2801, 1:762, 2:3046, 3:2313, 4:5589, 5:2616, 6:3696, 7:1988]...), fractionLoaded:0.99660903]
Sep 18 19:45:02 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:45:02,254 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@4c44a24a
Sep 18 19:45:02 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:45:02 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:45:02 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:45:02 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:45:02 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:45:25 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:45:25,483 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@151b8ec0
Sep 18 19:45:25 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:45:25 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:45:25 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:45:25 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:45:25 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:45:40 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:45:40,927 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@6c146741
Sep 18 19:45:40 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:45:40 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:45:40 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:45:40 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:45:40 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:45:48 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:45:48,237 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@380335c5
Sep 18 19:45:48 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:45:48 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:45:48 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:45:48 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:45:48 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:46:04 htm-psycho-401.zxz.su oryx-run.sh[13885]: 2018-09-18 19:46:04,786 WARN LoggingCallable:58 Unexpected error in com.cloudera.oryx.app.als.PartitionedFeatureVectors$$Lambda$188/259218113@5de3f4a
Sep 18 19:46:04 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
Sep 18 19:46:04 htm-psycho-401.zxz.su oryx-run.sh[13885]: Sep 18, 2018 7:46:04 PM org.apache.catalina.core.StandardWrapperValve invoke
Sep 18 19:46:04 htm-psycho-401.zxz.su oryx-run.sh[13885]: SEVERE: Servlet.service() for servlet [Jersey] in context with path [Oryx] threw exception [java.lang.IllegalStateException: java.lang.ClassCastException] with root cause
Sep 18 19:46:04 htm-psycho-401.zxz.su oryx-run.sh[13885]: java.lang.ClassCastException
from oryx.
stiv-yakovenko
commented on July 16, 2024
After restarting serving layer bug gone away. (temporary?)
from oryx.
stiv-yakovenko
commented on July 16, 2024
But some time later error is back....
from oryx.
srowen
commented on July 16, 2024
BTW you can just post a small relevant snippet of logs, rather than a huge section of them.
This is a duplicate of #304 which I never really figured out. It's a very strange error. But there are some alternatives there.
from oryx.
stiv-yakovenko
commented on July 16, 2024
I don't see a workaround. How can I diagnose this to help you?
from oryx.
srowen
commented on July 16, 2024
I honestly could not figure out how it could possibly happen. Any hint about a reproduction would help a lot.
from oryx.
stiv-yakovenko
commented on July 16, 2024
Collection is assumed to have only Strings inside, but having Object provokes crash, right? I propose to supply some logging to all places which put objects into this collection and check data type. Can you do it?
from oryx.
stiv-yakovenko
commented on July 16, 2024
I don't know how to reproduce it. I can send you hdfs.
from oryx.
stiv-yakovenko
commented on July 16, 2024
I've observed kafka was dead (probably because /tmp run out of space), so may be kafka sent some object, which is not String but rather error message which was serialised to the collection.
from oryx.
stiv-yakovenko
commented on July 16, 2024
I've also observed the problem without hdd overflow. My guess is that I was calling /ingest from multiple threads, so that data streams of different blocks has collided. Do you have synchronized block on /ingest handler?
from oryx.
srowen
commented on July 16, 2024
I don't think it's related to ingest directly; that's not where the issue occurs. It's actually in the path that reads updates from the speed layer. You can check the source code, but yes the data structure here is protected by a read/write lock. I checked it again and can't see any issue. It's not a concurrent modification issue, but somehow a raw Object gets in here, where nothing in the whole framework should use or allocate an Object. No idea how it happens. You're right that it's worth looking at how values are read from Kafka and parsed, but anything that I can imagine causing an Object to be created would cause other failures first. I think an internal array is resized somewhere with the wrong type or something. Truly puzzling, I looked at this for a long time and could never reproduce it or guess the problem.
from oryx.
stiv-yakovenko
commented on July 16, 2024
So what about adding checking code?
from oryx.
srowen
commented on July 16, 2024
I've tried lots of that sort of thing before and never been able to reproduce it locally. Other reporters couldn't reproduce it even with similar debug code.
from oryx.
stiv-yakovenko
commented on July 16, 2024
There is definitely something wrong here:
(the class is in jar of course)
from oryx.
srowen
commented on July 16, 2024
That is something else; best not to mix issues in one thread.
I have never seen that, but indeed seems like a deployment problem of some kind rather than issue with the project. I am unclear how you are deploy the serving layer. Please use the mailing list for questions and discussion.
from oryx.
stiv-yakovenko
commented on July 16, 2024
I've meditated a bit on FeatureVectorsPartitions and MutableLHashParallelKVObjObjMapGO a bit and I have hypothesis explaining whats going on during crash. Crash happens near this line:
if ((key = (K) tab[i]) != FREE) {
if (filter.test(new MutableEntry(mc, i, key, (V) tab[i + 1]))) {
and you can see from MutableLHashParallelKVObjObjMapGO code that inside this data structure they keep data in table[] array like this: [K1,V1,K2,V2,...]. Due to some unknown reason it turns out that key gets assigned from tab[i], while tab[i] is float[]. Generics are removed after compilation, so inside .class file key is just Object. On the next line we are about to construct MutableEntry object and java tries to cast key to String, but this is impossible, because it contains float[].
Now, the question is what might have caused this data corruption of this hashtable? I don't know. It may be an algorithmic bug, which happens under some special circumstances. Or it may be concurrency problem (I see that all calls to vectors are under lock, but i don't have 100% understanding of the code). So I'd propose simply to replace this hashmap with traditional java HashMap or even concurrent hashmap, which doesn't have this inner problems. What do you think?
from oryx.
srowen
commented on July 16, 2024
Objects in the JVM still have a type at runtime, after erasure, and an Object is not the same type as a float[]. The exception shows the actual runtime type of the object, and it's an Object, not a float[] or String. I don't think a float[] can be put into this data structure; it would cause a compile-time error or require casting to happen, and I don't see that.
An Object is typically only used as a marker object in internal data structures. There's little reason to allocate an Object otherwise. It's likely somehow related to this, that the internal state is corrupted an a special marker object is treated as data. I can't see how it happens yet nor see if it's a client-side problem or library side, if so.
from oryx.
stiv-yakovenko
commented on July 16, 2024
HI! Yes, this is a piece of code where this Object I think is created:
static final Object REMOVED = new Object(), FREE = new Object();
and these FREE/REMOVED are used to identify something in tab[]
from oryx.
stiv-yakovenko
commented on July 16, 2024
This issue is bug inside this collection, I propose to replace it with regular hashmap:
from oryx.
stiv-yakovenko
commented on July 16, 2024
essentially it looks like hashtable contains REMOVED entry and when you try to removeIf, then removeIf has no check on REMOVED, they only check on FREE, which looks like a bug. Think about removing these unsafe collections completely, they have not been tested under really heavy load and are not supported since 2016....
from oryx.
srowen
commented on July 16, 2024
Hey good find. That does quite sound like the same problem. We can change the implementation, though it will cost some performance. I'll have a look later and see if there's any other way to work around this.
from oryx.
Related Issues (20)
- Kafka 0.11.x support HOT 27
- Customer ID update HOT 2
- Multiple jobs on oryx HOT 3
- Update to Tomcat 9 / Jersey 2.26 HOT 1
- Error while runnig Oryx batch HOT 1
- Spark 2.3 and java.lang.NoClassDefFoundError: kafka/admin/RackAwareMode HOT 64
- java.lang.NoClassDefFoundError: scala/collection/Seq HOT 3
- 2.7.0 - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka010/LocationStrategies HOT 1
- mvn install error HOT 1
- SpeedLayer read kafka offset HOT 1
- KryoException: Buffer overflow. Available: 1, required: 4 HOT 19
- Trouble handling huge data HOT 3
- Reseting model HOT 3
- /knownItems/uid doesn't return all items HOT 50
- Can't build oryx 8.0 HOT 8
- Recommend products FROM <IDs> HOT 1
- Oryx tutorial HOT 2
- Application is added to the scheduler and is not yet activated. HOT 2
- question about "fold-in" implementation HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oryx.