cwensel / cascading Goto Github PK

Using an aws endpoint and, separately, an s3 URL of the form s3://my_tenant:my-bucket (including underscores)

In the S3Tap constructor (line 824 in my version) this line is causing problems:
this.bucketName = identifier.getHost();

The URI host is null. However the URI authority contains 'my_tenant:my-bucket', which is the full name of the bucket I need to access.

I am encountering an issue when running unit tests in a custom library that has Cadence as a dependency. This block of code has an issue when casting a Class to a String. The call to the PropertyUtil function is coming from AppProps.getApplicationJarClass. Here is the stack trace:

java.lang.ClassCastException: java.lang.Class cannot be cast to java.lang.String at cascading.property.AppProps.getApplicationJarClass(AppProps.java:113) at cascading.flow.hadoop.planner.HadoopPlanner.initialize(HadoopPlanner.java:166) at com.anon.cascading_ext.flow.LoggingHadoopPlanner.initialize(LoggingHadoopPlanner.java:75) at com.anon.cascading_ext.flow.LoggingFlowConnector.connect(LoggingFlowConnector.java:68) at cascading.flow.FlowConnector.connect(FlowConnector.java:421) at cascading.flow.FlowConnector.connect(FlowConnector.java:270) at cascading.flow.FlowConnector.connect(FlowConnector.java:215) at cascading.flow.FlowConnector.connect(FlowConnector.java:197) at com.anon.formats.mapred.inputformat.TestDirectoryLineInputFormat.testGetCurrentTapSourcePath(TestDirectoryLineInputFormat.java:61)

This could be solved by checking if defaultValue is an instance of String and using the Class.toString function. My org is in the process of upgrading from a very old version of Cascading to Cascading 4.5.0. In the version that we are using, AppProp.getApplicationJarClass actually passes a null Class object as the default value.

Is there a recommended method of fixing this casting issue?

Running into this on JDK 8, Cascading 4.5.0

Step/SliceCounters sums durations in wall clock milliseconds, but some read or write operations may be very short.

The fear is we are introducing a lot of error by summing ms interval durations. Using a Stopwatch with nanos granularity, that then is periodically published on a shared timer may improve relative accuracy.

Instead is using the #compile() method to generate a pointer, subsequently it is not capturing nested structures.

This operation may need to be updated to apply patterns to each child element in the pointer.allAt method.

I have created a fork of wip-4.0 with preliminary Hadoop 3 support, https://github.com/jonshea/cascading/tree/wip-4.5-hadoop-3 . I would love to be able to include this in the cascading main-line releases. What’s the best way forward here? Do you want to make a wip-4.5 branch that I make a pull request against?

Even if the name is declared, $n should still resolve appropriately.

For some reason, Cascading integration was removed from Parquet, apache/parquet-mr@907314c.

There was much room for improvement, so we can add it back here.

The upgrade requires some refactoring as the script and expression evaluators no longer have a shared base class.

By default, we don't leverage threading server/processing side since it's a highly constrained resource.

But in local mode, and when partitioning sink data into multiple files, parallelizing the close operation makes sense as the purges may include more than one partition.

Providing hooks to wrap the cache of open partitions purge and close operations would benefit applications that were designed specifically for partitioning data if the container hosting the application can provide cpus for the threads.

The Github Artifacts require authentication for read. This makes them hard to use seamlessly in other open source projects. Would it be possible to cross-publish to maven central or another maven-compatible repo with read enabled without auth?

hey @cwensel any chance we could keep java8 on the 4.5 branch at least? AWS EMR 6.X (latest release) still defaults on java8 for the hadoop3 flavors and it would be much appreciated to keep this support before they drop it. I would imagine that EMR is a very common use case for this type of library.

This type should support String representations via Base64.

something that was noticed when trying to use new versions of cascading on other projects like scalding (https://github.com/twitter/scalding/pull/1586/files example but this was abondoned) is an awkwardness between old API org.apache.hadoop.mapred and new API org.apache.hadoop.mapreduce and casting between them.

would cascading accept contributions to incrementally refactor parts against the new API?

Seeing this exception sometimes talking to S3. Could be a race condition on directory (key) creates on S3, but there should be no task cleaup when using S3.

    at cascading.tap.hadoop.io.TapOutputCollector.close(TapOutputCollector.java:184)
    at cascading.tuple.TupleEntrySchemeCollector.close(TupleEntrySchemeCollector.java:245)
    at cascading.tap.partition.BasePartitionTap$PartitionCollector.closeCollector(BasePartitionTap.java:205)
    at cascading.tap.partition.BasePartitionTap$PartitionCollector.close(BasePartitionTap.java:190)
    at cascading.flow.stream.element.SinkStage.cleanup(SinkStage.java:148)
    at cascading.flow.stream.graph.StreamGraph.cleanup(StreamGraph.java:187)
    at cascading.flow.local.planner.LocalStepRunner.call(LocalStepRunner.java:204)
    at cascading.flow.local.planner.LocalStepRunner.call(LocalStepRunner.java:53)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://bucket/test/_temporary/_attempt_002147483647_0000_m_000000_0/path/
    at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1931)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1822)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1763)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1585)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1561)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:326)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.commitTask(Hadoop18TapUtil.java:174)
    at cascading.tap.hadoop.io.TapOutputCollector.close(TapOutputCollector.java:171)
    ... 11 more```

Exception in thread "main" java.lang.VerifyError: Inconsistent stackmap frames at branch target 83
Exception Details:
Location:
cascading/stats/hadoop/HadoopStepStats.captureDetail(Lcascading/stats/CascadingStats$Type;)V @83: aload_0
Reason:
Type '[Lorg/apache/hadoop/mapred/TaskCompletionEvent;' (current frame, locals[4]) is not assignable to '[Lorg/apache/hadoop/mapreduce/TaskCompletionEvent;' (stack map, locals[4])
Current Frame:
bci: @77
flags: { }
locals: { 'cascading/stats/hadoop/HadoopStepStats', 'cascading/stats/CascadingStats$Type', 'org/apache/hadoop/mapreduce/Job', integer, '[Lorg/apache/hadoop/mapred/TaskCompletionEvent;' }
stack: { integer }
Stackmap Frame:
bci: @83
flags: { }
locals: { 'cascading/stats/hadoop/HadoopStepStats', 'cascading/stats/CascadingStats$Type', 'org/apache/hadoop/mapreduce/Job', integer, '[Lorg/apache/hadoop/mapreduce/TaskCompletionEvent;' }
stack: { }

Hadoop version: 2.8.0

Currently the empty string in hashed, optionally this should be null.

According to the implementation of TapOutputCollector#close and Hadoop18TapUtil#needsTaskCommit, work is always committed as long as something exists in the work path. As a result, I am seeing partial output files committed in S3 when I use a PartitionTap and my task fails after it has written some output.

Is this expected behaviour? I am seeing this in Cascading 3.1, which I understand is a few years old now, but any information here would be of great help

Thanks

see this graph on 3.2.1
https://www.dropbox.com/s/iffadh9x7unrg5w/01-BalanceAssembly-init.dot.png?dl=0

You can see the full planner logs here:
https://www.dropbox.com/s/7qyc4a9pxtstwio/E552D2.tgz?dl=0\

We are merging two HashJoins after some Each operations. In this particular graph, it is possible to fix the issue by adding Checkpoints after all but one of the HashJoins it seems. This is not a great solution since even knowing what a graph will look like when you combine many pipes with functions is not very clear.

It would be great to have either a clear rule that we need to follow in generating the graphs, or to remove this restriction since we would like to using cascading 3 in scalding by default.

Thanks.

JSONGetFunction (or sibling) should optionally account for values in lists and generate tuples for each item.

ObjectMapper#treeToValue is the natural way to convert a TreeNode/JsonNode to a requested type

Hadoop 3 can run under Java 11.

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions

Depends on #75

Local mode should set this property so that operations have access to the current file being processed.

I've been testing scalding with newer cascading as a demo on a branch here: daniel-sudz/scalding#1.

I currently have the following bad output:

[info] - should merge and joinWithTiny shouldn't duplicate data *** FAILED ***
[info]   Set((1,3), (2,3), (3,3), (4,2)) was not equal to Set((1,2), (2,2), (3,2), (4,1)) (PlatformTest.scala:466)

it looks like there is some duplication going on considering 3 > 2 and 2 > 1. I saw that there was some previous discussion around this when cascading3 scalding branch was being developed before it got stalled. twitter/scalding#1592. The resolution there seemed to be a higher hadoop version so not really applicable here.

Not sure where to begin debugging this but would love some pointers.