Comments (7)
noiano
commented on July 24, 2024
install apache thrift, then ant clean and ant
from commoncrawl.
commented on July 24, 2024
Hi, org.commoncrawl.protocol.shared.ArcFileItem is generated via the file protocol.jr. As the previous comment mentioned, try doing an ant clean and then ant. Normally gen-records-shared and gen-records targets (which generate the missing java files) should run before compile-core-classes.
from commoncrawl.
Eugene56
commented on July 24, 2024
Many thanks for help. ant clean fixed the problem. But one more occured. I have javac 1.7.0_01. When compiling with ant I receivce errors like:
compile-core-classes:
[javac] Compiling 173 source files to /root/commoncrawl/commoncrawl-commoncrawl-24052ae/build/classes
[javac] /root/commoncrawl/commoncrawl-commoncrawl-24052ae/src/org/commoncrawl/query/Query.java:942: incomparable types: java.lang.Enum<capture#256 of ?> and org.commoncrawl.hadoop.mergeutils.SequenceFileMerger.Counters
[javac] if (key == SequenceFileMerger.Counters.PCT_COMPLETED) {
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error
...
When added <property name="javac.args" value="-Xlint:unchecked -Xlint:deprecation"/> to build.xml errors chaned to:
compile-core-classes:
[javac] Compiling 173 source files to /root/commoncrawl/commoncrawl-commoncrawl-24052ae/build/classes
[javac] /root/commoncrawl/commoncrawl-commoncrawl-24052ae/src/org/commoncrawl/hadoop/io/ARCInputFormat.java:23: warning: [deprecation] org.apache.hadoop.mapred.InputFormat in org.apache.hadoop.mapred has been deprecated
[javac] import org.apache.hadoop.mapred.InputFormat;
[javac] ^
[javac] /root/commoncrawl/commoncrawl-commoncrawl-24052ae/src/org/commoncrawl/hadoop/io/ARCInputFormat.java:24: warning: [deprecation] org.apache.hadoop.mapred.InputSplit in org.apache.hadoop.mapred has been deprecated
[javac] import org.apache.hadoop.mapred.InputSplit;
[javac] ^
...
...
skipped
...
...
[javac] for (InputSplit split : splits) {
[javac] ^
[javac] /root/commoncrawl/commoncrawl-commoncrawl-24052ae/src/org/commoncrawl/samples/BasicArcFileReaderSample.java:105: warning: [deprecation] org.apache.hadoop.mapred.InputSplit in org.apache.hadoop.mapred has been deprecated
[javac] public InputSplit getInputSplit()
[javac] ^
[javac] /root/commoncrawl/commoncrawl-commoncrawl-24052ae/src/org/commoncrawl/samples/BasicArcFileReaderSample.java:139: warning: [unchecked] unchecked call to next(K,V) as a member of the raw type org.apache.hadoop.mapred.RecordReader
[javac] while (reader.next(url, item)) {
[javac] ^
[javac] /root/commoncrawl/commoncrawl-commoncrawl-24052ae/src/org/commoncrawl/util/shared/Filter.java:58: warning: [unchecked] unchecked cast
[javac] found : java.lang.Object
[javac] required: org.commoncrawl.util.shared.ICompactSerializer<org.commoncrawl.util.shared.Filter>
[javac] return (ICompactSerializer<Filter>) method.invoke(null);
[javac] ^
[javac] 1 error
[javac] 89 warnings
BUILD FAILED
/root/commoncrawl/commoncrawl-commoncrawl-24052ae/build.xml:266: Compile failed; see the compiler error output for details.
I can't get any error in compiler output...only tonns of [deprecation] and [unchecked] warnings. I suggest the problem can be due to improper JAVA or ant or maven version in my system. Could one suggest optimum versions of java/ant/maven/thrift packeges to use for commoncrawl compilation? I have EC2 FC8 instance with:
java-1.7.0-icedtea-devel-1.7.0.0-0.20.b21.snapshot.fc8
java-1.7.0-icedtea-1.7.0.0-0.20.b21.snapshot.fc8
ant-javadoc-1.7.0-1jpp.2.fc8
ant-1.7.0-1jpp.2.fc8
maven2-2.0.4-10jpp.8.fc8
and thrift 0.7.0 compiled from source;
Eugene
from commoncrawl.
noiano
commented on July 24, 2024
Well I just have java 1.6.0.29 and ant 1.8.2 and everything works fine
from commoncrawl.
commented on July 24, 2024
Hi Eugene,
I will take a look at building against Java 7. The deprecation warnings are an issue, and are related to the need for us to migrate to the mapreduce Hadoop package (instead of mapred). Hopefully we can fix this in an upcoming commit.
Ahad.
from commoncrawl.
Eugene56
commented on July 24, 2024
Just switched to sun java 1.6.0.29 and successfully compiled the package. Thanks for help.
Eugene
from commoncrawl.
Schdoop
commented on July 24, 2024
Hi all,
Sorry for commenting on the closed issue, but I have very similar one, with ant 1.8.0, java 1.6.0.18:
compile-core-classes:
[javac] Compiling 173 source files to /env/commoncrawl-commoncrawl-b835665/build/classes
[javac] /env/commoncrawl-commoncrawl-b835665/src/org/commoncrawl/query/Query.java:942: incomparable types: java.lang.Enum<capture#565 of ?> and org.commoncrawl.hadoop.mergeutils.SequenceFileMerger.Counters
[javac] if (key == SequenceFileMerger.Counters.PCT_COMPLETED) {
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error
BUILD FAILED
/env/commoncrawl-commoncrawl-b835665/build.xml:266: Compile failed; see the compiler error output for details.
What can be done with that?
Many thanks,
Oleg.
from commoncrawl.
Related Issues (8)
- Add jar to maven central repository?
- Index for WET files? HOT 5
- Broken link on main site HOT 2
- comment on public ARCFileReader constructor is confusing HOT 2
- WARN[0060] error instantiating commoncrawl: commoncrawl.apiResult: decode slice: expect [ or n, but found , error found in #0 byte of ...||..., bigger context ...||... HOT 1
- VerifyError HOT 1
- Update binaries path in build.xml
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from commoncrawl.