google / mr4c Goto Github PK

License: Apache License 2.0

Shell 0.69% ApacheConf 0.01% HTML 0.04% Makefile 0.86% C++ 36.02% C 0.75% XML 0.90% Java 60.74%

mr4c's Introduction

Introduction to the MR4C repo

About MR4C

MR4C is an implementation framework that allows you to run native code within the Hadoop execution framework. Pairing the performance and flexibility of natively developed algorithms with the unfettered scalability and throughput inherent in Hadoop, MR4C enables large-scale deployment of advanced data processing applications.

Map to this repo

This repository includes user guide, tutorials and source code for the MR4C framework created by Google Inc. We suggest you run through this repo in the following order:

Make sure that you have all dependencies and build (see below).
Test that MR4C install was successful - Run test_mr4c.sh from the test directory
Study up on MR4C - README.md in the UserGuide directory covers the basic concepts behind MR4C
Run through the example algorithms in the tutorial directory
Build your own algorithm using the examples as templates and let us know if you have questions or comments!

Dependencies

tested with Ubuntu 12.04 and CentOS 6.5
tested with CDH 5.2.0 (either MRV1 or YARN)
ant (1.8.2 min)
java (1.6 min)
ivy (2.1 min)
make (3.8.1 min)
g++ (4.6.3 min)
log4cxx (0.10.0)
jansson (2.2.1 min)
cppunit (1.12.1 min)
proj4 (4.8.0 min)
gdal (1.10 min)

Build

There are four scripts included to build, clean, deploy and/or remove mr4c. Build with:

./build_all

Clean previous builds with:

./clean_all

Deploy to /usr/local/mr4c using:

./deploy_all

Remove all components with:

./remove_all

If you get stuck, have questions, or would like to provide any feedback, please don’t hesitate to contact us at [email protected]. Let’s do big things together.

mr4c's People

Contributors

Stargazers

Watchers

Forkers

nischalsamji asifiqbal ablimit nivertech ruo91 chubbymaggie sophiste hubberwisdom oraclexbw iamsile ccl0326 zhengshuxin qlw clren mahak xiaojiaqi idkwim cherudim9 yzhwang chenleo towords lubronzhan caoyu0 lorlor luyuncan mmallad vovoma w359405949 yummykaya maxwellxxx zhangf911 yiliaofan shenh10 mrwangxc koodex chenkaigithub inginx soloice shyamalschandra clion0003 xiaofengleo mbrukman yanbin-ha oddukgi potatoxin ydm2011 yodamaster eagle518 ray-jing vshanyiao innoaus easyfmxu yonglehou fanyeren zhengxle hhmmhh520 7472741 superneo simonzhangsm fenggeorgeyu theseusyang home201448 lizhanhui williamren teardemon hbi980 superlzhi fmx jhouyang bingo cyclefusion vanloswang rhelfan liufeiran123 xc889078 kuner jackyzzy libin8216 u20024804 abioy yiiwood hesix kaisatec leonzhu rchunping moerstw lidaoduo7 seishuchen htoooth zsjss yoya darbean kiranvajja ecnumjc thurday substancettt 551618846 rm3l c11 tomaseliu

mr4c's Issues

When is there going to be a version for 10.10.2+?

I am waiting for that day. Thanks! Instructions to build would be nice too!

Build_all error

All dependencies are installed. But it still doesn't work properly.

error.txt

I'm having a lot of errors, too, don't know if that is normal.

collect2: error: ld returned 1 exit status
makefile:152: recipe for target 'exe/run_tests' failed
make: *** [exe/run_tests] Error 1

Those are the last output lines on terminal.
I'm running on ubuntu.

I am unable to ./build_all.

I get stuck with the following:

test/cpp/tests/MR4CTests.cpp:17:33: fatal error: cppunit/TestFixture.h: No such file or directory
#include <cppunit/TestFixture.h>

Can you provide instructions on how to get the dependencies for the different platforms?

Logger does not run on YARN.

When a program runs on YARN, context.log does not run.
If a program runs on YARN, does Logger normally output log?

Could you put links to the tar.gz of all the dependencies in the README?

Can you please do that ASAP?

Thanks!

Shyamal

my map phase looks like looping forever

After installation, I'd like to practice which I learned from tutorial. Below is my test code to read and write file in hdfs.

  void executeAlgorithm(AlgorithmData& data, AlgorithmContext& context)
  {
    Dataset* input = data.getInputDataset("input");
    Dataset* outputHist = data.getOutputDataset("output");

    std::set<DataKey> keys = input->getAllFileKeys();
    for ( std::set<DataKey>::iterator i = keys.begin(); i != keys.end(); i++ ) {
      DataKey myKey = *i;
      std::cout << myKey << std::endl;

      RandomAccessFile* randIn = input->getDataFileForRandomAccess(myKey);
      DataFile * fileOut = new DataFile("text/plain");
      WritableRandomAccessFile * randOut = outputHist->addDataFileForRandomAccess(myKey, fileOut);
      size_t fileSize = randIn->getFileSize();
      randOut->setFileSize(fileSize);
      size_t read = 0;
      size_t bufferSize = 1024 * 1024;
      char* data = new char[bufferSize];
      read = randIn->read(data, bufferSize);
      delete[] data;
      randIn->close();
      randOut->close();
    }
  }

Map phase looks like looping forever thought input is not so big. The percentage goes from 0% to some% again and again(Now it's executing for 6+hours).

...
21345055 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 17% reduce 0%
21346070 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 0% reduce 0%
21359136 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 33% reduce 0%
21360141 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 44% reduce 0%
21465685 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 0% reduce 0%
21479771 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 44% reduce 0%
21585825 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 0% reduce 0%
21598939 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 44% reduce 0%
...

Something wrong in my test code? Let me know please.
Thanks,
Jun

build_all error (run_tests error)

I downloaded mr4c on my production Hadoop-2.6.0 and I installed all dependencies

•ant (1.9.3)
•java (1.8)
•ivy (2.2)
•make (3.8.1)
•g++ (4.8.4)
•log4cxx (0.10.0)
•jansson (2.2.1)
•cppunit (1.12.1)
•proj4 (4.7.0)
•gdal (1.10)

and when I ran : ./build_all I get the following error

…..

2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte fil
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
2015-07-30 13:09:44,621 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 4 byte file
: OK
MR4C::TestMBTilesUtils::testZoomToTileCount : OK
MR4C::TestMBTilesUtils::testTileCountToZoom : OK
MR4C::TestMBTilesUtils::testZoomToTileSize : OK
MR4C::TestMBTilesUtils::testTileSizeToZoom : OK
MR4C::TestMBTilesUtils::testFlipTileY : OK
MR4C::TestMBTilesUtils::testTileIndex : OK
MR4C::TestMBTilesUtils::testToBoundingBox : OK
MR4C::TestMBTilesUtils::testToBoundingBoxSet : OK
MR4C::TestMBTilesUtils::testFindTilesInBoundingBox : OK
MR4C::TestImageBox::testEqual : OK
MR4C::TestImageBox::testNotEqualWidth : OK
MR4C::TestImageBox::testNotEqualHeight : OK
MR4C::TestImageBox::testNotEqualBound : OK
MR4C::TestImageBox::testNotEqualX : OK
MR4C::TestImageBox::testNotEqualY : OK
MR4C::TestImageBox::testAssignment : OK
MR4C::TestImageBox::testCopy : OK
MR4C::TestImageBox::testXYCalcs : OK
MR4C::TestImageBox::testWindow : OK
MR4C::TestImageBox::testCreateZeroWidth : OK
MR4C::TestImageBox::testCreateZeroHeight : OK
MR4C::TestImageBox::testCreateNegativeX1 : OK
MR4C::TestImageBox::testCreateNegativeY1 : OK
MR4C::TestBoundingBox::testEqual : OK
MR4C::TestBoundingBox::testNotEqualNW : OK
MR4C::TestBoundingBox::testNotEqualSE : OK
MR4C::TestBoundingBox::testAssignment : OK
MR4C::TestBoundingBox::testCopy : OK
MR4C::TestBoundingBox::testCreateFromEastNorth : OK
MR4C::TestBoundingBox::testCreateFromLatLon : OK
MR4C::TestBoundingBox::testCreateFromNormMerc : OK
MR4C::TestBoundingBox::testCreateInvalidOrientationX : OK
MR4C::TestBoundingBox::testCreateInvalidOrientationY : OK
MR4C::TestBoundingBox::testCreateZeroWidth : OK
MR4C::TestBoundingBox::testCreateZeroHeight : OK
MR4C::TestBoundingBox::testIntersecting : OK
MR4C::TestBoundingBox::testIntersect : OK
MR4C::TestLatLonCoord::testEqual : OK
MR4C::TestLatLonCoord::testNotEqualLat : OK
MR4C::TestLatLonCoord::testNotEqualLon : OK
MR4C::TestLatLonCoord::testAssignment : OK
MR4C::TestLatLonCoord::testCopy : OK
MR4C::TestLatLonCoord::testDegreesToRadians : OK
MR4C::TestLatLonCoord::testRoundTripConversion : OK
MR4C::TestEastNorthCoord::testEqual : OK
MR4C::TestEastNorthCoord::testNotEqualEast : OK
MR4C::TestEastNorthCoord::testNotEqualNorth : OK
MR4C::TestEastNorthCoord::testAssignment : OK
MR4C::TestEastNorthCoord::testCopy : OK
MR4C::TestNormMercCoord::testEqual : OK
MR4C::TestNormMercCoord::testNotEqualX : OK
MR4C::TestNormMercCoord::testNotEqualY : OK
MR4C::TestNormMercCoord::testAssignment : OK
MR4C::TestNormMercCoord::testCopy : OK
MR4C::TestNormMercCoord::testFromLatLon : OK
MR4C::TestNormMercCoord::testRoundTripConversion : OK
MR4C::TestGDALCoordTrans::testRoundTripConversion : OK
MR4C::TestGDALMemoryFile::testMemoryFile2015-07-30 13:09:44,710 INFO mr4c.native.dataset.SimpleDataFileSource src/cpp/impl/dataset/SimpleDataFileSource.cpp(99): Freeing 153481 byte file
: OK
MR4C::TestGDALLocalFile::testLocalFile : OK
MR4C::TestGDALUtils::testCopyGDALDataset : OK
MR4C::TestGDALUtils::testCopyGDALImage : OK
MR4C::TestGDALUtils::testGenerateCoordinateTransform : OK
make: *** [run_tests] error 1

I would appreciate any guidance and help on this issue

another build issue

the last step ./tools/build_yarn. on the end ,all test failed ,no other error.
do-test:
[junit] TEST com.google.mr4c.nativec.ExternalAlgorithmDataSerializerTest FAILED
[junit] TEST com.google.mr4c.nativec.ExternalAlgorithmSerializerTest FAILED
[junit] TEST com.google.mr4c.nativec.ExternalDatasetSerializerTest FAILED
[junit] TEST com.google.mr4c.nativec.jna.JnaExternalEntryTest FAILED
[junit] Tests FAILED

BUILD FAILED
then in dist directory ,no any jar package ,how to solve?

build jnaerate failed

I tried to build jnaerate object in ant, but failed, see below error message. I have already copied jna-3.4.0.jar and jnaerator-0.11.jar to java/lib. Is there any other depended jars jnaerate need? Thanks

jnaerate:

BUILD FAILED
java.lang.NoClassDefFoundError: com.ochafik.lang.jnaerator.parser.Identifier$QualifiedIdentifier
at java.lang.Class.forNameImpl(Native Method)
at java.lang.Class.forName(Class.java:237)
at org.apache.tools.ant.taskdefs.ExecuteJava.execute(ExecuteJava.java:135)
at org.apache.tools.ant.taskdefs.Java.run(Java.java:764)
at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:218)
at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:132)
at org.apache.tools.ant.taskdefs.Java.execute(Java.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.Target.performTasks(Target.java:385)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
at org.apache.tools.ant.Main.runBuild(Main.java:758)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: java.lang.ClassNotFoundException: com.ochafik.lang.jnaerator.parser.Identifier$QualifiedIdentifier
at org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1400)
at org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1341)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1094)
at java.lang.ClassLoader.loadClass(ClassLoader.java:827)
... 23 more

ABOUT gdaal ISSUE

Now I want save file into hdfs ,then read them by gdal interface(RastIO) ,then write result to other HDFS file by gdal interface (etc :create ,,rastIO) .Can anyone tell me the code outline of this process?
Dataset* input = data.getInputDataset("imageIn");
std::set keys = input->getAllFileKeys();
Dataset* outputHist = data.getOutputDataset("out");
for ( std::set::iterator i = keys.begin(); i != keys.end(); i++ ) {
DataKey myKey = i;
DataFile skyFile = input->getDataFile(myKey);
std::string inputFileName = myKey.toName("__")+"_input";
GDALMemoryFile inputMemFile(inputFileName, *skyFile);
GDALDataset * poDataset = inputMemFile.getGDALDataset();

    //then how to create new hdfs file  by gdal and add to outputHist 
    .....

A question about the keyspace.

   I have run the examples. And I have a question about the kayspace. I read the description of keyspace. 
   "The keyspace is an index of unique elements in the dataset. 
    Each key refers to a particular peice of the data without having to keep track of a lot paths. This can be especially handy when we are operating on a large cluster where all of the files are not necessarily local."
   I think it means that every file has a key instead of every record in the file. Am I right?

MessageConsumer API missing callback to recieveMessage (for http)

The current implementation of mr4c provides a default HttpMessageHandler which provides plumbing to send messages to a registered topic URL.

This works, and sends topic messages to the remote URL.

HttpMessageHandler

There is currently no mechanism to receive any http messages via a same / similar route.

The API is there MessageConsumer for which you implement virtual void receiveMessage(const Message& msg)

However mr4c is missing a Http Message Handler for receiving messages, and sending them through to any registered MessageConsumers

That's a great step

Can it build and run on windows?

The destructor of the algorithm base class is not a virtual function.

The destructor of the algorithm base class is not a virtual function, so the inherited class is not released correctly.

when i build_yarn, do-test errors

there are no logs for this,what can i do?who can help me?thanks.

do-compile:

compile:

do-test:
[junit] TEST com.google.mr4c.AlgoRunnerTest FAILED
[junit] TEST com.google.mr4c.algorithm.AlgorithmDataTest FAILED
[junit] TEST com.google.mr4c.algorithm.AlgorithmSchemaTest FAILED
[junit] TEST com.google.mr4c.config.ConfigDescriptorTest FAILED
[junit] TEST com.google.mr4c.config.ConfigUtilsTest FAILED
[junit] TEST com.google.mr4c.config.algorithm.AlgorithmConfigTest FAILED
[junit] TEST com.google.mr4c.config.algorithm.DimensionConfigTest FAILED
[junit] TEST com.google.mr4c.config.category.CategoryBuilderTest FAILED
[junit] TEST com.google.mr4c.config.category.CategoryConfigTest FAILED
[junit] TEST com.google.mr4c.config.category.CategoryParserTest FAILED
[junit] TEST com.google.mr4c.config.category.MR4CConfigBuilderTest FAILED
[junit] TEST com.google.mr4c.config.category.MR4CConfigTest FAILED
[junit] TEST com.google.mr4c.config.diff.DiffConfigTest FAILED
[junit] TEST com.google.mr4c.config.execution.DatasetConfigTest FAILED
[junit] TEST com.google.mr4c.config.execution.DirectoryConfigTest FAILED
[junit] TEST com.google.mr4c.config.execution.ExecutionConfigTest FAILED
[junit] TEST com.google.mr4c.config.execution.LocationsConfigTest FAILED
[junit] TEST com.google.mr4c.config.execution.MapConfigTest FAILED
[junit] TEST com.google.mr4c.config.execution.PatternMapperConfigTest FAILED
[junit] TEST com.google.mr4c.config.resources.LimitSourceTest FAILED
[junit] TEST com.google.mr4c.config.resources.ResourceConfigTest FAILED
[junit] TEST com.google.mr4c.config.resources.ResourceLimitTest FAILED
[junit] TEST com.google.mr4c.config.resources.ResourceRequestTest FAILED
[junit] TEST com.google.mr4c.config.site.ClusterConfigTest FAILED
[junit] TEST com.google.mr4c.config.site.SiteConfigTest FAILED
[junit] TEST com.google.mr4c.config.test.AlgoTestConfigTest FAILED
[junit] TEST com.google.mr4c.content.RelativeContentFactoryTest FAILED
[junit] TEST com.google.mr4c.content.S3CredentialsTest FAILED
[junit] TEST com.google.mr4c.dataset.DataFileTest FAILED
[junit] TEST com.google.mr4c.dataset.DatasetDiffTest FAILED
[junit] TEST com.google.mr4c.dataset.DatasetTest FAILED
[junit] TEST com.google.mr4c.dataset.DatasetTransformerTest FAILED
[junit] TEST com.google.mr4c.dataset.LogsDatasetBuilderTest FAILED
[junit] TEST com.google.mr4c.hadoop.ClusterTest FAILED
[junit] TEST com.google.mr4c.hadoop.DataKeyListTest FAILED
[junit] TEST com.google.mr4c.hadoop.DataLocalizerTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CArgumentParserTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CGenericOptionsParserTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CGenericOptionsTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CInputFormatTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CInputSplitTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CMRJobTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CMapperTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CRecordWriterTest FAILED
[junit] TEST com.google.mr4c.hadoop.MR4CReducerTest FAILED
[junit] TEST com.google.mr4c.keys.BasicDataKeyFilterTest FAILED
[junit] TEST com.google.mr4c.keys.BasicElementFilterTest FAILED
[junit] TEST com.google.mr4c.keys.CompoundDataKeyTest FAILED
[junit] TEST com.google.mr4c.keys.DataKeyComparatorTest FAILED
[junit] TEST com.google.mr4c.keys.DataKeyDimensionTest FAILED
[junit] TEST com.google.mr4c.keys.DataKeyElementTest FAILED
[junit] TEST com.google.mr4c.keys.DataKeyUtilsTest FAILED
[junit] TEST com.google.mr4c.keys.DimensionBasedKeyFilterTest FAILED
[junit] TEST com.google.mr4c.keys.ElementTransformerTest FAILED
[junit] TEST com.google.mr4c.keys.KeyDimensionPartitionerTest FAILED
[junit] TEST com.google.mr4c.keys.KeyTransformerTest FAILED
[junit] TEST com.google.mr4c.keys.KeyspaceDimensionTest FAILED
[junit] TEST com.google.mr4c.keys.KeyspacePartitionerTest FAILED
[junit] TEST com.google.mr4c.keys.KeyspaceTest FAILED
[junit] TEST com.google.mr4c.keys.SimpleDataKeyTest FAILED
[junit] TEST com.google.mr4c.mbtiles.MBTilesFileTest FAILED
[junit] TEST com.google.mr4c.mbtiles.TileFormatTest FAILED
[junit] TEST com.google.mr4c.mbtiles.TileKeyTest FAILED
[junit] TEST com.google.mr4c.mbtiles.TileTest FAILED
[junit] TEST com.google.mr4c.message.MessageTest FAILED
[junit] TEST com.google.mr4c.metadata.MetadataArrayTest FAILED
[junit] TEST com.google.mr4c.metadata.MetadataFieldTest FAILED
[junit] TEST com.google.mr4c.metadata.MetadataKeyExtractorTest FAILED
[junit] TEST com.google.mr4c.metadata.MetadataKeyTest FAILED
[junit] TEST com.google.mr4c.metadata.MetadataListTest FAILED
[junit] TEST com.google.mr4c.metadata.MetadataMapTest FAILED
[junit] TEST com.google.mr4c.nativec.ExternalAlgorithmDataSerializerTest FAILED
[junit] TEST com.google.mr4c.nativec.ExternalAlgorithmSerializerTest FAILED
[junit] TEST com.google.mr4c.nativec.ExternalDatasetSerializerTest FAILED
[junit] TEST com.google.mr4c.nativec.jna.JnaExternalEntryTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.BeanBasedAlgorithmSerializerTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.BeanBasedDatasetSerializerTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.BeanBasedKeyspaceSerializerTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.algorithm.AlgorithmSchemaBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.dataset.DataFileBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.dataset.DatasetBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.keys.DataKeyBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.keys.DataKeyElementBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.metadata.MetadataArrayBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.metadata.MetadataFieldBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.metadata.MetadataKeyBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.metadata.MetadataListBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.bean.metadata.MetadataMapBeanTest FAILED
[junit] TEST com.google.mr4c.serialize.json.JsonAlgorithmBeanSerializerTest FAILED
[junit] TEST com.google.mr4c.serialize.json.JsonConfigSerializerTest FAILED
[junit] TEST com.google.mr4c.serialize.json.JsonDatasetBeanSerializerTest FAILED
[junit] TEST com.google.mr4c.serialize.json.JsonKeyspaceBeanSerializerTest FAILED
[junit] TEST com.google.mr4c.serialize.json.JsonPropertiesSerializerTest FAILED
[junit] TEST com.google.mr4c.sources.ArchiveDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.BinaryDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.CompositeKeyFileMapperTest FAILED
[junit] TEST com.google.mr4c.sources.DiskFileSourceTest FAILED
[junit] TEST com.google.mr4c.sources.FilesDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.HDFSFileSourceTest FAILED
[junit] TEST com.google.mr4c.sources.HeterogenousFileSourceTest FAILED
[junit] TEST com.google.mr4c.sources.InMemoryArchiveSourceTest FAILED
[junit] TEST com.google.mr4c.sources.InMemoryFileSourceTest FAILED
[junit] TEST com.google.mr4c.sources.LogsDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.MBTilesDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.MapFileSourceDFSTest FAILED
[junit] TEST com.google.mr4c.sources.MapFileSourceLocalTest FAILED
[junit] TEST com.google.mr4c.sources.MetafilesDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.PatternKeyFileMapperTest FAILED
[junit] TEST com.google.mr4c.sources.RandomAccessFileSinkTest FAILED
[junit] TEST com.google.mr4c.sources.RandomAccessFileSourceTest FAILED
[junit] TEST com.google.mr4c.sources.SimpleDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.SourceUtilsTest FAILED
[junit] TEST com.google.mr4c.sources.StagedDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.sources.TransformedDatasetSourceTest FAILED
[junit] TEST com.google.mr4c.util.ByteBufferInputStreamTest FAILED
[junit] TEST com.google.mr4c.util.CollectionUtilsTest FAILED
[junit] TEST com.google.mr4c.util.CombinatoricUtilsTest FAILED
[junit] TEST com.google.mr4c.util.CustomFormatTest FAILED
[junit] TEST com.google.mr4c.util.MR4CLoggingTest FAILED
[junit] TEST com.google.mr4c.util.NamespacedPropertiesTest FAILED
[junit] TEST com.google.mr4c.util.PartitionerTest FAILED
[junit] TEST com.google.mr4c.util.PathUtilsTest FAILED
[junit] TEST com.google.mr4c.util.SetAnalysisTest FAILED
[junit] Tests FAILED

BUILD FAILED
/home/zhangsh/code/mr4c/java/build.xml:231: if=true

Total time: 11 seconds

Anything wrong in the way to use getBytes() in my code?

  void executeAlgorithm(AlgorithmData& data, AlgorithmContext& context)
  {
    Dataset* input = data.getInputDataset("input");

    std::set<DataKey> keys = input->getAllFileKeys();
    for ( std::set<DataKey>::iterator i = keys.begin(); i != keys.end(); i++ ) {
      DataKey myKey = *i;
      DataFile* myFile = input->getDataFile(myKey);
      long fileSize = myFile->getSize();
      char * fileBytes=myFile->getBytes();
    }
  }

Adding the last line char * fileBytes = myFile->getBytes();, failed while without the line, succeeded.
Anything wrong in my code?

build_all issue

g++ -std=c++0x -I./src/cpp/api -I./test/cpp/api -fPIC -Wall -o objs/suite/Suite.o -c test/cpp/suites/Suite.cpp
g++ -std=c++0x -Wall -rdynamic -o exe/run_tests ./objs/impl/context/Logger.o ./objs/impl/context/Message.o ./objs/impl/context/AlgorithmContext.o ./objs/impl/algorithm/AlgorithmRunner.o ./objs/impl/algorithm/AlgorithmAutoRegister.o ./objs/impl/algorithm/Algorithm.o ./objs/impl/algorithm/AlgorithmRegistry.o ./objs/impl/algorithm/AlgorithmConfig.o ./objs/impl/algorithm/AlgorithmData.o ./objs/impl/error/ErrorReporter.o ./objs/impl/error/JsonErrorSerializer.o ./objs/impl/error/Error.o ./objs/impl/util/IOUtil.o ./objs/impl/util/MR4CEnvironment.o ./objs/impl/util/StackUtil.o ./objs/impl/util/MR4CLogging.o ./objs/impl/util/Properties.o ./objs/impl/util/MR4CTempFiles.o ./objs/impl/keys/KeyspaceDimension.o ./objs/impl/keys/DataKey.o ./objs/impl/keys/DataKeyBuilder.o ./objs/impl/keys/KeyspaceBuilder.o ./objs/impl/keys/Keyspace.o ./objs/impl/keys/DataKeyDimension.o ./objs/impl/keys/DataKeyElement.o ./objs/impl/metadata/MetadataKey.o ./objs/impl/metadata/MetadataField.o ./objs/impl/metadata/MetadataMap.o ./objs/impl/metadata/MetadataList.o ./objs/impl/metadata/MetadataElement.o ./objs/impl/metadata/MetadataArray.o ./objs/impl/metadata/Primitive.o ./objs/impl/catalog/DimensionCatalog.o ./objs/impl/catalog/ImageTypes.o ./objs/impl/serialize/json/JsonAlgorithmConfigSerializer.o ./objs/impl/serialize/json/JsonCommonSerializer.o ./objs/impl/serialize/json/JsonKeyspaceSerializer.o ./objs/impl/serialize/json/JsonDatasetSerializer.o ./objs/impl/serialize/json/JsonPropertiesSerializer.o ./objs/impl/serialize/json/JanssonUtil.o ./objs/impl/serialize/json/JsonSerializerFactory.o ./objs/impl/serialize/json/JsonAlgorithmSerializer.o ./objs/impl/serialize/SerializerRegistry.o ./objs/impl/dataset/SimpleDataFileSource.o ./objs/impl/dataset/DataFile.o ./objs/impl/dataset/DatasetContext.o ./objs/impl/dataset/LocalTempFile.o ./objs/impl/dataset/LocalDataFileSink.o ./objs/impl/dataset/LocalDataFileSource.o ./objs/impl/dataset/Dataset.o ./objs/impl/dataset/DataFileSource.o ./objs/impl/external/CExternalDataFile.o ./objs/impl/external/CExternalContext.o ./objs/impl/external/ExternalDatasetContext.o ./objs/impl/external/ExternalDataFile.o ./objs/impl/external/CExternalDataset.o ./objs/impl/external/CExternalAlgorithmData.o ./objs/impl/external/ExternalRandomAccessFileSource.o ./objs/impl/external/CExternalEntry.o ./objs/impl/external/ExternalEntry.o ./objs/impl/external/ExternalAlgorithmData.o ./objs/impl/external/CExternalDataFileSource.o ./objs/impl/external/CExternalAlgorithm.o ./objs/impl/external/ExternalAlgorithmSerializer.o ./objs/impl/external/ExternalContext.o ./objs/impl/external/ExternalDataFileSink.o ./objs/impl/external/ExternalDataset.o ./objs/impl/external/ExternalAlgorithmDataSerializer.o ./objs/impl/external/CExternalRandomAccessFileSource.o ./objs/impl/external/CExternalDataFileSink.o ./objs/impl/external/ExternalRandomAccessFile.o ./objs/impl/external/ExternalRandomAccessFileSink.o ./objs/impl/external/ExternalDatasetSerializer.o ./objs/impl/external/ExternalAlgorithm.o ./objs/impl/external/ExternalDataFileSource.o ./objs/impl/external/CExternalRandomAccessFileSink.o ./objs/test/context/ContextTests.o ./objs/test/context/TestMessage.o ./objs/test/algorithm/TestAlgorithmConfig.o ./objs/test/algorithm/TestAlgorithmData.o ./objs/test/algorithm/AlgorithmTests.o ./objs/test/algorithm/AlgorithmDataTestUtil.o ./objs/test/error/ErrorTests.o ./objs/test/error/TestJsonErrorSerializer.o ./objs/test/error/TestError.o ./objs/test/util/TestArrayUtil.o ./objs/test/util/UtilTests.o ./objs/test/util/TestMR4CTempFiles.o ./objs/test/util/TestIOUtil.o ./objs/test/util/TestProperties.o ./objs/test/util/TestMR4CLogging.o ./objs/test/util/TestMR4CEnvironment.o ./objs/test/keys/TestKeyspaceDimension.o ./objs/test/keys/TestDataKey.o ./objs/test/keys/KeyspaceTestUtil.o ./objs/test/keys/TestKeyspace.o ./objs/test/keys/TestKeyspaceBuilder.o ./objs/test/keys/TestDataKeyDimension.o ./objs/test/keys/KeysTests.o ./objs/test/keys/TestDataKeyBuilder.o ./objs/test/keys/TestDataKeyElement.o ./objs/test/metadata/TestPrimitive.o ./objs/test/metadata/TestMetadataList.o ./objs/test/metadata/TestMetadataArray.o ./objs/test/metadata/TestMetadataKey.o ./objs/test/metadata/TestMetadataMap.o ./objs/test/metadata/TestMetadataField.o ./objs/test/metadata/MetadataTests.o ./objs/test/multithread/MultithreadTests.o ./objs/test/multithread/TestExternalRandomAccessFileSinkMultithread.o ./objs/test/multithread/SimultaneousThreadRunner.o ./objs/test/multithread/TestExternalRandomAccessFileMultithread.o ./objs/test/multithread/TestDatasetMultithread.o ./objs/test/multithread/TestDataFileMultithread.o ./objs/test/MR4CTests.o ./objs/test/serialize/json/TestJsonAlgorithmConfigSerializer.o ./objs/test/serialize/json/TestJsonKeyspaceSerializer.o ./objs/test/serialize/json/TestJsonAlgorithmSerializer.o ./objs/test/serialize/json/TestJsonPropertiesSerializer.o ./objs/test/serialize/json/JsonTests.o ./objs/test/serialize/json/TestJsonDatasetSerializer.o ./objs/test/dataset/TestLocalDataFileSink.o ./objs/test/dataset/TestDataFile.o ./objs/test/dataset/TestLocalDataFileSource.o ./objs/test/dataset/TestDataset.o ./objs/test/dataset/DatasetTests.o ./objs/test/dataset/DatasetTestUtil.o ./objs/test/dataset/TestSimpleDataFileSource.o ./objs/test/dataset/TestLocalTempFile.o ./objs/test/external/TestExternalEntry.o ./objs/test/external/ExternalTests.o ./objs/test/external/TestExternalAlgorithmDataSerializer.o ./objs/test/external/TestExternalDatasetSerializer.o ./objs/suite/Suite.o -llog4cxx -ljansson -lcppunit
/usr/local/glibc/lib/libpthread.so.0: undefined reference to `memcpy@GLIBC_2.14'
collect2: error: ld returned 1 exit status
make: *** [exe/run_tests] Error 1
how to solve? i installed glibc 2.14 and give its lib path in LD_LIBRARY_PATH variable.

Implement MessageConsumer and sendMessage using Apache Kafka

Implement a KafkaMessageHandler for sending and receiving messages via Apache Kafka

MR4C does not use HADOOP_CONF_DIR

MR4C uses the hard-coded directory /etc/hadoop/conf as the location for the Hadoop configuration files. It would been more convenient to be able to control this location via the standard Hadoop environment variable HADOOP_CONF_DIR.