guherbozdogan2 / yelp_challenge Goto Github PK

Docker is necessary for building.

• Docker Version: 18.09.2

To establish the execution of spark tasks and docker container run, only the following script should be executed with providing the path of the data file:

	  chmod +x *.sh
	  ./run.sh TAR_GZ_FILE_LOCATION

To test the db, the cqlsh client of cassandra container should be executed as:

	>docker exec -ti  yelp_challenge_cassandra_1 bash
	>cqslh
	>describe test.tables
	>select * from test.photo limit 10

The generated image used to create the "task" named container process (task service docker-compose,yml) is taken from following repository: docker pull guherbozdogan2/repo:latest

The generation has been established via this scala-sbt project via sbt docker plugin(via sbt sbt docker:publishLocal command inside this project folder). The command generates Docker file and it's necessary lib/bin folders (with populated applications). Accordingly the above docker hub repo has been generated from the repo with the following steps: (The below steps are not necessary to do for to run project. Only ./run.sh command execution is enough to run project as since the docker image is alreadily existing in docker hub repo: guherbozdogan2/repo:latest )

    >sbt update clean compile package
    >sbt docker:publishLocal
    >cd docker/target/stage

Change the line of: 'ENTRYPOINT ["/opt/docker/bin/yelp-conversion-task"]' to 'ENTRYPOINT /opt/docker/bin/main-task && /opt/docker/bin/main-test-task' and build local image with:

    >docker build -t repo .

• The spark application is ran lke a standalone spark application. The best ran condition would be creating a new Docker image based on Spark images for Spark Executors (Docker using Mesos' project's spark executors) and using spark-submit instead of standalone application. This solution currently uses a stand alone spark application instead of utilizing spark-submit( spark client/cluster mode).
• Unit tests are missing
• Integration test cases have low coverage and should be automated wth a test framework
• CI tool integration for builds/tests + log server integration for spark logs/docker compose logs analysis

In integration tests, the following 2 cases are tested for each table: (This task executes right after the migration completes and the tests results are written with Success/Error logs output in console currently(With having prefix of "******************" )

• The equivalence of cardinality of RDD data and Cassandra table based on partition key (grouped by partition key)
• Whether sampled 1000 different records (With different partititon keys from RDD) also exist in cassandra table

Sample outputs in console after executing ./run.sh:

******************Testing whether cardinality of RDD is same as CQL Table's cardinality in table:business
******************Success in:Testing whether cardinality of RDD is same as CQL Table's cardinality in table:business

******************Testing randomly sampled 1000 records existence in table:business
******************Success in: Testing randomly sampled 1000 records existence in table:business