Comments (9)
doddys
commented on July 2, 2024
Skipping HDFS Lab: Replicate to another cluster since using internal IP
from sebc.
doddys
commented on July 2, 2024
Starting HDFS Lab: Test HDFS throughput
from sebc.
doddys
commented on July 2, 2024
Added user doddys to linut
sudo useradd -u 2701 doddys
Create User Dir in HDFS
[centos@hadoop5 ~]$ sudo -u hdfs hadoop fs -mkdir /user/doddys
[centos@hadoop5 ~]$ sudo -u hdfs hadoop fs -chown dodys /user/doddys
from sebc.
doddys
commented on July 2, 2024
Generated 10GB file with 32MB block size
[centos@hadoop5 ~]$ time sudo -u doddys hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar teragen -Ddfs.blocksize=33554432 -Dmapreduce.job.maps=4 100000000 /user/doddys/tgen512m
18/05/15 03:02:49 INFO client.RMProxy: Connecting to ResourceManager at hadoop5.expecc.com/172.30.1.187:8032
18/05/15 03:02:50 INFO terasort.TeraGen: Generating 100000000 using 4
18/05/15 03:02:50 INFO mapreduce.JobSubmitter: number of splits:4
18/05/15 03:02:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526343652645_0002
18/05/15 03:02:50 INFO impl.YarnClientImpl: Submitted application application_1526343652645_0002
18/05/15 03:02:50 INFO mapreduce.Job: The url to track the job: http://hadoop5.expecc.com:8088/proxy/application_1526343652645_0002/
18/05/15 03:02:50 INFO mapreduce.Job: Running job: job_1526343652645_0002
18/05/15 03:02:58 INFO mapreduce.Job: Job job_1526343652645_0002 running in uber mode : false
18/05/15 03:02:58 INFO mapreduce.Job: map 0% reduce 0%
18/05/15 03:03:16 INFO mapreduce.Job: map 8% reduce 0%
18/05/15 03:03:19 INFO mapreduce.Job: map 15% reduce 0%
18/05/15 03:03:21 INFO mapreduce.Job: map 23% reduce 0%
18/05/15 03:03:22 INFO mapreduce.Job: map 25% reduce 0%
18/05/15 03:03:25 INFO mapreduce.Job: map 27% reduce 0%
18/05/15 03:03:27 INFO mapreduce.Job: map 31% reduce 0%
18/05/15 03:03:28 INFO mapreduce.Job: map 32% reduce 0%
18/05/15 03:03:31 INFO mapreduce.Job: map 35% reduce 0%
18/05/15 03:03:33 INFO mapreduce.Job: map 38% reduce 0%
18/05/15 03:03:34 INFO mapreduce.Job: map 41% reduce 0%
18/05/15 03:03:37 INFO mapreduce.Job: map 43% reduce 0%
18/05/15 03:03:39 INFO mapreduce.Job: map 47% reduce 0%
18/05/15 03:03:40 INFO mapreduce.Job: map 49% reduce 0%
18/05/15 03:03:43 INFO mapreduce.Job: map 52% reduce 0%
18/05/15 03:03:45 INFO mapreduce.Job: map 55% reduce 0%
18/05/15 03:03:46 INFO mapreduce.Job: map 57% reduce 0%
18/05/15 03:03:49 INFO mapreduce.Job: map 59% reduce 0%
18/05/15 03:03:51 INFO mapreduce.Job: map 62% reduce 0%
18/05/15 03:03:52 INFO mapreduce.Job: map 64% reduce 0%
18/05/15 03:03:55 INFO mapreduce.Job: map 66% reduce 0%
18/05/15 03:03:57 INFO mapreduce.Job: map 68% reduce 0%
18/05/15 03:03:58 INFO mapreduce.Job: map 70% reduce 0%
18/05/15 03:04:01 INFO mapreduce.Job: map 72% reduce 0%
18/05/15 03:04:03 INFO mapreduce.Job: map 75% reduce 0%
18/05/15 03:04:04 INFO mapreduce.Job: map 76% reduce 0%
18/05/15 03:04:07 INFO mapreduce.Job: map 77% reduce 0%
18/05/15 03:04:09 INFO mapreduce.Job: map 79% reduce 0%
18/05/15 03:04:10 INFO mapreduce.Job: map 81% reduce 0%
18/05/15 03:04:13 INFO mapreduce.Job: map 83% reduce 0%
18/05/15 03:04:15 INFO mapreduce.Job: map 87% reduce 0%
18/05/15 03:04:21 INFO mapreduce.Job: map 93% reduce 0%
18/05/15 03:04:27 INFO mapreduce.Job: map 98% reduce 0%
18/05/15 03:04:29 INFO mapreduce.Job: map 100% reduce 0%
18/05/15 03:04:29 INFO mapreduce.Job: Job job_1526343652645_0002 completed successfully
18/05/15 03:04:29 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=595120
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=344
HDFS: Number of bytes written=10000000000
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=319232
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=319232
Total vcore-milliseconds taken by all map tasks=319232
Total megabyte-milliseconds taken by all map tasks=326893568
Map-Reduce Framework
Map input records=100000000
Map output records=100000000
Input split bytes=344
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=1051
CPU time spent (ms)=138890
Physical memory (bytes) snapshot=723148800
Virtual memory (bytes) snapshot=6264893440
Total committed heap usage (bytes)=911736832
org.apache.hadoop.examples.terasort.TeraGen$Counters
CHECKSUM=214760662691937609
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=10000000000
real 1m42.428s
user 0m5.707s
sys 0m0.319s
from sebc.
doddys
commented on July 2, 2024
Finish terasort test. Results is in the lab folder.
There are 2 test , 10GB and 1GB.
from sebc.
doddys
commented on July 2, 2024
Staring Snapshot Lab
from sebc.
doddys
commented on July 2, 2024
Finish Snapshot Lab, result is submitted
from sebc.
doddys
commented on July 2, 2024
Starting HDFS Lab: Enable HDFS HA
from sebc.
doddys
commented on July 2, 2024
Finished JDFS Lab
I am not sure what is the extra step required after performing the wizard.
When I tested the manual failover, it works. Zookeeper Failover controller also active.
from sebc.
Related Issues (20)
- Challenges Setup HOT 2
- Install Database HOT 4
- Install CM HOT 4
- Install CDH HOT 5
- Test HDFS HOT 2
- Kerberize cluster HOT 5
- Install Sentry HOT 3
- Exit-test: Setup HOT 1
- Exit-test: Install Database HOT 2
- Exit-test: Install CM HOT 2
- Exit-test: Install CDH HOT 7
- Exit-test: Test HDFS HOT 1
- Exit-test: Kerberize cluster HOT 3
- Exit-test: Install Sentry HOT 3
- Exit-test: CM chart HOT 1
- Resource Lab HOT 6
- CM managing multiple different CDH Cluster
- CM Monitoring Lab HOT 5
- Security Lab HOT 14
- HUE Lab HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sebc.