some useful scripts
-
prerequisites:
- pip install paramiko:Paramiko is a Python (2.7, 3.4+) implementation of the SSHv2 protocol [1], providing both client and server functionality.While it leverages a Python C extension for low level cryptography (Cryptography), Paramiko itself is a pure Python interface around SSH networking concepts.
- pip install configparser:used to parse the configuration file add_new_node_config.ini
-
pip install tqdm: set progress
-
documents
- to use these scripts, you need to do following preparations:
- download the archive file hadoop-version and decompose it
- configure what you want to configure,maybe you have to pay attention to the following files:
- hadoop-version/etc/hadoop/hadoop-env.sh
- export JAVA_HOME=java_location
- hadoop-version/etc/hadoop/core-site.xml
- hadoop-version/etc/hadoop/hdfs-site.xml
- hadoop-version/etc/hadoop/slaves
- hadoop-version/etc/hadoop/hadoop-env.sh
- add all ip-hostname mappings in the cluster to the file /etc/hosts
- run add_system_env_path.py: this script will do the followings:
- install java-1.8.0-openjdk-devel and add it's related environment variables JAVA_HOME, JRE_HOME, CLASSPATH and PATH to the file /etc/profile;
- find hadoop's location you have just configurated and use it as the value of a environment variable HADOOP_HOME and also add hadoop-related environment variable PATH in the file /etc/profile .
- run authentication.py
- you must provide following information:
- input a user name which you will use it as the user in linux to manage the whole cluster;
- input a password which will be used to the password of the provided user;
- input root password to login into the linux cluster.
- this script will do the followings:
- use the user name you provided to create a user in every machine in cluster;
- execute ssh-keygen command to produce key-pair in each machine of cluster;
- execute ssh-copy-id command for leting any two machines in the cluster log on to the other machine without password;
- copy all the ip-host mappings concerned about machine in the cluster in the file /etc/hosts on the running machine to the other machines in the cluster;
- shutdown the firewall.
- you must provide following information:
- run configurate_distribution.py
- you should provide root password;
- this script will copy the hadoop you have configured on running machine to the other machines in cluster.
- run add_new_node_for_cluster.py
- you must complete the configuration file add_new_node_config.ini
- ips: these IPs indicate which machines you want to add to the cluster
- hostnames: the machines' name corresponding ips
- user: the user who manage the whole cluster
- password: the login password of the user you specified
- this script will do the followings:
- add ip-host mappings from the configuration file to /etc/hosts of each machine in cluster;
- add hostname file to hadoop_version/etc/hadoop/slaves in every machine in cluster;
- install openjdk and add it's related to the file /etc/profile for machines configured in the configuration file;
- add hadoop-related environment variables to the file /etc/profile for machines configured in the configuration file;
- distribute modified hadoop package to each machine configured in the configuration file;
- create new user for machines configurated in the configuration file to manage the cluster;
- let any two machines in the cluster log on to the other machine without password;
- shutdown the firewall in these machines configurated in the configuration file;
- execute hadoop command yarn-daemon.sh start nodemanager and start-balamcer.sh to create new datanode thread in machines configurated in the configurate file.
- you must complete the configuration file add_new_node_config.ini
- to use these scripts, you need to do following preparations: