coursework's People
Forkers
michaeldye cloofa dyejon rajeshthallam nathanieljblack anewm rboberg sjstanley badpaper qyjoan sarahneff sayantansatpati maktrix16 malinimittal favrecr raybuhr rockb1017 iy-yi yiransheng prabhakar9 ericwhyne jmjasper nkrishnaswami sahaba rocket-ron thomasatkins kutsumigit latuji kyungguyeo venamax letslego yanglinfang saifrahmed nandu163 mahedi-kaysar satishkt j-z0 gunnarklee abessou divpat18 motasem-salem prajp haroonchoudery saurabhjaju2 midsjoe2016 keriwheatley dbalck kevinjfoley jillzhang kbelsvik firstthefacts chabobo ysun1020 noopliskin gsubraman anusham1990 timothymalt planetceres mikegruzynski ucbiyyq steveyang90 anhnguyendepocen nathannam kippig goku8765 sohagdesai mamrotha saurav-datta acarl005 ye-pang hbulcha cameronbell75 vamseeragi yankinschweiz kckenneth ucb-mids revgizmo-mids hodgsim zijianwu jade-chiachunhou drminix haoyuzhang89 vanbantam limdomcoursework's Issues
Additional instructions / typo
Hi all -
I was able to get this running successfully. A few points that may help other people out:
- I didn't have a masters file in the /usr/local/hadoop/etc/hadoop folder. Creating the file and adding the master line worked just fine.
- mapred-site.xml needs to be copied from mapred-site.xml.template before making any edits
- There's a broken line on the config text for yarn-site.xml, if you copy paste and don't join the line you'll get an error starting the namenode services:
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</
name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
Should be:
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
CENTOS7_64 should be CENTOS_7_64 in lab 1
The slcli create command in lab 1 should have CENTOS_7_64 as the OS.
[HW2] global name 'RSA' is not defined
Students are getting errors related to 'RSA', for example per https://ucbischool.slack.com/archives/C1VFF2Y3U/p1495496191162070
[root@saltmaster ~]# salt-cloud -p sl_centos7_small mytests
[INFO ] salt-cloud starting
[INFO ] POST https://api.softlayer.com/xmlrpc/v3.1/SoftLayer_Account
[ERROR ] There was a profile error: global name 'RSA' is not defined
Resolved the issue by installing dependencies first, code below should be added to instructions
yum install -y python-pip && pip install SoftLayer apache-libcloud
yum install openssl-devel
yum install gcc
yum install python-devel
pip install M2Crypto
curl -o /tmp/install_salt.sh -L https://bootstrap.saltstack.com && sh /tmp/install_salt.sh -Z -M git 2015.5
HW5 needs submission info
We need to update HW5 to specify what the students turn in.
HW6 missing a command
Need to include instruction on which directory to run from. Need to cd to README dir before running spark shell
HW3 IP addresses
Homework 3 requests that IP addresses are created, but are never used.
Need submission info
There is no submission information for hw5. We need to update them to reflect the deliverables.
[HW1] Authentication Error
Getting authentication error caused by using the wrong username for slcli
Instructions should specify to use the SL id displayed when viewing API key.
Issues in lab 5: hdfs_2gram_data_load
- the link '(see Hadoop v1 Installation)' is broken (it points to https://github.com/MIDS-scaling-up/coursework/blob/master/week5/hw/version_1/README.md) which is a 404
- after setup,
/usr/local/hadoop/contrib/streaming/hadoop-streaming-1.2.1.jar
(in Part 2: Load data) is not valid---maybe this should be updated to/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.1.jar
? - when I fix the previous issue with the suggested workaround, and then do 'Part 2: Load data', I get a lot of errors like the following:
Container [pid=32025,containerID=container_1443213849013_0005_01_000002] is running beyond virtual memory limits. Current usage: 159.9 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
...
ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!
Invalid keyword 'str' for variable 'master_type'
Hello
When running the command
' salt-cloud -p sl_centos7_small mytestvs'
I am running into this error:
Invalid keyword 'str' for variable 'master_type'
Anyone else having the same problem?
HW9 Java Installation for Spark
The Spark installation indicates installing JDK 1.8, but apparently the latest CENTOS 7 x64 image / RPMs installs both 1.7 and 1.8 and sets the default to 1.7. To correct this, issue the following command:
[root@spark1 ~]# alternatives --config java
There are 2 programs which provide 'java'.
Selection Command
1 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre/bin/java
*+ 2 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91-2.6.2.1.el7_1.x86_64/jre/bin/java
Enter to keep the current selection[+], or type selection number: 1
Once done, do java --version to check that the correct version is selected.
How to suppress Spark logging?
The output is buried in the sea of log messages. Tried having a log4j.properties files in the project directory with spark log level set to WARN but doesn't seem to work.
log4j.logger.org.apache.spark=WARN
Week 2 lab has backwards instructions
SSL generation should happen before changing the ssl.sls file.
Week 2 homework: SL no longer offers VLANs
We need to remove the "Explore the VLAN Ordering Form" section because SL doesn't offer this option any longer.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.