ucberkeley / bce Goto Github PK

Trying out the latest version, I get this error when I run "BCE_PROVISION=BCE packer build BCE-14.10-amd64.json" and the script stops executing:

1 error(s) occurred:

• Bad source 'packer-vbox-out.log': stat packer-vbox-out.log: no such file or directory

  ---

If I run "touch packer-vbox-out.log" first the script does start building correctly.

Since Globus Connect Personal v2.1.3 was included in BCE, I thought I'd pass on this security announcement. The vulnerability sounds fairly low-severity according to the email and I don't think many people are (yet) using GCP in BCE, but it is likely to be more used over time as more people begin using the the Savio DTN. So, upgrading this in whatever your next release might be would be a good.

Impact of Logjam vulnerability on Globus Services

On May 20th, 2015, a new vulnerability known as “Logjam” was discovered, as described in this
NIST announcement. We reviewed the severity and impact to Globus services and posted the
findings in our support forum. We do not anticipate additional updates on this issue.

The vulnerability was identified and assessed to be of low severity due to the difficulty to complete
an attack. An enhancement was expedited and implemented to prevent the vulnerability in the
future. Please see the forum post for all recommended actions.

In particular, we ask that Globus Connect Personal users (which includes almost all Globus users)
update to the latest version by following the instructions here.

If you have any concerns about this issue, please contact our support team.

Aaron asked for mosh and git-annex to be installed post-install (perhaps only on EC2?).

Installation via apt-get update gives this error (illustrating here with mosh)

Processing triggers for libc-bin (2.19-0ubuntu6.5) ...
Processing triggers for ufw (0.34~rc-0ubuntu2) ...

*** Please tell me who you are.

Run

git config --global user.email "[email protected]"
git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: unable to auto-detect email address (got 'root@master.(none)')
E: Problem executing scripts DPkg::Post-Invoke 'if [ -x /usr/bin/etckeeper ]; then etckeeper post-install; fi'
E: Sub-process returned an error code

also, create a patch for this for spring-2015 and link to on website

Here are the json changes needed to build with spot instance pricing (https://www.packer.io/docs/builders/amazon-ebs.html#spot_price):

Add to the amazon-ebs builder in BCE-summer-2015.json:
"spot_price": "auto",
"spot_price_auto_product": "Linux/UNIX",

(Actual patch to come sometime.)

From a small sample size (n=3, k=1), this drops the m3.medium instance pricing from $0.06-$0.07 to $0.001 (83% cost reduction) and only took an extra few minutes to get the spot instance, even at noon.

We could then use the cost-savings to use a beefier instance than m3.medium and potentially reduce the build time. I'm not sure how sensitive the build time is to the instance power though.

I'm also guessing that since the build time is not that long, we won't need to worry about the instance getting terminated before the build completes. I don't know how realistic that is.

need to set up provisioning to use package versions saved from an initial creation of a given release

this also needs to deal with making sure all dependencies used saved source packages as well

Ryan plans to look into this.

I had a screen resizing bug on my machine (OS X, Yosemite). Updating the guest additions by hand solved this.

Strangely, after a logout and log back in (following upgrade of guest additions), the system reverted to the stock XFCE background. Not sure if this is related, but I can file a separate bug report if anyone likes.

Using shared folders changes the permissions of everything (it makes things executable). I don't know that we can fix this, but we should at least document it.

I thought I had this issue in the the old repo. I'm guessing we should migrate issues that we still want to address?

BCE needs to have an automated test suite to catch release bugs. It should focus on the functioning of important applications and the desktop environment.

The kernel prints "ACPI PCC mode failed" and systemd is prints "starting version 219". These messages overwrite the plymouth boot graphic and the former distress users.

(ACPI PCC patched)
http://thread.gmane.org/gmane.linux.acpi.devel/73411/focus=73585

(systemd fixed for 15.10)
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1432171

The fixes are not yet available in vivid-backports.

It would be handy to install dropbox by default to ease file transfer, esp. for cloud environments. From https://www.dropbox.com/en/help/246 it seems straightforward:

Ubuntu

Add the following line to /etc/apt/sources.list. Replace natty with your build's name below.

deb http://linux.dropbox.com/ubuntu natty main
To import our GPG keys into your apt repository, perform the following command from your terminal shell:

$ sudo apt-key adv --keyserver pgp.mit.edu --recv-keys 5044912E

Have a currently running process, closed a terminal, history was empty. This is easy to reproduce with something like the following as a running process, and click the 'x' on the terminal window:

while [ 0 -lt 1 ] ; do echo 'o crap'; done

Any ideas?

Hi,
What would it take to get packer to create a kvm-compatible image (qcow2)?

Just wanted to open an issue here and point out that I'm overhearing conversations in the python training along these lines. It is absolutely crucial that appropriate introduction / instruction is given around BCE, or it simply should not be used. We should develop some standards around this.

We'll be having a post-intensive discussion about this, but I wanted to open an issue here. I'll be adding notes to the issue as I get details.

VPN connection works over command line openconnect but I could not figure out how to configure this successfully using Network Manager (GUI). See notes: https://github.com/cboettig/berkeley-linux-config/blob/master/vpn.md

This seems like an easy windows approach: http://mobaxterm.mobatek.net/

I had a list of potential options somewhere... but don't know where now :\

This makes it easier to see what's being typed. Dav has an example - just ask him.

Noting for future reference, when building on OSX you can change the --audio parameter in the json from "alsa" to "coreaudio" for the sound to be built in properly. Maybe there is a way to detect Linux vs. OSX and set that automatically, e.g. within the makefile?

Is anyone else getting this error with Beautiful Soup failing? I luckily noticed the error message during the build and thought I'd note it since everything else seems to have worked.

virtualbox-iso: Downloading BeautifulSoup-3.2.1.tar.gz
virtualbox-iso: Traceback (most recent call last):
virtualbox-iso: File "<string>", line 20, in <module
virtualbox-iso: Complete output from command python setup.py egg_info:
virtualbox-iso: Traceback (most recent call last):
virtualbox-iso:
virtualbox-iso: File "<string>", line 20, in <module
virtualbox-iso:
virtualbox-iso: ----------------------------------------
virtualbox-iso:     Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-delbg1u1/BeautifulSoup

Would it be possible to use CRAN's version of R rather than the default one in Ubuntu? I have found that I have to re-install R from CRAN since the current one on BCE is pretty old (3.0.2 rather than ~3.2.0) and has bugs that prevent parallel machine learning in caret from working.

Switching to Ubuntu 15.04 is helpful but still gets us a somewhat old version of R (3.1) - https://launchpad.net/ubuntu/vivid/i386/r-base-core

According to http://askubuntu.com/questions/506560/how-to-install-package-of-r we would need to add a CRAN mirror to the apt packages list that is specific to the Ubuntu version, then that will be used when installing R.

The default keybindings with xfce4 conflict with fundamental keyboard shortcuts in Emacs.

This is a known problem and has a straightforward solution.

We may want to consider documenting this in a BCE known problems list and/or just changing the default in the next BCE release since it is more likely to affect our community than a changed keybinding in the ibus system.

Details: http://lists.gnu.org/archive/html/help-gnu-emacs/2014-05/msg00282.html

I first opened IBus ($ ibus-setup) and there I changed Ctrl<space> into
Alt<space>. I hope it won't destroy anything, but at least C-<SPC> now sets
the mark in emacs.

http://www.freedesktop.org/software/systemd/man/os-release.html
http://0pointer.de/blog/projects/os-release.html

Utilize /etc/os-release so that our OS tags are consumable by downstream tools.

NAME="BCE"
VERSION="2015 Fall"
ID="bce"
VERSION_ID="2015-fall"
PRETTY_NAME="BCE 2015 Fall"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:ucberkeley:bce:2015.3"
HOME_URL="https://bce.berkeley.edu/"
BUG_REPORT_URL="https://github.com/ucberkeley/bce/issues"

This is something @davclark does manually before running the machine using the "snapshots" tab (upper right on OS X version, no sure on Win / Lin).

I think focusing on the master branch for bug fixes and new features would be a better model git workflow for this project, particularly given that the modification velocity is somewhat sporadic. It doesn't seem to me that the overhead of a separate dev branch is beneficial for the circumstances of this project. Under the current policy "dev" is the useful branch and "master" is generally out of date, plus there isn't a big userbase that is building from master for whom we need to carefully guard against the minor chance of bugs creeping in.

Right now master is 72 commits behind dev and master hasn't been updated in 6 months. I would argue that under the BCE use case (single digit usage of the source repo, 99% of users are only downloading the generated .ovas or using the AMIs) there is not really a benefit to holding "production" from all of the improvements that have been made since January.

Here are a few related articles that I reviewed as food for thought:

1. http://blog.endpoint.com/2014/05/git-workflows-that-work.html
2. http://www.barryodonovan.com/2012/07/03/two-git-branching-models
3. http://nvie.com/posts/a-successful-git-branching-model/

As one proof of concept, Caret is the main machine learning framework in R that does this with a much larger source userbase: https://github.com/topepo/caret

So I would like to put in a vote to remove the "dev" branch approach in favor of simply merging directly to master.

We should provide a page for each release documenting the versions of at least the major software packages. Theoretically this information can be generated from dpkg, conda, R, etc.

The MRAN url copied into /etc/R/Rprofile.site should be https rather than http in order to prevent a big red warning message every time RStudio is opened:

"WARNING: Your CRAN mirror is set to "http://mran.revolutionanalytics.com/snapshot/2015-05-06" which has an insecure (non-HTTPS) URL. The repository was likely specified in .Rprofile or Rprofile.site so if you wish to change it you may need to edit one of those files. You should either switch to a repository that supports HTTPS or change your RStudio options to not require HTTPS downloads.

To learn more and/or disable this warning message see the "Use secure download method for HTTP" option in Tools -> Global Options -> Packages."

Seems like we should use an educational editor, ya?

I just spoke with someone from Haas, though, and there is apparently a community of Spyder users. So, perhaps we should switch to that...

There are OVAs and Vagrant boxes here:

https://cloud-images.ubuntu.com/ubuntu-core/15.04/core/stable/current/

That's probably cheaper than using EC2, but the last time I checked, the packer handling of OVAs was not fully implemented, but I don't remember what the issue was. This is the most recent docs:

https://packer.io/docs/builders/virtualbox-ovf.html

Dav is excited about snappy because it has the promise of composability (as compared to Docker, which only works with revisions / layers).

I don't know exactly how to do this without the GUI, but I can probably figure it out. The best interpreter link would be ~oski/miniconda/envs/py34/bin/python3.4

For the EC2 AMI target, it could be helpful to have a few benchmark images for comparison and food for thought. I think Louis Aslett's is probably a good comparison: http://www.louisaslett.com/RStudio_AMI/

Unfortunately it doesn't appear to be open source but it does describe some of the key default features:
10GB EBS storage — compact, but enables storage of more sizeable datasets.
Full LaTeX support enabling regular or Sweave document compiles within RStudio.
GDAL dependencies for GIS packages.
GSL and CURL libraries.
ODBC drivers for database access.
Git and Subversion support out of the box.
Stan (RStan) installed and ready to use for Hamiltonian Monte Carlo sampling.
OpenGL R interfaces supported.
Swap space for compiling of large packages on constrained memory instances (such as rugarch).
Defaults to fast SSD storage.
Arbitrary precision arithmetic and number theory libraries supported out of the box (GMP, MPFR, FLINT).
Optimised BLAS for automatically faster matrix operations than base R libraries (OpenBLAS).

At present, one can find it and start an instance using the AMI by going to the EC2 dashboard and then clicking on AMIs tab on the left and then searching "BCE-spring-2015" under "Public Images".

But if one directly from the dashboard clicks on Launch, then at Step 1 of "Choose an AMI", BCE is not listed under the Ubuntu images under Community.

Related to #38, but likely the solution is different.

Currently, if you run the IPython notebook from the application menu, you get the python 2 version, and no way to run the python 3 version. This can be set up with kernelspecs, which I learned about here:

http://stackoverflow.com/questions/28831854/how-do-i-add-python3-kernel-to-jupyter-ipython

(not sure where the official docs are.)

By default, kernelspecs are installed system-wide. The following enables python3 in the notebook for the current user only (omitting --user attempts to install to /usr/local/share/jupyter):

. activate py34
ipython kernelspec install-self --user

A similar approach can make any other python kernel always available. I guess we should do both the base miniconda 2.7 and the miniconda py34?

This could also be part of the setup_ipython_notebook.sh script, which I tend to use elsewhere as well... I'm happy to implement this, but wanted a chance to get other's feedback.

I think we used to have git-annex. It's not there now. I'll add other packages here as I discover them.

• git-annex

[I wanted to mention, @ryanlovett, I miss your old GitHub handle... ]

The general issue is that staff laptops are heavily locked down in terms of the software they can run. However, there are times when a script can replace weeks of clicking. So, it would be of high value to figure out how to enable technical staff to have access to more advanced tools (like those installed in BCE).

@ryanlovett, I'm assigning this to you, as Aaron suggested you are the best equipped to know what makes sense to do both from our perspective and also from Ben Gross' perspective.

The main issue for using BCE is that users can't install VirtualBox. If we could solve that, the rest would be solved inside VirtualBox.

An alternative would be to use a remote solution, but we'd still need to set up use for some set of tools:

• Remote Desktop / X11
• SSH into remote server
• Citrix (this is more an engineering problem on the BCE side - I think staff can get Citrix no problem)

But it would be great if we could get VirtualBox available with BigFix. Maybe aligning our release cycles so we standardize on the version of VirtualBox that's on BigFix (or however it's called now).

One question that I'm still not clear on: can folks install user-land programs in their own directory?

All of this came out of a conversation with @tgallag1, who is a business analyst on campus who wants to be able to run Python.

This seems quite similar to the work @agcinsf is doing in UC Purchasing, and so it might be worth roping him in as well.

cc @aculich

They are stripey on my machine. I can post a screenshot if you don't see the same.

It would be nice if we provided a simple RStudio-server installation script. It's a very handy service that I think a lot of people may use, although maybe not worth auto-enabling by default.

The commands are essentially:

wget https://download2.rstudio.org/rstudio-server-0.99.467-amd64.deb
sudo gdebi rstudio-server-0.99.467-amd64.deb
sudo adduser rstudio

• The adduser will generate a password for the rstudio user. Maybe we could/should use oski instead?
• The latest .deb is posted at https://www.rstudio.com/products/rstudio/download-server/

I will try to implement this sometime if someone doesn't beat me to it.

And in theory, we might be able to use instance meta-data (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) so that people can add a tag for "rstudio-server" when creating the instance so that the script runs without requiring an ssh and manual execution. Not sure how to implement that yet though.

Chris ran into a possibly similar issue on AMD machines on the SCF that we addressed so he can go back to his notes and try to see if this is a related issue.

Alternatively, link numpy to openblas and see if the issue persists. At the moment reinstallation of numpy after install of openblas is NOT detecting openblas.

openblas_info:
libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE

Not sure why as /usr/lib contains openblas-base and libopenblas.so.0

Also, look into whether python/numpy should be linking to system blas (/usr/lib/libblas.so{,.3}), which (after install of openblas) is pointed to openblas via the alternatives system.

Tim Pope has a github repo for this, we can just get this directly: https://raw.githubusercontent.com/tpope/vim-sensible/master/plugin/sensible.vim

I've run into this earlier this summer as well in spring-2015...

Downloading a test 150 Mb file using wget in BCE downloads at ~50 KB/s when the host is a Linux machine. Natively downloading on the host Linux machine it is 100 MB/s. Similarly in BCE on a Mac laptop, downloading is ~50 KB/s.

I thought I requested this... not a big deal, but it's small and decent for looking at git history.

We could use a better readme.md for github marketing. Will try to work on that soon.

Would it be possible to add PyCharm to the Gnome Applications menu? I believe it will be used pretty often in the DLab python trainings, and it seems like it would be a good usability improvement to not require typing "pycharm" on a terminal to run it.

Alternatively, a desktop icon could be the second best option.

Does the summer 2015 release exist as an EC2 AMI? I can only find the spring 2015 release :(

E.g.:

• PyCharm Edu
• Miniconda
• RStudio
• etc.

Enabling this in a Savio-compatible way would allow mobility of code from a local HPC cluster to Azure cloud-based HPC cluster-- without an end-user having to change their code or submit scripts.

See azure-quickstart-templates#422 for further discussion.

appears to be because summer-2015 is Ubuntu 15.04 while spring-2015 was Ubuntu 14.04. Starcluster uses upstart to start up the NFS daemon while 15.04 uses systemd. I'm going to try out a hack with /etc/init.d/nfs passing things through to systemd, but this is on the edge of my sysadmin skills so will also check in with Ryan.

I should also file an issue with Starcluster.

I'm not sure where / why this happens. Not really a problem until someone starts changing their config and then it doesn't work!

Should the MRAN repo be based on the current day that the build is being created, rather than being hardcoded? Or find the most recent MRAN archive to use? http://mran.revolutionanalytics.com/

It seems kind of unfortunate to have to update the URL whenever building or otherwise having packages potentially worth upgrading immediately after the build is created. E.g. right now the MRAN is 2.5 months old.

This affects usage of MPI on BCE, in particular some uses of Rmpi.

Chris is writing up a modification to the post-install script to install openMPI 1.8.4 from source.