berkeley-cs162 / vagrant Goto Github PK

View Code? Open in Web Editor NEW

59.0 53.0 55.0 491 KB

The Vagrant Virtual Machine for CS 162 students.

Shell 16.47% Puppet 7.17% Makefile 0.23% Perl 52.80% C 19.82% Dockerfile 0.79% Vim Script 2.72%

vagrant's Introduction

DEPRECATED.

PLEASE SEE THE NEW WORKSPACE INSTEAD

Vagrant VM for CS162

Running on Vagrant

Download the source code (don't forget --recursive if you're using git clone) and run vagrant up inside the root of the project directory.
The Vagrantfile specifies ubuntu/bionic64 as the base box for this VM, so Vagrant will download that box from the Internet, which may take some time.
Once the download is complete, Vagrant will import the VM appliance and run our Puppet provisioner.

This provisioner is the only modification we've made to the base box, which means that you should be able to run it on any x86 Ubuntu 18.04 machine.

Running on your own infrastructure

Vagrant+Virtualbox is the recommended way to run the VM. If you don't have VT-x, or you would prefer to use your own infrastructure, you can run the provisioner manually.

1. Set up a Ubuntu 18.04 VM

You can set this up on AWS, DigitalOcean, your home server, etc. Anything that runs this operating system is okay. Both 64-bit and 32-bit versions are okay, but the underlying system must be x86, not ARM.

2. Log in to your VM as root, and download the source code for this project.

If you can't log in as root, just type sudo su when you log in, and you will become root (assuming you have sudo permissions).

The easiest way to download the source code is through a git clone. You may need to install git first, so:

$ apt-get install git
$ git clone --recursive https://github.com/Berkeley-CS162/vagrant.git

3. Add the vagrant user (even if you aren't using vagrant)

There will be some things that are installed for the vagrant user and some things that are placed in the vagrant user's home directory, so regardless of whether you are using vagrant, you will need a user named vagrant with a home directory at /home/vagrant for this to work.

You should use the adduser utility script to do this, NOT useradd (unless you are already familiar with the command-line switches of useradd).

$ adduser vagrant

Make sure your password isn't easily guessable! The personal information section doesn't matter.

4. Give password-less sudo access to the vagrant user (not optional)

You will need to edit /etc/sudoers to give vagrant password-less sudo access. This just means you need to add one line to the end of /etc/sudoers. Open up /etc/sudoers with your favorite command-line text editor and add this line:

vagrant ALL=(ALL) NOPASSWD: ALL

5. Install puppet using the package manager

You will need to install Puppet. This is as easy as:

apt-get install puppet

6. Run the provisioner

Make sure you are in the project root, and then just run the provisioner.

$ cd path/to/project/vagrant/
$ sudo puppet apply manifests/site.pp --modulepath modules/

7. Log in as the vagrant user (not optional)

You need to actually log in as the vagrant user now, since the .bashrc has ~/.bin in the PATH for the vagrant user, which has some nifty utilities. You can do this by logging in with ssh, or if you are already logged in with root, just run:

$ su vagrant

You're done! You can use this box just the same as you would a vagrant box. Just remember to log in with the vagrant user when you're developing code.

SMB Server

There is a Samba server that you can connect to with any SMB client. You can log in with vagrant as both the username and the password. You can also use the vboxsf mount on /vagrant that is connected to the host's home folder. Or, you can set up rsync or sshfs. You can either add your own SSH key to the vagrant user's authorized keys (don't replace the entire file, because Vagrant needs to be able to log in as well to manage the VM), or you can use SSH's IdentityFile option to use the same private key that Vagrant does. The IP address of the VM should be 192.168.162.162 always, unless another interface on the host is using that subnet.

Building the distributable box

The dist/ directory has components to build the final box file. I'm trying a thing where the box image and components are all from the ubuntu/bionic64 box, and we just put a puppet provisioner on top of it. To build

vagrant box add ubuntu/bionic64
cd dist/
(Replace the MAC address in Vagrantfile with real MAC address)
./make.sh

~~You can find the real MAC address to use in the Vagrantfile in the ~/.vagrant.d/boxes/ubuntu-..../ directory.~~

Update Fall 2019: This semester we upgraded the VM to Ubuntu 18.04. I couldn't find a way to make the scripts in dist work with this version of Ubuntu. Ubuntu starts to boot but then hangs at some point in the boot procedure. I've left my attempt at porting this to Ubuntu 18.04 in the dist directory, so the code there DOES NOT WORK. I didn't spend too much time on it, so it's possible the fix is trivial---maybe a future TA will find out where I went wrong and revive this.

Until then, just build the vagrant box for distribution like normal. Run vagrant up to provision the VM locally, vagrant halt to stop the VM, and then vagrant package --vagrantfile Vagrantfile --output fall2019.box (choose the correct semester name). After the provisioning the VM (in between vagrant up and vagrant halt) you may choose to SSH in the VM, clear the apt cache, and zero out the disk so that the packaged .box file is smaller after compression. That can be done by running sudo apt clean, sudo dd if=/dev/zero of=/EMPTY bs=1M, and sudo rm -f /EMPTY inside the VM. You may also choose to clear the history with history -c (and clear the .bash_history if it is present). Here's a pretty good tutorial: https://scotch.io/tutorials/how-to-create-a-vagrant-base-box-from-an-existing-one.

vagrant's People

Contributors

Stargazers

Watchers

vagrant's Issues

can't specify id_rsa to use

I met this git issue that said

ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

when trying to access my private cs162 git repo within the vagrant box and I have set up a id_rsa key on the host machine. But the problem is I how two id_rsa in my .ssh dir for my two different github account. But it seems like the vagrant box didn't forward the right one. Is there a way to update the vagrant file so that we can forward a specific id_rsa key to use?

Branch cloud-setup is missing rust

Solution: merge main into cloud-setup

The command Pintos can not find

Hi,

I used the vagrant to build the virtual machine, when I try command pintos run, the system threw an error like pintos can not be found. I also try to update the PATH variable， but the $HOME/.bin does not exist. Could anyone give me some help? Thank you.

Container-first strategy

I have had various issues (described later) with provisioning VMs using the existing Puppet method. If you have confidence in the Docker image,¹ as it seems you do, there is a different approach you can take which I find very elegant:

You maintain a Dockerfile, as opposed to a Puppet configuration
- This can be an incremental change: slowly remove dependencies from Puppet and bring them into Docker. For the sake of the general strategy, it doesn't matter whether you internally use Puppet; it only matters from an optimization and readability standpoint. The minimal change uses the existing Docker image. See #18 for optimization.²
You as CS 162 build images with this Dockerfile
- You can create different tags for different use cases. For example, if you don't need rustup (the autograder almost surely doesn't), you can shave about 1.2 GB off the image!
- Your Docker build will be much faster than any student's Puppet provisioning, because yours will be cached and parallelized. It also only has to happen ~once.
Students simply install docker/podman on their x86 machine, and pull and run the image.
- Docker commit is now possible, in case ever necessary
  - For the most part, there is no state on the virtual machine besides git config, so you're not adding any duplicated work by using containers. Docker commit can be done by the student (or script) after setting up git, and then this new image should contain all necessary state for the remainder of the class session.
  - Of course, you can also use volumes or bind mounts as well
- Non-x86-users can emulate at the Docker level (not recommended), or they can still run in a VM. But in this case, the VM is not a critical dependency, can be swapped with others, and is very fast and easy to provision.
- Users not running in a VM will experience much better performance
- The only issue I forsee in the entire end-to-end process is certain students (those on instructional machines) not having root access to run the daemon. I have no single solution for this use case,³ but most people will not have this issue because Docker on Mac is supposed to support most x86 images, and also direct use of QEMU is possible. It looks like the use of instructional machines is already discouraged, so this is probably not making the problem worse.

I spent over a week in compute just provisioning the VM, and had various issues at various stages on various machines.⁴ This effort would no longer be necessary.

i.e., you think containerization is enough and you don't need a dedicated VM ↩
You can also keep the existing Puppet provisioner and leave the current approach as a backup, though the remainder of this document³ suggests that this is not necessary. ↩
162 can use their privileged account to start the Docker daemon and run the containers as non-root on behalf of the student, so that student the attaches to an existing container. 162 can run their own x86 VMs (say, within an instructional machine). Students can use a free instance on Oracle Cloud. Students can work directly on the instructional machines as they are doing now. ↩ ↩²
One particularly terrible issue that occurs on only some of my machines is that the provisioning makes the machine extremely slow (ls in home dir takes 20 seconds and new ssh connections can't be established). And the build gets stuck at the stage after matplotlib. ↩

C utilities should be built by the provisioner

From a student:

When I tried running 'pintos run alarm-multiple' I got an error stating squish-pty couldn't be executed (cannot execute binary file: Exec format error). To fix this, I went into /home/vagrant/.bin and ran 'gcc squish-pty.c -o squish-pty'. I did this because the error message suggested that squish-pty had been compiled for a 64-bit arch. After this, 'pintos run alarm-multiple' worked fine and ran all the tests.

I just wanted to let the staff know about this in case the provisioner should be modified to build squish-pty from source during provisioning.

Issues with Vagrant Up on Ubuntu 18.04 as host OS

I tried downloading this repo and running vagrant up. I get the following output.

The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

/sbin/ifdown 'enp0s8' || true
/sbin/ip addr flush dev 'enp0s8'
# Remove any previous network modifications from the interfaces file
sed -e '/^#VAGRANT-BEGIN/,$ d' /etc/network/interfaces > /tmp/vagrant-network-interfaces.pre
sed -ne '/^#VAGRANT-END/,$ p' /etc/network/interfaces | tac | sed -e '/^#VAGRANT-END/,$ d' | tac > /tmp/vagrant-network-interfaces.post

cat \
  /tmp/vagrant-network-interfaces.pre \
  /tmp/vagrant-network-entry \
  /tmp/vagrant-network-interfaces.post \
  > /etc/network/interfaces

rm -f /tmp/vagrant-network-interfaces.pre
rm -f /tmp/vagrant-network-entry
rm -f /tmp/vagrant-network-interfaces.post

/sbin/ifup 'enp0s8'

Stdout from the command:



Stderr from the command:

bash: line 5: /sbin/ifdown: No such file or directory
bash: line 21: /sbin/ifup: No such file or directory

To Fix:
After running vagrant up run
vagrant ssh
sudo apt-get install ifupdown
Reload the VM.

I do not know if we can install this beforehand to avoid this issue.

Switch to only Docker

Currently build times are ~30 minutes using puppet. We can optimize this by utilizing docker staged builds which cache results of building programs. A new repo for this project should be created.

Intentionally optimize Docker image

The current Docker image is huge—even with recent optimizations (#13, #16), it is around 3.9 GB!

Part of why is just that it has so many packages. But part of why is that it is oriented around the existing Puppet provisioner, which isn't amenable to the types of optimizations that container images are.

For example, it is a lot easier in a Dockerfile to initiate an install of some software, and then copy over only the relevant portions (leaving behind caches and even parts of the program which will never be used). You can download each dependency in a parallel stage, and selectively copy it over to the final image, dramatically improving performance of builds and rebuilds. Better, you can copy the software directly from prebuilt official images; the software providers have done the hard work of isolating the necessary components, and the software is already built.

Intentionally optimizing for containerized loads might be worthwhile. This is best done together with #17; see that issue for the general strategy, as well as other suggestions such as slim versions.¹

But with a lot more work, it is also possible to rewrite the Dockerfile so that it completely bypasses Puppet, while still having students use Puppet in a VM. ↩