seattletestbed / custominstallerbuilder Goto Github PK

The demokit should contain the user's keys (or at least it should be possible to download one that does). It's a bit of a pain in the butt to download 3 separate things.

This task should not be started until ticket 29 is resolved

In the link https://seattle.poly.edu/wiki/CustomInstallerBuilderConfiguration we see that repy_runtime should be placed under ~/custominstallerbuilder/repy_runtime but actually we build repy_runtime under ~ dir. when we configure apache it throws an error if repy_runtime is not found under ~ dir. i tried to modify wsgi.py to actually tell apache to search repy_runtime under ~/custominstallerbuilder/ dir. Do we need to mention the same in our installation instruction?

The custominstallerbuilder config contains the public key which gets embedded in the "service vessel" (i.e. the small reserved space+resources that contains the nodemanager and softwareupdater logs on any install),

The implications of this are twofold:

• We can track installs that came from CIBs that we didn't set up, because they will advertise "our" service vessel pubkey. (Nice for us to have, but not strictly necessary).
• The admins of those CIB sites have no way to collect NM/SWUP logs from their nodes. (A pain for them, but they might not be aware of this in the first place).

I think that the latter is way more important to fix than the fact that this means we will lose one way of accounting for nodes across testbeds. (See SeattleTestbed/nodemanager#72 for an alternative proposal for tracking nodecounts.)

Opinions?

Clearinghouse and Custominstallerbuilder use python's xmlrpclib to communicate with each other. xmlrpclib in turn is based on python's httplib which was changed in Python 2.7.9+ to raise an exception during handshake when issuing a request via HTTPS and the server uses a self-signed certificate or the CommonName of the certificate does not match the requested host. (c.f. PEP 474 for further background.)

While this behavior is actually preferred in a production environment, it is a nuisance in a testing setup. Possible remedies are:

• sign your certificates, e.g. with let's encrypt

• Add an unverified ssl context to requests in debug mode, e.g. :

  # https://github.com/SeattleTestbed/clearinghouse/blob/master/website/html/views.py#L1110
  if settings.DEBUG:
      xmlrpclib.ServerProxy(settings.SEATTLECLEARINGHOUSE_INSTALLER_BUILDER_XMLRPC, context=ssl._create_unverified_context())
  else:
      xmlrpclib.ServerProxy(settings.SEATTLECLEARINGHOUSE_INSTALLER_BUILDER_XMLRPC)

@yyzhuang reports (and @asm582 confirms) that the combination of updated Clearinghouse and updated Custominstallerbuilder does not work together correctly. "Updated" in this context refers to the changes made to support Django 1.6.x and mod_wsgi as a replacement for mod_python.

Although the CIB website works fine and builds working installers, the XML-RPC interface (that the CH uses to request "crediting" installers for a CH user account) is broken. From the CIB Apache error log:

[Mon Mar 30 07:44:13 2015] [error] Error while generating https://sensibilityclearinghouse.poly.edu/custominstallerbuilder/xmlrpc/
[Mon Mar 30 07:44:13 2015] [error]   Traceback (most recent call last):
[Mon Mar 30 07:44:13 2015] [error]     File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
[Mon Mar 30 07:44:13 2015] [error]     File "/home/custominstallerbuilder/custominstallerbuilder/xmlrpc/views.py", line 59, in xmlrpc_handler
[Mon Mar 30 07:44:13 2015] [error]   AttributeError: 'WSGIRequest' object has no attribute 'raw_post_data'

I'm working to fix this.

Feedback from the SAS 2018 Sensibility Hackathon: The Custom Installer Builder creates new key pairs for users if you don't upload public keys for them. After the Build step, three separate downloads are offered:

• An archive containing all private keys (if any),
• An archive containing all public keys, including those that were uploaded,
• And finally, the customized installers themselves.

Also, the keys archives each contain and thus extract into a directory, public_keys/ and private_keys/, respectively. Navigating these dirs to just copy over the keys to a demokit dir has been described as "a pain", particularly on Windows.

A suggested solution is to,

1. Put all keys / key pairs into a single archive (to avoid handling multiple archives)
2. Make the archive "flat", i.e. not contain a subdirectory (so that its contents will extract right into the dir that the archive is placed in)

The place to modify is this bit in packager.py. We'll need to check that the workflow where pubkeys are uploaded keeps working, as our Clearinghouses depend on it.

@yyzhuang, @JustinCappos, let me hear what you think.

The CIB repo contains a directory named local_template which the operator is supposed to rename to local when setting up their install. The idea for this layout probably was to be able to update the template from SVN (back then) without overwriting the local config, and relying on the user to merge any changes into the live config.

This workflow makes little sense with Git (which will warn you and prompt you to merge when it detects deviations between your local repo and origin), so let's not waste the operator's time and rename the local_template directory to local in the source repo right away. This also requires a change to the CIB installation docs.

Note: This issue, and some additional remarks retaining to the existence of two settings files, are based on discussion with @lukpueh over at #17.

Trunk contains many referenes to blackbox.cs.washington.edu. Many of them must be updated in order to function correctly. Of the rest, they are part of previous published research so we shouldn't change those.

The current custominstallerbuilder HTML has lots of mentions of "Seattle", Seattle-specific terminology, and "Custom Installer Builder". These are hardcoded which makes it hard for people forking the custominstallerbuilder to, e.g., use a different testbed name than we do.

The obvious fix is to replace said instances by constants that will be substituted by Django using the {{ CONSTANT_NAME }} syntax in the HTML code, and defining these constants in local_template/settings.py. (Grep the sources for MEDIA_URL to get an idea how this approach works).

(See also SeattleTestbed/clearinghouse#139)

I was trying to regenerate cib.publickey and cib.privatekey and for some weird reason its giving me four values into the cib.publickey file.

I wonder if the CIB installation docs are misleading. When I setup the CIB django app as suggested I get the following error when I try to download the installer:

[Errno 2] No such file or directory: '/home/cib/custominstallerbuilder/html/static/installers/base/seattle_linux.tgz'

As far as I understand the custom installer bundle is created by common/packager.py which builds/packages it from the base installer at BASE_INSTALLER_ROOT -- a default setting found in settings_base.py that points to the directory seen in the error message above.

The docs don't mention to change this setting. But they do suggest to install the base installers at /var/www/dist this section.

Am I missing something?

The custominstallerbuilder app currently runs on Django 1.6 (which just had its final, end-of-life version released). Django 1.8 will be the new Long-Term Supported version, so this is should be our goal.

(See SeattleTestbed/clearinghouse#152 for potential constraints, and first tests.)

This is to reflect the current state of the project, given that we are no longer under the GENI project.

Pages complete:

Pages in progress:
'''(file/link references left unchanged)'''
https://seattle.cs.washington.edu/wiki/SeattleGeniProductionHttp
https://seattle.cs.washington.edu/wiki/SeattleGeniInstallation
https://seattle.cs.washington.edu/wiki/EducationalAssignments/TakeHome
https://seattle.cs.washington.edu/wiki/ContributorsPage
https://seattle.cs.washington.edu/wiki/SeattleGeniClientLib
https://seattle.cs.washington.edu/wiki/SeattleGeniApi

https://seattle.cs.washington.edu/wiki/SeattleLib (stopped after "serialize.py")

Pages not started (INCOMPLETE):

(This was originally filed as SeattleTestbed/dist#133).

The custominstallerbuilder installer packager script takes a base installer for an OS and adds the vesselinfo file to it (which contains the share sizes and user keys for the customized Seattle VMs). For the Linux installer, it does so by gunzipping the base installer, piping it through tar, appending file ./seattle/seattle_repy/vesselinfo to the tar file, then regzipping it. Note that all of the other contents are in a slightly different dir, with identical expansion on an interactive terminal, but still distinguishable in the tarball:

seattle/seattle_repy/deserialize.repy
seattle/seattle_repy/math.repy
seattle/seattle_repy/servicelookup.repy
# etc., etc.
./seattle/seattle_repy/
./seattle/seattle_repy/vesselinfo 

When the person installing Seattle on their machine from the completed, downloaded installer tarball uses a GUI tool like Gnome's File Roller to extract the contents, they might only choose the "seattle" directory, but miss the "." dir that is shown separately. Too bad, "." contains the vesselinfo!

If you happen to extract the tarball on the command line instead, tar+shell result in "seattle" and "./seattle" expanding to the same directory, and the vesselinfo file will be placed where it should be.

The fix will be to make packager.py use the correct dir (without the leading ./).

packager.py takes the compressed base installers, unpacks them, adds an installer-specific vesselinfo file to the resulting tar file or directory and then re-compresses to create the final, downloadable installer.

However, storing the base installers in compressed form really only generates CPU and disk load, without solving any problems. Modifying the tar file did get us into trouble previously, see #9.

I therefore propose to store the base installers in uncompressed form, i.e. as plain directories, and only compress them as the last step of the packager.