homebysix / recipe-robot Goto Github PK

I often run Recipe Robot multiple times in a row while making small tweaks. The end result is only a handful of recipes, but the cumulative count increases on every run.

Might be more accurate not to increment the count if an existing recipe is overwritten (see #32).

Right now the generate_recipes() function and various generate_x_recipe() functions individually determine whether they can proceed, given the info in the facts dict. It might be more efficient to have a single fact_validator() function that centrally determines which recipes can be built and which should be skipped.

Once #18 is built out, we should prompt users to create a GitHub token if they don't already have one. I don't think we should actually handle creation of the tokens, but at least a [WARNING] that tells them it's a good idea would be nice.

Here are the facts we need to gather from the PackageInfo and Payload files inside of an expanded flat package:

• bundle identifier
• install location
• Does it contain an app? If so, extract it and inspect_app().

This is not working 100% reliably at the moment.

And what if the payload contains multiple apps? Do we go with the one with the shortest path, as would be the case below?

$ gunzip -c /Users/elliot/Library/Caches/Recipe\ Robot/expanded/ChronoSync.pkg/Payload | p
ax | egrep "\.app$"
./Applications/ChronoSync.app
./Applications/ChronoSync.app/Contents/Library/LoginItems/ChronoSync Scheduler.app
./Library/Application Support/ChronoSync/ChronoAgentLocal.app

Or should we use the one with the largest file size? (How can we determine file size without extracting first?)

Or should we use the one with a Sparkle feed?

Right now we're doing what AutoPkg does, which is this:

try:
    from recipe_robot_lib import FoundationPlist
except:
    print '[WARNING] importing plistlib as FoundationPlist'
    import plistlib as FoundationPlist

@sheagcraig mentioned that we might use neither FoundationPlist nor plistlib, and instead "use the preferences system correctly." I don't yet know what this means, but I want to make an issue here for reference.

When a png icon file or a recipe already exists in the output directory, should we warn about that? I'm leaning towards no, since I often run Recipe Robot multiple times in a row while making small tweaks, and overwriting existing files is the desired behavior.

If we don't prompt for overwriting, we may want to choose a different default output path than ~/Library/AutoPkg/RecipeOverrides, in order to prevent overwriting existing overrides.

Word on the street is that urllib2 isn't great at downloading large files. Our inspect_download_url() function is almost exclusively going to be handling large files, so is this something we need to consider?

One example of a "chunking" method: https://gist.github.com/gourneau/1430932

Would be useful for the Popen() command that implements #25.

Is there such a thing as a Sparkle feed that doesn't use http://www.andymatuschak.org/xml-namespaces/sparkle as its namespace? If so, Recipe Robot will probably choke on it.

Here's an example of how this might be resolved:
https://github.com/autopkg/autopkg/blob/master/Code/autopkglib/SparkleUpdateInfoProvider.py#L115-L119

Most autopkg repos group recipes into folders by developer name or by product name. Should Recipe Robot do that, or leave it up to the user?

SourceForge requires automated programs to limit themselves to one hit per 30 minutes per feed. In the scenario where somebody is repeatedly trying to create a SourceForge-based recipe, Recipe Robot could exceed this limit.

Potential options:

a. When the same RSS feed is tried a second time, issue an error and say "try again in X minutes."
b. Keep a local cache file of the RSS feed, and refer to that instead of the live version on subsequent runs.

Both rather complex.

We can't count on SourceForgeURLProvider.py to remain at the same place in jessepeterson-recipes forever, so we should have a backup plan if urlopen() fails to retrieve it. I vote we use the copy in recipe_robot_lib as a backup option.

Right now Recipe Robot looks for the files:extra-info tag marked as data in the RSS feed, which seems to work for a subset of SourceForge-hosted files, but not all of them.

For example, here are the filenames and extra-info tags for each file in a GrandPerspective release:

The one in bold is the actual binary we want to download.

As a counterexample, the latest release of VLC has an extra-info tag x86 boot sector (bzip2 compressed data).

Trying to find the unifying factor that determines "yes, this is the binary you want to download."

The function defined here is used to define the user identifier for the autopkg recipes: https://github.com/homebysix/recipe-robot/blob/master/scripts/recipe-robot#L372

It defaults to: com.github.your_name_here

It would be interesting if it could default to: com.github.(configured git user)

Typically this is contained in the file: ~/.gitconfig or ~/.config/git/config or /etc/gitconfig

The format is like an INI file, with sections, keys, values:

[user]
    name = jgstew

Or else, it would default to: com.github.( os.getusername() )

com.github.your_name_here could be a fall back value.

References:

• http://stackoverflow.com/questions/842059/is-there-a-portable-way-to-get-the-current-username-in-python
• https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup
• http://stackoverflow.com/questions/8884188/how-to-read-and-write-ini-file-with-python
• https://bigfix.me/relevance/details/3003662
• https://bigfix.me/relevance/details/3003664

Are there keys other than SUFeedURL and SUOriginalFeedURL that would help us download an app, given its Info.plist?

I'm guessing the answer is no.

Even if a valid output folder is specified, sometimes clicking Reveal Recipes will produce an error.

Going back to the preferences and selecting the same output folder again seems to fix this problem.

Any value set here:

$ defaults read com.elliotjordan.recipe-robot RecipeIdentifierPrefix
com.github.homebysix

Should be used as the default value on next --config run:

$ recipe-robot --config

                      -----------------------------------
                     |  Welcome to Recipe Robot v0.2.5.  |
                      -----------------------------------
                                \   _[]_
                                 \  [oo]
                                   d-||-b
                                     ||
                                   _/  \_

Showing configuration options...

Recipe identifier prefix
This is your default identifier, in reverse-domain notation.
(If you have a GitHub account, it's customary to use com.github.<your GitHub username>.)

[com.github.elliot]: 

It looks like Recipe Robot can get confused about which recipes it's building if the previous app's cache remains in the cache folder (either by using --keep-cache or by interrupting a run).

Steps to reproduce:

1. Start this, but interrupt it when it gets to the unzipping part:

   recipe-robot --verbose http://www.coconut-flavour.com/downloads/coconutBattery_3_2_1.zip
   

2. Run this fully:

   recipe-robot --include-existing --verbose http://delicious-monster.com/downloads/DeliciousLibrary3.zip
   

3. Observe that the resulting recipes are for coconutBattery, not Delicious Library.

Only two hard things in computer science, they say: cache invalidation, naming things, and off-by-one errors.

I suspect something was lost in a refactor. Recipe Robot is no longer adding the download URL input variable and associated URLDownloader argument as it used to.

Correct recipe produced by version 0.0.3 on the left, incorrect recipe produced by version 0.2.3 on the right:

Right now Recipe Robot doesn't know how to create filewave recipes based on pkg downloads.

Here's an example download URL we can use for testing: http://zoom.us/client/latest/ZoomRooms.pkg

Without lots of passing facts around, robo_print can't add messages to the appropriate facts lists (previously "report"). Also, I don't feel like a print function should be tasked with storing lists of messages for later consumption, or for exiting the program.

SO, here's what I'm going to do:

1. Add an error_handler function. It will robo_print a message, and then raise a RoboException.
2. All functions that can potentially call an ERROR will be moved inside of the main try/except/finally; upon catching an ERROR, the message will get added to the local facts object (written immediately), and then sys.exit(1) will get called.
3. All functions that print a WARNING or REMINDER will instead keep a local list to return at the end of the function. Docstrings and invocation will all be updated to match.
4. This allows us to greatly compact robo_print.
5. Finally, I'll refactor the try/except/finally into a robo_runner function, and put all of the error-prone stuff from main in it. All config/args/prefs handling stuff will be orchestrated either by main, or get factored into a setup manager function. This leaves main basically doing just a few lines of work. Thumbs up!

Some of the setup stuff may need its own exception handling written in, since it will be outside of the main recipe generation try/except/finally.

This will ease my OCD over single-responsibility-principle violations!

Because Recipe Robot produces entire recipe chains, you shouldn't be able to select a specific recipe type in preferences unless its parent recipe type is also selected.

Here are the dependencies:

• munki requires download
• pkg requires download
• install requires download
• jss requires pkg
• absolute requires pkg
• sccm requires pkg
• filewave requires download

Running ../recipe-robot/scripts/recipe-robot --config

Results in "too few arguments" error
Simply appending any arbitrary path to the end fixes the issue.
Not sure where this is exactly but probably want to set the input_path required flag to false when running --config

This will handle situations in which the name of the app doesn't match the filename of the app bundle.

When set to True, this will strictly follow the style here. This will be useful for people who are creating recipes they wish to submit to jss-recipes.

The only difference I can think of between official style and Recipe Robot's existing style is which Identifier is used (com.github.homebysix.jss.Foo vs com.github.jss-recipes.jss.Foo), but it might be useful to have this preference setting anyway for future reference.

Right now %NAME%.app is our sole blocking application when creating Munki recipes. However, if we encountered other apps in our travels (specifically while running inspect_pkg()) I think those apps should also be included by default.

Do we want to have Recipe Robot actually run the recipe and make sure the exitcode is zero? I'm leaning towards no, because of the risks of running newly-created .munki, .install, or .jss recipes on a production Mac.

What is the goal of timing the recipe generation?

Is it just for fun?

I think it's fun ;)

But if so, do we need to include timing for the argparsing, welcome message, etc?

Right now Recipe Robot reads all the sparkle:shortVersionString and sparkle:version information for every item in the feed, and does a basic string comparison to determine which item is the "latest" version. However, this fails easily: "999" > "1000"

A better approach would be to get the most recent pubDate.

When somebody chooses Help > Recipe Robot Help, they should arrive at this URL in their default browser.

At the moment, Recipe Robot doesn't build a pkg recipe if there's no bundle_id in facts. However, I don't think we've considered that there are other possible ways to determine the bundle identifier even if we weren't able to collect it during inspect_app().

• Many Sparkle feeds provide bundle identifiers that SparkleUpdateInfoProvider grabs.
• AppDmgVersioner can be used to determine a bundle identifier. As long as we include that before the PkgCreator processor, we should be good to go.

If for example we don't have the bundle identifier when we arrive at generate_pkg_recipe(), it issues an early return back to the parent generate_recipes() function. However, at the end of the generate_recipes() function it still happily writes a non-functional recipe to disk because it doesn't know any better.

I have an idea of how to fix this:

        if len(recipe["keys"]["Process"]) > 0:
            FoundationPlist.writePlist(recipe["keys"], dest_path)

Would that work?

Thanks so much for this tool. I was excited to see what it would come up with for some apps that I've already written manual recipes for. Alas, this generates a fatal error.

It would be nice to compare my old recipes with what RR comes up with. Would it be possible to tell the nice robot to ignore an existing recipe and continue anyway?

Thanks

Right now Recipe Robot works really well with apps that support CodeSignatureVerifier, but less so with pkgs that support CodeSignatureVerifier.

Here's what we're collecting to verify an app:

• input_path (string)
• requirements (string)

Here's what we need to collect to verify a pkg:

• input path (string)
• expected_authority_names (array of strings)

We may have to separate codesign_status and codesign_reqs into app-specific and pkg-specific keys in the facts dict.

Right now an interactive prompt appears when the Recipe Robot script is first run. The user is prompted for their preferred identifier, recipe save location, and preferred recipe types.

During this prompt, it would be excellent to "refresh" the screen while retaining scrollback. Right now the act of toggling recipe types on/off causes the entire list of recipes to appear again.

I believe the right way to do this is using the curses module, although I'm not familiar with it yet.

This will not affect the Mac app, which will have a separate interface for prompting for preferences.

When creating a lot of recipes in a short time, it's possible that users could run up against GitHub's API rate limits. It would be excellent if we could leverage API tokens (as AutoPkg does) to prevent this limitation.

If the app we want to use as the basis for recipe-building isn't on the root level of the dmg or zip, do we want to dig much farther, or should we error-out?

Two easy examples, both of which require the user to drag a folder to their /Applications folder, rather than an app:

Thinking about this might be out of scope for Recipe Robot, but just wanted to put it out there.

When creating absolute, sccm, and filewave recipes, certain repos need to be added in order for the resulting recipe to run successfully. Remind people of this, if the repo isn't already present.

In order to release our public beta, I think we need these features:

• When the com.elliotjordan.recipe-robot.plist preferences file doesn't exist, the Recipe Robot app should automatically show the prefs window and create one. (Right now it fails to process anything if prefs don't exist.)
• Ability to access --ignore-existing from the preferences. Preferably the checkbox should appear only while holding Option, or another such minor deterrent. (Right now there's no way to access this option via the UI.)
• Ability to paste in a URL as the input path. (Currently it only accepts dragged-in apps.)

After the Recipe Robot script finishes, it should output a report-plist to /tmp or the Cache folder, so that the planned Mac app can read the status of the last run, any errors, and other information. Once this is done we should be able to go full speed ahead on the Mac app.

Recipe Robot was originally intended to utilize existing recipes as a foundation to build additional recipes. However, this seems like an incredibly complex thing to undertake:

• Many download recipes don't use code signature verification. Do other recipes we create need to start with code signature verification, or should that be the responsibility of the download recipe author?
• Some munki recipes were written before ParentRecipe existed, and therefore handle the entire download process too. Do we duplicate the download components into a new download recipe, then build pkg and other recipes on top of that?
• It wouldn't be sufficient to check whether the parent recipe has an Unarchiver processor, for example. We'd need to actually parse the destination path to figure out what the state of the cache directory will be when our recipes take over.

Given this, and given our initial time constraints, I'm starting to consider dropping this feature from the 1.0 release. Instead, Recipe Robot would only generate full recipe chains.

I still want to urge people not to build recipes that already exist and upload them to GitHub, in order to reduce confusion caused by recipe duplication in autopkg search results. Maybe this would mean that if any recipe exists, Recipe Robot will politely decline to build more (unless forced to build a full set).

Many recipes based on download URLs benefit from a filename argument when including the URLDownloader processor:

• http://macrabbit.com/espresso/get
• https://srv3.airdroid.com/p14/web/getbinaryredirect?type=dmg
• http://download.bjango.com/skalapreview/
• https://central.github.com/mac/latest
• https://hephaestus.infinit.io/macosx/installer/Infinit.dmg?t=1441494215512

Others are just fine as-is:

• http://www.ohanaware.com/appwrapper/appWrapper3.zip
• https://download-cdn.getsync.com/stable/osx/BitTorrent-Sync.dmg
• http://static-download.s3-website-us-east-1.amazonaws.com/sketch.zip
• http://www.sonnysoftware.com/Bookends.dmg

How can Recipe Robot determine which is which and apply the filename argument intelligently?

Currently, the recipes are displayed in this order:

  [ ] 0. filewave - Imports a fileset into your FileWave server.
  [ ] 1. munki - Imports into your Munki repository.
  [ ] 2. install - Installs the app on the computer running AutoPkg.
  [ ] 3. jss - Imports into your Casper JSS and creates necessary groups, policies, etc.
  [ ] 4. download - Downloads an app in whatever format the developer provides.
  [ ] 5. absolute - Imports into your Absolute Manage server.
  [ ] 6. ds - Imports into your DeployStudio Packages folder.
  [ ] 7. sccm - Creates a cmmac package for deploying via Microsoft SCCM.
  [ ] 8. pkg - Creates a standard pkg installer file.

I think they should be displayed in this order, based in part on the ParentRecipe chain and in part on popularity.

  [ ] 0. download - Downloads an app in whatever format the developer provides.
  [ ] 1. pkg - Creates a standard pkg installer file.
  [ ] 2. munki - Imports into your Munki repository.
  [ ] 3. install - Installs the app on the computer running AutoPkg.
  [ ] 4. jss - Imports into your Casper JSS and creates necessary groups, policies, etc.
  [ ] 5. ds - Imports into your DeployStudio Packages folder.
  [ ] 6. absolute - Imports into your Absolute Manage server.
  [ ] 7. filewave - Imports a fileset into your FileWave server.
  [ ] 8. sccm - Creates a cmmac package for deploying via Microsoft SCCM.

As of right now, the master branch is no longer producing AppStoreApp recipes. Here's what happens when I attempt to run DisplayMenu:

recipe-robot --verbose /Applications/Display\ Menu.app 

                      -----------------------------------
                     |  Welcome to Recipe Robot v0.2.5.  |
                      -----------------------------------
                                \   _[]_
                                 \  [oo]
                                   d-||-b
                                     ||
                                   _/  \_

Processing /Applications/Display Menu.app ...
Input path looks like an app.
Validating app...
    App seems valid
Getting app name...
    App name is: Display Menu
Getting bundle identifier...
    Bundle idenfitier is: de.milchimgemuesefach.Display-Menu
Checking for a Sparkle feed...
    No Sparkle feed
Determining whether app was downloaded from the Mac App Store...
    App came from the App Store
Looking for version key...
    Version key is: CFBundleShortVersionString (2.2.2)
Looking for app icon...
    App icon is: /Applications/Display Menu.app/Contents/Resources/DisplayMenu
Getting app description from MacUpdate...
    Description: Get your display menu back for OS X mavericks.
Gathering code signature information...
    Code signature verification requirements recorded
    3 authority names recorded
Searching for existing AutoPkg recipes for "Display Menu"...
    No results
Searching for existing AutoPkg recipes for "DisplayMenu"...
    No results
Generating munki recipe...
[REMINDER] I've created at least one AppStoreApp override for you. Be sure to add the nmcspadden-recipes repo and install pyasn1, if you haven't already. (More information: https://github.com/autopkg/nmcspadden-recipes#appstoreapp-recipe)
    /Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/MAS-Display Menu.munki.recipe
[WARNING] Skipping install recipe, because this app was downloaded from the App Store.
    /Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/Display Menu.install.recipe
[WARNING] Skipping download recipe, because this app was downloaded from the App Store.
    /Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/Display Menu.download.recipe
Generating pkg recipe...
[REMINDER] I've created at least one AppStoreApp override for you. Be sure to add the nmcspadden-recipes repo and install pyasn1, if you haven't already. (More information: https://github.com/autopkg/nmcspadden-recipes#appstoreapp-recipe)
    /Users/elliot/Library/AutoPkg/Recipe Robot Output/Display Menu/MAS-Display Menu.pkg.recipe

No recipes are actually being created. The output folder is created, but no files exist in that folder.

Right now Recipe Robot requires you to press "y" to agree to the SLA on certain disk images.

[WARNING] Please type Y and press Enter to accept the software license agreement.
Agree Y/N? y

In order to link this up with the Mac app, we'll need to make this non-interactive.

Might not be possible, but I want to see if we can find a way around SSLv3 handshake errors when trying to access various URLs. Here's an example:

$ recipe-robot -v http://www.macroplant.com/latest-binaries/adapter-mac.dmg

                      -----------------------------------
                     |  Welcome to Recipe Robot v0.0.4.  |
                      -----------------------------------
                                \   _[]_
                                 \  [oo]
                                   d-||-b
                                     ||
                                   _/  \_

Processing http://www.macroplant.com/latest-binaries/adapter-mac.dmg ...
Input path looks like a download URL.
    Download URL is: http://www.macroplant.com/latest-binaries/adapter-mac.dmg
Downloading file for further inspection...
[WARNING] I got an SSLv3 handshake error, and I don't yet know what to do with that. (<urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)>)
[ERROR] I wasn't able to determine the name of this app, so I can't make any recipes.

How should we handle apps with spaces in their names? Current behavior is as follows:

a. Recipe identifier does not contain spaces.
b. Recipe filenames don't contain spaces, except for JSS recipes, which do.
c. Recipe %NAME% input variable does contain spaces.
d. Packages, downloads, etc based on %NAME% do contain spaces.

(a) and (c) are logical and well accepted. I'm not as sure about (b) and (d).

No need to wrap a flat package in a dmg before importing to Munki.

If it's a standard "drag this to your PreferencePanes" or "drag this to your Internet Plugins" dmg, we should be able to detect and build recipes for that. Likewise if there's a single .prefPane or .plugin file enclosed in a zip.

Not essential for 1.0, though, since there are so few preference panes and internet plugins that don't already have recipes.

For example, this AutoCasperNBI run failed because there's a single curly quote in the repo's description.

Processing https://github.com/macmule/AutoCasperNBI ...
Input path looks like a GitHub URL.
Getting GitHub repo...
    GitHub repo is: macmule/AutoCasperNBI
Getting app name...
    App name is: AutoCasperNBI
Getting GitHub description...
    GitHub description is: AutoCasperNBI is an app that automates the creation of NetBoot Images (read: NBI’s) for use with Casper Imaging.
[ERROR] Recipe Robot exploded with unexpected error: 
Traceback (most recent call last):
  File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe-robot", line 103, in main
    process_input_path(facts)
  File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/inspect.py", line 127, in process_input_path
    facts = inspect_func(input_path, args, facts)
  File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/inspect.py", line 998, in inspect_github_url
    facts["description"] = description
  File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/facts.py", line 79, in __setitem__
    val = NotifyingString(self.default_suffix, val)
  File "/Users/elliot/Developer/My Repos/recipe-robot/scripts/recipe_robot_lib/facts.py", line 141, in __new__
    instance = super(NotifyingString, cls).__new__(cls, text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 80: ordinal not in range(128)