gslab-econ / template Goto Github PK

It can simplify dependency tracking if SCons allows directories to be tracked as sources and targets, not just files within directories. In this issue, I'll explore how feasible a setup like this is.

Followup task to the lunch discussion with @gentzkow on January 24, 2017.

I have discovered that running

sconsign .sconsign.dblite

print the file in a human readable format

sconsign .sconsign.dblite > sconsign.txt

give you this.

It's not super useful, but still quite useful as files have hash that you can use to track whether the most recent version was used to build other files.

Two things:

1. I suggest we integrate this line into the code so that it runs everytime we scons. This can be versioned if necessary.

2. The file seems to be machine-specific. I suspect this being the reason Scons force a rebuild every time I switch branches (@arosenbe and I have same issue) ? In that case, should we stop versioning this file? If so, does caching still work?

Follow up to gslab-econ/gslab_python#8

No documentation and confusing names.

Clarify what Debrief does. (2h)

• Add a descriptive comment.
• Switch away from source/target/env keyword arguments to avoid conflation with SCons build commands.

• Force installation of all our stata ado files by default if using Stata in a repo.
  • Document in readme which ones are necessary and how this prerequisite can be changed.
• Create our own YAML parser that stores variables as globals. Profusely acknowledge prior parser.

Following release 4.0.0 of template and gslab_python, I would like us to do another team review to discuss next steps. The following is a list of proposed next steps I'd like us to consider. I would suggest that others add additional items they think should be considered then we schedule a call to discuss.

• Raw / external data

  • All raw data in Dropbox raw data store; clear instructions in RA manual + directory template
  • Print manifest of contents of all external directories plus contents of provenance.log if exists (or specify other name)

• constants.yaml & user-config.yaml

  • propose to rename constants.yaml -> config-global.yaml and user-config.yaml -> config-user.yaml
  • Create config-user-template.yaml committed to repo; if config-user.yaml is missing build configuration_test.py copies from config-user-template.yaml

• Top-level reorganization

  • Separate top-level subs for /analysis/, and /paper-slides/
  • Follow standard export protocol to move release contents from one to /input/ sub of the next

• Reduce requirements, include optional languages

  • Make git-lfs optional; switch in config-global.yaml to require git-lfs
  • Make Stata, R, Lyx optional; switches in config-global.yaml to require
  • /analysis/source/R/ and /analysis/source/stata/; instructions for deleting those not being used; template set to require neither
  • /paper-slides/source/paper-lyx/ and /paper-slides/source/paper-latex/
  • Only true dependencies are python w/ gslab_python and scons

• Stata scripts

  • Default Stata executable is statamp on Mac/Linux and whatever it's supposed to be on Windows
  • We should go back to .do file templates in SVN and port over what we think is essential (e.g., prerequisites)
  • Minimal version of gslab_stata that we can require be installed; separate library that includes all our legacy commands
  • Need to define paths as globals rather than locals (otherwise can't call in subfunctions); probably need to create our own version of stata yaml

• Large output files

  • Put in /release/lg/

• minor

  • standardize underscore vs dash in filenames
  • state-of-repo.log should be in /release/ along with state-of-externals.log and sconstruct.log
  • Make gencat part of gslab_misc python tools
  • Stata tool for saving tables
  • Tables & figs hard coded in /source/paper/ Lyx files; not declared as dependencies
  • SConstruct
    • migrate debrief_env to constants.yaml?
    • how to cut down the amount of code here?
    • what does Default() command do?

• Change names to config_user.yaml and config_global.yaml.
• Have a config_user_template.yaml that gets copied to config_user.yaml if the latter does not exist.
• Determine how we prevents users from committing their person's config_user.yaml (don't spend much time on this).
  • Commit the template file with a different filename, and discuss name changes in the readme.
  • Add note in the readme telling users not to commit their own file.
• Add comments to yaml files with specific instructions on what to put where.
• Reflect change in code and documentation.

• Large files from build that aren't intended for release go here.
• Added to .gitignore
• Released to remote data store but not GitHub.
• Add clear documentation in RA-Manual/Template

• Two subdirectories at top level: analysis and paper_slides. Each has its own SConsctruct and source, build, input, and release directories.
• Tables and figures from analysis are installed in the input directory of paper_slides.
  • Make sure the installation has a provenance (see raw data protocol).
  • Is this input directory flat?
  • how are paths specified from paper writing code to input directory?

Many things have changed in this release. The goal of this task is to make sure the readme and all other documentation (ra manual, notes on release.py) is up to date with the changes.

This should be the last thing we do in the sprint.

Right now, we only walk one level of the dictionary created by our yaml configuration files. However, we often nest dictionaries to more than one level. Nesting dictionaries raises a TypeError when we try to determine if the dictionary is a file. The goal of this task is to fix our method for grabbing paths from the input dictionary.

Follow up to gslab-econ/gslab_python#61 (comment). Drop .sconsign.dblite from being versioned in template (add to .gitignore). Open appropriate pull requests to implement the same change in congress_members and congress_text.

1. @arosenbe I found something that I think is potentially a day-one bug for the template's configuration.py. Line 47 and 48 will throw the following error if the config_user file is blank. I don't think this is the desired behavior. We can discuss follow ups, potentially creating a specialized error message for that case or simply warning the user of this possible error.

   PrerequisiteError: Inappropriate input values: `d` must be a dictionary, and at least one of `d[global][executable_names]` and `d[user][executable_names]` must be a dictionary as well.:
   

Add that we need to update config_global

2. please update the instruction in README.md on how to specify Stata executable name

3. Add warning that upgrading users need to update .gitignore

On dev_four template+gslab_python (and their most recent version to be pulled in #68-rmgoogledrive), on Windows, running Scons produce the following:

C:\Users\quanle\template>scons
scons: Reading SConscript files ...
IndexError: list index out of range:

C:\Users\quanle\template>scons --debug=stacktrace
scons: Reading SConscript files ...
IndexError: list index out of range:

The last time we got an IndexError with empty stacktrace, we were accidentally passing a None object to one of Scons' deep function (NoCache was the offender), so that's where I would start. My suggestion is to comment out lines in the SConstruct until the error stops.

Thanks @lboxell

Hi @gentzkow ,

We were thinking of this SCons-scenario and want to ask if you have a suggestion on how to deal with it.

DATA BUILD      ---->           ANALYSIS                 ---->   PLOTTING

SConscript                      Cloud computing                  SConscript
source=/Dropbox/data.txt        source=build/data.txt            source=build/analysis.txt    
target=build/data.txt           target=build/analysis.txt        output=release/plot.png

In this scenario, the data build and the plotting steps can be done locally, but the analysis step must be done on the cloud (e.g. AWS or Sherlock). Our two current suggested solutions are:

1. Do everything on the cloud. Then the analysis step can be co-opted into the SCons framework.

2. Break the data build and the plotting into two separate SConstruct files and manage the analysis files manually (copy build/data.txt to the cloud server, run computations, then download build/analysis.txt to the local build folder).

This is going to be a problem in ad-price-drivers and is currently small problem in divergence (except that the "cloud" step is at the beginning so it's not such a big deal to manage it manually). Thanks!

• Port template code for Stata, R, Python, MATLAB, etc. from SVN (perhaps with slight modifications).
  • Code should conform to our style guides.
  • Use gslab_stata functions to export Stata tables. preliminaries for Stata.
• Each subdirectory has same implementation coded in multiple languages.
  • Determine if LyX and LaTeX follow this convention or are in separate paper_slides_lyx and paper_slides_Llatex subs.

• Examine python virtual environments (e.g., conda) and related tools in R (e.g., packrat).
• Find what works and what gets done in practice.
• Don't include any systems in this round, but make note of them for larger projects with many dependencies.

This issue tracks problems that arise during installation of the die repo by fgwlab (note that the repo is private). Once all users are able to run the die repo, the problems tracked here will be turned into a proposal on how to improve the template. The proposal should be delivered to @gentzkow.

SCons is currently on release 3.0.1. We're using 2.5.1, and that's behind by one major and one minor release. The goal of this task is to decide if we want to stick at 2.5.1 or update.

• Release notes
• Change log

I don't think this would be a difficult change, since we can just drop in the newer scons local archive.

My sense is that most of the changes aren't relevant to us, with the possible exception of

3.0.0 should be slightly faster than 2.5.1. Changes yielded a 15% speed up for null incremental builds.

Thoughts?

I seem to be having an issue with the pip.main() call in the python script when pip 10.0.1 is installed.

Levi1$ easy_install --upgrade pip
Searching for pip
Reading https://pypi.python.org/simple/pip/
Best match: pip 10.0.1
Processing pip-10.0.1-py2.7.egg
pip 10.0.1 is already the active version in easy-install.pth
Installing pip script to /Users/Levi1/anaconda/bin
Installing pip2.7 script to /Users/Levi1/anaconda/bin
Installing pip2 script to /Users/Levi1/anaconda/bin

Using /Users/Levi1/anaconda/lib/python2.7/site-packages/pip-10.0.1-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip
DN0a232986:brazil_census Levi1$ python
Python 2.7.13 |Anaconda 2.2.0 (x86_64)| (default, Dec 20 2016, 23:05:08)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
import pip
pip.main()
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'module' object has no attribute 'main'

I switched to 9.0.2 and things worked for me again.

Levi1$ sudo pip uninstall pip
Password:
The directory '/Users/Levi1/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Uninstalling pip-10.0.1:
Would remove:
/Users/Levi1/anaconda/bin/pip
/Users/Levi1/anaconda/bin/pip2
/Users/Levi1/anaconda/bin/pip2.7
/Users/Levi1/anaconda/lib/python2.7/site-packages/pip-10.0.1.dist-info/*
/Users/Levi1/anaconda/lib/python2.7/site-packages/pip/*
Proceed (y/n)? y
Successfully uninstalled pip-10.0.1
The directory '/Users/Levi1/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
You are using pip version 8.1.0, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
DN0a232986:brazil_census Levi1$ easy_install pip==9.0.2
Searching for pip==9.0.2
Best match: pip 9.0.2
Processing pip-9.0.2-py2.7.egg
Removing pip 10.0.1 from easy-install.pth file
Adding pip 9.0.2 to easy-install.pth file
Installing pip script to /Users/Levi1/anaconda/bin
Installing pip2.7 script to /Users/Levi1/anaconda/bin
Installing pip2 script to /Users/Levi1/anaconda/bin

Using /Users/Levi1/anaconda/lib/python2.7/site-packages/pip-9.0.2-py2.7.egg
Processing dependencies for pip==9.0.2
Finished processing dependencies for pip==9.0.2
DN0a232986:brazil_census Levi1$ python
Python 2.7.13 |Anaconda 2.2.0 (x86_64)| (default, Dec 20 2016, 23:05:08)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org

import pip
pip>>> pip.version
'9.0.2'
pip.main()

Usage:
pip [options]

This impacts the python configuration code in the template. I'm not sure if this is idiosyncratic to me, or a broader issue, so I'm flagging here for you guys.

SCons executes scripts under the variables defined in its environment object. We currently load the PATH into the object, so scripts are passed to the right executable, but we don't load PYTHONPATH. This makes it impossible for a homebrewed python installation to find its associated packages.

see title

The readme has become a little messy, to the point where it's difficult to track and install dependencies. The goal of this task is to spruce up the readme.

@gentzkow what do you think about revisiting the decision not to put plots and tables in release?

I like being able to call all the inputs to the draft and slides via git for working offline on planes etc. and just for greater portability.

(Arguably these are part of the formal release of a paper in the sense that image files are typically required for typesetting anyway.)

Follow up to gslab-econ/gslab_python#46.

See https://github.com/gslab-econ/template/tree/gslab_python-issue26_clarg.

The goal is to make the readme more user-friendly, see here.

• The only true dependencies are Python (2.7.X), GSLab Python (4.1.0), PyYAML, and SCons.
• Create switches to implement our automated checks of other software and run example scripts.
  • Comment out lines in SConstruct or SConscript.
  • Add instructions on turning on dependencies in readme.

YAML MATLAB tool

How do we read YAML config files in MATLAB?

Explore, select, and incorporate a MATLAB YAML parser (8h)

• See notes in here
• Survey the options.
• Pick the best.
• Include it in our MATLAB scripts.

We are going to be doing a review of our template and gslab_python shortly. It would be good to make sure you are familiar with both (particularly the gslab_scons portion of gslab_python).

While you're going through both, if you see things to improve or seem off, you can probably start making a list on this issue (as we'll be doing something similar shortly as a team).

When SCons executes in dry-run mode (--dry-run included as command line argument), don't produce sconstruct.log, state_of_repo.log, or state_of_externals.log. Turning off the logging help keep the repository clean during the automated release process. See thread starting here.

See gslab-econ/gslab_python#134 (comment). All five points are template tasks.

Test using a symlink to some shared drive (such as hsfiles) that is mounted locally as the cache.

Follow-up to gslab-econ/gslab_python#134 (comment).

I'd like you guys to give us an assessment of how much effort it would be to convert each of our currently active repositories to the new format. Based on that we can prioritize for which it is worth paying that fixed cost.

See [template task]'s in gslab-econ/gslab_python#58 (comment)

Combined meeting notes:

Meeting note

• Echo command for user-config.yaml. [Template task]

• Allow stata-flavour yaml to be empty. Change stata-flavour to stata-exec. Make comment that Windows user can specify StataMP-64.exe. [Template task]

• Prompt user input for yaml file if it does not exist. [Template task]

• setup.py script name is not ideal in template (setup.py means something in the general Python context) [Template task]

• drop default support for gslab mtheme [Template task]

• It's not clear to me why we force the user to have a cache. We should at least discuss that if we make them write it into their user-config. (we decided to drop this as default requirement and add it as part of the "extra features" section in the readme.md) [Template task]

• Remove demonstration of Stata/R, add it to FAQ (build_stata vs build_python) so forth [template task]

• check_cache(cache_dir) [template task]

• The entire error message in check_cache is not printed when there's an error. [template task]

• fix use of ~ [template task]

• don't produce .pyc [template task]

• remove references to mode=release [template task]

• GSLab specific import statement should be kept together at the top of the SConstruct. [template task]

We previously had a check_lfs() function (see https://github.com/gslab-econ/admin/issues/7#issuecomment-235720593). This got dropped when we switched to scons. Need to re-implement.

Our current SConstruct writes STDOUT to a log file /output/SConstruct.log.

I would like to improve this in 3 ways:

(1) We should write both STDOUT and STDERR

(2) We should write to the terminal and the log file, not to the log file alone

(3) We should write the log to the top-level directory rather than in /output/ and get rid of the SConstruct commands to test whether /output/ exists and create it if it doesn't. We should then use an install builder to move the log file to the output directory in a final step.

Evaluation of template in parallel with gslab_scons module in gslab-econ/gslab_python#16.

Both small template (master) and large template (large_template).

Leave comments in task and I'll aggregate later.

This task will figure out a place to put this custom syntax highlighting. After putting this file in ~/Library/Application\ Support/Sublime\ Text\ 3/Packages/User/ and restarting Sublime Text, you see that it appears in the bottom right corner and is automatically applied whenever you open a *.log file.

There is no default *.log syntax highlighting in Sublime Text so this shouldn't clash with anything shipped with stock Sublime Text.

It basically highlights all text after three asterisks *** and before newline, making GSLab log demarcations look snazzy.

Suggested location is keeping it in a gist file and linked to from template README.md. Or maybe we can keep it in a comment (this one) and link to the comment from README.md. Or keep it in README.md but hide it using <details> tag.

File name:gslab_scons_log.tmLanguage

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>fileTypes</key>
	<array>
		<string>log</string>
	</array>
	<key>name</key>
	<string>GSLab Sconstruct Log</string>
	<key>patterns</key>
	<array>
		<dict>
			<key>comment</key>
			<string>Mark GSLAB-raised lines</string>
			<key>match</key>
			<string>\*\*\*(.)*\n</string>
			<key>name</key>
			<string>keyword.other.sconstructlog</string>
		</dict>
	</array>
	<key>scopeName</key>
	<string>source.sconstructlog</string>
	<key>uuid</key>
	<string>ed77d45b-7334-4f32-9e2f-23e212ef4f47</string>
</dict>
</plist>

Develop and document protocol for raw data storage. (8h)

• Follow SVN protocol for documenting and storing raw data (readme w/ download date, who, and where; license/citation, codebooks), but do in Dropbox.
• Determine what provenance.log looks like. (e.g., checksum, hash, file enumeration + file size)
• Simple data cleaning code is allowed to live in Dropbox.
• Update RA Manual to reflect these practices.

Document protocol for data transfer. (2h)

• Manual installation of externals. User points to location via config_user.yaml.
• Update RA Manual (and template readme) to reflect these practices.

Adds command line arguments and custom log names to FAQ. See #52 (comment).

Following this conversation we want to add a /release/release.txt file as a release asset that will keep track of the file list, drive path, and time stamp in each release.

This task includes updating release procedure in congress_text and congress_members.

Toby pointed out that if the last build step in a tree fails, the scons script doesn't indicate this. This isn't ideal. Scons should indicate explicitly that the build step failed.

Hi @stanfordquan,

I checked out the template to help @zkashner on his practice task and am having some issues following the steps listed in the readme. See my output below:

DN0a1e6122:Research Levi1$ rm -rf template/
DN0a1e6122:Research Levi1$ git clone -b 4.1.0 [email protected]:gslab-econ/template.git
Cloning into 'template'...
remote: Counting objects: 2751, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 2751 (delta 3), reused 8 (delta 3), pack-reused 2738
Receiving objects: 100% (2751/2751), 4.02 MiB | 3.74 MiB/s, done.
Resolving deltas: 100% (1082/1082), done.
Checking connectivity... done.
Downloading config/scons.zip (648.17 KB)
Downloading paper_slides/release/paper/ondeck.pdf (17.57 KB)
Downloading paper_slides/release/paper/online_appendix.pdf (44.59 KB)
Downloading paper_slides/release/paper/paper.pdf (57.72 KB)
Downloading paper_slides/release/paper/text.pdf (17.57 KB)
Downloading paper_slides/release/talk/slides.pdf (29.26 KB)
Checking out files: 100% (50/50), done.
DN0a1e6122:Research Levi1$ cd template/
DN0a1e6122:template Levi1$ python config/config_python.py

Collecting git+http://[email protected]/gslab-econ/[email protected]
Cloning http://[email protected]/gslab-econ/gslab_python.git (to v4.1.0) to /private/var/folders/9n/r307fbt12rjfch7x2cc3v5k00000gp/T/pip-VWFVOA-build
Could not find a tag or branch 'v4.1.0', assuming commit.
error: pathspec 'v4.1.0' did not match any file(s) known to git.
Command "git checkout -q v4.1.0" failed with error code 1 in /private/var/folders/9n/r307fbt12rjfch7x2cc3v5k00000gp/T/pip-VWFVOA-build

Note sure what the issue here is because I'm not familiar with the new 4.1.0 features in template or gslab_python. (Feel free to re-assign as you see fit, I just saw you were on the 4.1.0 review.)

Do this in both template and gslab_python.

Hyphens are not valid characters in object names in Python and R. We thus prefer underscores.

Make the change everywhere and document in RA Manual.

Develop and document protocol for provenance.log when releasing data to Dropbox (6h)

• Determine whether releases to Dropbox should have the exact same provenance.log structure as raw data stored in dropbox, or some other derivative.
• Implement.

Develop and implement protocol for logging external data assets. (8h)

• Print contents of every provenance/checksum/readme/GH release hash from imported data into a log. Make sure the log is committed.
  • Make sure the crawler we write doesn't go forever (use command line tools and subprocess in builder).
  • Make sure the code is general enough for use with any sort of external dependency.

Determine if we want to have automatic revision checks. (1h)

• Determine whether it is feasible to specify a specific revision stable link on Dropbox.
• If not, is there some other way of approximating this behavior?
• Determine whether we want to implement this.
• Create new issue to implement if we want this feature.

We should be running make_provenance() every time we run SCons.

First follow up to #19. The below points are a combination of comments from different people (many from MG's #19 (comment)).

Steps:

• Combine large and small templates. We should only have one template in master. This should contain the superset of directories in small and large templates.

Readme / documentation

• Lab-specific protocol info should be moved to RA manual
• Google drive should be dropped as a requirement; assume generic storage
• Discussion of requirements could be pared down / simplified further
• Specify scons 2.4 or greater as (suggested?) pre-requisite because earlier versions use a different cache naming scheme (single alphanumeric for directories versus two alphanumerics for directories).

Setup

• I had to explicitly install the Python requests library to run the large template. Looks like this is something gslab_scons depends on. Is there a way to specify things so this (and any other dependencies) get installed automatically when we install gslab_python?
• I would like to define an explicit setup.py (or equivalent step within scons) which verifies that all prerequisites are satisfied. The goal should be that we would trap most errors at this stage so if a new user passes the setup step the repository runs without error with very high probability. It could also be part of this step to check out externals from SVN + create sim links to external raw data / code / etc.

Other

• The SConscript in /data/ in the lg template should use build_r not env.Command()
• What about renaming the builders we define in SConstruct BuildLyx, BuildStata, etc. rather than Lyx, Stata, etc.?
• We should not use from xx import *; switch to import xx or from xx import specific-commands
• We should think about the templates we want R and Stata scripts to follow and implement those here. For Stata, this will mean revisiting our gslab-specific tools like preliminaries -- whether we want to keep using them and also how we want to install them.
• I don't think we want very short scripts like the current build.R to use the

main <- function() {
}
main()

structure.

• Add ondeck.lyx and text.lyx to /paper directory.
• Create ./tables and ./figures subdirectories that live are at the same level as ./paper. These subdirectories should store a .lyx figure/table file for each figure or table that gets included in the paper.
• Implement grammar suggestions from #19 (comment)

See if we can add something to setup.py that prevents scons from continuing to run if the cache directory is not already created.

As for implementation, we would add a log builder function to gslab_socns.py. The function would detect the operating system and use switches to appropriately redirect the stdout and stderr streams. For mac and linux, we can use my current implementation to send all output to the terminal and to a log. For windows, we would send all output to a log and not worry about printing the output.