Git Product home page Git Product logo

scriptorium's Introduction

Latest Version Downloads License Code Health

Scriptorium

Framework for easily using MultiMarkdown and LaTeX based system to write academic papers, especially those with shared templates for organization. This system is designed with several important design guidelines and observations:

  1. Many academic publications are collaborative, continuously evolving works.
  2. Many academic publications provide LaTeX templates to meet the formatting guidelines for submission.
  3. Research groups often use common templates for documents, such as submitting to a particular conference repeatedly over the years, or thesis/dissertation requirements at a particular university.
  4. LaTeX can have a high learning curve for people unfamiliar with its syntax and style. Simpler markup langauges can sacrifice control for ease of use or readability. Ideally, a system would provide tools that cover the continuum between simplicity and control.
  5. Academic publications can have a mix of visibility/availability between groups. Some papers may be private to an individual, a group, some mix of groups, or publicly available. The tools for building papers may not share this variability of privacy.

In light of these observations, this framework aims to provide:

  1. Version control for documents, allowing concurrent collaboration, full history tracking, branching for edits, etc.
  2. User friendly in syntax and operation, minimizing the requirement to learn gory internal details.
  3. Provide the ability to selectively access lower level commands for power users to fine tune results.
  4. A base for groups to use to distribute/share a common working base, with flexibility for individuals to choose documents which are present on their system.
  5. Fine grained privacy controls for controlling access to documents and intellectual property.
  6. Plain text as the basis for documents, ensuring that a wide variety of workflows and tools can be used to edit documents.

Installation

Linux Installation

  1. Install external dependencies:
    1. git
    2. LaTeX
      1. Some distributions don't include biber, which should be installed as well
    3. Python
    4. cmake, which is required to build MultiMarkdown
  2. Execute pip install scriptorium
  3. Install the MultiMarkdown shared library by executing sudo python -c "import pymmd; pymmd.build_mmd()"

Mac Installation

  1. Install external dependencies:
    1. git
    2. MacTeX
    3. pip
  2. Execute pip install scriptorium

Windows Setup

These instructions provide a method to configure Scriptorium to work on Windows with a minimum of fuss. There are many other ways to configure this system, and cleaner instructions would be appreciated in a pull request

  1. Install the GitHub Desktop Client, and follow the directions to configure it with your GitHub account.
  2. Install MikTex
  3. Install Python
  4. Modify the Environment Variables to add Python to the PATH variable. Based on the helpful instructions here:
    1. Right click on "My Computer" and select Properties
    2. In the System Properties window, click on Advanced
    3. Click on the "Environment Variables" button
    4. Select the PATH variable, and add Python. The default values would be C:\Python27 and C:\Python27\Scripts for Python 2.7, or C:\Python35 and C:\Python35\Scripts for Python 3.5, although this would changed if the installation directories were changed in previous steps.
  5. Open the "Git Shell" installed by GitHub, and verify that python.exe and pip are recognized commands.
  6. Execute pip install scriptorium

Tutorial

Scriptorium can be invoked directly from the command line using the name scriptorium.

Check that all external dependencies are installed and detected correctly, by veryifying the following command returns nothing:

scriptorium doctor

You can check where templates will be installed:

scriptorium config TEMPLATE_DIR

or change the directory:

scriptorium config TEMPLATE_DIR ~/.scriptorium/templates

Install some example templates:

scriptorium template -i https://github.com/jasedit/simple_templates.git

To list which templates are currently available in scriptorium:

scriptorium template -l

To create a new paper in the directory example_report using the report template previously installed:

scriptorium new example_report -t report -c author "John Doe" -c title "My Example Report"

Adding example content using the command:

echo "

# Introduction

This is an introductory section." >> example_report/paper.mmd

The PDF of the report can be built using:

scriptorium build example_report

or, if inside example_report:

scriptorium build

Papers Organization

Since papers in development are generally not open-source, this framework pushes papers into standalone folders. Storing these folders in version control is STRONGLY encouraged, though not strictly required by the system. Generally, version control repositories don't handle binary files (e.g. images) particularly well, so it is recommended to break up papers into more repositories to require less overhead storing history, as well as providing finer granularity in sharing papers.

Paper Metadata

In order to integrate the template system, the MultiMarkdown metadata header requires a few important statements. Consider an example header, as shown below.

Base Header Level: 3
latex author: Author
Title: Paper Title
myemail: [email protected]
latex input: template/setup.tex
latex footer: template/footer.ex

The Base Header Level is important for configuring MultiMarkdown to avoid section levels which may not be supported by the template being used. Level 1 is the \chapter command in LaTeX, which is often unused in conference papers. The latex author key bypasses input sanitization, allowing LaTeX specific commands in the authors title. myemail is the author's e-mail address. The input and footer are used to read the template preamble and footer. Some templates will also read a metadata.tex file, which provides a direct LaTeX file for specifying metadata when LaTeX specific commands are necessary.

Template Organization

A template defines the latex setup defining how a paper is going to be laid out, which packages it will use, etc. For reference, consider templates in the simple templates repository. A template is made in a few steps:

  1. A folder inside the templates directory. The name of this folder is what is used to reference the template in a MultiMarkdown paper, by LaTeX's recursive subdirectory search.
  2. A LaTeX file named setup.tex inside this folder, which contains the template preamble. The preamble should include everything at the start of the document before the content, through the \begin{document} statement. More may be included in this preamble, such as seen in the IEEEtran example in the simple templates.
  3. A LaTeX file named footer.tex inside this folder, which contains any LaTeX which should be appended to the end of the file. This often includes the bibliography commands. The IEEEtran footer.tex file is a good example of such a footer.
  4. An optional frontmatter.mmd and/or metadata.tex file, which contains a default values, minus the input and footer values. Any field can have a value starting with a dollar sign, and capital alphanumeric and _, ., or -, which are replaceable during the new command.

scriptorium's People

Contributors

jasedit avatar johnpeterson avatar mangusman07 avatar newtonj avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

scriptorium's Issues

Scriptorium new command fails when making a new paper

Scriptorium fails when creating a new paper. I tried with both the ieeetran template and the report template. Traceback is below.

Traceback (most recent call last):
File "/usr/local/bin/scriptorium", line 9, in
load_entry_point('scriptorium==2.0.0', 'console_scripts', 'scriptorium')()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 78, in main
args.func(args)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 43, in create
if not scriptorium.create(args.output, args.template, force=args.force, config=args.config):
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/papers.py", line 117, in create
shutil.copyfile(giname, os.path.join(paper_dir))
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 21] Is a directory: 'test'

_MMD_LIB set to null ?

I get this error whatever I try to build:

C:\Users\Roland\Desktop\Thesis\Compile>scriptorium build mypaper.mmd
Traceback (most recent call last):
File "c:\users\roland\appdata\local\programs\python\python39\lib\runpy.py", line 197, in run_module_as_main
return run_code(code, main_globals, None,
File "c:\users\roland\appdata\local\programs\python\python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Roland\AppData\Local\Programs\Python\Python39\Scripts\scriptorium.exe_main
.py", line 7, in
File "c:\users\roland\appdata\local\programs\python\python39\lib\site-packages\scriptorium_main
.py", line 181, in main
args.func(args)
File "c:\users\roland\appdata\local\programs\python\python39\lib\site-packages\scriptorium_main
.py", line 16, in build_cmd
pdf = scriptorium.to_pdf(args.paper, use_shell_escape=args.shell_escape, flatten=args.flatten,
File "c:\users\roland\appdata\local\programs\python\python39\lib\site-packages\scriptorium\papers.py", line 137, in to_pdf
tex_fp.write(pymmd.convert(mmd_fp.read(), fmt=pymmd.LATEX, dname=mmd, ext=pymmd.SMART))
File "c:\users\roland\appdata\local\programs\python\python39\lib\site-packages\pymmd-0.4.0-py3.9.egg\pymmd\mmd.py", line 123, in convert
source, _ = _expand_source(source, dname, fmt)
File "c:\users\roland\appdata\local\programs\python\python39\lib\site-packages\pymmd-0.4.0-py3.9.egg\pymmd\mmd.py", line 83, in _expand_source
_MMD_LIB.g_string_new.restype = ctypes.POINTER(GString)
AttributeError: 'NoneType' object has no attribute 'g_string_new'

Error during setup

When trying to setup scriptorium on a new Ubuntu 16.04 computer, when the command sudo python -c "import pymmd; pymmd.build_mmd()" is run, the following is reported

Entering 'submodules/greg'
Already up-to-date.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/robert/.local/lib/python2.7/site-packages/pymmd/download.py", line 65, in build_mmd
    subprocess.call(cmake_cmd)
  File "/usr/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

This prevent the rest of the install from working.

Template update fails with invalid expression

Called template update on both ieeetran and report. Same result.

Ryans-MacBook-Pro-2:scriptorium ryan$ scriptorium template -u report
Traceback (most recent call last):
File "/usr/local/bin/scriptorium", line 9, in
load_entry_point('scriptorium==2.0.0', 'console_scripts', 'scriptorium')()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 102, in main
args.func(args)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 41, in template_cmd
scriptorium.update_template(args.update)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/templates.py", line 78, in update_template
treeish_re = re.compile(r'[A-Za-z0-9_-.]+')
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 190, in compile
return _compile(pattern, flags)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 244, in _compile
raise error, v # invalid expression
sre_constants.error: bad character range

Invoking scriptorium make fails.

When trying out the new Python system, running the following on a properly configured paper directory bombs out:

scriptorium make papers/thesis/main

The error produced indicates a failure to import a module.

Traceback (most recent call last):
  File "/usr/local/bin/scriptorium", line 11, in <module>
    load_entry_point('scriptorium==2.0.0', 'console_scripts', 'scriptorium')()
  File "/home/hakujin/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 565,
 in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/hakujin/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2598,
 in load_entry_point
    return ep.load()
  File "/home/hakujin/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2258,
 in load
    return self.resolve()
  File "/home/hakujin/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2264,
 in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ImportError: No module named main

As far as I know, all python dependencies are satisfied at this point.

Breaking out abstracts into separate files.

Current template implementations such as vtthesis have sections such as the abstract and acknowledgements embedded in the metadata.tex file. It'd be snazzy to have the ability to define those as external multimarkdown files to ease creation and maintenance.

Missing package latex

When running the scriptorium doctor command, it returns Missing package latex on a new install of Ubuntu 16.04.

Template command with list option fails with Unicode error

Attempted to check the template list that is installed. I verified that the TEMPLATE_DIR is correct in the scriptorium config file.

Ryans-MacBook-Pro-2:scriptorium ryan$ scriptorium template -l
Traceback (most recent call last):
File "/usr/local/bin/scriptorium", line 9, in
load_entry_point('scriptorium==2.0.0', 'console_scripts', 'scriptorium')()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 102, in main
args.func(args)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 44, in template_cmd
templates = scriptorium.all_templates(args.template_dir)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/templates.py", line 15, in all_templates
for dirpath, _, filenames in os.walk(dname):
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 278, in walk
names = listdir(top)
TypeError: coercing to Unicode: need string or buffer, NoneType found

Fatal LaTeX errors can cause cause assignment errors in Scriptorium.

A fatal LaTeX error can break error output statements inside Scriptorium. The stack trace and causative error are included below.

Traceback (most recent call last):
  File "/usr/local/bin/scriptorium", line 11, in <module>
    load_entry_point('scriptorium==2.0.0', 'console_scripts', 'scriptorium')()
  File "/usr/local/lib/python2.7/dist-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/__main__.py", line 78, in main
    args.func(args)
  File "/usr/local/lib/python2.7/dist-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/__main__.py", line 12, in make
    pdf = scriptorium.to_pdf(args.paper, use_shell_escape=args.shell_escape)
  File "/usr/local/lib/python2.7/dist-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/papers.py", line 76, in to_pdf
    print('LaTeX conversion failed with the following output:\n', output)
UnboundLocalError: local variable 'output' referenced before assignment
LaTeX Warning: Citation 'mrx05' on page 1 undefined on input line 16.

pdfTeX warning (ext4): destination with the same identifier (name{page.1}) has
been already used, duplicate ignored
<to be read again>
                   \relax
l.18 \chapter
             {Bibliography} [1

]
Chapter 2.

(/home/hakujin/scriptorium/bin/../templates/./simple_templates/report/footer.te
x

! Package biblatex Error: '\bibliographystyle' invalid.

See the biblatex package documentation for explanation.
Type  H <return>  for immediate help.
 ...

l.4   \bibliographystyle{ieeetr}

Here is how much of TeX's memory you used:
 10908 strings out of 495000
 178786 string characters out of 6181170
 614717 words of memory out of 5000000
 14062 multiletter control sequences out of 15000+600000
 8501 words of font info for 30 fonts, out of 8000000 for 9000
 14 hyphenation exceptions out of 8191
 37i,6n,32p,992b,729s stack positions out of 5000i,500n,10000p,200000b,80000s

!  ==> Fatal error occurred, no output PDF file produced!

Expand unit tests

Unit tests are recently added to the repository - adding more to improve coverage would be greatly appreciated. These are relatively simple to write, and I'm happy to review any PRs that introduce additional unit tests.

Consider allowing papers and templates to provide scriptorium configuration values

Currently, certain configuration values are defined on a per-user basis, such as the LATEX_CMD configuration option. This isn't necessarily a per-user setting, however - particular papers may be limited in which LaTeX engine may be used for layout.

Is the LATEX_CMD the only option which would be worth considering for generalization in this fashion? Would the --flatten or --keep-comments be worth including?

Templates should provide their own example frontmatter

Right now, all papers derive from a generic paper example, which suffers from not providing the appropriate metadata each template requires for correct formatting. Templates should be able to provide an example frontmatter with default values inserted.

This would also allow a paper to not require the use of metadata.tex for templates which are simple enough to not require it.

Shell Completions

In the same vein as git's subcommands such as git add, etc. having the ability to tab-complete things like scriptorium bu to scriptorium build would be a quality of life improvement for command-line users.

Remove bash dependency

Windows does not have a good method of generally supporting the bash dependency. Switching to a python script for performing operations could be useful in helping this.

Make as a dependency could also be removed if this were the case.

Subtle difference in how Python 2 mmap stuff works causing error when building papers.

When running scriptorium build under Python 2, the following TypeError exception is thrown:

Traceback (most recent call last):
  File "/usr/local/bin/scriptorium", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/scriptorium/__main__.py", line 161, in main
    args.func(args)
  File "/usr/local/lib/python2.7/dist-packages/scriptorium/__main__.py", line 16, in build_cmd
    pdf = scriptorium.to_pdf(args.paper, use_shell_escape=args.shell_escape, flatten=args.flatten)
  File "/usr/local/lib/python2.7/dist-packages/scriptorium/papers.py", line 61, in to_pdf
    fname = paper_root(paper_dir)
  File "/usr/local/lib/python2.7/dist-packages/scriptorium/papers.py", line 28, in paper_root
    if get_template(fname):
  File "/usr/local/lib/python2.7/dist-packages/scriptorium/papers.py", line 48, in get_template
    idx = mm.find(bytearray(blank_line, 'utf-8'))
TypeError: find() argument 1 must be string or read-only buffer, not bytearray

This does not occur when running under Python 3.

Error on building document

Attempted to build a new document after completing the configuration in paper.mmd, but python errors occurred. Paper files are attached below.

tree_classification.zip

$ scriptorium build
Traceback (most recent call last):
File "/usr/local/bin/scriptorium", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.4/dist-packages/scriptorium/main.py", line 181, in main
args.func(args)
File "/usr/local/lib/python3.4/dist-packages/scriptorium/main.py", line 17, in build_cmd
keep_comments=args.keep_comments)
File "/usr/local/lib/python3.4/dist-packages/scriptorium/papers.py", line 116, in to_pdf
fname = paper_root(paper_dir)
File "/usr/local/lib/python3.4/dist-packages/scriptorium/papers.py", line 33, in paper_root
if get_template(fname):
File "/usr/local/lib/python3.4/dist-packages/scriptorium/papers.py", line 54, in get_template
idx = mmf.find(_BLANK_LINK)
TypeError: must be read-only pinned buffer, not bytearray

Error with scriptorium make output file name

Called scriptorium make from directory with paper.mmd (folder name test).

Ryans-MacBook-Pro-2:test ryan$ scriptorium make -o test
Traceback (most recent call last):
File "/usr/local/bin/scriptorium", line 9, in
load_entry_point('scriptorium==2.0.0', 'console_scripts', 'scriptorium')()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 78, in main
args.func(args)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/main.py", line 12, in make
pdf = scriptorium.to_pdf(args.paper, use_shell_escape=args.shell_escape)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scriptorium-2.0.0-py2.7.egg/scriptorium/papers.py", line 76, in to_pdf
print('LaTeX conversion failed with the following output:\n', output)
UnboundLocalError: local variable 'output' referenced before assignment

Add clean command to remove LaTeX temporary files

LaTeX can get confused sometimes by cruft in temporary files, and having a command to automatically remove these files would be useful. These should not removed every time, since those files are normally required for building the document.

People discuss this issue in related areas and tools (see here), which can serve as a decent starting point.

At a simple cut, this should be a clean function which:

  1. Determines which files are the temporary files (ignored by git, matching temporary files derived from the paper root).
  2. Can remove those files.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.