nedbat / cog Goto Github PK

View Code? Open in Web Editor NEW

324.0 324.0 26.0 340 KB

Small bits of Python computation for static files

License: MIT License

Makefile 1.26% Python 98.74%

cog's Introduction

I'm Ned Batchelder, a Python software developer and community organizer.

My personal site is nedbatchelder.com.
I'm between gigs. My most recent was at 2U/edX working on the Open edX project.
I'm an organizer of Boston Python.

You can find me at:

Mastodon: @[email protected].
Bluesky: @nedbat.com.
Libera IRC: nedbat in #python.
Discord: nedbat in the Python Discord.

My latest blog posts:

My flexbox layout, 19 Apr
I recently had to reverse engineer the layout of this site. I created it once upon a time, but had forgotten the details, and to save myself the work five years from now when I have to do it again, I’m noting down what I learned about how it works. (read..)
Try it: function/class coverage report, 15 Apr
I’ve added experimental function and class coverage reports to coverage.py. I’d like feedback about whether they behave the way you want them to. (read..)
Is this for autistic people?, 20 Mar
Special Olympics swim practices just started. A new young athlete joined us, and he asked a question that has stuck with me: is this for autistic people? (read..)
Does Python have pointers?, 11 Mar
People sometimes ask, “Does Python have pointers?” I hate to be the typical senior engineer, but this is one of those questions where the answer is, it depends what you mean by pointer. (read..)
and many more..

I maintain a few Python packages, including:

Coverage.py: The code coverage tool for Python
Cog: Small bits of Python computation for static files
Dinghy: A GitHub activity digest tool
Scriv: Changelog management tool
WatchGHA: Live display of current GitHub action runs
Aptus: Mandelbrot fractal viewer

I've also made a few informal projects, some mathy art, some small utilities:

Flourish is a harmonograph explorer. Blog post and live site.
Stilted is a toy PostScript implementation. Blog post.
Truchet images explores Truchet tiles, and rendering images with them. Blog post.
Gefilte Fish is a Python-based DSL for writing Gmail filters. Blog post.
Pydoctor shows details of your Python environment, for troubleshooting.
pkgsample, an simple example of how to package a Python project.

(made with cog at 2024-05-11 02:34 UTC)

cog's People

Contributors

Stargazers

Watchers

cog's Issues

Generate multiple output files with cog

Is it possible to generate multiple output files with cog?

For example I have python dictionary:

MODULE_ACTIONS_DICT = {
"contractors": ["add", "delete", "edit", "find", "get"],
"invoices": ["add", "delete", "download", "edit", "find"],
"expenses": ["find","get"],
........
}

I would like to use cog to create multiple source files, one file per dictionary key and using dictionary values to parametrize those files.

I wonder if cog is right tool for this task... Alternatively I can use jinja2

Debian package: which release should be packaged?

Hi. I was going to package the latest tag: v3.3.0. But doing that produces two different executables in /usr/bin. It looks like you already fixed that here: 29a5e04

Should I wait for a new release of cog, or should I apply that change to v3.3.0? Debian is in a freeze right now (we're preparing our next release), so this won't make it into Debian for a few months anyway. So there's no rush on this at all.

Thanks!

picky about the command line interface

Dear Ned,
I really love the idea and the concept - all the time and work you put into cog.
After I heard about it on Python-Bytes I was planning to use and distribute it through different parts of my professional life and personal projects.

What I saw that made me itch to support you was the command line interface.

I really think there is a great standard with the short / long option style like -c / --crc
If you then give it a broad help keyword - including those /? from windows the command line interface is really a self-documenting thing.
At the moment you mainly use short options.

Would you be interested in a suggestion from me in form of a pull request?
Would it be fine with you to add click as a dependency? (this is nice but not necessary)

Sincerly
Felix

indirect-vs-inline-python

Overview

indirect-vs-inline-python
using cog without having to include any python in the cog_enabled_src_file

Context

python nedbat cog
most recent stable version as of 2023-03-11

Scenario

developer MurrayStella wants to invoke cog on a cog_enabled_src_file
MurrayStella wants to remove the inline python from the cog_enabled_src_file and use indirect python instead

Example

this simple example illustrates the desired outcome
MurrayStella wants to convert BEFORE into AFTER

BEFORE

## Simple demo just to show what we want
* just print a hello world message
* a random number of times for variability
* include a random integer for more variability

##@ [[[cog cog.out('Hello {vxx:03d} World!\n'.format(vxx=random.randint(0,999))*random.randint(1,3)) ]]]
Hello 009 World!
Hello 009 World!
##@ [[[end]]]

##@ [[[cog cog.out('Hello {vxx:03d} World!\n'.format(vxx=random.randint(0,999))*random.randint(1,3)) ]]]
Hello 371 World!
##@ [[[end]]]

AFTER

## Simple demo just to show what we want
* just print a hello world message
* a random number of times for variability
* include a random integer for more variability

##@ <cogbeg coggid="uu190hello1678463240" coguri="py003010" >
Hello 772 World!
Hello 772 World!
Hello 772 World!
##@ <cogend>

##@ <cogbeg coggid="uu195hello1678463240" coguri="py003010" >
Hello 371 World!
##@ <cogend>

Feature request

MurrayStella would like to use the syntax style in the AFTER example instead of BEFORE
This could be doable with a new cmd_line flag such as --idmarker= or --outputmarker=
- this flag is similar to the --marker cmd_line flag
- instead of changing the cog_markup_syntax as --marker does, it would instead remove the need to specify inline python with cog.out("PythonCodeHere")
- it would allow the user to specify any arbitrary prefix and suffix to delimit a cog_code_id
- the cog_code_id could simply reference a variable specified with the -D name=val cmd_line flag
- this would tell cog to get the auto-generated code from the variable instead of inline python code
- it would also allow arbitrary syntax in the cog_enabled_src_file that does not have to involve any visible python

Workaround

MurrayStella has already implemented a hacky sadness-inducing (but functional) workaround to provide this functionality
Workaround Steps:
- Step001: invoke cog from within python instead of from the cmd_line
  - this allows us to do our hacky sadness-inducing postproc and preproc
  - capt="seealso" ;; href="#19 (comment)"
- Step002: change the default cog_markup_syntax with the existing --marker option
  - this allows [[[cog ]]] [[[end]]] to become <cogbeg coguri= <cogend>
- Step003: extract out all the coggid unique_ids in the cog_enabled_src_file
- Step004: use those unique_ids to create cog-visible variables using the -D name=val cmd_line flag
- Step005: assign python-generated output to each of those unique_ids
  - this causes sadness because it is brittle
  - the -D name=val cmd_line flag is not friendly to python-generated output that spans more than one line
  - there are hacky workarounds for this, but they are less desirable compared to a full-fledged cogscript API
- Step006: muck around with the cog_enabled_src_file so that coggid="unique_id_here" becomes cog.out(MyPyOutputEngine.unique_id_here())
- Step007: invoke cog
- Step008: muck around with the cog_enabled_src_file so that cog.out(..) changes back to coggid="..."

Hosted module docs are blank

It appears that the module documentation is rendering as blank on RTD:

(Firefox, similar in Safari)

If you click to view the source the content appears to be there, and it also looks fine here on GitHub. All of the package's other pages seem fine. Unfortunately I'm not familiar with deploying to RTD so I'm not sure how to help diagnose further at the moment.

Using cog with a markdown file and fenced code blocks

Hello, I'm sorry to bother you but would it be possible to use cog with python snippets put between fenced code, inside a markdown file ?

For example can this markdown file:

This is my source code

```python
def sum_numbers(a, b):
    return a + b

sum_numbers(3, 4)    
```

and this is the result:

```cog
```

Be translated into this:

This is my source code

```python
def sum_numbers(a, b):
    return a + b

sum_numbers(3, 4)    
```

and this is the result:

```cog
7
```

And then, getting this final output from a markdown parser:

This is my source code

def sum_numbers(a, b):
    return a + b

sum_numbers(3, 4)

and this is the result:

Make cog usable as a library

https://twitter.com/simonw/status/1461372852901335043

Use of mutation testing in cog - Help needed

Hello there!

My name is Ana. I noted that you use the mutation testing tool in the project.
I am a postdoctoral researcher at the University of Seville (Spain), and my colleagues and I are studying how mutation testing tools are used in practice. With this aim in mind, we have analysed over 3,500 public GitHub repositories using mutation testing tools, including yours! This work has recently been published in a journal paper available at https://link.springer.com/content/pdf/10.1007/s10664-022-10177-8.pdf.

To complete this study, we are asking for your help to understand better how mutation testing is used in practice, please! We would be extremely grateful if you could contribute to this study by answering a brief survey of 21 simple questions (no more than 6 minutes). This is the link to the questionnaire https://forms.gle/FvXNrimWAsJYC1zB9.

Drop me an e-mail if you have any questions or comments ([email protected]). Thank you very much in advance!!

WIP: option go around

after the #7 there could be considered to move

-w no longer supported for --make-writable
-w as --warning (former -e)
-e as --encoding (former -n)
-E as (former -z
... to be continued...

Check flag

Is it possible to run cog to check if a file will be changed?
If not, would it be possible to add a --check flag that returns a non-zero status code for CI and pre-commit purposes?

Handling of old output slows down cog

Hi,

It seems that cog is quite slow for large files, especially when the -r flag is not used. For example, for an ~80k LoC (including cog output) file, cog -xc -o /path/to/output/file /path/to/input/file is taking ~40s whereas cog -rc /path/to/input/file is taking ~10s.

There are 2 distinct issues here. The first one being that cog -xc takes 4x the time cog -rc takes. If we change cog to always use an io.StringIO buffer to read the input file into, as it does when the -r flag is used, then the performance of cog -xc would match that of cog -rc.

~~The second issue is cog taking ~10s, even when not running the generator.~~ If, instead of concatenating each line of the previous cog output to a string, we append each line to a list of strings then we would see a significant performance improvement. For the same file as the initial example, cog -xc would take ~0.150s and cog -rc would take ~0.300s.

I have already made the proposed changes, since ~~they were~~ it was simple enough, and I'll be creating a PR shortly. However, I'm opening this issue in case you feel a deeper discussion is needed. Looking forward to hearing your thoughts!

Best,
Panayiotis

Cog fails to run on a FIPS-compliant system

I can't run Cog because my hosts operate in FIPS mode. Cog uses MD5, which isn't compliant with FIPS. By default in Python, hashes are considered a security item by FIPS. Python added a usedforsecurity argument to hashlib constructors for non-cryptographic usage: https://docs.python.org/3/library/hashlib.html#hash-algorithms. Also, FIPS generally approves the SHA-2 family of hashes: https://csrc.nist.gov/pubs/fips/180-4/upd1/final.

Here's output from running Cog on a FIPS-mode host:

[dev@6d507bf42c10 cog-example]$ cog -r cog-input
Cogging cog-input
Traceback (most recent call last):
  File "/home/dev/cog-example/.venv/bin/cog", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 845, in main
    return Cog().main(sys.argv)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 794, in main
    self.callableMain(argv)
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 782, in callableMain
    self.processArguments([a])
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 756, in processArguments
    self.processWildcards(args[0])
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 721, in processWildcards
    self.processOneFile(sMatchingFile)
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 693, in processOneFile
    sNewText = self.processString(sOldText, fname=sFile)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 631, in processString
    self.processFile(fOld, fNew, fname=fname)
  File "/home/dev/cog-example/.venv/lib64/python3.11/site-packages/cogapp/cogapp.py", line 467, in processFile
    self.cogmodulename = 'cog_' + hashlib.md5(sFileOut.encode()).hexdigest()
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

Sign the release tag with github GPG key

Please add a signature to the source code bundle or sign the release tag with github GPG key

Preserve file original line ending regardless the platform

As I am on Windows, Python open(file) will automatically use CRLF when saving the file, regardless whether the file was originally with LF line ending. By opening the file in binary mode, Python open(file) will stop converting the file line endings.

Add support for use as pre-commit hook

Hey,

Thanks for the useful tool! Using cog as a pre-commit hook would increase its usefulness as you could verify that all cog code is run before committing. The config is as simple as I have implemented here: https://github.com/nialov/cog/blob/feat-add-as-pre-commit-hook/.pre-commit-hooks.yaml.
I have configured the implemented hook locally in a project as such:

  - repo: https://github.com/nialov/cog
    rev: 6d73562c5cb38753bd94414a6e7612b263bc278d
    hooks:
      - id: cog
        files: "(docs_src/index.rst)"

However, what flags to use and what to put as the default is definitely an open question. In my example I have used the checksum (-c) and inplace conversion (-r) flags. These probably cannot be defaults as cog can be used in numerous different ways.

Secondly, the command line cog should not fail when ran with no files inputted. This does not have to be the default but should be enabled by e.g., a flag (--no-fail-on-no-filenames or something more succinct).

I would not be surprised if multiple have not already implemented this functionality in their own projects/forks. A caveat in the implementation as a pre-commit hook is that the Python interpreter will not have access to any other packages other than cog and the standard library of Python so more advanced use cases are probably out of scope of the hook implementation.

Most recent Python updates break -I parameter

With the most recent Python 3.9.5, 3.8.10 fixing bpo-43105, cogapp can no longer import modules from relative paths given by -I.

Test failures:

[   31s] ============================= test session starts ==============================
[   31s] platform linux -- Python 3.9.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
[   31s] rootdir: /home/abuild/rpmbuild/BUILD/cogapp-3.0.0, configfile: setup.cfg
[   31s] collected 130 items
[   31s] 
[   32s] cogapp/test_cogapp.py .................................................. [ 38%]
[   32s] ............................FFF...............................           [ 86%]
[   32s] cogapp/test_makefiles.py .....                                           [ 90%]
[   33s] cogapp/test_whiteutils.py .............                                  [100%]
[   33s] 
[   33s] =================================== FAILURES ===================================
[   33s] _____________________ CogIncludeTests.testTwoIncludePaths ______________________
[   33s] 
[   33s] self = <cogapp.test_cogapp.CogIncludeTests testMethod=testTwoIncludePaths>
[   33s] 
[   33s]     def testTwoIncludePaths(self):
[   33s]         # Test that two -I's add include directories properly.
[   33s]         makeFiles(self.dincludes)
[   33s]         self.cog.callableMain(['argv0', '-r', '-I', 'include', '-I', 'inc2', 'test.cog'])
[   33s] >       self.assertFilesSame('test.cog', 'test.out')
[   33s] 
[   33s] cogapp/test_cogapp.py:1551: 
[   33s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   33s] cogapp/test_cogapp.py:791: in assertFilesSame
[   33s]     self.assertEqual(text1, text2)
[   33s] E   AssertionError: b'//[[15 chars]rt mymodule\n//]]]\nHello from mymodule in inc2\n//[[[end]]]\n' != b'//[[15 chars]rt mymodule\n//]]]\nHello from mymodule\n//[[[end]]]\n'
[   33s] _____________________ CogIncludeTests.testTwoIncludePaths2 _____________________
[   33s] 
[   33s] self = <cogapp.cogapp.CogGenerator object at 0x7fc50518a1c0>
[   33s] cog = <cogapp.cogapp.Cog object at 0x7fc50518a460>
[   33s] globals = {'__builtins__': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, ...}, 'cog': <module 'cog'>}
[   33s] fname = '<cog test.cog:1>'
[   33s] 
[   33s]     def evaluate(self, cog, globals, fname):
[   33s]         # figure out the right whitespace prefix for the output
[   33s]         prefOut = whitePrefix(self.markers)
[   33s]     
[   33s]         intext = self.getCode()
[   33s]         if not intext:
[   33s]             return ''
[   33s]     
[   33s]         prologue = "import " + cog.cogmodulename + " as cog\n"
[   33s]         if self.options.sPrologue:
[   33s]             prologue += self.options.sPrologue + '\n'
[   33s]         code = compile(prologue + intext, str(fname), 'exec')
[   33s]     
[   33s]         # Make sure the "cog" module has our state.
[   33s]         cog.cogmodule.msg = self.msg
[   33s]         cog.cogmodule.out = self.out
[   33s]         cog.cogmodule.outl = self.outl
[   33s]         cog.cogmodule.error = self.error
[   33s]     
[   33s]         self.outstring = ''
[   33s]         try:
[   33s] >           eval(code, globals)
[   33s] 
[   33s] cogapp/cogapp.py:169: 
[   33s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   33s] 
[   33s] >   ???
[   33s] E   ModuleNotFoundError: No module named 'mymodule'
[   33s] 
[   33s] <cog test.cog:1>:2: ModuleNotFoundError
[   33s] 
[   33s] During handling of the above exception, another exception occurred:
[   33s] 
[   33s] self = <cogapp.test_cogapp.CogIncludeTests testMethod=testTwoIncludePaths2>
[   33s] 
[   33s]     def testTwoIncludePaths2(self):
[   33s]         # Test that two -I's add include directories properly.
[   33s]         makeFiles(self.dincludes)
[   33s] >       self.cog.callableMain(['argv0', '-r', '-I', 'inc2', '-I', 'include', 'test.cog'])
[   33s] 
[   33s] cogapp/test_cogapp.py:1556: 
[   33s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   33s] cogapp/cogapp.py:751: in callableMain
[   33s]     self.processArguments([a])
[   33s] cogapp/cogapp.py:725: in processArguments
[   33s]     self.processWildcards(args[0])
[   33s] cogapp/cogapp.py:690: in processWildcards
[   33s]     self.processOneFile(sMatchingFile)
[   33s] cogapp/cogapp.py:666: in processOneFile
[   33s]     sNewText = self.processString(sOldText, fname=sFile)
[   33s] cogapp/cogapp.py:605: in processString
[   33s]     self.processFile(fOld, fNew, fname=fname)
[   33s] cogapp/cogapp.py:545: in processFile
[   33s]     sGen = gen.evaluate(cog=self, globals=globals, fname=sFile)
[   33s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   33s] 
[   33s] self = <cogapp.cogapp.CogGenerator object at 0x7fc50518a1c0>
[   33s] cog = <cogapp.cogapp.Cog object at 0x7fc50518a460>
[   33s] globals = {'__builtins__': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, ...}, 'cog': <module 'cog'>}
[   33s] fname = '<cog test.cog:1>'
[   33s] 
[   33s]     def evaluate(self, cog, globals, fname):
[   33s]         # figure out the right whitespace prefix for the output
[   33s]         prefOut = whitePrefix(self.markers)
[   33s]     
[   33s]         intext = self.getCode()
[   33s]         if not intext:
[   33s]             return ''
[   33s]     
[   33s]         prologue = "import " + cog.cogmodulename + " as cog\n"
[   33s]         if self.options.sPrologue:
[   33s]             prologue += self.options.sPrologue + '\n'
[   33s]         code = compile(prologue + intext, str(fname), 'exec')
[   33s]     
[   33s]         # Make sure the "cog" module has our state.
[   33s]         cog.cogmodule.msg = self.msg
[   33s]         cog.cogmodule.out = self.out
[   33s]         cog.cogmodule.outl = self.outl
[   33s]         cog.cogmodule.error = self.error
[   33s]     
[   33s]         self.outstring = ''
[   33s]         try:
[   33s]             eval(code, globals)
[   33s]         except CogError:
[   33s]             raise
[   33s]         except:
[   33s]             typ, err, tb = sys.exc_info()
[   33s]             frames = (tuple(fr) for fr in traceback.extract_tb(tb.tb_next))
[   33s]             frames = find_cog_source(frames, prologue)
[   33s]             msg = "".join(traceback.format_list(frames))
[   33s]             msg += "{}: {}".format(typ.__name__, err)
[   33s] >           raise CogUserException(msg)
[   33s] E           cogapp.cogapp.CogUserException:   File "test.cog", line 2, in <module>
[   33s] E               def func():
[   33s] E           ModuleNotFoundError: No module named 'mymodule'
[   33s] 
[   33s] cogapp/cogapp.py:178: CogUserException
[   33s] ____________________ CogIncludeTests.testUselessIncludePath ____________________
[   33s] 
[   33s] self = <cogapp.cogapp.CogGenerator object at 0x7fc5051515e0>
[   33s] cog = <cogapp.cogapp.Cog object at 0x7fc505151b80>
[   33s] globals = {'__builtins__': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, ...}, 'cog': <module 'cog'>}
[   33s] fname = '<cog test.cog:1>'
[   33s] 
[   33s]     def evaluate(self, cog, globals, fname):
[   33s]         # figure out the right whitespace prefix for the output
[   33s]         prefOut = whitePrefix(self.markers)
[   33s]     
[   33s]         intext = self.getCode()
[   33s]         if not intext:
[   33s]             return ''
[   33s]     
[   33s]         prologue = "import " + cog.cogmodulename + " as cog\n"
[   33s]         if self.options.sPrologue:
[   33s]             prologue += self.options.sPrologue + '\n'
[   33s]         code = compile(prologue + intext, str(fname), 'exec')
[   33s]     
[   33s]         # Make sure the "cog" module has our state.
[   33s]         cog.cogmodule.msg = self.msg
[   33s]         cog.cogmodule.out = self.out
[   33s]         cog.cogmodule.outl = self.outl
[   33s]         cog.cogmodule.error = self.error
[   33s]     
[   33s]         self.outstring = ''
[   33s]         try:
[   33s] >           eval(code, globals)
[   33s] 
[   33s] cogapp/cogapp.py:169: 
[   33s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   33s] 
[   33s] >   ???
[   33s] E   ModuleNotFoundError: No module named 'mymodule'
[   33s] 
[   33s] <cog test.cog:1>:2: ModuleNotFoundError
[   33s] 
[   33s] During handling of the above exception, another exception occurred:
[   33s] 
[   33s] self = <cogapp.test_cogapp.CogIncludeTests testMethod=testUselessIncludePath>
[   33s] 
[   33s]     def testUselessIncludePath(self):
[   33s]         # Test that the search will continue past the first directory.
[   33s]         makeFiles(self.dincludes)
[   33s] >       self.cog.callableMain(['argv0', '-r', '-I', 'inc3', '-I', 'include', 'test.cog'])
[   33s] 
[   33s] cogapp/test_cogapp.py:1562: 
[   33s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   33s] cogapp/cogapp.py:751: in callableMain
[   33s]     self.processArguments([a])
[   33s] cogapp/cogapp.py:725: in processArguments
[   33s]     self.processWildcards(args[0])
[   33s] cogapp/cogapp.py:690: in processWildcards
[   33s]     self.processOneFile(sMatchingFile)
[   33s] cogapp/cogapp.py:666: in processOneFile
[   33s]     sNewText = self.processString(sOldText, fname=sFile)
[   33s] cogapp/cogapp.py:605: in processString
[   33s]     self.processFile(fOld, fNew, fname=fname)
[   33s] cogapp/cogapp.py:545: in processFile
[   33s]     sGen = gen.evaluate(cog=self, globals=globals, fname=sFile)
[   33s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   33s] 
[   33s] self = <cogapp.cogapp.CogGenerator object at 0x7fc5051515e0>
[   33s] cog = <cogapp.cogapp.Cog object at 0x7fc505151b80>
[   33s] globals = {'__builtins__': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, ...}, 'cog': <module 'cog'>}
[   33s] fname = '<cog test.cog:1>'
[   33s] 
[   33s]     def evaluate(self, cog, globals, fname):
[   33s]         # figure out the right whitespace prefix for the output
[   33s]         prefOut = whitePrefix(self.markers)
[   33s]     
[   33s]         intext = self.getCode()
[   33s]         if not intext:
[   33s]             return ''
[   33s]     
[   33s]         prologue = "import " + cog.cogmodulename + " as cog\n"
[   33s]         if self.options.sPrologue:
[   33s]             prologue += self.options.sPrologue + '\n'
[   33s]         code = compile(prologue + intext, str(fname), 'exec')
[   33s]     
[   33s]         # Make sure the "cog" module has our state.
[   33s]         cog.cogmodule.msg = self.msg
[   33s]         cog.cogmodule.out = self.out
[   33s]         cog.cogmodule.outl = self.outl
[   33s]         cog.cogmodule.error = self.error
[   33s]     
[   33s]         self.outstring = ''
[   33s]         try:
[   33s]             eval(code, globals)
[   33s]         except CogError:
[   33s]             raise
[   33s]         except:
[   33s]             typ, err, tb = sys.exc_info()
[   33s]             frames = (tuple(fr) for fr in traceback.extract_tb(tb.tb_next))
[   33s]             frames = find_cog_source(frames, prologue)
[   33s]             msg = "".join(traceback.format_list(frames))
[   33s]             msg += "{}: {}".format(typ.__name__, err)
[   33s] >           raise CogUserException(msg)
[   33s] E           cogapp.cogapp.CogUserException:   File "test.cog", line 2, in <module>
[   33s] E               def func():
[   33s] E           ModuleNotFoundError: No module named 'mymodule'
[   33s] 
[   33s] cogapp/cogapp.py:178: CogUserException
[   33s] =============================== warnings summary ===============================
[   33s] cogapp/cogapp.py:14
[   33s]   /home/abuild/rpmbuild/BUILD/cogapp-3.0.0/cogapp/cogapp.py:14: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
[   33s]     import imp
[   33s] 
[   33s] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[   33s] =========================== short test summary info ============================
[   33s] FAILED cogapp/test_cogapp.py::CogIncludeTests::testTwoIncludePaths - Assertio...
[   33s] FAILED cogapp/test_cogapp.py::CogIncludeTests::testTwoIncludePaths2 - cogapp....
[   33s] FAILED cogapp/test_cogapp.py::CogIncludeTests::testUselessIncludePath - cogap...
[   33s] =================== 3 failed, 127 passed, 1 warning in 2.19s ===================

Workaround:
Replace

cog/cogapp/cogapp.py

Line 307 in 6cc0fff

self.addToIncludePath(a)

with

self.addToIncludePath(os.path.abspath(a))

Write output without cog source

Is it possible to write the generator code without the cog python source code embedded in it.

I don't think it would be too hard to parse the output myself and remove all cog statements however I'd prefer to use the libraries functionality if I can.

Make [[[end]]] optional to allow inline templating

Firstly, thanks for Cog, it is such a great way to generate files against a template using Python.

However, the major downside of the current implementation (for me at least) is that it is line-based. There doesn't appear to be a way to do inline templating. For example, let's say I want to generate a Doxygen C source file header:

[[[cog import cog ]]] [[[end]]]
/**
 * @file [[[cog cog.out(filename) ]]] [[[end]]]
 * @copyright Copyright (c) [[[ cog.out(current_year + " " + company_name) ]]] [[[end]]]
 * @author [[[cog cog.out(file_author) ]]] [[[end]]]
 */

However, the Cog source code mandates that the [[[end]]] tag is present, and is on a different line to the Python code block, which prevents this rather useful ability. If the -d option is used to delete the source code and the [[[end]]] tag was optional, I could just do:

/**
 * @file [[[cog cog.out(filename) ]]]
 * @copyright Copyright (c) [[[ cog.out(current_year + " " + company_name) ]]]
 * @author [[[cog cog.out(file_author) ]]]
 */

which would get transformed rather neatly into:

/**
 * @file my_first_file.h
 * @copyright Copyright (c) 2020 AcmeCorp
 * @author Joe Bloggs
 */

This is kinda similar to how PHP works inline with HTML, and would be a really useful feature IMO.

bash: /cygdrive/f/Python/Scripts/cog.py: No such file or directory

I just installed this and it does not run:

Professional@PROFESSIONAL-PC$ cog.py -r tinyformat.h
F:\python\python.exe: can't open file '/cygdrive/f/Python/Scripts/cog.py': [Errno 2] No such file or directory

Professional@PROFESSIONAL-PC$ pip3 uninstall cogapp
Uninstalling cogapp-3.0.0:
  Would remove:
    f:\python\lib\site-packages\cogapp-3.0.0.dist-info\*
    f:\python\lib\site-packages\cogapp\*
    f:\python\scripts\cog.exe
    f:\python\scripts\cog.py
Proceed (y/n)? y
  Successfully uninstalled cogapp-3.0.0

Professional@PROFESSIONAL-PC$ /usr/bin/pip3.6 install cogapp
Collecting cogapp
  Downloading https://files.pythonhosted.org/packages/83/16/463c1487939f55f5f5f13d82b9928762692bd8cf59db711f2345fed9ad0e/cogapp-3.0.0-py2.py3-none-any.whl
Installing collected packages: cogapp
Successfully installed cogapp-3.0.0
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Professional@PROFESSIONAL-PC$ cog.py -r tinyformat.h
bash: /cygdrive/f/Python/Scripts/cog.py: No such file or directory

Professional@PROFESSIONAL-PC$ /usr/bin/pip3.6 uninstall cogapp
Uninstalling cogapp-3.0.0:
  Would remove:
    /usr/bin/cog
    /usr/bin/cog.py
    /usr/lib/python3.6/site-packages/cogapp-3.0.0.dist-info/*
    /usr/lib/python3.6/site-packages/cogapp/*
Proceed (y/n)? y
  Successfully uninstalled cogapp-3.0.0

Professional@PROFESSIONAL-PC$ /usr/bin/pip2.7 install cogapp
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting cogapp
  Using cached https://files.pythonhosted.org/packages/83/16/463c1487939f55f5f5f13d82b9928762692bd8cf59db711f2345fed9ad0e/cogapp-3.0.0-py2.py3-none-any.whl
Installing collected packages: cogapp
Successfully installed cogapp-3.0.0
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Professional@PROFESSIONAL-PC$ cog.py -r tinyformat.h
bash: /cygdrive/f/Python/Scripts/cog.py: No such file or directory

Professional@PROFESSIONAL-PC$

Test suite fails

Hi. I'm making a cogapp package for Debian. I'm on a recent Debian system, and there are a bunch of failures of the test suite. Does everything pass for you?

I do this:

python3 -m unittest discover -v

There are many loud arnings about unclosed files. This patch makes the complaint happy:

diff --git a/cogapp/cogapp.py b/cogapp/cogapp.py
index 1258c0e..0baf05f 100644
--- a/cogapp/cogapp.py
+++ b/cogapp/cogapp.py
@@ -639,7 +639,8 @@ class Cog(Redirectable):
             if self.options.sMakeWritableCmd:
                 # Use an external command to make the file writable.
                 cmd = self.options.sMakeWritableCmd.replace('%s', sOldPath)
-                self.stdout.write(os.popen(cmd).read())
+                with os.popen(cmd) as f:
+                    self.stdout.write(f.read())
                 if not os.access(sOldPath, os.W_OK):
                     raise CogError("Couldn't make %s writable" % sOldPath)
             else:
diff --git a/cogapp/test_cogapp.py b/cogapp/test_cogapp.py
index 09f5444..3b55f33 100644
--- a/cogapp/test_cogapp.py
+++ b/cogapp/test_cogapp.py
@@ -786,8 +786,10 @@ class TestCaseWithTempDir(TestCase):
         shutil.rmtree(self.tempdir)
 
     def assertFilesSame(self, sFName1, sFName2):
-        text1 = open(os.path.join(self.tempdir, sFName1), 'rb').read()
-        text2 = open(os.path.join(self.tempdir, sFName2), 'rb').read()
+        with open(os.path.join(self.tempdir, sFName1), 'rb') as f:
+            text1 = f.read()
+        with open(os.path.join(self.tempdir, sFName2), 'rb') as f:
+            text2 = f.read()
         self.assertEqual(text1, text2)
 
     def assertFileContent(self, sFName, sContent):

But this still leaves two test failures:

FAIL: test_error_report (cogapp.test_cogapp.TestMain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dima/debianstuff/cogapp/cogapp/test_cogapp.py", line 943, in test_error_report
    self.check_error_report()
  File "/home/dima/debianstuff/cogapp/cogapp/test_cogapp.py", line 970, in check_error_report
    assert expected == s
AssertionError

======================================================================
FAIL: test_error_report_with_prologue (cogapp.test_cogapp.TestMain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dima/debianstuff/cogapp/cogapp/test_cogapp.py", line 946, in test_error_report_with_prologue
    self.check_error_report("-p", "#1\n#2")
  File "/home/dima/debianstuff/cogapp/cogapp/test_cogapp.py", line 970, in check_error_report
    assert expected == s
AssertionError

Thanks

message line will not apear on a new line (on Windows 10?)

if I use the cog.msg( ... ) the first call will be directly printed on the Cogging ... output line.

Every following message will be printed on a own separate line.

Suggestion: Add a new_line to the Cogging output