Git Product home page Git Product logo

zettelcon's Introduction

zettelcon

An external CLI tool for Zettlr note collections to automatically add backlinks to your note files. Edits files in-place, so be careful and try on a copy of your files first. Run whenever you feel like an update would be worth it.

Also check out zettelwarmer for finding new interconnections between your notes.

Assumptions

  • Zettlr-standard wiki-links are used ([[...]])
  • Single-line markdown syntax for headings is used (# some heading)
  • Note IDs are unique, also relative to the names of the notes
  • The backlink section is the last thing of a page
  • Two spaces are used for list indentation

Features

  • Single python file, no dependencies, under 250 sloc
  • Can run multi-core for large collections
  • Supports any note ID syntax without explicit regex
  • Supports collections that contain more than one note ID style
$ python3 zettelcon.py --help
usage: zettelcon.py [-h] -f FOLDER [-s SUFFIX] [-c] [-n NPROCS] [-ic]

Tool to insert automatic backlinks into Zettlr note collections or other
interlinked (markdown) files.

optional arguments:
  -h, --help            show this help message and exit
  -f FOLDER, --folder FOLDER
                        Path to folder with all the zettels in it.
  -s SUFFIX, --suffix SUFFIX
                        Suffix for the files to consider. Defaults to .md
  -c, --clear-backlinks
                        Instead of generating backlinks, revert all files to a
                        no-backlinks state
  -n NPROCS, --nprocs NPROCS
                        Number of worker processes to run for file reading and
                        writing.
  -ic, --ignore-cache   Don't use zettelcon's cache, force writing to _all_
                        Zettel files (even the ones where backlinks haven't
                        changed).


Future Work

  • Only cite a few words before and after the citation
  • Add horizontal break before backlinks
  • Output additional info such as islands, sinks, sources, etc.
  • Check out what happens if a file that links to another doesn't have a title
  • Somehow reduce the number of files that are written
    • Zettlr takes quite a while to update its indices after the files get changed so it would be benefitial to reduce write operations only to files that actually get new backlinks
  • Add option to clear all backlinks
  • Make file writing multi core
  • Add "last edited XXX" info field to markdown
  • Do an analysis of computation complexity as a function of number of files (or links)
    • As expected, it's linear in the number of files
    • Doubling the cores increases performance by ~1.5x
    • Ca. 1500 notes per second dual core, 2300 n/s quad core

Discarded Ideas

  • Add option to commit everything to git before updating links
    • Not needed, because you can simply define a shell function to do that
    • gen-backlinks() {
        cd $ZETTELPATH && git add -A && git commit -m "pre-zettelcon backup" ;\
        python ~/...path.../zettelcon.py -f $ZETTELPATH
      }

zettelcon's People

Contributors

whateverforever avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

lodestone

zettelcon's Issues

non-wiki/markdown links

I'm interested in using this for a bunch of inter-linked markdown files which don't use wiki-links and instead use more vanilla markdown syntax, like this:
[filename](./filename.md)

If I wanted to modify this to work with such links, do I just need to edit the regex in line 17?

Or is there other code I'd need to change as well?

Getting UnicodeDecodeErrors

I have tried to run this script on my files, but I got UnicodeDecodeError. I added to the open(...) functions the option encoding="utf8", and added a try statement here:

def get_file_outlinks(path):
    with open(path, "r", encoding='utf8') as fh:
        try:
            contents = fh.read()
            print(path)
        except UnicodeDecodeError:
            print('----', path)

With this I discovered some of my files were ANSI encoded, not utf-8 (?!?). I then converted them and run the altered script again. This time things went smoothly. It just complained many articles didn't have back references, which is actually quite cool, but could perhaps be printed a bit better, it's hard to read.

However, I downloaded the original and ran it, got UnicodeDecodeError again, line 187, function get_file_outlinks. Adding encoding="utf8" fixed this. What could be causing this? I think that adding some safeguard against encoding errors would be cool.

Full traceback of this last part,:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\Labore-05-Rosangela\Anaconda3\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\Labore-05-Rosangela\Anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Users\Labore-05-Rosangela\Dropbox\Zettelkasten - Copy\zettelcon.py", line 185, in get_file_outlinks
    contents = fh.read()
  File "C:\Users\Labore-05-Rosangela\Anaconda3\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 5445: character maps to <undefined>
"""

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File ".\zettelcon.py", line 230, in <module>
    main()
  File ".\zettelcon.py", line 49, in main
    process_directory(**params)
  File ".\zettelcon.py", line 58, in process_directory
    res = pool.map(get_file_outlinks, files)
  File "C:\Users\Labore-05-Rosangela\Anaconda3\lib\multiprocessing\pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\Labore-05-Rosangela\Anaconda3\lib\multiprocessing\pool.py", line 657, in get
    raise self._value
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 5445: character maps to <undefined>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.