Git Product home page Git Product logo

wikmd's Introduction

wikmd

GitHub Workflow Status GitHub Repo stars GitHub release (latest by date) GitLab

ℹ️ Information can be found in the docs: https://linbreux.github.io/wikmd/

What is it?

It’s a file-based wiki that aims to simplicity. Instead of storing the data in a database I chose to have a file-based system. The advantage of this system is that every file is directly readable inside a terminal etc. Also when you have direct access to the system you can export the files to anything you like.

To view the documents in the browser, the document is converted to html.

preview

Features

  • knowledge graph
  • NEW plugin system: drawio integration, alerts, embedded pages
  • git support (version control)
  • image support including sizing and referencing
  • math/latex
  • code highlight
  • file searching
  • file based
  • dark theme
  • codemirror for editing
  • password protection

Using the wiki

Installation

Detailed installation instruction can be found here.

Development

Instructions on the easiest way to develop on the project can be found here.

Plugins & Knowledge graph (beta)

More info can be found in the docs.

wikmd's People

Contributors

artivis avatar barakstout avatar dependabot[bot] avatar fabacab avatar gtsitsiridis avatar jerakin avatar kura avatar linbreux avatar mik3rizzo avatar oliver-hanikel avatar p-vitt avatar poblabs avatar potatoes1286 avatar reactorcoremeltdown avatar testoe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

wikmd's Issues

Some links try to load invalid files

My markdown files already have relative links because most of the time when I'm editing them I'm using VS Code and similar tools. If I click on links that reference another markdown file the extension will be added twice and the page fails to load.

for example if I have a link like this [Some random text](OtherFile.md) it will generate a URL that ends with /OtherFile.md but that page than tries to load /OtherFile.md.md

Other cases would be files that seem to be supported but are not .md files, for example like .txt and .ini. Those files are listed (on the /list pages) but when I click them the URL is generated without the extension and than the page tries to load an .md file that doesn't exist.

My guess is that originally one wanted to keep the wiki URLs clean and without extensions but that seems to break in a lot of cases.

At this point I think there would be two options to solve those issues:

  1. Always just use the original filename including extension in the URL. This would be simple and straight forward but the URLs would be a bit less clean.
  2. Another solution would be to change the way files are looked up. First check the URL if it already points to a valid file and if not try other extensions. So if I open /Folder/PageA test if there really is a file without extension and open that. If there is not try to append ".md" and test if that file exists. This would allow for clean URLs but fix the issues mentioned above.

I like Option 1 better and it would be the least surprising to users, but I guess there was a reason to drop the ".md" from the URLs in the first place.

UnicodeDecodeError on search

I just tried the server withing the docker container as per the instructions and filled it with a few markdown files to test it.
When typing in a search term and sending it, I get an UnicodeDecodeError as follows:

UnicodeDecodeError

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
Traceback (most recent call last)

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in __call__

    return self.wsgi_app(environ, start_response)

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2450, in wsgi_app

    response = self.handle_exception(e)

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1867, in handle_exception

    reraise(exc_type, exc_value, tb)

    File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise

    raise value

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app

    response = self.full_dispatch_request()

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request

    rv = self.handle_user_exception(e)

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception

    reraise(exc_type, exc_value, tb)

    File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise

    raise value

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request

    rv = self.dispatch_request()

    File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

    File "/app/wiki.py", line 179, in file_page

    return search()

    File "/app/wiki.py", line 80, in search

    fin = f.read()

    File "/usr/lib/python3.6/codecs.py", line 321, in decode

    (result, consumed) = self._buffer_decode(data, self.errors, final)

    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Temp created as root

Hi

I'm using docker-compose to start service :

version: "2.1"
services:
  wikmd:
    image: linbreux/wikmd:latest
    container_name: wikmd
    environment:
      - PUID=1001
      - PGID=1001
      - TZ=Europe/Warsaw
      - WIKI_DIRECTORY=/pages
      - HOMEPAGE=homepage.md
      - HOMEPAGE_TITLE=Homepage
      - SYNCH_WITH_REMOTE=1
      - REMOTE_URL="XXXXX"
      - GIT_USER=wikmdSYnc
      - [email protected]
      - MAIN_BRANCH_NAME=workingcopy
    volumes:
    - pages:/pages
    ports:
      - 5000:5000
    restart: unless-stopped
volumes:
  pages:

After docker-compose up two errors are showing up:

image

/wiki is still used as default or just wiki tries to empty out folder. Application is trying to move data from /wiki it to non-existing temp folder not permitted to user with GID 1001 (non-root user on host machine) -> it causes loads of logs and machine load

/pages is created and correct permission are added

New local mode feature throws errors due to missing js directory

When enabling the new local mode feature the container fails to start up due to a missing js directory.

[2022-08-25 18:55:04,926] INFO in web_dependencies: Downloading dependency https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/js/bootstrap.bundle.min.js
[2022-08-25 18:55:05,167] INFO in web_dependencies: Writing dependency >>> /app/wikmd/static/js/bootstrap.bundle.min.js
Traceback (most recent call last):
  File "/app/wikmd/wiki.py", line 45, in <module>
    "web_deps": get_web_deps(cfg.local_mode, app.logger)
  File "/app/wikmd/web_dependencies.py", line 74, in get_web_deps
    download_web_deps(logger)
  File "/app/wikmd/web_dependencies.py", line 96, in download_web_deps
    with open(dep_file_path, "wb") as file:

PermissionError with default config

I just spin up a docker container as basic as it can be.

version: "2.1"
services:
  wikmd:
    image: linbreux/wikmd:latest
    container_name: wikmd
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Madrid
    ports:
      - 5000:5000
    restart: unless-stopped

I start the application and the default page is shown, but if I try to edit it or create a new one I am having this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/wikmd/wiki.py", line 272, in edit_homepage
    save(page_name)
  File "/app/wikmd/wiki.py", line 66, in save
    content = plugin.process_md(content)
  File "/app/wikmd/plugins/draw/draw.py", line 26, in process_md
    return self.search_for_pattern_and_replace_with_uniqueid(md)
  File "/app/wikmd/plugins/draw/draw.py", line 83, in search_for_pattern_and_replace_with_uniqueid
    self.create_draw_file(filename)
  File "/app/wikmd/plugins/draw/draw.py", line 68, in create_draw_file
    shutil.copyfile(os.path.join(self.this_location, "default_draw"), path_to_file)
  File "/usr/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/app/wikmd/plugins/draw/drawings/draw_e03c70da-ab4a-4c53-bbe4-ead5a7fa36ee'

Maybe something related with permissions on /app/wikmd/ path?

Usability and refactoring

Hi,
Nice work again (I'm the guy from Reddit)!

I've found some small usability issues related to item positioning, general consistency and style. I'm trying to improve the design of the website (I've switched to bootstrap 5.1) in order to reach a better user interaction.
I'm also trying to refactor a little bit the project, both for the structure side and from the coding side (comments, etc.).

I've also some ideas, but I am not sure I'll manage to implement them soon, so I'll leave them as a suggest:

  • sidebar to show the hierarchy of the wiki, maybe collapsible
  • breadcrumbs to show the current page and the path
  • .json/.yaml config file, in my opinion is handier than set up env variables
  • periodic git push to personal remote repo

Feel free to check out my fork.
One of these days I'll do a PR if you agree.

Best regards,
Michele

wikmd on Windows: Bug when converting markdown in internal editor

Hi,

It seems there is an issue when converting written markdown in the internal editor.
Tested on a German/European Windows version.
It seems it saved the md file as ANSI/ISO-8859-1 instead of UTF-8.
So Pandoc seems to complain about that:

INFO:wiki:Saving >>> 'Test' ...
INFO:wiki:Converting to HTML with pandoc >>> 'wiki\Test.md' ...
INFO:wiki:New git commit >>> Add page 'Test' on 2022-04-23 01:48:26.530440
INFO:wiki:Pandoc died with exitcode "92" during conversion: b'UTF-8 decoding error in wiki\Test.md at byte offset 31 (f6).\r\nThe input must be a UTF-8 encoded text.\r\n'

pandoc tries to convert the favicon

I noticed that two GET requests are made once a wiki page has been opened:

  • page request
  • favicon request

Adding a bit of logging details, this is the console out:

[2022-03-14 15:20:35,028] INFO in wiki: Converting wiki\How to use the wiki.md to HTML with pandoc
[2022-03-14 15:20:35,798] INFO in wiki: Showing html page of How to use the wiki
[2022-03-14 15:20:36,408] INFO in wiki: Converting wiki\favicon.ico.md to HTML with pandoc
[2022-03-14 15:20:36,409] INFO in wiki: source_file is not a valid path

Pandoc tries to convert to HTML both the page (that is correct) and the favicon (that is not desirable).
I'll try to fix that.

Docker setup fails with Git Error

Running wikmd as docker container with following error.

wikmd    | [2022-05-28 19:35:41,212] INFO in git_manager: Initializing existing repo >>> wiki ...
wikmd    | [2022-05-28 19:35:41,212] INFO in git_manager: Creating 'main' branch ...
wikmd    | [2022-05-28 19:35:41,216] ERROR in git_manager: Existing repo initialization failed >>> Cmd('git') failed due to: exit code(128)
wikmd    |   cmdline: git branch -M main
wikmd    |   stderr: 'fatal: unsafe repository ('/app/wikmd/wiki' is owned by someone else)
wikmd    | To add an exception for this directory, call:
wikmd    |
wikmd    |      git config --global --add safe.directory /app/wikmd/wiki'

Troubleshooting done,

  1. Configured PID GID to current user
  2. Ensured path/to/wiki has correct permission

Decoding issue in f.read() - Win10, python 3.11.2

I installed wikmd in Windows 10, python3.11.2 (after a little bit of tweaking #97 )

I've got another error just after the wiki started:

Traceback (most recent call last):
  File "D:\Wiki (static)\wikmd\wiki.py", line 465, in <module>
    run_wiki()
  File "D:\Wiki (static)\wikmd\wiki.py", line 457, in run_wiki
    setup_search()
  File "D:\Wiki (static)\wikmd\wiki.py", line 442, in setup_search
    search.index_all(cfg.wiki_directory, items)
  File "D:\Wiki (static)\wikmd\search.py", line 87, in index_all
    content = f.read()
              ^^^^^^^^
  File "C:\dev\python\python311\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 4807: character maps to <undefined>

It seems that f.read() tries to decode the file as CP1252 whereas it is encoded as UTF-8.

EDIT
I solved it adding encoding="utf8" where files are opened, like:

with open(file_path, encoding="utf8") as f:

Maybe I'll make a PR if I don't get other errors.

Pandoc error (MacOS)?

I tried installing wikmd on a MacOS with Python 3.9.2 and Pandoc 3.11, and got the error below. I'm wondering if the problem is because I was using the newest version of pandoc? (I couldn't find any reference to Pandoc or Python versions in the docs) I'm not sure what else it might be. Got this error:

`$ python3 wiki.py

[2023-03-07 18:34:08,508] INFO in git_manager: Initializing existing repo >>> wiki ...

  • Serving Flask app "wiki" (lazy loading)
  • Environment: production
    WARNING: This is a development server. Do not use it in a production deployment.
    Use a production WSGI server instead.
  • Debug mode: on
    [2023-03-07 18:34:14,037] INFO in wiki: Showing HTML page >>> 'homepage'
    [2023-03-07 18:34:14,037] INFO in wiki: Converting to HTML with pandoc >>> 'homepage' ...
    [2023-03-07 18:34:16,334] INFO in wiki: Showing HTML page >>> 'homepage'
    [2023-03-07 18:34:16,336] INFO in wiki: Converting to HTML with pandoc >>> 'homepage' ...
    [2023-03-07 18:34:17,098] ERROR in wiki: Conversion to HTML failed >>> Pandoc died with exitcode "83" during conversion: b'Traceback (most recent call last):\n File "/Users/this.user/Library/Python/3.9/bin/pandoc-xnos", line 8, in \n sys.exit(main())\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandocxnos/main.py", line 38, in main\n m.main(stdin, stdout)\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandoc_fignos.py", line 608, in main\n PANDOCVERSION = pandocxnos.init(args.pandocversion, doc)\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandocxnos/core.py", line 224, in init\n _PANDOCVERSION = _get_pandoc_version(pandocversion, doc)\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandocxnos/core.py", line 179, in _get_pandoc_version\n raise RuntimeError(msg)\nRuntimeError: Cannot understand pandocversion=3.1.1\nError running filter pandoc-xnos:\nFilter returned error status 1\n'
    [2023-03-07 18:34:17,830] ERROR in wiki: Conversion to HTML failed >>> Pandoc died with exitcode "83" during conversion: b'Traceback (most recent call last):\n File "/Users/this.user/Library/Python/3.9/bin/pandoc-xnos", line 8, in \n sys.exit(main())\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandocxnos/main.py", line 38, in main\n m.main(stdin, stdout)\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandoc_fignos.py", line 608, in main\n PANDOCVERSION = pandocxnos.init(args.pandocversion, doc)\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandocxnos/core.py", line 224, in init\n _PANDOCVERSION = _get_pandoc_version(pandocversion, doc)\n File "/Users/this.user/Library/Python/3.9/lib/python/site-packages/pandocxnos/core.py", line 179, in _get_pandoc_version\n raise RuntimeError(msg)\nRuntimeError: Cannot understand pandocversion=3.1.1\nError running filter pandoc-xnos:\nFilter returned error status 1\n'

$ pandoc --version

pandoc 3.1.1
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: /Users/this.user/.local/share/pandoc
Copyright (C) 2006-2023 John MacFarlane. Web: https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

`

Add support for a table of contents

I would like to be able to insert a table of contents in some pages, ideally with some macro like [toc]. Would this be a possible future addition.

Toggling darkmode results in an error

When pressing the toggle darkmode icon in the upper left, i get the following error:

FileNotFoundError FileNotFoundError: [Errno 2] No such file or directory: '/wiki/toggle-darktheme/None.md'

If i afterwards go back to the main page, or any other page, the theme has actually toggled, so its just the redirect that is not working.

I am running it from docker.

Stop using JS from CDNs and host locally instead

I don't really like that an app I am hosting myself on my network is requesting JS files from an external CDN, even with SRI. Can those JS files just be included in the repo and hosted locally instead?

Switching theme (light/dark) removes unsaved content

If you are on /add_new and switch to a different theme the content you were writing is discarded.

Preferably the switching theme shouldn't destroy the content, at minimum if you have written any content it should ask if you want to discard your changes or cancel.

Installation issue due to lxml - Win10, python 3.11.2

I tried to install the wikmd on Windows 10 (python 3.11.2, venv).

I got this error installing requirements.txt:

Running setup.py install for lxml ... error
  error: subprocess-exited-with-error

  × Running setup.py install for lxml did not run successfully.
  │ exit code: 1
  ╰─> [96 lines of output]

       ...

      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      *********************************************************************************
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of lxml
  ...
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lxml

Any cue? Thanks.

EDIT:
I solved the problem installing lxml==4.9.2 instead of 4.9.1 (that is in the requirements.txt).

Exception for custom wiki directory

I set up wiki_directory = "m-iki" and I got this exception:

Traceback (most recent call last):
  File "/opt/wikmd/wiki.py", line 39, in <module>
    wrm = WikiRepoManager(flask_app=app)
  File "/opt/wikmd/git_manager.py", line 45, in __init__
    self.__git_repo_init()
  File "/opt/wikmd/git_manager.py", line 55, in __git_repo_init
    self.__init_remote_repo()
  File "/opt/wikmd/git_manager.py", line 88, in __init_remote_repo
    if os.listdir(self.wiki_directory):
FileNotFoundError: [Errno 2] No such file or directory: 'm-iki'

Search button results in werkzeug.exceptions.BadRequestKeyError

Using the search button results in a werkzeug.exceptions.BadRequestKeyError.

werkzeug.exceptions.BadRequestKeyError: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand.
KeyError: 'PN'

Used the latest Docker build from Github.

Line edit fields are light in dark mode

Usually text fields are darker than the background in dark mode themes, which is a lot easier on the eyes.

Current look
image

Example of line edits with darker color
image

Branch rename failed when creating a new instance.

When launching a fresh new instance I get the following error,

ERROR in git_manager: New local repo initialization failed >>> Cmd('git') failed due to: exit code(128)
  cmdline: git branch -M main
  stderr: 'error: refname refs/heads/master not found
fatal: Branch rename failed'

After a quick look I think this line should really be a git checkout -b <branch_name>.

Permissions issue in MacOS

Using wikmd in MacOS. The previous version of wikmd worked perfectly. This new version however will not start, throwing the following error:

Initializing existing repo >>> wiki ...
DrawIO integration
Alerts system
Traceback (most recent call last):
  File "wiki.py", line 52, in <module>
    cache = Cache(cfg.cache_dir)
  File "/Users/jeff/Development/Python/wikmd/cache.py", line 14, in __init__
    os.makedirs(path)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 1] Operation not permitted: '/dev/shm' 

Any suggestions? Thanks!

Add saving via ctrl+s

Just an idea. Will implement this later.

document.addEventListener('keydown', e => { 
if (e.ctrlKey && e.key === 's') { 
// Prevent the Save dialog to open
 e.preventDefault(); 
// Place your code here
 console.log('CTRL + S'); } });

Wikilinks support?

Was wondering if there are any plans for this project to include wiki links support, like Obsidian

Something like this : [[link]]

Docker image is huge

The image is way too big for a wiki that tries to be lean.

Architecture Size¹
amd64 448MB
armhf 563MB
aarch64 703MB
¹Size of #67 images

I'll try to improve this by using alpine as the base image.
But there would also be other ways like reducing the amount of dependencies or switching out dependencies. For example the installed size of pandoc is 100MB on arm64 but markdown would only take up 57KB.

New search bug when using absolute paths and when moving documents

I spotted 2 bugs in the new search code after leaving it running in my home lab.

  1. When the wiki documents directory is an absolute path the watchdog indexer stores the abs path rather than rel path, breaking links in the results page. It works when an index_all() is called but not via the natural create/update functionality that the watchdog handles.
  2. Moving a document between directories causes the old document to remain in the index until a full reindex is done 'cos I didn't implement on_moved event handling... :(

Both of these should be pretty easy fixes, I'll work on a patch over lunch later today.

git checkout error for initialized repo

When starting again the wiki, the initialization of the repo fails and commits will not be added.

Here's the log:

cmdline: git checkout -b master
stderr: 'fatal: a branch named 'master' already exists'

I may have fixed it dropping the -b parameter if the repo has already been initialized.

An option to list only direct children

I have a pretty large wiki (multiple thousands of files and a ton of folders) but they are structured in a very hierarchical manner, so at every level there are only about a dozen elements. Currently if I list at a very high level (for example the root folder) it iterates all files and takes a very long time to render the page.

It would be great to have the option to only list direct children but not grand-children.

Commit bug due to (my) refactoring

This line:

repo.git.git_commit('-m', commit)

Should be:

repo.git.commit('-m', commit)

Otherwise commits don't work.

I was developing git push to remote repos when I discovered this annoying bug. I may have introduced it due to the automatic refactoring. Sometimes it could be a pain: my bad, sorry!

PS. I've already implemented configuration through an handy .yaml file ;-) Tomorrow I'll do a PR (with the above bug fixed as well).

Win: every commit adds new lines in the file

On windows every commit adds a new line for each original line of the file; the lines of the file are doubbled.

It should be a problem related to line ending, I'm currently work to fix that.

Docker file

Hi @Linbreux,

I've created a Docker file for wikmd that is loosely inspired by those of the linuxserver community and I've successfully tested it on amd64 and arm (raspberry pi + raspbian). You can find them on my fork.

I don't really plan on actively maintaining it as I won't fully transition to wikmd just yet. Are you interested in a PR? Maybe that could even support a wikmd application for support by the linuxserver community who knows.

Directory Listings Missing First Character

Hello,
I have observed that when listing files (and clicking the subsequent links), the first character of the top level directory is missing. As a written example:

  • Wiki/Technical/Cisco.md

with Wiki as the root, when you click "List all pages" you see:

  • echnical/Cisco

These files are contents searchable- so it's not a wiki wide issue. I've had a look to try and work it out, but couldn't see where in the code the issue was (apologies). When listing all pages, the wiki doesn't throw out any errors. When you click one of the bad links you get the below error in the log:

FileNotFoundError: [Errno 2] No such file or directory: '/home/sqshr/Wiki/echnical/Cisco.md'

I know this error isn't useful for the above issue (hence omitting the Traceback)- but want to show willing :)

Thanks!

Changing Webroot

I want to host my wiki at mydomain.xyz/wiki. I have set up the correct reverse proxy via caddy, but the frontend returned by wikmd does not have the correct links. How do I change the webroot?

Fails to render when running from systemd

Wanted to thank you for this project. I needed something like this to maintain my notes. Great work!

So I am running it locally and want to start the wiki via systemd.
When I run from shell, the wiki works well and renders properly.
When I run via systemd, it does not render the pages.

Systemd Unit

[Unit]
Description=My Notes wiki
After=network.target

[Service]
User=myusename
WorkingDirectory=/home/daksh/wiki/wikmd/
Environment=FLASK_APP=wiki.py
Environment=FLASK_DEBUG=1
Environment=FLASK_ENV=development
Environment=FLASK_CONFIG=development
ExecStart=/home/myusename/wiki/wikmd/.venv/bin/python3 -m flask run --host=0.0.0.0
Restart=always

[Install]
WantedBy=multi-user.target

Traces

* wikmd.service - My Notes wiki
   Loaded: loaded (/etc/systemd/system/wikmd.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-11-29 11:51:36 EST; 2s ago
 Main PID: 19283 (python3)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/wikmd.service
           ├─19283 /home/myusename/wiki/wikmd/.venv/bin/python3 -m flask run --host=0.0.0.0
           └─19311 /home/myusename/wiki/wikmd/.venv/bin/python3 -m flask run --host=0.0.0.0

Nov 29 11:51:36 systemd[1]: Started My Notes wiki.
Nov 29 11:51:36 python3[19283]:  * Serving Flask app "wiki.py" (lazy loading)
Nov 29 11:51:36 python3[19283]:  * Environment: development
Nov 29 11:51:36 python3[19283]:  * Debug mode: on
Nov 29 11:51:36 python3[19283]:  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Nov 29 11:51:36 python3[19283]:  * Restarting with stat
Nov 29 11:51:36 python3[19283]:  * Debugger is active!
Nov 29 11:51:36 python3[19283]:  * Debugger PIN: 125-541-320

NavBar

I am really missing sticky NavBar on the left for this library. i think it might be bit different request than #51
Virtually all docs have this.

image
image
image

[Feature request] optional/custom log path

Hi,

Could it be possible to make the file logging optional?

Furthermore, it would be great to be able to set a custom path for the logging file (wikmd.log). Maybe through an environment variable? Depending on how the wiki is launched, it may not have the permissions to write at the path it is launched from.

Thx.

.git folder listed in "List all pages"

I'm running the wiki on Windows 10, with Python 3.11.2 (venv) and a remote repo set up.

When I list all the pages also the files inside the .git folder are listed.

image

exception when deleting a wiki page

I discovered a little typing error here that leads to an exception while trying to delete a page:

[2022-04-05 00:03:37,800] INFO in wiki: Showing HTML page >>> 'test'
Traceback (most recent call last):                                                                          
  [...]
  File "C:\Users\mrizz\Desktop\Workspaces\wikmd\wiki.py", line 251, in remove
    git_sync_thread = Thread(target=wrm.git_sync, args=(page_name, "Remove"))
NameError: name 'page_name' is not defined

Redirect all .md files to wikimd?

For me, the ideal setup would be one where I am able to have all my markdown files rendered and displayed by wikimd on the "main" domain and on a subdomain (or /edit) the same URL would allow the file to be edited?
So for example, at domain.com/file.md I'd be able to just view the file as if it were a normal page then at edit.domain.com/file.md I'd be able to sign and edit the file.

Currently, I use Nginx to forward all URLs ending with .md to a simple js markdown "processor" that I wrote that renders and displays the file. For now, I edit these files through SSH + nano 😔
Would it possible to forward all .md files to wikimd to display even if they aren't ones that are in the scope of wikimd?

Further, would it be possible to hide the header for wikimd?

[bug] local variable 'repo' referenced before assignment

Hi,
I've hit the following error on l.147:

Traceback (most recent call last):
  File "wiki.py", line 315, in <module>
    gitcom()
  File "wiki.py", line 146, in gitcom
    repo.git.add("--all")
UnboundLocalError: local variable 'repo' referenced before assignment

Directory traversal vulnerabilities

I have discovered several directory traversal vulnerabilities. The capabilities are as follows:

  • Any .md file on the system can be read by an unauthenticated attacker
  • The contents of sibling directories to the wiki directory can be listed by unauthenticated attacker in certain circumstances.
  • Any .md file on the system can be overwritten by an unauthenticated attacker by default (authentication required if edit password protection is enabled)
  • Any .md file on the system can be deleted by an unauthenticated attacker by default (authentication required if edit password protection is enabled)
  • New directories and .md files can be created on the system with arbitrary contents and paths by an unauthenticated attacker by default (authentication required if edit password protection is enabled)

Details as follows:

Directory traversal in /list/

The /list/ request can be used to get a list of files in the wiki, or in a subdirectory of the wiki.

In wiki.py the list_wiki function that handles /list/ requests checks whether the canonical form of the requested path shares a common prefix with the configured "safe" wiki directory:

wikmd/wiki.py

Lines 102 to 107 in 04482d3

@app.route('/list/<path:folderpath>/', methods=['GET'])
def list_wiki(folderpath):
folder_list = []
safe_folder = os.path.realpath(cfg.wiki_directory)
requested_path = os.path.join(cfg.wiki_directory,folderpath)
if os.path.commonprefix((os.path.realpath(requested_path),safe_folder)) != safe_folder:

This is intended to prevent directory traversal. However, this does not prevent traversal to sibling directories whose names start with the same name as the configured wiki directory.

For example, consider the following directory structure:

home
└── user
    └── wikmd
        ├── wiki
        │   ├── homepage.md
        │   └── foo.md
        └── wiki_secret
            └── secret.md

If an attacker requests /list/../wiki_secret/, the following happens:

  1. folderpath is set to ../wiki_secret/
  2. cfg.wiki_directory is configured to serve out of the wiki directory, so safe_folder becomes /home/user/wikmd/wiki
  3. requested_path is created by joining cfg.wiki_directory and folderpath, which results in /home/user/wikmd/wiki/../wiki_secret/
  4. requested_path is canonicalised with os.path.realpath(), which returns /home/user/wikmd/wiki_secret.
  5. os.path.commonprefix() is called on the canonicalised request path and safe_folder, which searches character-by-character and returns the prefix substring up until the first difference. Since /home/user/wikmd/wiki and /home/user/wikmd/wiki_secret are the arguments, the result is /home/user/wikmd/wiki
  6. This is compared to safe_folder, which is /home/user/wikmd/wiki, and therefore the check passes.
  7. The contents of /home/user/wikmd/wiki_secret are listed.

This can be demonstrated simply by sending GET /list/../wiki_secret/ and two enter key presses to the server using netcat.

Directory traversal in page display

There is no directory traversal protection inside the file_page function. As such, any .md file on the system can be displayed.

This can be trivially demonstrated with any default install by sending GET /../README and two enter presses to the server using netcat. This will return the HTML rendered copy of the README.md file in the wikmd directory.

Directory traversal in page edit

There is no directory traversal protection inside the edit function. This can be used both for reading and writing any .md file on the system. This function is unauthenticated by default, but may be configured to require a password.

When a GET request is sent to an /edit/ route, this function displays the existing contents of the file. This can be demonstrated by sending GET /edit/../README and two enter presses to the server using netcat. This will display the HTML of the edit template with the contents of the README.md file in the wikmd directory included.

When a POST request is sent to an /edit/ route, the following code path is taken:

wikmd/wiki.py

Lines 276 to 282 in 04482d3

filename = os.path.join(cfg.wiki_directory, page + '.md')
if request.method == 'POST':
page_name = fetch_page_name()
if page_name != page:
os.remove(filename)
save(page_name)

The fetch_page_name() function acquires the page name from the PN form variable.

The page variable contains the portion of the URL after /edit/. No traversal check is performed. If the value provided in the PN form variable does not match the path in the URL, the file specified by the URL is deleted. Since the path is vulnerable to traversal, any .md file on the system can be deleted using this method.

The save() function saves the page. Since the page name is taken from the PN form variable without validation, this allows any .md file on the system to be written to. The save() function creates the full directory path to the target file name, so this enables the creation of directories anywhere on the system.

Directory traversal in new page creation

The add_new() function handles /add_new. This function is unauthenticated by default, but may be configured to require a password.

The PN form variable is used to specify the name/path of the page to create, but no traversal protection is performed. This function calls into save() in the same way that the page edit functionality does, with the same impact.

Directory traversal in remove

The remove() function handles /remove/ route requests. This function is unauthenticated by default, but may be configured to require a password.

The path is taken from the URL and is not validated or protected against traversal. This allows for any .md file on the system to be deleted.


Remediation

The existing commonprefix() traversal check in list_wiki() is broken and should be removed.

The safe_join() function provided by Flask should be utilised instead of os.path.join() in all cases where one or more of the path arguments is untrusted. This function will raise a NotFound exception if traversal out of the base directory is attempted.

Publish a new Docker image

Hey. Any chance you can publish newer Docker images? The latest image doesn't contain any of the recently added functionality.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.