iwillspeak / docket Goto Github PK

We should add in support for stemming and stopword removal. We should also consider switching from term frequency to
some variant of TFIDF for search too. This would normalise search term frequency accross documents to hopefully filter out common words. We could also consider some kind of cutoff to prevent words common to all texts from being included in the index.

Originally posted by @iwillspeak in #20 (comment)

• Stemming and stopword removal
• TFIDF
• Drop insignificant terms?

Currently titles have an #id assigned to them to allow linking from within the page. These links can be copied and used to share a deep link to a certain page section. We could do with a few ergonomic improvements however:

• Highlight the target item in the TOC if an ID link is active
• Add # or link icon on hover to the titles to make it more obvious that these can be copied. We could even add js onclick handling to copy them.

We're using quite an outdated copy of pulldown now. We should fix that.

This causes a lot of unwanted HTTP 301 redirects. Things would be a lot nicer if we just emitted a link to the folder directly.

Fingers crossed this should be as simple as adding / to the TOC links here.

It would be super nice to have support for tables. Pulldown should support this, we just need to enable the extension feature and provide css for their styling.

This might be the better option for the search box placement:

Originally posted by @iwillspeak in #20 (comment)

Features added in #10 are broken.

diff --git a/src/docket.rs b/src/docket.rs
index 3ad1920..1ade956 100644
--- a/src/docket.rs
+++ b/src/docket.rs
@@ -162,18 +162,18 @@ impl Docket {
 }
 
 /// Map a collection, using Rayon.
-#[cfg(par_render)]
+#[cfg(feature="par_render")]
 fn map_maybe_par<T, F, U>(input: Vec<T>, f: F) -> Result<Vec<U>, Error>
 where
     T: Sync,
-    F: Fn(T) -> Result<U, Error> + Send + Sync,
+    F: Fn(&T) -> Result<U, Error> + Send + Sync,
     U: Send,
 {
     use rayon::prelude::*;
-    input.par_iter().map(f).collect()
+    input.into_par_iter().map(f).collect()
 }
 
-#[cfg(not(par_render))]
+#[cfg(not(feature="par_render"))]
 fn map_maybe_par<T, F, U>(input: Vec<T>, f: F) -> Result<Vec<U>, Error>
 where
     F: Fn(T) -> Result<U, Error>,
diff --git a/src/main.rs b/src/main.rs
index b0a5b7d..de5a033 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -45,11 +45,11 @@ struct Args {
     flag_target: Option<String>,
 }
 
-/// On Erorr Behaviour
+/// On Error Behaviour
 ///
 /// Chooses what should happen if an error happens when running the build.
 #[derive(PartialEq, Copy, Clone)]
-#[cfg_attr(not(watch), allow(dead_code))]
+#[cfg_attr(not(feature="watch"), allow(dead_code))]
 enum OnError {
     Skip,
     Exit,
@@ -77,7 +77,7 @@ fn main() {
     let target = path_or_default(args.flag_target, "build/");
 
     if args.flag_watch {
-        #[cfg(watch)]
+        #[cfg(feature="watch")]
         {
             use notify::{watcher, RecursiveMode, Watcher};
             use std::sync::mpsc::channel;
@@ -99,7 +99,7 @@ fn main() {
                 }
             }
         }
-        #[cfg(not(watch))]
+        #[cfg(not(feature="watch"))]
         eprintln!("Watch not supported.");
     } else {
         run(&source, &target, OnError::Exit);

As raised in #11 we should add an example of highliting to the docs.

It's often nice to have the documentation that's being generated kept in line with the
input markdown. Update the main docket executable to add a --watch flag which
will re-generate documentation when a file changes.

In the documentation it is said that pages are sorted asciibetically. This doesn't happen right now so the order of pages in the main TOC tree is dependent on the underlying filesystem.

Update the Docket constructor to add in a sort on the Vec of pages. Somewhere just after we have collected all the pages should be good.

Update the output HTML to set the viewport width with the viewport meta tag (https://developer.apple.com/library/content/documentation/AppleApplications/Reference/SafariWebContent/UsingtheViewport/UsingtheViewport.html). This should probably be set to just a little wider than the fixed width of the content.

Currently hilighting takes place when the page is rendered with hilight.js. It would be nicer if we didn't have a JS dependency for this. This should be possible from Rust with https://github.com/trishume/syntect.

Currently docket comes 'all or nothing'. The watch support and rayon parallelisation come at quite a cost though.

Add a set of feature flags, enabled by default, which allow users to choose if they want those features:

• watch - inotify and friends have a lot of dependencies
• parallel - Some people may be willing to dispense with Rayon

For the watch feature the body of the watch branch in main.rs needs extracting to a function and wrapping with a cfg(watch). When watching isn't enabled we will need a stub implementation which prints an error and exits.

For Rayon removal we will need two copies of the renderd_pages in docket.rs. One which uses Rayon and one which doesn't.

I can't get the syntax highlighting to work:

I did open the docs from an HTTP server, not from disk:

Failure has security issues causing dependabot alerts. Switch the error system away from it to another crate. Just a manual implementaiton of Error and a dyn Error return might be enough.

Generate some kind of 'index.json' and have some javascript to search it maybe? Could stick the search box in the footer somewhere on most pages, and somewhere in the header on the index?

Even better if it is web assembly and compiled from Rust.

Currently only the markdown files are coped. Maybe every file should be copied, and just markdown transformed. This would be useful for including images in the final output without having to copy them across later.