minoru / hakyll-convert Goto Github PK

We already have an issue for extensive docs #4 (which has enormous scope and should be extensively separated IMO), but it would be good to just have simple brief instructions somewhere in README that show how this thing is supposed to be used. Nothing extraordinary or 100% correct. Just something. Because right now the repo creates the following feeling: alright we have a bunch of haskell stuff... What's next?

I just installed hakyll and hakyll-converter - both without issues using stack.
the hakyll test produced a nice web blog, but hakyll-converter is not working. I received from Google in the folder with the title of my blog a 3 files: feed.atom, followers.csv and settings.csv.

the command

/Desktop/gerastreeWeb/blogger_thinking$ hakyll-convert --format=blogger feed.atom posts

gives

hakyll-convert: user error (Could not understand Atom feed: feed.atom). 

did google change the configuration or did I do something wrong?

it would be good if it would work to prepare for input into hakyll.

ps. you could mention that the installation works with stack as well! thank you.

In the interim, we can suggest people delete all drafts before exporting, but eww.

Blogger.distill reduces a list of comment authors into a single string, separating names with spaces. We need to do some research and find out if it's actually possible for a comment to have multiple authors. If it is, then we probably need a nicer format, e.g. separate names with commas, and maybe put an "and" before the final author's name.

Commit c012802 fails to compile with Stackage LTS 21.6 (GHC 9.4.5):

% stack test --pedantic
hakyll-convert> build (lib + exe + test)
Preprocessing library for hakyll-convert-0.3.0.4..
Building library for hakyll-convert-0.3.0.4..
Preprocessing test suite 'golden' for hakyll-convert-0.3.0.4..
Building test suite 'golden' for hakyll-convert-0.3.0.4..
[1 of 6] Compiling Golden.GoldenTestHelpers

/home/minoru/src/hakyll-convert/test/golden/Golden/GoldenTestHelpers.hs:14:1: error: [-Worphans, -Werror=orphans]
    Orphan instance: instance Show DistilledPost
    Suggested fix:
      Move the instance declaration to the module of the class or of the type, or
      wrap the type with a newtype and declare the instance on the new type.
   |
14 | deriving instance Show DistilledPost
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Progress 1/2

--  While building package hakyll-convert-0.3.0.4 (scroll up to its section to see the error) using:
      /home/minoru/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_3.8.1.0_ghc-9.4.5 --verbose=1 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.8.1.0 build lib:hakyll-convert exe:hakyll-convert test:golden test:spec --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1

We have two tests that ensure error gets called in some edge cases. Those tests use tasty-expected-failure to catch the exception and turn the failure into an "expected failure", making the test suite pass. However, with tasty 1.4 this no longer works: the exceptions propagate past tasty-expected-failure and fail the tests.

Our version bounds disallow tasty 1.4, so our users are fine. However, Stackage is looking to upgrade to tasty 1.4 (commercialhaskell/stackage#5795), and this incompatibility might prevent hakyll-convert from being included in there.

I filed nomeata/tasty-expected-failure#24 in hopes that the developers of tasty-expected-failure might have some ideas of what's going on, so I can fix the issue before we're booted from Stackage.

I believe the missing pattern in case could've been found by the compiler, if only we compiled with -Wall. So let's add those to package.yaml.

hakyll-convert should is cross-platform. To keep it that way, we need to have a Windows builder in CI which will check that we didn't break something.

I wrote golden tests for Hakyll.Convert.Blogger.readPosts and Wordpress.readPosts. They rely on a Show instance for types that represent a post. The default implementation prints everything out in a single line, so when a test fails, it's quite hard to see what exactly the difference is.

Furthermore, Wordpress posts are represented by a type defined in an external library. Depending on its Show instance for testing is probably a bad idea.

These tests should be rewritten with savePost, which is our function. That will lean them towards integration testing, which is fine. We can then use tasty-mgolden to provide us with nice diffs on test failures.

Blocked on #19.

Hi,

It looks like the latest version of hakyll-convert currently published on Hackage has a dependency on feed package without an upper bound specified.

A new version of feed will be released soon, incorporating non-backwards compatible changes to feed interface (String was replaced with Text throughout and xml-types types were adopted). Changes are largely mechanical but you'll probably want to ensure that future version of hakyll-convert can build with newer feed.

To prevent breakage, you may want to release a new version of hakyll-convert with an upper bound specified now, then push a version compatible with feed-1.0 after that comes out later.

Use the tip of the feed PR to test your package against: https://github.com/dzhus/feed/commit/259bab0. Please report any issues to the aforementioned PR thread.

This project currently uses GitHub Actions, but its "cache" action turned out to be quite unreliable:

• on macOS, cached executables aren't executable after being restored. Our cache contains the compiler, so this is obviously a problem;
• Windows doesn't ship tar, so restoring the cache always fails. This is less catastrophic than the macOS one, since the job simply takes longer to finish (≈20 minutes versus ≈2 minutes).

What can I do about it?

1. be a good FLOSS citizen and fix the "cache" action. Yeah, no, I'm not that married to GitHub Actions.

2. disable the cache for macOS and Windows. That means waiting ≈20 minutes instead of ≈2 minutes. Don't like the wait.

3. move to a different CI provider. There's plenty of them out there, and that's what I'm inclined to do.

I'll decide after I slept on this.

(This was supposed to be the last step of #3, but it might be a bit too large a task, so let's split it out.)

We currently repeat some things like chunks of HTML in both Hakyll.Convert.Wordpress and Hakyll.Convert.Blogger. These should be factored out into Hakyll.Convert.Common.

tasty-mgolden provides a diff output when the result doesn't match the expectation. We should migrate to it from tasty-golden, which shows the whole result and the whole expected value.

(This is extracted from #20. This is currently impossible because LTS 6.9 doesn't have all the necessary dependencies.)

Ideally showing where in the Blogger settings to export, and what you need to do in your hakyll site (should be fairly minimal, just a pattern).

Also need to see if the Cool URI thing is possible (if Google would kindly redirect the domain to your own site)

Cf. https://neilmitchell.blogspot.fr/2017/04/hlint-on-travisappveyor.html

Ugh, there must be a best practice for this. The WP feed stores results in semi-HTML with paragraphs (?) separated by whitespace rather than tags. I'm not sure what the right way to deal with this is: generate p tags around each block? In the 0.1 release I stupidly “avoided” the issue by declaring the format to be markdown (which violates our aim about not formatting your data!), which has the flaw that some things are accidentally treated as markdown syntax that weren't meant to be (hence the format thing).

Thanks to @Mikolaj for pointing this out!

Most of the work is there. We should have a DistilledComment type perhaps. The default converter could have an option to include comments or not.

It'd be nice to know that hakyll-convert works on macOS as well. There were some issues with testing it on GitHub Actions (see #40), so I abandoned the idea for now. If anyone wants to help, please drop a comment!

I'm pretty bad at keeping the code style consistent. Need to add a tool that would:

1. make it easy for me to reformat the code before saving it;
2. make it easy for CI to check that I didn't mess up anything.

Travis CI recently introduced limits even for open-source projects. I'm not sure if we will ever reach 1000 minutes/month, but it's be annoying to drop everything and wait for them to up a limit for us. So let's pre-emptively move to a different CI.

hakyll-convert (the executable) contains an important function savePost that writes posts to disk. We need to ensure its correctness, so we have to write tests for it. To achieve that, the function must be moved into the hakyll-convert (the library).

While writing tests for Blogger.distill, I discovered that comments without an author are rendered as if the author name is empty.

If at all possible, we should claim that those are written by an author named "Unknown", to make it look better on the page:

On unknown date, Unknown wrote: …

I just created a new backup of my testing blog (blog-11-07-2020.xml.gz), and hakyll-convert fails to convert it:

$ stack run -- blog-11-07-2020.xml output2
hakyll-convert: Block of entries with no post?!
This is an example comment.

It would be good to have a couple of backup samples right in the codebase for testing purposes. We can also use them for integration testing on CI.

I just ran hakyll-convert on my exported WP posts and noticed that the generated files are named based on the wp:post_id. I think it would be better if the file name was based on wp:post_date instead, since it then matches better with the naming scheme suggested for Hakyll posts (statement based on the output of hakyll-init).