friends-of-hugo / contentful-export Goto Github PK

A failed HTTP connection is now directly a Fatal error. Consider a retry mechanism.

Check how all fields are rendered. Refactor into switch statement as required

Allow the program to run as a server, responding to webhooks from Contentful

Key purpose: Preview / Staging (combined with hugo server)

Add unit tests and work with simple mocking for https requests.

Change local packages into proper github-based packages

Though the chances of running in trouble with the number of content types is much smaller, still it is not elegant to have a limit hardcoded.

Aim: Refactor the pagination from the content items in such a way that most code is reused.

Locale is now hard-coded.

Possibilities:

• Config file
• Command line parameter
• Advanced: multiple locales resulting in multiple output directories

Certain functions do not return errors as expected.

Example: https://github.com/adriaandejonge/contentful-hugo/blob/9a944b96c7bc4eb0024ececd73a36105345f8a47/dump/worker.go#L75

To start, create a README

Check on startup

Specifically this instance:

https://github.com/adriaandejonge/contentful-hugo/blob/9a944b96c7bc4eb0024ececd73a36105345f8a47/dump/types.go#L15-L23

Mac 64-bit version generates the following error: (on macOS High Sierra 10.13.4)

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 1 [running]:
_/Users/adriaanthuis/Documents/projects/contentful-hugo/translate.(*TranslationContext).translateField(0xc42012faf8, 0x0, 0x0, 0xc420
3bc570, 0x8, 0xc4203bc578, 0x5, 0xc4203bc580, 0x4, 0x0, ...)
        /Users/adriaanthuis/Documents/projects/contentful-hugo/translate/translate.go:75 +0x484
_/Users/adriaanthuis/Documents/projects/contentful-hugo/translate.(*TranslationContext).convertContent(0xc42012faf8, 0xc4203bb4d0, 0x
c420134200, 0x7, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/adriaanthuis/Documents/projects/contentful-hugo/translate/translate.go:38 +0x1d9
_/Users/adriaanthuis/Documents/projects/contentful-hugo/translate.(*TranslationContext).Translate(0xc42012faf8, 0xc4203bd590, 0x5, 0x
0, 0x0, 0xc4203731e0, 0x16, 0xc4203bb440, 0xc420373200, 0x18, ...)
        /Users/adriaanthuis/Documents/projects/contentful-hugo/translate/translate.go:23 +0x77
_/Users/adriaanthuis/Documents/projects/contentful-hugo/extract.(*Extractor).processItems(0xc42012ff08, 0x1400800, 0x144a4f8, 0x12e5e
fd, 0x1a, 0xc42000c069, 0xc, 0xc42001400f, 0x40, 0x12dfcb6, ...)
        /Users/adriaanthuis/Documents/projects/contentful-hugo/extract/extractor.go:66 +0x403
_/Users/adriaanthuis/Documents/projects/contentful-hugo/extract.(*Extractor).ProcessAll(0xc42012ff08)
        /Users/adriaanthuis/Documents/projects/contentful-hugo/extract/extractor.go:43 +0x217
main.main()
        /Users/adriaanthuis/Documents/projects/contentful-hugo/main.go:26 +0x1a8

Certain site specific behavior is hard-coded.

E.g.:

• Different naming for items of type Homepage
• Content directory
• Fields like slug / mainContent and their behavior

Introduce a TOML based configuration for these.

Consider:

• URL creation
  • While reading
  • While writing
• TOML serialization (+++)

Make choice between TOML and YAML configurable

Currently, filenames are based on Content ID. Hugo can work with that. But it is not helpful for human readability. Especially for users who consider moving away from Contentful at some point.

Translating to readable names can be tricky for sites with more items than fit on a single page (>1000 items). An item may refer to another item that has not yet been loaded and therefore is more expensive to resolve.

Possible strategies:

• Request items not in current page
  • Slow and lots of network traffic
• Load all items in memory before processing
  • Memory-intensive
• Keep list of items that have unresolved references and resolve them when a new page comes
  • Probably less memory intensive but depends on specific use cases. Also complex
• Two-pass strategy first writing all items to a work-dir as a half-fabricate while building up an ID-name translation table. Then in the second pass, rename all items and references in the content.
  • Probably the most reasonable tradeoff between speed and memory use. Some impact due to increased disk use and need to ensure clean-up of the work-dir. (PREFERRED)

Add page size to the Read Config

Give HTTP handler its own package and Make the timeout configurable

Currently, the tool only gets the first 200 items from the REST API. If there'd be more than 200 items, part of the content would be missing.

To solve this, make use of the pagination that the Contentful API supports.

The tool should be capable of handling 10,000 items that come with the free plan.

Change MockStore to capture outputs and use this to add checks. Should work at least as a regression test. Right now, a lot of code coverage only proves that the code runs without errors, not that it gives expected results.

On reference fields, it only lists the filename.md in the front matter.

Example...

lessons:
- 3op5VIqGZiwoe06c8IQIMO.md
- 5p9qNpTOJaCE6ykC4a8Wqg.md
- 1zwAjpe38UC8iqKMM6gu0Q.md
- 3KinTi83FecuMeiUo0qGU4.md

And that causes a problem if you are refererring to multiple content types. Would it be possible to add the folder of the content type along with the filename in the the reference field, like so???

lessons:
- programming/3op5VIqGZiwoe06c8IQIMO.md
- administration/5p9qNpTOJaCE6ykC4a8Wqg.md
- programming/1zwAjpe38UC8iqKMM6gu0Q.md
- accounting/3KinTi83FecuMeiUo0qGU4.md

As a guy who is very Golang illiterate, this looks like the place you are defining this frontmatter?
https://github.com/adriaandejonge/contentful-hugo/blob/7ab0c690f2a5a78beba76d6c2d798bbb0120c04a/translate/translate.go#L103

Could you point in the direction to adding the directory of the content types?