I'm interested in accessing the espfs via vfs.

My project uses Espfs built-in to libesphttpd https://github.com/chmorgan/libesphttpd
but I want to access it using VFS from other modules in my project.

Can this module coexist with libesphttpd ?
Would it make sense to modify libesphttpd to use a external git module like this (instead of including the espfs code)?
@chmorgan thoughts?

This error is being generated on build with CMake (ESP-IDF Tools 2.1). It work fine thou when using legacy make with exact same sources.

[1059/1109] Performing configure step for 'mkespfsimage'
-- Configuring done
-- Generating done
-- Build files have been written to: D:/Arduino/Builds/concert_bt_cmake/esp-idf/esp32-espfs/mkespfsimage-prefix/src/mkespfsimage-build
[1062/1109] Performing build step for 'mkespfsimage'
[1/2] Building C object CMakeFiles/mkespfsimage.dir/main.c.obj
[2/2] Linking C executable mkespfsimage.exe
[1089/1109] Performing install step for 'mkespfsimage'
[1/3] Building C object CMakeFiles/mkespfsimage.dir/main.c.obj
[2/3] Linking C executable mkespfsimage.exe
[2/3] Install the project...
-- Install configuration: ""
-- Installing: D:/Arduino/Builds/concert_bt_cmake/esp-idf/esp32-espfs/bin/mkespfsimage.exe
[1098/1109] Generating espfs_image.bin
FAILED: esp-idf/esp32-espfs/espfs_image.bin
cmd.exe /C "cd /D D:\Arduino\Builds\concert_bt_cmake\esp-idf\esp32-espfs && D:\Espressif\.espressif\tools\cmake\3.13.4\bin\cmake.exe -E env BUILD_DIR=D:/Arduino/Builds/concert_bt_cmake/esp-idf/esp32-espfs CONFIG_ESPFS_PREPROCESS_FILES= CONFIG_ESPFS_CSS_MINIFY_UGLIFYCSS=n CONFIG_ESPFS_HTML_MINIFY_HTMLMINIFIER=n CONFIG_ESPFS_JS_CONVERT_BABEL=n CONFIG_ESPFS_JS_MINIFY_BABEL=n CONFIG_ESPFS_JS_MINIFY_UGLIFYJS=n CONFIG_ESPFS_UGLIFYCSS_PATH= CONFIG_ESPFS_HTMLMINIFIER_PATH= CONFIG_ESPFS_BABEL_PATH= CONFIG_ESPFS_UGLIFYJS_PATH= D:/Dropbox/Arduino/Sketches/Projects/ESP32/A2DP/a2dp_sink/components/esp32-espfs/tools/build-image.py D:/Dropbox/Arduino/Sketches/Projects/ESP32/A2DP/a2dp_sink/html"
%1 is not a valid Win32 application
[1099/1109] Linking CXX static library esp-idf\esp_https_ota\libesp_https_ota.a
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

I noticed my http server (https://github.com/chmorgan/libesphttpd) sending extra junk data after the contents of a gzip file. In fact, it sends as much junk as to fill the uncompressed size of the file it is sending.

I.e. a gzip file which uncompressed is 44KB. The server sends 44KB of the gzip file and then whatever bytes follow in the espfs image. Modern browsers seem to ignore this extra data, but chromium 53 does not. And it defeats the purpose of compression if we are padding the gzip file with junk so that the effective compression is 1:1.

I dug into it and found that the server just relies on the file size provided by espfs.
https://github.com/chmorgan/libesphttpd/blob/master/core/httpd-espfs.c#L202

I added some debug crumbs and espfs_stat also reports the incorrect (as if uncompressed) file size.
I.e. for the file /static/app-25efe0c865.min.css which is 44280 bytes uncompressed. When compressed it should be less than half of that.

espfs_stat_t s = {0};
espfs_fstat(file, &s);
ESP_LOGD(TAG, "espfs_fstat size: %d flags: %x comp: %x", s.size, s.flags, s.compression);

D (127935) httpdespfs: espfs_fstat size: 44280 flags: 3 comp: 0

I'm guessing the bug is probably in the generating of the espfs image, which writes the file size to the header. I'll keep digging...

The hiyapyco package is currently broken (zerwes/hiyapyco#45)

This is because it relies on a build system that ultimately depends on markupsafe (zerwes/hiyapyco#45) that made a change in a patch (pallets/markupsafe#284)

A workaround is to modify requirements.txt:

diff --git a/requirements.txt b/requirements.txt
index b21dbcf..cfcac7e 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +1,3 @@
 heatshrink2>=0.6.0
 hiyapyco>=0.4.16
+MarkupSafe==2.0.1
\ No newline at end of file

Implement --id switch for compressors so the compressor ID can be looked up by name without modifying mkfrogfs.py code.

Currently chosen hash function is colliding. In effect, any char X followed by Y so that Y = X+33 gives the same result as X+1 followed by X.
And since a path is very likely to have successive char with a distance of 33 between them, this is going to happen frequently.
For example, "2S" gives the same hash as "32"

Replacing:

static uint32_t hash_path(const char *path)
{
    uint32_t hash = 5381;
    const uint8_t *p = (uint8_t *)path;

    while (*p) {
        /* hash = hash * 33 + *p */
        hash = (hash << 5) + hash + *p;
        p++;
    }

    return hash;
}

by

static uint32_t hash_path(const char *path)
{
    uint32_t hash = 5381;
    const uint8_t *p = (uint8_t *)path;

    while (*p) {
        /* hash = hash * 257 + *p */
        hash = (hash << 8) + hash + *p;
        p++;
    }

    return hash;
}

will solve this issue since there is no char that's 257 higher than another.

Win 7. Legacy GNU Make (msys2/mingw32). ESP-IDF 3.3.1.

I unselected menuconfig->components->espfs->preprocess files.
Leads to:

#
# espfs
#
CONFIG_ESPFS_MAX_PARTITIONS=3
CONFIG_ESPFS_IMAGEROOTDIR="html"
CONFIG_ESPFS_PREPROCESS_FILES=
CONFIG_ESPFS_USE_HEATSHRINK=y
CONFIG_ESPFS_USE_GZIP=y
CONFIG_ESPFS_LINK_BINARY=y

get build error:

make[1]: *** No rule to make target 'node_modules', needed by 'espfs_image.bin'.  Stop.

Workaround with:

#
# espfs
#
CONFIG_ESPFS_MAX_PARTITIONS=3
CONFIG_ESPFS_IMAGEROOTDIR="html"
CONFIG_ESPFS_PREPROCESS_FILES=y
CONFIG_ESPFS_CSS_MINIFY_NONE=y
CONFIG_ESPFS_CSS_MINIFY_UGLIFYCSS=
CONFIG_ESPFS_HTML_MINIFY_NONE=y
CONFIG_ESPFS_HTML_MINIFY_HTMLMINIFIER=
CONFIG_ESPFS_JS_CONVERT_NONE=y
CONFIG_ESPFS_JS_CONVERT_BABEL=
CONFIG_ESPFS_JS_MINIFY_NONE=y
CONFIG_ESPFS_JS_MINIFY_BABEL=
CONFIG_ESPFS_JS_MINIFY_UGLIFYJS=
CONFIG_ESPFS_USE_HEATSHRINK=y
CONFIG_ESPFS_USE_GZIP=y
CONFIG_ESPFS_LINK_BINARY=y

Builds OK (CONFIG_ESPFS_PREPROCESS_FILES is y, but all sub-options off)

This looks like a great library/project. However, I'm using VSCode and Platform.io and not cmake. Any idea on how I would go about integrating this?

xxd is now a dependency

Hi. I'm trying to use target_add_frogfs with CONFIG parameter but it results in wrong invocation of mkfrogfs.py as it is necessary to pass additional --config cmd line argument that is missing in functions.cmake file. As result the next error is generated in build output:

usage: mkfrogfs.py [-h] [--config CONFIG] ROOT OUTPUT
mkfrogfs.py: error: unrecognized arguments: E:/Espressif/workspace/httpd/build/frogfs.bin

A quick fix is to modify line 45 in functions.cmake to look like:

COMMAND ${CMAKE_COMMAND} -E env CMAKEFILES_DIR=${BUILD_DIR}/CMakeFiles ${Python3_VENV_EXECUTABLE} ${frogfs_DIR}/tools/mkfrogfs.py ${directories} --config ${ARG_CONFIG} ${path} ${output}.bin

(pay attention --config added here)

But then it will break compilation without CONFIG argument.

I use espfs as the base read-only filesystem which holds factory-default files for my project. Then I use a FAT fs at run-time to store any changes. When a file is accessed, I have logic to first check FAT to see if the file exists there, then if not try to read it from espfs.
So it is essentially an OverlayFS.
This enables me to perform a factory-reset (clear all user settings) by just formatting the FAT partition and rebooting.

To avoid having to duplicate this logic (httpd and other tasks), I'm thinking it might be appropriate to add it to the espfs_vfs layer in this module.
So you would initialize espfs_vfs with an optional parameter that specifies the upper/overlay filesystem path, like this:

// Mount trivial read-only filesystem: espfs
esp_vfs_espfs_conf_t espfs_vfs_conf = {
  .base_path = "/espfs",
  .max_files = 5,
  .overlay_path = "/fatfs"
};
esp_vfs_espfs_register(&espfs_vfs_conf);

vfs_espfs_open() will check if overlay_path is not NULL, then somehow forward the open call to the upper/overlay filesystem. If the file is not found in upper, then handle as usual via espfs.

I'm sure I haven't thought this completely through... Perhaps it should be a separate OverlayFS module that specifies both "upper" and "lower" fs? Maybe it already exists?
Thoughts?

The espfs binary should optionally be flashable to a partition, which means linking the image should be optional. The espfs binary should also have a header field that contains the binary size. See #8

Lets find out how fast FrogFS is, and provide real-world test data to find out how well the filters shrink the data.

preprocess.py:49 gives error:
AttributeError: 'str' object has no attribute 'removeprefix'
Looks like this was added in python 3.9, but I have 3.8.

Since 3.8 is supported in IDF 5.1, I think it should be supported here too.

I suggest adding a branch for maintenance of version 3.
e85510e
seems like a good spot, since that is before the latest breaking changes.

I've got a couple of ideas kicking around for managing mime types in FrogFS.

1. A lookup table of uint16_t to mime type strings, with uint16_t field added to files.
2. A file on the file system that maps file extensions to mime types

There are pros and cons to both approaches. The former is fast, but requires modifying the binary format yet again. The later either requires memory caching of the mime types or slow string lookups, but it can be an optional feature.

As far as I understand, esp-idf support esp_partition_mmap function, and espfs is mostly already a memory mapped filesystem.
Is is possible to link both to get a partition on flash containing RO data we could access via mmap (that is getting access to espfs_file_t->raw_start and length) ?

I see multiple advantages to this scheme:

1. No need to copy the data from ROM to RAM (since RAM is very limited on ESP32) like with the VFS layer. Once could use the raw pointer as if it was a C array
2. Data is update-able via OTA (provided the right code is written for writing to the right partition). Obviously, care must be taken not to read from the partition while it's updated.
3. All other advantages to use ESPFS.

I'm still trying to track this one down, and in the meantime I'm opening this issue to document my findings.

The espfs build will silently fail on Windows when compressing files using npx:

[2/7] Building espfs binary CMakeFiles/blackmagic.elf.dir/espfs
'"npx uglifycss"' is not recognized as an internal or external command,
operable program or batch file.
0abb65ac xterm.css                          file 4.1 KiB   -> 0.0 KiB   (0.0%)
0f71a853 flash                              dir
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
0fd72300 xterm.js                           file 255.9 KiB -> 0.0 KiB   (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
52372b2f debug.html                         file 1.6 KiB   -> 0.0 KiB   (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
66e3de48 index.js                           file 94 B      -> 0 B       (0.0%)
'"npx uglifycss"' is not recognized as an internal or external command,
operable program or batch file.
69d0be6a flash/style.css                    file 517 B     -> 0 B       (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
70aae05d flash/index.html                   file 1.8 KiB   -> 0.0 KiB   (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
ad1d1634 flash/140medley.min.js             file 827 B     -> 0 B       (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
af538a40 index.html                         file 6.7 KiB   -> 0.0 KiB   (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
f4f97403 xterm-addon-fit.js                 file 2.3 KiB   -> 0.0 KiB   (0.0%)
[7/7] Generating binary image from built executable
esptool.py v3.2-dev

This is despite the fact that the npx command exists and is capable of running commands:

[15:31:33] E:/Code/esp/blackmagic-espidf> npx uglifycss --help
Usage: uglifycss [options] file1.css [file2.css [...]] > output
options:
--max-line-len n  add a newline every n characters
--expand-vars     expand variables
--ugly-comments   remove newlines within preserved comments
--cute-comments   preserve newlines within and around preserved comments
--convert-urls d  convert relative urls using the d directory as location target
--debug           print full error stack on error
--output f        put the result in f file
--help            show this help
--version         display version number
[15:31:47] E:/Code/esp/blackmagic-espidf>

The compilation does not fail when this occurs, and instead generates empty files.

Possible reasons for this:

• The command is getting quoted in its entirety, generating an invalid command -- indeed, there is no program named npx html-minifier, it's supposed to call "npx" "html-minifier"
• The idf.py shell is replacing PATH with one that is cleaned of any system PATH entries
• Something is generating paths inside the JS code that is using backslashes that are getting misinterpreted later on down the line
• It's bouncing back and forth between cmd.exe and powershell, however this is less likely because the error occurs as well with cmd.exe without powershell

There are two issues here:

1. A failure in the generation process is nonfatal
2. The build system is unable to find npx on Windows even though it exists and is on the PATH

Discussion for a future version

Directories currently do not know if an object is a direct descendant or not. Should they get their own sort table? This might replace the sort table that follows the hash table.

Compile error on ESP32 IDF Eclipse plugin. This may be an error on my part, I'm still learning about this plugin and CMake in particular. I added your frogfs component as a managed component. I then moved it to just a normal component. Compilation fails for frogfs.c because PATH_MAX is undefined. I guess this is defined in Linux headers but not in ESP32 IDF ones. I can , of course, hack in a value - but what it the correct solution? Am excited to get this working, it's exactly what I need.

Would be nice to have a get file size method in espfs

Also fill-out the VFS layer
so that this works:

size_t getFilesize(const char* filename) {
	struct stat st;
	stat(filename, &st);
	return st.st_size;
}

Anyway, It's a low priority feature that I'm going to work-around for now.
This method works:

fseek(f, 0, SEEK_END); // seek to end of file
size = ftell(f); // get current file pointer
fseek(f, 0, SEEK_SET); // seek back to beginning of file
// proceed with allocating memory and reading the file

For some reason, heatshrink does not work with the VFS code. It works OK with espfs code.

The implementation of frogfs_fseek() (or espfs_fseek in latest 3.0.0 branch) has a weakness with heatshrink compressed files when seeking backwards - at least when new position is not a multiple of 16. This could be made visible when performing either an fread() afterwards (comparing the read data with the known file content at that position) or just calling ftell() right after fseek().
I prepared a standalone test app to verify and analyze the issue (find attached below).

The example has a filesystem attached as byte array. It only consists of one heatshrink-compressed file named "file.bin". This file's raw content is included as byte array as well. When running the example it performs four seek and read operations and compares them against the expected result. Test 4 seeks backwards from file position left at test 3. At this point file position as well as read data do not match the expectation.

I TESTAPP: Test #1
I TESTAPP: Seek to offset 0x0000
I TESTAPP: Read 12 bytes:  DE 12 04 95 00 00 00 00 31 00 00 00
I TESTAPP: Test #2
I TESTAPP: Seek to offset 0x000c
I TESTAPP: Read 8 bytes:  1C 00 00 00 A4 01 00 00
I TESTAPP: Test #3
I TESTAPP: Seek to offset 0x03d4
I TESTAPP: Read 4 bytes:  26 00 00 00
I TESTAPP: Test #4
I TESTAPP: Seek to offset 0x0384
E TESTAPP: SEEK MISMATCH! Sought to 0x0384 but ftell() reports 0x0390
I TESTAPP: Read 4 bytes:  18 00 00 00
E TESTAPP: CONTENT MISMATCH - offset 0x0384 expects  28 00 00 00 
I TESTAPP: Done...

The root cause seems to be in frogfs_fseek() method at the end of the "heatshrink" branch. There all data from file begin to some position is read in order to "pump" it through heatshrink decoder (I guess this is due to the fact that heatshrinked data blocks rely on each other). But here the data is read in fixed 16 byte blocks, hence the exact seek position is usually missed.

        while (new_pos > f->file_pos) {
            uint8_t buf[16];
            frogfs_fread(f, buf, sizeof(buf));
        }

Test application:

• FrogFsTest.zip
• Extract, clone FrogFS into "frogfs" sub-directory
• call cmake -B build && cmake --build build
• run build/FrogFsTest

The new version of FrogFS appears to rely on venv in order to install just the right dependencies into the build system.

However, venv appears to be broken on the ESP version of Python, at least on Windows:

[11:29:05 pm] E:/Code/esp/blackmagic-esp32/components/frogfs> cmd.exe /C "cd /D E:\Code\esp\blackmagic-esp32\build && C:\Users\Sean\.espressif\python_env\idf4.4_py3.8_env\Scripts\python.exe -m venv ."
[11:29:05 pm] E:/Code/esp/blackmagic-esp32/components/frogfs> cmd.exe /C "cd /D E:\Code\esp\blackmagic-esp32\build && C:\Users\Sean\.espressif\python_env\idf4.4_py3.8_env\Scripts\python.exe --version"
Python 3.8.7
[11:30:37 pm] E:/Code/esp/blackmagic-esp32/components/frogfs>

• CMakeLists.txt for mkespfsimage
• Integrate mkespfsimage into component
• Write python script to run minifiers/babel against files
• Integrate python script into component
• Generate espfs image
• Link object to resulting binary

For IDF-specific CMake projects, this part of frogfs usage:

include(components/frogfs/cmake/functions.cmake)   # <<<
target_add_frogfs(my_project.elf html NAME frogfs CONFIG frogfs.yaml)

could be simplified by adding a project_include.cmake file into the component root directory. IDF build system includes project_include.cmake files, making the functions defined there available in component and project CMakeLists files.

Here are some examples:

• spiffs adds its spiffs_create_partition_image function this way
• and it is used in the example later

(Generic CMake projects can continue using include(...), as this is more idiomatic in CMake. For IDF-specific projects, though, making the function available immediately after adding the component to the build is more common.)

For quick development, I'd like to have the HTML compressed to a spiffs partition once instead of uploading both inside of the factory app partition (it costs time to upload every new change to the main factory app). Is there a way to make use of the spiffsgen.py tool to do this?

Hi @jkent, I wonder if you could publish this component in the IDF component registry to make it easier to add to IDF projects?

To do this, you only need to:

• add an idf_component.yml file (example 1, example 2, docs)
• log into components.espressif.com with your Github account and create a token
• add a Github Actions workflow to publish the component for every new tag (example)

(I have a talk & slides on this topic, in case it helps.)

Hello.
The directory structure is like this:
/.test_dot_dir/.test.html

Result:
       - Stage 1
         - test_dot_dir/.test.html
Traceback (most recent call last):
  File "E:\Espressif\workspace\test_proj\managed_components\frogfs\tools\mkfrogfs.py", line 511, in <module>
    dirty |= run_preprocessors(entries)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\Espressif\workspace\test_proj\managed_components\frogfs\tools\mkfrogfs.py", line 324, in run_preprocessors
    preprocess(entry)
  File "E:\Espressif\workspace\test_proj\managed_components\frogfs\tools\mkfrogfs.py", line 261, in preprocess
    with open(os.path.join(g_root_dir, path), 'rb') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '\\test_dot_dir/.test.html'

It looks like the dot is removed from directory name in mkfrogfs.py

Currently it is not possible to match just a directory and apply filters on it. You have to glob it like the following to apply to all descendants:

filters:
  'path/to/folder_name*':
    - discard

You should be able to just do the following:

filters:
  'path/to/folder_name':
    - discard

...but this is currently broken.

Please consider including new espFsAccess method:

https://esp32.com/viewtopic.php?t=698

I'll fork and PR if I get around to it.

Test with micropython

I need to know the espfs image size, so I can calculate MD5 on it and check if updating it over OTA was successful.
Is there a way to get the espfs image size other than traversing through the included files?

Cheers