jkent / frogfs Goto Github PK
View Code? Open in Web Editor NEWFast Read-Only General-purpose File System (ESP32/ESP8266 compatible)
Home Page: https://frogfs.readthedocs.io
License: Mozilla Public License 2.0
Fast Read-Only General-purpose File System (ESP32/ESP8266 compatible)
Home Page: https://frogfs.readthedocs.io
License: Mozilla Public License 2.0
For quick development, I'd like to have the HTML
compressed to a spiffs partition once instead of uploading both inside of the factory app partition (it costs time to upload every new change to the main factory app). Is there a way to make use of the spiffsgen.py
tool to do this?
Lets find out how fast FrogFS is, and provide real-world test data to find out how well the filters shrink the data.
The implementation of frogfs_fseek()
(or espfs_fseek
in latest 3.0.0 branch) has a weakness with heatshrink compressed files when seeking backwards - at least when new position is not a multiple of 16. This could be made visible when performing either an fread()
afterwards (comparing the read data with the known file content at that position) or just calling ftell()
right after fseek()
.
I prepared a standalone test app to verify and analyze the issue (find attached below).
The example has a filesystem attached as byte array. It only consists of one heatshrink-compressed file named "file.bin". This file's raw content is included as byte array as well. When running the example it performs four seek and read operations and compares them against the expected result. Test 4 seeks backwards from file position left at test 3. At this point file position as well as read data do not match the expectation.
I TESTAPP: Test #1
I TESTAPP: Seek to offset 0x0000
I TESTAPP: Read 12 bytes: DE 12 04 95 00 00 00 00 31 00 00 00
I TESTAPP: Test #2
I TESTAPP: Seek to offset 0x000c
I TESTAPP: Read 8 bytes: 1C 00 00 00 A4 01 00 00
I TESTAPP: Test #3
I TESTAPP: Seek to offset 0x03d4
I TESTAPP: Read 4 bytes: 26 00 00 00
I TESTAPP: Test #4
I TESTAPP: Seek to offset 0x0384
E TESTAPP: SEEK MISMATCH! Sought to 0x0384 but ftell() reports 0x0390
I TESTAPP: Read 4 bytes: 18 00 00 00
E TESTAPP: CONTENT MISMATCH - offset 0x0384 expects 28 00 00 00
I TESTAPP: Done...
The root cause seems to be in frogfs_fseek()
method at the end of the "heatshrink" branch. There all data from file begin to some position is read in order to "pump" it through heatshrink decoder (I guess this is due to the fact that heatshrinked data blocks rely on each other). But here the data is read in fixed 16 byte blocks, hence the exact seek position is usually missed.
while (new_pos > f->file_pos) {
uint8_t buf[16];
frogfs_fread(f, buf, sizeof(buf));
}
Test application:
cmake -B build && cmake --build build
build/FrogFsTest
Hello.
The directory structure is like this:
/.test_dot_dir/.test.html
Result:
- Stage 1
- test_dot_dir/.test.html
Traceback (most recent call last):
File "E:\Espressif\workspace\test_proj\managed_components\frogfs\tools\mkfrogfs.py", line 511, in <module>
dirty |= run_preprocessors(entries)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Espressif\workspace\test_proj\managed_components\frogfs\tools\mkfrogfs.py", line 324, in run_preprocessors
preprocess(entry)
File "E:\Espressif\workspace\test_proj\managed_components\frogfs\tools\mkfrogfs.py", line 261, in preprocess
with open(os.path.join(g_root_dir, path), 'rb') as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '\\test_dot_dir/.test.html'
It looks like the dot is removed from directory name in mkfrogfs.py
As far as I understand, esp-idf support esp_partition_mmap
function, and espfs is mostly already a memory mapped filesystem.
Is is possible to link both to get a partition on flash containing RO data we could access via mmap (that is getting access to espfs_file_t->raw_start
and length
) ?
I see multiple advantages to this scheme:
This looks like a great library/project. However, I'm using VSCode and Platform.io and not cmake. Any idea on how I would go about integrating this?
I'm interested in accessing the espfs via vfs.
My project uses Espfs built-in to libesphttpd https://github.com/chmorgan/libesphttpd
but I want to access it using VFS from other modules in my project.
Can this module coexist with libesphttpd ?
Would it make sense to modify libesphttpd to use a external git module like this (instead of including the espfs code)?
@chmorgan thoughts?
I've got a couple of ideas kicking around for managing mime types in FrogFS.
There are pros and cons to both approaches. The former is fast, but requires modifying the binary format yet again. The later either requires memory caching of the mime types or slow string lookups, but it can be an optional feature.
Test with micropython
Hi. I'm trying to use target_add_frogfs
with CONFIG
parameter but it results in wrong invocation of mkfrogfs.py
as it is necessary to pass additional --config
cmd line argument that is missing in functions.cmake
file. As result the next error is generated in build output:
usage: mkfrogfs.py [-h] [--config CONFIG] ROOT OUTPUT
mkfrogfs.py: error: unrecognized arguments: E:/Espressif/workspace/httpd/build/frogfs.bin
A quick fix is to modify line 45 in functions.cmake to look like:
COMMAND ${CMAKE_COMMAND} -E env CMAKEFILES_DIR=${BUILD_DIR}/CMakeFiles ${Python3_VENV_EXECUTABLE} ${frogfs_DIR}/tools/mkfrogfs.py ${directories} --config ${ARG_CONFIG} ${path} ${output}.bin
(pay attention --config added here)
But then it will break compilation without CONFIG argument.
preprocess.py:49 gives error:
AttributeError: 'str' object has no attribute 'removeprefix'
Looks like this was added in python 3.9, but I have 3.8.
Since 3.8 is supported in IDF 5.1, I think it should be supported here too.
Please consider including new espFsAccess method:
https://esp32.com/viewtopic.php?t=698
I'll fork and PR if I get around to it.
Directories currently do not know if an object is a direct descendant or not. Should they get their own sort table? This might replace the sort table that follows the hash table.
Currently chosen hash function is colliding. In effect, any char X followed by Y so that Y = X+33 gives the same result as X+1 followed by X.
And since a path is very likely to have successive char with a distance of 33 between them, this is going to happen frequently.
For example, "2S" gives the same hash as "32"
Replacing:
static uint32_t hash_path(const char *path)
{
uint32_t hash = 5381;
const uint8_t *p = (uint8_t *)path;
while (*p) {
/* hash = hash * 33 + *p */
hash = (hash << 5) + hash + *p;
p++;
}
return hash;
}
by
static uint32_t hash_path(const char *path)
{
uint32_t hash = 5381;
const uint8_t *p = (uint8_t *)path;
while (*p) {
/* hash = hash * 257 + *p */
hash = (hash << 8) + hash + *p;
p++;
}
return hash;
}
will solve this issue since there is no char that's 257 higher than another.
At first thanks for your great project! Nice to see all its progress and continuously better integration as ESP component.
Quite simple issue here: We get a compile warning which can be easily avoided in
Line 216 in bf6173f
The target of name is dynamically allocated so not so constant after all. free(name) causes a warning because free wants (obviously) a non-constant input.
Several easy solutions:
const char *name = frogfs_get_name(entry);
--> char *name = frogfs_get_name(entry);
free(name)
--> free((char*) name)
free(name)
--> free((void*) name)
I can create a PR for this if it helps but I am not sure which is your favorite solution.
I'm still trying to track this one down, and in the meantime I'm opening this issue to document my findings.
The espfs
build will silently fail on Windows when compressing files using npx
:
[2/7] Building espfs binary CMakeFiles/blackmagic.elf.dir/espfs
'"npx uglifycss"' is not recognized as an internal or external command,
operable program or batch file.
0abb65ac xterm.css file 4.1 KiB -> 0.0 KiB (0.0%)
0f71a853 flash dir
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
0fd72300 xterm.js file 255.9 KiB -> 0.0 KiB (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
52372b2f debug.html file 1.6 KiB -> 0.0 KiB (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
66e3de48 index.js file 94 B -> 0 B (0.0%)
'"npx uglifycss"' is not recognized as an internal or external command,
operable program or batch file.
69d0be6a flash/style.css file 517 B -> 0 B (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
70aae05d flash/index.html file 1.8 KiB -> 0.0 KiB (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
ad1d1634 flash/140medley.min.js file 827 B -> 0 B (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
af538a40 index.html file 6.7 KiB -> 0.0 KiB (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
f4f97403 xterm-addon-fit.js file 2.3 KiB -> 0.0 KiB (0.0%)
[7/7] Generating binary image from built executable
esptool.py v3.2-dev
This is despite the fact that the npx
command exists and is capable of running commands:
[15:31:33] E:/Code/esp/blackmagic-espidf> npx uglifycss --help
Usage: uglifycss [options] file1.css [file2.css [...]] > output
options:
--max-line-len n add a newline every n characters
--expand-vars expand variables
--ugly-comments remove newlines within preserved comments
--cute-comments preserve newlines within and around preserved comments
--convert-urls d convert relative urls using the d directory as location target
--debug print full error stack on error
--output f put the result in f file
--help show this help
--version display version number
[15:31:47] E:/Code/esp/blackmagic-espidf>
The compilation does not fail when this occurs, and instead generates empty files.
Possible reasons for this:
npx html-minifier
, it's supposed to call "npx" "html-minifier"
idf.py
shell is replacing PATH
with one that is cleaned of any system PATH entriescmd.exe
and powershell
, however this is less likely because the error occurs as well with cmd.exe
without powershellThere are two issues here:
npx
on Windows even though it exists and is on the PATHxxd is now a dependency
For IDF-specific CMake projects, this part of frogfs usage:
include(components/frogfs/cmake/functions.cmake) # <<<
target_add_frogfs(my_project.elf html NAME frogfs CONFIG frogfs.yaml)
could be simplified by adding a project_include.cmake
file into the component root directory. IDF build system includes project_include.cmake
files, making the functions defined there available in component and project CMakeLists files.
Here are some examples:
spiffs_create_partition_image
function this way(Generic CMake projects can continue using include(...)
, as this is more idiomatic in CMake. For IDF-specific projects, though, making the function available immediately after adding the component to the build is more common.)
The new version of FrogFS appears to rely on venv in order to install just the right dependencies into the build system.
However, venv appears to be broken on the ESP version of Python, at least on Windows:
[11:29:05 pm] E:/Code/esp/blackmagic-esp32/components/frogfs> cmd.exe /C "cd /D E:\Code\esp\blackmagic-esp32\build && C:\Users\Sean\.espressif\python_env\idf4.4_py3.8_env\Scripts\python.exe -m venv ."
[11:29:05 pm] E:/Code/esp/blackmagic-esp32/components/frogfs> cmd.exe /C "cd /D E:\Code\esp\blackmagic-esp32\build && C:\Users\Sean\.espressif\python_env\idf4.4_py3.8_env\Scripts\python.exe --version"
Python 3.8.7
[11:30:37 pm] E:/Code/esp/blackmagic-esp32/components/frogfs>
Hi @jkent, I wonder if you could publish this component in the IDF component registry to make it easier to add to IDF projects?
To do this, you only need to:
The hiyapyco
package is currently broken (zerwes/hiyapyco#45)
This is because it relies on a build system that ultimately depends on markupsafe (zerwes/hiyapyco#45) that made a change in a patch (pallets/markupsafe#284)
A workaround is to modify requirements.txt
:
diff --git a/requirements.txt b/requirements.txt
index b21dbcf..cfcac7e 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +1,3 @@
heatshrink2>=0.6.0
hiyapyco>=0.4.16
+MarkupSafe==2.0.1
\ No newline at end of file
The espfs binary should optionally be flashable to a partition, which means linking the image should be optional. The espfs binary should also have a header field that contains the binary size. See #8
For some reason, heatshrink does not work with the VFS code. It works OK with espfs code.
The python cache is generated in the frogfs directory thus triggering an IDF error after a reconfigure when using the component.
ERROR: Some components (jkent/frogfs) in the "managed_components" directory
were modified on the disk since the last run of the CMake. Content of this
directory is managed automatically.
To reproduce:
This triggers the error mentioned above.
Compile error on ESP32 IDF Eclipse plugin. This may be an error on my part, I'm still learning about this plugin and CMake in particular. I added your frogfs component as a managed component. I then moved it to just a normal component. Compilation fails for frogfs.c because PATH_MAX is undefined. I guess this is defined in Linux headers but not in ESP32 IDF ones. I can , of course, hack in a value - but what it the correct solution? Am excited to get this working, it's exactly what I need.
I use espfs as the base read-only filesystem which holds factory-default files for my project. Then I use a FAT fs at run-time to store any changes. When a file is accessed, I have logic to first check FAT to see if the file exists there, then if not try to read it from espfs.
So it is essentially an OverlayFS.
This enables me to perform a factory-reset (clear all user settings) by just formatting the FAT partition and rebooting.
To avoid having to duplicate this logic (httpd and other tasks), I'm thinking it might be appropriate to add it to the espfs_vfs layer in this module.
So you would initialize espfs_vfs with an optional parameter that specifies the upper/overlay filesystem path, like this:
// Mount trivial read-only filesystem: espfs
esp_vfs_espfs_conf_t espfs_vfs_conf = {
.base_path = "/espfs",
.max_files = 5,
.overlay_path = "/fatfs"
};
esp_vfs_espfs_register(&espfs_vfs_conf);
vfs_espfs_open() will check if overlay_path is not NULL, then somehow forward the open call to the upper/overlay filesystem. If the file is not found in upper, then handle as usual via espfs.
I'm sure I haven't thought this completely through... Perhaps it should be a separate OverlayFS module that specifies both "upper" and "lower" fs? Maybe it already exists?
Thoughts?
I suggest adding a branch for maintenance of version 3.
e85510e
seems like a good spot, since that is before the latest breaking changes.
I noticed my http server (https://github.com/chmorgan/libesphttpd) sending extra junk data after the contents of a gzip file. In fact, it sends as much junk as to fill the uncompressed size of the file it is sending.
I.e. a gzip file which uncompressed is 44KB. The server sends 44KB of the gzip file and then whatever bytes follow in the espfs image. Modern browsers seem to ignore this extra data, but chromium 53 does not. And it defeats the purpose of compression if we are padding the gzip file with junk so that the effective compression is 1:1.
I dug into it and found that the server just relies on the file size provided by espfs.
https://github.com/chmorgan/libesphttpd/blob/master/core/httpd-espfs.c#L202
I added some debug crumbs and espfs_stat also reports the incorrect (as if uncompressed) file size.
I.e. for the file /static/app-25efe0c865.min.css which is 44280 bytes uncompressed. When compressed it should be less than half of that.
espfs_stat_t s = {0};
espfs_fstat(file, &s);
ESP_LOGD(TAG, "espfs_fstat size: %d flags: %x comp: %x", s.size, s.flags, s.compression);
D (127935) httpdespfs: espfs_fstat size: 44280 flags: 3 comp: 0
I'm guessing the bug is probably in the generating of the espfs image, which writes the file size to the header. I'll keep digging...
Would be nice to have a get file size method in espfs
Also fill-out the VFS layer
so that this works:
size_t getFilesize(const char* filename) {
struct stat st;
stat(filename, &st);
return st.st_size;
}
Anyway, It's a low priority feature that I'm going to work-around for now.
This method works:
fseek(f, 0, SEEK_END); // seek to end of file
size = ftell(f); // get current file pointer
fseek(f, 0, SEEK_SET); // seek back to beginning of file
// proceed with allocating memory and reading the file
I need to know the espfs image size, so I can calculate MD5 on it and check if updating it over OTA was successful.
Is there a way to get the espfs image size other than traversing through the included files?
Cheers
Win 7. Legacy GNU Make (msys2/mingw32). ESP-IDF 3.3.1.
I unselected menuconfig->components->espfs->preprocess files.
Leads to:
#
# espfs
#
CONFIG_ESPFS_MAX_PARTITIONS=3
CONFIG_ESPFS_IMAGEROOTDIR="html"
CONFIG_ESPFS_PREPROCESS_FILES=
CONFIG_ESPFS_USE_HEATSHRINK=y
CONFIG_ESPFS_USE_GZIP=y
CONFIG_ESPFS_LINK_BINARY=y
get build error:
make[1]: *** No rule to make target 'node_modules', needed by 'espfs_image.bin'. Stop.
Workaround with:
#
# espfs
#
CONFIG_ESPFS_MAX_PARTITIONS=3
CONFIG_ESPFS_IMAGEROOTDIR="html"
CONFIG_ESPFS_PREPROCESS_FILES=y
CONFIG_ESPFS_CSS_MINIFY_NONE=y
CONFIG_ESPFS_CSS_MINIFY_UGLIFYCSS=
CONFIG_ESPFS_HTML_MINIFY_NONE=y
CONFIG_ESPFS_HTML_MINIFY_HTMLMINIFIER=
CONFIG_ESPFS_JS_CONVERT_NONE=y
CONFIG_ESPFS_JS_CONVERT_BABEL=
CONFIG_ESPFS_JS_MINIFY_NONE=y
CONFIG_ESPFS_JS_MINIFY_BABEL=
CONFIG_ESPFS_JS_MINIFY_UGLIFYJS=
CONFIG_ESPFS_USE_HEATSHRINK=y
CONFIG_ESPFS_USE_GZIP=y
CONFIG_ESPFS_LINK_BINARY=y
Builds OK (CONFIG_ESPFS_PREPROCESS_FILES is y, but all sub-options off)
Currently it is not possible to match just a directory and apply filters on it. You have to glob it like the following to apply to all descendants:
filters:
'path/to/folder_name*':
- discard
You should be able to just do the following:
filters:
'path/to/folder_name':
- discard
...but this is currently broken.
Implement --id switch for compressors so the compressor ID can be looked up by name without modifying mkfrogfs.py code.
This error is being generated on build with CMake (ESP-IDF Tools 2.1). It work fine thou when using legacy make with exact same sources.
[1059/1109] Performing configure step for 'mkespfsimage'
-- Configuring done
-- Generating done
-- Build files have been written to: D:/Arduino/Builds/concert_bt_cmake/esp-idf/esp32-espfs/mkespfsimage-prefix/src/mkespfsimage-build
[1062/1109] Performing build step for 'mkespfsimage'
[1/2] Building C object CMakeFiles/mkespfsimage.dir/main.c.obj
[2/2] Linking C executable mkespfsimage.exe
[1089/1109] Performing install step for 'mkespfsimage'
[1/3] Building C object CMakeFiles/mkespfsimage.dir/main.c.obj
[2/3] Linking C executable mkespfsimage.exe
[2/3] Install the project...
-- Install configuration: ""
-- Installing: D:/Arduino/Builds/concert_bt_cmake/esp-idf/esp32-espfs/bin/mkespfsimage.exe
[1098/1109] Generating espfs_image.bin
FAILED: esp-idf/esp32-espfs/espfs_image.bin
cmd.exe /C "cd /D D:\Arduino\Builds\concert_bt_cmake\esp-idf\esp32-espfs && D:\Espressif\.espressif\tools\cmake\3.13.4\bin\cmake.exe -E env BUILD_DIR=D:/Arduino/Builds/concert_bt_cmake/esp-idf/esp32-espfs CONFIG_ESPFS_PREPROCESS_FILES= CONFIG_ESPFS_CSS_MINIFY_UGLIFYCSS=n CONFIG_ESPFS_HTML_MINIFY_HTMLMINIFIER=n CONFIG_ESPFS_JS_CONVERT_BABEL=n CONFIG_ESPFS_JS_MINIFY_BABEL=n CONFIG_ESPFS_JS_MINIFY_UGLIFYJS=n CONFIG_ESPFS_UGLIFYCSS_PATH= CONFIG_ESPFS_HTMLMINIFIER_PATH= CONFIG_ESPFS_BABEL_PATH= CONFIG_ESPFS_UGLIFYJS_PATH= D:/Dropbox/Arduino/Sketches/Projects/ESP32/A2DP/a2dp_sink/components/esp32-espfs/tools/build-image.py D:/Dropbox/Arduino/Sketches/Projects/ESP32/A2DP/a2dp_sink/html"
%1 is not a valid Win32 application
[1099/1109] Linking CXX static library esp-idf\esp_https_ota\libesp_https_ota.a
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.