Tool for manipulating partition tables in MicroPython esp32 firmware image files and device flash storage.
mp-image-tool-esp32
manipulates micropython esp32 firmware files and flash
storage on serial-attached ESP32 devices. It has been tested to work with ESP32,
ESP32-S2 and ESP32-S3 firmware images and devices.
Contents: Features | Installation | Examples | OTA Updates | Usage
$ mp-image-tool-esp32 ESP32_GENERIC-20231005-v1.21.0.bin
Opening image file: ESP32_GENERIC-20231005-v1.21.0.bin...
Chip type: esp32
Flash size: 4MB
Micropython App size: 0x186bb0 bytes (1,562 KB)
Partition table (flash size: 4MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x6000 0xf000 0x0 (24.0 kB)
phy_init data phy 0xf000 0x1000 0x10000 0x0 (4.0 kB)
factory app factory 0x10000 0x1f0000 0x200000 0x0 (1.9 MB)
vfs data fat 0x200000 0x200000 0x400000 0x0 (2.0 MB)
Micropython app fills 78.8% of factory partition (421 kB free)
mp-image-tool-esp32
can operate on:
- micropython esp32 firmware files (including those downloaded from micropython.org) or
- flash storage in esp32 devices attached via serial port.
- Print a summary of the partition table of a micropython esp32 firmware file or
device
mp-image-tool-esp32 ESP32_GENERIC-20231005-v1.21.0.bin
- Change the size of the flash storage for the firmware file:
--resize-flash 8M
or-f 8M
- Rewrite or modify the partition table:
--table ota
: install an OTA-enabled partition table--table default
: install a default micropython table (non-OTA, without 'phy_init')--table original
: install the original default micropython table (non-OTA, with 'phy_init')--table nvs=6B,phy_init=1B,factory=0x1f0B,vfs=0
: specify a table layout--resize factory=2M,vfs=0x400K
: resize any partition (adjust other parts to fit)--delete phy_init --resize nvs=0
: delete 'phy_init' and expand 'nvs' to use free space--add vfs2=fat:2M:1M
: add a new FS data partition at offset 0x200000 with size 0x100000--app-size 0x200000
: change the size of theapp
partitions (use this with--table ota
)
- Extract the micropython application image (
.app-bin
) from the firmware or device--extract-app
: the extracted.app-bin
file can be used for OTA firmware updates.
- Modify the contents of partitions in the firmware:
--read factory=micropython.app-bin,nvs=nvs.bin
: read contents of partitions into files--write factory=micropython.app-bin
: write contents of files into partitions--write bootloader=bootloader.bin
: load a new bootloader from file--erase nvs,otadata
: erase partitions
When operating on micropython firmware files, mp-image-tool-esp32
will create
a copy of the firmware file with the partition table and partition contents
modified according to the options provided. The original firmware file is not
modified in any way. If no modification options are given, it will print the
partition table of filename
.
Use mp-image-tool-esp32 u0
to operate on the esp32 device attached to
/dev/ttyUSB0
. When operating on esp32 devices over a serial interface, the
following additional commands are available:
-
Erase 'vfs' filesystem partitions:
--erase-fs vfs
: erases the first 4 blocks of the partition- micropython will automatically build a fresh filesystem on the next boot
-
Use the
OTA
mechanism to perform a micropython firmware update over the serial interface to the device:--ota-update micropython.app-bin
-
Flash a firmware (including any changes) to an esp32 device, eg:
mp-image-tool-esp32 firmware.bin --flash u0
ormp-image-tool-esp32 firmware.bin -f 8M --table ota --flash /dev/ttyACM1
mp-image-tool-esp32
will automatically use the rightesptool
options to flash the firmware for your device (you don't need to remember if you should write to offset0x0
or0x1000
)
When operating on serial-attached esp32 devices, mp-image-tool-esp32
will
automatically erase any data
partitions (eg. nvs
, otadata
or vfs/fat
)
which have been moved or resized. Generally, micropython will re-initialise
these data
partitions on next boot. This prevents micropython attempting to
mount what appears to be a corrupt filesystem or nvs partition.
mp-image-tool-esp32
uses the
esptool.py
program to perform the
operations on attached esp32 devices:
- Select the specific method used to perform operations on the device:
-
--method direct/command/subprocess
subprocess
: Run the "esptool.py" command in a subprocess to interact with the device.command
: Run the esptool commands in this process using the esptool.main() function from the esptool module.direct
: (default): Use lower level functions from the esptool module to perform operations on the device. This is more efficient as it skips repeated initialisation and querying of the device.
Versions prior to 0.0.5, always used the
subprocess
method.
-
First, copy this github repo into a folder somewhere:
git clone https://github.com/glenn20/mp-image-tool-esp32
cd mp-image-tool-esp32
If you use a python virtual environment (recommended), make sure it is active.
To use without installing:
-
Prerequisites (
esptool
andcolorama
):pip install -r requirements.txt
-
Usage:
./mp-image-tool-esp32 ~/Downloads/ESP32_GENERIC-20231005-v1.21.0.bin
or, to install in your python environment:
-
Install in "editable mode":
pip install -e .
-
OR build and install a distributable
.whl
packagepython -m build pip install dist/mp_image_tool_esp32*.whl
$ mp-image-tool-esp32 ESP32_GENERIC-20231005-v1.21.0.bin -f 8M --resize vfs=0
Opening image file: ESP32_GENERIC-20231005-v1.21.0.bin...
Chip type: esp32
Flash size: 4MB
Micropython App size: 0x186bb0 bytes (1,562 KB)
Partition table (flash size: 4MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x6000 0xf000 0x0 (24.0 kB)
phy_init data phy 0xf000 0x1000 0x10000 0x0 (4.0 kB)
factory app factory 0x10000 0x1f0000 0x200000 0x0 (1.9 MB)
vfs data fat 0x200000 0x200000 0x400000 0x0 (2.0 MB)
Micropython app fills 78.8% of factory partition (421 kB free)
Resizing vfs partition to 0x600000 bytes.
Writing output file: ESP32_GENERIC-20231005-v1.21.0-8MB-vfs=0.bin...
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x6000 0xf000 0x0 (24.0 kB)
phy_init data phy 0xf000 0x1000 0x10000 0x0 (4.0 kB)
factory app factory 0x10000 0x1f0000 0x200000 0x0 (1.9 MB)
vfs data fat 0x200000 0x600000 0x800000 0x0 (6.0 MB)
Micropython app fills 78.8% of factory partition (421 kB free)
$ mp-image-tool-esp32 ESP32_GENERIC-20231005-v1.21.0-8MB.bin --table ota
Opening image file: ESP32_GENERIC-20231005-v1.21.0-8MB.bin...
Warning: End of last partition (0x400000) < flash size (0x800000).
Chip type: esp32
Flash size: 8MB
Micropython App size: 0x186bb0 bytes (1,562 KB)
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x6000 0xf000 0x0 (24.0 kB)
phy_init data phy 0xf000 0x1000 0x10000 0x0 (4.0 kB)
factory app factory 0x10000 0x1f0000 0x200000 0x0 (1.9 MB)
vfs data fat 0x200000 0x200000 0x400000 0x0 (2.0 MB)
Warning: End of last partition (0x400000) < flash size (0x800000).
Micropython app fills 78.8% of factory partition (421 kB free)
Writing output file: ESP32_GENERIC-20231005-v1.21.0-8MB-OTA.bin...
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x5000 0xe000 0x0 (20.0 kB)
otadata data ota 0xe000 0x2000 0x10000 0x0 (8.0 kB)
ota_0 app ota_0 0x10000 0x200000 0x210000 0x0 (2.0 MB)
ota_1 app ota_1 0x210000 0x200000 0x410000 0x0 (2.0 MB)
vfs data fat 0x410000 0x3f0000 0x800000 0x0 (3.9 MB)
Micropython app fills 76.3% of ota_0 partition (485 kB free)
- will automatically erase the first 4 blocks of any data partition which is
changed by the operation.
- micropython will automatically create a new filesystem on 'vfs' at next boot.
$ mp-image-tool-esp32 u0 -f 8M --resize vfs=0
Opening esp32 device at: /dev/ttyUSB0...
Warning: End of last partition (0x400000) < flash size (0x800000).
Chip type: esp32
Flash size: 8MB
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x6000 0xf000 0x0 (24.0 kB)
phy_init data phy 0xf000 0x1000 0x10000 0x0 (4.0 kB)
factory app factory 0x10000 0x1f0000 0x200000 0x0 (1.9 MB)
vfs data fat 0x200000 0x200000 0x400000 0x0 (2.0 MB)
Warning: End of last partition (0x400000) < flash size (0x800000).
Resizing vfs partition to 0x600000 bytes.
Writing new table to flash storage at /dev/ttyUSB0...
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x6000 0xf000 0x0 (24.0 kB)
phy_init data phy 0xf000 0x1000 0x10000 0x0 (4.0 kB)
factory app factory 0x10000 0x1f0000 0x200000 0x0 (1.9 MB)
vfs data fat 0x200000 0x600000 0x800000 0x0 (6.0 MB)
Setting flash_size in bootloader to 8.0MB...
Erasing data partition: vfs...
$ mp-image-tool-esp32 u0 -f 8M --table ota
Opening esp32 device at: /dev/ttyUSB0...
Warning: End of last partition (0x400000) < flash size (0x800000).
Chip type: esp32
Flash size: 8MB
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x6000 0xf000 0x0 (24.0 kB)
phy_init data phy 0xf000 0x1000 0x10000 0x0 (4.0 kB)
factory app factory 0x10000 0x1f0000 0x200000 0x0 (1.9 MB)
vfs data fat 0x200000 0x200000 0x400000 0x0 (2.0 MB)
Warning: End of last partition (0x400000) < flash size (0x800000).
Writing new table to flash storage at /dev/ttyUSB0...
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x5000 0xe000 0x0 (20.0 kB)
otadata data ota 0xe000 0x2000 0x10000 0x0 (8.0 kB)
ota_0 app ota_0 0x10000 0x200000 0x210000 0x0 (2.0 MB)
ota_1 app ota_1 0x210000 0x200000 0x410000 0x0 (2.0 MB)
vfs data fat 0x410000 0x3f0000 0x800000 0x0 (3.9 MB)
Setting flash_size in bootloader to 8.0MB...
Erasing data partition: nvs...
Erasing data partition: otadata...
Warning: app partition 'ota_1' does not contain app image.
Erasing data partition: vfs...
$ mp-image-tool-esp32 u0 --ota-update ESP32_GENERIC-20231005-v1.21.0.app-bin
Opening esp32 device: /dev/ttyUSB0...
Chip type: esp32
Flash size: 8MB
Partition table (flash size: 8MB):
# Name Type SubType Offset Size (End) Flags
nvs data nvs 0x9000 0x5000 0xe000 0x0 (20.0 kB)
otadata data ota 0xe000 0x2000 0x10000 0x0 (8.0 kB)
ota_0 app ota_0 0x10000 0x200000 0x210000 0x0 (2.0 MB)
ota_1 app ota_1 0x210000 0x200000 0x410000 0x0 (2.0 MB)
vfs data fat 0x410000 0x3f0000 0x800000 0x0 (3.9 MB)
Performing OTA firmware upgrade from 'ESP32_GENERIC-20231005-v1.21.0-8MB-OTA.app-bin'...
Writing firmware to OTA partition ota_1...
Updating otadata partition...
mp-image-tool-esp32
can be used to perform an OTA firmware update
over the serial interface to an OTA-enabled
esp32 device:
mp-image-tool-esp32 a1 --ota-update micropython.app-bin
ESP32 Over-The-Air updates
are intended to support firmware updates over wifi or bluetooth, but sometimes
it is convenient to push a firmware update to an OTA-enabled, USB-attached
device. This is the purpose of the --ota-update
option.
OTA updates must be performed using micropython app image firmware files. You can :
- download a
.app-bin
firmware file from micropython.org or - use
--extract-app
to extract a.app-bin
file from a micropython.bin
firmware file. For more options, see micropython app image.
OTA-enabled devices include those which:
- have been converted in place with
mp-image-tool-esp32 u0 --table ota
or - have been flashed with
OTA
enabled firmware files (see Firmware for OTA updates):- downloaded from micropython.org or
- produced by
mp-image-tool-esp32 filename --table ota
.
CAUTION: If you update the firmware with --ota-update
and OTA rollback
is enabled, the device will automatically reset into the new firmware on
completion. If you then connect to the device over the serial port, your IDE or
terminal software may reset the device again. If your startup files have not
marked the firmware as valid
, your device will have rolled back to the
previous firmware before you get to work with it.
You can stop the rollback by marking the new firmware as valid with:
esp32.Partition.mark_app_valid_cancel_rollback()
(docs) orota.rollback.cancel()
if you use the micropython-esp32-ota tool.- In practice, this means you should call
esp32.Partition.mark_app_valid_cancel_rollback()
after every restart.- Eg. in
main.py
-- or you might choose to wait till after your app has initialised wifi and any other devices you require so you know the new firmware is good for your app. - If it fails, just call
machine.hard_reset()
and the device will revert to the previous firmware on restart.
- Eg. in
Alternatively, you can use --no-rollback
to disable rollback for this update.
OTA rollback is enabled by default in the bootloader of:
- all micropython firmware files since
v1.21.0
and -OTA
micropython firmware files for earlier versions downloaded from micropython.org.
If you use --table ota
to convert a non-OTA, pre-v1.21.0
firmware, it will
NOT support OTA rollback. You can replace the bootloader with one from
an OTA-enabled firmware or device, but it is just easier to start with
rollback-enabled firmware.
See micropython-esp32-ota for an example of a tool which can perform conventional OTA firmware updates over the wifi interface.
NOTE: You can also update the app firmware of a non-OTA device with:
mp-image-tool-esp32 --write factory=ESP32_GENERIC-20231005-v1.21.0.app-bin
- This will upgrade the micropython firmware on the device without erasing
the
nvs
partition (which is usually placed before thefactory
partition on the device), as would be the case for writing a full micropython firmware to the device withesptool.py
.
usage: mp-image-tool-esp32 [-h] [-o FILE] [-q] [-d] [-n] [-x] [-f SIZE] [-a SIZE] [-m METHOD]
[--check-app] [--no-rollback] [--baud RATE] [--ota-update FILE]
[--from-csv FILE] [--table ota/default/NAME1=SUBTYPE:SIZE[,NAME2,...]]
[--delete NAME1[,NAME2]] [--add NAME1:SUBTYPE:OFFSET:SIZE[,NAME2,...]]
[--resize NAME1=SIZE1[,NAME2=SIZE2]] [--erase NAME1[,NAME2]]
[--erase-fs NAME1[,NAME2]]
[--read NAME1=FILE1[,NAME2=FILE2,bootloader=FILE,...]]
[--write NAME1=FILE1[,NAME2=FILE2,bootloader=FILE,...]]
[--flash DEVICE]
filename
Tool for manipulating MicroPython esp32 firmware files and flash storage on esp32 devices.
positional arguments:
filename the esp32 firmware filename or serial device
options:
-h, --help show this help message and exit
-o FILE, --output FILE
output firmware filename (auto-generated if not given)
-q, --quiet set debug level to WARNING (default: INFO)
-d, --debug set debug level to DEBUG (default: INFO)
-n, --no-reset leave device in bootloader mode afterward
-x, --extract-app extract .app-bin from firmware
-f SIZE, --flash-size SIZE
size of flash for new partition table
-a SIZE, --app-size SIZE
size of factory and ota app partitions
-m METHOD, --method METHOD
esptool method: subprocess, command or direct (default)
--check-app check app partitions and OTA config are valid
--no-rollback disable app rollback after OTA update
--baud RATE baud rate for serial port (default: 460800)
--ota-update FILE perform an OTA firmware upgrade over the serial port
--from-csv FILE load new partition table from CSV file
--table ota/default/original/NAME1=SUBTYPE:SIZE[,NAME2,...]
create new partition table, eg: "--table ota" (install an OTA-enabled
partition table), "--table default" (default (non-OTA) partition table),
"--table nvs=7B,factory=2M,vfs=0". SUBTYPE is optional in most cases
(inferred from name).
--delete NAME1[,NAME2]
delete the named partitions
--add NAME1:SUBTYPE:OFFSET:SIZE[,NAME2,...]
add new partitions to table
--resize NAME1=SIZE1[,NAME2=SIZE2]
resize partitions eg. --resize factory=2M,nvs=5B,vfs=0. If SIZE is 0,
expand partition to available space
--erase NAME1[,NAME2]
erase the named partitions
--erase-fs NAME1[,NAME2]
erase first 4 blocks of a partition on flash storage. Micropython will
initialise filesystem on next boot.
--read NAME1=FILE1[,NAME2=FILE2,bootloader=FILE,...]
copy partition contents (or bootloader) to file.
--write NAME1=FILE1[,NAME2=FILE2,bootloader=FILE,...]
write file(s) contents into partitions (or bootloader) in the firmware.
--flash DEVICE flash new firmware to the serial-attached device.
Where SIZE is a decimal or hex number with an optional suffix (M=megabytes, K=kilobytes, B=blocks
(0x1000=4096 bytes)). Options --erase-fs and --ota-update can only be used when operating on
serial-attached devices (not firmware files). If the --flash options is provided, the firmware
(including any changes made) will be flashed to the device, eg: `mp-image-tool-esp32 firmware.bin
--flash u0` is a convenient way to flash firmware to a device.