Comments (4)
kilobyte
commented on July 25, 2024
What does du say? Is this file sparse?
from compsize.
Zygo
commented on July 25, 2024
The method of creating the file wasn't included, but I'm guessing truncate -s 100G from the output given.
truncate -s 100G produces a file that occupies 0B of space and still allows compression on future writes, then mkfs.ext4 would add ~500MB of data to that. fallocate -l 100G would produce a file that occupies 100GB of space + space occupied by many later writes to the file, and prevents compression on future writes, but the file is smaller and compsize reports no prealloc extents. head -c 100G /dev/zero or similar would produce a file with minimum size ~3GB (100GB * 0.03 for the maximum compression ratio on btrfs) but the file is much smaller than that.
mke2fs would write highly compressible data, but sparsely, so the compression doesn't get much chance to work, hence the tiny amount of zstd data (or prealloc data for some versions of mkfs.ext4). There would be a large extent count from that, but the extent counts are missing from the output.
It looks fairly normal to me:
# truncate -s 100G test
# mkfs.ext4 test
mke2fs 1.46.2 (28-Feb-2021)
Discarding device blocks: done
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: 8db74c2a-4415-46aa-aec8-3347e73e4a4c
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information:
done
# compsize test
Processed 1 file, 98 regular extents (99 refs), 0 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 517M 517M 517M
none 100% 38M 38M 5.0M
prealloc 100% 478M 478M 512M
mkfs.ext4 does discards which turn into prealloc extents, so there's no zstd if you do it this way, but a copy of the test file looks like this:
# cp --reflink=never test test_cp
# compsize /tmp/test_cp
Processed 1 file, 100 regular extents (100 refs), 0 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 8% 464K 5.0M 5.0M
none 100% 204K 204K 204K
zstd 5% 260K 4.8M 4.8M
from compsize.
ant7511
commented on July 25, 2024
No, i copied from /dev/zero
dd if=/dev/zero of=./zeroimg.img bs=1M status=progress
102684950528 bytes (103 GB, 96 GiB) copied, 174 s, 590 MB/s^C
97936+0 records in
97936+0 records out
102693339136 bytes (103 GB, 96 GiB) copied, 174,014 s, 590 MB/s
compsize ./zeroimg.img
Type Perc Disk Usage Uncompressed Referenced
TOTAL 3% 2.9G 95G 95G
zstd 3% 2.9G 95G 95G
mkfs.ext4 ./zeroimg.img
compsize ./zeroimg.img
Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 517M 517M 517M
none 100% 517M 517M 517M
Now i checked the disk space after each step, the compsize shows the disk usage correctly, because after a dd copy 2.99M the disk usage,and after mkfs.ext2 the disk usage become only 432M.
from compsize.
Zygo
commented on July 25, 2024
mkfs.ext4 does hole punching, so the end result will usually be the same as the truncate:
# strace mkfs.ext4 test
[...]
fsync(3) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=107374182400, ...}) = 0
ioctl(3, BLKDISCARDZEROES, 0x7ffc1e027c10) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(3, BLKROGET, 0x7ffc1e027c10) = -1 ENOTTY (Inappropriate ioctl for device)
uname({sysname="Linux", nodename="tester39", ...}) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 0, 16777216) = 0
write(1, "Discarding device blocks: ", 26Discarding device blocks: ) = 26
write(1, " 4096/26214400", 17 4096/26214400) = 17
write(1, "\10\10\10\10\10\10\10\10\10\10\10\10\10) = 17
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 16777216, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 2164260864, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 4311744512, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 6459228160, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 8606711808, 2147483648) = 0
[...]
mkfs.ext4 discards any previous extents in the file, so the end result is the same as the truncate case. truncate is much faster because it doesn't waste 100GB of iops.
from compsize.
Related Issues (20)
- Exclusive / shared usage HOT 3
- wrong size: 1036 bytes file reported as 140M HOT 10
- Show sparse usage. HOT 1
- Change the name to btrfs-compsize HOT 4
- Error messages with Kernel 5.8.8 HOT 13
- compsize should also show dedupe% and total% not only comp% HOT 6
- Regular extent's header not 53 bytes (0) long?!? HOT 1
- compsize could be more user-friendly (from an enduser perspective) HOT 2
- SEARCH_V2 buffer is too big, slows compsize down HOT 8
- Support DESTDIR in Makefile
- Issues reported by static analysis HOT 1
- Option to always shows count of extents
- Makefile: Ensure that MAN_I directory exists
- Setuid root? HOT 3
- Add option to set fixed output format for size or at least raw bytes HOT 1
- Option to print each file HOT 4
- Feature request: compsize --find HOT 3
- Is the extent number accurate for telling file's fragmentation for compressed files? HOT 17
- Feature request: output only one value instead of a table HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from compsize.