cybershadow / btdu Goto Github PK

View Code? Open in Web Editor NEW

340.0 15.0 6.0 469 KB

sampling disk usage profiler for btrfs

License: GNU General Public License v2.0

D 96.69% Shell 2.14% Dockerfile 0.14% Roff 1.03%

btrfs

btdu's Introduction

btdu - sampling disk usage profiler for btrfs

Some btrfs features may make it difficult to estimate what disk space is being used for:

Subvolumes allow cheap copy-on-write snapshots of entire filesystem trees, with unmodified data being shared among snapshots
File and extent cloning allow creating cheap copies of files or parts thereof, with extents being stored only once
Compression transparently allows further reducing disk usage

For these reasons, classic disk usage analyzers such as ncdu cannot provide an accurate depiction of actual disk usage. (btrfs compression in particular is challenging to classic analyzers, and special tools must be used to query compressed usage.)

btdu is a sampling disk usage profiler for btrfs. It works according to the following algorithm:

Pick a random point on the disk in use
Find what is located at that point
Add the path to the results
Repeat the above steps indefinitely

Though it works by taking random samples, it is "eventually" accurate.

It differs from classic analyzers through the following properties:

btdu starts showing results instantly. Though wildly inaccurate at first, they become progressively more accurate the longer btdu is allowed to run.
btdu analyzes entire filesystems only. There is no way to analyze only a particular subdirectory or subvolume.
btdu counts extents used by multiple files only once. (The shortest path is used when placing the sample in the tree for visualization.)
By nature of its algorithm, btdu works correctly with compression and other btrfs filesystem features.
Because it queries raw filesystem metadata, btdu requires root privileges to run.

Use cases

Quickly summarize space usage

btdu needs to collect only 100 samples to achieve a ~1% resolution, which means it can identify space hogs very quickly. This is useful if the disk is full, and some space must be freed urgently to restore normal operation.
Estimate snapshot size

When an extent is in use by multiple files or snapshots, to decide where to place it in the browsable tree, btdu picks the path with the shortest length, or the lexicographically smaller path if the length is the same. An emergent effect of this property is that it can be used to estimate snapshot size, if your snapshots use a fixed-length lexicographically-ordered naming scheme (such as e.g. YYYY-MM-DD-HH-MM-SS): the size of snapshots displayed in btdu will thus indicate data that occurs in that snapshot or any later one, i.e. the amount of "new" data in that snapshot.
Estimate compressed data size

If you use btrfs data compression (whether to save space, improve performance, or conserve flash writes), btdu can be used to estimate how much real disk space compressed data uses.
Estimate unreachable extent size

A feature unique to btdu is the ability to estimate the amount of space used by unreachable parts of extents, i.e. data in extents containing older versions of file content which has since been overwritten. This btrfs "dark matter" can be an easily overlooked space hog, which can be eliminated by rewriting or defragmenting affected files.
Understand btrfs space usage

btdu shows explanations for hierarchy objects and common errors, which can help understand how btrfs uses disk space. The --expert mode enables the collection and display of additional size metrics, providing more insight into the allocation of objects with non-trivial sharing. Logical and physical sampling modes can help understand RAID space usage, especially when using multiple profiles.

Installation

btdu can be installed in one of the following ways:

Via package manager, if it is packaged by your distribution (see on the right).
Download a static binary from the releases page or the latest CI run.
Clone this repository and build from source (see below).

Building

Install a D compiler.
Note that you will need a compiler supporting D v2.097 or newer - the compiler in your distribution's repositories might be too old.
Install Dub, if it wasn't included with your D compiler.
Install libncursesw5-dev, or your distribution's equivalent package.
Run dub build -b release

Usage

Run btdu with root privileges as follows:

# btdu /path/to/filesystem/root

Note: The indicated path must be to the top-level subvolume (otherwise btdu will be unable to open other subvolumes for inode resolution). If in doubt, mount the filesystem to a new mountpoint with -o subvol=/,subvolid=5.

You can start browsing the results instantly; btdu will keep collecting samples to improve accuracy until it is stopped by quitting or pausing (which you can do by pressing p).

Run btdu --help for more usage information.

See CONCEPTS.md for information about some btdu / btrfs concepts, such as represented / exclusive / shared size.

Headless mode

With the --headless switch, btdu will run without the user interface. This is useful together with the --export option, which saves results to a file that can later be viewed in the UI using the --import option. For automated invocations, don't forget to specify a stop condition such as --max-time.

Example:

# btdu --headless --export=results.json --max-time=10m /path/to/filesystem/root
$ btdu --import results.json

Deleting

You can delete the selected file or directory from the filesystem by pressing d then ⇧ ShiftY. This will recursively delete the file or directory shown as "Full path".

Deleting files during a btdu run (whether via btdu or externally) skews the results. When deleting files from btdu, it will make a best-effort attempt to adjust the results to match. Statistics such as exclusive size may be inaccurate. Re-run btdu to obtain fresh results.

Marking

You can mark or unmark items under the cursor by pressing the space bar.

Press ⇧ ShiftM to view all marks, and ⇧ ShiftD then ⇧ ShiftY to delete all marked items.

Press * to invert marks on the current screen.

Marks are saved in exported .json files; a boolean field named "mark" will be present on marked nodes. Press ⇧ ShiftO to save an export file during an interactive session.

License

btdu is available under the GNU GPL v2. (The license is inherited from btrfs-progs.)

btdu's People

Contributors

Stargazers

Watchers

Forkers

stjordanis hongyunnchen abexamir xyyy1420 intelfx zll600

btdu's Issues

Feature suggestion: metric option when outputting in "du format".

Feature suggestion:
Be able to choose metric when running with --du flag. (Or perhaps, output all four metrics at once in four columns).

Often I am more interested in the distributed, shared and exclusive metrics, or to compare them with each other. (I have deduplicated the data with Bees.)

I also like the --du flag, since you can then sort the output like btdu-static-051-x86_64 --max-time=5m --du --headless /mnt | sort -n | tail -1000, to get a good overview of which directories and files take the most space (also when they might be under different top level directories).

Error when trying to import with v0.5.1

Hi,

When I try to import a file previously exported with btdu v0.5.1 I get this error:
Fatal error: Can't parse string: bool should be case-insensitive 'true' or 'false'

Export:
./btdu-static-x86_64-v0.5.1 /mnt/btrfsroot/ --minresolution=50000000 --headless --export=btduexport

Import:
./btdu-static-x86_64-v0.5.1 --import=btduexport

I'm on linux Mint 21.1 running the last x86_64 release downloaded from your github (without compiling it)

Strange/wrong size reported after a lot of samples

I have a raid5 array with more than 16TB data on it. At first, btdu shows numbers that look really good but if I let it run quite long (a few hours or so), the reported size is wrong completely (sometimes only a few GB).

btdu v0.2.0 @ /btrfs                                                              
--- / -----------------------------------------------------------------------------
  ~167.1 GiB [          ] /<RAID1>                                                  
  ~33.3 GiB [##########] /<RAID5>                                                 

--- Details: ----------------------------------------------------------------------
- Full path: /btrfs/                                                               
- Size: ~200.5 GiB (46054422 samples)                                              
- Average query duration: 10 ms, 723 μs, and 4 hnsecs                              

--- Explanation:                                                                   
Welcome to btdu. You are in the hierarchy root; results will be arranged according 
to their block group and profile, and then by path.                                

Use the arrow keys to navigate, press ? for help.                                  

Samples: 46054422  Resolution: ~403.8 KiB

This tool is BADASS - nicely done! Quick question

First thank you so much for building this app. I have been pulling my hair out (what's left of it) trying to figure out why my du -sh was showing only 4 TiB usage, but my df -lh was showing 8 TiB usage. Your tool showed me that about 3.47 TiB is unreachable.

I appreciate the great explanations you provided in the tool. Any idea how to clean up this unreachable part? NOTE: I am trying the defrag now to see if that works. Lifesaving tool --there are lots of people with similar questions on Stackoverflow which I will link this tool to answer.

Also please add a donation link I'd love to buy you a coffee. I'm adding this tool as a standard issue app to my servers now.

Add `du` compatible output formatting for batch mode

Misreported used size by btdu in pyshical mode (maybe)

I may be interpreting this data wrong, but it seems I may have found a bug in physical mode.

While the unallocated space and metadata(duplicated) show correctly, once I enter /SINGLE/DATA and start navigating my files, some of them show the wrong space used (much less than what the actual file takes up), and the free space (UNUSED) is wildly wrong.

Obs: my BTRFS install is a simple one, automatically done by manjaro in a single SSD.

Here are some commands output:

BTDU in -p mode:

btdu v0.5.0 @ /mnt/sda2                                                                                                                                      [PAUSED] 
--- /<SINGLE>/<DATA> --------------------------------------------------------------------------------------------------------------------------------------------------
  ~11.82 GiB [#         ]  <NO_INODE>                                                                                                                                  
  ~71.00 GiB [##########]  <UNUSED>
  ~21.00 GiB [##        ] /@
  ~4.756 GiB [          ] /@cache
  ~66.99 GiB [######### ] /@home
  ~2.037 GiB [          ] /@log
  ~4.491 GiB [          ] /timeshift-btrfs

--- Details: ----------------------------------------------------------------------------------------------------------------------------------------------------------
- Full path: /mnt/sda2
- Average query duration: 0.0002491 seconds
- Represented size: ~182.1 GiB (306451 samples), ±242.6 MiB
- Logical offsets: ..., 185807908152, 67499195313, 26414524317
- Physical offsets: ..., 1:336123374904, 1:68564548529, 1:25332393885`

BTDU in normal mode:

btdu v0.5.0 @ /mnt/sda2                                                                                                                                      [PAUSED] 
--- /<SINGLE>/<DATA> --------------------------------------------------------------------------------------------------------------------------------------------------
  ~19.84 GiB [##        ]  <NO_INODE>                                                                                                                                  
  ~19.38 GiB [#         ]  <UNUSED>
  ~27.50 GiB [##        ] /@
  ~7.785 GiB [          ] /@cache
  ~98.93 GiB [##########] /@home
  ~2.399 GiB [          ] /@log
  ~6.113 GiB [          ] /timeshift-btrfs

--- Details: ----------------------------------------------------------------------------------------------------------------------------------------------------------
- Full path: /mnt/sda2
- Average query duration: 0.0002802 seconds
- Represented size: ~182.0 GiB (276869 samples), ±102.6 MiB
- Logical offsets: ..., 227972342342, 98740561859, 78583809087

And sudo btrfs fi usage /:

Overall:
    Device size:                 210.53GiB
    Device allocated:            190.02GiB
    Device unallocated:           20.50GiB
    Device missing:                  0.00B
    Device slack:                  3.50KiB
    Used:                        167.52GiB
    Free (estimated):             39.77GiB      (min: 29.52GiB)
    Free (statfs, df):            39.77GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              387.14MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:182.01GiB, Used:162.74GiB (89.41%)
   /dev/sda2     182.01GiB

Metadata,DUP: Size:4.00GiB, Used:2.39GiB (59.74%)
   /dev/sda2       8.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB (0.59%)
   /dev/sda2      16.00MiB

Unallocated:
   /dev/sda2      20.50GiB

I'am also having an issue with NO_INODE link to my reddit post, but it is unrelated to this finding. (of course, if you know how to solve this please do tell me, I'am only commenting about this in this issue in case the NO_INODE part is somehow related to this).

Ehancement suggestion : Add export/import functionnality ?

Hello,

I like the export/import ability of ncdu, I export the scans of drives so I can quickly search what is taking space without plugging the drive or launching a new scan.

I wonder if it could be possible to do the same with btdu for btrfs?
It would maybe be harder to set because there are extra steps : let the scan running to wanted size level then press "p" key and "enter" key.

Btdu freezing

Hello,

I'm experiencing a issue with btdu freezing on my machine after a few seconds. I'm using NixOS 22.05 (I'm the maintainer of the package there), while using a lenovo ideapad 3 with a ryzen 5500U.

Best regards.

Feature request: write file paths chosen from UI into a log

Hi!

First, this is an excellent implementation of an excellent concept. It has made browsing my old archive disks so much easier and more efficient, so thanks!

I note you have a feature to delete things on the fly from the UI. That's nice, however, I would rather batch up my deletes, and have a log of what I have deleted.

To that end, it would be really useful to have a key which 'marks' a file, so and records those paths into a log, so that they may be filtered and deleted through other tooling.

Another issue here are shared extents - what to do if some extents belong to multiple snapshots or reflink copies. It would be quite useful to me to know if there are files which need to be deleted together to liberate the space relating to them in practice.

Thanks for considering the above, all the best.

Build Error on Debian 10.

I'm trying to build this from source on Debian 10 (Buster). Am I missing a dependency?

I'm getting the following error when trying to build with dub build -b release

Performing "release" build using /usr/bin/ldc2 for x86_64.
ncurses 0.0.149: target for configuration "minimal" is up to date.
btdu 0.2.1: building configuration "application"...
source/btdu/browser.d(24,8): Error: module `locale` is in file 'core/sys/posix/locale.d' which cannot be read
import path[0] = source/
import path[1] = /root/.dub/packages/ae-0.0.2833
import path[2] = /root/.dub/packages/btrfs-0.0.11/btrfs/src/
import path[3] = /root/.dub/packages/ncurses-0.0.149/ncurses/source/
import path[4] = /usr/lib/ldc/x86_64-linux-gnu/include/d/ldc
import path[5] = /usr/lib/ldc/x86_64-linux-gnu/include/d
import path[6] = /usr/include/d
/usr/bin/ldc2 failed with exit code 1.

My locale is set to:

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Maintainer of btdu not reachable via E-Mail in the AUR, Submitted btdu-bin into the AUR due to issues on aarch64

See title.

Mail to Vladimir Panteleev <arch-pkg at thecybershadow.net> was returned to sender.
Submission of https://aur.archlinux.org/packages/btdu-bin was required cause cannot compile the binaries on aarch64 due to the d compilers. Not even when resolving the required packages via AUR / pacman. Specifically on the m1 mac mini running Archlinux.

Btrfs as Root Filesystem

Hi!

I installed the latest version of btdu as follows.

wget https://github.com/CyberShadow/btdu/releases/download/v0.3.1/btdu-static-x86_64
mv btdu-static-x86_64 btdu
chmod +x btdu

However, during execution the following error occurs.

$ ./btdu /
Fatal error: / is not the root btrfs subvolume - please specify the path to a mountpoint mounted with subvol=/ or subvolid=5
E.g.: mkdir /tmp/sda2 && mount -o subvol=/ /dev/sda2 /tmp/sda2 && ./btdu /tmp/sda2

$ ./btdu /dev/sda2       
Fatal error: /dev/sda2 is not a btrfs filesystem

$ ./btdu /@rootfs 
Fatal error: open (No such file or directory)

$ ./btdu /\@rootfs
Fatal error: open (No such file or directory)

In my case Btrfs is mounted as root partition.

$ cat /etc/fstab | grep btrfs
UUID=2080d58a-d1b3-4bc7-bd8d-67662935e96a /               btrfs   defaults,space_cache=v2,subvol=@rootfs 0       0

$ df -Th -t btrfs    
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sda2      btrfs  9.8T   50G  9.8T   1% /

$ mount | grep btrfs | grep -v docker
/dev/sda2 on / type btrfs (rw,relatime,space_cache=v2,subvolid=256,subvol=/@rootfs)

$ findmnt -nt btrfs | grep -v docker
/                                       /dev/sda2[/@rootfs]                                      btrfs  rw,relatime,space_cache=v2,subvolid=256,subvol=/@rootfs

I am not sure: am I doing something wrong or is this a false positive?

Thanks,
Markus

Also release an ARMv7 version as well

For those who still use an older raspberry this would be useful.

Seeing "ENOENT" under "/SINGLE/DATA/ERROR/logical info"

I ran btdu against my btrfs partition mouted as read-only and saw the following. Tried defragment, no help.

 btdu v0.3.0 @ /mnt                                                                                                                                                                   [PAUSED]
--- /SINGLE/DATA/ERROR/logical ino ------------------------------------------------------------------------------------------------------------------------------------------------------------
  ~116.9 GiB [100.0% ##########]  ENOENT

--- Details: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- Average query duration: 0.0000153 seconds
- Represented size: ~116.9 GiB (123783 samples)
- Logical offsets: ..., 1004368585530, 178054220006, 421607607093

--- Explanation:
An error occurred while trying to look up which inodes use a particular logical offset.

Children of this node indicate the encountered error code, and may have a more detailed explanation attached.

 btdu v0.3.0 @ /mnt                                                                                                                                                                   [PAUSED]
--- INFO: /SINGLE/DATA/ERROR/logical ino/ENOENT -----------------------------------------------------------------------------------------------------------------------------------------------

--- Details:
- Error code: 2
- Error message: No such file or directory
- Average query duration: 0.0000153 seconds
- Represented size: ~116.9 GiB (123783 samples)
- Logical offsets: ..., 1004368585530, 178054220006, 421607607093

--- Explanation:
btrfs reports that there is nothing at the random sample location that btdu picked.

This most likely represents allocated but unused space, which could be reduced by running a balance on the DATA block grp.

Could anyone help

What does the error mean? Or what documents/manuals should I check?
What should I do with the error?
Do I really need to run balance even there is only one device to recover the space?

Thanks in advance.

Can't compile properly from source on Linux Mint Cinnamon

Hi,

I'm a total newbie so maybe I do something wrong.

I can't successfully compile from source I get this error when building :

sudo apt install dub
dub build -b release

Performing "release" build using /usr/bin/gdc for x86_64.
Build directory ../../.dub/packages/emsi_containers-0.9.0/emsi_containers/.dub/build/library-release-linux.posix-x86_64-gdc_2068-E53B02A4D0A6D2BC0039752B2E02A84F/ is not writable. Falling back to direct build in the system's temp folder.
emsi_containers 0.9.0: building configuration "library"...
../../.dub/packages/emsi_containers-0.9.0/emsi_containers/src/containers/dynamicarray.d:10:16: error: module lifetime is in file 'core/lifetime.d' which cannot be read
 private import core.lifetime : move, moveEmplace, copyEmplace, emplace;
                ^
import path[0] = /usr/lib/gcc/x86_64-linux-gnu/8/include/d
import path[1] = /home/zzz/.dub/packages/emsi_containers-0.9.0/emsi_containers/src
/usr/bin/gdc failed with exit code 1.

edit:
lsb_release -a

No LSB modules are available.
Distributor ID:	LinuxMint
Description:	Linux Mint 19.2 Tina
Release:	19.2
Codename:	tina

I tried to install dmd from https://s3.us-west-2.amazonaws.com/downloads.dlang.org/releases/2021/dmd_2.098.1-0_amd64.deb but I just realize that it failed :

Preparing to unpack .../dmd_2.098.1-0_amd64.deb ...
Unpacking dmd (2.098.1-0) ...
dpkg: error processing archive /tmp/mozilla_zzz0/dmd_2.098.1-0_amd64.deb (--install):
 trying to overwrite '/usr/bin/dub', which is also in package dub 1.8.0-2
dpkg-deb (subprocess): decompressing archive member: lzma write error: Broken pipe
dpkg-deb (subprocess): cannot copy archive member from '/tmp/mozilla_zzz0/dmd_2.098.1-0_amd64.deb' to decompressor pipe: failed to write (Broken pipe)
dpkg-deb: error: <decompress> subprocess returned error exit status 2
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...

Maybe it comes from here

edit2 :

As asked I tried to remove dub and give an other try with dmd this worked.
Then I tried reinstall dub:

sudo apt install dub

Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances       
Lecture des informations d'état... Fait
Les NOUVEAUX paquets suivants seront installés :
  dub
0 mis à jour, 1 nouvellement installés, 0 à enlever et 29 non mis à jour.
Il est nécessaire de prendre 1 443 ko dans les archives.
Après cette opération, 5 903 ko d'espace disque supplémentaires seront utilisés.
Réception de :1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 dub amd64 1.8.0-2 [1 443 kB]
1 443 ko réceptionnés en 2s (722 ko/s)
Sélection du paquet dub précédemment désélectionné.
(Lecture de la base de données... 551483 fichiers et répertoires déjà installés.)
Préparation du dépaquetage de .../archives/dub_1.8.0-2_amd64.deb ...
Dépaquetage de dub (1.8.0-2) ...
dpkg: erreur de traitement de l'archive /var/cache/apt/archives/dub_1.8.0-2_amd64.deb (--unpack) :
 tentative de remplacement de « /usr/bin/dub », qui appartient aussi au paquet dmd 2.098.1-0
dpkg-deb: erreur: coller subprocess was killed by signal (Relais brisé (pipe))
Des erreurs ont été rencontrées pendant l'exécution :
 /var/cache/apt/archives/dub_1.8.0-2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

edit3 :

As it was written on the doc, install dub if needed (but it already was with dmd) so let's compile again:

dub build -b release

Performing "release" build using /usr/bin/dmd for x86_64.
emsi_containers 0.9.0: building configuration "library"...
ncurses 0.0.149: building configuration "minimal"...
../../.dub/packages/ncurses-0.0.149/ncurses/source/deimos/ncurses/curses.d(1486,1): Deprecation: Usage of the `body` keyword is deprecated. Use `do` instead.
btdu ~master: building configuration "application"...
../../.dub/packages/ncurses-0.0.149/ncurses/source/deimos/ncurses/curses.d(1486,1): Deprecation: Usage of the `body` keyword is deprecated. Use `do` instead.
Linking...
/usr/bin/ld : ne peut trouver -lncursesw
collect2: error: ld returned 1 exit status
Error: linker exited with status 1
/usr/bin/dmd failed with exit code 1.

edit4 :

libncursesw5-dev was missing, so :

sudo apt install libncursesw5-dev

And now it's compiling, thank you for your help :)

no way to see the subvolume?

Sorry, I'm no btrfs expert but it seems the way my btrfs is mounted, I can't use this program?

root@s:/tmp# mount|grep btr
/dev/sda1 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=257,subvol=/rootsubvol)
/dev/sda1 on /var/lib/docker/btrfs type btrfs (rw,relatime,ssd,space_cache,subvolid=257,subvol=/rootsubvol)
root@s:/tmp#



root@s:/tmp# btrfs subv list /|grep -v docker
ID 257 gen 462716 top level 5 path rootsubvol
ID 374 gen 459325 top level 257 path srv/monica
ID 375 gen 172293 top level 374 path srv/monica/.snapshot-2020-11-16-163135
root@s:/tmp# ./btdu-static-x86_64 rootsubvol
Fatal error: open (No such file or directory)
root@s:/tmp# ./btdu-static-x86_64 /rootsubvol
Fatal error: open (No such file or directory)
root@s:/tmp#

Mounting top-level subvolume example and risk?

BTDU requires to mount the top-level subvolume (-o subvol=/,subvolid=5).

My environment is a Synology raid.

And so I'm wondering:

What is the bash command to mount the top-level subvolume (read-only)
Considering that child subvolumes are already mounted separately, is there any risk of corruption to mounting the top level one (as the child ones are inside of/beneath it)?
Could you consider adding the reply to the official help file so others can find out quicker?

Thanks a lot!.

Min Resolution: Floating Point Exception

Hi,

I get the following error.

$ ./btdu --headless --min-resolution=1G --du /mnt/sda2 
[1]    1073680 floating point exception  ./btdu --headless --min-resolution=1G --du /mnt/sda2

I have mounted my top-level btfs subvolume as follows.

$ mkdir /mnt/sda2
$ mount -o subvol=/ /dev/sda2 /mnt/sda2

Below is a list of commands that work and those that do not.

# works
./btdu /mnt/sda2
./btdu --headless --max-time=10s --du /mnt/sda2

# works not
./btdu --headless --min-resolution=1G --du /mnt/sda2
./btdu --headless --min-resolution=1000 --du /mnt/sda2
./btdu --headless --max-time=10s --min-resolution=1000 --du /mnt/sda2

So I think it has something to do with the --min-resolution option.

Could this be a bug or am I doing something wrong?

I use version v0.5.0.

$ ./btdu --help
btdu v0.5.0
(..)

Best,
Markus

How to track if files are deduplicated?

Let's say I copy the same 10MB file from an external device to BTRFS into two folders.

tmp\a\10mb.dat - Exclusive 10mb
tmp\b\10mb.dat - Exclusive 10mb
tmp\ - Exclusive 20mb

Now I create a snapshot and I get (as they share space with the snapshot):

tmp\a\10mb.dat - Exclusive 0mb
tmp\b\10mb.dat - Exclusive 0mb
tmp\ - Exclusive 0mb - Shared 20mb

This isn't helpful to know what are debup candidates. So I'm wondering can BTDU help me here to see that a & b are not deduplicated (without deleting all snapshots)?

So that I can see this after a dedup with ie cp --reflink=always I get (with and without snapshot)

tmp\a\10mb.dat - Exclusive 0mb
tmp\b\10mb.dat - Exclusive 0mb
tmp\ - Exclusive 0mb - Shared 10mb

"Status bar" (samples and resolution) does not appear anymore after ed368d5

synopsis

It appears ed368d5 introduced a regression. The "status bar" with the sample count and approximate resolution does not show for me anymore.

environment

Arch, GNOME Terminal + tmux

demo

working

regressed

GCC bug with btdu

I tried to use gdc (gcc d compiler), however it failed. It seems to be a bug in gcc(*). I was able to compile btdu with LLVM's ldc, which worked good.

To use another compiler than dmd, I used rhe following:

dub build --compiler=ldc2 -b release

It is interesting that ldc2 made a much smaller binary.

-rwxr-xr-x 1 forza forza  19M Aug 15 23:17 btdu.dmd*
-rwxr-xr-x 2 forza forza 4,2M Sep 30 23:15 btdu.ldc2*
-rwxr-xr-x 2 forza forza 5,7M Sep 30 23:33 btdu.ldc2.O3-lto*

(*) https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111650

Can't run btdu on ubuntu 22.04 wsl 2

btdu .
Fatal error: Unexpected ';' when converting from type const(char)[] to type int

I think btdu is looking at some windows environment variable or something, those are separated with ;. Without what the env is I can't help more I think

What does <ERROR> mean?

Feature requests: Allow to delete read-only snapshots

Hello! Thanks for great software.
It would be really handy to be able to delete btrfs snapshots right from btdu.

Thanks.

UI hang, 100% CPU usage

Hi,

This is on version v0.3.1, running on NixOS - I'm aware you've got a newer version, but I don't have a chance to try this out right now and thought you may be interested in the report anyway. Please feel free to close this issue for any reason.

I've used btdu a fair amount without problems. Today I tried applying it to new disk, larger than I normally use it on, which is NVMe. I observe a hang (instantly) after ~1000 samples (not repeatable exactly), where the UI subsequently doesn't update nor respond to input.

CPU usage remains at 100% though. I captured the following report of what btdu is doing using sudo perf record --all-user --call-graph dwarf,65528 -F 100 btdu and perf report.

-   92.63%     0.00%  btdu     btdu                  [.] _start                                                                                                              ▒
     _start                                                                                                                                                                  ▒
     __libc_start_main_alias_2 (inlined)                                                                                                                                     ▒
     __libc_start_call_main                                                                                                                                                  ◆
     _d_run_main                                                                                                                                                             ▒
     _d_run_main2                                                                                                                                                            ▒
     _D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv                                                                                                                       ▒
     _Dmain                                                                                                                                                                  ▒
     _D2ae5utils6funopt__TQkS_D4btdu4main7programFSQBsQBsQBp__T11_OptionImplVEQCtQCtQCq10OptionTypei2TAyaVQea45_5061746820746f2074686520726f6f74206f66207468652066696c6573797▒
   - _D4btdu4main7programFS2ae5utils6funopt__T11_OptionImplVEQBiQBiQBf10OptionTypei2TAyaVQea45_5061746820746f2074686520726f6f74206f66207468652066696c6573797374656d20746f2061▒
      - 88.54% _D4btdu7subproc10Subprocess11handleInputMFZv                                                                                                                  ▒
         - 88.46% _D4btdu5proto__T5parseTSQw7subproc10SubprocessZQBfFKAhKQBgZm                                                                                               ▒
            - 56.05% _D4btdu7subproc10Subprocess13handleMessageMFSQBr5proto13ResultMessageZv                                                                                 ▒
               - _D4btdu5paths7SubPath8__mixin210appendNameMFIAaZPSQBwQBuQBr                                                                                                 ▒
                    19.93% __memcmp_avx2_movbe                                                                                                                               ▒
                    0.81% bcmp@plt                                                                                                                                           ▒
            - 32.25% _D4btdu7subproc10Subprocess13handleMessageMFSQBr5proto16ResultEndMessageZv                                                                              ▒
               - 31.45% _D4btdu5paths11BrowserPath8__mixin710appendPathMFIPSQByQBw7SubPathZ7recurseMFPSQCzQCxQCuIQBnZQq                                                      ▒
                  - 29.91% _D4btdu5paths11BrowserPath8__mixin710appendNameMFSQBwQBu__T9InlineArrTyaVmi23ZQtZPSQDdQDbQCy                                                      ▒
                       10.13% __memcmp_avx2_movbe                                                                                                                            ▒
                  - 1.54% _D4btdu5paths11BrowserPath8__mixin710appendPathMFIPSQByQBw7SubPathZ7recurseMFPSQCzQCxQCuIQBnZQq                                                    ▒
                       1.30% _D4btdu5paths11BrowserPath8__mixin710appendNameMFSQBwQBu__T9InlineArrTyaVmi23ZQtZPSQDdQDbQCy                                                    ▒
      + 4.05% _D4btdu6sample14subprocessMainFAyaZv

Accidental O(N^2) behaviour?

This is somewhat academic since you clearly get very good resolution with a low number of samples, but maybe there are non-academic issues lurking you might be interested in.

I observe that if I double the samples, it takes approx 4x as long to generate an export. Is there some accidental N^2 complexity somewhere? I don't think this is necessary a big deal (so feel free to close), but thought you might like the data:

Note I'm using handwavy order-of-magnitude math but I might have expected the latter to take (5.3s x 4 = ) ~20s rather than ~40s.

Collected 237700 samples (achieving a resolution of ~9.999 MiB) in 5 secs, 143 ms, 833 μs, and 4 hnsecs.
real	0m5.338s / user	0m5.083s / sys	0m0.198s

Collected 475616 samples (achieving a resolution of ~4.997 MiB) in 16 secs, 432 ms, and 132 μs.
real	0m16.710s / user	0m16.259s / sys	0m0.420s

Collected 950729 samples (achieving a resolution of ~2.500 MiB) in 45 secs, 119 ms, 284 μs, and 8 hnsecs.
real	0m45.520s / user	0m44.655s / sys	0m0.854s

X axis: # samples
Y axis: Time per sample in microseconds (i.e, it's getting more costly per sample)

Automatic remount with `-o subvolid=5`?

Hi! Is there any reason why btdu does not just try to remount the filesystem of given path with -o subvolid=5?

i. e. can we just do this? https://github.com/intelfx/bin/blob/master/wrappers/btdu

Can't seem to find root

Hello, I must be misunderstanding something basic. I'm trying to run btdu on my system, which is a Garuda Linux (Arch derivative) distribution.

Here's the fstab entry for /. Isn't this enough to run? Instead I get the error that I'm

UUID=.... / btrfs subvol=/@,defaults,noatime,space_cache,noautodefrag,compress-force

Instead I get the error

Fatal error: / is not the root btrfs subvolume

Probably I need to temporarily mount something, but I'm not exactly sure how.

Packaging for OpenSUSE distros

Opening a new issue so as not to hijack #25

Thank you for the kind words!

I agree that a package file provided by us wouldn't be too helpful in practice. What would be helpful though is to get btdu packaged in your distro. If that's something you'd be interested in doing, it will make the tool more accessible to that distro's users. I'm also happy to help with packaging issues.

You are most welcome. I'm certainly interested in helping to get this into OpenSUSE's distros. The first step in getting it into OpenSUSE will be that I will package it in my personal public repository on their build server, and then from there I submit it to the main repo and make any required adjustments until they are satisfied with the package, and then upon acceptance it will be available in the distros' package managers

The one thing I know will be the 'trick' is that this distro builds packages in RPM format, and adheres to redhat's practice of building from source on an 'island' which means no binaries in sources and notably no internet access during the build, so I'm sure that dub won't work in the usual way. For node and rust (npm/cargo) the way this is dealt with, is to 'vendor' the packages fetched by the build tool, into an additional source archive, and during the build, run those tools in a manner that prevents them from attempting to go online.

I did have a quick look at this and found that dub had a similar option to npm's --offline argument, but I'm not 100% across it yet - specifically I'm not sure how to get it to download the remote sources to form the 'vendored source' archive, and then where to put them so it will access them. I'm confident it's documented, but if you know the way to do this off the top of your head, I'd be very grateful for any tips - but if you're unsure, don't worry, leave it with me and I'll figure it out and ask here if I get stuck.

I do have some health issues so I am a little unreliable with times, so I may not be the fastest, but the way I see it, I owe you several hours of fixing my PC that would have happened without btdu to save me, so this is a high priority to me.

How to get Checksum from releases

Hi,

I love you work (thank you again),
How can we verify the checksum of a binary file downloaded from your repo?
I usually I download the file, do a sha256sum locally and compare the result with the site were I downloaded it.