namjaejeon / ntfsprogs-plus Goto Github PK

Validate variables of mft entry, If user give all yes or auto fix option, repair it and write to disk.

I tried to apply address sanitizer to ntfsck

tests/mismatch_index_filename found below issues

==4022496==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2176 byte(s) in 16 object(s) allocated from:
#0 0x7f37f5e16a06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x7f37f5c9777a in ntfs_calloc /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/misc.c:47
#2 0x7f37f5c790f7 in __ntfs_inode_allocate /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:95
#3 0x7f37f5c79330 in ntfs_inode_real_open /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:171
#4 0x7f37f5c7b49d in ntfs_inode_open /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:486
#5 0x56444a4b6c59 in ntfsck_check_system_files /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2647
#6 0x56444a4b71a6 in main /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2740
#7 0x7f37f5a24082 in __libc_start_main ../csu/libc-start.c:308

Direct leak of 512 byte(s) in 1 object(s) allocated from:
#0 0x7f37f5e16808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7f37f5c977e3 in ntfs_malloc /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/misc.c:57
#2 0x7f37f5cc4a38 in ntfsck_verify_boot_sector /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/volume.c:571
#3 0x7f37f5cc53fa in ntfs_volume_startup /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/volume.c:703
#4 0x7f37f5cc6c42 in ntfs_device_mount /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/volume.c:1075
#5 0x7f37f5cca001 in ntfs_mount /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/volume.c:1564
#6 0x56444a4b69b6 in ntfsck_mount /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2590
#7 0x56444a4b7190 in main /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2738
#8 0x7f37f5a24082 in __libc_start_main ../csu/libc-start.c:308

Direct leak of 136 byte(s) in 1 object(s) allocated from:
#0 0x7f37f5e16a06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x7f37f5c9777a in ntfs_calloc /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/misc.c:47
#2 0x7f37f5c790f7 in __ntfs_inode_allocate /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:95
#3 0x7f37f5c79330 in ntfs_inode_real_open /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:171
#4 0x7f37f5c7b49d in ntfs_inode_open /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:486
#5 0x56444a4b4fa3 in ntfsck_scan_index_entries_btree /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2311
#6 0x56444a4b65f4 in ntfsck_scan_index_entries /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2529
#7 0x56444a4b724e in main /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2748
#8 0x7f37f5a24082 in __libc_start_main ../csu/libc-start.c:308

Direct leak of 136 byte(s) in 1 object(s) allocated from:
#0 0x7f37f5e16a06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x7f37f5c9777a in ntfs_calloc /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/misc.c:47
#2 0x7f37f5c790f7 in __ntfs_inode_allocate /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:95
#3 0x7f37f5c79330 in ntfs_inode_real_open /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:171
#4 0x7f37f5c7b49d in ntfs_inode_open /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/inode.c:486
#5 0x56444a4b5f48 in list_dir_entry /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2455
#6 0x7f37f5c5d005 in ntfs_readdir /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/libntfs-3g/dir.c:1156
#7 0x56444a4b63cc in ntfsck_scan_index_entries_bitmap /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2508
#8 0x56444a4b6609 in ntfsck_scan_index_entries /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2531
#9 0x56444a4b724e in main /home/pinocc/Projects/filesystems/ntfs/ntfsprogs/ntfsprogs/ntfsck.c:2748
#10 0x7f37f5a24082 in __libc_start_main ../csu/libc-start.c:308

Parse #1: Check check system files...
Parse #2: Replay logfile...
Parse #3: Check index entries in volume...
Parse #4: Check mft entries in volume...
Clear the bit of mft no(28) in the $MFT/$BITMAP corresponding orphaned MFT record(y/N)? N
Parse #5: Check cluster bitmap...
1 errors found.

$ sudo ./src/ntfsck -C /dev/sdf1 -v
Boot sector verification complete. Proceeding to $MFTCheck volume: Volume is dirty.

Validate variables of attribute, If user give all yes or auto fix option, repair it and write to disk.

How to handle if two mft cluster runs are pointing same lcn cluster number.

<src/ntfsck.c>
2668 /* The first index entry. /
2669 next = (INDEX_ENTRY)((u8*)&ir->index +
2670 le32_to_cpu(ir->index.entries_offset));
2671
2672 if (next->ie_flags & INDEX_ENTRY_NODE) { <=== at here

if ir->index.entries_offset has very large value, illegal memory access occurs

Currently we just remove orphan inode(mft entry), We need to move it to lost&found directory.

The system meta file-map differs according to the ntfs version. Currently, only ntfs 3.0 is checked, but for lower versions, other checks are required according to other system meta file-map...

linkinjeon@linkinjeon-Precision-Tower-3620:~/git/ntfs_work/tool/ntfsprogs$ sudo ./src/ntfsck -n /dev/sdc1
Parse #1: Check system files...
Parse #2: Replay logfile...
Parse #3: Check index entries in volume...
Reparse tag is different (IDX/$FN:0000002d MFT/$FN:0000002d) on inode(3614, libtest.la) (y/N)? N
Reparse tag is different (IDX/$FN:0000002d MFT/$FN:0000002d) on inode(2000, 035.out) (y/N)? N
Parse #4: Check mft entries in volume...
Parse #5: Check cluster bitmap...
2 errors found, 0 fixed

On a freshly formatted volume, inodes 0x0B to 0x0F are marked as in use, but empty. Inodes 0x10 to 0x17 are marked as free and not used. This doesn't change until the volume is under a lot of stress. how do fsck check it ?

After checking filesystem with ntfsck, some issues are still remained.
(corrupted image is from https://github.com/namjaejeon/linux-ntfs/issues/90)

$ ./ntfsprogs/ntfsprogs/ntfsck -a /dev/sdc1
Parse #1: Check check system files...
Parse #2: Replay logfile...
Volume is dirty.
Parse #3: Check index entries in volume...
Sizes of index allocation is corrupted, Removing and recreating attribute(y/N)? y
Parse #4: Check mft entries in volume...
Found an orphaned file(mft no: 39). Try to add index entry(y/N)? y
Index lookup failed, inode 38: No such file or directory
Failed to sync FILE_NAME (inode 39): Input/output error
Parse #5: Check cluster bitmap...
Found missing cluster bit set : 56644 in cluster bitmap. Set orphaned cluster bit into cluster bitmap(y/N)? y
Found missing cluster bit set : 56645 in cluster bitmap. Set orphaned cluster bit into cluster bitmap(y/N)? y
Last cluster bit for backup boot sector is not set in $Bitmap
Clean, No errors found

$ ./ntfsprogs/ntfsprogs/ntfsck -a /dev/sdc1
Parse #1: Check check system files...
Parse #2: Replay logfile...
Parse #3: Check index entries in volume...
Parse #4: Check mft entries in volume...
Parse #5: Check cluster bitmap...
Found orphaned cluster bit : 56644. Clear orphaned cluster bit in cluster bitmap(y/N)? y
Found orphaned cluster bit : 56645. Clear orphaned cluster bit in cluster bitmap(y/N)? y
Last cluster bit for backup boot sector is not set in $Bitmap
Clean, No errors found

1. read
2. readdir

Maximum 32MB lcn bitmap buffer can be allocated with 1TB HDD.

Is attribute order corrupted possible?
And if possible, how to fix it's order. We need to decide that place to check, and how to fix.

Validate variables of index entry, If user give all yes or auto fix option, repair it and write to disk.

Can reproduce it using make_disk_corruption.sh. Windows could recover this case.

$ ./make_disk_corruption.sh 8197 4096 /dev/sdb1
OFFSET:8197 , END:12293 , FILE:/dev/sdb1

== zero 1byte patten corruption test ==
OFFSET:8197 , PATTERN:0x00
wrote 1/1 bytes at offset 8197
1.000000 bytes, 1 ops; 0.0051 sec (195.694716 bytes/sec and 195.6947 ops/sec)
OFFSET:8197 , PATTERN:0xFF
wrote 1/1 bytes at offset 8197
1.000000 bytes, 1 ops; 0.0008 sec (1.218 KiB/sec and 1246.8828 ops/sec)
ntfs_mst_post_read_fixup_warn: magic: 0x454c4946 size: 1024 usa_ofs: 65328 usa_count: 3: Invalid argument
MFT record(0)'s attribute start offset is corrupted
ntfs_mst_post_read_fixup_warn: magic: 0x454c4946 size: 1024 usa_ofs: 65328 usa_count: 3: Invalid argument
MFT record(0)'s attribute start offset is corrupted
1 errors left (errors:1, fixed:0)

if data variables of non-resident of normal file are corrupted, how do fsck fix it ?
data variables means:

1. data_size, allocated size, initialized size of non-resident.
2. cluster runs, mapping offset and length, etc..
3. cluster bit in lcn bitmap is clear.

if mapping offset of non-resident of normal file and directory are corrupted, windows try to remove this attribute and recreate.

1. if lcn bitmap data size and allocated size are different ?
2. data size and number of clusters in cluster run are different ?

Because this repository is changed to private, github action of linux-ntfs cannot clone this repository.

how does fsck validate hard links(several FILENAME attributes) in mft entry.

Validate variables of index block, If user give all yes or auto fix option, repair it and write to disk.

After a disk is unplugged, 100% core usage holds for a long time.