I'm considering using this library in rclone.

It works very well according to my tests.

However I find that it imports more stuff than I'd like ideally. Specifically to use the squashfs library as an archiver (like archive/zip) pulls in github.com/CalebQ42/fuse and github.com/seaweedfs/fuse which I don't need and will take up space in my binary.

It looks possible to re-arrange the library so it has an internal "pure" squashfs implementation then add the fuse implementation on top of that.

I think it could even be done in a backwards compatible way with some cunning use of type aliases, though it might be cleaner to declare a v2.

What do you think?

How can I extract a specific file with this library. I have a big squashfs file BUT I only want to extract a small file from it. We can do that with unsquashfs like

unsquashfs my.squashfs -extract-file meta/snap.yaml

Hi there,

Thanks for your nifty library and it's been a real help for me.

But I've got something wrong when reading the squashfs on Windows hosts.

After some investigation, it seems line 55 here

shall be altered to

	split := strings.Split(name, string(os.PathSeparator))

When running on Windows, a path separator will be modified to \ after filepath.Clean even if it was passed with /
So I think it's better to split its path by os specified separator rather than hardcoding /

What do you think?

BRs.

Due to libfuse2 vanishing from mainstream distributions, we are forced to look into alternatives.

One possible alternative would be to write a new AppImage runtime in Go. https://github.com/orivej/static-appimage/ has done it, but using zip rather than squashfs. https://github.com/kost/static-appimage/releases has binaries to try out.

I can see a couple of advantages when using Go for the AppImage runtime, but as most things, this is a question of tradeoffs.

Do you think it would be possible to make https://github.com/orivej/static-appimage/ work with squashfs using https://github.com/CalebQ42/squashfs? Would it be a huge undertaking?

Seeing the issue on linux.

Architecture: x86_64
goos: linux
goarch: amd64

github.com/CalebQ42/squashfs/fuse_linux.go:3:15: undefined: fuse

probably import "github.com/CalebQ42/fuse" is missing.

The library can't be used in macOS due to a missing fuse.ENODATA symbol.

Is it possible to replace it with something go-generic, like os.ErrNotExists?

# github.com/CalebQ42/squashfs
../../../go/pkg/mod/github.com/!caleb!q42/[email protected]/fuse2.go:103:19: undefined: fuse.ENODATA
../../../go/pkg/mod/github.com/!caleb!q42/[email protected]/fuse2.go:115:14: undefined: fuse.ENODATA
../../../go/pkg/mod/github.com/!caleb!q42/[email protected]/fuse3.go:101:19: undefined: fuse.ENODATA
../../../go/pkg/mod/github.com/!caleb!q42/[email protected]/fuse3.go:113:14: undefined: fuse.ENODAT

Thanks @CalebQ42 for this extremely useful library.

When I wrote go-appimage, one area that I would have loved to have is the option to follow (dereference) symlinks when extracting from squashfs. Sometimes want to extract the symlinks as such, sometimes you want to extract the files/directories they point to.

I think the go-appimage code could be significantly cleaned up if this capability could become part of this library.

Hi @CalebQ42, thanks again for the great module!

I'm troubleshooting an issue with Anchore Syft (anchore/syft#1150) via its underlying stereoscope module (anchore/stereoscope#140), which is a consumer of version 0.5.4 of this module. The issue seems to present itself on Open is called on an inode that references a fragment block. At the user level, this manifests itself as an error being returned from Open. In the specific example referenced by anchore/syft#1150, the error is:

open opt/intel/oneapi/compiler/2022.1.0/linux/lib/oclfpga/host/linux64/bin/perl/lib/5.30.3/pod/perlpod.pod: unexpected EOF

Having immersed myself in your code as well as https://dr-emann.github.io/squashfs/squashfs.html, I believe I've identified a few issues with fragment handling. I'll open a PR shortly with patches that seem to fix the issue.

Thanks!

I'm getting an error when calling Open on a path obtained while running a WalkDir on an squashfs.FS. The path in question is a tad odd, namely, bin/[, but is otherwise unremarkable.

Having traced through the code, I think the problem is here:

At this point in the code, path.Match("[", "[") ends up being called, but this returns false for a match. I believe this is because [ is treated as a special character in Match. A direct check for equality may be more appropriate here? I will open a PR shortly with that for review.

Summary: A file with 321 NULLs unpacks as 128k of NULLs.

I've made a small test archive - unzip the bug.sqsh to demonstrate:

bug.sqfs.zip

go install github.com/CalebQ42/squashfs/...@main
go-unsquashfs bug.sqfs bug-out-go-unsquashfs
unsquashfs -d bug-out-unsquashfs bug.sqfs

Comparing the two extractions

$ ls -l bug-out-go-unsquashfs
total 128
-rw-rw-r-- 1 ncw ncw 131072 Aug 11 17:37 file2

vs

$ ls -l bug-out-unsquashfs
total 4
-rw-rw-r-- 1 ncw ncw 321 Sep  7  2020 file2

file2 is 321 bytes of NULLs where it seems to have unpacked as 131072 bytes (so one block) of NULLs with go-unsquashfs

bug.sqsh was made like this, so with default options for mksquashfs.

mksquashfs source/ bug.sqfs

Thanks for this awesome library. It would be great to be able to read the current directory with "." on fs.FS. This would also allow using fs.Walkdir starting at the current directory.

I tried to implement it and provided a PR #6

Hi

I am seeing an error while extracting.

"Unsupported file type. Inode type: 5"

This library is declared as MIT, however, depends on a library (statically) with a GPL-2.0 license (github.com/rasky/go-lzo). How much of a problem is this? I'm not certain... I'm not a lawyer 😄 . But, this is really easy to miss and and wanted to make certain you were aware of this.

(squashfs is awesome btw ❤️ !)

When I use this program to unpack an archive (I wrote this to learn the squashfs API)

package main

import (
	"context"
	"fmt"
	"io"
	"io/fs"
	"log"
	"os"
	"path/filepath"
	"runtime"

	"github.com/CalebQ42/squashfs"
	"golang.org/x/sync/errgroup"
)

func unpackSquashfs(inputFile, outputDir string) (err error) {
	ctx := context.Background()
	f, err := os.Open(inputFile)
	if err != nil {
		return fmt.Errorf("squashfs open failed: %w", err)
	}
	defer func() {
		closeErr := f.Close()
		if err == nil {
			err = closeErr
		}
	}()

	rd, err := squashfs.NewReader(f)
	if err != nil {
		return fmt.Errorf("squashfs reader creation failed: %w", err)
	}

	g, gCtx := errgroup.WithContext(ctx)
	g.SetLimit(runtime.NumCPU())

	err = fs.WalkDir(rd.FS, ".", func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		outPath := filepath.Join(outputDir, path)
		fmt.Printf("Writing %q\n", path)

		if d.IsDir() {
			return os.MkdirAll(outPath, 0777)
		}

		// Run file copies in parallel
		g.Go(func() error {
			if gCtx.Err() != nil {
				return nil
			}

			r, err := rd.FS.Open(path)
			if err != nil {
				return fmt.Errorf("unpack open source failed: %w", err)
			}
			defer r.Close()

			w, err := os.Create(outPath)
			if err != nil {
				return fmt.Errorf("unpack open destination failed: %w", err)
			}
			defer w.Close()

			_, err = io.Copy(w, r)
			if err != nil {
				return fmt.Errorf("unpack copy failed: %w", err)
			}
			return nil
		})

		return nil
	})
	if err != nil {
		return fmt.Errorf("squashfs read walk failed: %w", err)
	}

	err = g.Wait()
	if err != nil {
		return fmt.Errorf("squashfs copies failed: %w", err)
	}

	return nil

}

func main() {
	if len(os.Args) < 3 {
		log.Printf("Syntax: %s <image.sqfs> <directory>", os.Args[0])
		os.Exit(1)
	}
	imageFile, inputDirectory := os.Args[1], os.Args[2]

	err := unpackSquashfs(imageFile, inputDirectory)
	if err != nil {
		log.Fatalf("Failed: %v", err)
	}
}

then sometimes I get

Failed: squashfs read walk failed: readdir usr/local/lib/python3.11/dist-packages/tensorflow/include/external/gemmlowp/meta: unexpected EOF

I can reproduce this with go-unsquashfs but it appears to have a memory leak so unless you have a very large amount of RAM it won't finish. This probably deserves another issue!

Exactly which file returns the error varies

Failed: squashfs read walk failed: readdir usr/local/lib/python3.11/dist-packages/grpc/_cython: unexpected EOF

I tracked this error down to the binary.Read returning ErrUnexpectedEOF here

It does appear to be reading near the end of the file at the time the unexpected EOF is generated so I conjecture there is some padding at the end or something which is causing the binary.Read to read a few extra bytes.

I don't know why it is inconsistent though.

You can generate a squashfs which demonstrates the problem with this script

#! /usr/bin/bash
ORG=${ORG:-tensorflow}
IMG=${IMG:-tensorflow}
TAG=${TAG:-latest-gpu-jupyter}
docker export $(docker create $ORG/$IMG:$TAG) -o $IMG.tar.gz
mkdir -p $IMG && tar xf $IMG.tar.gz -C $IMG
[ -f $IMG.sqfs ] && rm $IMG.sqfs
mksquashfs $IMG $IMG.sqfs  -comp zstd -Xcompression-level 3 -b 1M -no-xattrs -all-root

If that doesn't demonstrate the problem I can send you one which does, but it is 3GB so will take a long time to upload with my puny internet!

PS Actually with the magic of docker you can probably generate an identical archive if I tell you the ID of the image I used is a2cf87758fef

The latest version v0.7.9 has broken squashfs.ENODATA value due to missing import: https://github.com/calebq42/squashfs/blob/v0.7.9/fuse_linux.go

The master branch is fixed. Can you please ship a new release with the fix?