Describe the bug

We compile xz as a Meson subproject. We have AppVeyor VS2019 as CI job, they recently updated their toolchain to LLVM/Clang 16.0.1, we use clang-cl to compile everything. After the upgrade, it started producing this error:

..\subprojects\xz-5.4.1\src\common\tuklib_integer.h(647,2): error: call to undeclared library function '_BitScanReverse' with type 'unsigned char (unsigned long *, unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        _BitScanReverse(&i, n);
        ^
..\subprojects\xz-5.4.1\src\common\tuklib_integer.h(647,2): note: include the header <intrin.h> or explicitly provide a declaration for '_BitScanReverse'
..\subprojects\xz-5.4.1\src\common\tuklib_integer.h(749,2): error: call to undeclared library function '_BitScanForward' with type 'unsigned char (unsigned long *, unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        _BitScanForward(&i, n);
        ^

See full log here:
https://ci.appveyor.com/project/rizinorg/rizin/builds/46815593/job/ny9b339jydox5pft#L5611

This is the wrap file we use: https://github.com/rizinorg/rizin/blob/dev/subprojects/liblzma.wrap
These are Meson build files: https://github.com/rizinorg/rizin/tree/dev/subprojects/packagefiles/xz-5.4.1

Version

5.4.1

Operating System

Windows Server 2019 (AppVeyor VS2019 image)

Relevant log output

No response

https://tukaani.org/xz-backdoor/review.html discusses the crc_attr_no_sanitize_address (i.e. __attribute__((__no_sanitize_address__))) in crc_x86_clmul.h:

This attribute is obviously scary but it is unfortunately required with the x86 SIMD code. The code makes aligned 16-byte reads which may read up to 15 bytes before the beginning or past the end of the buffer if the buffer is misaligned. The unneeded bytes are then ignored. It cannot cross page boundaries and thus cannot cause access violations.

Disabling sanitzers is indeed scary. But also, I don't think the disabling is required to use x86 SIMD.

Instead, you can add crc_simd_body preconditions that its buf and size arguments must be 16-byte aligned. Make the callers (crc32_arch_optimized and crc64_arch_optimized) responsible for the leading/lagging bytes that aren't 16-byte aligned. Out-of-bounds reads are no longer needed.

diff --git a/src/liblzma/check/crc_x86_clmul.h b/src/liblzma/check/crc_x86_clmul.h
index f1254ece..7ecd16ee 100644
--- a/src/liblzma/check/crc_x86_clmul.h
+++ b/src/liblzma/check/crc_x86_clmul.h
@@ -91,6 +91,9 @@ crc_simd_body(const uint8_t *buf, const size_t size, __m128i *v0, __m128i *v1,
        //     [skip_start][size][skip_end]
        //     [     size2      ]
        //
+       // This code can be simplified. skip_start and skip_end are now always 0,
+       // size is a positive multiple of 16 and size2 = size.
+       //
        // A and D are 16-byte aligned. B and C are 1-byte aligned.
        // skip_start and skip_end are 0-15 bytes. size is at least 1 byte.
        //
@@ -253,6 +256,15 @@ crc32_arch_optimized(const uint8_t *buf, size_t size, uint32_t crc)
                return crc;
 #endif
 
+       // Process any leading bytes (before the 16-byte chunks).
+       crc = ~crc;
+       while (15 & (uintptr_t)buf) {
+               if (size == 0)
+                       return ~crc;
+               crc = lzma_crc32_table[0][*buf++ ^ (crc & 0xFF)] ^ (crc >> 8);
+               --size;
+       }
+
        // uint32_t poly = 0xedb88320;
        const int64_t p = 0x1db710640; // p << 1
        const int64_t mu = 0x1f7011641; // calc_lo(p, p, 32) << 1 | 1
@@ -264,21 +276,34 @@ crc32_arch_optimized(const uint8_t *buf, size_t size, uint32_t crc)
        const __m128i vfold8 = _mm_set_epi64x(0, k5);
        const __m128i vfold16 = _mm_set_epi64x(k4, k3);
 
-       __m128i v0, v1, v2;
+       // Process 16-byte chunks that are 16-byte aligned.
+       size_t size1 = size & 15;
+       size_t size0 = size - size1;
+       if (size0) {
+               __m128i v0, v1, v2;
+
+               crc_simd_body(buf,  size, &v0, &v1, vfold16,
+                               _mm_cvtsi32_si128((int32_t)~crc));
+               buf += size0;
+
+               v1 = _mm_xor_si128(
+                               _mm_clmulepi64_si128(v0, vfold16, 0x10), v1); // xxx0
+               v2 = _mm_shuffle_epi32(v1, 0xe7); // 0xx0
+               v0 = _mm_slli_epi64(v1, 32);  // [0]
+               v0 = _mm_clmulepi64_si128(v0, vfold8, 0x00);
+               v0 = _mm_xor_si128(v0, v2);   // [1] [2]
+               v2 = _mm_clmulepi64_si128(v0, vfold4, 0x10);
+               v2 = _mm_clmulepi64_si128(v2, vfold4, 0x00);
+               v0 = _mm_xor_si128(v0, v2);   // [2]
+               crc = ~(uint32_t)_mm_extract_epi32(v0, 2);
+       }
 
-       crc_simd_body(buf,  size, &v0, &v1, vfold16,
-                       _mm_cvtsi32_si128((int32_t)~crc));
-
-       v1 = _mm_xor_si128(
-                       _mm_clmulepi64_si128(v0, vfold16, 0x10), v1); // xxx0
-       v2 = _mm_shuffle_epi32(v1, 0xe7); // 0xx0
-       v0 = _mm_slli_epi64(v1, 32);  // [0]
-       v0 = _mm_clmulepi64_si128(v0, vfold8, 0x00);
-       v0 = _mm_xor_si128(v0, v2);   // [1] [2]
-       v2 = _mm_clmulepi64_si128(v0, vfold4, 0x10);
-       v2 = _mm_clmulepi64_si128(v2, vfold4, 0x00);
-       v0 = _mm_xor_si128(v0, v2);   // [2]
-       return ~(uint32_t)_mm_extract_epi32(v0, 2);
+       // Process any lagging bytes (after the 16-byte chunks).
+       while (size1--) {
+               crc = lzma_crc32_table[0][*buf++ ^ (crc & 0xFF)] ^ (crc >> 8);
+       }
+
+       return ~crc;
 }
 #endif // BUILDING_CRC32_CLMUL
 
@@ -343,6 +368,8 @@ crc64_arch_optimized(const uint8_t *buf, size_t size, uint64_t crc)
                return crc;
 #endif
 
+       // Ditto.
+
        // const uint64_t poly = 0xc96c5795d7870f42; // CRC polynomial
        const uint64_t p  = 0x92d8af2baf0e1e85; // (poly << 1) | 1
        const uint64_t mu = 0x9c3e466c172963d5; // (calc_lo(poly) << 1) | 1

Describe the bug

After updating to v5.2.12 it seems --disable-assembler flag is not fully affecting liblzma

compiler is not standard but for this case it can be considered as clang, also tried v5.2.11 and it's working fine so I'm guessing it must be a configuration changes between these two versions

Version

v5.2.12

Operating System

docker/ubuntu:jammy

Relevant log output

22.71 make[3]: Entering directory '/opt/xz-5.2.12/src/liblzma'
22.71 /bin/bash ../../libtool  --tag=CC   --mode=compile /emsdk/upstream/emscripten/emcc -DHAVE_CONFIG_H -I. -I../..  -I../../src/liblzma/api -I../../src/liblzma/common -I../../src/liblzma/check -I../../src/liblzma/lz -I../../src/liblzma/rangecoder -I../../src/liblzma/lzma -I../../src/liblzma/delta -I../../src/liblzma/simple -I../../src/common -DTUKLIB_SYMBOL_PREFIX=lzma_ -I/usr/local/include/ -I/opt/zlib-1.3 -I/opt/bzip2-1.0.8 -I/opt/openssl-1.0.2s/include -I/opt/openssl-1.0.2s/test -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -g -O2 -MT liblzma_la-common.lo -MD -MP -MF .deps/liblzma_la-common.Tpo -c -o liblzma_la-common.lo `test -f 'common/common.c' || echo './'`common/common.c
22.76 libtool: compile:  /emsdk/upstream/emscripten/emcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/liblzma/api -I../../src/liblzma/common -I../../src/liblzma/check -I../../src/liblzma/lz -I../../src/liblzma/rangecoder -I../../src/liblzma/lzma -I../../src/liblzma/delta -I../../src/liblzma/simple -I../../src/common -DTUKLIB_SYMBOL_PREFIX=lzma_ -I/usr/local/include/ -I/opt/zlib-1.3 -I/opt/bzip2-1.0.8 -I/opt/openssl-1.0.2s/include -I/opt/openssl-1.0.2s/test -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -g -O2 -MT liblzma_la-common.lo -MD -MP -MF .deps/liblzma_la-common.Tpo -c common/common.c  -fPIC -DPIC -o .libs/liblzma_la-common.o
22.87 <inline asm>:1:1: error: unknown directive
22.87     1 | .symver lzma_get_progress_522,lzma_get_progress@XZ_5.2.2
22.87       | ^
22.87 <inline asm>:2:1: error: unknown directive
22.87     2 | .symver lzma_get_progress_52,lzma_get_progress@@XZ_5.2
22.87       | ^
22.89 2 errors generated.

Describe the bug

Not necessarily a bug, but the behavior is confusing. lzma_str_to_filters() is successful with the filter string "lzma1" when LZMA_STR_ALL_FILTERS flag is used, as expected. However, lzma_str_to_filters() fails with "lzma1" when LZMA_STR_NO_VALIDATION. This is confusing because the "validation" is too vague. Checking if a filter can be used with the .xz format is technically validation. So, either the code should change or the documentation should be updated to clarify this.

Version

5.4.1

Operating System

Ubuntu

Relevant log output

No response

https://nvd.nist.gov/vuln/detail/CVE-2020-22916

This link seems to be inaccessible：https://github.com/snappyJack/CVE-request-XZ-5.2.5-has-denial-of-service-vulnerability

Is there a fix for CVE xz?
[1] If not, what is the repair plan for xz?
[2] If yes, can you indicate which submissions fix CVE-2020-22916?

Will be installed on Windows x86-64, if it makes any difference.

Describe the bug

Recently, changes were made to the error handling code in the xz utility's libarchive project (reference to the specific MR or commit). While the intention behind these changes may have been to improve error reporting, there are concerns about potential security implications, especially in light of the recent discovery of a backdoor in the xz utility.

Upon reviewing the changes, it appears that safe fprintf calls were replaced with unsafe fprintf calls in the error handling code. This alteration raises concerns about the possibility of introducing vulnerabilities, particularly if terminal escape sequences or other forms of exploitation are inadvertently included in error messages.

Given the sensitivity of the situation and the potential for similar vulnerabilities, I believe it's crucial for the project maintainers to thoroughly assess these changes and ensure that they do not inadvertently introduce security risks. Additionally, considering the recent backdoor discovery, it's important to prioritize the security and integrity of the xz utility.

Proposed Actions:

Revert the changes made to the error handling code, restoring the use of safe fprintf calls.
Use archive_errno instead of errno for error handling to mitigate potential risks associated with including terminal escape sequences in error messages.
Conduct a thorough review of the codebase to identify and address any other potential security concerns.

Additional Context:

Link to the relevant MR or commit where the changes were made.
Reference to any discussions or concerns raised by other contributors or community members.
Any other relevant information or insights that could aid in understanding the issue and its implications.

Request for Response:
I kindly request a response from the project maintainers regarding these security concerns and proposed actions. Transparency and collaboration are essential in addressing potential security vulnerabilities and ensuring the continued trust and reliability of the xz utility.

Thank you for your attention to this matter.

Version

5.6.0/5.6.1

Operating System

Linux

Relevant log output

No response

Describe the bug

The following tests have been reported to fail on Solaris 10:

• FAIL: test_compress_prepared_bcj_sparc
• FAIL: test_compress_prepared_bcj_x86
• FAIL: test_compress_generated_abc
• FAIL: test_compress_generated_random
• FAIL: test_compress_generated_text

Thanks to both İhsan Doğan and Dennis Clarke for reporting this.

Version

xz and liblzma 5.4.0

Operating System

Solaris 10

Relevant log output

Log file provided by İhsan Doğan <[email protected]> 

cat work/solaris10-sparc/build-isa-sparcv8plus/xz-5.4.0/tests/test-suite.log 
==========================================
   XZ Utils 5.4.0: tests/test-suite.log
==========================================

# TOTAL: 16
# PASS:  11
# SKIP:  0
# XFAIL: 0
# FAIL:  5
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test_compress_prepared_bcj_sparc
======================================

./test_compress.sh: bad substitution
FAIL test_compress_prepared_bcj_sparc (exit status: 1)

FAIL: test_compress_prepared_bcj_x86
====================================

./test_compress.sh: bad substitution
FAIL test_compress_prepared_bcj_x86 (exit status: 1)

FAIL: test_compress_generated_abc
=================================

./test_compress.sh: bad substitution
FAIL test_compress_generated_abc (exit status: 1)

FAIL: test_compress_generated_random
====================================

./test_compress.sh: bad substitution
FAIL test_compress_generated_random (exit status: 1)

FAIL: test_compress_generated_text
==================================

./test_compress.sh: bad substitution
FAIL test_compress_generated_text (exit status: 1)

Describe the bug

Our Gentoo Tinderbox reported a test failure at bug 914170

By looking at test-suite.log I can see:

==1161==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5614ecd418a0 at pc 0x7f8d20216905 bp 0x7ffd6a482040 sp 0x7ffd6a482038
READ of size 16 at 0x5614ecd418a0 thread T0
    #0 0x7f8d20216904 in crc64_clmul /var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4/src/liblzma/check/crc64_fast.c:284:40
    #1 0x5614ecd2a53c in test_lzma_crc64 /var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4/tests/test_check.c:106:2
    #2 0x5614ecd2a237 in tuktest_run_test /var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4/tests/tuktest.h:596:4
    #3 0x5614ecd29cec in main /var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4/tests/test_check.c:383:2
    #4 0x7f8d1fe23c89  (/lib64/libc.so.6+0x23c89)
    #5 0x7f8d1fe23d44 in __libc_start_main (/lib64/libc.so.6+0x23d44)
    #6 0x5614ecc553f0 in _start (/var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4-abi_x86_64.amd64/tests/.libs/test_check+0x203f0)

0x5614ecd418a0 is located 32 bytes before global variable '.str.42' defined in '/var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4/tests/test_check.c:78' (0x5614ecd418c0) of size 51
  '.str.42' is ascii string 'assert_uint: '%s == %lu' but expected '... %s %lu''
0x5614ecd418a9 is located 0 bytes after global variable 'test_string' defined in '/var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4/tests/test_check.c:22' (0x5614ecd418a0) of size 9
SUMMARY: AddressSanitizer: global-buffer-overflow /var/tmp/portage/app-arch/xz-utils-5.4.4/work/xz-5.4.4/src/liblzma/check/crc64_fast.c:284:40 in crc64_clmul
Shadow bytes around the buggy address:
  0x5614ecd41600: 07 f9 f9 f9 f9 f9 f9 f9 00 00 00 05 f9 f9 f9 f9
  0x5614ecd41680: 00 00 00 03 f9 f9 f9 f9 00 00 00 00 00 00 00 04
  0x5614ecd41700: f9 f9 f9 f9 06 f9 f9 f9 06 f9 f9 f9 07 f9 f9 f9
  0x5614ecd41780: 00 07 f9 f9 00 04 f9 f9 00 02 f9 f9 00 00 00 00
  0x5614ecd41800: 00 06 f9 f9 f9 f9 f9 f9 00 00 00 00 00 02 f9 f9
=>0x5614ecd41880: f9 f9 f9 f9[00]01 f9 f9 00 00 00 00 00 00 03 f9
  0x5614ecd41900: f9 f9 f9 f9 00 00 00 00 00 00 f9 f9 f9 f9 f9 f9
  0x5614ecd41980: 03 f9 f9 f9 00 04 f9 f9 00 00 00 00 00 00 07 f9
  0x5614ecd41a00: f9 f9 f9 f9 04 f9 f9 f9 00 00 00 02 f9 f9 f9 f9
  0x5614ecd41a80: 00 00 00 00 00 00 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x5614ecd41b00: 00 00 07 f9 f9 f9 f9 f9 00 00 00 03 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1161==ABORTING

I didn't look deeply into this issue so I don't know if the bug is in the unittest itself or in the involed libraries/daemons, if so please check for any security implications. I'm not doing a private report since it is already public on gentoo bugzilla.
If I can do further, please let me know.

Version

5.4.4

Operating System

Gentoo

Relevant log output

No response

Hello,

Have you changed your key on https://tukaani.org/misc/lasse_collin_pubkey.txt ?

>>> Emerging (1 of 2) sec-keys/openpgp-keys-lassecollin-20230213::gentoo
 * Fetching files in the background.
 * To view fetch progress, run in another terminal:
 * tail -f /var/log/emerge-fetch.log
>>> Downloading 'https://mirror.isoc.org.il/pub/gentoo/distfiles/layout.conf'
--2024-04-16 03:45:15--  https://mirror.isoc.org.il/pub/gentoo/distfiles/layout.conf
Resolving mirror.isoc.org.il (mirror.isoc.org.il)... 2a01:4280:2:20::2, 192.115.2.70
Connecting to mirror.isoc.org.il (mirror.isoc.org.il)|2a01:4280:2:20::2|:443... connected.
ERROR: cannot verify mirror.isoc.org.il's certificate, issued by ‘CN=ZeroSSL RSA Domain Secure Site CA,O=ZeroSSL,C=AT’:
  Issued certificate has expired.
To connect to mirror.isoc.org.il insecurely, use `--no-check-certificate'.
!!! Couldn't download '.layout.conf.mirror.isoc.org.il'. Aborting.
>>> Downloading 'https://mirror.isoc.org.il/pub/gentoo/distfiles/openpgp-keys-lassecollin-20230213-lasse_collin_pubkey.txt'
--2024-04-16 03:45:17--  https://mirror.isoc.org.il/pub/gentoo/distfiles/openpgp-keys-lassecollin-20230213-lasse_collin_pubkey.txt
Resolving mirror.isoc.org.il (mirror.isoc.org.il)... 2a01:4280:2:20::2, 192.115.2.70
Connecting to mirror.isoc.org.il (mirror.isoc.org.il)|2a01:4280:2:20::2|:443... connected.
ERROR: cannot verify mirror.isoc.org.il's certificate, issued by ‘CN=ZeroSSL RSA Domain Secure Site CA,O=ZeroSSL,C=AT’:
  Issued certificate has expired.
To connect to mirror.isoc.org.il insecurely, use `--no-check-certificate'.
>>> Downloading 'https://tukaani.org/misc/lasse_collin_pubkey.txt'
--2024-04-16 03:45:19--  https://tukaani.org/misc/lasse_collin_pubkey.txt
Resolving tukaani.org (tukaani.org)... 5.44.245.25
Connecting to tukaani.org (tukaani.org)|5.44.245.25|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3151 (3.1K) [text/plain]
Saving to: ‘/usr/portage/distfiles/openpgp-keys-lassecollin-20230213-lasse_collin_pubkey.txt.__download__’

/usr/portage/distfi 100%[===================>]   3.08K  --.-KB/s    in 0s      

2024-04-16 03:45:20 (1.43 GB/s) - ‘/usr/portage/distfiles/openpgp-keys-lassecollin-20230213-lasse_collin_pubkey.txt.__download__’ saved [3151/3151]

!!! Fetched file: openpgp-keys-lassecollin-20230213-lasse_collin_pubkey.txt VERIFY FAILED!
!!! Reason: Filesize does not match recorded size
!!! Got:      3151
!!! Expected: 4658
Refetching... File renamed to '/usr/portage/distfiles/openpgp-keys-lassecollin-20230213-lasse_collin_pubkey.txt._checksum_failure_.hi7__t65'

!!! Couldn't download 'openpgp-keys-lassecollin-20230213-lasse_collin_pubkey.txt'. Aborting.
 * Fetch failed for 'sec-keys/openpgp-keys-lassecollin-20230213', Log file:
 *  '/var/tmp/portage/sec-keys/openpgp-keys-lassecollin-20230213/temp/build.log'

Dear @Larhzu,

Several people would like that you do new releases (to remove the username of the backdoor author) and good asset files:

• https://github.com/tukaani-project/xz/releases

And maybe good to remove the 5.6.x tags too?

• https://github.com/tukaani-project/xz/tags

Note: Same in primary git source place.

Thanks in advance.

Well. Not really a bug, but perhaps more a question of default and helpful information.
Please correct me if I'm wrong here.
I've always lived with the assumption that using xz with more than one thread isn't deterministic.
Ie, the compression result will vary with threads?
By changing the default, various usages will start to perhaps experience different results when building on different machines.
Like packagers, embedded build envs etc.

Has anything changed in the determinism department or can users just expect that default is to create variable results with the defaults?

Version

5.6

Operating System

Linux

Relevant log output

No response

Describe the Feature

A Security Policy should provide clear guidance on how to report potential vulnerabilities and inform the vulnerabilities disclosure window for this repo. It recommended by Github and Scorecard.

Expected Complications

No.

Will I try to implement this new feature?

Yes

Additional context

I'm Gabriela and I work on behalf of Google and the OpenSSF suggesting supply-chain security changes :)

If you agree, I can open a PR to suggest a Security Policy, and we can work together to communicate how the repo can best handle vulnerability reports.

Describe the bug

Reference: INSTALL.generic | ./configure is present as a configuration instruction.

Hello, In Bash Shell:

$ ./configure
bash: ./configure: No such file or directory
$ ls | grep -i configure
configure.ac

Version

5.4.5; as source code (tar.gz)

Operating System

Fedora; edition 39

Relevant log output

No response

Could release-pipeline and test-pipeline be separated? I mean release-pipeline should not be complicated or run any extra step, and test-pipeline should not release any artifact. Just to make easier to review release,

Also, maybe it is good to ban binary blobs. These blobs could be generated in the pipeline step or it should be mandatory to share steps to create these with the commit so it can be reviewed.

A backdoor was discovered in xz-5.6.0 and xz-5.6.1 by Andres Freund.

Updates are being posted at https://tukaani.org/xz-backdoor/ and the links at the bottom of the page.

Please be patient while everything is cleaned up. It will take time. In particular, please avoid speculation here which would be better placed in other venues.

Describe the bug

If --format is set to anything other than raw, xz will only print the list information, or an error (--list works only on .xz files (--format=xz or --format=auto)). When --format=raw, xz will print:
Filter chain: --lzma2=dict=8MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 [or the actual filter chain specified]
9 MiB of memory is required. The limiter is disabled.
--list works only on .xz files (--format=xz or --format=auto)

This is not a critical bug, but it is a small inconsistency.

Version

master @ 8d372bd

Operating System

Ubuntu

Relevant log output

./src/xz/xz --format=raw --suffix=.xz --lzma2 -lvv  config.h.xz
xz: Filter chain: --lzma2=dict=8MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: 9 MiB of memory is required. The limiter is disabled.
xz: --list works only on .xz files (--format=xz or --format=auto)

-----------------------------------------------

./src/xz/xz --format=lzma --suffix=.xz --lzma2 -lvv  config.h.xz
xz: --list works only on .xz files (--format=xz or --format=auto)

-----------------------------------------------

./src/xz/xz --format=xz --suffix=.xz --lzma2 -lvv  config.h.xz
config.h.xz (1/1)
  Streams:            1
  Blocks:             1
  Compressed size:    4,044 B
  Uncompressed size:  16.5 KiB (16,852 B)
  Ratio:              0.240
  Check:              CRC64
  Stream padding:     0 B
  Streams:
    Stream    Blocks      CompOffset    UncompOffset        CompSize      UncompSize  Ratio  Check      Padding
         1         1               0               0           4,044          16,852  0.240  CRC64            0
  Blocks:
    Stream     Block      CompOffset    UncompOffset       TotalSize      UncompSize  Ratio  Check      CheckVal          Header  Flags        CompSize    MemUsage  Filters
         1         1              12               0           4,008          16,852  0.238  CRC64      b4fdba2a9a1834bb      12  --              3,987       9 MiB  --lzma2=dict=8MiB
  Memory needed:      9 MiB
  Sizes in headers:   No
  Minimum XZ Utils version: 5.0.0

Describe the bug

localhost:~/llvm_18$ release_BUILD/LIBXZ/OUT/bin/lzma
Error relocating release_BUILD/LIBXZ/BUILD/lzma: unsupported relocation type 37
Error relocating release_BUILD/LIBXZ/BUILD/lzma: unsupported relocation type 37

if (NOT EXISTS ${CMAKE_BINARY_DIR}/LIBXZ)
  execute_process(
    COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_CURRENT_SOURCE_DIR}/xz ${CMAKE_BINARY_DIR}/LIBXZ
  )
endif()

if (NOT EXISTS ${CMAKE_BINARY_DIR}/LIBXZ/OUT/lib/liblzma.a)
  execute_process(
    COMMAND
      ${CMAKE_COMMAND}
      -D CMAKE_BUILD_TYPE=Release
      -D "CMAKE_C_FLAGS=${CMAKE_C_FLAGS}"
      -D "CMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}"
      -D CMAKE_C_COMPILER=${CMAKE_C_COMPILER}
      -D CMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
      -D CMAKE_INSTALL_PREFIX:PATH=${CMAKE_BINARY_DIR}/LIBXZ/OUT
      -D BUILD_SHARED_LIBS=OFF
      -B ${CMAKE_BINARY_DIR}/LIBXZ/BUILD
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/LIBXZ
    COMMAND_ECHO STDOUT
  )
  execute_process(
    COMMAND
      ${CMAKE_COMMAND}
      --build ${CMAKE_BINARY_DIR}/LIBXZ/BUILD
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/LIBXZ
    COMMAND_ECHO STDOUT
  )
  execute_process(
    COMMAND
      ${CMAKE_COMMAND}
      --install ${CMAKE_BINARY_DIR}/LIBXZ/BUILD
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/LIBXZ
    COMMAND_ECHO STDOUT
  )
endif()

Version

5.5.0 git master 837ea40

Operating System

alpine linux

Relevant log output

No response

Describe the bug

When given empty buffers with next_in == NULL and avail_in == 0 (or same for out), liblzma sometimes computes NULL + 0, which has undefined behavior in C (it is defined in C++).

This gets detected by ubsan.

The following places are affected (possibly more):

Version

5.4.1

Operating System

Linux

Relevant log output

No response

Describe the bug

If LZMA_MF_HC3 or LZMA_MF_HC4 are disabled for preset level <=3, or if LZMA_MF_BT4 is disabled for the other levels, then lzma_lzma_preset still returns success. When lzma_options_lzma * is used, it will cause an error.

The solution to this can be done either in code or in documentation. The code change would be simple to check if the needed match finders were configured.

Instead, a documentation change in lzma12.h could make it clear that the preset is valid, but might not work depending on the liblzma configuration (in this case, which match finders were built).

Version

5.5.0alpha

Operating System

N/A

Relevant log output

No response

Describe the bug

Fails to build on armv7l. built with --enable-shared

Version

5.6.0

Operating System

ChromeOS, M91

Relevant log output

make  all-recursive
make[1]: Entering directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0'
Making all in src
make[2]: Entering directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src'
Making all in liblzma
make[3]: Entering directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/liblzma'
Making all in api
make[4]: Entering directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/liblzma/api'
make[4]: Nothing to be done for 'all'.
make[4]: Leaving directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/liblzma/api'
make[4]: Entering directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/liblzma'
make[4]: Nothing to be done for 'all-am'.
make[4]: Leaving directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/liblzma'
make[3]: Leaving directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/liblzma'
Making all in xzdec
make[3]: Entering directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/xzdec'
armv7l-cros-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I../..  -DTUKLIB_GETTEXT=0 -I../../src/common -I../../src/liblzma/api -I../../lib  -pthread -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wshift-overflow=2 -Wstrict-overflow=3 -Walloc-zero -Wduplicated-cond -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wdate-time -Wsign-conversion -Wfloat-conversion -Wlogical-op -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls -O2 -pipe -ffat-lto-objects -fPIC -mfloat-abi=hard -mthumb -mfpu=vfpv3-d16 -march=armv7-a+fp -fuse-ld=mold  -flto=auto -c -o xzdec-xzdec.o `test -f 'xzdec.c' || echo './'`xzdec.c
xzdec.c: In function ‘sandbox_enter’:
xzdec.c:329:36: error: ‘SYS_landlock_create_ruleset’ undeclared (first use in this function)
  329 |         int landlock_abi = syscall(SYS_landlock_create_ruleset,
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~
xzdec.c:329:36: note: each undeclared identifier is reported only once for each function it appears in
xzdec.c:349:29: error: ‘SYS_landlock_restrict_self’ undeclared (first use in this function)
  349 |                 if (syscall(SYS_landlock_restrict_self, ruleset_fd, 0U) != 0)
      |                             ^~~~~~~~~~~~~~~~~~~~~~~~~~
make[3]: *** [Makefile:610: xzdec-xzdec.o] Error 1
make[3]: Leaving directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src/xzdec'
make[2]: *** [Makefile:427: all-recursive] Error 1
make[2]: Leaving directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0/src'
make[1]: *** [Makefile:597: all-recursive] Error 1
make[1]: Leaving directory '/usr/local/tmp/crew/xzutils.20240224160935.dir/xz-5.6.0'
make: *** [Makefile:487: all] Error 2
There was a build error.

Describe the bug

make  all-recursive
make[1]: Entering directory '/Depot/jjj/xz-5.6.0'
Making all in src
make[2]: Entering directory '/Depot/jjj/xz-5.6.0/src'
Making all in liblzma
make[3]: Entering directory '/Depot/jjj/xz-5.6.0/src/liblzma'
Making all in api
make[4]: Entering directory '/Depot/jjj/xz-5.6.0/src/liblzma/api'
make[4]: Nothing to be done for 'all'.
make[4]: Leaving directory '/Depot/jjj/xz-5.6.0/src/liblzma/api'
make[4]: Entering directory '/Depot/jjj/xz-5.6.0/src/liblzma'
make[4]: Nothing to be done for 'all-am'.
make[4]: Leaving directory '/Depot/jjj/xz-5.6.0/src/liblzma'
make[3]: Leaving directory '/Depot/jjj/xz-5.6.0/src/liblzma'
Making all in xzdec
make[3]: Entering directory '/Depot/jjj/xz-5.6.0/src/xzdec'
gcc -DHAVE_CONFIG_H -I. -I../..  -DTUKLIB_GETTEXT=0 -I../../src/common -I../../src/liblzma/api -I../../lib  -pthread -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wshift-overflow=2 -Wstrict-overflow=3 -Walloc-zero -Wduplicated-cond -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wdate-time -Wsign-conversion -Wfloat-conversion -Wlogical-op -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls -O2 -fPIC -fno-strict-overflow -Wno-error -MT xzdec-xzdec.o -MD -MP -MF .deps/xzdec-xzdec.Tpo -c -o xzdec-xzdec.o `test -f 'xzdec.c'     || echo './'`xzdec.c
xzdec.c: In function ‘sandbox_enter’:
xzdec.c:329:36: error: ‘SYS_landlock_create_ruleset’ undeclared (first use in this function)
  329 |         int landlock_abi = syscall(SYS_landlock_create_ruleset,
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~
xzdec.c:329:36: note: each undeclared identifier is reported only once for each function it appears in
xzdec.c:349:29: error: ‘SYS_landlock_restrict_self’ undeclared (first use in this function)
  349 |                 if (syscall(SYS_landlock_restrict_self, ruleset_fd, 0U) != 0)
      |                             ^~~~~~~~~~~~~~~~~~~~~~~~~~
make[3]: *** [Makefile:606: xzdec-xzdec.o] Error 1
make[3]: Leaving directory '/Depot/jjj/xz-5.6.0/src/xzdec'
make[2]: *** [Makefile:427: all-recursive] Error 1
make[2]: Leaving directory '/Depot/jjj/xz-5.6.0/src'
make[1]: *** [Makefile:597: all-recursive] Error 1
make[1]: Leaving directory '/Depot/jjj/xz-5.6.0'
make: *** [Makefile:487: all] Error 2

I can compile older xz releases fine by the way, on the same system, so 5.4.x,
5.2.x. Not sure what changed.

My system is a custom built slackware; I recompiled most
things from source following mostly LFS/BLFS instructions.
xz 5.6.x is not yet appearing on LFS, so I just poke in the
dark here really.

glibc is 2.33, gcc is 11.4.0.

Version

5.6.0

Operating System

linux

Relevant log output

No response

Describe the Feature

A makefile target that downloads adequate data to run a real-world benchmark (compression and decompression). Or something similar.

Expected Complications

No response

Will I try to implement this new feature?

No

basically 5.6.1 minus backdoor

I understand why the author(s) of the analysis of the backdoor being distributed by this project decided not to notify upstream first since it looks like either the upstream is the compromise or at least are compromised themselves, so reporting here first would have done nothing except give the culprit time to wiggle. But the cat is out of the bag now and I see comments all over the place pinging the author. I think it's high time a bug report here notifies people following this tracker there is an issue. Also this serves as a request for a postmortem. Either this project has members that are bad actors or the members have themselves been deeply compromised. Most evidence seems to support the former. If there is evidence of the latter I suggest getting it out there post-haste.

With the recent attention, I'm sure plenty of people will be willing to donate a few dollars to help support you.

Please enable the "Sponsor" feature on your repo : https://docs.github.com/en/sponsors/sponsoring-open-source-contributors/sponsoring-an-open-source-contributor-through-github

Describe the Feature

Hello. The program acquired from this repository and installed works as intended. Nonetheless I'd rather keep the benefit of having a release installed as packaged by the Linux distribution I am using in order to obtain its automatic updates. However, the uninstallation is uncovered in theINSTALL.generic file. This could be rectified by an addition either of a methodology into this file or script dedicated for that purpose.

Expected Complications

No response

Will I try to implement this new feature?

Yes

Describe the Feature

Both MinGw and CMake Windows builds should be tested.

Expected Complications

Downloading the correct build environment dependencies could be tricky. GitHub might have some reusable components to help with this.

Will I try to implement this new feature?

Yes

Short version

Public domain has (real or perceived) legal issues in some jurisdictions. To avoid those issues, a widely-accepted public-domain-equivalent license that requires no attribution is considered for future versions. Feedback is wanted to know if this idea is good.

Why the XZ projects are in the public domain

LZMA SDK used to be available under the GNU LGPL and a few other license choices.
In late 2008, LZMA SDK became public domain (PD). Since XZ Utils, XZ Embedded, and XZ for Java are derived from the LZMA SDK code, I felt that it made no sense to have more restrictions on the XZ projects code than what LZMA SDK had. Thus the XZ projects have been PD as well.

For example, the MIT License, BSD 2-Clause "Simplified" License, or ISC License would have been more restrictive than PD as those require keeping the copyright notice and license notice when distributing copies. Pure PD has no such requirements.

I knew even in 2008 that PD might be legally complicated in some jurisdictions but I didn't see ideal alternatives (a common advice was to not create a new license). In practice it seemed that PD projects were accepted in major distros with strict policies (like Debian and Fedora). A few discussions around the problems of PD have occurred over the years though and in general it's waste of everyone's time.

Public-domain-equivalent licenses

In 2008 there were no widely-recognized and widely-accepted public-domain-equivalent licenses. Nowadays there are at least BSD Zero Clause License (0BSD) and MIT No Attribution license (MIT-0) which look good.

A few other PD-equivalent licenses exist too. For example:

• WTFPL uses unprofessional language (and lacks a warranty disclaimer in case it matters).

• CC0 is also a PD dedication with a fallback license. It explicitly lacks patent license which has made Fedora consider it unacceptable for free software in 2022 (still fine for files other than code).

• The Unlicense is a PD dedication with a fallback license. Compared to 0BSD and MIT-0, a PD dedication doesn't seem to add much practical value but it can fuel discussions about how the PD-dedication, fallback license, and warranty disclaimer interact with each other and so on.

Both 0BSD and MIT-0 are simple modifications to existing license texts and thus more than one person might have created the same variants independently over the years. The difficult thing is making the license texts widely accepted. 0BSD came from toybox which is included in Android. MIT-0 got visibility because Amazon uses it for example code. 0BSD is slightly shorter than MIT-0 while both should have the same legal effect. 0BSD got wide recognition a little earlier:

• 0BSD in SDPX license list v2.2 on 2015-09-30 and OSI approval on 2015-10-14

• MIT-0 in SDPX license list v3.1 in 2018 and OSI approval in 2020

GitHub has over four times 0BSD-licensed repositories than MIT-0-licensed repositories.

Google doesn't allow contributions to PD projects or certain PD-equivalent projects: WTFPL, CC0, and The Unlicense are explicitly mentioned as prohibited. However, 0BSD is explicitly listed as allowed. (It's spelled as BSD0 on that page but it links to 0BSD.) MIT-0 isn't mentioned at all.

So clearly the legal department in one large company is happy with 0BSD. And since Android contains 0BSD-licensed code, a few other companies must be OK with 0BSD too. Even Microsoft has released code under 0BSD.

With the above considerations, my impression is that 0BSD is currently be the best of the well-known public-domain-equivalent licenses for software although the difference to MIT-0 is minuscule. The reasons to prefer 0BSD are that it's more popular, clearly accepted by legal departments of more than one large company, and that it's slightly shorter while being equivalent in legal meaning.

The Plan

The change would affect future versions only. The public domain code in the old releases would obviously remain in the public domain.

For simplicity, all currently-PD code would be marked as 0BSD (with per-file SPDX license identifiers). If one wishes to know which parts are available as PD, one can look at the old releases or Git commit history. Those aren't going away.

We have already asked a few authors that they indeed are OK that their code would be under 0BSD. It feels a bit silly to ask since PD allows this already, but this way only a fairly small amount of code will rely solely on PD dedications. In any case, some code would remain that is from PD-only source.

Switching to 0BSD should affect users and distributors very little. PD code can be distributed without any notices about where the code came from. The same can be done with 0BSD.

(There is one tiny difference still: With 0BSD-licensed code, one shouldn't claim that it is in the public domain. Code under 0BSD is copyrighted, thus it's not literally in the public domain. Example: It is fine to take PD code and re-release it under The Unlicense. But one cannot re-release someone else's 0BSD-licensed code under The Unlicense because The Unlicense has a PD dedication and one cannot dedicate code into PD if one isn't its copyright holder. 0BSD to MIT-0 should be fine because there is no PD dedication in MIT-0.)

Questions

1. Would using 0BSD instead of PD make legal considerations easier for distributors and contributors, even if some code still relies on PD dedications?

2. Should the XZ projects switch from PD to 0BSD for future releases?

When answering, if you represent an organization, please mention it. Otherwise the answer is assumed be an opinion of a private person.

Thank you!

how stable is cmake support in xz ?

as i plan to build for windows via cmake and add xz as a library in my project

Describe the bug

Building the multilib package for Arch Linux (https://archlinux.org/packages/multilib/x86_64/lib32-xz/) fails. The lines producing the error did not change recently, but the version before (5.4.1) built without problems (and still builds with current toolchain).

Version

latest

Operating System

Arch Linux

Relevant log output

In file included from lz/lz_encoder_mf.c:16:
../../src/liblzma/common/memcmplen.h: In function 'lzma_memcmplen':
../../src/liblzma/common/memcmplen.h:92:36: error: conversion to 'uint32_t' {aka 'const unsigned int'} from 'int' may change the sign of the result [-Werror=sign-conversion]
   92 |                 const uint32_t x = 0xFFFF ^ _mm_movemask_epi8(_mm_cmpeq_epi8(
      |                                    ^~~~~~

Describe the Feature

Restrict access in your workflow jobs to your repository. This is preventive action to keep your workflows safe. The ci.yml workflow jobs need just minimum permissions contents: read, and the default GitHub permissions are higher.

This is considered good-practice and is recommended by GitHub itself and other security tools, such as Scorecards and StepSecurity.

Additional Context

I'm Gabriela and I work on behalf of Google and the OpenSSF suggesting supply-chain security changes :)

Expected Complications

No.

Will I try to implement this new feature?

Yes

Describe the bug

With a plain ./configure I get:

./src/liblzma/.libs/liblzma.a
./src/liblzma/.libs/liblzma.so.5.4.2
./src/liblzma/.libs/liblzma.so.5
./src/liblzma/.libs/liblzma.so

but as soon as I add --disable-threads the .so.5.4.2 file and links are missing:

./src/liblzma/.libs/liblzma.a

Is this the intended behaviour?

Version

5.4.2

Operating System

Linux AMD64

Relevant log output

No response

Describe the bug

When building with the mold linker and -flto added to CFLAGS, this package seems to have some issues in the linking phase regarding symbols not being found. I've turned mold's warnings into errors with -Wl,--fatal-warnings, so I can catch them more easily.

GCC version used is gcc (Gentoo 13.1.1_p20230527 p3) 13.1.1 20230527
Mold is built from git at commit b04aba89d3a1931470983212925443e7aefca1e1

Steps to reproduce:

1. Clone this repo and cd into the xz folder
2. run export CFLAGS="-O2 -pipe -flto=auto" and export LDFLAGS="-fuse-ld=mold -Wl,--fatal-warnings"
3. run autoreconf -vfi
4. run ./configure --enable-threads --prefix=/usr
5. build the package with make V=1

In the log output I am only going to only add the last few lines of the log as I think the entire log would be way too much, and then add the full logs from ./configure and make as an attachment

xz-configure.log
xz-make.log

Version

commit 66bdcfa

Operating System

Gentoo Linux

Relevant log output

libtool: link: gcc -shared  -fPIC -DPIC  .libs/liblzma_la-tuklib_physmem.o .libs/liblzma_la-tuklib_cpucores.o .libs/liblzma_la-common.o .libs/liblzma_la-block_util.o .libs/liblzma_la-easy_preset.o .libs/liblzma_la-filter_common.o .libs/liblzma_la-hardware_physmem.o .libs/liblzma_la-index.o .libs/liblzma_la-stream_flags_common.o .libs/liblzma_la-string_conversion.o .libs/liblzma_la-vli_size.o .libs/liblzma_la-hardware_cputhreads.o .libs/liblzma_la-outqueue.o .libs/liblzma_la-alone_encoder.o .libs/liblzma_la-block_buffer_encoder.o .libs/liblzma_la-block_encoder.o .libs/liblzma_la-block_header_encoder.o .libs/liblzma_la-easy_buffer_encoder.o .libs/liblzma_la-easy_encoder.o .libs/liblzma_la-easy_encoder_memusage.o .libs/liblzma_la-filter_buffer_encoder.o .libs/liblzma_la-filter_encoder.o .libs/liblzma_la-filter_flags_encoder.o .libs/liblzma_la-index_encoder.o .libs/liblzma_la-stream_buffer_encoder.o .libs/liblzma_la-stream_encoder.o .libs/liblzma_la-stream_flags_encoder.o .libs/liblzma_la-vli_encoder.o .libs/liblzma_la-stream_encoder_mt.o .libs/liblzma_la-microlzma_encoder.o .libs/liblzma_la-alone_decoder.o .libs/liblzma_la-auto_decoder.o .libs/liblzma_la-block_buffer_decoder.o .libs/liblzma_la-block_decoder.o .libs/liblzma_la-block_header_decoder.o .libs/liblzma_la-easy_decoder_memusage.o .libs/liblzma_la-file_info.o .libs/liblzma_la-filter_buffer_decoder.o .libs/liblzma_la-filter_decoder.o .libs/liblzma_la-filter_flags_decoder.o .libs/liblzma_la-index_decoder.o .libs/liblzma_la-index_hash.o .libs/liblzma_la-stream_buffer_decoder.o .libs/liblzma_la-stream_decoder.o .libs/liblzma_la-stream_flags_decoder.o .libs/liblzma_la-vli_decoder.o .libs/liblzma_la-stream_decoder_mt.o .libs/liblzma_la-microlzma_decoder.o .libs/liblzma_la-lzip_decoder.o .libs/liblzma_la-check.o .libs/liblzma_la-crc32_table.o .libs/liblzma_la-crc32_fast.o .libs/liblzma_la-crc64_table.o .libs/liblzma_la-crc64_fast.o .libs/liblzma_la-sha256.o .libs/liblzma_la-lz_encoder.o .libs/liblzma_la-lz_encoder_mf.o .libs/liblzma_la-lz_decoder.o .libs/liblzma_la-lzma_encoder_presets.o .libs/liblzma_la-lzma_encoder.o .libs/liblzma_la-lzma_encoder_optimum_fast.o .libs/liblzma_la-lzma_encoder_optimum_normal.o .libs/liblzma_la-fastpos_table.o .libs/liblzma_la-lzma_decoder.o .libs/liblzma_la-lzma2_encoder.o .libs/liblzma_la-lzma2_decoder.o .libs/liblzma_la-price_table.o .libs/liblzma_la-delta_common.o .libs/liblzma_la-delta_encoder.o .libs/liblzma_la-delta_decoder.o .libs/liblzma_la-simple_coder.o .libs/liblzma_la-simple_encoder.o .libs/liblzma_la-simple_decoder.o .libs/liblzma_la-x86.o .libs/liblzma_la-powerpc.o .libs/liblzma_la-ia64.o .libs/liblzma_la-arm.o .libs/liblzma_la-armthumb.o .libs/liblzma_la-arm64.o .libs/liblzma_la-sparc.o   -lpthread  -O2 -flto=auto -Wl,--version-script=../../src/liblzma/liblzma_linux.map -fuse-ld=mold -Wl,--fatal-warnings   -pthread -Wl,-soname -Wl,liblzma.so.5 -o .libs/liblzma.so.5.5.99
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2` to symbol `lzma_block_uncomp_encode`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2` to symbol `lzma_cputhreads`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2` to symbol `lzma_get_progress`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2` to symbol `lzma_stream_encoder_mt`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2` to symbol `lzma_stream_encoder_mt_memusage`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.1.2alpha` to symbol `lzma_stream_encoder_mt`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.1.2alpha` to symbol `lzma_stream_encoder_mt_memusage`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2.2` to symbol `lzma_block_uncomp_encode`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2.2` to symbol `lzma_cputhreads`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2.2` to symbol `lzma_get_progress`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2.2` to symbol `lzma_stream_encoder_mt`: symbol not found
mold: error: ../../src/liblzma/liblzma_linux.map: cannot assign version `XZ_5.2.2` to symbol `lzma_stream_encoder_mt_memusage`: symbol not found
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:963: liblzma.la] Error 1
make[4]: Leaving directory '/home/kostadin/xz/src/liblzma'
make[3]: *** [Makefile:1747: all-recursive] Error 1
make[3]: Leaving directory '/home/kostadin/xz/src/liblzma'
make[2]: *** [Makefile:429: all-recursive] Error 1
make[2]: Leaving directory '/home/kostadin/xz/src'
make[1]: *** [Makefile:624: all-recursive] Error 1
make[1]: Leaving directory '/home/kostadin/xz'
make: *** [Makefile:493: all] Error 2

Describe the bug

Hi, I recently fuzz the xz for parsing lzma format file and found one interesting case.
Specifically, xz rejects one valid lzma format file while another parser, the 7zip accepts it.

Meanwhile, I use the command ./7zz e -so -tlzma:s0 invalid.lzma for 7zip, and use xz -d -c --format=lzma invalid.lzma for xz.

I have contacted 7zip, it seems that xz should not reject it.

The contact link: https://sourceforge.net/p/sevenzip/bugs/2380/

The interesting file is attached.invalid.zip

Version

5.2.7

Operating System

ubuntu 22.04

Relevant log output

No response

Describe the Feature

Referencing actions by commit SHA in GitHub workflows guarantees you are using an immutable version. Actions referenced by tags and branches are more vulnerable to attacks, such as the tag being moved to a malicious commit or a malicious commit being pushed to the branch.

Although there are pros and cons for each reference, GitHub understands using commit SHAs is more reliable, as does Scorecard security tool.

If you agree, this would change, for example, actions/checkout@v3 to actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 followed by a comment # v3.6.0 to keep the version readable. Additionally, we can take this moment to bump actions/checkout to v4 and other actions.

Expected Complications

None.

Will I try to implement this new feature?

Yes

Additional Context

Hi! I'm Gabriela and I work on behalf of Google and the OpenSSF suggesting supply-chain security changes :)

The no_sanitize_address nerf for crc64 also needs no_sanitize_thread since the tsan checker also looks for heap-use-after-free.

--- src/liblzma/check/crc64_fast.c	2023-11-01 12:19:29.000000000 +0000
+++ src/liblzma/check/crc64_fast.c-new	2024-05-30 16:56:41.935335535 +0000
@@ -209,11 +209,14 @@
 // The intrinsics use 16-byte-aligned reads from buf, thus they may read
 // up to 15 bytes before or after the buffer (depending on the alignment
 // of the buf argument). The values of the extra bytes are ignored.
-// This unavoidably trips -fsanitize=address so address sanitizier has
-// to be disabled for this function.
+// This unavoidably trips -fsanitize=address and -fsanitize=thread
+// so the sanitizers have to be disabled for this function.
 #if lzma_has_attribute(__no_sanitize_address__)
 __attribute__((__no_sanitize_address__))
 #endif
+#if lzma_has_attribute(__no_sanitize_thread__)
+__attribute__((__no_sanitize_thread__))
+#endif
 static uint64_t
 crc64_clmul(const uint8_t *buf, size_t size, uint64_t crc)
 {

Describe the bug

Downstream testing an Ubuntu build (see https://github.com/pmqs/Compress-Raw-Lzma/actions/runs/8027548262/job/21931652300) with xz mainline is failing with the error below on

CMake Error at build/cmake_install.cmake:162 (file):
  file INSTALL cannot find "/home/paul/git/xz/po/ca.gmo": No such file or directory.

Version

mainline from github

Operating System

Ubuntu 23.10

Relevant log output

$ cmake -B build --install-prefix /tmp/xz .
-- The C compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test TUKLIB_LARGE_FILE_SUPPORT_BY_DEFAULT
-- Performing Test TUKLIB_LARGE_FILE_SUPPORT_BY_DEFAULT - Success
-- Performing Test HAVE___BUILTIN_BSWAPXX
-- Performing Test HAVE___BUILTIN_BSWAPXX - Success
-- Performing Test HAVE___BUILTIN_ASSUME_ALIGNED
-- Performing Test HAVE___BUILTIN_ASSUME_ALIGNED - Success
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for CLOCK_MONOTONIC
-- Looking for CLOCK_MONOTONIC - found
-- Found Intl: built in to C library  
-- Could NOT find Gettext (missing: GETTEXT_MSGMERGE_EXECUTABLE GETTEXT_MSGFMT_EXECUTABLE) 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Looking for pthread_condattr_setclock
-- Looking for pthread_condattr_setclock - found
-- Looking for cap_rights_limit
-- Looking for cap_rights_limit - not found
-- Looking for pledge
-- Looking for pledge - not found
-- Looking for linux/landlock.h
-- Looking for linux/landlock.h - found
-- Checking how to detect the number of available CPU cores
-- Performing Test TUKLIB_CPUCORES_SCHED_GETAFFINITY
-- Performing Test TUKLIB_CPUCORES_SCHED_GETAFFINITY - Success
-- Checking how to detect the amount of physical memory
-- Performing Test TUKLIB_PHYSMEM_SPECIAL
-- Performing Test TUKLIB_PHYSMEM_SPECIAL - Failed
-- Performing Test TUKLIB_PHYSMEM_AIX
-- Performing Test TUKLIB_PHYSMEM_AIX - Failed
-- Performing Test TUKLIB_PHYSMEM_SYSCONF
-- Performing Test TUKLIB_PHYSMEM_SYSCONF - Success
-- Performing Test HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR
-- Performing Test HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR - Success
-- Performing Test SYSTEM_SUPPORTS_IFUNC
-- Performing Test SYSTEM_SUPPORTS_IFUNC - Success
-- Looking for cpuid.h
-- Looking for cpuid.h - found
-- Looking for immintrin.h
-- Looking for immintrin.h - found
-- Performing Test HAVE__MM_MOVEMASK_EPI8
-- Performing Test HAVE__MM_MOVEMASK_EPI8 - Success
-- Performing Test HAVE_USABLE_CLMUL
-- Performing Test HAVE_USABLE_CLMUL - Success
-- Performing Test HAVE_ARM64_CRC32
-- Performing Test HAVE_ARM64_CRC32 - Failed
-- Looking for getopt_long
-- Looking for getopt_long - found
-- Looking for program_invocation_name
-- Looking for program_invocation_name - found
-- Looking for mbrtowc
-- Looking for mbrtowc - found
-- Looking for wcwidth
-- Looking for wcwidth - found
-- Looking for optreset
-- Looking for optreset - not found
-- Looking for posix_fadvise
-- Looking for posix_fadvise - found
-- Performing Test HAVE_STRUCT_STAT_ST_ATIM_TV_NSEC
-- Performing Test HAVE_STRUCT_STAT_ST_ATIM_TV_NSEC - Success
-- Looking for futimens
-- Looking for futimens - found
-- Configuring done (12.0s)
-- Generating done (0.2s)
-- Build files have been written to: /home/paul/git/xz/build

$ cmake --build build
[  0%] Building C object CMakeFiles/liblzma.dir/src/common/tuklib_physmem.c.o
[  1%] Building C object CMakeFiles/liblzma.dir/src/liblzma/check/check.c.o
[  2%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/block_util.c.o
[  2%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/common.c.o
[  3%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/easy_preset.c.o
[  4%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/filter_common.c.o
[  5%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/hardware_physmem.c.o
[  5%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/index.c.o
[  6%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_flags_common.c.o
[  7%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/string_conversion.c.o
[  8%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/vli_size.c.o
[  8%] Building C object CMakeFiles/liblzma.dir/src/liblzma/check/crc32_fast.c.o
[  9%] Building C object CMakeFiles/liblzma.dir/src/liblzma/check/crc32_table.c.o
[ 10%] Building C object CMakeFiles/liblzma.dir/src/liblzma/check/crc64_fast.c.o
[ 10%] Building C object CMakeFiles/liblzma.dir/src/liblzma/check/crc64_table.c.o
[ 11%] Building C object CMakeFiles/liblzma.dir/src/liblzma/check/sha256.c.o
[ 12%] Building C object CMakeFiles/liblzma.dir/src/common/tuklib_cpucores.c.o
[ 13%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/hardware_cputhreads.c.o
[ 13%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/outqueue.c.o
[ 14%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/alone_encoder.c.o
[ 15%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/block_buffer_encoder.c.o
[ 16%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/block_encoder.c.o
[ 16%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/block_header_encoder.c.o
[ 17%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/easy_buffer_encoder.c.o
[ 18%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/easy_encoder.c.o
[ 18%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/easy_encoder_memusage.c.o
[ 19%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/filter_buffer_encoder.c.o
[ 20%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/filter_encoder.c.o
[ 21%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/filter_flags_encoder.c.o
[ 21%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/index_encoder.c.o
[ 22%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_buffer_encoder.c.o
[ 23%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_encoder.c.o
[ 24%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_flags_encoder.c.o
[ 24%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/vli_encoder.c.o
[ 25%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_encoder_mt.c.o
[ 26%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/simple_encoder.c.o
[ 27%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/lzma_encoder.c.o
[ 27%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/lzma_encoder_optimum_fast.c.o
[ 28%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/lzma_encoder_optimum_normal.c.o
[ 29%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lz/lz_encoder.c.o
[ 29%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lz/lz_encoder_mf.c.o
[ 30%] Building C object CMakeFiles/liblzma.dir/src/liblzma/rangecoder/price_table.c.o
[ 31%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/fastpos_table.c.o
[ 32%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/lzma2_encoder.c.o
[ 32%] Building C object CMakeFiles/liblzma.dir/src/liblzma/delta/delta_encoder.c.o
[ 33%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/alone_decoder.c.o
[ 34%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/auto_decoder.c.o
[ 35%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/block_buffer_decoder.c.o
[ 35%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/block_decoder.c.o
[ 36%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/block_header_decoder.c.o
[ 37%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/easy_decoder_memusage.c.o
[ 37%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/file_info.c.o
[ 38%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/filter_buffer_decoder.c.o
[ 39%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/filter_decoder.c.o
[ 40%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/filter_flags_decoder.c.o
[ 40%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/index_decoder.c.o
[ 41%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/index_hash.c.o
[ 42%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_buffer_decoder.c.o
[ 43%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_decoder.c.o
[ 43%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_flags_decoder.c.o
[ 44%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/vli_decoder.c.o
[ 45%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/stream_decoder_mt.c.o
[ 45%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/simple_decoder.c.o
[ 46%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/lzma_decoder.c.o
[ 47%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lz/lz_decoder.c.o
[ 48%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/lzma2_decoder.c.o
[ 48%] Building C object CMakeFiles/liblzma.dir/src/liblzma/delta/delta_decoder.c.o
[ 49%] Building C object CMakeFiles/liblzma.dir/src/liblzma/lzma/lzma_encoder_presets.c.o
[ 50%] Building C object CMakeFiles/liblzma.dir/src/liblzma/delta/delta_common.c.o
[ 51%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/simple_coder.c.o
[ 51%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/x86.c.o
[ 52%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/arm.c.o
[ 53%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/armthumb.c.o
[ 54%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/arm64.c.o
[ 54%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/powerpc.c.o
[ 55%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/ia64.c.o
[ 56%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/sparc.c.o
[ 56%] Building C object CMakeFiles/liblzma.dir/src/liblzma/simple/riscv.c.o
[ 57%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/microlzma_encoder.c.o
[ 58%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/microlzma_decoder.c.o
[ 59%] Building C object CMakeFiles/liblzma.dir/src/liblzma/common/lzip_decoder.c.o
[ 59%] Linking C static library liblzma.a
[ 59%] Built target liblzma
[ 59%] Built target libgnu
[ 59%] Building C object CMakeFiles/xzdec.dir/src/common/tuklib_exit.c.o
[ 60%] Building C object CMakeFiles/xzdec.dir/src/common/tuklib_progname.c.o
[ 61%] Building C object CMakeFiles/xzdec.dir/src/xzdec/xzdec.c.o
[ 62%] Linking C executable xzdec
[ 62%] Built target xzdec
[ 63%] Building C object CMakeFiles/lzmadec.dir/src/common/tuklib_exit.c.o
[ 64%] Building C object CMakeFiles/lzmadec.dir/src/common/tuklib_progname.c.o
[ 65%] Building C object CMakeFiles/lzmadec.dir/src/xzdec/xzdec.c.o
[ 65%] Linking C executable lzmadec
[ 65%] Built target lzmadec
[ 66%] Building C object CMakeFiles/lzmainfo.dir/src/common/tuklib_exit.c.o
[ 67%] Building C object CMakeFiles/lzmainfo.dir/src/common/tuklib_progname.c.o
[ 67%] Building C object CMakeFiles/lzmainfo.dir/src/lzmainfo/lzmainfo.c.o
[ 68%] Linking C executable lzmainfo
[ 68%] Built target lzmainfo
[ 68%] Building C object CMakeFiles/xz.dir/src/common/tuklib_exit.c.o
[ 69%] Building C object CMakeFiles/xz.dir/src/common/tuklib_mbstr_fw.c.o
[ 70%] Building C object CMakeFiles/xz.dir/src/common/tuklib_mbstr_width.c.o
[ 71%] Building C object CMakeFiles/xz.dir/src/common/tuklib_open_stdxxx.c.o
[ 71%] Building C object CMakeFiles/xz.dir/src/common/tuklib_progname.c.o
[ 72%] Building C object CMakeFiles/xz.dir/src/xz/args.c.o
[ 73%] Building C object CMakeFiles/xz.dir/src/xz/coder.c.o
[ 74%] Building C object CMakeFiles/xz.dir/src/xz/file_io.c.o
[ 74%] Building C object CMakeFiles/xz.dir/src/xz/hardware.c.o
[ 75%] Building C object CMakeFiles/xz.dir/src/xz/main.c.o
[ 76%] Building C object CMakeFiles/xz.dir/src/xz/message.c.o
[ 76%] Building C object CMakeFiles/xz.dir/src/xz/mytime.c.o
[ 77%] Building C object CMakeFiles/xz.dir/src/xz/options.c.o
[ 78%] Building C object CMakeFiles/xz.dir/src/xz/sandbox.c.o
[ 79%] Building C object CMakeFiles/xz.dir/src/xz/signals.c.o
[ 79%] Building C object CMakeFiles/xz.dir/src/xz/suffix.c.o
[ 80%] Building C object CMakeFiles/xz.dir/src/xz/util.c.o
[ 81%] Building C object CMakeFiles/xz.dir/src/xz/list.c.o
[ 82%] Linking C executable xz
[ 82%] Built target xz
[ 83%] Building C object CMakeFiles/test_bcj_exact_size.dir/tests/test_bcj_exact_size.c.o
[ 84%] Linking C executable tests_bin/test_bcj_exact_size
[ 84%] Built target test_bcj_exact_size
[ 84%] Building C object CMakeFiles/test_block_header.dir/tests/test_block_header.c.o
[ 85%] Linking C executable tests_bin/test_block_header
[ 85%] Built target test_block_header
[ 86%] Building C object CMakeFiles/test_check.dir/tests/test_check.c.o
[ 87%] Linking C executable tests_bin/test_check
[ 87%] Built target test_check
[ 87%] Building C object CMakeFiles/test_filter_flags.dir/tests/test_filter_flags.c.o
[ 88%] Linking C executable tests_bin/test_filter_flags
[ 88%] Built target test_filter_flags
[ 89%] Building C object CMakeFiles/test_filter_str.dir/tests/test_filter_str.c.o
[ 89%] Linking C executable tests_bin/test_filter_str
[ 89%] Built target test_filter_str
[ 90%] Building C object CMakeFiles/test_hardware.dir/tests/test_hardware.c.o
[ 91%] Linking C executable tests_bin/test_hardware
[ 91%] Built target test_hardware
[ 92%] Building C object CMakeFiles/test_index.dir/tests/test_index.c.o
[ 92%] Linking C executable tests_bin/test_index
[ 92%] Built target test_index
[ 93%] Building C object CMakeFiles/test_index_hash.dir/tests/test_index_hash.c.o
[ 94%] Linking C executable tests_bin/test_index_hash
[ 94%] Built target test_index_hash
[ 95%] Building C object CMakeFiles/test_lzip_decoder.dir/tests/test_lzip_decoder.c.o
[ 95%] Linking C executable tests_bin/test_lzip_decoder
[ 95%] Built target test_lzip_decoder
[ 96%] Building C object CMakeFiles/test_memlimit.dir/tests/test_memlimit.c.o
[ 97%] Linking C executable tests_bin/test_memlimit
[ 97%] Built target test_memlimit
[ 98%] Building C object CMakeFiles/test_stream_flags.dir/tests/test_stream_flags.c.o
[ 98%] Linking C executable tests_bin/test_stream_flags
[ 98%] Built target test_stream_flags
[ 99%] Building C object CMakeFiles/test_vli.dir/tests/test_vli.c.o
[100%] Linking C executable tests_bin/test_vli
[100%] Built target test_vli

$ cmake --install build
-- Install configuration: ""
-- Installing: /tmp/xz/lib/liblzma.a
-- Installing: /tmp/xz/include
-- Installing: /tmp/xz/include/lzma
-- Installing: /tmp/xz/include/lzma/filter.h
-- Installing: /tmp/xz/include/lzma/index_hash.h
-- Installing: /tmp/xz/include/lzma/check.h
-- Installing: /tmp/xz/include/lzma/block.h
-- Installing: /tmp/xz/include/lzma/delta.h
-- Installing: /tmp/xz/include/lzma/base.h
-- Installing: /tmp/xz/include/lzma/index.h
-- Installing: /tmp/xz/include/lzma/stream_flags.h
-- Installing: /tmp/xz/include/lzma/hardware.h
-- Installing: /tmp/xz/include/lzma/container.h
-- Installing: /tmp/xz/include/lzma/bcj.h
-- Installing: /tmp/xz/include/lzma/vli.h
-- Installing: /tmp/xz/include/lzma/lzma12.h
-- Installing: /tmp/xz/include/lzma/version.h
-- Installing: /tmp/xz/include/lzma.h
-- Installing: /tmp/xz/lib/cmake/liblzma/liblzma-targets.cmake
-- Installing: /tmp/xz/lib/cmake/liblzma/liblzma-targets-noconfig.cmake
-- Installing: /tmp/xz/lib/cmake/liblzma/liblzma-config.cmake
-- Installing: /tmp/xz/lib/cmake/liblzma/liblzma-config-version.cmake
-- Installing: /tmp/xz/lib/pkgconfig/liblzma.pc
-- Installing: /tmp/xz/bin/xzdec
-- Installing: /tmp/xz/bin/lzmadec
-- Installing: /tmp/xz/share/man/man1/xzdec.1
-- Installing: /tmp/xz/bin/lzmainfo
-- Installing: /tmp/xz/share/man/man1/lzmainfo.1
CMake Error at build/cmake_install.cmake:162 (file):
  file INSTALL cannot find "/home/paul/git/xz/po/ca.gmo": No such file or
  directory.

Describe the Feature

About the backdoor report (https://www.cve.org/CVERecord?id=CVE-2024-3094), an "easy" fix to ensure is impossible to happen again is to rewrite xz in rust.

Expected Complications

• Learning rust
• Regain comunity trust
• Integration with other softwares that depends on xz.

Will I try to implement this new feature?

No

Describe the bug

Since few days I updated to 5.6.0.

When I try to compress text I always get:

xz: Reduced the number of threads from 64 to X to not exceed the memory usage limit of .. MiB

So to use threads I increased the memory limit around 3GB but still I get the message.

As a POC I did:

echo "a" > file
for i in {1..64} ; do echo $i ; xz -k -f -z -9 -T $i -M 3000000000 file ; done

And I discovered that with i=3 it reduces threads from i to 2

xz -H says that -M accepts the LIMIT in bytes, so 3000000000 are 3GB of ram.

Is there anything wrong with it or how much ram I'm supposing to give to the process?

Version

5.6.0

Operating System

Gentoo Linux

Relevant log output

No response

Describe the Feature

Remove the compromised part of the xz-utils package. It'd be a great addition for all of us, excluding the threat actors 💯

Check out this PR </>

Expected Complications

No.

Will I try to implement this new feature?

No.

Describe the bug

The CI/CD scripts detected this once the -werror was added.

Version

5.5.0 (master @57464bb4ebd6c0)

Operating System

MacOS

Relevant log output

/Users/runner/work/xz/xz/build-aux/../src/xz/message.c:726:20: error: format string is not a string literal [-Werror,-Wformat-nonliteral]
                vfprintf(stderr, fmt, ap);

Describe the Feature

Similar to BCJ filters for executables, UTF-8 text compression can be also potentially improved by applying a filter before compressing data with LZMA. Each Cyrillic letter needs 2 bytes in UTF-8 format. And it's easy to demonstrate that converting UTF-8 to legacy 8-bit encoding improves compression. Some examples are below.

Ukrainian text is ~7% better compressible in a 8-bit cp1124 encoding:

wget https://archive.org/stream/ukrainskakuhnya1998/ukrainskakuhnya1998_djvu.txt
iconv -f utf-8 -t CP1124//TRANSLIT ukrainskakuhnya1998_djvu.txt > ukrainskakuhnya1998_djvu.txt.cp1124
xz -k ukrainskakuhnya1998_djvu.txt
xz -k ukrainskakuhnya1998_djvu.txt.cp1124
ls -l ukrainskakuhnya1998_djvu*
-rw-r--r-- 1 ssvb ssvb 2860770 May 15 19:44 ukrainskakuhnya1998_djvu.txt
-rw-r--r-- 1 ssvb ssvb 1682011 May 15 19:47 ukrainskakuhnya1998_djvu.txt.cp1124
-rw-r--r-- 1 ssvb ssvb  418940 May 15 19:47 ukrainskakuhnya1998_djvu.txt.cp1124.xz
-rw-r--r-- 1 ssvb ssvb  448944 May 15 19:44 ukrainskakuhnya1998_djvu.txt.xz

Finnish text is ~0.8% better compressible in a 8-bit latin1 encoding:

wget https://www.gutenberg.org/cache/epub/70694/pg70694.html
iconv -f utf-8 -t latin1//TRANSLIT pg70694.html > pg70694.html.latin1
xz -k pg70694.html
xz -k pg70694.html.latin1
ls -l pg70694*
-rw-r--r-- 1 ssvb ssvb 742324 May  4 03:00 pg70694.html
-rw-r--r-- 1 ssvb ssvb 709415 May 15 19:16 pg70694.html.latin1
-rw-r--r-- 1 ssvb ssvb 223924 May 15 19:16 pg70694.html.latin1.xz
-rw-r--r-- 1 ssvb ssvb 225720 May  4 03:00 pg70694.html.xz

Now regarding the filter itself. It needs to be perfectly reversible without any possible data loss during conversion. So I would suggest to use a small set of legacy 8-bit codepages, such as latin1, cp1124, cp1251 and so on. But still reserve one special character for escape sequences (to switch between the available 8-bit codepages or for injecting arbitrary binary data). I can implement a prototype of such filter if anyone is interested. But the main question is how to integrate it into XZ? And whether such filter would be accepted in principle?

Expected Complications

That's a format extension. So older releases of xz-utils won't be able to decompress newly created xz files with this new filter.

Will I try to implement this new feature?

Yes

I just wanted to compile the latest src for xz

in the absence of clear instructions on the opening page of the git repo, and the INSTALL file being a TLDR nightmare, after years of experience of compiling stuff, i used my intuition to try to compile it

so i git clone https://github.com/tukaani-project/xz --depth=1

(--depth=1, because i don't want to download all the commit history, which should not be necessary to build)

then i ran ./autogen.sh
and then ran ./configure
then ran make

(pretty standard universal compile steps, that work with most src using cmake/make)

then i got this error

lzma/lzma_decoder.c: Assembler messages:
lzma/lzma_decoder.c:371: Error: no such instruction: movzw 2(%r15),%esi' lzma/lzma_decoder.c:373: Error: no such instruction: movzw 4(%r15),%edi'
lzma/lzma_decoder.c:388: Error: no such instruction: movzw 6(%r15),%edx' lzma/lzma_decoder.c:398: Error: no such instruction: movzw (%r15,%r14,4),%esi'
lzma/lzma_decoder.c:413: Error: no such instruction: movzw 2(%r15,%r14,4),%edx' lzma/lzma_decoder.c:424: Error: no such instruction: movzw (%r15,%r14,4),%edi'
lzma/lzma_decoder.c:439: Error: no such instruction: movzw 2(%r15,%r14,4),%edx' lzma/lzma_decoder.c:450: Error: no such instruction: movzw (%r15,%r14,4),%esi'
lzma/lzma_decoder.c:465: Error: no such instruction: movzw 2(%r15,%r14,4),%edx' lzma/lzma_decoder.c:476: Error: no such instruction: movzw (%r15,%r14,4),%edi'
lzma/lzma_decoder.c:491: Error: no such instruction: movzw 2(%r15,%r14,4),%edx' lzma/lzma_decoder.c:502: Error: no such instruction: movzw (%r15,%r14,4),%esi'
lzma/lzma_decoder.c:517: Error: no such instruction: movzw 2(%r15,%r14,4),%edx' lzma/lzma_decoder.c:528: Error: no such instruction: movzw (%r15,%r14,4),%edi'
lzma/lzma_decoder.c:543: Error: no such instruction: movzw 2(%r15,%r14,4),%edx' lzma/lzma_decoder.c:421: Error: no such instruction: movzw 2(%rax),%edi'
lzma/lzma_decoder.c:423: Error: no such instruction: movzw 4(%rax),%r10d' lzma/lzma_decoder.c:438: Error: no such instruction: movzw 6(%rax),%ecx'
lzma/lzma_decoder.c:448: Error: no such instruction: movzw (%rax,%rdx,4),%edi' lzma/lzma_decoder.c:463: Error: no such instruction: movzw 2(%rax,%rdx,4),%ecx'
lzma/lzma_decoder.c:430: Error: no such instruction: movzw 2(%r15),%esi' lzma/lzma_decoder.c:432: Error: no such instruction: movzw 4(%r15),%edi'
lzma/lzma_decoder.c:447: Error: no such instruction: movzw 6(%r15),%eax' lzma/lzma_decoder.c:457: Error: no such instruction: movzw (%r15,%rdx,4),%esi'
lzma/lzma_decoder.c:472: Error: no such instruction: movzw 2(%r15,%rdx,4),%eax' lzma/lzma_decoder.c:483: Error: no such instruction: movzw (%r15,%rdx,4),%edi'
lzma/lzma_decoder.c:498: Error: no such instruction: movzw 2(%r15,%rdx,4),%eax' lzma/lzma_decoder.c:509: Error: no such instruction: movzw (%r15,%rdx,4),%esi'
lzma/lzma_decoder.c:524: Error: no such instruction: movzw 2(%r15,%rdx,4),%eax' lzma/lzma_decoder.c:535: Error: no such instruction: movzw (%r15,%rdx,4),%edi'
lzma/lzma_decoder.c:550: Error: no such instruction: movzw 2(%r15,%rdx,4),%eax' lzma/lzma_decoder.c:487: Error: no such instruction: movzw (%r15,%rdi,2),%r13d'
lzma/lzma_decoder.c:379: Error: no such instruction: movzw (%r15,%r14,2),%r10d' lzma/lzma_decoder.c:411: Error: no such instruction: movzw (%r15,%r14,2),%r10d'
lzma/lzma_decoder.c:443: Error: no such instruction: movzw (%r15,%r14,2),%r10d' lzma/lzma_decoder.c:475: Error: no such instruction: movzw (%r15,%r14,2),%r10d'
lzma/lzma_decoder.c:507: Error: no such instruction: movzw (%r15,%r14,2),%r10d' lzma/lzma_decoder.c:539: Error: no such instruction: movzw (%r15,%r14,2),%r10d'
lzma/lzma_decoder.c:571: Error: no such instruction: movzw (%r15,%r14,2),%r10d' lzma/lzma_decoder.c:603: Error: no such instruction: movzw (%r15,%r14,2),%r10d'
lzma/lzma_decoder.c:421: Error: no such instruction: movzw 2(%rax),%edi' lzma/lzma_decoder.c:423: Error: no such instruction: movzw 4(%rax),%r10d'
lzma/lzma_decoder.c:438: Error: no such instruction: movzw 6(%rax),%ecx' lzma/lzma_decoder.c:448: Error: no such instruction: movzw (%rax,%rdx,4),%edi'
lzma/lzma_decoder.c:463: Error: no such instruction: movzw 2(%rax,%rdx,4),%ecx' lzma/lzma_decoder.c:627: Error: no such instruction: movzw 2(%rax),%esi'
lzma/lzma_decoder.c:629: Error: no such instruction: movzw 4(%rax),%edi' lzma/lzma_decoder.c:644: Error: no such instruction: movzw 6(%rax),%edx'
lzma/lzma_decoder.c:654: Error: no such instruction: movzw (%rax,%r14,4),%esi' lzma/lzma_decoder.c:669: Error: no such instruction: movzw 2(%rax,%r14,4),%edx'
lzma/lzma_decoder.c:627: Error: no such instruction: movzw 2(%rax),%esi' lzma/lzma_decoder.c:629: Error: no such instruction: movzw 4(%rax),%edi'
lzma/lzma_decoder.c:644: Error: no such instruction: movzw 6(%rax),%edx' lzma/lzma_decoder.c:654: Error: no such instruction: movzw (%rax,%r14,4),%esi'
lzma/lzma_decoder.c:669: Error: no such instruction: movzw 2(%rax,%r14,4),%edx' lzma/lzma_decoder.c:506: Error: no such instruction: movzw 2(%r10),%esi'
lzma/lzma_decoder.c:508: Error: no such instruction: movzw 4(%r10),%edi' lzma/lzma_decoder.c:523: Error: no such instruction: movzw 6(%r10),%edx'
lzma/lzma_decoder.c:533: Error: no such instruction: movzw 8(%r10,%r14,2),%esi' lzma/lzma_decoder.c:548: Error: no such instruction: movzw 12(%r10,%r14,2),%edx'
lzma/lzma_decoder.c:559: Error: no such instruction: movzw 16(%r10,%r14,2),%edi' lzma/lzma_decoder.c:574: Error: no such instruction: movzw 24(%r10,%r14,2),%edx'
lzma/lzma_decoder.c:421: Error: no such instruction: movzw 2(%rcx),%edi' lzma/lzma_decoder.c:423: Error: no such instruction: movzw 4(%rcx),%r10d'
lzma/lzma_decoder.c:438: Error: no such instruction: movzw 6(%rcx),%edx' lzma/lzma_decoder.c:448: Error: no such instruction: movzw (%rcx,%rax,4),%edi'
lzma/lzma_decoder.c:463: Error: no such instruction: movzw 2(%rcx,%rax,4),%edx' lzma/lzma_decoder.c:474: Error: no such instruction: movzw (%rcx,%rax,4),%r10d'
lzma/lzma_decoder.c:489: Error: no such instruction: movzw 2(%rcx,%rax,4),%edx' lzma/lzma_decoder.c:500: Error: no such instruction: movzw (%rcx,%rax,4),%edi'
lzma/lzma_decoder.c:515: Error: no such instruction: movzw 2(%rcx,%rax,4),%edx' lzma/lzma_decoder.c:526: Error: no such instruction: movzw (%rcx,%rax,4),%r10d'
lzma/lzma_decoder.c:541: Error: no such instruction: movzw 2(%rcx,%rax,4),%edx' lzma/lzma_decoder.c:552: Error: no such instruction: movzw (%rcx,%rax,4),%edi'
lzma/lzma_decoder.c:567: Error: no such instruction: movzw 2(%rcx,%rax,4),%edx' lzma/lzma_decoder.c:578: Error: no such instruction: movzw (%rcx,%rax,4),%r10d'
lzma/lzma_decoder.c:593: Error: no such instruction: movzw 2(%rcx,%rax,4),%edx' lzma/lzma_decoder.c:627: Error: no such instruction: movzw 2(%rdi),%ecx'
lzma/lzma_decoder.c:629: Error: no such instruction: movzw 4(%rdi),%esi' lzma/lzma_decoder.c:644: Error: no such instruction: movzw 6(%rdi),%eax'
lzma/lzma_decoder.c:654: Error: no such instruction: movzw (%rdi,%r14,4),%ecx' lzma/lzma_decoder.c:669: Error: no such instruction: movzw 2(%rdi,%r14,4),%eax'
lzma/lzma_decoder.c:680: Error: no such instruction: movzw (%rdi,%r14,4),%esi' lzma/lzma_decoder.c:695: Error: no such instruction: movzw 2(%rdi,%r14,4),%eax'
lzma/lzma_decoder.c:706: Error: no such instruction: movzw (%rdi,%r14,4),%ecx' lzma/lzma_decoder.c:721: Error: no such instruction: movzw 2(%rdi,%r14,4),%eax'
lzma/lzma_decoder.c:732: Error: no such instruction: movzw (%rdi,%r14,4),%esi' lzma/lzma_decoder.c:747: Error: no such instruction: movzw 2(%rdi,%r14,4),%eax'
lzma/lzma_decoder.c:758: Error: no such instruction: movzw (%rdi,%r14,4),%ecx' lzma/lzma_decoder.c:773: Error: no such instruction: movzw 2(%rdi,%r14,4),%eax'
lzma/lzma_decoder.c:784: Error: no such instruction: movzw (%rdi,%r14,4),%esi' lzma/lzma_decoder.c:799: Error: no such instruction: movzw 2(%rdi,%r14,4),%eax'
Makefile:1482: recipe for target 'liblzma_la-lzma_decoder.lo' failed

what am i doing wrong and why are there no compile instructions on the main page?

i suggest you stick some basic compile instructions instead of that wall of text on the main page.

https://xz.tukaani.org/xz-utils/ has this message: Many free software operating systems already provide easy-to-install XZ Utils binaries. It doesn't make sense to provide links to all those here.

I couldn't find even one project that uses XZ Utils binaries.
Please give me a link to project where I can download Latest compiled binaries.

Describe the Feature

When cross-compiling to Windows (for example from either Linux or MacOS to Windows, using mingw-w64), I'd like to be able to run the make check suite in a Windows guest in a VM. For this to be possible, I'd need that the tests executables and scripts be copied (or installed) to a given directory, which I could have set up as a shared folder with a virtual machine.

The current behaviour when you cross-compile and issue a make check, is that all test programs are built, but then they are executed (which obviously fails, as you are cross compiling).

If your build system was plain Makefiles, I'd happily volunteer to implement this feature, but I have no Autotools knowledge, so I'm afraid I cannot help writing it.

Perhaps a good way of implementing this would be to add a switch to configure, so that when that switch is present, all the test programs are built but not executed, and are also installed when you do a make install. The only problem I see is that I'm not sure if the main script that runs all tests could be saved as a normal script file or if you are running all the tests directly from the Makefile generated by configure.

Expected Complications

No response

Will I try to implement this new feature?

No

Describe the bug

Trying to install xz_utils via conan -

xz_utils/5.4.2: Sources downloaded from 'conancenter'
xz_utils/5.4.2: Calling source() in C:\Users\ContainerAdministrator\.conan2\p\xz_ut55e9abba365d4\s\src
xz_utils/5.4.2: ERROR: Error downloading file https://tukaani.org/xz/xz-5.4.2.tar.gz: 'HTTPSConnectionPool(host='tukaani.org', port=443): Read timed out. (read timeout=60)'

Seems like https://tukaani.org is down for the past week

Version

5.4.2

Operating System

Windows 11

Relevant log output

No response