Git Product home page Git Product logo

filesystem's Introduction

Supported Platforms License: MIT CMake Build Matrix Build Status Build Status Coverage Status Latest Release Tag

Filesystem

This is a header-only single-file std::filesystem compatible helper library, based on the C++17 and C++20 specs, but implemented for C++11, C++14, C++17 or C++20 (tightly following the C++17 standard with very few documented exceptions). It is currently tested on macOS 10.12/10.14/10.15/11.6, Windows 10, Ubuntu 18.04, Ubuntu 20.04, CentOS 7, CentOS 8, FreeBSD 12, Alpine ARM/ARM64 Linux and Solaris 10 but should work on other systems too, as long as you have at least a C++11 compatible compiler. It should work with Android NDK, Emscripten and I even had reports of it being used on iOS (within sandboxing constraints) and with v1.5.6 there is experimental support for QNX. The support of Android NDK, Emscripten, QNX, and since 1.5.14 GNU/Hurd and Haiku is not backed up by automated testing but PRs and bug reports are welcome for those too and they are reported to work. It is of course in its own namespace ghc::filesystem to not interfere with a regular std::filesystem should you use it in a mixed C++17 environment (which is possible).

Test coverage is well above 90%, and starting with v1.3.6 and in v1.5.0 more time was invested in benchmarking and optimizing parts of the library. I'll try to continue to optimize some parts and refactor others, striving to improve it as long as it doesn't introduce additional C++17/C++20 compatibility issues. Feedback is always welcome. Simply open an issue if you see something missing or wrong or not behaving as expected and I'll comment.

Motivation

I'm often in need of filesystem functionality, mostly fs::path, but directory access too, and when beginning to use C++11, I used that language update to try to reduce my third-party dependencies. I could drop most of what I used, but still missed some stuff that I started implementing for the fun of it. Originally I based these helpers on my own coding- and naming conventions. When C++17 was finalized, I wanted to use that interface, but it took a while, to push myself to convert my classes.

The implementation is closely based on chapter 30.10 from the C++17 standard and a draft close to that version is Working Draft N4687. It is from after the standardization of C++17 but it contains the latest filesystem interface changes compared to the Working Draft N4659. Staring with v1.4.0, when compiled using C++20, it adapts to the changes according to path sorting order and std::u8string handling from Working Draft N4860.

I want to thank the people working on improving C++, I really liked how the language evolved with C++11 and the following standards. Keep on the good work!

Why the namespace GHC?

If you ask yourself, what ghc is standing for, it is simply gulraks helper classes, yeah, I know, not very imaginative, but I wanted a short namespace and I use it in some of my private classes (so it has nothing to do with Haskell, sorry for the name clash).

Platforms

ghc::filesystem is developed on macOS but CI tested on macOS, Windows, various Linux Distributions, FreeBSD and starting with v1.5.12 on Solaris. It should work on any of these with a C++11-capable compiler. Also there are some checks to hopefully better work on Android, but as I currently don't test with the Android NDK, I wouldn't call it a supported platform yet, same is valid for using it with Emscripten. It is now part of the detected platforms, I fixed the obvious issues and ran some tests with it, so it should be fine. All in all, I don't see it replacing std::filesystem where full C++17 or C++20 is available, it doesn't try to be a "better" std::filesystem, just an almost drop-in if you can't use it (with the exception of the UTF-8 preference).

ℹ️ Important: This implementation is following the "UTF-8 Everywhere" philosophy in that all std::string instances will be interpreted the same as std::u8string encoding wise and as being in UTF-8. The std::u16string will be seen as UTF-16. See Differences in API for more information.

Unit tests are currently run with:

  • macOS 10.12: Xcode 9.2 (clang-900.0.39.2), GCC 9.2, Clang 9.0, macOS 10.13: Xcode 10.1, macOS 10.14: Xcode 11.2, macOS 10.15: Xcode 11.6, Xcode 12.4
  • Windows: Visual Studio 2017, Visual Studio 2015, Visual Studio 2019, MinGW GCC 6.3 (Win32), GCC 7.2 (Win64), Cygwin GCC 10.2 (no CI yet)
  • Linux (Ubuntu): GCC (5.5, 6.5, 7.4, 8.3, 9.2), Clang (5.0, 6.0, 7.1, 8.0, 9.0)
  • Linux (Alpine ARM/ARM64): GCC 9.2.0 (The Drone build scripts stopped working, as they where a contribution, I couldn't figure out what went wrong, any help appreciated.)
  • FreeBSD: Clang 8.0
  • Solaris: GCC 5.5

Tests

The header comes with a set of unit-tests and uses CMake as a build tool and Catch2 as test framework. All tests are registered with in CMake, so the ctest commando can be used to run the tests.

All tests against this implementation should succeed, depending on your environment it might be that there are some warnings, e.g. if you have no rights to create Symlinks on Windows or at least the test thinks so, but these are just informative.

To build the tests from inside the project directory under macOS or Linux just:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make
ctest

This generates the test binaries that run the tests and the last command executes them.

If the default compiler is a GCC 8 or newer, or Clang 7 or newer, it additionally tries to build a version of the test binary compiled against GCCs/Clangs std::filesystem implementation, named std_filesystem_test as an additional test of conformance. Ideally all tests should compile and succeed with all filesystem implementations, but in reality, there are some differences in behavior, sometimes due to room for interpretation in in the standard, and there might be issues in these implementations too.

Usage

Downloads

The latest release version is v1.5.14 and source archives can be found here.

The latest pre-native-backend version is v1.4.0 and source archives can be found here.

The latest pre-C++20-support release version is v1.3.10 and source archives can be found here.

Currently only the latest minor release version receives bugfixes, so if possible, you should use the latest release.

Using it as Single-File-Header

As ghc::filesystem is at first a header-only library, it should be enough to copy the header or the include/ghc directory into your project folder or point your include path to this place and simply include the filesystem.hpp header (or ghc/filesystem.hpp if you use the subdirectory).

Everything is in the namespace ghc::filesystem, so one way to use it only as a fallback could be:

#if _MSVC_LANG >= 201703L || __cplusplus >= 201703L && defined(__has_include)
    // ^ Supports MSVC prior to 15.7 without setting /Zc:__cplusplus to fix __cplusplus
    // _MSVC_LANG works regardless. But without the switch, the compiler always reported 199711L: https://blogs.msdn.microsoft.com/vcblog/2018/04/09/msvc-now-correctly-reports-__cplusplus/
    #if __has_include(<filesystem>) // Two stage __has_include needed for MSVC 2015 and per https://gcc.gnu.org/onlinedocs/cpp/_005f_005fhas_005finclude.html
        #define GHC_USE_STD_FS

        // Old Apple OSs don't support std::filesystem, though the header is available at compile
        // time. In particular, std::filesystem is unavailable before macOS 10.15, iOS/tvOS 13.0,
        // and watchOS 6.0.
        #ifdef __APPLE__
            #include <Availability.h>
            // Note: This intentionally uses std::filesystem on any new Apple OS, like visionOS
            // released after std::filesystem, where std::filesystem is always available.
            // (All other __<platform>_VERSION_MIN_REQUIREDs will be undefined and thus 0.)
            #if __MAC_OS_X_VERSION_MIN_REQUIRED && __MAC_OS_X_VERSION_MIN_REQUIRED < 101500 \
             || __IPHONE_OS_VERSION_MIN_REQUIRED && __IPHONE_OS_VERSION_MIN_REQUIRED < 130000 \
             || __TV_OS_VERSION_MIN_REQUIRED && __TV_OS_VERSION_MIN_REQUIRED < 130000 \
             || __WATCH_OS_VERSION_MAX_ALLOWED && __WATCH_OS_VERSION_MAX_ALLOWED < 60000
                #undef GHC_USE_STD_FS
            #endif  
        #endif
    #endif
#endif

#ifdef GHC_USE_STD_FS
    #include <filesystem>
    namespace fs = std::filesystem;
#else
    #include "filesystem.hpp"
    namespace fs = ghc::filesystem;
#endif

If you want to also use the fstream wrapper with path support as fallback, you might use:

#if _MSVC_LANG >= 201703L || __cplusplus >= 201703L && defined(__has_include)
    // ^ Supports MSVC prior to 15.7 without setting /Zc:__cplusplus to fix __cplusplus
    // _MSVC_LANG works regardless. But without the switch, the compiler always reported 199711L: https://blogs.msdn.microsoft.com/vcblog/2018/04/09/msvc-now-correctly-reports-__cplusplus/
    #if __has_include(<filesystem>) // Two stage __has_include needed for MSVC 2015 and per https://gcc.gnu.org/onlinedocs/cpp/_005f_005fhas_005finclude.html
        #define GHC_USE_STD_FS

        // Old Apple OSs don't support std::filesystem, though the header is available at compile
        // time. In particular, std::filesystem is unavailable before macOS 10.15, iOS/tvOS 13.0,
        // and watchOS 6.0.
        #ifdef __APPLE__
            #include <Availability.h>
            // Note: This intentionally uses std::filesystem on any new Apple OS, like visionOS
            // released after std::filesystem, where std::filesystem is always available.
            // (All other __<platform>_VERSION_MIN_REQUIREDs will be undefined and thus 0.)
            #if __MAC_OS_X_VERSION_MIN_REQUIRED && __MAC_OS_X_VERSION_MIN_REQUIRED < 101500 \
             || __IPHONE_OS_VERSION_MIN_REQUIRED && __IPHONE_OS_VERSION_MIN_REQUIRED < 130000 \
             || __TV_OS_VERSION_MIN_REQUIRED && __TV_OS_VERSION_MIN_REQUIRED < 130000 \
             || __WATCH_OS_VERSION_MAX_ALLOWED && __WATCH_OS_VERSION_MAX_ALLOWED < 60000
                #undef GHC_USE_STD_FS
            #endif  
        #endif
    #endif
#endif

#ifdef GHC_USE_STD_FS
    #include <filesystem>
    namespace fs {
        using namespace std::filesystem;
        using ifstream = std::ifstream;
        using ofstream = std::ofstream;
        using fstream = std::fstream;
    }
#else
    #include "filesystem.hpp"
    namespace fs {
        using namespace ghc::filesystem;
        using ifstream = ghc::filesystem::ifstream;
        using ofstream = ghc::filesystem::ofstream;
        using fstream = ghc::filesystem::fstream;
    }
#endif

Now you have e.g. fs::ofstream out(somePath); and it is either the wrapper or the C++17 std::ofstream.

ℹ️ Be aware, as a header-only library, it is not hiding the fact, that it uses system includes, so they "pollute" your global namespace. Use the forwarding-/implementation-header based approach (see below) to avoid this. For Windows it needs Windows.h and it might be a good idea to define WIN32_LEAN_AND_MEAN or NOMINMAX prior to including filesystem.hpp or fs_std.hpp headers to reduce pollution of your global namespace and compile time. They are not defined by ghc::filesystem to allow combination with contexts where the full Windows.his needed, e.g. for UI elements.

ℹ️ Hint: There is an additional header named ghc/fs_std.hpp that implements this dynamic selection of a filesystem implementation, that you can include instead of ghc/filesystem.hpp when you want std::filesystem where available and ghc::filesystem where not.

Using it as Forwarding-/Implementation-Header

Alternatively, starting from v1.1.0 ghc::filesystem can also be used by including one of two additional wrapper headers. These allow to include a forwarded version in most places (ghc/fs_fwd.hpp) while hiding the implementation details in a single cpp file that includes ghc/fs_impl.hpp to implement the needed code. Using ghc::filesystem this way makes sure system includes are only visible from inside the cpp file, all other places are clean.

Be aware, that it is currently not supported to hide the implementation into a Windows-DLL, as a DLL interface with C++ standard templates in interfaces is a different beast. If someone is willing to give it a try, I might integrate a PR but currently working on that myself is not a priority.

If you use the forwarding/implementation approach, you can still use the dynamic switching like this:

#if _MSVC_LANG >= 201703L || __cplusplus >= 201703L && defined(__has_include)
    // ^ Supports MSVC prior to 15.7 without setting /Zc:__cplusplus to fix __cplusplus
    // _MSVC_LANG works regardless. But without the switch, the compiler always reported 199711L: https://blogs.msdn.microsoft.com/vcblog/2018/04/09/msvc-now-correctly-reports-__cplusplus/
    #if __has_include(<filesystem>) // Two stage __has_include needed for MSVC 2015 and per https://gcc.gnu.org/onlinedocs/cpp/_005f_005fhas_005finclude.html
        #define GHC_USE_STD_FS

        // Old Apple OSs don't support std::filesystem, though the header is available at compile
        // time. In particular, std::filesystem is unavailable before macOS 10.15, iOS/tvOS 13.0,
        // and watchOS 6.0.
        #ifdef __APPLE__
            #include <Availability.h>
            // Note: This intentionally uses std::filesystem on any new Apple OS, like visionOS
            // released after std::filesystem, where std::filesystem is always available.
            // (All other __<platform>_VERSION_MIN_REQUIREDs will be undefined and thus 0.)
            #if __MAC_OS_X_VERSION_MIN_REQUIRED && __MAC_OS_X_VERSION_MIN_REQUIRED < 101500 \
             || __IPHONE_OS_VERSION_MIN_REQUIRED && __IPHONE_OS_VERSION_MIN_REQUIRED < 130000 \
             || __TV_OS_VERSION_MIN_REQUIRED && __TV_OS_VERSION_MIN_REQUIRED < 130000 \
             || __WATCH_OS_VERSION_MAX_ALLOWED && __WATCH_OS_VERSION_MAX_ALLOWED < 60000
                #undef GHC_USE_STD_FS
            #endif  
        #endif
    #endif
#endif

#ifdef GHC_USE_STD_FS
    #include <filesystem>
    namespace fs {
        using namespace std::filesystem;
        using ifstream = std::ifstream;
        using ofstream = std::ofstream;
        using fstream = std::fstream;
    }
#else
    #include "fs_fwd.hpp"
    namespace fs {
        using namespace ghc::filesystem;
        using ifstream = ghc::filesystem::ifstream;
        using ofstream = ghc::filesystem::ofstream;
        using fstream = ghc::filesystem::fstream;
    }
#endif

and in the implementation hiding cpp, you might use (before any include that includes ghc/fs_fwd.hpp to take precedence:

#if _MSVC_LANG >= 201703L || __cplusplus >= 201703L && defined(__has_include)
    // ^ Supports MSVC prior to 15.7 without setting /Zc:__cplusplus to fix __cplusplus
    // _MSVC_LANG works regardless. But without the switch, the compiler always reported 199711L: https://blogs.msdn.microsoft.com/vcblog/2018/04/09/msvc-now-correctly-reports-__cplusplus/
    #if __has_include(<filesystem>) // Two stage __has_include needed for MSVC 2015 and per https://gcc.gnu.org/onlinedocs/cpp/_005f_005fhas_005finclude.html
        #define GHC_USE_STD_FS

        // Old Apple OSs don't support std::filesystem, though the header is available at compile
        // time. In particular, std::filesystem is unavailable before macOS 10.15, iOS/tvOS 13.0,
        // and watchOS 6.0.
        #ifdef __APPLE__
            #include <Availability.h>
            // Note: This intentionally uses std::filesystem on any new Apple OS, like visionOS
            // released after std::filesystem, where std::filesystem is always available.
            // (All other __<platform>_VERSION_MIN_REQUIREDs will be undefined and thus 0.)
            #if __MAC_OS_X_VERSION_MIN_REQUIRED && __MAC_OS_X_VERSION_MIN_REQUIRED < 101500 \
             || __IPHONE_OS_VERSION_MIN_REQUIRED && __IPHONE_OS_VERSION_MIN_REQUIRED < 130000 \
             || __TV_OS_VERSION_MIN_REQUIRED && __TV_OS_VERSION_MIN_REQUIRED < 130000 \
             || __WATCH_OS_VERSION_MAX_ALLOWED && __WATCH_OS_VERSION_MAX_ALLOWED < 60000
                #undef GHC_USE_STD_FS
            #endif  
        #endif
    #endif
#endif

#ifndef GHC_USE_STD_FS
    #include "fs_impl.hpp"
#endif

ℹ️ Hint: There are additional helper headers, named ghc/fs_std_fwd.hpp and ghc/fs_std_impl.hpp that use this technique, so you can simply include them if you want to dynamically select the filesystem implementation.

Git Submodule and CMake

Starting from v1.1.0, it is possible to add ghc::filesystem as a git submodule, add the directory to your CMakeLists.txt with add_subdirectory() and then simply use target_link_libraries(your-target ghc_filesystem) to ensure correct include path that allow #include <ghc/filesystem.hpp> to work.

The CMakeLists.txt offers a few options to customize its behavior:

  • GHC_FILESYSTEM_BUILD_TESTING - Compile tests, default is OFF when used as a submodule, else ON.
  • GHC_FILESYSTEM_BUILD_EXAMPLES - Compile the examples, default is OFF when used as a submodule, else ON.
  • GHC_FILESYSTEM_WITH_INSTALL - Add install target to build, default is OFF when used as a submodule, else ON.
  • GHC_FILESYSTEM_BUILD_STD_TESTING - Compile std_filesystem_test, the variant of the test suite running against std::filesystem, defaulting to GHC_FILESYSTEM_BUILD_TESTING. This is only done if the compiler is detected as being able to do it.
  • GHC_FILESYSTEM_TEST_COMPILE_FEATURES can be set to a list of features to override CMAKE_CXX_COMPILE_FEATURES when the detection of C++17 or C++20 for additional tests is not working (e.g. cxx_std_20 to enforce building a filesystem_test_cpp20 with C++20).

Bazel

Please use hedronvision/bazel-cc-filesystem-backport, which will automatically set everything up for you.

Versioning

There is a version macro GHC_FILESYSTEM_VERSION defined in case future changes might make it needed to react on the version, but I don't plan to break anything. It's the version as decimal number (major * 10000 + minor * 100 + patch).

ℹ️ Note: Only even patch versions will be used for releases and odd patch version will only be used for in between commits while working on the next version.

Documentation

There is almost no documentation in this release, as any std::filesystem documentation would work, besides the few differences explained in the next section. So you might head over to https://en.cppreference.com/w/cpp/filesystem for a description of the components of this library.

When compiling with C++11, C++14 or C++17, the API is following the C++17 standard, where possible, with the exception that std::string_view parameters are only supported on C++17. When Compiling with C++20, ghc::filesysytem defaults to the C++20 API, with the char8_t and std::u8string interfaces and the deprecated fs::u8path factory method.

ℹ️ Note: If the C++17 API should be enforced even in C++20 mode, use the define GHC_FILESYSTEM_ENFORCE_CPP17_API. Even then it is possible to create fws::path from std::u8string but fs::path::u8string() and fs::path::generic_u8string() return normal UTF-8 encoded std::string instances, so code written for C++17 could still work with ghc::filesystem when compiled with C++20.

The only additions to the standard are documented here:

ghc::filesystem::ifstream, ghc::filesystem::ofstream, ghc::filesystem::fstream

These are simple wrappers around std::ifstream, std::ofstream and std::fstream. They simply add an open() method and a constructor with an ghc::filesystem::path argument as the fstream variants in C++17 have them.

ghc::filesystem::u8arguments

This is a helper class that currently checks for UTF-8 encoding on non-Windows platforms but on Windows it fetches the command line arguments as Unicode strings from the OS with

::CommandLineToArgvW(::GetCommandLineW(), &argc)

and then converts them to UTF-8, and replaces argc and argv. It is a guard-like class that reverts its changes when going out of scope.

So basic usage is:

namespace fs = ghc::filesystem;

int main(int argc, char* argv[])
{
    fs::u8arguments u8guard(argc, argv);
    if(!u8guard.valid()) {
        std::cerr << "Bad encoding, needs UTF-8." << std::endl;
        exit(EXIT_FAILURE);
    }

    // now use argc/argv as usual, they have utf-8 encoding on windows
    // ...

    return 0;
}

That way argv is UTF-8 encoded as long as the scope from main is valid.

Note: On macOS, while debugging under Xcode the code currently will return false as Xcode starts the application with US-ASCII as encoding, no matter what encoding is actually used and even setting LC_ALL in the product scheme doesn't change anything. I still need to investigate this.

Differences

As this implementation is based on existing code from my private helper classes, it derived some constraints of it. Starting from v1.5.0 most of the differences between this and the standard C++17/C++20 API where removed.

LWG Defects

This implementation has switchable behavior for the LWG defects #2682, #2935, #2936 and #2937. The currently selected behavior (starting from v1.4.0) is following #2682, #2936, #2937 but not following #2935, as I feel it is a bug to report no error on a create_directory() or create_directories() where a regular file of the same name prohibits the creation of a directory and forces the user of those functions to double-check via fs::is_directory if it really worked. The more intuitive approach to directory creation of treating a file with that name as an error is also advocated by the newer paper WG21 P1164R0, the revision P1161R1 was agreed upon on Kona 2019 meeting see merge and GCC by now switched to following its proposal (GCC #86910).

Not Implemented on C++ before C++17

// methods in ghc::filesystem::path:
path& operator+=(basic_string_view<value_type> x);
int compare(basic_string_view<value_type> s) const;

These are not implemented under C++11 and C++14, as there is no std::basic_string_view available and I did want to keep this implementation self-contained and not write a full C++17-upgrade for C++11/14. Starting with v1.1.0 these are supported when compiling ghc::filesystem under C++17 of C++20.

Starting with v1.5.2 ghc::filesystem will try to allow the use of std::experimental::basic_string_view where it detects is availability. Additionally if you have a basic_string_view compatible c++11 implementation it can be used instead of std::basic_string_view by defining GHC_HAS_CUSTOM_STRING_VIEW and importing the implementation into the ghc::filesystem namespace with:

namespace ghc {
    namespace filesystem {
        using my::basic_string_view;
    }
}

before including the filesystem header.

Differences in API

To not depend on any external third party libraries and still stay portable and compact, this implementation is following the "UTF-8 Everywhere" philosophy in that all std::string instances will be interpreted the same as std::u8string encoding wise and as being in UTF-8. The std::u16string will be seen as UTF-16 and std::u32string will be seen as Unicode codepoints. Depending on the size of std::wstring characters, it will handle std::wstring as being UTF-16 (e.g. Windows) or char32_t Unicode codepoints (currently all other platforms).

Differences of Specific Interfaces

Starting with v1.5.0 ghc::filesystem is following the C++17 standard in using wchar_t and std::wstring on Windows as the types internally used for path representation. It is still possible to get the old behavior by defining GHC_WIN_DISABLE_WSTRING_STORAGE_TYPE and get filesystem::path::string_type as std::string and filesystem::path::value_type as wchar_t.

If you need to call some Windows API, with v1.5.0 and above, simply use the W-variant of the Windows-API call (e.g. GetFileAttributesW(p.c_str())).

ℹ️ Note: When using the old behavior by defining GHC_WIN_DISABLE_WSTRING_STORAGE_TYPE, use the path::wstring() member (e.g. GetFileAttributesW(p.wstring().c_str())). This gives you the Unicode variant independent of the UNICODE macro and makes sharing code between Windows, Linux and macOS easier and works with std::filesystem and ghc::filesystem.

std::string path::u8string() const;
std::string path::generic_u8string() const;
vs.
std::u8string path::u8string() const;
std::u8string path::generic_u8string() const;

The return type of these two methods is depending on the used C++ standard and if GHC_FILESYSTEM_ENFORCE_CPP17_API is defined. On C++11, C++14 and C++17 or when GHC_FILESYSTEM_ENFORCE_CPP17_API is defined, the return type is std::string, and on C++20 without the define it is std::u8string.

Differences in Behavior

I created a wiki entry about quite a lot of behavioral differences between different std::filesystem implementations that could result in a mention here, but this readme only tries to address the design choice differences between ghc::filesystem and those. I try to update the wiki page from time to time.

Any additional observations are welcome!

fs.path (ref)

Since v1.5.0 the complete inner mechanics of this implementations fs::path where changed to the native format as the internal representation. Creating any mixed slash fs::path object under Windows (e.g. with "C:\foo/bar") will lead clean path with "C:\foo\bar" via native() and "C:/foo/bar" via generic_string() API. On all platforms redundant additional separators are removed, even if this is not enforced by the standard and other implementations mostly not do this.

Additionally this implementation follows the standards suggestion to handle posix paths of the form "//host/path" and USC path on windows also as having a root-name (e.g. "//host"). The GCC implementation didn't choose to do that while testing on Ubuntu 18.04 and macOS with GCC 8.1.0 or Clang 7.0.0. This difference will show as warnings under std::filesystem. This leads to a change in the algorithm described in the standard for operator/=(path& p) where any path p with p.is_absolute() will degrade to an assignment, while this implementation has the exception where *this == *this.root_name() and p == preferred_separator a normal append will be done, to allow:

fs::path p1 = "//host/foo/bar/file.txt";
fs::path p2;
for (auto p : p1) p2 /= p;
ASSERT(p1 == p2);

For all non-host-leading paths the behavior will match the one described by the standard.

Open Issues

Windows

Symbolic Links on Windows

As symbolic links on Windows, while being supported more or less since Windows Vista (with some strict security constraints) and fully since some earlier build of Windows 10, when "Developer Mode" is activated, are at time of writing (2018) rarely used, still they are supported wiit th this implementation.

Permissions

The Windows ACL permission feature translates badly to the POSIX permission bit mask used in the interface of C++17 filesystem. The permissions returned in the file_status are therefore currently synthesized for the user-level and copied to the group- and other-level. There is still some potential for more interaction with the Windows permission system, but currently setting or reading permissions with this implementation will most certainly not lead to the expected behavior.

Release Notes

v1.5.15 (wip)

  • Fix for #166, extension() did return non empty result for the directory name ".."
  • Pull request #163, build support for Haiku (also fixes #159)
  • Pull request #162, fix for directory iterator treating all files subsequent to a symlink as symlink on Windows
  • Pull request #161, the CMake alias ghcFilesystem::ghc_filesystem is now set unconditionally
  • Fix for #160, the cmake config now only sets install targets by default if the project is no subproject, as documented
  • Fix for #157, suppress C4191 warning on MSVC for GetProcAddress casts
  • Fix for #156, on POSIX stem(), filename() and extension() of fs::path would return wrong result if a colon was in the filename
  • Pull request #154, build support for GNU/Hurd
  • Pull request #153, fixed fs::last_write_time(path, time, ec) setter on iOS, tvOS and watchOS
  • Fix for #151, fs::directory_entry::refresh() now, consistently with status() will not throw on symlinks to non-existing targets, but make the entry have file_type::not_found as the type
  • Pull request #149, add version to CMake project and export it
  • Fix for #146, handle EINTR on POSIX directory iteration and file copy to avoid errors on network filesystems
  • Pull request #145, fix for Y2038 bug in timeToFILETIME on Windows
  • Pull request #144, fs::copy_file() now also copies the permissions
  • Pull request #143, fix for fs::copy_file() ignoring the skip_existing option.
  • Fix for #142, removed need for GHC_NO_DIRENT_D_TYPE on systems that don't support dirent::d_type and fixed build configuration and tests to support Solaris as new platform.
  • Pull request #138, if the platform uses the POSIX backend and has no PATH_MAX, one is defined.
  • Pull request #137, update of Catch2 to version v2.13.7
  • Added macOS 11 to the automatically tested platforms.
  • Pull request #136, the Windows implementation used some unnecessary expensive shared pointer for resource management and these where replaced by a dedicated code.
  • Fix for #132, pull request #135, fs::remove_all now just deletes symbolic links instead of following them.
  • Pull request #133, fix for fs::space where a numerical overflow could happen in a multiplication.
  • Replaced travis-ci.org with GitHub Workflow for the configurations: Ubuntu 20.04: GCC 9.3, Ubuntu 18.04: GCC 7.5, GCC 8.4, macOS 10.15: Xcode 12.4, Windows 10: Visual Studio 2019
  • Fix for #125, where fs::create_directories on Windows no longer breaks on long filenames.
  • Fix for #124, ghc::filesystem treated mounted folder/volumes erroneously as symlinks, leading fs::canonical to fail on paths containing those.
  • Fix for #122, incrementing the recursive_directory_iterator will not try to enter dead symlinks.
  • Fix for #121, on Windows backend the fs::remove failed when the path pointed to a read-only entry, see also (microsoft/STL#1511) for the corresponding issue in std::fs on windows.
  • Fix for #119, added missing support for char16_t and char32_t and on C++20 char8_t literals.
  • Pull request #118, when running tests as root, disable tests that would not work.
  • Pull request #117, added checks to tests to detect the clang/libstdc++ combination.
  • Fix for #116, internal macro GHC_NO_DIRENT_D_TYPE allows os detection to support systems without the dirent.d_type member, experimental first QNX compile support as initial use case, fixed issue with filesystems returning DT_UNKNOWN (e.g. reiserfs).
  • Pull request #115, added string_view support when clang with libstdc++ is detected.
  • Fix for #114, for macOS the pre-Catalina deployment target detection worked only if <Availability.h> was included before <ghc/fs_std.hpp> or <ghc/fs_std_fwd.hpp>/<ghc/fs_std_impl.hpp>.
  • Fix for #113, the use of standard chapter numbers was misleading since C++17 and C++20 std::filesystem features are supported, and was replaced by the tag-like chapter names that stay (mostly) consistent over the versions.
  • Pull request #112, lots of cleanup work on the readme, thanks!
  • Enhancement for #111, further optimization of directory iteration, performance for recursive_directory_iterator over large trees now somewhere between libc++ and libstdc++.
  • Enhancement for #110, ghc::filesystem now has preliminary support for Cygwin. Changes where made to allow the tests to compile and run successfully (tested with GCC 10.2.0), feedback and additional PRs welcome as it is currently not part of the CI configuration.
  • Pull request #109, various spelling errors in error messages and comments fixed.
  • Pull request #108, old style casts removed.
  • Fix for #107, the error handling for status calls was suppressing errors on symlink targets.
  • Pull request #106, fixed detection of AppleClang for compile options.
  • Pull request #105, added option GHC_FILESYSTEM_BUILD_STD_TESTING to override additional build of std::filesystem versions of the tests for comparison and the possibility to use GHC_FILESYSTEM_TEST_COMPILE_FEATURES to prefill the used compile features defaulting to CMAKE_CXX_COMPILE_FEATURES when not given.
  • Enhancement #104, on POSIX backend: optimized reuse of status information and reduced directory_entry creation leads to about 20%-25% in tests with recursive_directory_iterator over a larger directory tree.
  • Pull request #103, wchar_t was not in the list of supported char types on non-Windows backends.
  • Pull request #102, improved string_view support makes use of <string_view> or <experimental/string_view> when available, and allows use of custom basic_string_view implementation when defining GHC_HAS_CUSTOM_STRING_VIEW and importing the string view into the ghc::filesystem namespace before including filesystem header.
  • Pull request #101, fix for #100, append and concat type of operations on path called redundant conversions.
  • Pull request #98, on older linux variants (GCC 7/8), the comparison std::filesystem tests now link with -lrt to avoid issues.
  • Fix for #97, on BTRFS the test case for fs::hard_link_count failed due to the filesystems behavior, the test case was adapted to take that into account.
  • Pull request #96, the export attribute defines GHC_FS_API and GHC_FS_API_CLASS are now honored when when set from outside to allow override of behavior.
  • Fix for #95, the syntax for disabling the deprecated warning in tests in MSVC was wrong.
  • Pull request #93, now the CMake configuration file is configured and part of the make install files.
  • Fix for #91, the way the CMake build options GHC_FILESYSTEM_BUILD_TESTING, GHC_FILESYSTEM_BUILD_EXAMPLES and GHC_FILESYSTEM_WITH_INSTALL where implemented, prohibited setting them from a parent project when using this via add_subdirectory, this fix allows to set them again.
  • Major refactoring for #90, the way, the Windows version of fs::path was originally created from the POSIX based implementation was, by adaption of the incoming and outgoing strings. This resulted in a mutable cache inside fs::pathon Windows, that was inherently not thread-safe, even for const methods. To not add additional patches to a suboptimal solution, this time I reworked the path code to now store native path-representation. This changed a lot of code, but when combined with wchar_t as value_type helped to avoid lots of conversion for calls to Win-API.
    As interfaces where changed, it had to be released in a new minor version. The set of refactorings resulted in the following changes:
    • fs::path::native() and fs::path::c_str() can now be noexcept as the standard mandates
    • On Windows wchar_t is now the default for fs::path::value_type and std::wstring is the default for fs::path::string_type.
    • This allows the implementation to call Win-API without allocating conversions
    • Thread-safety on const methods of fs::path is no longer an issue
    • Some code could be simplified during this refactoring
    • Automatic prefixing of long path on Windows can now be disabled with defining GHC_WIN_DISABLE_AUTO_PREFIXES, for all other types of prefixes or namespaces the behavior follows that of MSVC std::filesystem::path
    • In case the old char/std::string based approach for Windows is still needed, it can be activated with GHC_WIN_DISABLE_WSTRING_STORAGE_TYPE
  • Enhancement for #89, fs::file_status now supports operator== introduced in std::filesystem with C++20.
  • Refactoring for #88, fs::path::parent_path() had a performance issue, as it was still using a loop based approach to recreate the parent from elements. This created lots of temporaries and was too slow especially on long paths.
  • Enhancements for #71, when compiled with C++20:
    • char8_t and std::u8string are supported where Source is the parameter type
    • fs::path::u8string() and fs::path::generic_u8string() now return a std::u8string
    • The spaceship operator <=> is now supported for fs::path
    • With the define GHC_FILESYSTEM_ENFORCE_CPP17_API ghc::filesystem will fall back to the old fs::path::u8string() and fs::path::generic_u8string() API if preferred
  • Bugfix for fs::proximate(p, ec) where the internal call to fs::current_path() was not using the error_code variant, throwing possible exceptions instead of setting ec.
  • Enhancement LWG_2936_BEHAVIOUR is now on by default.
  • Some cleanup work to reduce preprocessor directives for better readability and remove unneeded template specializations.
  • Fix for #81, fixed issues with handling Source parameters that are string views.
  • Fix for #79, the bit operations for filesystem bitmasks that should be are now constexpr.
  • Refactoring for #78, the dynamic switching helper includes are now using __MAC_OS_X_VERSION_MIN_REQUIRED to ensure that std::filesystem is only selected on macOS if the deployment target is at least Catalina.
  • Bugfix for #77, the directory_iterator and the recursive_directory_iterator had an issue with the skip_permission_denied option, that leads to the inability to skip SIP protected folders on macOS.
  • Enhancement for #76, _MSVC_LANG is now used when available, additionally to __cplusplus, in the helping headers to allow them to work even when /Zc:__cplusplus is not used.
  • Bugfix for #75, NTFS reparse points to mapped volumes where handled incorrect, leading to false on fs::exists or not-found-errors on fs::status. Namespaced paths are not filtered anymore.
  • Pull request #74, on Windows symlink evaluation used the wrong reparse struct information and was not handling the case of relative paths well, thanks for the contribution.
  • Refactoring for #73, enhanced performance in path handling. the changes lead to much fewer path/string creations or copies, speeding up large directory iteration or operations on many path instances.
  • Bugfix for #72, the TestAllocator in filesystem_test.cpp was completed to fulfill the requirements to build on CentOS 7 with devtoolset-9. CentOS 7 and CentOS 8 are now part of the CI builds.
  • Bugfix for #70, root names are now case insensitive on Windows. This fix also adds the new behavior switch LWG_2936_BEHAVIOUR that allows to enable post C++17 fs::path::compare behavior, where the comparison is as if it was an element wise path comparison as described in LWG 2936 and C++20 [fs.path.compare]. It is default off in v1.3.6 and will be default starting from v1.4.0 as it changes ordering.
  • Pull request #69, use wchar_t versions of std::fstream from ghc::filesystem::fstream wrappers on Windows if using GCC with libc++.
  • Bugfix for #68, better handling of permission issues for directory iterators when using fs::directory_options::skip_permission_denied and initial support for compilation with emscripten.
  • Refactoring for #66, unneeded shared_ptr guards where removed and the file handles closed where needed to avoid unnecessary allocations.
  • Bugfix for #63, fixed issues on Windows with clang++ and C++17.
  • Pull request #62, various fixes for better Android support, thanks for the PR
  • Pull request #61, ghc::filesystem now supports use in projects with disabled exceptions. API signatures using exceptions for error handling are not available in this mode, thanks for the PR (this resolves #60 and #43)
  • Bugfix for #58, on MinGW the compilation could fail with an error about an undefined ERROR_FILE_TOO_LARGE constant.
  • Bugfix for #56, fs::lexically_relative didn't ignore trailing slash on the base parameter, thanks for PR #57.
  • Bugfix for #55, fs::create_directories returned true when nothing needed to be created, because the directory already existed.
  • Bugfix for #54, error_code was not reset, if cached result was returned.
  • Pull request #53, fix for wrong handling of leading whitespace when reading fs::path from a stream.
  • Pull request #52, an ARM Linux target is now part of the CI infrastructure with the service of Drone CI.
  • Pull request #51, FreeBSD is now part of the CI infrastructure with the service of Cirrus CI.
  • Pull request #50, adaptive cast to timespec fields to avoid warnings.
  • Important: ghc::filesystem is re-licensed from BSD-3-Clause to MIT license. (see #47)
  • Pull request #46, suppresses unused parameter warning on Android.
  • Bugfix for #44, fixes for warnings from newer Xcode versions.
  • The Visual Studio 2019 compiler, GCC 9.2 and Clang 9.0 where added to the CI configuration.
  • Bugfix for #41, fs::rename on Windows didn't replace an existing regular file as required by the standard, but gave an error. New tests and a fix as provided in the issue was implemented.
  • Bugfix for #39, for the forwarding use via fs_fwd.hpp or fs_std_fwd.hpp there was a use of DWORD in the forwarding part leading to an error if Windows.h was not included before the header. The tests were changed to give an error in that case too and the useage of DWORD was removed.
  • Bugfix for #38, casting the return value of GetProcAddress gave a warning with -Wcast-function-type on MSYS2 and MinGW GCC 9 builds.
  • Pull request #30, the CMakeLists.txt will automatically exclude building examples and tests when used as submodule, the configuration options now use a prefixed name to reduce risk of conflicts.
  • Pull request #24, install target now creates a ghcFilesystemConfig.cmake in ${CMAKE_INSTALL_LIBDIR}/cmake/ghcFilesystem for find_package that exports a target as ghcFilesystem::ghc_filesystem.
  • Pull request #31, fixes error: redundant redeclaration of 'constexpr' static data member deprecation warning in C++17 mode.
  • Pull request #32, fixes old-style-cast warnings.
  • Pull request #34, fixes TOCTOU situation on fs::create_directories, thanks for the PR!
  • Feature #35, new CMake option to add an install target GHC_FILESYSTEM_WITH_INSTALL that is defaulted to OFF if ghc::filesystem is used via add_subdirectory.
  • Bugfix for #33, fixes an issue with fs::path::lexically_normal() that leaves a trailing separator in case of a resulting path ending with .. as last element.
  • Bugfix for #36, warnings on Xcode 11.2 due to unhelpful references in path element iteration.
  • Pull request #23, tests and examples can now be disabled in CMake via setting BUILD_TESTING and BUILD_EXAMPLES to NO, OFF or FALSE.
  • Pull request #25, missing specialization for construction from std::string_view when available was added.
  • Additional test case when std::string_view is available.
  • Bugfix for #27, the fs::path::preferred_separator declaration was not compiling on pre C++17 compilers and no test accessed it, to show the problem. Fixed it to an construction C++11 compiler should accept and added a test that is successful on all combinations tested.
  • Bugfix for #29, stricter warning settings where chosen and resulting warnings where fixed.
  • Enabled stronger warning switches and resulting fixed issues on GCC and MinGW
  • Bugfix for #22, the fs::copy_options where not forwarded from fs::copy to fs::copy_file in one of the cases.
  • Fix for (#21), when compiling on Alpine Linux with musl instead of glibc, the wrong strerror_r signature was expected. The complex preprocessor define mix was dropped in favor of the usual dispatch by overloading a unifying wrapper.
  • Added MinGW 32/64 and Visual Studio 2015 builds to the CI configuration.
  • Fixed additional compilation issues on MinGW.
  • Pull request (#13), set minimum required CMake version to 3.7.2 (as in Debian 8).
  • Pull request (#14), added support for a make install target.
  • Bugfix for (#15), the forward/impl way of using ghc::filesystem missed a <vector> include in the windows case.
  • Bugfix for (#16), VS2019 didn't like the old size dispatching in the utf8 decoder, so it was changed to a sfinae based approach.
  • New feature (#17), optional support for standard conforming wchar_t/std::wstring interface when compiling on Windows with defined GHC_WIN_WSTRING_STRING_TYPE, this is default when using the ghc/fs_std*.hpp header, to enhance compatibility.
  • New feature (#18), optional filesystem exceptions/errors on Unicode errors with defined GHC_RAISE_UNICODE_ERRORS (instead of replacing invalid code points or UTF-8 encoding errors with the replacement character U+FFFD).
  • Pull request (#20), fix for file handle leak in fs::copy_file.
  • Coverage now checked in CI (~95% line coverage).
  • Additional Bugfix for (#12), error in old unified readdir/readdir_r code of fs::directory_iterator; as readdir_r is now deprecated, I decided to drop it and the resulting code is much easier, shorter and due to more refactoring faster
  • Fix for crashing unit tests against MSVC C++17 std::filesystem
  • Travis-CI now additionally test with Xcode 10.2 on macOS
  • Some minor refactorings
  • Bugfix for (#11), fs::path::lexically_normal() had some issues with ".."-sequences.
  • Bugfix for (#12), fs::recursive_directory_iterator could run into endless loops, the methods depth() and pop() had issues and the copy behavior and input_iterator_tag conformance was broken, added tests
  • Restructured some CMake code into a macro to ease the support for C++17 std::filesystem builds of tests and examples for interoperability checks.
  • Some fixes on Windows tests to ease interoperability test runs.
  • Reduced noise on fs::weakly_canonical() tests against std::fs
  • Added simple du example showing the recursive_directory_iterator used to add the sizes of files in a directory tree.
  • Added error checking in fs::file_time_type test helpers
  • fs::copy() now conforms LWG #2682, disallowing the use of `copy_option::create_symlinks' to be used on directories
  • Restructuring of the project directory. The header files are now using hpp as extension to be marked as c++ and they where moved to include/ghc/ to be able to include by <ghc/filesystem.hpp> as the former include name might have been to generic and conflict with other files.
  • Better CMake support: ghc::filesystem now can be used as a submodul and added with add_subdirectory and will export itself as ghc_filesystem target. To use it, only target_link_libraries(your-target ghc_filesystem) is needed and the include directories will be set so #include <ghc/filesystem.hpp> will be a valid directive. Still you can simply only add the header file to you project and include it from there.
  • Enhancement (#10), support for separation of implementation and forwarded api: Two additional simple includes are added, that can be used to forward ghc::filesystem declarations (fs_fwd.hpp) and to wrap the implementation into a single cpp (fs_impl.hpp)
  • The std::basic_string_view variants of the fs::path api are now supported when compiling with C++17.
  • Added CI integration for Travis-CI and Appveyor.
  • Fixed MinGW compilation issues.
  • Added long filename support for Windows.
  • Bugfix for (#9), added missing return statement to ghc::filesystem::path::generic_string()
  • Added checks to hopefully better compile against Android NDK. There where no tests run yet, so feedback is needed to actually call this supported.
  • filesystem.h was renamed filesystem.hpp to better reflect that it is a c++ language header.
  • Bugfix for (#6), where ghc::filesystem::remove() and ghc::filesystem::remove_all() both are now able to remove a single file and both will not raise an error if the path doesn't exist.
  • Merged pull request (#7), a typo leading to setting error code instead of comparing it in ghc::filesystem::remove() under Windows.
  • Bugfix for ((#8), the Windows version of ghc::filesystem::directory_iterator now releases resources when reaching end() like the POSIX one does.
  • Bugfix for (#4), missing error_code propagation in ghc::filesystem::copy() and ghc::filesystem::remove_all fixed.
  • Bugfix for (#5), added missing std namespace in ghc::filesystem::recursive_directory_iterator::difference_type.
  • Bugfix for (#3), fixed missing inlines and added test to ensure including into multiple implementation files works as expected.
  • Building tests with -Wall -Wextra -Werror and fixed resulting issues.
  • Updated catch2 to v2.4.0.
  • Refactored fs.op.permissions test to work with all tested std::filesystem implementations (gcc, clang, msvc++).
  • Added helper class ghc::filesystem::u8arguments as argv converter, to help follow the UTF-8 path on windows. Simply instantiate it with argc and argv and it will fetch the Unicode version of the command line and convert it to UTF-8. The destructor reverts the change.
  • Added examples folder with hopefully some usefull example usage. Examples are tested (and build) with ghc::filesystem and C++17 std::filesystem when available.
  • Starting with this version, only even patch level versions will be tagged and odd patch levels mark in-between non-stable wip states.
  • Tests can now also be run against MS version of std::filesystem for comparison.
  • Added missing fstream include.
  • Removed non-conforming C99 timespec/timeval usage.
  • Fixed some integer type mismatches that could lead to warnings.
  • Fixed chrono conversion issues in test and example on clang 7.0.0.
  • Bugfix: ghc::filesystem::canonical now sees empty path as non-existant and reports an error. Due to this ghc::filesystem::weakly_canonical now returns relative paths for non-existant argument paths. (#1)
  • Bugfix: ghc::filesystem::remove_all now also counts directories removed (#2)
  • Bugfix: recursive_directory_iterator tests didn't respect equality domain issues and dereferencapable constraints, leading to fails on std::filesystem tests.
  • Bugfix: Some noexcept tagged methods and functions could indirectly throw exceptions due to UFT-8 decoding issues.
  • std_filesystem_test is now also generated if LLVM/clang 7.0.0 is found.

This was the first public release version. It implements the full range of C++17 std::filesystem, as far as possible without other C++17 dependencies.

filesystem's People

Contributors

actboy168 avatar albert-github avatar begasus avatar bugdea1er avatar cookieplmonster avatar cpsauer avatar cybik avatar despair86 avatar dvzrv avatar fenglc avatar gulrak avatar jnhyatt avatar jonasvautherin avatar jwnimmer-tri avatar kkaefer avatar kleuter avatar ligfx avatar nightlark avatar noexcept avatar okhowang avatar philbucher avatar phprus avatar rikyoz avatar soilros avatar sthibaul avatar tohammer avatar touraill-adobe avatar vgeorgiev avatar zer0xff avatar zero9178 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

filesystem's Issues

create_directories should return false when the path already exists

Describe the bug
For the methods filesystem::create_directories, the document N4687 says (see 30.10.15.6)

Returns: true if a new directory was created, otherwise false. The signature with argument ec returns false if an error occurs.

However, the current implementation in v1.3.0 returns true everytime.

To Reproduce

// should return false as the folder already exists
std::cout << boolalpha << std::filesystem::create_directories(std::filesystem::current_path()) << std::endl

Expected behavior
In case the path already exists and this is a directory, the returned value should be false. This is the behaviour implemented within VS2019 (16.4) and Xcode 11.

Additional context
A way to fix this issue could be to test if the path already exists. For example

// Around line 3400
GHC_INLINE bool create_directories(const path& p, std::error_code& ec) noexcept
{
    path current;
    ec.clear();
    // BEGIN_PROPOSITION
    std::error_code tec;
    auto fs = status(p, tec);
    if (status_known(fs) && exists(fs) && is_directory(fs)) {
        return false;
    }
    // END_PROPOSITION
    for (path::string_type part : p) {
        current /= part;
//...

Use After Free Error // Heap Overflow

Describe the bug
When using a debug build (-DCMAKE_BUILD_TYPE=Debug) and a recursive directory iterator. If the directory to iterate is fairly large, a use after free or heap overflow occurs

To Reproduce

  1. Debug Build - Enable address sanitizer
  2. Use a recursive directory iterator on a directory that has at least 12 levels of sub
    directories

Expected behavior
After some execution you should see a crash located at filesystem.hpp:4570:

==32507==ERROR: AddressSanitizer: unknown-crash on address 0x62d0011f6328 at pc 0x555e71d4f2d5 bp 0x7ffd5f191200 sp 0x7ffd5f1911f0
READ of size 280 at 0x62d0011f6328 thread T0
.
.
.
0x62d0011f6430 is located 0 bytes to the right of 32816-byte region [0x62d0011ee400,0x62d0011f6430)
allocated by thread T0 here:
    #0 0x7f25c1f1db50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f25c064c9b5 in opendir (/lib/x86_64-linux-gnu/libc.so.6+0xdf9b5)

SUMMARY: AddressSanitizer: unknown-crash {PATH}/filesystem.hpp:4570 in i_readdir_r
Shadow bytes around the buggy address:
  0x0c5a80236c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c5a80236c60: 00 00 00 00 00[00]00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c80: 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==32507==ABORTING

Additional context
For some reason it is listed as an unknown-crash but it has all the symptoms of a heap overflow or a use-after-free error.

Warnings when building with Xcode 11.2

Describe the bug
I get the following warning when building with Xcode 11.2:

Loop variable 's' has type 'const ghc::filesystem::path::string_type &' (aka 'const basic_string<char> &') but is initialized with type 'const ghc::filesystem::path' resulting in a copy

Inside path::parent_path implementation, this line:
for (const string_type& s : input_iterator_range<iterator>(begin(), --end())) {

Loop variable 's' has type 'const ghc::filesystem::path::string_type &' (aka 'const basic_string<char> &') but is initialized with type 'const ghc::filesystem::path' resulting in a copy

Inside path::lexically_normal implementation, this line:
for (const string_type& s : *this) {

Loop variable 'part' has type 'const path::string_type &' (aka 'const basic_string<char> &') but is initialized with type 'const ghc::filesystem::path' resulting in a copy
Inside create_directories implementation, this line:
for (const path::string_type& part : p) {

Would it be possible to remove them?

Building with clang c++17, wide chars

Describe the bug
Building in Visual Studio with the wide character flag: GHC_WIN_WSTRING_STRING_TYPE
With clang 11, and c++17, I have these warnings/errors:

1>In file included from D:\filesystem\test\filesystem_test.cpp:62: 1>D:\filesystem\include\ghc/filesystem.hpp(4509,50): error : suggest braces around initialization of subobject [-Werror,-Wmissing-braces] 1>D:\filesystem\include\ghc/filesystem.hpp(4510,42): error : suggest braces around initialization of subobject [-Werror,-Wmissing-braces] 1>D:\filesystem\include\ghc/filesystem.hpp(4511,46): error : suggest braces around initialization of subobject [-Werror,-Wmissing-braces] 1>D:\filesystem\include\ghc/filesystem.hpp(1649,13): error : no matching function for call to 'toUtf8' 1>D:\filesystem\include\ghc/filesystem.hpp(2628,29): message : in instantiation of function template specialization 'ghc::filesystem::path::path<std::basic_string_view<wchar_t, std::char_traits<wchar_t> >, ghc::filesystem::path>' requested here 1>D:\filesystem\include\ghc/filesystem.hpp(1536,20): message : candidate template ignored: could not match 'basic_string' against 'basic_string_view' 1>D:\filesystem\include\ghc/filesystem.hpp(1542,20): message : candidate template ignored: could not match 'basic_string' against 'basic_string_view' 1>D:\filesystem\include\ghc/filesystem.hpp(1571,20): message : candidate template ignored: could not match 'basic_string' against 'basic_string_view' 1>D:\filesystem\include\ghc/filesystem.hpp(1581,20): message : candidate template ignored: could not match 'const charT *' against 'std::basic_string_view<wchar_t, std::char_traits<wchar_t> >'

Please find my Visual Studio project file for filesystem_test attached. (just trim the .txt extension)

Note! clang compiles with only the first three warnings with c++11 (without the actual basic_string_view error)

To create the Visual Studio solution and projects, I ran:
cmake -G "Visual Studio 15 2017 Win64"

I am working with latest on master: commit 9a047b9

To Reproduce
Compile as above

Expected behavior
Compiles, no errors. Warnings are negotiable. :)

Additional context

Thank you for creating this library. I was working with the Windows filesystem, which I found did not support path appending as expected. And I want to have standard cross-platform support.

filesystem_test.vcxproj.txt

Compiler warnings

Is your feature request related to a problem? Please describe.
Not a problem, just some compiler warnings I got in the latest macOS with the latest Xcode.

Describe the solution you'd like
Comment and possibly act on these:

filesystem.hpp:3561:44: warning: zero as null pointer constant [-Wzero-as-null-pointer-constant]
if (::getcwd(buffer.get(), pathlen) == NULL) {
^~~~
nullptr

filesystem.hpp:2559:12: note: call 'std::move' explicitly to avoid copying on older compilers
return fn;
^~
std::move(fn)

filesystem.hpp:2559:12: warning: prior to the resolution of a defect report against ISO C++11, local variable 'fn' would have been copied despite being returned by name, due to its not matching the function return type ('ghc::filesystem::path' vs 'ghc::filesystem::path::impl_string_type' (aka 'basic_string')) [-Wreturn-std-move-in-c++11]
return fn;
^~

Performance issues on Windows

ghc::filesystem is slower than std::filesystem on Windows when working with directories that contain many files.
I tested it in directory which contains about 16600 files.
Enumerating files via ghc::filesystem takes about 50-60 seconds on my PC (NTFS, HDD).
Enumerating files via std::filesystem used from example code takes 2.5-5 seconds (same directory).

  fs::path object_path = fs::u8path(start_path);

  for (auto &entry_path : fs::directory_iterator(object_path)) {
    fs::file_status entry_status = fs::status(entry_path);
    // some simple operations like push names to list
  }

I think the problem corresponds to non-optimized iterator operators and 'status' operations.
Please look on call stack on image. Simple iterator increment operation requires call for 'status', which processed all data gathering, strings assigning, etc. So, for single pass for one file, 'status' operation called so many times with gathering a lot of not necessary information.

image

Maybe caching of file names, extensions, root paths, etc. instead calculation on each call, will make library faster.

copy and remove_all with error_code throws

First of all - finally one working impl. Great job!

Describe the bug
copy and remove_all with error_code throws exceptions due to the error code not being propagated to the directory_iterator's constructor when iterating over the files.

        for (const directory_entry& x : directory_iterator(from)) {

    for (const directory_entry& de : directory_iterator(p)) {

shoud be

        for (const directory_entry& x : directory_iterator(from, ec)) {

    for (const directory_entry& de : directory_iterator(p, ec)) {

Expected behavior
These should not throw when using the error_code api

Missing include <vector> in forward declarations

Seems like std::vector is being used but not included. This is with forward declaration by including ghc/fs_std_fwd.hpp and happens only on Windows.

Check in ghc/filesystem.hpp:1009

Solved by adding #include just before #endif // GHC_EXPAND_IMPL

Web platform support using emscripten

First of all, thanks for this extremely useful project!
I would like to have it run in the browser by compiling it to web-assembly using emscripten. Emscripten itself does not currently provide support for std::filesystem and it seems to me this might be in scope for your library.

I was able to have it compile using emscripten by simply adding

#elif defined(__EMSCRIPTEN__)
#define GHC_OS_WEB

to your platform determination part in filesystem.hpp.

It was immediately usable, but the following simple example gives a crash, so I suppose there may be some more effort necessary to get things running correctly:

for(auto& p: fs::recursive_directory_iterator("."))
        std::cout << p.path() << '\n';

fails after two prints

./proc
./proc/self
Operation not permitted: './proc/self'

Note that the "filesystem" in this case is the virtual browser filesystem -- I have not yet researched the nuances that might make the browser platform different from others.

Unusual shared_ptr use in copy_file

The copy_file function uses shared_ptr as a RAII wrapper around close through a custom deleter.

std::shared_ptr<void> guard_in(nullptr, [in](void*) { ::close(in); });

std::shared_ptr<void> guard_out(nullptr, [out](void*) { ::close(out); });

It seems like this code is using shared_ptr as a mechanism for RAII. shared_ptr isn't necessarily free: it will likely allocate storage for the lamdas and their captures because it has to type erase the custom deleter internally. I find it confusing to see shared_ptr used this way. This is not really its intended use, so the intent of the code wasn't really clear on my first read.

I don't see why RAII is advantageous here: it would make sense if any of the functions called in copy_file could throw, because the destructor would ensure close is called. However, the function is noexcept. If anything throws in here, the application will be terminated anyway.

This is just something I noticed while I was digging around the source. I could submit a PR addressing these comments if you agree with them. Otherwise, I'd be really interested to hear why the code is written like this.

Cheers!

lexically_relative should ignore trailing slash on base

Expected behavior

fs::path("a/b").lexically_relative("a") == "b";
fs::path("a/b").lexically_relative("a/") == "b";

Actual behavior

fs::path("a/b").lexically_relative("a/") == "b";
fs::path("a/b").lexically_relative("a/") == "../b";

(tested on AppleClang 11.0.0 on macOS 10.15.3)

error_code not cleared in directory_entry::exists?

Describe the bug
I use the class filesystem::directory_entry and I am checking some attributes using the methods is_regular_file(), is_directory(), and exists(std::error_code &). For the later, I also check the content of the error_code object. In case, I reuse the same std::error_code object and one of the previous path is not valid, then the content of the reused std::error_code object is always false (i.e. the value() is not equal to 0) for the other paths even if the method exists returns true. Based on C++ reference, the implementation proposed by Visual Studio, or the one included in XCode 11 with macOS 10.15, the error_code object should be cleared. I also checked the document N4687 but it is not mentioned what to do in this case (see 30.10.12.3).

To Reproduce

    std::error_code ec;
    auto d1 = std::filesystem::directory_entry();
    std::cout << d1.exists(ec) << std::endl; // should be false
    std::cout << ec.value() << std::end; // should not be equal to 0
    auto d2 = std::filesystem::directory_entry(std::filesystem::current_path());
    std::cout << d2.exists(ec) << std::endl; // should be true
    std::cout << ec.value() << std::end; // expected to be equal to 0 but this is not!

Expected behavior
The std::error_code object should be cleared if the directory entry exists

Additional context
From my comprehension of the source code, the problem is in the method directory_entry::status(std::error_code &) (see line 4699).

In case, the status' type is not none, then the std::error_code object should be cleared. I proposed the following implementation to fix the issue.

GHC_INLINE file_status directory_entry::status(std::error_code& ec) const noexcept
{
    if (_status.type() != file_type::none) {
        ec.clear();
        return _status;
    }
    return filesystem::status(path(), ec);
}

fs::remove_all cannot delete non-directory files

Describe the bug
fs::remove_all should allow deletion of a file.

To Reproduce

assert(fs::remove_all("test.txt") == 1)
assert(fs::remove_all("test.txt") == 0)

Expected behavior
But now it will throw an exception.

Additional context

Optional support of the more standard conforming wchar_t/wstring interface when compiling on Windows

The C++17 standard demands for std::filesystem::path: "For Windows-based operating systems, value_type is wchar_t and preferred_separator is the backslash character (L’\\’)." (30.10.8-2)

While I still don't really like the decision, I will try to help those in need of a more std::filesystem conforming implementation for C++11/14 by implementing an option to build ghc::filesystem
with ghc::filesystem::path::value_type as wchar_t and ghc::filesystem::path::string_type as std::basic_string<wchar_t> on Windows.

It might be a good idea to actually activate that option from the helper header files that try to detect std::filesystem and include ghc::filesystem only when no standard version is available, as in these situations the resulting fs::path should have the same interface.

Incorrect handling of NTFS Reparse points to volumes

Describe the bug

exists() returns false when given a path to an NTFS reparse point that points to a volume.
Additionally, status() returns not_found for such path.

To Reproduce

In Disk Manager, format a new volume. Instead of assigning a drive-letter, assign a mount-point at an empty directory on an NTFS volume.

Call exists() and pass in the path to the mount point.

Observe it returning false, even though the volume exists and is mounted.

Expected behavior

exists() is expected to return true.

Additional context

The problem arises from resolving a "symlink", in status_ex. The symlink being an IO_REPARSE_TAG_MOUNT_POINT is handled here.

The symlink resolves to something of this form: "\??\Volume{<GUID>}". This path is correct. However, it's assigned into a path object (result = ). The path assign() member function will strip the \??\ prefix, which makes the path incorrect.

Recursing down into another call to status_ex() (here) will then fail saying the path doesn't exist (because it doesn't).

Removing the code that's stripping the prefix fixes the problem:

--- a/filesystem.hpp
+++ b/filesystem.hpp
@@ -1663,9 +1663,9 @@ GHC_INLINE void path::postprocess_path_with_format(path::impl_string_type& p, pa
                         p[0] = '\\';
                     }
                 }
-                else if (detail::startsWith(p, std::string("\\??\\"))) {
-                    p.erase(0, 4);
-                }
+//                else if (detail::startsWith(p, std::string("\\??\\"))) {
+//                    p.erase(0, 4);
+//                }
             }
             for (auto& c : p) {
                 if (c == '\\') {

However, that is likely introducing other issues. Perhaps the best solution would be to have a resolveLink return a plain native sting (instead of path) and also make status_ex() accept a plain string, to avoid round-tripping via path in this case.

Small append differences on Windows, compared with boost

Our unit tests all harness boost::filesystem::path, which has some differences from std::filesystem::path and ghc::filesystem::path.
Are these strictly defined in the standard?

ex1/

p1 = "c:/bbb"
p2 = "";
p3 = p1 / p2;   == "c:/bbb/"    (with ghc)
p3 = p1 / p2;   == "c:/bbb"     (with boost)

ex2/

p1 = "c:/bbb"
p2 = "/ccc";
p3 = p1 / p2;   == "c:/ccc"       (with ghc)
p3 = p1 / p2;   == "c:/bbb/ccc"   (with boost)

If I want the same behaviour, can you recommend how?

I considered making path::append virtual, to:
ex1/ skip appending if the rhs is empty
ex2/ strip the leading or trailing separator on entry

Or add a ghc::filesystem::path member to my path class, instead of inheriting. But this requires forwarders for every path function.

Undeclared identifier DWORD

Hi.
Believe I've found a bug:
Error C2065 'DWORD': undeclared identifier ghc\filesystem.hpp(1089)

Noticed this now (on Windows, vs2019) when upgrading from an older release of your repo.

So I get this when using the forward declarations. Apparently it now relies on having DWORD already defined which wasn't the case before. My application does not include windows.h, hence there's no DWORD.

I see a few options:

  • Change the declaration of make_system_error() to not use DWORD.
  • Include windows.h always, not only for implementation.
  • typedef unsigned long DWORD if not already defined.

Not sure what works best here but the latter two are probably best to avoid if possible.

MIT license

Is your feature request related to a problem? Please describe.
The BSD-3-Clause license type complicates the filesystem library usage.

Describe the solution you'd like
The MIT license is more permissive, so it would be nice to apply it instead.

Describe alternatives you've considered
Any other permissive licenses are welcome.

Additional context
Our team is going to open source some library (under MIT license) and having a dependency on the BSD license is not okay then. So if the license type is not changed, we'll have to find another filesystem implementation.

path::lexically_normal() behaves slightly differently from the standard

Hi!
First of all, thank you for this library!

Describe the bug
I'm trying to use this library as a fallback when the compiler does not support the final standard version of <filesystem> library.
However, I've found that in some cases the result returned by path::lexically_normal() of ghc is different from the one returned by the C++17 standard version.
In particular, the normal version of path("../") should be "..", while ghc retains the trailing '/' (eventually converted to '\\' on Windows).
On Windows, the same applies also to path("..\\"), which should be normalized to ".." and instead still retains the trailing '\\'.

To Reproduce
To reproduce the behavior, I've created a small program to compare the different results between ghc::filesystem and std::filesystem:

#include <iostream>
#include <filesystem>
#include "ghc/filesystem.hpp"

using std::cout;
using std::endl;

int main() {
    cout << "[std] path(\"../\").lexically_normal() == ";
    cout << std::filesystem::path("../").lexically_normal() << endl;
    
    cout << "[ghc] path(\"../\").lexically_normal() == ";
    cout << ghc::filesystem::path("../").lexically_normal() << endl;

#ifdef _WIN32
    cout << "[std] path(\"..\\\\\").lexically_normal() == ";
    cout << std::filesystem::path("..\\").lexically_normal() << endl;
    
    cout << "[ghc] path(\"..\\\\\").lexically_normal() == ";
    cout << ghc::filesystem::path("..\\").lexically_normal() << endl;
#endif
    return 0;
}

I've tested the program using various compilers and operating systems.

  • On Windows 10 using

    • MSVC 2017
    • MinGW-w64 9.2.0
    • clang 9.0

    the program output is:

    [std] path("../").lexically_normal() == ".."
    [ghc] path("../").lexically_normal() == "..\\"
    [std] path("..\\").lexically_normal() == ".."
    [ghc] path("..\\").lexically_normal() == "..\\"
  • On Ubuntu 18.04 LTS using

    • g++ 8.3.0
    • clang 6.0

    the program output is:

    [std] path("../").lexically_normal() == ".."
    [ghc] path("../").lexically_normal() == "../"

Expected behavior
In all these cases path::lexically_normal() of ghc should return a path("..") in order to conform to the standard.

External References

  • std::filesystem::path from cppreference.com

    A path can be normalized by following this algorithm:
    ...
    7. If the last filename is dot-dot, remove any trailing directory-separator.

  • Relative Paths from Open Standards

    A path in normal form has no redundant current directory (dot) elements, no redundant parent directory (dot-dot) elements, and no redundant directory-separators.

Use std::error_code instead of exceptions

Is your feature request related to a problem? Please describe.
I'm currently trying to use this library in a codebase that should be compiled without exceptions.

Describe the solution you'd like
Replace throwing exceptions with std::error_code. Since exceptions seem to be only used in the detail:: namespace this should not break compatibility with the std:: implementation.

Describe alternatives you've considered
Write my own?

C++20 support?

As you know, MacOS support -std=c++20 even in 10.14 but does not support <filesystem> until 10.15.

char8_t, std::u8string are coming soon. std::filesystem::path::string() will returns std::u8string instead of std::string.

The project does not compile with GCC 7

Describe the bug

/home/builder/build/test/bench/filesystem.hpp: In instantiation of 'ghc::filesystem::path::path(const Source&, ghc::filesystem::path::format) [with Source = std::basic_string_view<char>; <template-parameter-1-2> = ghc::filesystem::path]':
/home/builder/build/test/bench/filesystem.hpp:2409:35:   required from here
/home/builder/build/test/bench/filesystem.hpp:1455:27: error: no matching function for call to 'toUtf8(const std::basic_string_view<char>&)'
     : _path(detail::toUtf8(source))
             ~~~~~~~~~~~~~~^~~~~~~~
/home/builder/build/test/bench/filesystem.hpp:1342:20: note: candidate: template<class charT, class traits, class Alloc, typename std::enable_if<(sizeof (charT) == 1), int>::type size> std::__cxx11::string ghc::filesystem::detail::toUtf8(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&)
 inline std::string toUtf8(const std::basic_string<charT, traits, Alloc>& unicodeString)
                    ^~~~~~
/home/builder/build/test/bench/filesystem.hpp:1342:20: note:   template argument deduction/substitution failed:
/home/builder/build/test/bench/filesystem.hpp:1455:27: note:   'const std::basic_string_view<char>' is not derived from 'const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>'
     : _path(detail::toUtf8(source))
             ~~~~~~~~~~~~~~^~~~~~~~

To Reproduce
Compile with G++ 7 on Debian.

Expected behavior
Compiles.

Additional context
The GCC7 seems to be missing string_view or is not included.

Compilation failed

Hello,
I'm compiling with MinGW gcc 9.2.0 on Windows 10 and
the constant ERROR_FILE_TOO_LARGE (line 4228) is not defined.

On Windows, when the find directory is finished, the handle is not closed.

Describe the bug
I'm not sure if this meets the C++ standard, but both msvc and mingw implementations do this.

To Reproduce

auto path = fs::path("test/");
fs::create_directory(path);
auto itor = fs::directory_iterator(path);
while (itor != fs::directory_iterator()) {
    ++itor;
}
fs::remove_all(path);
fs::create_directory(path); // throw an exception

When itor is equal to end(), it releases the resources held by itor. fs::create_directory will not throw an exception.

Expected behavior
The second fs::create_directory works fine.

Additional context

Fix conversion warnings

Is your feature request related to a problem? Please describe.

This is a header-only library therefore it is usually built with the warning level of the project which includes it. With some additional warnings enabled the filesystem.hpp does not compile cleanly.

Describe the solution you'd like

Build the project with -Wconversion -Wsign-conversion -Wpedantic and fix the reported warnings.

[Question] iOS support

Hi,

We are currently checking for alternatives to boost filesystem.
As of now, unless mistaken, std::filesystem is still not supported on iOS < 13, and after searching a bit, I stumbled on your project's page.

I did not see any mention of iOS in the supported platforms. Is this just a lack of time to test on iOS, or would this require extra work for this platform?

Thanks!

toUtf8 template argument from 'const Source'

Here's the errors I get when compiling a project with filesystem as a submodule:

D:\jonas\Code\WololoKingdoms\libwololokingdoms\third_party\filesystem\include\ghc\filesystem.hpp:1455:
Error: C2672: 'ghc::filesystem::detail::toUtf8': no matching overloaded function found
Error: C2784: 'std::string ghc::filesystem::detail::toUtf8(const charT *)': could not deduce template argument for 'const charT *' from 'const Source'
Error: C2784: 'std::string ghc::filesystem::detail::toUtf8(const std::basic_string<_Elem,_Traits,_Alloc> &)': could not deduce template argument for 'const std::basic_string<_Elem,_Traits,_Alloc> &' from 'const Source'

This error has occasionally appeared and disappeared before, so it's quite possible that it's a problem on my end, but this time I can't seem to make it go away with updating/cleaning as was the case before. I'm not sure how to handle this, so any tips/pointers would be appreciated if someone has an idea what the issue might be.

cannot include header in multiple place.

Describe the bug
As the implementation and declaration stays in one file, i can not include this file in different source files
if these files would be linked together. Because multiple definition exist in object files.

You may separate the implementation into .cc file ?

v1.3.1 release ETA?

OpenRCT2 is a project that uses your library and we've recently received a bug report regarding inability to build on FreeBSD. I saw the support for this platform was added across #49 and #51, but it's only to be included in the upcoming, v1.3.1 release. According to https://github.com/gulrak/filesystem/milestone/17 that seems all done and 15 days overdue already and I wanted to check with you what's the ETA of this release before we start breakingapplying local changes to the filesystem library?

Inconsistent handling of `directory_options::skip_permission_denied`

Describe the bug
While testing the POSIX implementation of the recursive_directory_iterator on macOS, I found the skip_permission_deniedoption to not work when iterating over my complete home directory hitting the ~/Library/Application Support/MobileSync folder. The increment still threw a filesystem_error. The reason is not wrong permissions, but "System Integrety Protection". The opendir results in an EPERM and not EACESS, and while EPERM was added to the increment handling of directory_iterator it is missing from the opendir error handling.

To Reproduce
Just try to iterate over a macOS SIP protected folder.

Expected behavior
The iterator should not enter the protected folder but continue to scan next to it.

Test suite does not build on centos7

The test suite does not seem to build under centos7 with devtoolset-9. This seems to be due to a used TestAllocator not implementing rebind (deprecated in C++17, removed in C++20), e.g.,

/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h: In instantiation of ‘class std::basic_string<char, std::char_traits<char>, TestAllocator<char> >’:
/tmp/cirrus-ci-build/test/filesystem_test.cpp:588:116:   required from here
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3142:63: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3142 |       typedef typename _Alloc::template rebind<_CharT>::other _CharT_alloc_type;
      |                                                               ^~~~~~~~~~~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3160:68: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3160 |       typedef __gnu_cxx::__normal_iterator<pointer, basic_string>  iterator;
      |                                                                    ^~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3162:61: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3162 |                                                             const_iterator;
      |                                                             ^~~~~~~~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3163:53: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3163 |       typedef std::reverse_iterator<const_iterator> const_reverse_iterator;
      |                                                     ^~~~~~~~~~~~~~~~~~~~~~
/opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_string.h:3164:52: error: no class template named ‘rebind’ in ‘class TestAllocator<char>’
 3164 |       typedef std::reverse_iterator<iterator>      reverse_iterator;

I was exploring whether this library would be a good replacement for devtoolset on centos7 since the base system libstdc++ is fairly old there, and some the std filesystem library often requires compiled components (the devtoolset workflow is easiest to deploy if the produced artifacts have not runtime dependencies on devtoolset libraries).

I have tested this by extending the Cirrus CI setup with centos7 and centos8 platforms here. The build and test steps pass for centos8, but centos7 fails with above error.

Ideally it should be possible to use this library to statically compile in a libstdc++-neutral std::filesystem-like functionality (either by using header-only components or by a static link).

I understand that there is no official support for centos platforms, so please close this if it is out of scope.

Case-Sensitivity For Windows Drive Letters

Windows case-sensitivity for drive letters is not always interpreted correctly.

To Reproduce:

fs::path path = fs::relative("c:\\dev\\working\\directory\\file.txt");

Assuming the current working directory is C:\dev\working\directory, one would expect path to be file.txt. However, because the second (optional) argument to relative is evaluated as C:\dev\working\directory (capital 'C'), lexically_relative will exit early on its first check and simply return a default-constructed path.

A fix for this specific issue could be to force drive letters to be capital, say, in weakly_canonical. Now, even Windows can't guarantee case-insensitivity for all file names, but I do believe that drive letters are case-insensitive -- for example, I don't think you can have a d: and D: drive mounted simultaneously. Additionally, under MSVC, it appears that path equality checks are completely case-insensitive when using the standard template library's file system implementation. This does not seem to be the case when compiling under any other compiler.

Warning under mingw32-x86_64

filesystem.hpp: In function 'void ghc::filesystem::detail::create_symlink(const ghc::filesystem::path&, const ghc::filesystem::path&, bool, std::error_code&)':
filesystem.hpp:1605:159: warning: cast between incompatible function types from 'FARPROC' {aka 'long long int (*)()'} to 'ghc::filesystem::detail::CreateSymbolicLinkW_fp' {aka 'unsigned char (*)(const wchar_t*, const wchar_t*, long unsigned int)'} [-Wcast-function-type]
 1605 |     static CreateSymbolicLinkW_fp api_call = reinterpret_cast<CreateSymbolicLinkW_fp>(GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "CreateSymbolicLinkW"));
      |                                                                                                                                                               ^
filesystem.hpp: In function 'void ghc::filesystem::detail::create_hardlink(const ghc::filesystem::path&, const ghc::filesystem::path&, std::error_code&)':
filesystem.hpp:1622:147: warning: cast between incompatible function types from 'FARPROC' {aka 'long long int (*)()'} to 'ghc::filesystem::detail::CreateHardLinkW_fp' {aka 'unsigned char (*)(const wchar_t*, const wchar_t*, _SECURITY_ATTRIBUTES*)'} [-Wcast-function-type]
 1622 |     static CreateHardLinkW_fp api_call = reinterpret_cast<CreateHardLinkW_fp>(GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "CreateHardLinkW"));

Should probably use static_cast<> instead

Missing options parameter in copy

In line 3027 of filesystem.hpp, if the copy()-method resolves to a copy_file with two file paths given, the options parameter is not passed on.
copy_file(from, to, ec);
This causes issues e.g. if the target file already exists and overwrite_existing or update_existing was passed as an option.

On win32 fs::rename should overwrite file if exists

Currently , fs::rename() will return with error 183, if destination file already exists. The standard indicate that it must overwrite file , here is the patch to use MoveFileExW instead.

GHC_INLINE void rename(const path& from, const path& to, std::error_code& ec) noexcept
{
    ec.clear();
#ifdef GHC_OS_WINDOWS
    if (from != to) {
        if (!MoveFileExW(detail::fromUtf8<std::wstring>(from.u8string()).c_str(), detail::fromUtf8<std::wstring>(to.u8string()).c_str(), (DWORD) MOVEFILE_REPLACE_EXISTING)) {
            ec = detail::make_system_error();
        }
    }
#else
    if (from != to) {
        if (::rename(from.c_str(), to.c_str()) != 0) {
            ec = detail::make_system_error();
        }
    }
#endif
}

gcc 9.2 warning (-Wcast-function-type) on MSYS2

Describe the bug
warning in gcc 9.2 on MSYS2 environment

C:/msys64/home/phlpt/HELICS/build_gcc/include/helics_cxx/helics/external/filesystem.hpp: In function 'void ghc::filesystem::detail::create_symlink(const ghc::filesystem::path&, const ghc::filesystem::path&, bool, std::error_code&)':
C:/msys64/home/phlpt/HELICS/build_gcc/include/helics_cxx/helics/external/filesystem.hpp:1605:159: warning: cast between incompatible function types from 'FARPROC' {aka 'long long int (*)()'} to 'ghc::filesystem::detail::CreateSymbolicLinkW_fp' {aka 'unsigned char (*)(const wchar_t*, const wchar_t*, long unsigned int)'} [-Wcast-function-type]
 1605 |     static CreateSymbolicLinkW_fp api_call = reinterpret_cast<CreateSymbolicLinkW_fp>(GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "CreateSymbolicLinkW"));
      |                                                                                                                                                               ^

To Reproduce
Detailed steps to reproduce the behavior.

Expected behavior
It would be nice if the code were warning free

Additional context
MSYS2, gcc 9.2 compiler --std=c++14, most warnings turned on.
Tried with the latest release, but that didn't clear up the warning.

Incorrect handling of ../..

Describe the bug

With ../.. only 1 dir is unrolled.

To Reproduce

    const char *path = "ab/cd/ef/../../qw";
    ghc::filesystem::path srcPath = ghc::filesystem::u8path(path);
    std::string destPath = srcPath.lexically_normal().u8string();

Expected behavior
Should produce ab/qw, produces ab/cd/qw.

Attached is a test project.

filesystem_test.zip

UPD:

More test cases:

"\\/\\///\\/" produces //////, expected /.

"a/b/..\\//..///\\/../c\\\\/" produces a/b/c///, expected ../c/.

"a/b/../../../c" produces a/c, expected ../c.

"..a/b/..\\//..///\\/../c\\\\/" produces ..a/b/c///, expected ../c/.

Support for -fno-exception

Is your feature request related to a problem? Please describe.
my project is compiled with -fno-exception.
and throw is forbidden with -fno-exception

Describe the solution you'd like
maybe we can make a macro protect

Clean build on Visual Studio 2019

I know your tests are currently not being run on 2019. I gave it a shot and tried to compile anyways. I got a bunch of warnings and an error in one of the examples. Error was easy enough to fix (missing include) but the warnings are mostly about template magic in some system header. I don't get it exactly but seems to be originating from the utf8 conversion somehow.
..so VS2019 build would be awesome to have. Thanks for sharing this great lib.

Error when concatenating paths

OS: Windows 10

Describe the bug
The base path to which a sub path is to be concatenated, is not considered at all.

To Reproduce

using filesystem = ghc::filesystem;
auto base_path = filesystem::path("/binaries");
auto path = base_path / filesystem::path("./sub_path");

path is now "/sub_path"

Expected behavior
path should be "/binaries/sub_path".

Conditional directives for strerror_r don't work with musl

Describe the bug
Using musl instead of glibc results in the #else branch in systemErrorText getting used; musl follows the POSIX standard.

To Reproduce
Compiling a program using ghc::filesystem on Alpine Linux (uses musl as its libc by default) as part of a CI build is how I encountered this issue. Otherwise, setup another compiler to use musl instead of glibc.

Expected behavior
The variant of strerror_r that returns an int should be used.

Additional context
Line 208 of the log in https://cloud.drone.io/nightlark/HELICS-src/22/2/4 is where I first saw the problem.

Option to not install files in /usr/share/include or /usr/lib64/cmake

Is your feature request related to a problem? Please describe.
Issue of an app I was building jpd002/Play-#825
It includes a ghc::filesystem as dependency. If I run make install, it will install files in /usr/share/include and /usr/lib64/cmake.

Describe the solution you'd like
Here can be an option to not install these files if the package is included as a dependency of other apps.

Describe alternatives you've considered

Additional context

Optional way to use ghc::filesystem with forwarding header and one time included implementation

Is your feature request related to a problem? Please describe.

When ghc::filesystem is used in multiple places of a project, there is a compiletime overhead due to the header-only, everything-inline nature of the library. In such situations it might be nice to be able to concentrate the implementation into a single place and include only the nessesery elements in all the other places.

This also comes with the advantage, that the needed system includes (e.g. windows.h) will only be needed at the implementation place, so thes do not pollute the global namespace in every place filesystem is used.

Describe the solution you'd like

Two small wrapping headers should be added, they set some defines and include the normal filesystem.hpp header. One called fs_fwd.hpp will ensure everything needed to use ghc::filesystem is visible, the other, called fs_impl.hpp will lead to the expansion of all implementation code and use the system headers needed to implement the functionality.
The implementation header must be included in a cpp before the visibility of a fs_fwd.hpp to take precedence in the code extraction.

First implementation will be done on a feature branch.

This feature is related to some wishes from #3.

Option, how to react on unicode errors

There should be an option to select if unicode errors (invalid byte sequence or bad codepoint) should lead to errors or be replaced by the unicode replacement character (U+FFFD), like it is currently done.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.