cern-fts / davix Goto Github PK
View Code? Open in Web Editor NEWHigh-performance file management over WebDAV / HTTP
License: GNU Lesser General Public License v2.1
High-performance file management over WebDAV / HTTP
License: GNU Lesser General Public License v2.1
https://curl.se/docs/CVE-2022-32221.html
Affected versions: libcurl 7.7 to and including 7.85.0
Not affected versions: libcurl < 7.7 and >= 7.86.0
davix bundles 7.69.0
The bundled library is used in the EPEL 7 and EPEL 8 builds, because the system version is too old.
EPEL 9 and Fedora uses the system version.
I have backported the commit fixing the CVE (a one line patch) to the packages in EPEL 7 and EPEL 8.
with Davix 0.7.5, it doesn't seem possible to process more than one file at a time with a single davix-get/davix-cp command.
For instance, I've tried this syntax to get content from https source to locale:
davix-get -P grid https://example.org:PORT/folder/test1 https://example.org:PORT/folder/test2 > txtfile
which results into: (Davix::Tools::davix-get) Error: File not found for destination file https://example.org/folder/test2
Moreover, if I try to get from the same two files into a local folder:
davix-get -P grid https://example.org:PORT/folder/test1 https://example.org:PORT/folder/test2 ./localfolder/
I get: davix-get, Error: Wrong argument format
Finally, if I try:
davix-cp -P grid https://example.org:PORT/folder/test1 https://example.org:PORT/folder/test2 ./localfolder/
The result is: (Davix::HttpThirdPartyCopy) Error: Transfer failed: failure: SSLException while pushing https://example.org/folder/test2 Broken pipe (Write failed).
All the aforementioned commands have been tried also with the -r 1 option instead of -P, which however did not change the result.
Is it possible to have that functionality?
Thanks a lot for your attention and effort
Hi!
Davix hides error messages coming from the server, making it difficult to debug issues or use Davix as a client in automated tests.
Example:
$ davix-ls -H "Authorization: Bearer ${BT}" https://storm.example:8443/wlcg
(Davix::HttpRequest) Error: Authentification failed on server
There's a typo in the error message, it should read "Authentication failed on server", but the main problem
is that Davix hides error information coming from the server that could help the user understand what's
going on.
11:27 $ davix-ls --debug -H "Authorization: Bearer ${BT}" https://storm.example:8443/wlcg
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.
DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
...
DAVIX(socket): Running pre_send hooks
> PROPFIND /wlcg HTTP/1.1
> User-Agent: libdavix/0.7.6 neon/0.0.29
> Keep-Alive:
> Connection: Keep-Alive
> TE: trailers
> Host: storm.example:8443
> Depth: 1
> Authorization: Bearer eyJra....a3re25tgugeWlETlAcay1pxNwdGQCQSfpg
> Content-Length: 303
>
...
DAVIX(socket): Sending request body:
DAVIX(socket): Request body sent successfully
DAVIX(socket): Request sent; retry is 0.
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Bearer error="invalid_token", error_description="Unknown token issuer: https://tf.cloud.cnaf.infn.it", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Content-Length: 0
<
DAVIX(socket): End of headers.
DAVIX(socket): Running post_headers hooks
DAVIX(socket): Running post_send hooks
DAVIX(ssl): ah_post_send (#0), code is 401 (want 401), WWW-Authenticate is Bearer error="invalid_token", error_description="Unknown token issuer: https://tf.cloud.cnaf.infn.it", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"
DAVIX(ssl): auth: Got challenge (code 401).
DAVIX(ssl): auth: Ignored parameter: error = "invalid_token
DAVIX(ssl): auth: Ignored parameter: error_description = "Unknown token issuer: https://tf.cloud.cnaf.infn.it
DAVIX(ssl): auth: Ignored parameter: error_uri = "https://tools.ietf.org/html/rfc6750#section-3.1
Davix version (on MacOS Mojave):
11:35 $ davix-ls --version
Version: 0.7.6
Thanks!
A.
Maybe they were removed by accident in commit 011b247?
Notice that include(CTest)
line was removed from CMakeLists.txt
in that commit.
With davix-0.7.3-1, we do not manage to upload a local directory to http server:
$ davix-put -H "Authorization: Bearer ${AT}" test_folder/ https://example.org/dir1
(Davix::Tools::davix-put) Error: test_folder/is not a valid regular file
This should be possibile, according to man:
EXAMPLES
[...]
Upload local directory to a http server
davix-put mydir http://example.org/dir1
Davix has been recently packaged in Gentoo, but it is failing to build with testing turned on, as shown below. For more information, please see https://bugs.gentoo.org/651098.
CMakeFiles/g_test_unit_davix_exe.dir/parser/test_config_parser.cpp.o: In function `ConfigParser_Tokenizer_Test::TestBody()':
test_config_parser.cpp:(.text+0x22f): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
test_config_parser.cpp:(.text+0x576): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
test_config_parser.cpp:(.text+0x804): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
test_config_parser.cpp:(.text+0xc02): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
-------------------------------------------------------------------
This is an unstable amd64 chroot image at a tinderbox (==build bot)
name: 17.0-developer_20180319-191208
-------------------------------------------------------------------
gcc-config -l:
[1] x86_64-pc-linux-gnu-7.3.0 *
Available Python interpreters, in order of preference:
[1] python3.5
[2] python2.7 (fallback)
Davix's unit tests create a file in /tmp/davix-tests-tmp-file
. However, the test doesn't remove this file, so if you run the tests as another user on the same machine (i.e. as root to install davix system-wide), the file will already be there and creation of the file will fail for the second user (with permission denied, as it already exists), causing the test to get a segfault when dereferencing the FILE
pointer here, as fopen
returns NULL
. Debugged this last Friday with @joaopblopes.
-- Found uuid: /usr/local/include
CMake Error at src/CMakeLists.txt:40 (set_target_properties):
set_target_properties called with incorrect number of arguments.
FreeBSD 11.2
On MacOS the kernel provides uuid functions (e.g. uuid_generate_random()
) without the need to link to libuuid.so: https://developer.apple.com/documentation/kernel/1470612-uuid_generate_random?language=objc.
However the Finduuid.cmake
module checks for the library and fails to configure if it's not found.
Even on Linux the functions come from the kernel (on my system libuuid.so exists, but has no symbols).
Please consider updating the module to not require the library (or I could create a pull request, just let me know).
Another suggestion is to add a --quiet flag for displaying only error messages, and muting all progress messages.
[root@txhk2 ~]# export DAVIX_USE_LIBCURL=1
[root@txhk2 ~]# davix-ls --userlogin xxx --userpass xxx https://toi.teracloud.jp/dav/xxx-dat/ --debug
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.
DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
DAVIX(core): libdavix path /lib64/libdavix.so.0, version: 0.8.5
DAVIX(posix): -> opendirpp
DAVIX(http): Create HttpRequest for https://toi.teracloud.jp/dav/xxx-dat/
DAVIX(http): -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem, retry
DAVIX(http): -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem, retry
DAVIX(http): -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem, retry
DAVIX(http): -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem: eradicate session
DAVIX(http): <- negotiateRequest
DAVIX(http): <- negotiateRequest
DAVIX(http): <- negotiateRequest
DAVIX(http): <- negotiateRequest
DAVIX(core): Destroy HttpRequest
DAVIX(posix): <- opendirpp
(Davix::HttpRequest) Error: Failed to load certificate
[root@txhk2 ~]#
anybody can help ?
Affected versions: libcurl 7.69.0 to and including 8.3.0
Not affected versions: libcurl < 7.69.0 and >= 8.4.0
Davix bundles libcurl 7.69.0
on EPEL7 and EPEL8, which is affected.
On EPEL9 and recent Fedora versions, Davix uses the system curl.
A patch must be applied in the build process for the bundled libcurl
.
Consider the following case:
Use davix-get to retrieve a file from a remote server.
The retrieval is successful and the file is written. :)
Use davix-get again to retrieve an updated version of the same file.
The retrieval fails due to some error (e.g. name resolution failure), and the old file is overwritten with an empty file. :(
For some context, I use davix to synchronize SSL certificates between multiple remote computers. A temporary failure caused davix to overwrite the certificates with empty files, resulting in errors from the programs using those certificates.
how to use in wndows?
Test project /usr/ports/www/davix/work/.build
Start 1: g_test_unit_davix
1/1 Test #1: g_test_unit_davix ................***Failed 0.06 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.07 sec
The following tests FAILED:
1 - g_test_unit_davix (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util
OS: FreeBSD 11.2
fetch: http://grid-deployment.web.cern.ch/grid-deployment/dms/lcgutil/tar/davix/0.7.4/davix-0.7.4.tar.gz: size mismatch: expected 4343614, actual 4340677
See the full log here: http://beefy5.nyi.freebsd.org/data/120i386-default/506094/logs/davix-0.7.4.log
What happened? Did somebody hack your server?
When I do something like:
davix-put -r 4 a-directory https://somehost.com/owncloud/remote.php/webdav/remotedir/
it works but it ask for login credentials for every file it wants to upload. Could this please be cached?
Version 0.6.9 fails:
-- Looking for uuid_generate_random in uuid - not found
-- Found uuid: /usr/local/include
CMake Error at src/CMakeLists.txt:40 (set_target_properties):
set_target_properties called with incorrect number of arguments.
-- generate pkgconfig file for libdavix under /usr/ports/www/davix/work/davix-R_0_6_9/src/davix.pc
-- Configuring incomplete, errors occurred!
CMake Error at src/CMakeLists.txt:40 (set_target_properties):
set_target_properties called with incorrect number of arguments.
This is the offending instruction:
set_target_properties(libdavix PROPERTIES
VERSION ${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}
SOVERSION ${VERSION_MAJOR}
CLEAN_DIRECT_OUTPUT 1
OUTPUT_NAME ${OUTPUT_NAME_DAVIX})
Regression in version 0.7.1
FreeBSD 11.2 amd64
cmake-3.12.3
Current macOS does not come with python 2 anymore. CMake should be updated to use the new FindPackage Python, similar to what's done here: https://github.com/dpiparo/vdt/pull/12/files - that will happily find Python3.
This blocks building ROOT on current macOS systems, version 12.3.
The test seems to hang because of a version mismatch between strings in the test and the actual library.
I had to apply the following change for the tests to pass for me:
--- a/test/slow-unit/test-utils.hpp
+++ b/test/slow-unit/test-utils.hpp
@@ -33,7 +33,7 @@
#define SSTR(message) static_cast<std::ostringstream&>(std::ostringstream().flush() << message).str()
inline std::string getCurlUserAgent() {
- return SSTR("User-Agent: " << Davix::RequestParams().getUserAgent() << " libcurl/7.69.0-DEV\r\n");
+ return SSTR("User-Agent: " << Davix::RequestParams().getUserAgent() << " libcurl/7.85.0\r\n");
}
inline std::string getDefaultUserAgent() {
I was compiling against curl-7.85.0. This string may have to be generated for the right library version for the test to always work.
Hi,
when compiling DAVIX on debian/ubuntu 64bit the logic to determine the library path suffix is not correct: It checks for the existence of /usr/lib64 but this directory does (usually) not exist on debian/ubuntu. As such PKG_ARCH is usually set to i386 but that doesn't seem to be a problem. However we do have some issues wen using davix with ROOT as the ROOT compilation cannot find the library in some cases.
The official GNUInstallDirs module from cmake determines the path differently by checking if /etc/debian_version exists and doing a different logic in that case. Would it be possible to adapt the same logic for davix? Or just use GNUInstallDirs?
Cheers,
Martin
Hi,
I faced with an issue with davix-*
commands when trying to authenticate by username and password. It was working before but suddenly stopped and constantly returns: (Davix::HttpRequest) Error: HTTP 401 : Authentification Error
So after investigation I found out the davix client version in the machine was upgraded to version 0.8.0
so I tested it with older version successfully.
We connect to a server with valid certificate over https, but to reproduce the issue here; I use a docker image as a webdav server:
docker-compose.yml
file with content:version: '3'
services:
webdav:
image: bytemark/webdav
restart: always
ports:
- "80:80"
environment:
AUTH_TYPE: Digest
USERNAME: alice
PASSWORD: secret1234
volumes:
- ./dav:/var/lib/dav
docker compose up -d
to start the webdav server./dav/data
path to have some thing to show0.7.6
and 0.8.0
let's first try the older version that works:/usr/local/Cellar/davix/0.7.6/bin/davix-ls -l --userlogin alice --userpass secret1234 http://localhost
-rwxrwxrwx 0 10 2021-10-28 19:17:29 data1.txt
-rwxrwxrwx 0 16 2021-10-28 19:44:54 data2.txt
/usr/local/Cellar/davix/0.8.0/bin/davix-ls -l --userlogin alice --userpass secret1234 http://localhost
(Davix::HttpRequest) Error: HTTP 401 : Authentification Error
/usr/local/Cellar/davix/0.8.0/bin/davix-ls -l --debug --userlogin alice --userpass secret1234 http://localhost
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.
DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
DAVIX(core): libdavix path /usr/local/Cellar/davix/0.8.0/lib/libdavix.0.8.0.dylib, version: 0.8.0
DAVIX(posix): -> opendirpp
DAVIX(http): Create HttpRequest for http://localhost
DAVIX(http): -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): no cached ne_session, create a new one
HTTP session to http://localhost:80 begins.
DAVIX(http): configure session...
DAVIX(http): define connection timeout to 30
DAVIX(http): enable login/password authentication
DAVIX(http): enable client cert authentication by callback
DAVIX(ssl): ah_create, for WWW-Authenticate
DAVIX(socket): Running pre_send hooks
> PROPFIND / HTTP/1.1
> User-Agent: libdavix/0.8.0 neon/0.0.29
> Keep-Alive:
> Connection: Keep-Alive
> TE: trailers
> Host: localhost
> Depth: 1
> Content-Length: 303
>
DAVIX(socket): Sending request-line and headers:
DAVIX(socket): Doing DNS lookup on localhost...
DAVIX(socket): Sending request body:
DAVIX(socket): Request body sent successfully
DAVIX(socket): Request sent; retry is 0.
< HTTP/1.1 401 Unauthorized
< Date: Thu, 28 Oct 2021 18:01:19 GMT
< Server: Apache/2.4.37 (Unix)
< WWW-Authenticate: Digest realm="WebDAV", nonce="kv9pf23PBQA=87dcf2978519b77f6a33ecf1023df29d2c0c5cd6", algorithm=MD5, qop="auth"
< Content-Length: 381
< Keep-Alive: timeout=5, max=100
< Connection: Keep-Alive
< Content-Type: text/html; charset=iso-8859-1
<
DAVIX(socket): End of headers.
DAVIX(socket): Running post_headers hooks
DAVIX(ssl): Disable Session recycling
DAVIX(http): <- negotiateRequest
DAVIX(core): Destroy HttpRequest
DAVIX(socket): sess: Closing connection.
DAVIX(socket): sess: Connection closed.
DAVIX(ssl): Disable Session recycling
DAVIX(socket): Running destroy hooks.
DAVIX(socket): Request ends.
DAVIX(socket): sess: Destroying session.
DAVIX(posix): <- opendirpp
(Davix::HttpRequest) Error: HTTP 401 : Authentification Error
davix-get
, etc.Thanks a lot,
Shahram.
Version: built from latest github source
davix-put with the following options:
-r10 -k --no-100-continue
Getting errors that look like this:
Davix::DavixOp::PutOp Failed to PUT //2D_seismic_data/kevin_2D/SEG Y Export Job.LOG
(Davix::HttpRequest) Error: davs://dav.box.com/dav//2D_seismic_data/kevin_2D/SEG Y Export Job.LOG is not not a valid HTTP or Webdav URL
This seems to be happening with every file that has a space in the name. I think the URL is not being quoted (or special characters are not being URL encoded).
The reason is that the library is called libcurl.so
and needs to be linked as -lcurl
not -llibcurl
. However, the build system has no code to support the case in which the embedded library is not used. See the error below.
FAILED: src/libdavix.so.0.8.0
: && /usr/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=native -O2 -pipe -std=c++11 -Wl,-O1 -Wl,--as-needed -shared -Wl,-soname,libdavix.so.0 -o src/libdavix.so.0.8.0 deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_207.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_acl3744.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_alloc.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_auth.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_basic.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_compress.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_dates.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_i18n.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_locks.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_md5.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_ntlm.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_oldacl.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_pkcs11.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_props.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_redirect.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_request.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_session.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_socket.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_socks.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_sspi.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_string.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_uri.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_utils.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_xml.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_xmlreq.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_openssl.c.o src/CMakeFiles/libdavix.dir/auth/davix_openssl.cpp.o src/CMakeFiles/libdavix.dir/auth/davixauth.cpp.o src/CMakeFiles/libdavix.dir/auth/davixx509cred.cpp.o src/CMakeFiles/libdavix.dir/backend/BackendRequest.cpp.o src/CMakeFiles/libdavix.dir/backend/SessionFactory.cpp.o src/CMakeFiles/libdavix.dir/backend/StandaloneNeonRequest.cpp.o src/CMakeFiles/libdavix.dir/core/ContentProvider.cpp.o src/CMakeFiles/libdavix.dir/core/RedirectionResolver.cpp.o src/CMakeFiles/libdavix.dir/curl/CurlSession.cpp.o src/CMakeFiles/libdavix.dir/curl/CurlSessionFactory.cpp.o src/CMakeFiles/libdavix.dir/curl/HeaderlineParser.cpp.o src/CMakeFiles/libdavix.dir/curl/ResponseBuffer.cpp.o src/CMakeFiles/libdavix.dir/curl/StandaloneCurlRequest.cpp.o src/CMakeFiles/libdavix.dir/deprecated/httpcachetoken.cpp.o src/CMakeFiles/libdavix.dir/file/davfile.cpp.o src/CMakeFiles/libdavix.dir/file/davposix.cpp.o src/CMakeFiles/libdavix.dir/fileops/AzureIO.cpp.o src/CMakeFiles/libdavix.dir/fileops/chain_factory.cpp.o src/CMakeFiles/libdavix.dir/fileops/davix_reliability_ops.cpp.o src/CMakeFiles/libdavix.dir/fileops/davmeta.cpp.o src/CMakeFiles/libdavix.dir/fileops/fileutils.cpp.o src/CMakeFiles/libdavix.dir/fileops/httpiochain.cpp.o src/CMakeFiles/libdavix.dir/fileops/httpiovec.cpp.o src/CMakeFiles/libdavix.dir/fileops/iobuffmap.cpp.o src/CMakeFiles/libdavix.dir/fileops/S3IO.cpp.o src/CMakeFiles/libdavix.dir/fileops/SwiftIO.cpp.o src/CMakeFiles/libdavix.dir/hooks/davix_hooks.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/chrono/timepoint.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/crypto/base64.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/crypto/hmacsha.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/str/format.cpp.o src/CMakeFiles/libdavix.dir/libs/datetime/datetime_utils.cpp.o src/CMakeFiles/libdavix.dir/modules/modules_profiles.cpp.o src/CMakeFiles/libdavix.dir/neon/neonrequest.cpp.o src/CMakeFiles/libdavix.dir/neon/neonsession.cpp.o src/CMakeFiles/libdavix.dir/neon/neonsessionfactory.cpp.o src/CMakeFiles/libdavix.dir/params/davixrequestparams.cpp.o src/CMakeFiles/libdavix.dir/request/httprequest.cpp.o src/CMakeFiles/libdavix.dir/status/DavixStatus.cpp.o src/CMakeFiles/libdavix.dir/status/davixstatusrequest.cpp.o src/CMakeFiles/libdavix.dir/utils/checksum_extractor.cpp.o src/CMakeFiles/libdavix.dir/utils/CompatibilityHacks.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_azure_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_gcloud_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_logger.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_misc_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_cs3_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_s3_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_swift_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/simple_get_pass.cpp.o src/CMakeFiles/libdavix.dir/utils/stringutils.cpp.o src/CMakeFiles/libdavix.dir/utils/davixuri.cpp.o src/CMakeFiles/libdavix.dir/xml/azurepropparser.cpp.o src/CMakeFiles/libdavix.dir/xml/davdeletexmlparser.cpp.o src/CMakeFiles/libdavix.dir/xml/davix_ptree.cpp.o src/CMakeFiles/libdavix.dir/xml/davpropxmlparser.cpp.o src/CMakeFiles/libdavix.dir/xml/davxmlparser.cpp.o src/CMakeFiles/libdavix.dir/xml/metalinkparser.cpp.o src/CMakeFiles/libdavix.dir/xml/s3deleteparser.cpp.o src/CMakeFiles/libdavix.dir/xml/S3MultiPartInitiationParser.cpp.o src/CMakeFiles/libdavix.dir/xml/s3propparser.cpp.o src/CMakeFiles/libdavix.dir/xml/swiftpropparser.cpp.o src/CMakeFiles/libdavix.dir/davixcontext.cpp.o -llibcurl -lssl -lcrypto -lxml2 -lpthread -luuid -lz && :
/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -llibcurl
collect2: error: ld returned 1 exit status
For now, in order to package Davix, I am using the following patch:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index c65ed46b..35950a9a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -117,6 +117,8 @@ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CXX11_FLAG_ENABLE}")
if(EMBEDDED_LIBCURL)
include(buildCurl REQUIRED)
buildCurl()
+else()
+ find_package(CURL REQUIRED)
endif()
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/deps/libneon/src/ ${UUID_INCLUDE_DIRS})
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 79a8858b..53455abb 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -108,7 +108,7 @@ add_library(libdavix SHARED ${DAVIX_SOURCES} )
target_include_directories(libdavix PRIVATE ${DAVIX_INTERNAL_INCLUDES} )
target_link_libraries(libdavix
- PRIVATE libcurl
+ PRIVATE CURL::libcurl
PUBLIC ${LIBSSL_PKG_LIBRARIES}
${LIBXML2_LIBRARIES}
${CMAKE_THREAD_LIBS_INIT}
@@ -146,7 +146,7 @@ set_target_properties(libdavix_static PROPERTIES
target_include_directories(libdavix_static PRIVATE ${DAVIX_INTERNAL_INCLUDES} )
target_link_libraries(libdavix_static
- PRIVATE libcurl
+ PRIVATE CURL::libcurl
PUBLIC ${OPENSSL_LIBRARIES}
${CMAKE_THREAD_LIBS_INIT}
dl ${UUID_LIBRARIES}
However, that is not a full solution, as that breaks the case with embedded libcurl. It would be nice to have davix support building against all external libraries rather than only the embedded ones. Thanks!
Hi guys,
there's a bug in error handling for these two utilities:
$ davix-cp -P grid https://a.example/a https://b.example/b
(Davix::HttpRequest) Error: copy Domain name resolution failed
$ echo $?
0
$ davix-mkdir -P grid https://a.example/a
(Davix::HttpRequest) Error: Domain name resolution failed
$ echo $?
0
$ davix-cp --version
Version: 0.6.9-
Thanks!
A.
Hi,
I'm trying to recursively delete a directory with davix-rm (as reported here https://www.systutorials.com/docs/linux/man/1-davix-rm/) but it does not seem to work.
This is the directory:
davix-ls -l --capath /etc/grid-security/certificates/ -H "Authorization: Bearer ${BEARER_TOKEN}" davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12
-rwxrwxrwx 0 9442 2021-05-19 09:24:14 certs.p12
-rwxrwxrwx 0 494 2021-05-19 09:24:37 oidc-agent-stat.sh
And this is what I see when I try to delete it:
davix-rm --capath /etc/grid-security/certificates/ -H "Authorization: Bearer ${BEARER_TOKEN}" davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/
(Davix::rm) Error: HTTP 412 : Server Error with url davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/
Anyways, gfal-rm -r seems to work on the same directory:
gfal-rm -r davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12 2>/dev/null
davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/certs.p12 DELETED
davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/oidc-agent-stat.sh DELETED
davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12 RMDIR
davix-ls -l --capath /etc/grid-security/certificates/ -H "Authorization: Bearer $AT" davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12
(Davix::HttpRequest) Error: HTTP 404 : File not found
Is it possible to make davix-rm behave the same way as gfal-rm -r with respect to directories?
For your information, these are the versions involved:
gfal-rm --version
gfal2-util version 1.5.3 (gfal2 2.18.1)
dcap-2.18.1
file-2.18.1
gridftp-2.18.1
http-2.18.1
lfc-2.18.1
rfio-2.18.1
sftp-2.18.1
srm-2.18.1
xrootd-2.18.1
davix-rm --version
Version: 0.7.5
Thanks for your attention.
building on FreeBSD fails because BSD sed -i requires a prefix argument (it is optional in GNU sed)
PR follows
When trying to compile davix 0.7.4 I ran into a compiler error because a function definition changed in the bundled version of googletest since the latest release 1.8.1
In file included from davix/deps/googletest/googletest/src/gtest-all.cc:45:
davix/deps/googletest/googletest/src/gtest-port.cc:1211:13: error: ambiguating new declaration of 'const char* testing::internal::StringFromGTestEnv(const char*, const char*)'
const char* StringFromGTestEnv(const char* flag, const char* default_value) {
^~~~~~~~~~~~~~~~~~
In file included from common/include/gtest/internal/gtest-internal.h:40,
from /common/include/gtest/gtest.h:58,
from /davix/deps/googletest/googletest/src/gtest-all.cc:39:
include/gtest/internal/gtest-port.h:2549:13: note: old declaration 'std::__cxx11::string testing::internal::StringFromGTestEnv(const char*, const char*)'
std::string StringFromGTestEnv(const char* flag, const char* default_val);
^~~~~~~~~~~~~~~~~~
googletest 1.8.1 is installed in a common include directory with libuuid so I cannot easily hide it from davix.
As a simple solution, would it be possible to include an option to not build the tests at all?
Environment:
The traceback
#10 0x00007fd5daf680d3 in std::string::size() const () from /lib64/libstdc++.so.6
#11 0x00007fd5db51f3b7 in std::operator+<char, std::char_traits<char>, std::allocator<char> > (__lhs=0x7fd5cd0cdc3f " davix/", __rhs=<error reading variable: Cannot access memory at address 0xffffffffffffffe8>) at /usr/include/c++/4.8.2/bits/basic_string.tcc:700
#12 0x00007fd5ccf09ba8 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /mnt/build/jenkins/workspace/root-package/BUILDTYPE/Release/COMPILER/native/LABEL/cc7/VERSION/master/root/net/davix/src/TDavixFile.cxx:60
#13 0x00007fd5ccf09c78 in _GLOBAL__sub_I_TDavixFile.cxx(void) () at /mnt/build/jenkins/workspace/root-package/BUILDTYPE/Release/COMPILER/native/LABEL/cc7/VERSION/master/root/net/davix/src/TDavixFile.cxx:721
#14 0x00007fd5dbb6f4e3 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#15 0x00007fd5dbb73b04 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#16 0x00007fd5dbb6f2f4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#17 0x00007fd5dbb7321b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#18 0x00007fd5d9f9d02b in dlopen_doit () from /lib64/libdl.so.2
#19 0x00007fd5dbb6f2f4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#20 0x00007fd5d9f9d62d in _dlerror_run () from /lib64/libdl.so.2
#21 0x00007fd5d9f9d0c1 in dlopen
The problem is that TDavixFile.cxx uses Davix::version() in an static initialization and is probably not initialized yet.
Hi, I'm trying to upload files with an encrypted PEM key, but I get the following:
tlater ~/projects/webdav-test $ davix-put --key tlater.key.pem --cert tlater.cert.pem test dav://server/test
(Davix::OpenSSL) Error: parse PEM private key failed tlater.key.pem : error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
(Davix::HttpRequest) Error: (Neon): SSL handshake failed, client certificate was requested: sslv3 alert handshake failure
Unencrypted key files do work, and curl will happily use this one, so I think there's something wrong here.
I highly appreciate your effort in providing open source software and supporting legacy equipment like older macs the same.
Naturally, Homebrew is a valid option. However, one may prefer the option to use the MacPorts package manager and there is a davix port available too.
http://dmc.web.cern.ch/projects/davix/home - Installation - Easy Install
Please change from:
OSX: brew install davix
to
OSX: brew install davix
with Homebrew or sudo port install davix
with MacPorts
Ref to https://ports.macports.org/port/davix/summary
To my own knowledge the davix port installs nicely on Mac OS X / macOS 10.12.6 Sierra as:
A clean install in a fresh environment on the a.m. OS X would result in:
$ port installed
The following ports are currently installed:
autoconf @2.69_5 (active)
automake @1.16.1_1 (active)
bzip2 @1.0.8_0 (active)
cmake @3.15.4_0 (active)
curl @7.66.0_0+ssl (active)
curl-ca-bundle @7.66.0_1 (active)
davix @0.7.5_1 (active)
db48 @4.8.30_4 (active)
expat @2.2.9_0 (active)
gdbm @1.18.1_1 (active)
gettext @0.19.8.1_2 (active)
glib2 @2.58.3_0+x11 (active)
gperf @3.1_0 (active)
help2man @1.47.11_0 (active)
icu @65.1_0 (active)
libarchive @3.4.0_1 (active)
libcxx @5.0.1_4 (active)
libedit @20190324-3.1_0 (active)
libffi @3.2.1_0 (active)
libiconv @1.16_0 (active)
libidn2 @2.2.0_0 (active)
libpsl @0.21.0-20190419_0 (active)
libtool @2.4.6_6 (active)
libunistring @0.9.10_0 (active)
libuv @1.33.1_0 (active)
libxml2 @2.9.9_3 (active)
lz4 @1.9.2_0 (active)
lzo2 @2.10_0 (active)
ncurses @6.1_0 (active)
openssl @1.1.1d_0 (active)
p5.28-locale-gettext @1.70.0_0 (active)
pcre @8.43_0 (active)
perl5 @5.26.1_0+perl5_28 (active)
perl5.28 @5.28.2_0 (active)
pkgconfig @0.29.2_0 (active)
python3_select @0.0_1 (active)
python37 @3.7.5_1 (active)
python_select @0.3_8 (active)
readline @8.0.000_0 (active)
sqlite3 @3.30.1_0 (active)
texinfo @6.7_0 (active)
xz @5.2.4_0 (active)
zlib @1.2.11_0 (active)
zstd @1.4.3_0 (active)
Hope this helps.
Happy hacking. 👍
The davix library leaks one file descriptor for write-opened file.
lrwx------. 1 root root 64 Feb 11 13:25 66 -> /tmp/.davix_tmp_file_XXXXXXXXXXXXXXXXXXyzgG0q (deleted)
lrwx------. 1 root root 64 Feb 11 13:25 65 -> /tmp/.davix_tmp_file_XXXXXXXXXXXXXXXXXXZ6E10r (deleted)
lrwx------. 1 root root 64 Feb 11 13:25 63 -> /tmp/.davix_tmp_file_XXXXXXXXXXXXXXXXXXItaC8s (deleted)
The error is in the desctructor here, it unlinks the temporary file, but does not close _fd.
struct IOBufferLocalFile{
IOBufferLocalFile(int fd, const std::string & filepath): _fd(fd), _filepath(filepath){}
virtual ~IOBufferLocalFile(){
DAVIX_SLOG(DAVIX_LOG_TRACE, DAVIX_LOG_CHAIN, "Delete tmp file {}", _filepath);
unlink(_filepath.c_str());
}
int _fd;
std::string _filepath;
};
It would be also very nice, if the stage location could be modified because /tmp/ is not an optimal place for staging possibly large files.
Davix-libs 0.8.0-1, gfal2 2.20.0-1
Even though Google claims it's compatible with multi-part uploads via their S3 compatibility endpoints, it is not working:
$ gfal-copy -v -v -v testfile.4000M s3://burt-test-1.storage.googleapis.com/big5
yields a 403 error after uploading the first chunk with the following output:
It also fails with davix-libs 0.7.6-1 quickly with a different error (413).
I tried recompiling davix 0.7.6-1 and forcing https://github.com/cern-fts/davix/blob/devel/src/fileops/S3IO.cpp#L39 to return false; then the transfer succeeds.
I am unable to build using the tarball at
https://github.com/cern-fts/davix/releases/download/R_0_7_3/davix-0.7.3.tar.gz
due to the following error:
Traceback (most recent call last):
File "/home/conda/feedstock_root/build_artifacts/davix_1557317461225/work/genversion.py", line 253, in <module>
main()
File "/home/conda/feedstock_root/build_artifacts/davix_1557317461225/work/genversion.py", line 224, in main
gitDescribe = GitDescribe(sh("git describe --dirty").strip())
File "/home/conda/feedstock_root/build_artifacts/davix_1557317461225/work/genversion.py", line 164, in sh
raise Exception("Command {0} exited with code {1}".format(cmd, retcode))
Exception: Command git describe --dirty exited with code 128
Configuring cmake for davix version: 0.7.3
This occurs both when configuring with CMake and running make
. The script used for the build and can be found here and works when checking out the tag directly.
You can see a full build log here (line 465 onwards).
The download link on the release page does not result in a file of the correct format.
If I download:
$ wget https://github.com/cern-fts/davix/releases/download/R_0_8_4/davix-0.8.4.tar.gz
The retrieved file is a zip file, not a gzipped tar archive.
$ file davix-0.8.4.tar.gz
davix-0.8.4.tar.gz: Zip archive data, at least v2.0 to extract, compression method=deflate
If I inflate this file
$ mkdir z
$ cd z
$ unzip ../davix-0.8.4.tar.gz
Archive: ../davix-0.8.4.tar.gz
inflating: davix-0.8.4.tar.gz
$ file davix-0.8.4.tar.gz
davix-0.8.4.tar.gz: gzip compressed data, last modified: Thu Feb 2 11:17:28 2023, from Unix, original size modulo 2^32 36597760
I get the correct file. So the link give a zipped gzipped tar archive instead of a gzipped archive.
The corresponding link for the previous release gives the right format.
$ wget https://github.com/cern-fts/davix/releases/download/R_0_8_3/davix-0.8.3.tar.gz
$ file davix-0.8.3.tar.gz
davix-0.8.3.tar.gz: gzip compressed data, last modified: Tue Sep 13 10:19:00 2022, from Unix, original size modulo 2^32 36587520
(The davix-0.8.4-el7.x86_64.tar.gz link also gives the wrong format.)
Hi,
Are we (ROOT) configuring / building davix incorrectly? We get
15:18:07 [7702/10606] Linking CXX shared library lib/libRDAVIX.so.6.27.01
15:18:07 FAILED: : && /usr/bin/c++ -fPIC -fdiagnostics-color=always -std=c++14 -pipe -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -O3 -Wl,--no-undefined -Wl,--hash-style="both" -shared -Wl,-soname,libRDAVIX.so.6.27 -o lib/libRDAVIX.so.6.27.01 net/davix/CMakeFiles/G__RDAVIX.dir/G__RDAVIX.cxx.o net/davix/CMakeFiles/RDAVIX.dir/src/RRawFileDavix.cxx.o net/davix/CMakeFiles/RDAVIX.dir/src/TDavixFile.cxx.o net/davix/CMakeFiles/RDAVIX.dir/src/TDavixSystem.cxx.o -Wl,-rpath,/mnt/build/workspace/root-pullrequests-build/build/lib: lib/libNet.so.6.27.01 lib/libRIO.so.6.27.01 builtins/davix/DAVIX-prefix/lib/libdavix.a builtins/davix/DAVIX-prefix/lib/libneon.a -luuid /usr/lib/x86_64-linux-gnu/libssl.so /usr/lib/x86_64-linux-gnu/libcrypto.so /usr/lib/x86_64-linux-gnu/libxml2.so -ldl lib/libThread.so.6.27.01 lib/libCore.so.6.27.01 -lpthread && :
15:18:07 c++: error: builtins/davix/DAVIX-prefix/lib/libneon.a: No such file or directory
That's coming from https://github.com/root-project/root/blob/master/builtins/davix/CMakeLists.txt#L19-L22
foreach(lib davix neon)
set(libname ${CMAKE_STATIC_LIBRARY_PREFIX}${lib}${CMAKE_STATIC_LIBRARY_SUFFIX})
list(APPEND DAVIX_LIBRARIES ${DAVIX_PREFIX}/lib/${libname})
endforeach()
Does Davix export CMake targets (Davix::Davix
or similar) these days? I didn't find those.
IIUC, https://github.com/cern-fts/davix/blob/devel/src/CMakeLists.txt#L150 is relevant here and it "includes" curl, so we don't even need to link against neon anymore?
Thanks for any insights :-)
Axel
Hi,
there seems to be an auth problem when running davix-get
with DAVIX_USE_LIBCURL
[root@d1b4129fd75a davix]# davix-get --capath /etc/grid-security/certificates/ --cert $X509_USER_PROXY https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml zozo.xml
[root@d1b4129fd75a davix]# export DAVIX_USE_LIBCURL=1
[root@d1b4129fd75a davix]# davix-get --capath /etc/grid-security/certificates/ --cert $X509_USER_PROXY https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml zozo.xml
(Davix::HttpRequest) Error: Result HTTP 401 : Authentification Error after 3 attempts
This is done on a cern/cc7-base
docker container with the lcg-CA
and the following davix rpms
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-devel-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-libs-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-debuginfo-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-doc-0.7.6.200.3c48eab-1.el7.cern.noarch.rpm
Here is an extract of the debug logs that keeps repeating
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 41 locks.
DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
DAVIX(core): libdavix path /lib64/libdavix.so.0, version: 0.7.6.200.3c48eab
DAVIX(chain): -> readToFd
DAVIX(chain): request size 0
DAVIX(http): Create HttpRequest for https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml
DAVIX(http): -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): add CA PATH /etc/grid-security/certificates/
< GET /pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml HTTP/1.1
< Host: lhcbwebdav-kit.gridka.de:2880
< Accept: */*
< User-Agent: libdavix/0.7.6.200.3c48eab libcurl/7.69.0-DEV
> HTTP/1.1 401 login failed
> Date: Fri, 06 Aug 2021 09:06:53 GMT
> Server: dCache/6.2.24
> WWW-Authenticate: Basic realm=""
> Cache-Control: must-revalidate,no-cache,no-store
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 379
DAVIX(http): -> Error when using reycling of session/redirect : cancel and try again
DAVIX(http): -> negotiateRequest
davix does not compile with openssl 1.1.0, as reported to Debian BTS:
Davix 0.4.1 in epel seems affected by two regression bugs:
This is affecting the Data Bridge operation.
Davix 0.4.1-dev works perfectly instead. I suspect a bad error in the last push to epel, or an unfortunate push to the git head.
Hello,
I would like to use DavFile::get
to download some files. For the few cases I have tried it seems that DavFile::get
returns -1 instead of the number of bytes read, but as far as I can tell, the read succeeds and no DavixError
is thrown. I wanted to ask if I am indeed making an error here.
e.g.
#include <davix.hpp>
#include <iostream>
#include <vector>
int main() {
Davix::Context c;
Davix::RequestParams p;
Davix::DavFile file(c, p, Davix::Uri("https://root.cern/files/LHCb002_data_v1.root"));
std::vector<char> buf;
auto ret = file.get(nullptr, buf);
// file size is 11461 B
if (buf.size() != 11461) {
std::cout << "expected 11461 bytes read, got " << buf.size() << "\n";
}
std::cout << "got ret: " << ret << "\n";
return 0;
}
prints
got ret: -1
Thank you very much for your time.
Setup
devel
branchHi,
In most projects I've seen, new features are committed to the master
branch. Although this repo has a master
branch, it seems that work is mostly committed to the devel
branch.
I think it would help if there was some documentation (perhaps the README.md
file) that included a brief description on how the different branches are used.
Cheers,
Paul.
In
davix/test/unit/response-buffer.cpp
Line 136 in 454950a
Related issue in the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1012467
For authentication, would it be possible to add ~/.netrc support for providing credentials? This feature is used a lot by FTP command line tools.
PS: thanks so much davix (I use it for WebDAV access)!
To better debug failing HTTP-TPC transfers, it would be nice to expose to users of libdavix the hostname actually responsible for the transfer after redirection, if any.
I originally thought that adding a getCurrentUri
analogous to getOriginalUri
to expose the protected _current
variable from BackendRequest would be sufficient to get things going
davix/src/backend/BackendRequest.hpp
Line 178 in 23a591d
... but this is the wrong place to solve it. In DavixCopyInternal::copy
, there is logic to work through the redirections and then find the "correct" host for one side of the transfer:
davix/src/modules/copy/copy.cpp
Lines 229 to 298 in 23a591d
There isn't the analogous logic for the 2nd side of the transfer (which is stored in the destination
variable regardless of push/pull mode), the Uri is passed as-is as a Source/Destination header, then the resolution of the redirects happens on the machine we're connected to: (the top-level redirector for each site is xrootd-vanderbilt.sites.opensciencegrid.org
and xrootd.cmsaf.mit.edu
, note that the Source side is the redirector and not the "resolved" version)
DEBUG Davix: > COPY ///store/user/meloam/TESTMV.txt?authz=BearerXXX HTTP/1.1
> User-Agent: gfal2-util/1.7.1 gfal2/2.20.5 neon/0.0.29
> Keep-Alive:
> Connection: Keep-Alive
> TE: trailers
> Host: xrootd-se2-vanderbilt.sites.opensciencegrid.org:1094
> Source: https://xrootd.cmsaf.mit.edu:1094//store/temp/user/meloam/hosts.txt
> X-Number-Of-Streams: 0
> Secure-Redirection: 1
> TransferHeaderAuthorization: xx
> Credential: none
> Authorization: xx
> RequireChecksumVerification: false
Since libdavix doesn't resolve the COPY redirects for the destination, there's no way to report back the actual destination host (in this case). But, we do resolve the PROPFIND verb for the destination:
DEBUG Davix: > PROPFIND //store/temp/user/meloam/hosts.txt HTTP/1.1
> User-Agent: gfal2-util/1.7.1 gfal2/2.20.5 neon/0.0.29
> Keep-Alive:
> Connection: Keep-Alive
> TE: trailers
> Host: xrootd.cmsaf.mit.edu:1094
> Depth: 0
> Authorization: xxx
>
DEBUG Davix: Sending request-line and headers:
DEBUG Davix: Doing DNS lookup on xrootd.cmsaf.mit.edu...
DEBUG Davix: clicert callback
DEBUG Davix: call client cert callback
DEBUG Davix: Request sent; retry is 0.
DEBUG Davix: < HTTP/1.1 302 Redirect
DEBUG Davix: < Connection: Keep-Alive
DEBUG Davix: < Content-Length: 0
DEBUG Davix: < Location: https://xrootd6.cmsaf.mit.edu:1094/store/temp/user/meloam/hosts.txt?authz=BearerXXX
DEBUG Davix: <
DEBUG Davix: End of headers.
DEBUG Davix: Running post_headers hooks
DEBUG Davix: Add cached redirection <PROPFIND davs://xrootd.cmsaf.mit.edu:1094//store/temp/user/meloam/hosts.txt https://xrootd6.cmsaf.mit.edu:1094/store/temp/user/meloam/hosts.txt?authz=BearerXXX>
Is this on purpose? If instead DavixCopyInternal::copy()
resolved redirects and stashed them into Davix::DavixCopyInternal
, then accessors could be added to DavixCopy
for current source/dest Uris. The changes seem relatively straightforward to implement, but it is a behavioral change, so I don't want to implement if there would be no desire to add that functionality.
The failing test:
Start 2: slow-unit-tests
2/10 Test #2: slow-unit-tests .............................................................................***Failed 3.02 sec
Running main() from gtest_main.cc
[==========] Running 10 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from DrunkServer
[ RUN ] DrunkServer.NoClients
[ OK ] DrunkServer.NoClients (1000 ms)
[ RUN ] DrunkServer.ClientInteraction
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/drunk-server.cpp:37: Failure
Value of: connSrv
Actual: false
Expected: true
[ FAILED ] DrunkServer.ClientInteraction (1001 ms)
[----------] 2 tests from DrunkServer (2001 ms total)
[----------] 4 tests from Standalone_Neon_Request
[ RUN ] Standalone_Neon_Request.BasicSanity
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/standalone-request.cpp:66: Failure
Value of: request->startRequest().ok()
Actual: false
Expected: true
[ FAILED ] Standalone_Neon_Request.BasicSanity (2 ms)
[ RUN ] Standalone_Neon_Request.Redirect
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/standalone-request.cpp:173: Failure
Value of: request->startRequest().ok()
Actual: false
Expected: true
[ FAILED ] Standalone_Neon_Request.Redirect (0 ms)
[ RUN ] Standalone_Neon_Request.NetworkError
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/standalone-request.cpp:257: Failure
Value of: "(Neon): Could not read status line: connection was closed by server"
Expected: st.getErrorMessage()
Which is: "Could not connect to server"
[ FAILED ] Standalone_Neon_Request.NetworkError (1 ms)
[ RUN ] Standalone_Neon_Request.StopNoStart
[ OK ] Standalone_Neon_Request.StopNoStart (0 ms)
[----------] 4 tests from Standalone_Neon_Request (3 ms total)
[----------] 4 tests from Standalone_Curl_Request
[ RUN ] Standalone_Curl_Request.BasicSanity
CONSUMED: GET /chickens HTTP/1.1
CONSUMED: Host: localhost:22222
CONSUMED: Accept: */*
CONSUMED: I like: Turtles
WRITING RESPONSE
Response written successfully
[ OK ] Standalone_Curl_Request.BasicSanity (1005 ms)
[ RUN ] Standalone_Curl_Request.Redirect
CONSUMED: GET /test HTTP/1.1
CONSUMED: Host: localhost:22222
CONSUMED: Accept: */*
CONSUMED: User-Agent: libdavix/0.8.5 libcurl/8.2.1
WRITING RESPONSE
Response written successfully
[ OK ] Standalone_Curl_Request.Redirect (1 ms)
[ RUN ] Standalone_Curl_Request.NetworkError
[ OK ] Standalone_Curl_Request.NetworkError (0 ms)
[ RUN ] Standalone_Curl_Request.StopNoStart
[ OK ] Standalone_Curl_Request.StopNoStart (0 ms)
[----------] 4 tests from Standalone_Curl_Request (1006 ms total)
[----------] Global test environment tear-down
[==========] 10 tests from 3 test cases ran. (3010 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 4 tests, listed below:
[ FAILED ] DrunkServer.ClientInteraction
[ FAILED ] Standalone_Neon_Request.BasicSanity
[ FAILED ] Standalone_Neon_Request.Redirect
[ FAILED ] Standalone_Neon_Request.NetworkError
4 FAILED TESTS
The several next tests hang.
Version: 0_8_5
FreeBSD 13.2
In DAVIX, it appears that a blocking read is utilized when waiting on performance markers:
https://github.com/cern-it-sdc-id/davix/blob/devel/src/modules/copy/copy.cpp#L296
The request->readLine
method will only return when 1024 bytes of data are available (for the xrootd-tpc
implementation of the HTTP COPY verb, this is once every 20 markers or so).
However, the underlying NEON interface can return data once every encoded chunk (in xrootd-tpc
, this is once per performance marker). This means we could provide the performance markers once they come in, instead of buffering into 1KB batches.
It looks like davix cannot be used in child processes if already used in the parent forking the child.
Spurious buffers maybe present in new connections open by children. In a test ROOT application the problem is systematically occurring when more than 3 child processes are created, rare with 1 or 2 children.
It looks like something does not get re-initialized on subsequent open calls.
Any hint would be welcome.
Is there a way to completely reset Davix in a new process?
A ROOT reproducer can be provided if needed.
Could Davix please offer the possibility to build against an external libneon package? I intend to package Davix in Gentoo Linux for use as a dependency by ROOT, but in Gentoo packages must not bundle any dependencies. When trying to use an external libneon, I noticed that neon-0.30 (available in Gentoo) is incompatible with neon-0.29 bundled in Davix.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.