Git Product home page Git Product logo

Comments (21)

JonathanHenson avatar JonathanHenson commented on June 12, 2024

You can return anything that inherits from std::iostream in your closure, we haven't implemented something that does what you need, but if you want to do that, you can create some sort of adapter to make it happen.

As for that comment, if the http response returns an error, we write the payload out to the stream, but if the in flag is not turned on, we can't parse the error. Nothing will break if you don't pass that flag, but you will recieve an "UNKNOWN" error if something goes wrong.

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

One thing you could do is create your file stream in the closure, and then seek it to where you want it to start. The system should start writing at the seeked position.

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024

And I should not expect any internal retry mechanisms to conflict with this logic?

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

You can turn retries off or override the retry strategy if you are worried about it. But no, the response stream is recreated for each new request/response cycle (so a retry would just recreate the stream)

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

One word of caution. This is a use case I haven't thought about very much and in the event of an error coming across the wire, the error will be written to the stream you provide, so you'll either need to handle that, or possibly maintain your file io stuff separately and just use our default response stream, then write the response to your file manually after you've checked the error code.

I'm not to worried about that sort of thing for normal file stream use cases, but yours is special and as a result my be problematic if an xml doc error payload is in the middle of your 40gb file.

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

S3 also has a torrent api, just fyi

http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024

Okay, looks like I will steer clear of the untrodden path for now and take the hit from the disk copy. Thanks.

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

You don't have to do a disk copy, if you have sufficient ram available, the default stream will use a stringbuffer, but you need to be aware of how large the objects are that you are downloading.

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024

So I tried to implement this, and it appears to be segfaulting.

#include <gtest/gtest.h>
#include <glog/logging.h>

#include <fstream>
#include <memory>

#include "rcp_net/NetFile.hpp"

#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpRequest.h>
#include <aws/core/http/HttpResponse.h>

using namespace Aws;
using namespace Client;
using namespace Http;


Aws::Client::ClientConfiguration Config;

TEST(NET, Basic) {
    std::shared_ptr<HttpClientFactory> factory(new HttpClientFactory);
    std::shared_ptr<HttpClient> client(factory->CreateHttpClient(Config)); // sigh

    std::string filePath("someTestPath");
    size_t offset = 10;

    std::shared_ptr<HttpRequest> request(
        factory->CreateHttpRequest(
            Aws::String("http://google.com"),
            HttpMethod::HTTP_GET,
            [&filePath, &offset]() -> Aws::IOStream* {
                constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
                Aws::IOStream* ofs = new Aws::FStream(filePath, OpenFlag);
                CHECK(ofs->good());
                return ofs;
            }));

    std::shared_ptr<HttpResponse> response = client->MakeRequest(*request); //<-- *** Segfaults there ***
    ASSERT_EQ(HttpResponseCode::OK,
              response->GetResponseCode());

$> valgrind --tool=memcheck ./gtests --gtest_filter='NET.*'
==10845== Memcheck, a memory error detector
==10845== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10845== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==10845== Command: ./gtests --gtest_filter=NET.*
==10845== 
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN      ] NET.Basic
==10845== Invalid read of size 8
==10845==    at 0x64F1309: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Invalid read of size 8
==10845==    at 0x64F132B: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098

TEST(NET, Basic) {
    std::shared_ptr<HttpClientFactory> factory(new HttpClientFactory);
    std::shared_ptr<HttpClient> client(factory->CreateHttpClient(Config)); // sigh

    std::string filePath("someTestPath");
    size_t offset = 10;

    std::shared_ptr<HttpRequest> request(
        factory->CreateHttpRequest(
            Aws::String("http://google.com"),
            HttpMethod::HTTP_GET,
            [&filePath, &offset]() -> Aws::IOStream* {
                constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
                Aws::OFStream* ofs = new Aws::OFStream(filePath, OpenFlag);
                CHECK(ofs->good());
                return dynamic_cast<Aws::IOStream*>(ofs);
            }));


    std::shared_ptr<HttpResponse> response = client->MakeRequest(*request); //<-- *** Segfaults there ***
    ASSERT_EQ(HttpResponseCode::OK,
              response->GetResponseCode());


$> valgrind --tool=memcheck ./gtests --gtest_filter='NET.*'
==10845== Memcheck, a memory error detector
==10845== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10845== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==10845== Command: ./gtests --gtest_filter=NET.*
==10845== 
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN      ] NET.Basic
==10845== Invalid read of size 8
==10845==    at 0x64F1309: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Invalid read of size 8
==10845==    at 0x64F132B: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Invalid read of size 8
==10845==    at 0x64F16D1: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Jump to the invalid address stated on the next line
==10845==    at 0x8B48FFF9A57AE810: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0x8b48fff9a57ae810 is not stack'd, malloc'd or (recently) free'd
==10845== 
*** Aborted at 1443730215 (unix time) try "date -d @1443730215" if you are using GNU date ***
PC: @ 0x8b48fff9a57ae810 (unknown)
==10845== Syscall param msync(start) points to uninitialised byte(s)
==10845==    at 0x4E46B3D: ??? (syscall-template.S:81)
==10845==    by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02643: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C04EAA: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C06151: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C064E8: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02A30: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845==    by 0x8B48FFF9A57AE80F: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==  Address 0xffeffe050 is on thread 1's stack
==10845== 
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845==    at 0x4E46B3D: ??? (syscall-template.S:81)
==10845==    by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02B31: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845==    by 0x8B48FFF9A57AE80F: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==  Address 0x8b48fff9a57ae000 is not stack'd, malloc'd or (recently) free'd
==10845== 
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845==    at 0x4E46B3D: ??? (syscall-template.S:81)
==10845==    by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02BC1: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845==    by 0x8B48FFF9A57AE80F: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==  Address 0xe56f920 is 0 bytes after a block of size 35,984 alloc'd
==10845==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x7E34529: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E435F5: curl_easy_init (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59083EF: Aws::Http::CurlHandleContainer::CheckAndGrowPool() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x59085B1: Aws::Http::CurlHandleContainer::AcquireCurlHandle() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x590964E: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
*** SIGSEGV (@0x8b48fff9a57ae810) received by PID 10845 (TID 0x405f580) from PID 18446744072190879760; stack trace: ***
    @          0x4e47340 (unknown)
    @ 0x8b48fff9a57ae810 (unknown)
==10845== 
==10845== HEAP SUMMARY:
==10845==     in use at exit: 576,891 bytes in 6,082 blocks
==10845==   total heap usage: 8,543 allocs, 2,461 frees, 740,617 bytes allocated
==10845== 
==10845== LEAK SUMMARY:
==10845==    definitely lost: 0 bytes in 0 blocks
==10845==    indirectly lost: 0 bytes in 0 blocks
==10845==      possibly lost: 9,521 bytes in 211 blocks
==10845==    still reachable: 567,370 bytes in 5,871 blocks
==10845==         suppressed: 0 bytes in 0 blocks
==10845== Rerun with --leak-check=full to see details of leaked memory
==10845== 
==10845== For counts of detected and suppressed errors, rerun with: -v
==10845== Use --track-origins=yes to see where uninitialised values come from
==10845== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0)

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

Not sure if this is the problem or not but I don't think ofstream is a
subclass of iostream
On Oct 1, 2015 1:13 PM, "Christopher J. Hanks" [email protected]
wrote:

So I tried to implement this, and it appears to be segfaulting.

#include <gtest/gtest.h>
#include <glog/logging.h>

#include
#include

#include "rcp_net/NetFile.hpp"

#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpRequest.h>
#include <aws/core/http/HttpResponse.h>
using namespace Aws;using namespace Client;using namespace Http;

Aws::Client::ClientConfiguration Config;
TEST(NET, Basic) {
std::shared_ptr factory(new HttpClientFactory);
std::shared_ptr client(factory->CreateHttpClient(Config)); // sigh

std::string filePath("someTestPath");
size_t offset = 10;

std::shared_ptr<HttpRequest> request(
    factory->CreateHttpRequest(
        Aws::String("http://google.com"),
        HttpMethod::HTTP_GET,
        [&filePath, &offset]() -> Aws::IOStream* {
            constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
            Aws::OFStream* ofs = new Aws::OFStream(filePath, OpenFlag);
            CHECK(ofs->good());
            return dynamic_cast<Aws::IOStream*>(ofs);
        }));


std::shared_ptr<HttpResponse> response = client->MakeRequest(*request); //<-- *** Segfaults there ***
ASSERT_EQ(HttpResponseCode::OK,
          response->GetResponseCode());

$> valgrind --tool=memcheck ./gtests --gtest_filter='NET.'
==10845== Memcheck, a memory error detector
==10845== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10845== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==10845== Command: ./gtests --gtest_filter=NET.
==10845==
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN ] NET.Basic
==10845== Invalid read of size 8
==10845== at 0x64F1309: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64F16BD: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845== at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x64D0497: std::basic_filebuf<char, std::char_traits >::M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64D4541: std::basic_filebuf<char, std::char_traits >::open(char const, std::Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x43511B: std::Function_handler<std::iostream (), NET_Basic_Test::TestBody()::{lambda()#1}>::M_invoke(std::Any_data const&) (fstream:281)
==10845== by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
==10845== Invalid read of size 8
==10845== at 0x64F132B: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64F16BD: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845== at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x64D0497: std::basic_filebuf<char, std::char_traits >::M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64D4541: std::basic_filebuf<char, std::char_traits >::open(char const, std::Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x43511B: std::Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::M_invoke(std::Any_data const&) (fstream:281)
==10845== by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const_) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
==10845== Invalid read of size 8
==10845== at 0x64F16D1: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845== at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x64D0497: std::basic_filebuf<char, std::char_traits >::M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64D4541: std::basic_filebuf<char, std::char_traits >::open(char const, std::Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x43511B: std::Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::M_invoke(std::Any_data const&) (fstream:281)
==10845== by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const_) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
==10845== Jump to the invalid address stated on the next line
==10845== at 0x8B48FFF9A57AE810: ???
==10845== by 0x64F16E2: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== Address 0x8b48fff9a57ae810 is not stack'd, malloc'd or (recently) free'd
==10845==
_* Aborted at 1443730215 (unix time) try "date -d @1443730215" if you are using GNU date ***
PC: @ 0x8b48fff9a57ae810 (unknown)
==10845== Syscall param msync(start) points to uninitialised byte(s)
==10845== at 0x4E46B3D: ??? (syscall-template.S:81)
==10845== by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02643: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C04EAA: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C06151: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C064E8: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02A30: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x5677442: google::GetStackTrace(void*, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845== by 0x8B48FFF9A57AE80F: ???
==10845== by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== Address 0xffeffe050 is on thread 1's stack
==10845==
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845== at 0x4E46B3D: ??? (syscall-template.S:81)
==10845== by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02B31: ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845== by 0x8B48FFF9A57AE80F: ???
==10845== by 0x64F16E2: std::ostream::write(char const, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== Address 0x8b48fff9a57ae000 is not stack'd, malloc'd or (recently) free'd
==10845==
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845== at 0x4E46B3D: ??? (syscall-template.S:81)
==10845== by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02BC1: ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x5677442: google::GetStackTrace(void_, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845== by 0x8B48FFF9A57AE80F: ???
==10845== by 0x64F16E2: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== Address 0xe56f920 is 0 bytes after a block of size 35,984 alloc'd
==10845== at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x7E34529: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E435F5: curl_easy_init (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59083EF: Aws::Http::CurlHandleContainer::CheckAndGrowPool() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x59085B1: Aws::Http::CurlHandleContainer::AcquireCurlHandle() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x590964E: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
*_* SIGSEGV (@0x8b48fff9a57ae810) received by PID 10845 (TID 0x405f580) from PID 18446744072190879760; stack trace: ***
@ 0x4e47340 (unknown)
@ 0x8b48fff9a57ae810 (unknown)
==10845==
==10845== HEAP SUMMARY:
==10845== in use at exit: 576,891 bytes in 6,082 blocks
==10845== total heap usage: 8,543 allocs, 2,461 frees, 740,617 bytes allocated
==10845==
==10845== LEAK SUMMARY:
==10845== definitely lost: 0 bytes in 0 blocks
==10845== indirectly lost: 0 bytes in 0 blocks
==10845== possibly lost: 9,521 bytes in 211 blocks
==10845== still reachable: 567,370 bytes in 5,871 blocks
==10845== suppressed: 0 bytes in 0 blocks
==10845== Rerun with --leak-check=full to see details of leaked memory
==10845==
==10845== For counts of detected and suppressed errors, rerun with: -v
==10845== Use --track-origins=yes to see where uninitialised values come from
==10845== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0)

—
Reply to this email directly or view it on GitHub
#21 (comment).

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024

Yes apologies, I have corrected the example so that no wild casting is required (that was a clip from one of my debugging permutations).

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

Did you rerun after making that change?
On Oct 1, 2015 1:19 PM, "Christopher J. Hanks" [email protected]
wrote:

Yes apologies, I have corrected the example so that no wild casting is
required (that was a clip from one of my debugging permutations).

—
Reply to this email directly or view it on GitHub
#21 (comment).

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024 
==11261== Memcheck, a memory error detector
==11261== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==11261== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==11261== Command: ./gtests --gtest_filter=NET.*
==11261== 
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN      ] NET.Basic
==11261== Mismatched free() / delete / delete []
==11261==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x5911E80: Aws::Utils::Stream::ResponseStream::ReleaseStream() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x5907C86: Aws::Http::Standard::StandardHttpResponse::~StandardHttpResponse() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x435B78: NET_Basic_Test::TestBody() (shared_ptr_base.h:144)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0xe56fbf0 is 0 bytes inside a block of size 528 alloc'd
==11261==    at 0x4C2B0E0: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x4350CA: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (test_net.cpp:36)
==11261==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x4355C4: NET_Basic_Test::TestBody() (test_net.cpp:41)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435D0E: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435DCE: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435DF7: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435DA5: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Mismatched free() / delete / delete []
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435E20: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0xe5089f0 is 0 bytes inside a block of size 84 alloc'd
==11261==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x58E5D87: char* std::basic_string<char, std::char_traits<char>, Aws::Allocator<char> >::_S_construct<char*>(char*, char*, Aws::Allocator<char> const&, std::forward_iterator_tag) (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x58FD22B: Aws::Client::ComputeUserAgentString() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x58FD50D: Aws::Client::ClientConfiguration::ClientConfiguration() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x43545B: NET_Basic_Test::TestBody() (test_net.cpp:23)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261== 
[       OK ] NET.Basic (1543 ms)
[----------] 1 test from NET (1558 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1587 ms total)
[  PASSED  ] 1 test.
==11261== 
==11261== HEAP SUMMARY:
==11261==     in use at exit: 180,644 bytes in 3,357 blocks
==11261==   total heap usage: 8,593 allocs, 5,240 frees, 744,842 bytes allocated
==11261== 
==11261== LEAK SUMMARY:
==11261==    definitely lost: 0 bytes in 0 blocks
==11261==    indirectly lost: 0 bytes in 0 blocks
==11261==      possibly lost: 1,652 bytes in 25 blocks
==11261==    still reachable: 178,992 bytes in 3,332 blocks
==11261==         suppressed: 0 bytes in 0 blocks
==11261== Rerun with --leak-check=full to see details of leaked memory
==11261== 
==11261== For counts of detected and suppressed errors, rerun with: -v
==11261== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)

So it is "PASS"ing with valgrind, but definitely failing.

*** Error in `./gtests': free(): invalid pointer: 0x00007f38bf3e5500 ***
*** Aborted at 1443730984 (unix time) try "date -d @1443730984" if you are using GNU date ***
PC: @     0x7f38bdc6dcc9 (unknown)
*** SIGABRT (@0x3e800002c1a) received by PID 11290 (TID 0x7f38c0028780) from PID 11290; stack trace: ***
    @     0x7f38bfc37340 (unknown)
    @     0x7f38bdc6dcc9 (unknown)
    @     0x7f38bdc710d8 (unknown)
    @     0x7f38bdcaa394 (unknown)
    @     0x7f38bdcb666e (unknown)
    @           0x435d0f NET_Basic_Test::TestBody()

    @           0x46e603 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x467217 testing::Test::Run()
    @           0x4672be testing::TestInfo::Run()
    @           0x4673c5 testing::TestCase::Run()
    @           0x467668 testing::internal::UnitTestImpl::RunAllTests()
    @           0x467907 testing::UnitTest::Run()
    @           0x410ea2 main
    @     0x7f38bdc58ec5 (unknown)
    @           0x413d07 (unknown)
    @                0x0 (unknown)

It appears that the error is occurring during some form of static destructor from within the AWS library, because my request completes successfully at this point.

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

If you compiled the library with custom memory management turned on, are you passing -DAWS_CUSTOM_MEMORY_MANAGEMENT to your compiler for your test program?

It looks like the allocators and deallocators are mismatched.

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024 
# LIBRARY BUILD
buildro=install_root
cmake \
    CMakeLists.txt \
    -DCMAKE_BUILD_TYPE=Release  \
    -DCMAKE_INSTALL_PREFIX=${buildro}/usr \
    -DSDK_INSTALL_BINARY_PREFIX=${buildro}/usr/lib \
    -DTARGET_ARCH=LINUX

make -j8
make install

// Recreate.cpp
#include <gtest/gtest.h>
#include <glog/logging.h>

#include <fstream>
#include <memory>

#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpRequest.h>
#include <aws/core/http/HttpResponse.h>

using namespace Aws;
using namespace Client;
using namespace Http;

int
main() {
    Aws::Client::ClientConfiguration Config;
    std::shared_ptr<HttpClientFactory> factory(new HttpClientFactory);
    std::shared_ptr<HttpClient> client(factory->CreateHttpClient(Config)); // sigh

    std::string filePath("someTestPath");
    size_t offset = 10;

    std::shared_ptr<HttpRequest> request(
        factory->CreateHttpRequest(
            Aws::String("http://google.com"),
            HttpMethod::HTTP_GET,
            [&filePath, &offset]() -> Aws::IOStream* {
                constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
                Aws::IOStream* ofs = new Aws::FStream(filePath, OpenFlag);
                CHECK(ofs->good());
                return ofs;
            }));

    do {
        std::shared_ptr<HttpResponse> response = client->MakeRequest(*request);
        assert(HttpResponseCode::OK == response->GetResponseCode());
    } while (0);
    return 0;
}
*** Error in `/tmp/a.out': free(): invalid pointer: 0x00007ffff7ba1500 ***
...
(gdb) bt
#0  0x00007ffff6fe5cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6fe90d8 in __GI_abort () at abort.c:89
#2  0x00007ffff7022394 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ffff7130b28 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff702e66e in malloc_printerr (ptr=<optimized out>, str=0x7ffff712cc19 "free(): invalid pointer", action=1) at malloc.c:4996
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5  0x00007ffff76581c0 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x0000000000401f10 in Aws::Client::ClientConfiguration::~ClientConfiguration() ()
#7  0x0000000000401972 in main ()

Error is same with or without -DAWS_CUSTOM_MEMORY_MANAGEMENT

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

by default, the library script you just specified will use custom memory management with dynamic linking.

How are you compiling Recreate.cpp ? Can I see the command?

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024

% g++ Recreate.cpp -std=c++11 -lglog -laws-cpp-sdk-core -DDAWS_CUSTOM_MEMORY_MANAGEMENT

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024

it's just -DAWS_CUSTOM_MEMORY_MANAGMENT not
-DDAWS_CUSTOM_MEMORY_MANAGEMENT

Also, you are mixing non managed and managed types.

comments in line:

int
main() {
Aws::Client::ClientConfiguration Config;
std::shared_ptr factory(new HttpClientFactory);
std::shared_ptr client(factory->CreateHttpClient(Config)); // sigh

<------
//sorry, I know that's inconvenient. I'd just do
HttpClientFactory factory;
std::shared_ptr client(factory.CreateHttpClient(Config));
------>

std::string filePath("someTestPath");

<------
//This is going to cause problems when you pass it to a type we manage. Instead, do this:
Aws::String filePath("someTestPath");
//or
const char* filePath = "someTestPath";
------>
size_t offset = 10;

std::shared_ptr<HttpRequest> request(
    factory->CreateHttpRequest(
        Aws::String("http://google.com"),
        HttpMethod::HTTP_GET,
        [&filePath, &offset]() -> Aws::IOStream* {
            constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
            Aws::IOStream* ofs = new Aws::FStream(filePath, OpenFlag);
            CHECK(ofs->good());
            return ofs;
        }));

<----------------
//The "new Aws::FStream" is going to segfault.
//we take ownership of this pointer and clean it up when we finish using our deallocators
//it should be
Aws::IOStream* ofs = Aws::NewAws::FStream("some tag", filePath, OpenFlag);
-------------->

do {
    std::shared_ptr<HttpResponse> response = client->MakeRequest(*request);
    assert(HttpResponseCode::OK == response->GetResponseCode());
} while (0);
return 0;

}

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024

Okay, so we are pretty close...

// g++ -std=c++11 Recreate.cpp -laws-cpp-sdk-core  -DAWS_CUSTOM_MEMORY_MANAGEMENT -lglog
#include <gtest/gtest.h>
#include <glog/logging.h>

#include <fstream>
#include <memory>
#include <string>

#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpRequest.h>
#include <aws/core/http/HttpResponse.h>

using namespace Aws;
using namespace Client;
using namespace Http;

int
main() {
    Aws::Client::ClientConfiguration Config;
    HttpClientFactory factory;
    std::shared_ptr<HttpClient> client(factory.CreateHttpClient(Config));

    Aws::String filePath("someTestPath");
    size_t offset = 10;

    std::shared_ptr<HttpRequest> request(
        factory.CreateHttpRequest(
            Aws::String("http://google.com"),
            HttpMethod::HTTP_GET,
            [&filePath, &offset]() -> Aws::IOStream* {
                constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
                Aws::IOStream* ofs = Aws::New<Aws::FStream>("BOGUS", filePath.c_str(), OpenFlag);
                CHECK(ofs->good());
                CHECK(ofs->seekp(offset, ofs->beg));
                return ofs;
            }));

    std::shared_ptr<HttpResponse> response = client->MakeRequest(*request);
    assert(HttpResponseCode::OK == response->GetResponseCode());

    return 0;
}

This code does everything my heart currently desires. But there a few curious things.

  1. I had to change Aws::NewAws::FStream to Aws::New<Aws::FStream>, is that correct?
  2. Why must the second argument of Aws::New<Aws::FStream>( be converted to a C string? Without it the overload is apparently ambiguous
% g++ -std=c++11 test_net.cpp -laws-cpp-sdk-core  -DAWS_CUSTOM_MEMORY_MANAGEMENT -lglog
In file included from /usr/include/aws/core/utils/memory/stl/AWSAllocator.h:20:0,
                 from /usr/include/aws/core/utils/memory/stl/AWSString.h:20,
                 from /usr/include/aws/core/http/Scheme.h:20,
                 from /usr/include/aws/core/client/ClientConfiguration.h:21,
                 from test_net.cpp:8:
/usr/include/aws/core/utils/memory/AWSMemory.h: In instantiation of ‘T* Aws::New(const char*, ArgTypes&& ...) [with T = std::basic_fstream<char>; ArgTypes = {std::basic_string<char, std::char_traits<char>, Aws::Allocator<char> >&, const std::_Ios_Openmode&}]’:
test_net.cpp:33:88:   required from here
/usr/include/aws/core/utils/memory/AWSMemory.h:54:77: error: no matching function for call to ‘std::basic_fstream<char>::basic_fstream(std::basic_string<char, std::char_traits<char>, Aws::Allocator<char> >&, const std::_Ios_Openmode&)’
     T *constructedMemory = new (rawMemory) T(std::forward<ArgTypes>(args)...);
                                                                             ^
/usr/include/aws/core/utils/memory/AWSMemory.h:54:77: note: candidates are:
In file included from test_net.cpp:4:0:
/usr/include/c++/4.8/fstream:831:7: note: std::basic_fstream<_CharT, _Traits>::basic_fstream(const string&, std::ios_base::openmode) [with _CharT = char; _Traits = std::char_traits<char>; std::string = std::basic_string<char>; std::ios_base::openmode = std::_Ios_Openmode]
       basic_fstream(const std::string& __s,
       ^
/usr/include/c++/4.8/fstream:831:7: note:   no known conversion for argument 1 from ‘std::basic_string<char, std::char_traits<char>, Aws::Allocator<char> >’ to ‘const string& {aka const std::basic_string<char>&}’
/usr/include/c++/4.8/fstream:816:7: note: std::basic_fstream<_CharT, _Traits>::basic_fstream(const char*, std::ios_base::openmode) [with _CharT = char; _Traits = std::char_traits<char>; std::ios_base::openmode = std::_Ios_Openmode]
       basic_fstream(const char* __s,
       ^
/usr/include/c++/4.8/fstream:816:7: note:   no known conversion for argument 1 from ‘std::basic_string<char, std::char_traits<char>, Aws::Allocator<char> >’ to ‘const char*’
/usr/include/c++/4.8/fstream:803:7: note: std::basic_fstream<_CharT, _Traits>::basic_fstream() [with _CharT = char; _Traits = std::char_traits<char>]
       basic_fstream()
       ^
/usr/include/c++/4.8/fstream:803:7: note:   candidate expects 0 arguments, 2 provided
  1. Should I expect this code to work if I disable memory management?

Thank you for your help.

from aws-sdk-cpp.

JonathanHenson avatar JonathanHenson commented on June 12, 2024
  1. Yes, that's what I typed but markdown and all........
  2. I'm not sure, if I had to guess,

http://en.cppreference.com/w/cpp/io/basic_fstream/basic_fstream

the stdlib authors hard coded std::string which is incompatible with the aliased types (if memory management is turned on).

  1. Yes, this code should work regardless of whether or not you are actually using custom memory managers.

from aws-sdk-cpp.

cjhanks avatar cjhanks commented on June 12, 2024

Thanks.

from aws-sdk-cpp.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.