Creating and joining a number of threads only known at runtime about photonlibos HOT 22 CLOSED

You may want to use this function:
void threads_create_join(uint64_t n, thread_entry start, void* arg, uint64_t stack_size = DEFAULT_STACK_SIZE);

from photonlibos.

Alternatively, you can create a thread pool first, then create threads with that pool, finally join that pool.

from photonlibos.

@aley1 The waitAll() is just an encapsulation. Inside it you still need to manage the thread ids.

Is that you met some issues when using our API model?

    auto th = photon::thread_create11([&]{
        // ...    
    });
    photon::thread_enable_join(th);
    photon::thread_join((photon::join_handle*) th);

    auto th = new photon_std::thread([&]{
        // ...    
    });
    th->join();
    delete th;

from photonlibos.

Hi @lihuiba

You may want to use this function: void threads_create_join(uint64_t n, thread_entry start, void* arg, uint64_t stack_size = DEFAULT_STACK_SIZE);

• I believe I have to pass the same arg value to all the threads when using this function? I am looking to pass different values to different threads to so that each thread can work on different set of data. Otherwise this function is close to what I need.

Hi @beef9999 I think the photon library is great and API is simple to use. Just that I have a use case where I need to call x number of threads (only known during runtime) and pass different data to each thread and then join them all.

In your example, I need to know many threads to create when I am writing the application. But in my use case I don't know the amount - could be 10-20 threads. And each thread needs to be given different data.

I was wondering if you have functions that make it simple to do this. Here is an example how C# tasks does it:
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.waitall?view=net-8.0

from photonlibos.

I think this is just how C++ programs supposed to be written, no other unnecessary encapsulations except for std:: or STL.

std::vector<YourDataStruct> data(input_num);
for (int i = 0; i < data.size(); ++i)
    photon::thread_create11([&](int index){
        data[index];
    }, i);

Is that what you wanted?

from photonlibos.

Sorry I am still fairly new to C++ but I am learning it along the way as I work on a project.
Yes that's it, thank you, and I presume still need to join them or this code will take care of it?

from photonlibos.

Yes, join is still needed.

BTW, creating a new photon thread would consume some CPU resources, even it’s relatively less than the std::thread. To minimize the performance impact to the lowest, you may want to use the thread_pool

from photonlibos.

I am looking to pass different values to different threads to so that each thread can work on different set of data. Otherwise this function is close to what I need.

So you may want to use the thread_pool to create the threads, and join the entire thread_poll.

from photonlibos.

Thank you all for your replies. I should have read these examples in /main/thread/test/test.cpp before posting my comment.

I'm now using a thread pool and trying to the code below for my use case. It's working except still trying to understand passing arguments to my function within each thread create. it seems to be restrictive with the (void* arg) pointer. I believe I should create an array that holds the data and then just pass the index of the array to each thread.

TEST(ThreadPool, test)
{
ThreadPool<64> pool(641024);
vector<TPControl> ths;
ths.resize(FLAGS_ths_total);
for (int i = 0; i<FLAGS_ths_total; i++)
ths[i] = pool.thread_create_ex(&::func1, nullptr, true);
// LOG_INFO("----------");
for (int i = 0; i<FLAGS_ths_total; i++) {
LOG_DEBUG("wait thread: `", ths[i]->th);
pool.join(ths[i]);
}
// LOG_INFO("???????????????");
}

from photonlibos.

void* arg is commonly used in low-level c/c++ interfaces. Actually you can cast any compatible types to that, and cast back in the thread body of course.

Compatible types includes (not limited to): all pointers, all integers not bigger than 8 bytes, char, bool, etc.

For example, suppose you want to pass an int to the new thread, you can write cast as (void*)(int64_t)x, and (int)(int64_t)y to cast back.

from photonlibos.

I have tested the thread library, tcp server/client, and all works fine.
But I have problems with the http server. using code from your /tests dir.

class SimpleHandler : public photon::net::http::HTTPHandler {
public:
    int handle_request(photon::net::http::Request& req, photon::net::http::Response& resp, std::string_view) {
        auto target = req.target();
        resp.set_result(200);
        resp.headers.content_length((size_t) 4096);
        resp.headers.insert("Server", "nginx/1.14.1");
        resp.headers.insert("Content-Type", "application/octet-stream");
        string r = "http server online";
        std::byte header_bytes[r.length()];
        std::memcpy(header_bytes, r.data(), r.length());
          return 0;
    }
};
```

```
int HttpServer(){
    auto tcpserv = photon::net::new_tcp_socket_server();
    tcpserv->bind(5000);
    tcpserv->listen();
    DEFER(delete tcpserv);
    auto http_srv = photon::net::http::new_http_server();
    DEFER(delete http_srv);
    SimpleHandler handler;

    tcpserv->set_handler(http_srv->get_connection_handler());
    tcpserv->start_loop(true);

    return 0;
}
```

Whenever I try to build I get an error like this:
_undefined reference to `photon::net::http::Response::set_result(int, std::basic_string_view<char, std::char_traits<char> >)'
undefined reference to `photon::net::http::HeadersBase::insert(std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, int)'_

I am not sure why it cannot link or find the reference for these methods in Response. 
Everything else works except this few lines of code.

from photonlibos.

I have tested the thread library, tcp server/client, and all works fine. But I have problems with the http server. using code from your /tests dir.

class SimpleHandler : public photon::net::http::HTTPHandler {
public:
    int handle_request(photon::net::http::Request& req, photon::net::http::Response& resp, std::string_view) {
        auto target = req.target();
        resp.set_result(200);
        resp.headers.content_length((size_t) 4096);
        resp.headers.insert("Server", "nginx/1.14.1");
        resp.headers.insert("Content-Type", "application/octet-stream");
        string r = "http server online";
        std::byte header_bytes[r.length()];
        std::memcpy(header_bytes, r.data(), r.length());
          return 0;
    }
};

int HttpServer(){
    auto tcpserv = photon::net::new_tcp_socket_server();
    tcpserv->bind(5000);
    tcpserv->listen();
    DEFER(delete tcpserv);
    auto http_srv = photon::net::http::new_http_server();
    DEFER(delete http_srv);
    SimpleHandler handler;

    tcpserv->set_handler(http_srv->get_connection_handler());
    tcpserv->start_loop(true);

    return 0;
}

Whenever I try to build I get an error like this:
undefined reference to photon::net::http::Response::set_result(int, std::basic_string_view<char, std::char_traits<char> >)' undefined reference to photon::net::http::HeadersBase::insert(std::basic_string_view<char, std::char_traits >, std::basic_string_view<char, std::char_traits >, int)'

I am not sure why it cannot link or find the reference for these methods in Response.
Everything else works except this few lines of code.

Could your show your CMakeLists.txt ?

string -> std::string ?

from photonlibos.

cmake_minimum_required(VERSION 3.24)
project(myproject)

set(CMAKE_CXX_STANDARD 20)
add_library(libphoton SHARED IMPORTED)

add_executable(myproject main.cpp)
find_library(PHOTON libphoton.so lib)
target_link_libraries(myproject LINK_PUBLIC ${PHOTON}

from photonlibos.

string -> std::string ?

FetchContent is recommended. https://github.com/alibaba/PhotonLibOS/blob/main/doc/CMakeLists.txt

from photonlibos.

std::string does not affect it. I tried with the CMakeList example with FetchContent. Still the same error. not sure why.

from photonlibos.

Looks like the root cause is gcc version or c++ std version. What is your environment like?

Photon is using C++14. Theoretically 17 and 20 is supported. But had not been tested. Could you change the CMAKE_CXX_STANDARD to 17 and try again?

from photonlibos.

BTW, would you like to disclose what kind of project it is that you're working on? Just a survey.

from photonlibos.

I am working on stream processing project - parsing unstructured data to json then doing aggregations on it as well.

from photonlibos.

Looks like the root cause is gcc version or c++ std version. What is your environment like?

Photon is using C++14. Theoretically 17 and 20 is supported. But had not been tested. Could you change the CMAKE_CXX_STANDARD to 17 and try again?

Tried with 17 and same error. I cannot set it to lower than 17 for my project. It only seems the http Response object is affected. Perhaps there is code in there that 17 and above does not support?

from photonlibos.

I have just reproduced this error with C++17. Will try to fix it

from photonlibos.

The reason seems to relate to the <string_view> header.

C++14 and C++17 have different symbols of std::string_view. One is std::basic_string_view<char, std::char_traits > and the other is std::basic_string_view<char>

Photon's CMakeLists.txt has explicitly specified C++14, while your app is C++17. You might need a patch to change Photon's C++ version, and place it in the FetchContent part of your Cmake.

FetchContent_Declare(
        photon
        GIT_REPOSITORY https://github.com/alibaba/PhotonLibOS.git
        GIT_TAG v0.4.0
        PATCH_COMMAND git apply a.patch
)
FetchContent_MakeAvailable(photon)

from photonlibos.

Thanks for the tip! I managed to get it to build now. I have building PhotonLibOS separately and then attaching the library to my project (not using FetchContent). I just updated PhotonLibOS c++ version in CMakeLists.txt to 17 and build again.With the new output library in my project I can now compile successfully. It should be sufficient for now as I am still testing and prototyping

from photonlibos.