tomasbrod / tbboinc Goto Github PK

Currently serialization is handles using hand-written write and parse functions. This is cumbersome, due to the sheer amount of code written, and error-prone, because these two functions are separate. We also have separate functions to marshall and unmarshall objects into SQL database. I propose to unify and clean up these using either table-driven or template-driven automatic serialization.
Existing libraries include: boost/archive, A Working Library, Cereal, eyalz800 and the bitcoin serializer.
I think, none of them would work as-is and custom microframework for our use will be needed. Anyway, I predict, it will be less code, than what there currently is.

Approach a) each serializable structure defines a table containing description of the struct fields, xml-name, type and maybe custom handler. Macros can help defining this table. Then a set of general functions will, with the help of the table, write or parse the struct.
Approach b) use templates and overloading to make to implement the same (like boost/archive and cereal). Currently the codebase is limited to C++03, which may, or may not affect this approach.

Gains: More developers interested in working with the code once they see it's not utter mess. Less duplicate code. Possibility to switch to a better format than XML in the future. Make it more developer-friendly.

Compatibility with the old clients (thus XML) should be maintained.

The XML part of the framework should be based on some standard SAX parsing library, see: BOINC/boinc#2070 . Using DOM library may make it easier to implement, but also slower. Thoughts?

// sample structure with the approach B
struct MSG_FROM_HOST_DESC {
  char variety[256];
  std::string msg_text;
  template<clsss Seralizer> serialize_func(Ser& s) {
    SFIELD(variety);
    SFIELD(msg_text);
  }
};

Allow passing small opaque XML fragment from scheduler response into the app on client.
This is to avoid sending few-byte inputs via standalone files.

Currently the scheduler is monolithic and is (effectively) limited to assigning tasks from the feeder shared memory generated by transitioner. Only few projects are comfortable with pre-generated tasks. Some projects could benefit from:

• on-demand task generation and
• more customized work assignment

For this reason, I decided to re-design the scheduler to be modular - either dlopen loading feeder plugins, or IPC to running feeders. Additionally, the database schema for workunits and results will be redesigned.

Building block: Queue:
Project can define multiple queues with priorities. Each queue has a feeder name and arguments associated to it. Scheduler weights the queues by priority and then sequentially asks their associated feeders to assign tasks to the host, until enough work has been assigned or there is no more. Users should be able to influence the selection of queues (opt-in, opt-out, priority).

Building block: Feeder:
A feeder acts as a plugin to the scheduler, either in-process in for of dlopened shared library, or separate process accesible via IPC (unix-domain sockets, most likely). Job of the feeder is to assign appropriate tasks to the requesting host, obeying the host's hardware capabilities.

Workunit feeder:
This feeder will be designed to replace the standard assignment of jobs to hosts. The wu-feeder will select workunit from the database, based on some configured criteria, and assign it. In this sense, it takes part of the role of a transitioner.

The boinc FUH is trash. I redone it in my project to write to database instead, because I need that. It would not be good for other boinc projects performance-vise.
The signing of uploads with RSA is superfluous. Simple MAC (or keyed-hash) is enough. Submit that.
In my impl i need to add check whether result is in right state to upload. But it NEVER overwrites already.
Check how the boinc fuh ensures immutability.