sociomantic-tsunami / ocean Goto Github PK

The following pattern has been seen:

scope it = new ChrSplitIterator('\n');
it.reset(data);

ChrSplitIterator could optionally accept data string in the constructor, so no call for unintuitive reset (in case of scope instances) is needed.

Those modules have been replaced by ocean.text.convert.Formatter and should be deprecated as they don't play well with D2.
Format depends on Layout_tango, as does Layout.

Once #203 and #187 have been pulled, we will be able to deprecate Layout_tango. As 203 includes a breaking change, this will only happen in v4.0.0 at best.

Feel free to assign me.

Now that we have UnixSocket support in DaemonApp, it would be nice to get a set of basic commands in (which can be overriden / skipped if the dev wants to).

One of those would be the ability to change logger configurations live, so one can temporarily enable / disable some loggers for debugging purpose.

Currently if you have a D2 only application and you decided to use ocean then you have to do the following:

• Make sure that you have d1to2fix installed on your system
• Add Ocean as a submodule
• Go to the ocean submodule directory and checkout the desired branch
• Add Makd as a submodule to ocean
• Run make d2conv
• If your project is under version-control, then you need to commit the converted code so you can have none-dirty commits.
• And only at that point you are ready to use ocean.

The above procedure may deter a lot of D2 users who would like to test Ocean. It would be much more convenient to have a new ocean-d2 repo that contains converted snapshots of the released tags of this repo.
Syncing the d2 repo can be automatically maintained by Travis.
Also, I don't know if current ocean is in dub, but certainly the converted Ocean can live there.

fyi @leandro-lucarella-sociomantic

Disclaimer: I'm a direct beneficiary of this issue. I will give a (2x-slot-)lightning talk about Ocean in DConf. I'd like the ease of using Ocean to be one of the selling points.

Add a static function to Time named for example fromUnixTime that converts a time_t to a Time. There seems to be no function of such functionality in ocean.

It should be possible to change toCString from

public static char* toCString ( ref mstring str )

to

public static Inout!(char)* toCString ( ref Inout!(char)[] str )

Because the implementation does not rely on str being mutable. Having it as Inout allows passing cstrings or even istrings.

Note this also applies to the Wchar overload.

The return value of the sink delegate (size_t delegate ( cstring chunk )) is currently unused. For more convenient usage, it'd be nice for Formatter to support void sink delegates.

There are many situations where you want to wait for a Task to finish or a certain time to pass. (e.g. connecting to a server with some bail-out logic if the connection isn't promptly established.)

It's a common request, and would be nice to have comprehensive docs on that.
The format is inherited from Layout, and is currently DMD-style documentated (read the code).

Currently every Logger allocates 2kB of data:

This is used by the Logger.format. However this introduces a limit in the output message, and an extra copy during logging.

I think with the Formatter approach of using delegates we should be able to directly output to the Appender(s), saving the memory allocation, the extra copy, and removing the limitation to 2048 char (or whatever the buffer length is).

@don-clugston-sociomantic commented on an internal sociomantic project:

The functions in ocean.text.Search are templated on the character type T. The constructor (actually opCall) takes this type from the parameter of the string passes to it. If it's a string literal it will be immutable char and if a variable it will be char. This is never what you want. Probably this could fixed by throwing Unqual everywhere in that module, but I don't think there's much point.

Those Fructs were originally created to work around bugs in D's template instantiation, which forced you to explicitly specify the types. The bugs were fixed about a decade ago. Now the Fructs do the opposite, they create type mismatch problems! And that's the only thing they do. They don't do anything useful.

It is probably best to deprecate and later remove this module to prevent someone from trying to use it.

cc @mathias-lang-sociomantic

Currently, TimerExt of DaemonApp is based on the TimerSet, which with few more inheritance layer goes back to TimeoutManager. TimeoutManager by default uses BucketElementGCAllocator for allocating internals, which uses GC to allocate.

However, TimeoutManager allows for configuring this, so another allocation strategies may be used, such malloc/free via BucketElementMallocAllocator, for example.

For the applications which want to avoid GC activity, it should be allowed to use some strategy other than BucketElementGCAllocator.

CodeCov as it is, is not being very useful, as we get red PRs that are OK.

I know there is #204, but it seems other project had issues with it, so maybe we can analyse taking similar steps.

Working on #37 I found myself wishing a way to interact with the user. Currently commands are one-shot: One send a command, an action is performed and one or multiple messages are sent back.

However, certain actions are complex or dangerous, in which case the handler might want to interact with the user. For example, here's an example of using adduser:

% sudo adduser test
Adding user `test' ...
Adding new group `test' (1002) ...
Adding new user `test' (1001) with group `test' ...
The home directory `/home/test' already exists.  Not copying from `/etc/skel'.
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
Changing the user information for test
Enter the new value, or press ENTER for the default
	Full Name []: Test
	Room Number []: 42   
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] Y

In the case of reconfiguring loggers it's even more sensible, because one might just want to change the logger level. However, the config currently allow to:

• Change the level
• Toggle write to console
• Change the file the logger is writing to
• Change the console layout
• Change the file layout
• Propagate settings

While it is possible to handle this via a variety of method, a mean to interact with the user would be welcome. However, AFAICS it would require some redesign of UnixSocketHandler.

Example logfile - 2017-04-07 13:32:22,893 cpu_user:89.94 cpu_system:9.89 cpu_total:99.83 vsz:224120832 rss:190791680 mem_percentage:inf

Server reports 62.7G RAM.

Extending the following unit test to check for the decoding of padded data shows that the decoding is broken.

https://github.com/sociomantic-tsunami/ocean/blob/v2.x.x/src/ocean/util/encode/Base64.d#L514

// Encode without padding
unittest
{
    istring str = "Hello, how are you today?";
    Const!(ubyte)[] payload = cast(Const!(ubyte)[]) str;
    cstring result = encode(payload, false);
    test!("==")(result, "SGVsbG8sIGhvdyBhcmUgeW91IHRvZGF5Pw");

  
    auto decoded = cast(cstring)decode(result);
    test!("==")(decoded, str);
}

results in

./src/ocean/util/encode/Base64.d:521: test error: expression 'Hello, how are you today == Hello, how are you today?' evaluates to false

I can't remember the reason why https://github.com/sociomantic-tsunami/ocean/blob/v2.x.x/src/ocean/task/Scheduler.d#L455 was added, will need to investigate that again.

After we added code coverage checks, it started to complain sporadically about unexpected changes in few modules, most often src/ocean/math/random/engines/KissCmwc.d. Initially I dismissed it as either codecov or dmd bug, but recently I have realized that it is becomes those are tested with actual random seed and resulting coverage is not deterministic.

Right now for some reason travis seems to be failing to cache the images that we pull and push to the docker registry to work as a cache, thus making the whole pulling and pushing just another time drain.

I already contacted Travis CI and they gave me some sort of experimental SSH access to the VM a job was run on, so I will be able to debug this and hopefully find a solution.

Since to be able to test this I need also for the docker pull/push is in a branch (PRs are not trusted with docker credentials), for now we should not disable the pull/push until I can have some time to investigate this. FYI @mihails-strasuns-sociomantic

The double-repo approach doesn't scale, so we'll use the same repo and just push tags with a +d2 build metadata information appended to the version (so tag v2.3.4 --- conversion D2 ---> v2.3.4+d2).

For reference, this is the PR that introduced the conversion and pushing: #100

Thanks to the new import rules in 2.071.0+, the design of LibFeatures doesn't work.

The definition is currently:

module ocean.LibFeatures;

static immutable has_features_2_7 = true;
static immutable has_features_2_6 = true;
static immutable has_features_2_5 = true;
static immutable has_features_2_4 = true;
static immutable has_features_2_3 = true;
static immutable has_features_2_1 = true;
static immutable has_features_2_0 = true;

And the usage is:

private bool hasFeaturesFromFunc ( istring libname, ulong major, ulong minor ) ()
{
    struct Library
    {
        mixin("import " ~ libname ~ ".LibFeatures;");
    }

   // Generates `return is(typeof(Library.has_features_2_0));` for example
    mixin("return is(typeof(Library.has_features_"
        ~ ctfe_i2a(major) ~ "_" ~ ctfe_i2a(minor) ~ "));");
}

However this doesn't work with new import rules, which make import scope not accessible from outside of an aggregate.
CC @mihails-strasuns-sociomantic

Follow-up for #94 to be done in next major release. Informational warning about using const with deserializer should be replaced with hard assertion and tests that ensure such code is rejected has to be re-enabled.

#24 disables the signalfd test due to a level of flakiness. This should be fixed / rewritten and re-enabled.

void main ()
{
    int[] arr;
    Stdout.print(format("{}\n", arr));

    arr = new int[5];
    arr.length = 0;
    Stdout.print(format("{}\n", arr));
}

This code must print [] twice. Right now it has special case for arr.ptr is null and will print null for a first call.

As this is a subtle breaking change, scheduling it to v4.0.0 for now.

Essentially another form of the goal to clean up assert usage in ocean to make -release flag feasible for our applications. Right now it doesn't even pass tests because some failed assertions are checked with testThrown.

Writing to a logger using ocean.util.log.Log ultimately does this:

Logger format (Level level, cstring fmt, TypeInfo[] types, ArgList args)
        {
                if (types.length)
                {
                    if (buffer_ is null)
                        formatWithDefaultBuffer(level, fmt, types, args);
                    else
                        formatWithProvidedBuffer(level, fmt, types, args);
                }
                else
                   append (level, fmt);
                return this;
        }

Note that if no types are provided, it simply does an append rather than formatting the string.
Logger behaves differently. It always formats. This means that it uses the buffer, which means the string is truncated at 2048 characters.

Code like the following is very common:

void doSomethingWithMyArray ( ref void[] buf )
{
  buf.length = 0;
  enableStomping(buf);
  fillArrayWithSomething(buf);
}

Live testing of calling such code in a tight loop has revealed that the impact of enableStomping (which, in turn, calls assumeSafeAppend) on the D2 build is significant. (One concrete example: iterating over a large database in memory and writing it out to disk was about 20% slower in the D2 build.)

One way to fix this is to never reduce the length of the referenced array and separately keep track of the actual content. (This is what AppendBuffer does, by the way.)

Buffer already handles a lot of D1/D2 array optimisation concerns (e.g. automatically calling enableStomping, where appropriate). It could probably be enhanced to support this optimisation as well, essentially merging AppendBuffer into Buffer. (We could then deprecate AppendBuffer.)

The ocean.text.convert.Float module defines a specialized format function that writes a formatted string of a floating point number into a provided buffer. This is used under the hood in ocean.text.convert.Formatter to handle floating-point values.

However, having this separate public format definition produces a naming clash any time anyone wants to import both ocean.text.convert.Float and ocean.text.convert.Formatter (e.g. if one wants to use, say, toFloat from the former and format from the second).

It would be better to bring floating-point formatting in under the hood inside Formatter and to deprecate the function in the Float module. In this way, Formatter becomes the single point of access for formatting variables to strings, while Float can concentrate on converting text to floating-point values.

Current tango inherited bigint implementation is absolutely useless for our goals as it allocates on mutations and doesn't have fixed binary layout. Fortunately no Sociomantic projects use it so it can be simply deprecated.

I will start working on simple replacement module ASAP.

The function toDString in ocean.text.util.String should be more general by accepting Const!(char)* instead of char*. Consider the case when you have a C string

Const!(char)* c_string = "test";
auto d_string = StringC.toDString(c_string); // does not compile with D2

Currently you cannot call toDString because c_string is const. You can easily end up with a const C string when calling a C library. As a work around you can copy c_string to a char*. To me the proper fix is allow passing Const!(char)* since toDString does not modify the memory pointed to by its argument. Also it is nice to indicate (via its signature) to the caller that it does not perform such modifications. Since the change accepts more input existing code is unaffected by the change.

Same reasoning holds for toDString accepting currently a Wchar*.

As found out by http://forum.dlang.org/post/[email protected], the https://github.com/sociomantic-tsunami/ocean/blob/v2.x.x/src/ocean/math/Probability.d module original code has a different license than the stated by the D module.

This module comes from Tango: http://www.dsource.org/projects/tango/browser/trunk/tango/math/Probability.d and we kept the original copyright notice, but it might be better to check with the original author if this license change is OK.

If it's not, then this module will probably have to go.

If you had an app that used a specific aquatic library that contained versioned structs (let's say it rhymes with brill), it would be really useful if the app emitted version information to the version log. For example:

2017-08-01 15:52:14,776 Info [ocean.util.app.ext.VersionArgsExt] - 
Moral version v2.6.3-3-g42bb468-dirty (compiled by 'NaNdrej Meetrowitch' on 
2017-08-01 15:51:58 UTC with DMD64 D Compiler v1.080.0 using 
crocodile:v5.1.0-6-g3d7e36d hashproto:v12.0.0 lunge:v4.2.0 
queueproto:v12.0.0 drill:v3.1.3 makd:v1.9.0 ocean:v3.1.3 
crowd:v4.0.1 tortoise:v7.0.1). Writing FieldProfile v3 and AdvertiserFieldProfile v2.

from man 7 ip:

       IP_FREEBIND (since Linux 2.4)
              If enabled, this boolean option allows binding to an IP
              address that is nonlocal or does not (yet) exist.  This
              permits listening on a socket, without requiring the
              underlying network interface or the specified dynamic IP
              address to be up at the time that the application is trying to
              bind to it.  This option is the per-socket equivalent of the
              ip_nonlocal_bind /proc interface described below.

e.g.:

union MyUnion
{
    int i;
    float f;
}

import ocean.core.Array : removeShift;

void func ( )
{
    MyUnion[] arr;
    removeShift(arr, 0);
}

produces this error:

./submodules/ocean/src/ocean/text/convert/Formatter.d(489): Error: static assert  "Type unsupported by ocean.text.convert.Formatter: MyUnion"
./submodules/ocean/src/ocean/text/convert/Formatter.d(475):        instantiated from here: handle!(MyUnion)
./submodules/ocean/src/ocean/text/convert/Formatter.d(203):        instantiated from here: handle!(MyUnion[])
./submodules/ocean/src/ocean/text/convert/Formatter.d(69):        instantiated from here: sformat!(MyUnion[],char[],MyUnion[])
./submodules/ocean/src/ocean/core/Enforce.d(427):        instantiated from here: format!(MyUnion[],char[],MyUnion[])
./submodules/ocean/src/ocean/core/Enforce.d(334):        instantiated from here: enforceImpl!("==",MyUnion[],MyUnion[])
./submodules/ocean/src/ocean/core/Test.d(80):        instantiated from here: enforceImpl!("==",TestException,MyUnion[],MyUnion[])
./submodules/ocean/src/ocean/core/Buffer.d(123):        instantiated from here: test!("==",MyUnion[],MyUnion[])
./submodules/ocean/src/ocean/core/array/Mutation.d(1174):        instantiated from here: Buffer!(MyUnion)
./src/myrepo/MyModule.d(1138):        instantiated from here: removeShift!(MyUnion)
make: *** [/home/devel/myrepo/build/devel/tmp/allunittests] Error 1

It's not immediately obvious to the user why calling removeShift causes unittests to fail.

Deserializer does not correctly deserialise const(T[])[] struct fields. The problem is a qualifier-incompatible way of checking for a dynamic array in sliceSubArrays.
For the reference type qualifier inexperienced, this is how it fails:

alias const(char[])[] T;
// Check if T is an array, and if so, derive the element type:
static if (is (T V : V[]))
{
    // V is const(char)
    // Now check if T is a dynamic array:
    static if (is(V[] == T)) // compares const(char)[] with const(char[]) -- not the same
    { /* ... */ }
    else
    { /* False negative, T is a dynamic array. */ }
}

So far so good, this can be fixed. But then another problem occurs: sliceSubArrays cannot call sliceArray because the out parameter of sliceArray would need to be the array element type const(char[]) -- which is illegal, an out parameter cannot be const.
This conflict is a symptom of a more fundamental problem with deserialising const. Given a struct

struct S
{
    const(T[]) a;
}

What the deserialiser essentially does is the following:

void[] input_data;
void[] slices_buf;
S dst;
dst.a = cast(const(T[]))(slices_buf[
    0 .. numberOfElements(input_data) * const(T[]).sizeof
]);
foreach (ref element; dst.a)
{
    element = deserialiseNextElement(data);
}

The problem here is that the assignment in the foreach loop is illegal because typeof(element) is const(T[]) -- one cannot assign to const.
So what can we do here? The only options that come to my mind is to either ban such a const or cast it away when populating the array, considering slices_buf to be uninitialised memory.

Any ideas, anyone?

Currently this test fails with contiguity contract violated:

unittest
{
    static struct Inner { mstring s; }
    static struct Outer { Contiguous!(Inner) inner; }

    auto s1 = Inner("abcd".dup);
    Outer s2;
    copy(s1, s2.inner);

    Contiguous!(Outer) s3; // nested contiguous struct
    copy(s2, s3);
    s3.ptr.inner.enforceIntegrity(); // fails
}

./src/ocean/util/serialize/contiguous/Contiguous.d:203: unexpected exception object.Exception: expression '0X00007F76C4F08FB8 >= 0X00007F76C4F214D8' evaluates to false

Problem happens because Contiguous struct is simply a wrapper on top of void[] from the PoV of the type system. As such, it will be serialized same as plain void[] array, as a raw sequence of bytes - meaning any internal pointers it contains won't be updated to refer to new buffer.

The only possible fix is to add special support of Contiguous structs to serializer itself, processing them as struct type instead.

https://issues.dlang.org/show_bug.cgi?id=16521

The issue itself is worked around in ocean, but with 2.074 it generates X deprecation messages for each instantiation of sformat - where X is the number of arguments. That makes deprecations completely unusable for us starting from 2.074.

Hopefully the last one!

Right now format("{}", someTimestamp) prints { ticks_: 636247008000000000 } instead of 03/10/17 00:00:00

Good old Tango supports it via this horror.

With the Formatter, we can just provide a toString method. Feel free to assign me.

In the section D2 Compatibility, the second paragraph ends with the sentence:

The patches needed to compile the dmd-transitional compiler are located in
docker/dmd-transitional/patches.

However, the link pointed to by docker/dmd-transitional/patches is broken.

(not making a PR as I don't know what the correct link is)

For files which are registered with ReopenableFilesExt, we need to be able to reopen a specific file via an UnixSocket command.
This should speed up the process for applications with many opened files.

For example it would be used as echo "App reopen /srv/httpd/log/root.log /srv/httpd/log/important.log" | tee /srv/httpd/httpd.socket.
@andrew-stevenson-sociomantic would that cover your needs ?

It's old code which needlessly allocates.
Like a lot. No really, I mean it.

It is however quite popular so the phasing out should be gradual.

For now Ocean for D2 is stuck at 2.070.x, but even then there are custom modifications that are needed to be able to build / use it (this compiler is built as part of Ocean automated testing by applying patches to the upstream compiler.

These are the issues:

• GC.stats (dlang/druntime#1610), available starting with 2.072
• GCC 5.1 fixup (dlang/dmd#5686), available starting with 2.072
• Extra C/POSIX bindings in druntime, available starting with 2.073
• Add a way to get a Throwable message that can be overridden by the user (reused) without re-allocating (Ocean reuse buffers for Exception to avoid allocating at all cost).
• #303 Remove pokemon catch
• #306 scope with different callback type as mentioned here.
• DMD bug

After all the issues are solved upstream, we still need to upgrade to an upstream DMD that have all the fixes, which might take a while too.

In case explicit padding, base64 decoder will throw an exception when there are more than two padding bytes. It should do the same when implicit padding is in use. Currently, old behavior is that it silently discards the last byte.

See #193

Summary

Create a logo for Sociomantic main core library we just open-sourced called Ocean.

Brief Description

This is base library for D focused on high-performance applications. It's used to create applications that are fast and use networks mainly.

Why Ocean is cool / important?

Ocean allows developers to create fast and efficient applications. Is the core library powering the whole Sociomantic high-performance backend.

Ideas for the Logo

Obviously something oceany would be a must, but it would be nice to somehow convey also the feeling of speed, of something fast (but also vast, because the library includes a lot of different kind of utilities). Also something related to building blocks could be a possibility.

Sociomantic colours seem to be a good fit for something watery, but on the other hand there is also D involved, so red could be an option too.

Here is a similar library as an example: http://www.boost.org/

Final Output

To be used on GitHub repo, stickers, etc. potentially even external website (if one is made).

Extra Goodies

Colour scheme and suggested font for further work on a potential website.

In case logging system doesn't have enough permission to write to the log file, it can happen that
the application fails on permission denied exception, before trying to look for the other application instance.

While this effectively prevents the application to run, it would be nice if the PidLock extension would check if the another instance is already running, before any other systems are started.