npalix / rathaxes Goto Github PK

Today you cannot control where the ::decl chunks are generated in the C code 
output.

They are just generated first and that means they are above the #includes, I 
have to move them manually each time I re-generate the code in order to compile 
it since the Rathaxes types I define use types defined by the system (and 
included via #includes).

I should be able to do something like:

provided pointcut ::decl();

In an interface and then define it in a blt. In other words ::decl should just 
be a regular pointcut except that you can use it to declare Rathaxes variables.

Currently the comments are ignored, and never kept to output in the generated 
code.
We need it to generated readable/commentable driver code.

In some case, we need to be able to write a whole block of sequence calls with 
a specific behavior. For instance :

ATOMIC
{
   SET(register, register.field.value);
   SET(register, register.field.value);
   SET(register, register.field.value);
}

would mean that the three SETs are done in ONE final operation. We could 
imagine an ASYNC attribute key, too.

We have to think about this feature, about how we can implement it in order for 
the language to be modulated with new attribute keys.

The middle-end documentation is not complete as of now, and some parts may 
still be unclear.

For instance, the chunk of code presented in section 3.3 is not clear, and the 
chunks and pointcuts should have non-confusing anmes like foo and bar.

Most builtins aren't coded yet, but should be as soon as the builtins 
management has been fixed.

This will display the output of the tests that fail, making failing tests 
easier to debug!

Currently, the command-line arguments of the rathaxes script are parsed kind of 
sloppily.

We shoul import the args module from the cnorm library.

Currently, the type-checking tries to work his way up the interface's 
inheritance tree to resolve the scope of an item.

It then selects the first one to match, instead of going on to check whether it 
could be an ambiguous declaration.

Since we chose to be strict on these aspects, we should not resolve but output 
an error telling it's ambiguous, with the prospects for the resolution, to help 
fix the issue.

In the .blt files we can write templates sequences that are not meant to be 
called from the frontend. This kind of sequences are used only on the backend 
and just inject themselves through pointcuts in other parts of the backends.

An example of this kind of sequences are PCI::probe() and PCI::remove() in the 
e1000 sample. These two functions just inject themselves in the "parent" 
pointcut LKM::code which simply corresponds to the place —in the generated .c 
sources— where functions are defined.

When you write some erroneous Rathaxes code -let's say that you mistype a 
template's implementation in a .rtx file-, an error or warning is outputted, 
but nothing tells you where exactly this error comes from.

We should parse and build the tree with the idea in mind that we want to be 
able to find at least the line involved in this error.

When writing a driver, we may use a statement to describe a behavior :

ATOMIC
{
    SET(register, register.field.value);
}

Here, the "ATOMIC" Would mean that every SET op is to be done into a temporary 
register, which would be set at once at the end of the block.

We must think of a way to write such statements and allow them to be written as 
templates to help extend easily the language.

You can't declare an array with an "automatic" size. In, 
rathaxes/samples/e1000/pci.blt:

static struct pci_device_id rtx_pci_device_table[] = {
    { ${config.vendor_id}, ${config.product_id}, PCI_ANY_ID, PCI_ANY_ID },
    { 0, }
};

In that case, blt-register will fail with:

havePH?:pci_device_id
function 'rtxTpl_compile(local_node : node, compile : node, node_idx : value)' 
hasn't been implemented
rathaxes.cws(127): main scope
rathaxes.cws(234): main scope
rtxCompile.inc.cws(61): rtxTpl_walk
rtxCompile.inc.cws(54): rtxTpl_walk
rtxCompile.inc.cws(24): rtxTpl_walk
rtxCompile.inc.cws(42): rtxTpl_walk
rtxCompile.inc.cws(111): rtxTpl_compile
rtxCompile.inc.cws(140): rtxTpl_compile
rtxTpl_compile

However it does work correctly if you specify the size:

static struct pci_device_id rtx_pci_device_table[2] = {
    { ${config.vendor_id}, ${config.product_id}, PCI_ANY_ID, PCI_ANY_ID },
    { 0, }
};

Given a type declared in an interface an implemented in a template, for example 
PCI::Device.

From a chunk in a template, if you try to declare a pointer to a variable of 
this type, like this:

{{{
${PCI::Device} *dev;
}}}

Rathaxes fails to generate the code with the following error:

{{{
rtxLink:<LOG> LOADING Script 'F01578E8B4437E08E68C6D8EC69C1858.cws'
rtxLink:<LOG> Resolution script and tree loaded.
Generated Script : inject for key _1 : (: ${PCI::Device})
inject<"__rtx_var__">:<LOG> Injecting code into placeHolder ${PCI::Device}
Generated Script : inject for key _2 : (: ${PCI::Device})
inject<"__rtx_var__">:<LOG> Injecting code into placeHolder ${PCI::Device}
inject<"__rtx_var__">:<LOG> Could not find any type manipulation for injection.
Element ->Device<- from "c_tree.body.compile[\"_2\"].node.body.identifiers" =  {
        [
                "0": {
                        .type = "__rtx_id__",
                        .scope = "PCI",
                        .name = "Device",
                        .global = ""
                }
        ]
} could not be found in the ResolverValues : "resVals" = ""
rathaxes.cws(127): main scope
rathaxes.cws(170): main scope
rtxGenerate.inc.cws(145): rtxFrontGen_iterate
rtxGenerate.inc.cws(135): rtxFrontGen_iterate
rtxGenerate.inc.cws(123): rtxFrontGen_iterate
rtxGenerate.inc.cws(99): rtxFrontGen_Device
rathaxes.cws(90): instanceChunk
rtxResolve.inc.cws(420): inject
rtxResolve.inc.cws(883): inject
rtxResolve.inc.cws(795): inject_pointcut
rathaxes.cws(47): instanceChunk
rtxResolve.inc.cws(420): inject
rtxResolve.inc.cws(573): inject
rtxNodeBack.inc.cws(230): rtxNodeResVal_GetField
}}}

It would be cool if types could provide pointcuts like sequences can do.

It would allow building of more complex types through aggregation; and maybe 
other stuff I didn't think of.

I don't know if this will be a good idea in the long term, but I think it's 
worth to test.

The types of those parameters should be checked.

Currently, most of the resolver code is based on the pre-conceived idea that 
the placeHolder is contained in a block.

Thus, resolving the palceHolder means deleting the previous and inserting 
(maybe) multiple rows instead of the deleted one.

We must change the way to view this mecanism. Maybe the placeHolder hasn't the 
block for parent (syntax and tree-wise).

The first idea for a solution is to use a templatised function for inserting 
code, instead of using a fixed function for the blocks. This way we'll be more 
supple in the way we resolve code and insert resolved code into the code to 
resolve.

meaning:

parentNode
insertModifier(parentNode, ...);
rtxResolve_insertCode(parentNode, ...);

would become:
rtxResolve_InsertCode<parentNode.type>(parentNode, ...);

Currently, the MiddleEnd type checking functions mostly throw errors when an 
error is detected.

This prevents the compiler from logging as many errors as possible before 
stopping the compilation, and thus impairs maintainability and consistency of 
the compiler

Calling a sequence provided by Rathaxes from a chunk of instrumented C code 
fails with the following error:

Generated Script : inject for key _2 : (: ${PCI::Device})
inject<"__rtx_var__">:<LOG> Injecting code into placeHolder ${PCI::Device}
inject<"__rtx_var__">:<LOG> Injecting a type name.
"c_tree.body.block[\"0\"].body.block[\"2\"]" =  {
        .ctype =  {
                .pointer =  {
                        .type = "__decl__",
                        .ctype =  {
                                .type = "__ctype__",
                                .cdecl_type = "__primary__",
                                .qualifier = "auto",
                                .specifier = "auto",
                                .funspecifier = "auto",
                                .identifier = "_2"
                        }
                },
                .type = "__ctype__",
                .cdecl_type = "__pointer__",
                .qualifier = "auto",
                .storage = "auto",
                .identifier = ""
        },
        .type = "__decl__",
        .name = "dev",
        .assign_expr =  {
                .type = "__expr__",
                .operator = "__id__",
                .expr_type = "__terminal__",
                .id = "NULL"
        }
}
rtxLink:<LOG> LOADING Script 'DA1912110026BDF93BE6D2E21EDA0139.cws'
rtxLink:<LOG> Resolution script and tree loaded.
Generated Script : inject for key _3 : (__rtx_pointcut__: ${pointcut 
::IMPLEMENTATION})
rtxLink:<LOG> LOADING Script '35B959FA6A8955857AB12CF7B0D4CD2F.cws'
rtxLink:<LOG> Resolution script and tree loaded.
Generated Script : inject for key _1 : (__rtx_pointcut__: ${pointcut 
::IMPLEMENTATION})
[Error] Could not resolve pointcut '${pointcut LKM::code}' from template 'With'.
[Error] Could not resolve global code.
[Error] Driver could not be generated.
make[3]: *** [rathaxes/samples/lkm/lkm_Linux.c] Error 6
make[2]: *** [rathaxes/samples/lkm/CMakeFiles/lkm.dir/all] Error 2
make[1]: *** [rathaxes/samples/lkm/CMakeFiles/lkm.dir/rule] Error 2
make: *** [rathaxes/samples/lkm/CMakeFiles/lkm.dir/rule] Error 2

You can reproduce this error by applying this revision of my patch queue:

https://www.kalessin.fr/projs/hg/epitech/mq/rathaxes/rev/efee5f0249e2

Over the changeset: dd92091de0ed.

I tried to reproduce this error in the helloworld sample (by calling Log::info 
from a chunk too) and it failed with an entirely different and scarier error: 
that starts with:

rtxLink:<LOG> LOADING Script '50F7F43ADD558B585487EA74FB511987.cws'
rtxLink:<ERROR> Could not load resolution script : impossible to find file 
"/home/kalessin/projs/rathaxes/rathaxes//backend_library/50F7F43ADD558B585487EA7
4FB511987.cws" into include paths

Some type-checking pass should do it (And we may as well remove the associated 
resolution code too).

This will allow us to use a rathaxes type as follows:

${Rtx::Type} hw;
${Rtx::Type} *hwp = &hw;

This is possible thanks to the unstrict cnorm.

do { } while is not implemented in rtxIntrospect and a do { } while fails with 
the following error message:

IntrospectingInstrumentedCcode...
rtxIntrospect_stmt<__do__>isnotimplemented.
rathaxes.cws(127):mainscope
rathaxes.cws(255):mainscope
rtxIntrospect.inc.cws(648):rtxIntrospect
rtxIntrospect.inc.cws(611):rtxIntrospect_walk
rtxIntrospect.inc.cws(606):rtxIntrospect_walk
rtxIntrospect.inc.cws(617):rtxIntrospect_walk
rtxIntrospect.inc.cws(623):rtxIntrospect_walk
rtxIntrospect.inc.cws(628):rtxIntrospect_walk
rtxIntrospect.inc.cws(596):rtxIntrospectChunk
rtxIntrospect.inc.cws(275):rtxIntrospect_code
rtxIntrospect.inc.cws(426):rtxIntrospect_stmt
rtxIntrospect.inc.cws(421):rtxIntrospect_stmt
rtxIntrospect.inc.cws(467):rtxIntrospect_cond
rtxIntrospect.inc.cws(163):rtxIntrospect_code
rtxIntrospect.inc.cws(275):rtxIntrospect_code
rtxIntrospect.inc.cws(55):rtxIntrospect_stmt

You can find a do { } while in e1000.blt.

The configure step of cmake fails because the RATHAXES_COPY_LATEX_CLASSES macro 
tries to copy files which are not generated yet.

In the meantime it is possible to run cmake a second time to finally configure 
the project and generate the build directory.

It's been long since we talked about it, but there is a need for this feature !
See devs'mailing-list for more details.

If I define a configuration block in a .rtx with hexadecimal variables, like 
this:

configuration
{
    PCI::vendor_id = 0x8080;
    PCI::product_id = 0x4242;
}

These variables will be recognized as ::strings.

Further investigations showed that the rtxType_Parse functions are never called 
for the variables in the configuration block. Could that be related to the 
un-implemented function rtxTypeCheck_Config in rtxInterfaces.inc.cws?

Currently, in an interface, there's nothing to tell the blt/rtx devs what to 
use/implement as a mapping. This must be controlled and checked.

Currently, the management of the identifiers in each backend template is void, 
and will soon be done through rathaxes functions generating unique identifiers.

In a far future, It would be nice to have a pass on the backend's tree, to 
modify the identifiers used each time there is a confliting name with a parent 
tree, in order to allow using the same identifiers in differents templates 
without risking any conflict at frontend compilation.

To sum it up :
Some magic is required to make every single identifier unique, but that should 
be for much later, as we need first some functional features :)

Currently, some tests still set the cnorm dialect manually before calling the 
parsing.

Now that we have proper functions encapsulating this, we should use them, in 
order to avoid test failures due to a forgotten update of the tests.

We need to think about how to make the definition of template statements 
dynamic.

A statement would be kind of like a sequence, but with additionnal things, ie:
CONCAT(buffer, register)
{
   PRE // Is a pre-condition for the loop inside concat (hidden)
   {
      WAIT(register, register.field.value);
   }
}

Here, this means that there will be ONE concat action after each time the 
pre-condition is met, until the buffer is full.

In the end it is a kind of parameterized sequence.

The issue is that such a statement could be used without any  content in the 
block, and thus behaving the same way the sequences do.

We need to think about how to define such a feature, bearing in mind the fact 
that we plan to do an attribute key system (see Issue #33), and the fact that 
we want to prevent anyone to be able to write the same thing with two distinct 
ways.

It doesn't work when you try to provide a pointcut in a sequence template, e.g:

test.rti:

interface Test
{
    provided sequence   Test::seq()
    {
        provided pointcut   Test::code;
        provided chunk      Test::main;
    }
}

test.blt:

with Test
{
    template sequence   Test::seq()
    {
        chunk   Test::main()
        {
            ${pointcut Test::code};
        }
    }
}

It doesn't work with “required” sequences either.

Currently, some abhorrent "AST" designs are left, and the whole thing is not 
normalized.

In order to ease future development, we absolutely need to normalize it all.

If you try to do a “rathaxes blt-register” on a blt that contains 
instrumented C code with if statements, it fails with:

Introspecting Instrumented C code...
function 'rtxIntrospect_code(local_node : node, theChunk : node, blockIds : 
node)' hasn't been implemented

But it seems implemented in rtxIntrospect.inc.cws.

a type cannot use a type declaration block using other rathaxes types ex:

decl {
   typedef struct ethernet_dev {
       ${PCI::Device} pci_dev;
       ${NET::Device} net_dev;
   } *ether_devp;
}

does not resolves the palceholders, which stay as _1 and _2, which causes 
compilation failure.

A call made outside a __block__ will not the substitute the code of the 
sequence leading to a placeholder (`_X') in the generated code.

For example in e1000.blt:

{
    ${Log::info("e1000::create: mac address loaded from the EEPROM")};
}

Only works with { }.

There is a need to explicit how to convert a rathaxes type to a c-type and 
conversely.

ex: ${call test::plop(local.an_integer)}; <-- local.an_integer is a C int, and 
should be able to be converted as a rathaxes type ::number.

The builtin types improvement is to be enhanced.

For this, we must take into account :
 * The cache must be used the same way it is for non-builtin templates
 * We must have a way to check whether the functions are implemented or not when adding the builtin type (parsing, node formatting, value resolution, mappings)

Thus the tasks are :
 * Update the builtin loader to fill the cache with the builtins
 * Set the cache items 'load' field to true (already loaded)

Currently, the macros (#define and such) are not kept in the generated code.

We need to be able to keep them.

Here are the main parts to refactor asap:
 * Extract builtins from the resolver code
 * Relocate and separate type-checking code depending on type checked
 * possibly rename some scripts for more consistency
 * Many other tasks

In order to make the development of a driver easier, we need to be able to give 
the line of the real source code of a template that failed to resolve, when 
generating a driver.

In order to do this, we must keep the original source of the template.

The testing suite is currently incomplete.
We should improve this by adding some independent tests for things like the 
typing system.

If you do a for loop in a chunk in a blt, like this:

for (i = 0; i != 10; i++) { }

Then the first expression “i = 0” is not generated in the C output.

Currently, a pointcut pulls in every related data, after selection filtered by 
configuration. The issue is double:

 * An optional template will be selected, and try to resolve itself, generating an error when trying to find the associated implem in the front-end..
 * The generated code is ridiculously HUGE, with too many things that are totally useless.

This is a critical flaw for the current compiler.

I think this is much easier to understand with an example:

I have a configuration named `e1000::rx_buffer_len' defined in my .rtx

I my .blt I would like to use it to generate the name of an enum field:

E1000_RCTL_BSIZE_${config.rx_buffer_len}

But this generate only ${config.rx_buffer_len} (i.e: E1000_RCTL_BSIZE_ is not 
output).

Here is my example in the code:

http://code.google.com/p/rathaxes/source/browse/rathaxes/samples/e1000/e1000.blt
?spec=svn675aa6c230476368c195b739218a4f18c2028325&r=675aa6c230476368c195b739218a
4f18c2028325#723

I guess this doesn't only apply to enum values, also maybe I shouldn't do this?

We should add a logging module to manage all the log of the compiler.
For instance, the messages could be stored as a list in 'this', and then dumped 
depending on the log level.

anyways, it should at least allow to format the error message in an unified 
manner.

This would be the counterpart of va_arg in C.

This would be useful for sequences like Log().

In my latest patches I'm trying to:

- Cast a local C variable into a Rathaxes variable;
- Then call a Rathaxes method of this variable (type).

Here is the actual code:

chunk ::dump_infos()
{
    unsigned short ethernet_proto = be16_to_cpu(${self}->protocol);
    ${cast local.ethernet_proto as Ethernet::ProtocolId};

    pr_info(
            "\tprotocol = %#-5x (%s)\n"
            "\t     len = %-5u data_len = %-5u head_len = %-5u\n"
            "\tnr_frags = %u\n"
            "\tgso_size = %-5u gso_segs = %-5u gso_type = %-5u\n",
            ethernet_proto, ${local.ethernet_proto.to_str()}, // XXX
            …
    );
}

The blt-register fails with:

>>out file:Socket::SKBuff()_chunk__dump_infos
rtxGen:<LOG> Starting code generation for __rtx_chunk__
rtxGen:<LOG> Generating cws for placeHolder _1
rtxGen:<LOG> Generating cws for placeHolder _3
function 'rtxRTypeName(theNode : node)' hasn't been implemented
rathaxes.cws(90): main scope
rathaxes.cws(241): main scope
rtxGen.inc.cws(433): rtxGen
rtxGen.inc.cws(440): rtxGen
rtxGen.inc.cws(464): rtxGen
rtxGen.inc.cws(448): rtxGen
rtxGen.inc.cws(334): gencodeworker
rtxGen.inc.cws(122): rtxGenPlaceHolder
rtxRTypeName
make[2]: *** [rathaxes/samples/e1000/e1000_src_Linux.c] Error 255
make[2]: Leaving directory `/home/louis/projs/rathaxes/build'
make[1]: *** [rathaxes/samples/e1000/CMakeFiles/e1000.dir/all] Error 2
make[1]: Leaving directory `/home/louis/projs/rathaxes/build'
make: *** [all] Error 2

I guess it's talking about the node generated earlier:

Built node for template_expr : ${local.ethernet_proto.to_str()}

Currently, we retrieve only one possible configuration instead of creating a 
list of all possible configurations and generate the driver for all of them. 
(in the front-end processing routine)

Many parts of the language may induce confusions.

for instance, a type contains chunks, methods, and attributes, but the 
associated template contains only chunks and a mapping.
Actually, it should contain:
 - chunks (required by interface)
 - a decl (to be inserted when declaring variables of the type, parametrized by the front-end if need be)
 - methods (instead of "chunk") for the methods described by the interface.

Many parts of the language are similar, and not clearly defined. Thus, some 
confusions may arise, which we want to avoid.

For a coherent typing system, we need to give a full description of builtin 
type as attributes,methods and chunks...

interface LKM
{
   provided builtintype configuration
   {
         attribute listString devices;
   }

   provided variable configuration config;
}

I'm trying to declare a C scope directly in a chunk in a blt, like this:

template sequence foo()
{
    chunk ::CALL()
    {
        {
            /* A variable declaration I'd like to scope */
            int i;

            /* Some useful code */
            i = 0;
            i++;
        }
    }
}

But it fails with:

Template sequence block!
CHUNK
An error Occured while parsing :
Trace:
rathaxes.cws(213): main scope
cnorm.cwp(76): rtxParseFile
rtx.cwp(26): translation_unit
rtx.cwp(37): rtx_declaration
rtxBack.cwp(126): rtx_stmt<"with">
rtx.cwp(37): rtx_declaration
rtxBack.cwp(28): rtx_stmt<"template">
rtxBack.cwp(37): rtx_stmt_template_content<"sequence">
rtxBack.cwp(210): rtx_stmt<"chunk">
rtxTemplatedLanguage.cwp(25): rtx_templatedLang_body<"C">
cnormDeclarations.cwp(592): initializer_list
rtx.cwp(11): initializer

Error: line 998, col 19:
bad initializer_list after '{'

Currently, there is no way to describe an extension for an interface depending 
on some config values.

Just do it.

Even with r7adecea6d0fd compiling a single file Rathaxes source (i.e: 
everything in a .rtx file) doesn't seem to work correctly: I'm getting 
different results between a single file source and a multiple-files source.

You can reproduce my problem from the two first patches of my patch queue 
(https://www.kalessin.fr/projs/hg/epitech/mq/rathaxes/file/c908de343118).

The first patch add a regression test for an issue I'll submit next. The test 
sources are split into rti, blt and rtx files; the compilation fails with:

[Error] Placeholder '${pointcut ::IMPLEMENTATION(local.p)}' could not be 
validated.
        at 
[Error] Could not validate sequence template 'Test::entry_point(Test::Test)' in 
interface 'Test'.
        at /home/louis/projs/rathaxes/rathaxes/compiler/tests/pointer_to_rathaxes_type/pointer_to_rathaxes_type.blt:17:14

The second patch rewrites the regression test into a single .rtx file (and also 
moves it into rathaxes/compiler/tests/language, pay attention to that when you 
apply the patch and re-compile). This single file version of the test "fails" 
to compile with:

Device c_tree:With_Test
Generated Script : inject for key _1 : ( __rtx_pointcut__ : ${pointcut 
Test::code()})
[Warning] inject_pointcut<"Test::code">: No chunk associated with the pointcut 
Test::code
[Warning] inject_pointcut<T("Test::code")>: No valid chunk, falling back to 
default behaviour.
Resulting include:
Resulting code: 
Writing to file: test_pointer_to_rathaxes_type_Linux.c
[100%] Built target test_pointer_to_rathaxes_type

I'm expecting the single file version to fail like the multiple file version.