This issue was copied from microsoft/checkedc-clang#4

This issue was copied from microsoft/checkedc-clang#24

This issue was copied from microsoft/checkedc-clang#25

This issue was copied from microsoft/checkedc-clang#46

It turns out that the unexpected test failures we were seeing were caused by LLVM/clang having some tests that depend on Unix line endings. On Windows, Git can alter the line endings to match the Windows line ending convention, breaking those tests. The fix recommended by the LLVM/clang project documentation is to set autocrlf to false, matching the behavior of subversion.

This change:

• Updates the enlistment instructions so that line endings for
  source files and test files are configured to be Unix line endings
  on Windows.
• Removes the test baseline instructions. We don't need baselines,
  as there should be no unexpected failures.
• Adds some information about line endings to the test section
  because they can cause unexpected test failures.
• Updates the working on Windows section to caution developers to
  terminate files with line feeds on Windows.

This issue was copied from microsoft/checkedc-clang#48

The logic in NewTyp::mkTypForConstrainedType is ugly and probably wrong in some cases. It should be replaced with either a ginormous switch statement (which is more intuitive from an ML-persons point of view, but not LLVM idiomatic) or a RecursiveASTVisitor (which is ugly from an ML-persons point of view because functionality is smeared across a bunch of tiny functions, but is LLVM idiomatic).

This issue was copied from microsoft/checkedc-clang#17

This change addresses issue #7 (process bounds expressions for parameters
with all the parameters available). Currently, bounds expressions for
parameters are being parsed and type checked in the scope of the parameters
seen so far in a parameter list. They need to be typechecked in a scope with
all the parameters for the parameter list.

To do this, parsing and type checking of bounds expressions for parameters
is deferred until the entire parameter list has been seen. This is done
in two steps. During the initial parse, the tokens for a superset of bounds
expressions are parsed and stored in a list. They are then re-parsed using
ParseBoundsExpression after all parameters have been seen.

This change also improves error messages when a bounds expression is parsed
with misspelled contextual keyword. For example, int x : boounds(x, x + 5).
Before, the code would simply give up after parsing the misspelled keyword
and the '('. This led to additional error messages when the code tried to
match the closing ')'. This fix is to skip all the tokens up to but not
including the closing ')'.

Testing:

• Passes existing clang test suite.
• Added additional tests for parsing of function parameter lists with bounds
  expressions. This will be committed separately to the Checked C repo.

This issue was copied from microsoft/checkedc-clang#23

This change adds parsing of member bounds declarations (issue #19). Member bounds
declarations use member bounds expressions, which can only use other
members of the structure. The changes include:

• IR: no change is needed to represent member bounds expressions. Because
  clang handles C++, it already can represent members directly in
  the IR for expressions. Bounds expressions are a subclass of expressions.

  We need to attach bounds expressions to member declarations. We already
  have bounds expressions on function declarations and variable declarations,
  which have the same base class as member declarations. Instead of having
  a 3rd copy of the code, we factor the common code for representing bounds
  expressions and move it to the common base.

• Parsing: we modify the parsing of member declarators that are followed
  by ':' to parse either a bounds expression or a constant-expression
  that is a bit-field size. We check whether the first token after the
  ':' is an identifier that is contextual keyword for bounds expressions.

  The parsing of bounds expressions is deferred until all members of the
  structure have been seen. This is so that member bounds expressions may
  refer to any member of the structure. Constant expressions for bitfield
  sizes continue to be eagerly parsed.

  The decision to just look for a bounds expression keyword can lead
  to several error messages from clang if there is a spelling
  error in a contextual keyword. Clang interprets the misspelled keyword
  as an implicit function declaration and issues unknown identifier errors
  for uses of members.

  For now, we'll live this because this specific case is a fairly
  low priority to address. The type for the identifier in the declarator
  hasn't been built yet at the time that we make the parsing decision.
  We could move that up or defer parsing of constant expressions. Either
  change is largely orthogonal to this change.

• Semantic processing: member bounds expressions are restricted to accessing
  only members. We add a state flag to Sema that changes the lookup of C
  identifiers during expression parsing to look up identifiers in the
  member namespace instead of the ordinary namespace. The state flag is
  protected by an RAII class.

Testing:

• Wrote new feature tests of parsing member bounds
  declaration. These will be checked in separately to the Checked C
  repo in parsing\member_bounds.c.
• Passes existing clang test baseline.

This issue was copied from microsoft/checkedc-clang#34

Initial version of the checked-c conversion tool. Current features:

• Processes single C files or multiple C files (up to whole programs)
• Infers "ptr-ness" for local variables used in those programs.
• Re-write many source files
• Supports many aspects of the C grammar
• Contains unit tests

Current weaknesses:

• Macro support is limited
• Casting support is limited, needs structural type equality checks
• No inherent support for standard library functions.
• Doesn't honor existing checked C types

This issue was copied from microsoft/checkedc-clang#49

Let's say you have a typedef that looks like this:

typedef struct _A { 
  int a;
  int b;
} A, *PA;

There are two variable declarations somewhere in the program:

PA p1 = foo();

and

PA p2 = bar(); 

In one scenario, both foo and bar are unconstrained and can be PTR. The re-writer could make one edit, in the typedef, to change the definition of PA to be ptr<struct _A> PA. Then this will just work.

However, let's say that the return value of foo is PTR but bar is constrained to not PTR. So one of these will be re-written to ptr and another won't. Can we do better? One option is to create a new typedef, but there are questions to be answered. What is this typedef called? How do we disambiguate it from the old one? And so on.

Come up with a spectrum of answers to this question and implement them.

This issue was copied from microsoft/checkedc-clang#37

The Checked C specification has relative bounds expressions. We need to extend the IR to represent relative bounds expressions, add parsing support for that, and add typechecking for that.

This issue was copied from microsoft/checkedc-clang#38

Bounds expressions and where clauses can only use non-modifying expressions. The checking for this needs to be implemented.

This issue was copied from microsoft/checkedc-clang#30

For interoperation, we will redeclare existing library functions with additional bounds information. We assume that modifying library source code is not possible when modifying programs that use libraries. We'll have to add bounds information separately. C allows functions and extern variable to be redeclared, so long as the redeclared versions are compatible with the existing version.

We need to extend the code for checking redeclaration to take bounds into account. For parameters or variables of array_ptr type, bounds will have to match. For now, we will use syntactic equality. If they have unchecked pointer type, if they both of have bounds, they must match. Otherwise, the bounds will be regard as being part of a more "complete" declaration of the parameter or variable.

This issue was copied from microsoft/checkedc-clang#39

Section 3.1.6 describes storage class-related restrictions on variables used in bounds expressions for bounds declarations. This needs to be implemented.

This issue was copied from microsoft/checkedc-clang#43

#NAME?

This issue was copied from microsoft/checkedc-clang#44

Many changes to deal with running on "real" code:

• In certain situations (macros) two AST nodes can occupy precisely the same source location. This is a problem for the re-writer that maintains a strong mapping between source locations and constraint variables, especially for whole-program analysis. The rewriter now does something conservative where two constraint variables that occupy the same source location implicitly alias.
• In other situations, it's difficult to know what to do when re-writing structure typedefs or other kinds of typedefs. We now do something conservative where structure typedefs are left alone.
• There is now a bunch of defensive programming to keep the rewriting logic from trying to do rash things like rewrite a declaration that starts in one file and ends in another file.

This issue was copied from microsoft/checkedc-clang#31

If a variable has unchecked pointer type and aptr interop annotations, this allows implicit conversions from checked pointer types to the unchecked pointer type. This issue is to implement the additional implicit conversions

This issue was copied from microsoft/checkedc-clang#28

This work item is to check bounds declarations to pointers to constant-sized data, where the bounds have the form bounds(_x_ + _const1_, _y_ + const2).   We will check bounds declarations for a subset of expressions that are useful for creating ptr-typed values.  These include:

• address-of operators
• uses of unchecked arrays with known dimensions
• uses of unchecked arrays with known dimensions that are not parameters
• function calls.  For function calls, we will need to substitute constant argument  expressions for parameter variables occurring in the return count.  We will then need to determine whether the resulting expression is a constant-expression.
• casts

It also includes checking bounds at

• Simple assignments
• Function calls

This issue was copied from microsoft/checkedc-clang#16

This change only allows the -fcheckedc-extension flag to be used for C programs
in clang. The clang driver will reject the use of -fcheckedc-extension for
other C family languages supported by clang, including C++, Objective
C, OpenCL, and CUDA. This addresses issue #9 in the checked-clang Github repo.

We are currently not modifying clang to support these other languages, which
is why need to disallow using the extension with them.

Testing:

• Add 4 new tests to clang. They test that use of the extension flag is
  rejected for C++, Objective C, CUDA, and OpenCL.
• As recommended by the clang documentation, I placed the tests in with
  other similar tests in the test tree.
• I updated the testing baselines to reflect the new tests. I also updated
  the documentation to reflect the fact that we have Checked C specific tests
  in clang. We need to take the new tests into account when updating to new
  versions of the clang/LLVM sources.

This issue was copied from microsoft/checkedc-clang#3

We added code for serializing and deserializing clang IR with the new checked types. This is necessary to get the code to compile. However, the serialization and deserialization of IR with these types has no tests. We should add tests before people start using compiler functionality that depends on this.

This issue was copied from microsoft/checkedc-clang#1

First of all, thanks a lot for putting this out in the public-domain.

I have a few questions, not sure if this is the best place to ask, but here I go:

A lot more users will directly benefit form this if this is provided as a package through distro-repositories for major OS distributions. So my questions are around how the teams sees this going forward:

• Is the team working with upstream to merge changes back to clang/llvm codebases? If so, what release is being targeted/what timeline?
• If merge-back is not on the cards, what is the long-term plan wrt packaging for various OS distro-flavors? Is the aim to make it available as separate packages on major distros (eg. Debian)?
• If the plan is to have a separate distribution in the long-term, what does this mean for work that happens on Clang/LLVM upstream? Will patches from Clang/LLVM be merged back from time to time? How will the releases be planned?

This issue was copied from microsoft/checkedc-clang#18

This change adds parsing of variable declarations with bounds declarations (issue #13).
For each declarator in a declaration, the declarator is parsed and then the
optional Checked C bounds declaration is parsed. The bounds declaration is
parsed before the optional initializing expression for the declarator. Because
the declarator has already been parsed and added to the current scope, the
bounds expression can be eagerly parsed.

One surprise with clang was that placing declarators for a declaration
on multiple lines caused a parsing error in the initial implementation,
while having all the declarators on one line did not. I traced this back to
special case code that looks for typographical mistakes
at line endings by calling MightBeDeclarator and generating an
error if MightBeDeclarator is false. MightBeDeclarator returns true
for syntactic items that might start a declarator. It has special
case checks to make sure that an identifier is followed by something
that might also be part of a declarator. For Checked C, an identifier
that starts a declarator may be followed by ':' and a bounds expression,
so allow ':' when the language options include Checked C.

This change also improves error handling during the parsing of bounds
expressions.

• When an error occurs after having parsed an identifier and a left parenthesis,
  always scan for the matching right parenthesis. The scan for the matching
  right parenthesis was only happening in one specific case, leading to
  hard-to-understand spurious parsing errors.
• Make a best effort to continue if an error occurs while parsing a
  bounds expression of the form bounds '(' e1 ',' e2, ')'. clang does not
  differentiate during parsing of expressions between semantic errors and
  parsing failures. It is important to continue parsing so that a semantic
  error does not cause a cascade of parsing errors.

These problems were uncovered during testing of parsing of variable declarations
with bounds expressions. Specifically, using an incorrect bounds expression
in a variable declaration with an initializer caused a spurious parsing
errors.

Testing:

• Created a new feature test for parsing of declarations with bounds
  (parsing_bounds_var_declarations.c). This will be committed separately to the
  checkedc repo.
• Passes existing Checked C tests.
• Passes existing clang base line tests.

This issue was copied from microsoft/checkedc-clang#13

Parse in-line bounds declaratons for variable declarations.

This issue was copied from microsoft/checkedc-clang#20

There needs to be a representation of bounds information in function types. Currently the bounds information is not being included in function types. This is problematic because in clang, typedefs are represent as names that map to type object. Uses of typedef'ed names refer to those function objects. It is also problematic for typechecking pointers to function types.

One approach is to use DeBruijn indices: number all the arguments and then change bounds expressions used in types to refer to those arguments. We would need a way to represent uses of those arguments in bounds expressions. For now, we could require that two bounds expressions be exactly identical after this transformation. Later on, we could canonicalize bounds expressions and require that they have the same canonical representation. Even later on, we might check to see if one bounds expression implies another.

This issue was copied from microsoft/checkedc-clang#22

This fixes a crash in clang that can occur during typo correction in C mode.
I hit this crash while testing out parsing of incorrect bounds declarations on
structure members.

The crashes were fixed in the mainline clang sources for binary operators
with SVN r272587. I've modified the fix to cover conditional operators as well.
The modification is out for code review for the mainline clang sources. I'm
checking in the modified fix ported to the Checked C clang branch now.

This issue was copied from microsoft/checkedc-clang#21

This extends parsing of function declarators to parse bounds expressions for
function return values. A return bounds expression is declared by following
a parameter list with ':' bounds-expression.

The return bounds information is propagated to the representation of function
declarations in clang. It is not propagated yet to function types because
there is no way to represent it in function types. I have opened issue #20
to track that and added a few TODO's to the code flagging where the
information will need to be added.

Testing:

• Add feature tests for parsing return bounds. The tests are in
  the file return_bounds.c, which will be committed separately to the checkedc
  Github repo.
• Passes existing clang regression test baseline.

This issue was copied from microsoft/checkedc-clang#36

This addresses issue #26.

This issue was copied from microsoft/checkedc-clang#47

• Only de-sugar typedefs if the immediate value is a pointer value, and then, only perform one "step" of de-sugaring. This reduces cases where wchar_t gets re-written as unsigned short but there is still a more elegant fix waiting to be implemented.
• Started adding documentation
• Fixed a bug where we would only ever constrain the top-most constraint variable in a Decl. Added test cases.
• Constrain compound assignment of pointers via +=.

This issue was copied from microsoft/checkedc-clang#35

The rewriter should have some documentation, both externally facing and developer documentation. They should be added to the folder docs\checked.

This issue was copied from microsoft/checkedc-clang#2

Fix a few typos in Setup-and-Build.md. Add a missing
hyperlink for how to update to newer versions of the
LLVM/clang sources.

This issue was copied from microsoft/checkedc-clang#27

We decided to stick with the existing rule in the specification that all dimensions of a multi-dimensional array must all be checked or unchecked. There is no need for the ability to declare unchecked arrays using the 'unchecked' keyword, so we should remove this feature.

This issue was copied from microsoft/checkedc-clang#15

This issue was copied from microsoft/checkedc-clang#42

This change implements checking of type requirements for bounds declarations for variables, members, and function return values. It implements the checking described in Chapter 3 and Chapter 5 (interoperation) of the Checked C specification. This addresses Github issue #41.

The requirements in Chapter 3 include:

• Bounds declaration are not allowed for variables, members, and return values with ptr type.
• They are also not allowed for function types and function pointer types.
• Count bounds expressions are allowed for variables or members that have pointer type or array type. They are also allowed for pointer-typed function return values. The pointer type must be a non-void pointer type.
• byte_countand bounds bounds expressions are allowed for variables and members with pointer or array types. They are also allowed for pointer-typed function return values..

The interoperation requirements include:

• Local variables with unchecked pointer or array types cannot have bounds declared for them. Other declarations (global variables, members, and function return values) with unchecked pointer or array types can have bounds declared for them. This is how bounds-safe interfaces are described.
• Integer-typed variables, members, and return values can have bounds declared for them using byte_count or bounds expressions

This change also:

• Fixes the implementation of clang type tests for BoundsExpr and its subclasses. The static classof methods were not implemented. In addition, the kind information for CountBoundsExpr and NullaryBoundsExpr was not properly padded, so it was overwriting the superclass kind information.
• Renames ActOnBoundsExpr to ActOnBoundsDecl, to more accurately describe what the function is doing.

Testing:

• New tests have been added to typechecking\bounds.c. This will be committed separately to the GitHub CheckedC repo. The new tests systematically test the cross product of variable/member/function return bounds declarations, the different kinds of bounds expressions, and the different types of variables.
• Passes existing clang tests.

This issue was copied from microsoft/checkedc-clang#14

Type check bounds declarations and bounds expressions that use bounds, following the rules outlined in the Checked C design. These can be used in parameter declarations, variables declarations, and for function return, following the rules outline.

This issue was copied from microsoft/checkedc-clang#11

This issue was copied from microsoft/checkedc-clang#40

Section 3.2 describes restrictions on variables at external scope that have bounds declarations. These restrictions need to be implemented.

This issue was copied from microsoft/checkedc-clang#10

We need to test printing, AST serialization, AST transformation, and AST serialization with the new Checked C features. This work item is to understand the tests that exist for these features and how to extend them to Checked C features.

This issue was copied from microsoft/checkedc-clang#29

This issue was copied from microsoft/checkedc-clang#41

An expression with a count bounds expressions must have a pointer type. The type cannot be a pointer to void.

This issue was copied from microsoft/checkedc-clang#7

Bounds expressions need to processed during clang's semantic checking/processing phase in a scope with all the parameters available. I am about to submit a pull request where they are being processed in a scope that contains only the parameters seen so far.

This is a little complicated to implement in clang's semantic checking/processing phase. You have to delay parsing of the bounds expressions because parsing and semantic processing are intermixed.

This issue was copied from microsoft/checkedc-clang#8

This change extends the clang IR to represent Checked C bounds expressions and optional bounds expressions for variable declarations. It also adds support for parsing bounds expressions and modifies parsing of function parameter lists to parse optional bounds expressions.

Bounds expressions are represented in the IR by adding a new abstract class BoundsExpr and subclassing it for count bounds expressions (count(e1) and byte_count(e1)), range bounds expressions (bounds(e1, e2)), and nullary bounds expressions(bounds(none)). AST printing, serialization, traversal, and tree transformations are extended handle the new expressions.

Bounds expressions are attached to variable declarations by adding an additional member to VarDecls. Many VarDecls will not have bounds expressions, so this adds extra space overhead to the representation of VarDecls. We can revisit this later if it becomes an issue.

To test the new bounds expressions, we add parsing of bounds expressions for function parameter lists and attach the parsed bounds expressions to the VarDecls for the parameters.

Bounds expressions for parameters need to be processed in a scope with all the parameters available. They are currently being processed in a scope that contains the parameters seen so far. This is a little complicated to implement in clang. You have to delay parsing of the bounds expressions. I will come back to this after getting basic parsing of bounds expressions working. I've opened issue #7 to track this.

Testing:

• This passes the current test baseline for this snapshot of clang:
• Wrote new feature tests of parsing of parameters with bounds declarations. There will be a separate pull request to the Github CheckedC repo for these tests.
• Passes the existing Checked C tests.

  Expected Passes    : 8942
  Expected Failures  : 21
  Unsupported Tests  : 206
  Unexpected Failures: 3

• We still need to test AST printing, serialization, traversal, and tree
  transformations. I've opened issues #4 , #3, #5, and #6 to track this.

This issue was copied from microsoft/checkedc-clang#5

This issue was copied from microsoft/checkedc-clang#19

Parse in-line bounds declarations for struct members.

This issue was copied from microsoft/checkedc-clang#9

We should only allow the Checked C extension flag to be used for C in clang. We should issue an error message for other languages. We're not currently doing the implementation work to enable all the language features for other C family languages supported by clang (such as C++). We're also not testing the other C family languages.

This issue was copied from microsoft/checkedc-clang#12

Parse inline bounds declarations so that we can attach bounds information to function parameters.

This issue was copied from microsoft/checkedc-clang#45

This change adds type checking for bounds expressions of the form (bounds(e1, e2). The type checking rules are:

• The types of e1 and e2 must both be pointer types.
• The types e1 and e2 must both be pointers to objects or incomplete types, and
• The types of e1 and e2 must both be pointers to the same type or one of them must be a pointer to void.

Testing:

• New tests will be added to typechecking\bounds.c in the Checked C GitHub repo. These will be committed separately.
• Passes existing clang tests.

This issue was copied from microsoft/checkedc-clang#50

It would be good to have one kind field on bounds expressions instead of specialized kinds per type of bounds expressions. This would make it easier to write code that traverses bounds expressions. Currently, we have to test the type of a bounds expression and then look at the kind field.

This issue was copied from microsoft/checkedc-clang#6

This issue was copied from microsoft/checkedc-clang#32

The implementation of Checked C in clang allows dimensions of multi-dimensional arrays to have different checked properties. A dimension could be declared as unchecked by prefixing it with the unchecked keyword. The Checked C specification says all dimensions of multi-dimensional arrays have to either be all checked or all unchecked. We thought about changing the spec, but have decided to stick with it because allowing dimensions to have different checked properties is confusing.

This changes removes the ability to use the `unchecked' keyword to declare unchecked dimensions for multi-dimensional arrays (addressing GitHub issue #27). In addition, it adds error checking that dimensions in a multi-dimensional array are either all checked or unchecked.

There are some corner cases to handle due to typedefs. In C, an array is an array of arrays. A typedef can define an array type that is used as a nested array type. The case where the enclosing array and the nested array differ in their checked property because of a typedef is now an error. We take some care to generate an error message that explains what is going on.

In addition, we also have to handle parenthesized types. The checked property for a multi-dimensional array propagates from the outer array type to inner array types declared as part of the a declaration (int a checked[10][10] declares a checked array of checked arrays). Testing for this change uncovered that the checked property was not being propagating properly to nested types when a declarator was parenthesized (int (a checked[10])[10]). This change fixes that.

Testing:

• Wrote new feature tests for testing that array dimensions have consistent checked properties. This changes will be checked in separately to the Checked C repo in typechecking\checked_arrays.c.
• Wrote new feature tests for testing parenthesized array declarators for checked arrays.
• Code still passes the existing clang baseline tests.

This issue was copied from microsoft/checkedc-clang#26

This should be fairly simple: ensure that the expression has an integral type.

This issue was copied from microsoft/checkedc-clang#33

• Processes single C files or multiple C files (up to whole programs)
• Infers "ptr-ness" for local variables used in those programs.
• Re-write many source files
• Supports many aspects of the C grammar
• Contains unit tests

Current weaknesses:

• Macro support is limited
• Casting support is limited, needs structural type equality checks
• No inherent support for standard library functions.
• Doesn't honor existing checked C types