maldil / javafypy Goto Github PK

adapting Java AST mining tools to Python

Roff 0.03% HTML 3.35% CSS 0.19% JavaScript 0.01% Java 96.11% Batchfile 0.01% Shell 0.01% Rascal 0.01% GAP 0.30% Python 0.01%

python java ast python-machine-learning

javafypy's Introduction

JavaFyPy : A mechanism to adapt Java AST analysis tools to Python

Over the years, researchers have developed a solid toolset to analyse Java software systems. JavaFyPy enables adapting these tools to Python.

JavaFyPy creates a Python AST that the Java AST mining tools can process. The above diagram shows the steps of AST transformation performed by JavaFyPy.

This repositoy presents all the above tools

You can follow the following steps to adapt Java AST mining tools to Python.

JPythonParser is a Python parser developed in Java. You can use the class PyASTParser to create the initial AST of the Python code content as below.
```
mod ast = PyASTParser.parsePython(content);
```
Then the class PythonASTUtil:createPyCompilationUnit in SyntaxtTranformer transforms the syntax and augments the AST with type information using the TypeAugmentor. You can use the type information in https://github.com/mlcodepatterns/PythonTypeInformation for type augmentation.
As the next step, migrate the existing Java parser of your tool to our eclipse JDT parser provided in CustomizedEclipseJDT

Using JavaFyPy, we adapted Java RefactoringMiner and CPATMiner to Python. We provide two example usages below.

Py-RefactoringMiner - click-me
R-CPATMiner - click-me

javafypy's People

Contributors

Stargazers

Watchers

Forkers

mlcodepatterns

javafypy's Issues

make super().init() parsable

    class Norm(base_norm_cls):

        def __init__(self, *args, **kwargs):
            ba = bound_init_signature.bind(*args, **kwargs)
            ba.apply_defaults()
            super().__init__(
                **{k: ba.arguments.pop(k) for k in ["vmin", "vmax", "clip"]})

Python nonlocal results

if more_results is not None:
                    for key, value in more_results.items():
                        all_more_results[key].extend(value)

                nonlocal results

Parsing problem with there is in expression with lambdas

Test case

file544() - testPandas

Check
consumeInExpression()
consumeNotInExpression() methods in Parser.java

Implement async functions, async with, await

async def run_clang_format_on_file(filename, semaphore, verbose=False):
    """
    Run clang-format on the provided file.
    """
    # -style=file picks up the closest .clang-format, -i formats the files inplace.
    cmd = "{} -style=file -i {}".format(CLANG_FORMAT_PATH, filename)
    async with semaphore:
        proc = await asyncio.create_subprocess_shell(cmd)
        _ = await proc.wait()
    if verbose:
        print("Formatted {}".format(filename))

except expression with non typed expressions

        try:
            function(*args, **kwargs)
        except (Exception if function_fails else tuple()):
            pass

Make the Floor division

def _synth_regression_dataset(n_samples=100000, n_features=100,
                              dtype=np.float32):
    X, y = make_regression(n_samples=n_samples, n_features=n_features,
                           n_informative=n_features // 10, noise=50,
                           random_state=0)

    X, X_val, y, y_val = train_test_split(X, y, test_size=0.1, random_state=0)
    return X, X_val, y, y_val

InfixExpression.Operator.FLOORDIV

support floor division assign

        while math_ops.reduce_sum(x) > s:
          x //= self.called_member(a)
        return x

Function calls for binary operators

def test_kernel_operator_commutative():
    # Adding kernels and multiplying kernels should be commutative.
    # Check addition
    assert_almost_equal((RBF(2.0) + 1.0)(X),
                        (1.0 + RBF(2.0))(X))

While with else

            while self._rcvd_idx < self._send_idx:
                info = self._task_info[self._rcvd_idx]
                worker_id = info[0]
                if len(info) == 2 or self._workers_status[worker_id]:  # has data or is still active
                    break
                del self._task_info[self._rcvd_idx]
                self._rcvd_idx += 1
            else:
                # no valid `self._rcvd_idx` is found (i.e., didn't break)
                if not self._persistent_workers:
                    self._shutdown_workers()
                raise StopIteration```

Yield From

        yield from self._staged_raw_predict(X)

List comprehension with conditionals

{
        key: np.asarray([score[key] for score in scores])
        if isinstance(scores[0][key], numbers.Number)
        else [score[key] for score in scores]
        for key in scores[0]
    }

for loops with Tuples

for est_idx, (name, (estimator, param_grid)) in \
        enumerate(zip(names, classifiers)):
    ax = axes[ds_cnt, est_idx + 1]

Calling returned function

res_outputs = (self._residual_fn or default_residual_fn)(inputs, outputs)
    return (res_outputs, new_state)

List comp with conditional

        if weights is None:
            y_pred = np.array([np.mean(_y[ind, :], axis=0)
                               if len(ind) else empty_obs
                               for (i, ind) in enumerate(neigh_ind)])

        else:
            y_pred = np.array([np.average(_y[ind, :], axis=0,
                               weights=weights[i])
                               if len(ind) else empty_obs
                               for (i, ind) in enumerate(neigh_ind)])

power assign

if inplace_csr_row_normalize is inplace_csr_row_normalize_l2:
                    X_csr.data **= 2

yield expression and yield from expression

    class Pop(nn.Module):
        def forward(self, input):
            foo = yield pop("foo")
            return foo # noqa

    def __iter__(self) -> Iterator[Tensor]:
        if self.atomic:
            yield self.tensor
        else:
            yield from self.tensors