google / rerast Goto Github PK

A tool for transforming Rust code using rules

License: Apache License 2.0

Rust 99.85% Shell 0.15%

rerast's Introduction

Rerast

Rerast is a search/replace tool for Rust code using rules. A rule consists of a search pattern, a replacement and possibly some placeholders that can appear in both the search pattern and the replacement. Matching is done on syntax, not on text, so formatting doesn't matter. Placeholders are typed and must match the type found in the code for the rule to apply.

Rerast is deprecated. We suggest using the Structured Search Replace feature available in rust-analyzer. It is available either in vscode or from the command line (and possibly also vim). If you are missing any particular feature that Rerast supported (or didn't), please comment on this issue.

If you'd like to still use Rerast, we suggest using Rerast version 0.1.88 with Rust version nightly-2020-02-27. There are a few newer version of Rerast, but there are some broken features in the newer versions.

Installation

rustup toolchain add nightly-2020-02-27
rustup component add --toolchain nightly-2020-02-27 rustc-dev
cargo +nightly-2020-02-27 install --version 0.1.88 rerast

Usage

Basic operations can be performed entirely from the command line

cargo +nightly-2020-02-27 rerast --placeholders 'a: i32' --search 'a + 1' --replace_with 'a - 1' --diff

Alternatively you can put your rule in a Rust file

fn rule1(a: i32) {
  replace!(a + 1 => a - 1);
}

then use

cargo +nightly-2020-02-27 rerast --rules_file=my_rules.rs

Putting your rules in a file is required if you want to apply multiple rules at once.

If you'd like to actually update your files, that can be done as follows:

cargo +nightly-2020-02-27 rerast --placeholders 'a: i32' --search 'a + 1' --replace_with 'a - 1' --force --backup

You can control which compilation roots rerast will inject the rule into using the --file argument, e.g.:

cargo +nightly-2020-02-27 rerast --rules_file=my_rules.rs --targets tests --file tests/testsuite/main.rs --diff

Here's a more complex example

use std::rc::Rc;
fn rule1<T>(r: Rc<T>) {
  replace!(r.clone() => Rc::clone(&r))
}

Here we're replacing calls to the clone() method on an Rc with the more explicit way of cloning an Rc - via Rc::clone.

"r" is a placeholder which will match any expression of the type specified. The name of the function "rule1" is not currently used for anything. In future it may be possible to selectively enable/disable rules by specifying their name, so it's probably a good idea to put a slightly descriptive name here. Similarly, comments placed before the function may in the future be displayed to users when the rule matches. This is not yet implemented.

A function can contain multiple invocations of the replace! macro, with earlier rules taking precedence. This is useful if you want to do several replacements that make use of the same placeholders or if you want to handle certain special patterns first, ahead of a more general match.

Besides replace! there are several other replacement macros that can be used:

replace_pattern! - this replaces patterns. e.g. &Some(a). Such a pattern might appear in a match arm or if let. Irrefutable patterns (those that are guaranteed to always match) can also be matched within let statements and function arguments.
replace_type! - this replaces types. It's currently a bit limited in that it doesn't support placeholders. Also note, if your type is just a trait you should consider using replace_trait_ref! instead, since trait references can appear in contexts where types cannot - specifically generic bounds and where clauses.
replace_trait_ref! - this replaces references to the named trait

Replacing statements is currently disabled pending a good use-case.

Matching macro invocations

Macro invocations can be matched so long as they expand to code that can be matched. Note however that a macro invocation will not match against the equivalent code, nor the invocation of a different, but identical macro. This is intentional. When verifying a match, we check that the same sequence of expansions was followed. Also note, that if a macro expands to something different every time it is invoked, it will never match. println! is an example of such a macro, since it generates a constant that is referenced from the expanded code and every invocation references a different constant.

Order of operations

Suppose you're replacing foo(a, b) with a && !b. Depending on what the placeholders end up matching and what context the entire expression is in, there may be need for extra parenthesis. For example if the matched code was !foo(x == 1, y == 2), if we didn't add any parenthesis, we'd end up with !x == 1 && !y == 2 which clearly isn't correct. Rerast detects this and adds parenthesis as needed in order to preserve the order or precedence found in the replacement. This would give !(x == 1 && !(y == 2)).

Formatting of code

No reformatting of code is currently done. Unmatched code will not be affected. Replacement code is produced by copying the replacement code from the rule and splicing in any matched patterns. In future, we may adjust identation for multi-line replacements. Running rustfmt afterwards is probably a good idea since some identation and line lengths may not be ideal.

Recursive and overlapping matches

The first matched rule wins. When some code is matched, no later rules will be applied to that code. However, code matched to placeholders will be searched for further matches to all rules.

Automatically determining a rule from a source code change

If you're about to make a change multiple times throughout your source code and you're using git, you can commit (or stage) your changes, make one edit then run:

cargo +nightly-2020-02-27 rerast --replay_git --diff

This will locate the changed expression in your project (of which there should be only one) then try to determine a rule that would have produced this change. It will print the rule, then apply it to your project. If you are happy with the changes, you can run again with --force to apply them, or you could copy the printed rule into a .rs file and apply it with --rules_file.

The rule produced will use placeholders to the maximum extent possible. i.e. wherever a subexpression is found in both the old and the new code, it will be replaced with a placeholder.
This only works for changed expressions at the moment, not for statements, types, patterns etc.
Your code must be able to compile both with and without the change.

Limitations

Use statements are not yet updated, so depending on your rule, may need to be updated after the rule is applied. This should eventually be fixed, there just wasn't time before release and it's kind of tricky.
Your code must be able to compile for this to work.
The replacement code must also compile. This means rerast is better at replacing a deprecated API usage with its non-deprecated equivalent than dealing with breaking changes. Often the best workaround is to create a new API temporarily.
Code within rustdoc is not yet processed and matched.
Conditional code that disabled with a cfg attribute isn't matched. It's suggested to enable all features if possible when running so that as much code can be checked as possible.
replace_type! doesn't yet support placeholders.
Probably many bugs and missing features. Please feel free to file bugs / feature requests.

Known issues

If you have integration tests (a "tests" directory) in your project, you might be no matches. Not sure why. This started from nightly-2020-04-10. You might be able to work around this issue by passing --targets '' to cargo rerast. Unfortunately then you won't get matches in non-integration tests (i.e. cfg(test)). Alternatively you could install an older version of rust and the corresponding rerast version.
Using ? in the replacement is currently broken. This broke I think in February 2020. Something changed with spans.

More examples

See the Rerast Cookbook for more examples.

Groups

Questions?

Feel free to just file an issue on github.

Authors

See Cargo.toml

Contributing

See CONTRIBUTING.md

Code of conduct

This project defers to the Rust code of conduct. If you feel someone is not adhering to the code of conduct in relation to this project, please contact David Lattimore. My email address is in Cargo.toml.

Disclaimer

This is not an official Google product. It's released by Google only because the (original) author happens to work there.

rerast's People

Contributors

Stargazers

Watchers

rerast's Issues

Build failure with latest nightly: ExprKind::AddrOf variant changed

With an up to date nightly (rustc 1.41.0-nightly (412f43ac5 2019-11-24)), I get the following build failure:

   Compiling rerast v0.1.76
error[E0023]: this pattern has 3 fields, but the corresponding tuple variant has 2 fields
    --> /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/rerast-0.1.76/src/rule_finder.rs:249:24
     |
249  |                 if let hir::ExprKind::AddrOf(_, _, ref expr) = addr_expr.kind {
     |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 2 fields, found 3

error[E0023]: this pattern has 3 fields, but the corresponding tuple variant has 2 fields
    --> /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/rerast-0.1.76/src/rule_matcher.rs:516:18
     |
516  |                 &ExprKind::AddrOf(p_kind, p_mut, ref p_expr),
     |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 2 fields, found 3

error[E0023]: this pattern has 3 fields, but the corresponding tuple variant has 2 fields
    --> /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/rerast-0.1.76/src/rule_matcher.rs:517:18
     |
517  |                 &ExprKind::AddrOf(c_kind, c_mut, ref c_expr),
     |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 2 fields, found 3

Macros are lost when compiling nickel.rs

@davidlattimore, you're on a roll! Here's another one: https://github.com/nickel-org/nickel.rs

warning: unused #[macro_use] import
--> examples/https.rs:1:1
|
1 | #[macro_use] extern crate nickel;
| ^^^^^^^^^^^^
|
= note: #[warn(unused_imports)] on by default

error: cannot find macro lazy_static! in this scope
--> examples/integration_testing.rs:279:9
|
279 | lazy_static! {
| ^^^^^^^^^^^

error: cannot find macro middleware! in this scope
--> examples/integration_testing.rs:81:21
|
81 | server.get("/", middleware!( "Hello World" ));
| ^^^^^^^^^^

replace_type macro fails on rust edition 2018.

When running a replace_type! macro I am getting the following compile error.

error[E0432]: unresolved import `__rerast_internal`
  --> src/__rerast_internal.rs:58:19
   |
58 |             use ::__rerast_internal::GetMode;
   |                   ^^^^^^^^^^^^^^^^^ did you mean `crate::__rerast_internal`?
   |
  ::: src/__rerast_rules.rs:5:5
   |
5  |     replace_type!(Result<r, Box<Error>> => Result<r, Box<dyn Error>>);
   |     ------------------------------------------------------------------ in this macro invocation
   |
   = note: `use` statements changed in Rust 2018; read more at <https://doc.rust-lang.org/edition-guide/rust-2018/module-system/path-clarity.html>

How can I use a type from the crate I'm fixing?

I tried this code:

extern crate rcc;

use rcc::data::Location;

fn rule1(l: Location) -> Location {
    replace!(l.clone() => l);
    unreachable!()
}

but I get a very long JSON error saying 'can't find crate for rcc'.

Rerast requires `rustup`

When using a nightly rust outside of rustup, rerast fails to compile because RUSTUP_HOME and RUSTUP_TOOLCHAIN aren't defined.

dot async example

Is it possible to replace await macros to dot await syntax?

fn rule1<T>(r: T) {
  replace!(await!(r) => r.await)
}

Processing src/main.rs
error[E0728]: `await` is only allowed inside `async` functions and blocks
 --> src/__rerast_rules.rs:3:12
  |
2 | fn rule1<T>(r: T) {
  |    ----- this is not `async`
3 |   replace!(await!(r) => r.await)
  |            ^^^^^^^^^ only allowed inside `async` functions and blocks

error[E0728]: `await` is only allowed inside `async` functions and blocks
 --> src/__rerast_rules.rs:3:25
  |
2 | fn rule1<T>(r: T) {
  |    ----- this is not `async`
3 |   replace!(await!(r) => r.await)
  |                         ^^^^^^^ only allowed inside `async` functions and blocks

error: aborting due to 2 previous errors

build failure with latest nightly

rustc 1.26.0-nightly (521d91c6b 2018-03-14)

repo was at @ 1a31cb5

   Compiling rerast v0.1.7 (file:///tmp/rerast)
error[E0061]: this function takes 0 parameters but 1 parameter was supplied
   --> src/rule_matcher.rs:333:33
    |
333 |                 let param_env = ty::ParamEnv::empty(Reveal::All);
    |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 0 parameters
error[E0061]: this function takes 0 parameters but 1 parameter was supplied
   --> src/rule_matcher.rs:362:22
    |
362 |             .can_sub(ty::ParamEnv::empty(Reveal::All), a, b)
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 0 parameters
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0061`.
error: Could not compile `rerast`.

Is it possible to replace try! with ?

I'm trying to write a rule that replaces try!(x) with x?, but it seems tricky.

This seems to compile (I've had to add a return type):

fn rule1<T,E,X: From<E>>(r: Result<T,E>) -> Result<T,X> {
    replace!(try!(r) => r?);
    Ok(r?)
}

but it doesn't match anything. Without return type I can't use try! nor ? in the pattern, since they expand to return, which rustc typechecks.

Can this replacement be done?

Matching of string literals

String literals are somewhat special in Rust in that some built-in macros only accept them and not arbitrary &'static str.

I've tried to run:

cargo +nightly rerast --placeholders 's: &str, a: impl std::fmt::Display' --search 'eprintln!(s, a)' --replace_with 'info!(s, a)' --diff

but this fails with

error: error: format argument must be a string literal
 --> src\__rerast_rules.rs:1:68
  |
1 | pub fn rule(s: &str, a: impl std::fmt::Display){replace!(eprintln!(s, a) => info!(s, a));}
  |                                                                    ^
  |
help: you might be missing a string literal to format with
  |
1 | pub fn rule(s: &str, a: impl std::fmt::Display){replace!(eprintln!("{} {}", s, a) => info!(s, a));}
  |                                                                    ^^^^^^^^

because the passed argument is, well, an argument and not an actual literal.

Span needs thread-locals initialized

Transforming glutin on Linux (4.4.0-116-generic #140-Ubuntu x86_64), rustc 1.27.0-nightly (ac3c2288f 2018-04-18), with rules:

fn rule1<T,E,X: From<E>>(r: Result<T,E>) -> Result<T,X> {
    replace!(try!(r) => r?);
    unreachable!()
}

I get

thread 'main' panicked at 'cannot access a scoped thread local variable without calling `set` first', /cargo/registry/src/github.com-1ecc6299db9ec823/scoped-tls-0.1.1/src/lib.rs:186:9
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:205
   3: std::panicking::default_hook
             at libstd/panicking.rs:221
   4: rustc::util::common::panic_hook
   5: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:461
   6: std::panicking::begin_panic
   7: <scoped_tls::ScopedKey<T>>::with
   8: syntax_pos::<impl core::cmp::PartialOrd for syntax_pos::span_encoding::Span>::partial_cmp
   9: alloc::slice::<impl [T]>::sort::{{closure}}
  10: alloc::slice::merge_sort
  11: rerast::code_substitution::FileRelativeSubstitutions::merge
  12: rerast::RerastOutput::merge
  13: cargo_rerast::cargo_rerast
  14: cargo_rerast::main

rust-lang/rustfmt#2532 suggests calling syntax::with_globals before using parser.

Intermittent panics in rustc related to incremental compilation

Seen when running rerast on glutin. @kornelski suggested it's due to incremental compilation, and indeed setting CARGO_INCREMENTAL=0 seems to stop it. Not sure if it's a bug in incremental compilation or just an incompatibility between the way we're using rustc and incremental compilation.

thread 'main' panicked at 'Forcing query with already existing DepNode.

query-key: ParamEnvAnd { param_env: ParamEnv { caller_bounds: Slice([]), reveal: UserFacing }, value: unsafe extern "rust-intrinsic" fn(*const std::os::raw::c_void) -> extern "system" fn(u32, i32, *mut i32) {std::intrinsics::transmute::<*const std::os::raw::c_void, extern "system" fn(u32, i32, *mut i32)>} }
dep-node: IsCopy(7cbec90d2168a3bd-88dbb5e68c554118)', librustc/ty/maps/mod.rs:97:1
stack backtrace:
0: std::sys::unix::backtrace::tracing:imp:unwind_backtrace
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: std::sys_common::backtrace::print
at libstd/sys_common/backtrace.rs:71
at libstd/sys_common/backtrace.rs:59
2: std::panicking::default_hook::{{closure}}
at libstd/panicking.rs:206
3: std::panicking::default_hook
at libstd/panicking.rs:222
4: rustc::util::common::panic_hook
5: std::panicking::rust_panic_with_hook
at libstd/panicking.rs:401
6: std::panicking::begin_panic_fmt
at libstd/panicking.rs:347
7: rustc::ty::maps::<impl rustc::ty::maps::queries::is_copy_raw<'tcx>>::force_with_lock
8: rustc::ty::maps::<impl rustc::ty::maps::queries::is_copy_raw<'tcx>>::try_get
9: rustc::ty::maps::TyCtxtAt::is_copy_raw
10: rustc::middle::mem_categorization::MemCategorizationContext::type_moves_by_default
11: rustc::middle::expr_use_visitor::ExprUseVisitor::consume_expr
12: rustc::middle::expr_use_visitor::ExprUseVisitor::walk_expr
13: rustc::middle::expr_use_visitor::ExprUseVisitor::walk_expr
14: rustc::middle::expr_use_visitor::ExprUseVisitor::walk_expr
15: <rustc_passes::rvalue_promotion::CheckCrateVisitor<'a, 'tcx> as rustc::hir::intravisit::Visitor<'tcx>>::visit_nested_body
16: rustc_passes::rvalue_promotion::rvalue_promotable_map
17: rustc::dep_graph::graph::DepGraph::with_task_impl
18: rustc::ty::context::tls::with_related_context
19: rustc::ty::maps::<impl rustc::ty::maps::queries::rvalue_promotable_map<'tcx>>::force_with_lock
20: rustc::ty::maps::<impl rustc::ty::maps::queries::rvalue_promotable_map<'tcx>>::try_get
21: rustc::ty::maps::TyCtxtAt::rvalue_promotable_map
22: rustc::ty::maps::<impl rustc::ty::context::TyCtxt<'a, 'tcx, 'lcx>>::rvalue_promotable_map
23: rustc_passes::rvalue_promotion::const_is_rvalue_promotable_to_static
24: rustc::dep_graph::graph::DepGraph::with_task_impl
25: rustc::ty::context::tls::with_related_context
26: rustc::ty::maps::<impl rustc::ty::maps::queries::const_is_rvalue_promotable_to_static<'tcx>>::force_with_lock
27: rustc::ty::maps::<impl rustc::ty::maps::queries::const_is_rvalue_promotable_to_static<'tcx>>::try_get
28: rustc::ty::maps::TyCtxtAt::const_is_rvalue_promotable_to_static
29: rustc::ty::maps::<impl rustc::ty::context::TyCtxt<'a, 'tcx, 'lcx>>::const_is_rvalue_promotable_to_static
30: rustc_passes::rvalue_promotion::check_crate
31: rustc::ty::context::tls::enter_context
32: std::thread::local::LocalKey::with
33: rustc::ty::context::TyCtxt::create_and_enter
34: rustc_driver::driver::compile_input
35: rustc_driver::run_compiler_impl
36: syntax::with_globals
37: rustc_driver::run_compiler
38: rerast::CompilerInvocationInfo::run_compiler
at src/lib.rs:165
39: rerast::run_compiler
at src/lib.rs:449
40: rerast::RerastCompilerDriver::apply_rules_to_code
at /home/dml/rerast/src/lib.rs:556
41: rerast::RerastCompilerDriver::apply_rules_from_string
at /home/dml/rerast/src/lib.rs:510
42: cargo_rerast::cargo_rerast
at src/bin/cargo-rerast.rs:364
43: cargo_rerast::main
at src/bin/cargo-rerast.rs:451
44: std::rt::lang_start::{{closure}}
at /checkout/src/libstd/rt.rs:74
45: std::panicking::try::do_call
at libstd/rt.rs:59
at libstd/panicking.rs:305
46: __rust_maybe_catch_panic
at libpanic_unwind/lib.rs:101
47: std::rt::lang_start_internal
at libstd/panicking.rs:284
at libstd/panic.rs:361
at libstd/rt.rs:58
48: std::rt::lang_start
at /checkout/src/libstd/rt.rs:74
49: main
50: __libc_start_main
51: start
query stack during panic:
#0 [rvalue_promotable_map] checking which parts of support::gl::Gles2::GetUniformiv are promotable to static
--> /home/dml/src/glutin/target/debug/build/glutin-7d863af2e66ed1de/out/test_gl_bindings.rs:1734:23
|
1734 | #[inline] pub unsafe fn GetUniformiv(&self, program: types::GLuint, location: types::GLint, params: *mut types::GLint) -> () { __gl_imports::mem::transmute::<, extern "system" fn(types::GLuint, types::GLint, *mut types::GLint) -> ()>(self.GetUniformiv.f)(program, location, params) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#1 [const_is_rvalue_promotable_to_static] const checking if rvalue is promotable to static support::gl::Gles2::GetUniformiv
--> /home/dml/src/glutin/target/debug/build/glutin-7d863af2e66ed1de/out/test_gl_bindings.rs:1734:23
|
1734 | #[inline] pub unsafe fn GetUniformiv(&self, program: types::GLuint, location: types::GLint, params: *mut types::GLint) -> () { __gl_imports::mem::transmute::<_, extern "system" fn(types::GLuint, types::GLint, *mut types::GLint) -> ()>(self.GetUniformiv.f)(program, location, params) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
end of query stack

Transform attribute values

Is it currently possible to change values of an attribute? This could then be used to automate the following one when creating new releases:

#![doc(html_root_url = "https://docs.rs/<crate_name>/<version>")]

Rewrite discards expression for variable

Given:

pub struct Execs {
    expect_exit_code: Option<i32>,
}

impl Execs {
    /// Verify the exit code from the process.
    pub fn with_status(mut self, expected: i32) -> Execs {
        self.expect_exit_code = Some(expected);
        self
    }
}

pub fn execs() -> Execs {
    Execs {
        expect_exit_code: Some(0),
    }
}

pub fn usage() {
    execs().with_status(0);
}

(derived from Cargo's testsuite support code)

The rewrite rule:

use Execs;

fn rule1(e: Execs) {
   replace!(e.with_status(0) => e);
}

applied with:

cargo +nightly rerast --rules_file=rewrite.rs --force

results in:

 pub fn usage() {
-    execs().with_status(0);
+    e;
 }

which doesn't compile. It should result in:

 pub fn usage() {
-    execs().with_status(0);
+    execs();
 }

Build diffing into the tool

Currently diffing it handled by calling out to an external diff command (e.g. diff or colordiff). To avoid people needing to have these installed, it'd be nice to build diff support (including colors) into the tool.

rustfmt has some diff code. Perhaps that could be pulled out into a separate crate that this crate could then depend on. https://github.com/rust-lang-nursery/rustfmt/blob/master/src/rustfmt_diff.rs

Unexpected character

Hi,

I'm trying this on a somewhat complex project, and I'm occasionally getting the following error message when I run cargo +nightly rerast --rules_file=changes.rs --diff

Unexpected character: R at (1:1)

(The second number in brackets -- maybe a character position -- changes every time I run the command)

Documentation examples of the replace_type and replace_trait_ref.

I've been trying to get a rules file that rewrites Result<r, Box<Error>> into Result<r, Box<dyn Error>> and failing. It would be useful to have an example for the replace_type macro since those are pretty much undocumented at the moment.

For reference my current attempt at this looks like

 use std::error::Error;

fn use_dyn_for_boxed_error_traits<T>(r: T) {
    replace_type!(Result<r, Box<Error>> => Result<r, Box<dyn Error>>);
    unreachable!()
}

deny(missing_docs) prevents the tool from working

For example, xz2-rs has

#![deny(missing_docs, warnings)]

which gets applied to rerast's own internals, breaking compilation:

error: missing documentation for a struct
--> src/__rerast_internal.rs:155:1
|
155 | pub struct _PatternRuleMarker;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: missing documentation for a struct
--> src/__rerast_internal.rs:157:1
|
157 | pub struct _TraitRefRuleMarker;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Perhaps an explicit #![allow(missing_docs, warnings)] in the internal file could reverse this?

Matches get duplicated if the same code is compiled in multiple targets

This is noticeable if you have a library with an integration test, since that triggers cargo to compile a non-test version of the library for use by the integration test. Rerast runs the compiler 3 times:

Compile lib
Compile lib with --test
Compile integration test

Any matches in non-test code in the library get duplicated in 1 and 2.

I have written a fix for this just now. Fixing this involved moving the code that produces new file contents from running inside the compiler, to running after the compiler has finished. This way we can merge the results from the separate compiler runs together before we produce the new file contents. In order to move it out, spans had to be made file relative, since once we're outside of a compiler execution, we no longer have access to the CodeMap - also each compilation has a separate CodeMap.

can't compile on latest nightly

Getting this error:

error[E0609]: no field `kind` on type `rustc_middle::ty::subst::GenericArg<'_>`
   --> /Users/davidwg/.cargo/registry/src/github.com-1ecc6299db9ec823/rerast-0.1.90/src/change_to_rule.rs:387:48
    |
387 |             if let TyKind::Param(..) = subtype.kind {
    |                                                ^^^^ unknown field

error: aborting due to previous error

I've also managed to compile with nightly-2020-04-06 and getting errors like this when I try to use the tool:

thread 'main' panicked at 'assertion failed: self.host', src/tools/cargo/src/cargo/core/profiles.rs:940:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

build failure with rustc 1.32.0-nightly (f4a421ee3 2018-12-13)

Hi, the latest release does not build with latest nightly unfortunately.

   Compiling rerast v0.1.31
error[E0023]: this pattern has 2 fields, but the corresponding tuple variant has 3 fields
  --> /home/matthias/.cargo/registry/src/github.com-1ecc6299db9ec823/rerast-0.1.31/src/change_to_rule.rs:86:13
   |
86 |             TokenTree::Delimited(_span, delimited) => {
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 3 fields, found 2

error: aborting due to previous error

For more information about this error, try `rustc --explain E0023`.
error: failed to compile `rerast v0.1.31`, intermediate artifacts can be found at `/tmp/cargo-instally8uCjp`

Replacing try macro in 2018 edition

I have a project which has moved to the 2018 edition and has had rustfix applied to replace try!(...) with r#try!(...). Should rerast work converting those to ?? I tried with the latest release, master, and the nightly-2019-05-13 version mentioned above; in all cases the diff is empty and --force has no effect.

Originally posted by @jugglerchris in #30 (comment)

Example of modifying a function definition?

I'd like to write a rule that modifies function declarations, such as deleting an argument of a specific type. For instance, I'd like to translate all functions of type fn foo(file: File, s: &str) to delete the s argument and all uses of it. However, in trying to do so, I received an error from replace!, saying that no rules expected the token fn.

How can I write a rule that would match fn foo(file: File, s: &str) and delete the s: &str argument? (And then additionally match inside the body of any matching function to make more changes?)

Cargo env vars are not present when compiling

fn test() {
    env!("OUT_DIR");
}

error: environment variable OUT_DIR not defined

Crates reading CARGO_PKG_VERSION, etc. won't compile without these variables being set. OUT_DIR is especially problematic, because it's used to dynamically include autogenerated files (FFI), so it can't be set to a dummy value.

Build targets that start with #![cfg(feature="something")] can cause panic

thread 'main' panicked at 'Internal error, failed to find rerast type definitions', libcore/option.rs:914:5

This is because rerast can't find its internal definitions and the rules that are to be applied due to the cfg attribute suppressing compilation of everything. We should just ignore this. There won't be any matches, but there might be for some other build targets and we don't want to prevent them being found. Noticed this while running rerast on xz2-rs while trying to reproduce #8.

try! example panics when fixing docs.rs

I'm trying to run rerast on https://github.com/rust-lang/docs.rs/, but it panics when I run the command. I'm not sure what I'm doing wrong and I don't understand the error message. I'm using the try! example directly from COOKBOOK.md.

$ cargo +nightly rerast --rules_file=rules.rs --force
thread 'rustc' panicked at 'Bad substitutions: [
    CodeSubstitution {
        span: src/__rerast_rules.rs:2:25: 2:26,
        new_code: "initialize_package_in_database(&conn, metadata_pkg)",
        needs_parenthesis: false,
        code_is_single_tree: false,
    },
    CodeSubstitution {
        span: src/__rerast_rules.rs:2:26: 2:27,
        new_code: "err",
        needs_parenthesis: false,
        code_is_single_tree: false,
    },
    CodeSubstitution {
        span: src/__rerast_rules.rs:2:26: 2:27,
        new_code: "err",
        needs_parenthesis: false,
        code_is_single_tree: false,
    },
    CodeSubstitution {
        span: src/__rerast_rules.rs:2:25: 2:27,
        new_code: "val",
        needs_parenthesis: false,
        code_is_single_tree: false,
    },
    CodeSubstitution {
        span: src/__rerast_rules.rs:2:25: 2:27,
        new_code: "val",
        needs_parenthesis: false,
        code_is_single_tree: false,
    },
]
For source: SourceChunk { source: "r?", start_pos: BytePos(214288) }', /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/rerast-0.1.73/src/code_substitution.rs:146:13

Rewrite file:

fn rule1<T,E,X: From<E>>(r: Result<T,E>) -> Result<T,X> {
    replace!(try!(r) => r?);
    unreachable!()
}

Backtrace

stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1030
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:188
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:205
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:464
  11: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:373
  12: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:328
  13: rerast::code_substitution::CodeSubstitution<syntax_pos::span_encoding::Span>::apply_with_source_map
  14: rerast::rule_matcher::substitions_for_matches
  15: rustc::ty::context::tls::enter_global
  16: rustc_interface::passes::BoxedGlobalCtxt::access::{{closure}}
  17: rustc_interface::passes::create_global_ctxt::{{closure}}
  18: <rerast::RerastCompilerCallbacks as rustc_driver::Callbacks>::after_analysis
  19: rustc_interface::interface::run_compiler_in_existing_thread_pool
  20: std::thread::local::LocalKey<T>::with
  21: scoped_tls::ScopedKey<T>::set
  22: syntax::with_globals

Build errors on installation

I'm getting the following errors when trying to install rerast:

$ cargo +nightly install rerast
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading rerast v0.1.19
  Installing rerast v0.1.19
 Downloading json v0.11.13
 Downloading rerast_macros v0.1.12
   Compiling unicode-xid v0.0.4
   Compiling quote v0.3.15
   Compiling cfg-if v0.1.3
   Compiling unicode-width v0.1.5
   Compiling libc v0.2.42
   Compiling rustc-demangle v0.1.8
   Compiling bitflags v1.0.3
   Compiling either v1.5.0
   Compiling ansi_term v0.11.0
   Compiling strsim v0.7.0
   Compiling lazy_static v0.2.11
   Compiling vec_map v0.8.1
   Compiling diff v0.1.11
   Compiling json v0.11.13
   Compiling rerast_macros v0.1.12
   Compiling textwrap v0.9.0
   Compiling synom v0.11.3
   Compiling itertools v0.5.10
   Compiling colored v1.6.0
   Compiling atty v0.2.10
   Compiling backtrace v0.3.8
   Compiling syn v0.11.11
   Compiling clap v2.31.2
   Compiling synstructure v0.6.1
   Compiling failure_derive v0.1.1
   Compiling failure v0.1.1
   Compiling rerast v0.1.19
error[E0412]: cannot find type `AnonConst` in module `hir`
    --> src/rule_matcher.rs:1087:25
     |
1087 | impl Matchable for hir::AnonConst {
     |                         ^^^^^^^^^ not found in `hir`

error[E0532]: expected tuple struct/variant, found unit variant `Crate`
    --> src/rule_matcher.rs:1125:14
     |
1125 |             (Crate(p_sugar), Crate(c_sugar)) => p_sugar == c_sugar,
     |              ^^^^^ not a tuple struct/variant
help: possible better candidate is found in another module, you can import it into scope
     |
15   | use syntax::ast::VisibilityKind::Crate;
     |

error[E0532]: expected tuple struct/variant, found unit variant `Crate`
    --> src/rule_matcher.rs:1125:30
     |
1125 |             (Crate(p_sugar), Crate(c_sugar)) => p_sugar == c_sugar,
     |                              ^^^^^ not a tuple struct/variant
help: possible better candidate is found in another module, you can import it into scope
     |
15   | use syntax::ast::VisibilityKind::Crate;
     |

error[E0053]: method `build_controller` has an incompatible type for trait
   --> src/change_to_rule.rs:260:5
    |
260 | /     fn build_controller(
261 | |         self: Box<Self>,
262 | |         sess: &rustc::session::Session,
263 | |         matches: &getopts::Matches,
...   |
271 | |         control
272 | |     }
    | |_____^ expected mutable reference, found struct `std::boxed::Box`
    |
    = note: expected type `fn(&mut change_to_rule::RCompilerCalls, &rustc::session::Session, &getopts::Matches) -> rustc_driver::driver::CompileController<'a>`
               found type `fn(std::boxed::Box<change_to_rule::RCompilerCalls>, &rustc::session::Session, &getopts::Matches) -> rustc_driver::driver::CompileController<'a>`

error[E0053]: method `build_controller` has an incompatible type for trait
   --> src/lib.rs:335:5
    |
335 | /     fn build_controller(
336 | |         self: Box<Self>,
337 | |         sess: &Session,
338 | |         matches: &getopts::Matches,
...   |
347 | |         control
348 | |     }
    | |_____^ expected mutable reference, found struct `std::boxed::Box`
    |
    = note: expected type `fn(&mut RerastCompilerCalls, &rustc::session::Session, &getopts::Matches) -> rustc_driver::driver::CompileController<'a>`
               found type `fn(std::boxed::Box<RerastCompilerCalls>, &rustc::session::Session, &getopts::Matches) -> rustc_driver::driver::CompileController<'a>`

error: aborting due to 5 previous errors

Some errors occurred: E0053, E0412, E0532.
For more information about an error, try `rustc --explain E0053`.
error: failed to compile `rerast v0.1.19`, intermediate artifacts can be found at `/var/folders/jy/jgttnxcd4750bdcw119bc9cw0000gn/T/cargo-installABdedY`

Caused by:
  Could not compile `rerast`.

To learn more, run the command again with --verbose.

Specifying type () in placeholders generates bad syntax

Specifying placeholders containing the unit type () causes rerast to generate incorrect syntax:

 cargo +nightly rerast --diff -p 'x: bool, y: ()' -s 'x' -r 'x'
error: error: expected one of `(` or `<`, found `:`
 --> src/__rerast_rules.rs:1:13
  |
1 | pub fn rulex: bool, y: (){replace!(x => x);}
  |             ^ expected one of `(` or `<`

rerast seems to have left out the parentheses around the placeholders.

Allow running when not in root directory

(bash) ~/.../import-test/src ▶️ cargo +nightly rerast --file src/lib.rs -p 'l1: crate::data::prelude::Location' -s 'l1.foo()' -r 'l1.bar()' --diff_cmd diff
error: No such file or directory (os error 2)
(bash) ~/.../import-test/src ▶️ cd ..
Cargo.lock  Cargo.toml  src  target
(bash) ~/.../rerast/import-test ▶️ cargo +nightly rerast --file src/lib.rs -p 'l1: crate::data::prelude::Location' -s 'l1.foo()' -r 'l1.bar()' --diff_cmd diff
27c27
<         assert_eq!(l.foo(), 42);
---
>         assert_eq!(l.bar(), 42);

I think this might be happening after the diff is calculated because on my large project it runs for a while before giving an error.

Build fails on 1.35-nightly

% rustc --version
rustc 1.35.0-nightly (94fd04589 2019-03-21)
% cargo install rerast
...
error[E0599]: no variant named `Ctor` found for type `rustc::hir::def::Def` in the current scope
   --> /Users/lazarus/.cargo/registry/src/github.com-1ecc6299db9ec823/rerast-0.1.46/src/change_to_rule.rs:572:20
    |
572 |             | Def::Ctor(..) => {
    |               -----^^^^---- variant not found in `rustc::hir::def::Def`

Features declared in Cargo.toml can cause files to not get checked without clean

If a crate has a non-empty [features] section, then it seems that not all targets get checked for matches unless the user runs "cargo clean" first.

Matching an expression expanded more than once by a macro causes panic

macro_rules! dupe {($a:expr) => (($a, $a))}

fn main() {
println!("{:?}", dupe!(41));
}

If we replace 41 with 42, we match 41 twice since it's in the HIR twice, but both matches have the same span which causes us to panic we we try to modify the file.

Running cargo rerast in workspaces fails

I was trying to replace r#try!(...) with ...? in a project which happens to be inside a workspace.

I've put an example here:

https://github.com/jugglerchris/temp_rerast_workspace_bug

Running cargo +nightly-2019-05-13 rerast --rules_file=rtry.rs --diff from inside crates/rerast_test inside that repo just outputs:

error: No such file or directory (os error 2)

If I rename the Cargo.toml in the workspace root to Cargo.toml_, then it works.

cargo check unexpectedly fails with status 101

On some projects, like https://github.com/tomaka/glutin rerast fails during cargo check phase, even though cargo check run without rerast works fine:

cargo +nightly rerast --rules_file=../rules.rs --force --verbose

Running cargo check in order to build dependencies and get rustc commands
cargo check failed (exit code = 101). Output follows:
RERAST_JSON_MARKER: [["…lots of normal output here, no mention of errors…"}]

This is surprising, because cargo check alone seems to work fine (it's even cached from rerast's run):

$ cargo check
    Finished dev [unoptimized + debuginfo] target(s) in 0.16 secs
$ echo $?
0

Better error messages

Right now if I do something wrong in rules.rs it gets printed as JSON and it's really hard to read. It would be nice to print just the rendered error.

Current:

{"message":"failed to resolve: use of undeclared type or module `rcc`","code":{"code":"E0433","explanation":"\nAn undeclared type or module was used.\n\nErroneous code example:\n\n```compile_fail,E0433\nlet map = HashMap::new();\n// error: failed to resolve: use of undeclared type or module `HashMap`\n```\n\nPlease verify you didn't misspell the type/module's name or that you didn't\nforget to import it:\n\n\n```\nuse std::collections::HashMap; // HashMap has been imported.\nlet map: HashMap<u32, u32> = HashMap::new(); // So it can be used!\n```\n"},"level":"error","spans":[{"file_name":"src/__rerast_rules.rs","byte_start":4,"byte_end":7,"line_start":1,"line_end":1,"column_start":5,"column_end":8,"is_primary":true,"text":[{"text":"use rcc::data::prelude::Location;","highlight_start":5,"highlight_end":8}],"label":"use of undeclared type or module `rcc`","suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[],"rendered":"\u001b[0m\u001b[1m\u001b[38;5;9merror[E0433]\u001b[0m\u001b[0m\u001b[1m: failed to resolve: use of undeclared type or module `rcc`\u001b[0m\n\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m--> \u001b[0m\u001b[0msrc/__rerast_rules.rs:1:5\u001b[0m\n\u001b[0m  \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m|\u001b[0m\n\u001b[0m\u001b[1m\u001b[38;5;12m1\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m| \u001b[0m\u001b[0muse rcc::data::prelude::Location;\u001b[0m\n\u001b[0m  \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m| \u001b[0m\u001b[0m    \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;9m^^^\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;9muse of undeclared type or module `rcc`\u001b[0m\n\n"}

Requested:

error[E0433]: failed to resolve: use of undeclared type or module `rcc`
 --> src/__rerast_rules.rs:1:5
  |
1 | use rcc::data::prelude::Location;
  |     ^^^ use of undeclared type or module `rcc`