Git Product home page Git Product logo

codetracking.jl's Introduction

CodeTracking

Build status Coverage

CodeTracking can be thought of as an extension of Julia's InteractiveUtils library. It provides an interface for obtaining:

  • the strings and expressions of method definitions
  • the method signatures at a specific file & line number
  • location information for "dynamic" code that might have moved since it was first loaded
  • a list of files that comprise a particular package.

CodeTracking is a minimal package designed to work with Revise.jl (for versions v1.1.0 and higher). CodeTracking is a very lightweight dependency.

Examples

@code_string and @code_expr

julia> using CodeTracking, Revise

julia> print(@code_string sum(1:5))
function sum(r::AbstractRange{<:Real})
    l = length(r)
    # note that a little care is required to avoid overflow in l*(l-1)/2
    return l * first(r) + (iseven(l) ? (step(r) * (l-1)) * (l>>1)
                                     : (step(r) * l) * ((l-1)>>1))
end

julia> @code_expr sum(1:5)
[ Info: tracking Base
quote
    #= toplevel:977 =#
    function sum(r::AbstractRange{<:Real})
        #= /home/tim/src/julia-1/base/range.jl:978 =#
        l = length(r)
        #= /home/tim/src/julia-1/base/range.jl:980 =#
        return l * first(r) + if iseven(l)
                    (step(r) * (l - 1)) * l >> 1
                else
                    (step(r) * l) * (l - 1) >> 1
                end
    end
end

@code_string succeeds in that case even if you are not using Revise, but @code_expr always requires Revise. (If you must live without Revise, you can use Meta.parse(@code_string(...)) as a fallback.)

"Difficult" methods are handled more accurately with @code_expr and Revise. Here's one that's defined via an @eval statement inside a loop:

julia> @code_expr Float16(1) + Float16(2)
:(a::Float16 + b::Float16 = begin
          #= /home/tim/src/julia-1/base/float.jl:398 =#
          Float16(Float32(a) + Float32(b))
      end)

whereas @code_string cannot return a useful result:

julia> @code_string Float16(1) + Float16(2)
"# This file is a part of Julia. License is MIT: https://julialang.org/license\n\nconst IEEEFloat = Union{Float16, Float32, Float64}"

Consequently it's recommended to use @code_expr in preference to @code_string wherever possible.

@code_expr and @code_string have companion functional variants, code_expr and code_string, which accept the function and a Tuple{T1, T2, ...} of types.

@code_expr and @code_string are based on the lower-level function definition; you can read about it with ?definition.

Location information

julia> using CodeTracking, Revise

julia> m = @which sum([1,2,3])
sum(a::AbstractArray) in Base at reducedim.jl:648

julia> Revise.track(Base)  # also edit reducedim.jl

julia> file, line = whereis(m)
("/home/tim/src/julia-1/usr/share/julia/base/reducedim.jl", 642)

julia> m.line
648

In this (ficticious) example, sum moved because I deleted a few lines higher in the file; these didn't affect the functionality of sum (so we didn't need to redefine and recompile it), but it does change the starting line number of the file at which this method appears. whereis reports the current line number, and m.line the old line number. (For technical reasons, it is important that m.line remain at the value it had when the code was lowered.)

Other methods of whereis allow you to obtain the current position corresponding to a single statement inside a method; see ?whereis for details.

CodeTracking can also be used to find out what files define a particular package:

julia> using CodeTracking, Revise, ColorTypes

julia> pkgfiles(ColorTypes)
PkgFiles(ColorTypes [3da002f7-5984-5a60-b8a6-cbb66c0b333f]):
  basedir: /home/tim/.julia/packages/ColorTypes/BsAWO
  files: ["src/ColorTypes.jl", "src/types.jl", "src/traits.jl", "src/conversions.jl", "src/show.jl", "src/operations.jl"]

You can also find the method-signatures at a particular location:

julia> signatures_at(ColorTypes, "src/traits.jl", 14)
1-element Array{Any,1}:
 Tuple{typeof(red),AbstractRGB}

julia> signatures_at("/home/tim/.julia/packages/ColorTypes/BsAWO/src/traits.jl", 14)
1-element Array{Any,1}:
 Tuple{typeof(red),AbstractRGB}

CodeTracking also helps correcting for Julia issue #26314:

julia> @which uuid1()
uuid1() in UUIDs at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\UUIDs\src\UUIDs.jl:50

julia> CodeTracking.whereis(@which uuid1())
("C:\\Users\\SomeOne\\AppData\\Local\\Julia-1.1.0\\share\\julia\\stdlib\\v1.1\\UUIDs\\src\\UUIDs.jl", 50)

A few details

CodeTracking has limited functionality unless the user is also running Revise, because Revise populates CodeTracking's internal variables. (Using whereis as an example, CodeTracking will just return the file/line info in the method itself if Revise isn't running.)

CodeTracking is perhaps best thought of as the "query" part of Revise.jl, providing a lightweight and stable API for gaining access to information it maintains internally.

Limitations (without Revise)

  • parsing sometimes starts on the wrong line. Line numbers are determined by counting '\n' in the source file, without parsing the contents. Consequently quoted- or in-code '\n' can mess up CodeTracking's notion of line numbering
  • default constructor methods for structs are not found

codetracking.jl's People

Contributors

dependabot[bot] avatar juliatagbot avatar keno avatar kristofferc avatar oscardssmith avatar oxinabox avatar ranocha avatar t-bltg avatar timholy avatar zentrik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

codetracking.jl's Issues

Wrong body output when a closure is defined of the first line of the method

julia> function main()
           todB(x) = 10*log10(x)
           println("100x is $(todB(100)) dB.")
       end
main (generic function with 1 method)

julia> CodeTracking.definition(String, @which main())
("    todB(x) = 10*log10(x)", 2)

Which is just the closure defined inside the method that we are querying the body for.
If we add something before the closure:

julia> function main()
           foo
           todB(x) = 10*log10(x)
           println("100x is $(todB(100)) dB.")
       end
main (generic function with 1 method)

julia> CodeTracking.definition(String, @which main())
("function main()\n    foo\n    todB(x) = 10*log10(x)\n    println(\"100x is \$(todB(100)) dB.\")\nend", 1)

Then we get the full body of the method we want.

should definition(Expr, meth) fall back to parsing, when Revise isn't loaded?

When Revise isn't loaded
it seems that definition(String, method) works (to my surprise),
but definition(Expr, method) returns nothing.

Seems like we could have a fallback to try parsing the definition(String, method),
if we were going to return nothing.

julia> using CodeTracking

julia> definition(String, @which abs(1))
("abs(x::Signed) = flipsign(x,x)", 136)

julia> definition(Expr, @which abs(1))

julia> Meta.parse(first(definition(String, @which abs(1))))
:(abs(x::Signed) = begin
          #= none:1 =#
          flipsign(x, x)
      end)

[Docs] Getting Started

CodeTracking [...] provides an interface for obtaining:

the strings and expressions of method definitions

How does CodeTracking do the above?

using CodeTracking, Revise

f = x -> 2 * x^3 + 1

julia> @code_expr f
ERROR: LoadError: expression is not a function call or symbol

julia> @code_string f(1)
ERROR: LoadError: MethodError: no method matching getindex(::Nothing, ::Int64)

julia> print(@code_expr f(1))
nothing 

I did not post on discourse because there are existing discussions and unfortunately I haven't been able to understand them. Can the above be done? If so, an example would be greatly appreciated. I'm opening an issue, so perhaps a word or two could be added to the readme. I won't be offended if you close it. :-)

Comments in these discussions point to CodeTracking, but unfortunately do not show examples of usage:

https://discourse.julialang.org/t/source-code-of-a-function/4499/12

https://discourse.julialang.org/t/printing-the-body-of-a-defined-function/23439/2

https://discourse.julialang.org/t/show-code-macro/46050

Discussion where the plot thickens:

https://discourse.julialang.org/t/get-function-as-expr/51886/5

Computing the end line can error

julia> using JuliaInterpreter, Zygote, CodeTracking

julia> l, back = Zygote.forward(x -> sum(x), ones(10,10,10))
(1000.0, getfield(Zygote, Symbol("##71#72")){typeof((getfield(Main, Symbol("##5#6"))()))}((getfield(Main, Symbol("##5#6"))())))

julia> frame = JuliaInterpreter.enter_call(back, 1f0);

julia> frame, pc = JuliaInterpreter.debug_command(frame, :si);

julia> frame, pc = JuliaInterpreter.debug_command(frame, :si);

julia> frame, pc = JuliaInterpreter.debug_command(frame, :si);

julia> m = frame.framecode.scope; # showing this will crash Julia due to https://github.com/FluxML/Zygote.jl/issues/134

julia> m.name
:Pullback

julia> CodeTracking.definition(String, m)
ERROR: Base.Meta.ParseError("unexpected \"end\"")
Stacktrace:
 [1] #parse#1(::Bool, ::Bool, ::Bool, ::Function, ::String, ::Int64) at .\meta.jl:129
 [2] parse(::String, ::Int64) at .\meta.jl:121
 [3] definition(::Type{String}, ::Method) at C:\Users\Kristoffer\Debugging\Debugger\dev\CodeTracking\src\CodeTracking.jl:209
 [4] top-level scope at none:0

src is at this point equal to

ignore(T) = all(T -> T <: Type, T.parameters)

@generated function _forward(ctx::Context, f, args...)
  T = Tuple{f,args...}
  ignore(T) && return :(f(args...), Pullback{$T}(()))
  g = try _lookup_grad(T) catch e e end
  !(g isa Tuple) && return :(f(args...), Pullback{$T}((f,)))
  meta, forw, _ = g
  argnames!(meta, Symbol("#self#"), :ctx, :f, :args)
  forw = varargs!(meta, forw, 3)
  forw = slots!(pis!(inlineable!(forw)))
  return IRTools.update!(meta, forw)
end

@generated function (j::Pullback{T})(Δ) where T
  ignore(T) && return :nothing
  g = try _lookup_grad(T)
  catch e
    rethrow(CompileError(T,e))
  end
  if g == nothing
    Δ == Nothing && return :nothing
    return :(error("Non-differentiable function $(j.t[1])"))
  end
  meta, _, back = g
  resize!(back.argtypes, 2)
  argnames!(meta, Symbol("#self#"), )
  back = slots!(inlineable!(back))
  return IRTools.update!(meta, back)
end

with istart = 477 and then:

julia> print(s[477:end])
end

@generated function (j::Pullback{T})(Δ) where T
  ignore(T) && return :nothing
  g = try _
...

starts with an end.

Signatures for standalone files

If I have a file myfile.jl and use Revise to include it via includet shouldn't it be possible to extract the signatures from that file? It seems that only package files are supported now.

definition(String, method) #265s

julia> f(x) = print("hello")
f (generic function with 1 method)

julia> m = @which f(2)
f(x) in Main at REPL[3]:1

julia> CodeTracking.definition(String, m)
("f(x) = print(\"hello\")", 1)

julia> f(x) = print("world")
f (generic function with 1 method)

julia> m = @which f(2)
f(x) in Main at REPL[6]:1

julia> CodeTracking.definition(String, m)
("f(x) = print(\"hello\")", 1)

julia> f(2)
world

internal error when trying to annotate_source

When doing some code revising (resulting in line numbers being shifted or the wrong file being parsed), it seems that CodeTracking can run off the end of the file and crash:

istart = findnext(eol, src, istart) + 1

shell> cd test

julia> descend(Base.structdiff, Tuple{NamedTuple, NamedTuple})                                                                                                                                                                                                                                                                        
ERROR: MethodError: no method matching +(::Nothing, ::Int64)                                                                                                       
                                                                                                                                                                   
Closest candidates are:                                                          
  +(::Any, ::Any, !Matched::Any, !Matched::Any...)                                                                                                                                                                                                                                                                                    
   @ Base operators.jl:587                                                                                                                                         
  +(!Matched::Base.CoreLogging.LogLevel, ::Integer)                                                                                                                
   @ Base logging.jl:131                                                                                                                                           
  +(!Matched::BigInt, ::Union{Int16, Int32, Int64, Int8})                                                                                                          
   @ Base gmp.jl:555                                                                                                                                               
  ...                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                      
Stacktrace:                                                                                                                                                                                                                                                                                                                           
  [1] definition(::Type{String}, method::Method)                                 
    @ CodeTracking ~/.julia/packages/CodeTracking/8vwW2/src/CodeTracking.jl:252                                                                                                                                                                                                                                                       
  [2] tsn_and_mappings(m::Method, src::Core.CodeInfo, rt::Any; warn::Bool, strip_macros::Bool, kwargs::@Kwargs{})                                                                                                                                                                                                                     
    @ TypedSyntax ~/.julia/dev/Cthulhu/TypedSyntax/src/node.jl:34                                                                                                  
  [3] get_typed_sourcetext(mi::Core.MethodInstance, src::Core.CodeInfo, rt::Any; warn::Bool)                                                                       
    @ Cthulhu ~/.julia/dev/Cthulhu/src/reflection.jl:354                                                                                                                                                                                                                                                                              
  [4] find_callsites(interp::Cthulhu.CthulhuInterpreter, CI::Core.CodeInfo, stmt_infos::Vector{Core.Compiler.CallInfo}, mi::Core.MethodInstance, slottypes::Vector{Any}, optimize::Bool, annotate_source::Bool)                                                                                                                       
    @ Cthulhu ~/.julia/dev/Cthulhu/src/reflection.jl:36                                                                                                            
  [5]                                                                                                                                                                                                                                                                                                                                 
    @ Cthulhu ~/.julia/dev/Cthulhu/src/Cthulhu.jl:465                                                                                                                                                                                                                                                                                 
  [6] _descend(term::REPL.Terminals.TTYTerminal, interp::Cthulhu.CthulhuInterpreter, mi::Core.MethodInstance; kwargs::@Kwargs{iswarn::Bool})                                                                                                                                                                                          
    @ Cthulhu ~/.julia/dev/Cthulhu/src/Cthulhu.jl:739                                                                                                              
  [7] _descend(term::REPL.Terminals.TTYTerminal, args::Any; interp::Core.Compiler.NativeInterpreter, kwargs::@Kwargs{iswarn::Bool})                                                                                                                                                                                                   
    @ Cthulhu ~/.julia/dev/Cthulhu/src/Cthulhu.jl:755                                                                                                              
  [8] __descend_with_error_handling(args::Any; terminal::Any, kwargs...)                                                                                           
    @ Cthulhu ~/.julia/dev/Cthulhu/src/Cthulhu.jl:211                                                                                                              
  [9] _descend_with_error_handling(f::Any, argtypes::Any; kwargs::@Kwargs{iswarn::Bool})
    @ Cthulhu ~/.julia/dev/Cthulhu/src/Cthulhu.jl:200
 [10] descend_code_typed(::Any, ::Vararg{Any}; kwargs::@Kwargs{})
    @ Cthulhu ~/.julia/dev/Cthulhu/src/Cthulhu.jl:158
 [11] top-level scope
    @ REPL[2]:1
Some type information was truncated. Use `show(err)` to see complete types.

Syntax highlighted code source

Hey, this seemed nifty so I made a variant macro @code_src that prints a syntax-highlighted @code_string using OhMyREPL.
I have no idea if optional dependencies are easy, or if this of interest, but just in case here's what I have:

using Tokenize, Crayons, InteractiveUtils
"""
    @code_src
Use *InteractiveUtils* to try to find the source for the function or macro call,
and colourise it using *Tokenize*, *Crayons*, and *OhMyREPL*.

```julia-repl
julia> @code_src 2^2
^(x::T, p::T) where {T<:Integer} = power_by_squaring(x,p)
```
"""
macro code_src(expr...)
    codestr = InteractiveUtils.gen_call_with_extracted_types_and_kwargs(CodeTracking, :code_string, expr)
    quote
        tokens = collect(Tokenize.tokenize($codestr))
        crayons = Vector{Crayon}(undef, length(tokens))
        fill!(crayons, Crayon())
        OhMyREPL.Passes.SyntaxHighlighter.SYNTAX_HIGHLIGHTER_SETTINGS(crayons, tokens, 0)
        print(stdout, "  ")
        OhMyREPL.untokenize_with_ANSI(crayons, tokens, 2)
    end
end

Nothing ever ends up in _pkgfiles ?

I'm not sure if I need to be on a special branch/tag of Revise.jl for this to work?

julia> using CodeTracking

julia> using DataStructures

julia> CodeTracking._pkgfiles
Dict{Base.PkgId,PkgFiles} with 0 entries

julia> using Revise

julia> Revise.revis
revise             revise_dir_queued   revise_file_now     revise_file_queued  revision_queue
julia> Revise.revise()

julia> CodeTracking._pkgfiles
Dict{Base.PkgId,PkgFiles} with 0 entries

julia> using DataFrames

julia> CodeTracking._pkgfiles
Dict{Base.PkgId,PkgFiles} with 0 entries

julia> DataFrame()
0×0 DataFrame


julia> CodeTracking._pkgfiles
Dict{Base.PkgId,PkgFiles} with 0 entries

Status:


(MagneticReadHead) pkg> st
Project MagneticReadHead v0.1.0
    Status `~/JuliaEnvs/MagneticReadHead/Project.toml`
  [7057c7e9] + Cassette v0.2.2+ [`dev/Cassette`]
  [da1fd8a2] + CodeTracking v0.1.0 [`dev/CodeTracking`]
  [1914dd2f] + MacroTools v0.4.5
  [78c3b35d] + Mocking v0.5.7
  [bac558e1] + OrderedCollections v1.0.2
  [ee283ea6] + Rebugger v0.2.2
  [295af30f] + Revise v1.0.2
    Status `~/JuliaEnvs/MagneticReadHead/Manifest.toml`
  [7057c7e9] + Cassette v0.2.2+ [`dev/Cassette`]
  [da1fd8a2] + CodeTracking v0.1.0 [`dev/CodeTracking`]
  [1914dd2f] + MacroTools v0.4.5
  [78c3b35d] + Mocking v0.5.7
  [bac558e1] + OrderedCollections v1.0.2
  [ee283ea6] + Rebugger v0.2.2
  [295af30f] + Revise v1.0.2

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

Should we rename `definition` to `code_expr` and `code_string`?

It belatedly occurs to me that definition is essentially part of the code_* family, returning the source AST and string associated with a method definition. Should we change the name, or do people prefer the current naming?

EDIT: to be completely clear, it might look like this:

julia> using CodeTracking, Revise

julia> using Colors

julia> c = RGB(1,0,0)
RGB{N0f8}(1.0,0.0,0.0)

julia> @code_expr red(c)
:(red(c::AbstractRGB) = begin
          #= /home/tim/.julia/packages/ColorTypes/TvB32/src/traits.jl:14 =#
          c.r
      end)

macro esc function cannot be used with @code_exp or @code_string

Guys, I use CodeTracking in my code, and I want to use @code_expr in a macro, but esc seems like a big problem here:

# file Foo.jl
using CodeTracking
macro foo(ex)
  quote
    bar = @code_string $esc(ex)
    # do something with bar
  end
end
myfn() = 1
@foo myfn()

ERROR: LoadError: expression is not a function call, or is too complex for @code_string to analyze; break it down to simpler parts if possible. In some cases, you may want to use Meta.@lower.

I believe it tries to @code_string the interpolate symbol.
Or in case: @code_string esc($ex)
It just returns the code_string of Escaping, what is not the intended behaviour, what we should have.

Failure on method with effect annotations

On Julia 1.9,

julia> using CodeTracking

julia> m = only(methods(Base._gcd))
_gcd(a::T, b::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}
     @ Base intfuncs.jl:64

julia> definition(String, m)

(i.e., nothing)

Boundserror in definition(String, method) when the method has inline macros without arguments

julia> function g()
           Base.@_inline_meta
           print("hello")
       end
g (generic function with 2 methods)

julia> CodeTracking.definition(String, @which g())
ERROR: BoundsError: attempt to access 2-element Array{Any,1} at index [3]
Stacktrace:
 [1] getindex(::Array{Any,1}, ::Int64) at .\array.jl:729
 [2] isfuncexpr(::Expr) at C:\Users\Kristoffer\.julia\environments\v1.1\dev\CodeTracking\src\utils.jl:5
 [3] definition(::Type{String}, ::Method) at C:\Users\Kristoffer\.julia\environments\v1.1\dev\CodeTracking\src\CodeTracking.jl:204
 [4] top-level scope at none:0

A call to `abspath` missing somewhere?

julia> includet("../myfile.jl")

julia> m = first(methods(f))
f(x) in Main at C:\Users\Kristoffer\Debugging\myfile.jl:2

julia> signatures_at(String(m.file), m.line)
1-element Array{Any,1}:
 Tuple{typeof(f),Any}

julia> signatures_at("../myfile.jl", 2)
ERROR: ../myfile.jl not found, perhaps the package is not loaded
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] signatures_at(::String, ::Int64) at C:\Users\Kristoffer\.julia\packages\CodeTracking\r4JU4\src\CodeTracking.jl:142
 [3] top-level scope at none:0

julia> abspath("../myfile.jl") == String(m.file)
true

Regression in finding source code for keyword argument functions

The following file (lookupA.jl)

func_1st_nokwarg() = true
func_2nd_kwarg(; kw=2) = true

using CodeTracking
m = Main.:(var"#func_2nd_kwarg#1")
CodeTracking.definition(String, first(methods(m)))

when run shows:

julia> include("lookupA.jl")
("func_1st_nokwarg() = true", 1)

instead of func_2nd_kwarg(; kw=2) = true.

Reverting 980dd13 makes it (correctly) show

julia> include("lookupA.jl")
("func_2nd_kwarg(; kw=2) = true", 2)

This also seems to have caused

using Debugger
H = rand(2,2)
@enter FiniteDiff.finite_difference_jacobian!(H, (out, in) -> begin; out[1] = in[2]*in[3]; out[2] = in[1]*in[3]; out[3]=in[1]*in[2]; end, ones(3))

to fail to give correct data.

Ref JuliaDebug/Debugger.jl#268

support return types

Hi,

Thank you for this package to obtain a function's source code!

I am just starting out experimenting and found, that return type annotations might not be supported:

function f(x::Int64)
  return x*x + x
end
function g(x::Int64)::Int64
  return x*x + x
end
println("f = $(definition(String,which(f,(Int64,))))")
println("g = $(definition(String,which(g,(Int64,))))")

gives

f = ("function f(x::Int64)\n  return x*x + x\nend", 901)
g = nothing

would it be difficult to add this functionality?

kind regards,
Christian

Make pkgfile(Main) work?

I think it would make some sense to have Main work as it if were a package.

So I could run include("foo.jl") (or includet) in the REPL,
and then pkgfiles(Main) would list "foo.jl" as one of my source files.

Though it might be a bit odd as its basedir would have to be nothing, I guess,
and all paths would need to be absolute?

I want a way to list all files that have been included into Main.

"definition" cannot read function defined within Jupyter notebook

When calling definition in a Jupyter notebook:

f(x,y) = x^2 + y^2
definition(String, @which(f(Int,Int)))

I get the following error:
SystemError: opening file "In[52]": No such file or directory

Seems to just not handled reading from the notebook's In because it works when reading from functions defined purely in the REPL.

`@code_expr` fails (?) with a method from IntervalArithmetic.jl

julia> using CodeTracking, Revise

julia> using IntervalArithmetic

julia> Revise.track(IntervalArithmetic)

julia> @code_expr +(0.1, 0.2, RoundDown)
┌ Warning: Tuple{typeof(+), T, T, RoundingMode} where T<:AbstractFloat was not found
└ @ Revise ~/.julia/packages/Revise/9lZUE/src/packagedef.jl:1083

Revise tracking CodeTracking.jl itself

It seems that Revise doesn't have the ability track changes functions to in CodeTracking.jl. Is this is fundamental limitation of the architecture due to CodeTracking relying on Revise, or something that could be enabled?

$ julia --startup-file=no

(@v1.10) pkg> activate --temp
(jl_3k5Dzh) pkg> dev CodeTracking
(jl_3k5Dzh) pkg> add Revise

julia> using Revise
julia> using CodeTracking
julia> CodeTracking.is_func_expr(:(a -> a + 1))
true

julia> @edit CodeTracking.is_func_expr(:(a -> a + 1))
# add a println statement to `is_func_expr`

julia> CodeTracking.is_func_expr(:(a -> a + 1))
true
# changes not reflected

CodeTracking.definition(String, m) can throw

julia> using LinearAlgebra

julia> B = Hermitian(hcat([one(BigFloat) + im]));

julia> m = @which cholesky(B)
cholesky(A::Union{StridedArray{T, 2} where T, Union{Hermitian{var"#s821",var"#s820"}, Hermitian{Complex{var"#s821"},var"#s820"}, Symmetric{var"#s821",var"#s820"}} where var"#s820"<:(StridedArray{T, 2} where T) where var"#s821"<:Real}) in LinearAlgebra at /home/kc/Applications/julia-1.5.1/share/julia/stdlib/v1.5/LinearAlgebra/src/cholesky.jl:344

julia> using CodeTracking

julia> CodeTracking.definition(String, m)
ERROR: Base.Meta.ParseError("extra token \")\" after end of expression")
Stacktrace:
 [1] parse(::String, ::Int64; greedy::Bool, raise::Bool, depwarn::Bool) at ./meta.jl:184
 [2] parse(::String, ::Int64) at ./meta.jl:176
 [3] definition(::Type{String}, ::Method) at /home/kc/.julia/packages/CodeTracking/g51CV/src/CodeTracking.jl:212
 [4] top-level scope at REPL[11]:1

Tests fail on windows

Line endings:

CodeTracking.jl: Test Failed at C:\Users\Kristoffer\.julia\environments\v1.1\dev\CodeTracking\test\runtests.jl:24
  Expression: src == "function f1(x, y)\n    # A comment\n    return x + y\nend\n"
   Evaluated: "function f1(x, y)\r\n    # A comment\r\n    return x + y\r\nend\r\n" == "function f1(x, y)\n    # A comment\n    return x + y\nend\n"

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

MethodError when using function defined via macro

Hi

Reproducible example:

macro makefn(name)
    name = Symbol(name)
    quote
        $(esc(name))() = print("hello")
    end
end

julia> @makefn f
f (generic function with 1 method)

julia> f()
hello

julia> @code_string f()
ERROR: MethodError: no method matching +(::Nothing, ::Int64)
Closest candidates are:
  +(::Any, ::Any, ::Any, ::Any...) at /usr/share/julia/base/operators.jl:655
  +(::T, ::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at /usr/share/julia/base/int.jl:87
  +(::Base.TwicePrecision, ::Number) at /usr/share/julia/base/twiceprecision.jl:279
  ...
Stacktrace:
 [1] definition(#unused#::Type{String}, method::Method)
   @ CodeTracking ~/.julia/packages/CodeTracking/zfc7r/src/CodeTracking.jl:230
 [2] code_string(f::Function, t::Type)
   @ CodeTracking ~/.julia/packages/CodeTracking/zfc7r/src/CodeTracking.jl:299
 [3] top-level scope
   @ REPL[5]:1

Revise should track macro-created functions defined in the REPL

macro makefn(name)
    name = Symbol(name)
    quote
        $(esc(name))() = print("hello")
    end
end

julia> @makefn f
f (generic function with 1 method)

julia> f()
hello

julia> @code_string f()
"        \$(esc(name))() = println(\"hello\")"

In an ideal world, @code_string f() would return something like f() = println(\"hello\")".

Multiple definitions; good design for limiting allocation?

Revise can get desynchronized from the file state by the following sequence of user actions:

  1. cut the definition of foo(x::T) from fileA.jl, but do not yet save the file
  2. paste definition into fileB.jl (another file in the same package) and save the file
  3. trigger a revision
  4. save fileA.jl and trigger a revision

After item 3, there are technically duplicated definitions of the method (and if the user had quit and rebuilt the package at this point s/he would get a precompile error). After item 4, Revise will helpfully delete the method, not realizing it had been duplicated. So attempts to use the method will henceforth fail.

I used to think of this as "user error" (the user should have saved fileA.jl before triggering the first revision) but lately I've come to think it should be treated as a Revise bug. The question is how best to implement a fix. It seems the most thorough fix would involve making method_info a sig=>list-of-definitions dict rather than a sig=>definition dict. If list-of-definitions is a Vector, though, there are going to be an awful lot of vectors with just one element. Any thoughts on a better way to do this?

Invalid Index: ...

I think this is fairly new (I'm on the latest released versions for all three packages):

julia> using Revise, JuliaInterpreter, CodeTracking

julia> bps=breakpoint(min);

julia> bps[1].framecode.scope
min(x::BigFloat, y::BigFloat) in Base.MPFR at mpfr.jl:693

julia> whereis(bps[1].framecode.scope)
[ Info: tracking Base
┌ Warning: error evaluating in module Base.MPFR: for (fJ, fC) = ((:si, :Clong), (:ui, :Culong))
│     #= /buildworker/worker/package_linux64/build/base/mpfr.jl:197 =#
│     #= /buildworker/worker/package_linux64/build/base/mpfr.jl:197 =# @eval begin
│             #= /buildworker/worker/package_linux64/build/base/mpfr.jl:198 =#
│             function BigFloat(x::$(Expr(:$, :fC)), r::MPFRRoundingMode=ROUNDING_MODE[]; precision::Integer=DEFAULT_PRECISION[])
│                 #= /buildworker/worker/package_linux64/build/base/mpfr.jl:199 =#
│                 z = BigFloat(; precision=precision)
│                 #= /buildworker/worker/package_linux64/build/base/mpfr.jl:200 =#
│                 ccall(($(Expr(:$, :(string(:mpfr_set_, fJ)))), :libmpfr), Int32, (Ref{BigFloat}, $(Expr(:$, :fC)), MPFRRoundingMode), z, x, r)
│                 #= /buildworker/worker/package_linux64/build/base/mpfr.jl:201 =#
│                 return z
│             end
│         end
│ end
│   exception =
│    ArgumentError: invalid index: breakpoint(min(x::T, y::T) where T<:Real in Base at promotion.jl:415, line 415) of type BreakpointRef
│    Stacktrace:
│     [1] to_index(::BreakpointRef) at ./indices.jl:270
│     [2] to_index(::Array{Any,1}, ::BreakpointRef) at ./indices.jl:247
│     [3] to_indices at ./indices.jl:298 [inlined]
│     [4] to_indices at ./indices.jl:295 [inlined]
│     [5] getindex at ./abstractarray.jl:927 [inlined]
│     [6] pc_expr(::Core.CodeInfo, ::BreakpointRef) at /home/pfitzseb/.julia/packages/JuliaInterpreter/62ZyY/src/utils.jl:55
│     [7] pc_expr(::JuliaInterpreter.FrameCode, ::BreakpointRef) at /home/pfitzseb/.julia/packages/JuliaInterpreter/62ZyY/src/utils.jl:56
│     [8] pc_expr(::Frame, ::BreakpointRef) at /home/pfitzseb/.julia/packages/JuliaInterpreter/62ZyY/src/utils.jl:57
│     [9] #methods_by_execution!#10(::Bool, ::Function, ::Any, ::Revise.CodeTrackingMethodInfo, ::Dict{Module,Array{Expr,1}}, ::Frame) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/lowered.jl:169
│     [10] (::getfield(Revise, Symbol("#kw##methods_by_execution!")))(::NamedTuple{(:define,),Tuple{Bool}}, ::typeof(Revise.methods_by_execution!), ::Function, ::Revise.CodeTrackingMethodInfo, ::Dict{Module,Array{Expr,1}}, ::Frame) at ./none:0
│     [11] #methods_by_execution!#9(::Bool, ::Function, ::Any, ::Revise.CodeTrackingMethodInfo, ::Dict{Module,Array{Expr,1}}, ::Module, ::Expr) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/lowered.jl:40
│     [12] #methods_by_execution! at ./none:0 [inlined]
│     [13] #eval_with_signatures#55(::Bool, ::Function, ::Module, ::Expr) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/Revise.jl:343
│     [14] #eval_with_signatures at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/Revise.jl:0 [inlined]
│     [15] instantiate_sigs!(::OrderedCollections.OrderedDict{Module,OrderedCollections.OrderedDict{Revise.RelocatableExpr,Union{Nothing, Array{Any,1}}}}) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/Revise.jl:351
│     [16] maybe_parse_from_cache!(::Revise.PkgData, ::String) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/pkgs.jl:224
│     [17] get_def at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/Revise.jl:742 [inlined]
│     [18] #get_def#62(::Set{Tuple{Revise.PkgData,String}}, ::Function, ::Method) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/Revise.jl:711
│     [19] get_def(::Method) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/Revise.jl:693
│     [20] #invokelatest#1 at ./essentials.jl:742 [inlined]
│     [21] invokelatest at ./essentials.jl:741 [inlined]
│     [22] whereis(::Method) at /home/pfitzseb/.julia/packages/CodeTracking/PPhte/src/CodeTracking.jl:55
│     [23] top-level scope at none:0
│     [24] eval(::Module, ::Any) at ./boot.jl:328
│     [25] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:85
│     [26] run_backend(::REPL.REPLBackend) at /home/pfitzseb/.julia/packages/Revise/ZblJU/src/Revise.jl:846
│     [27] (::getfield(Revise, Symbol("##66#68")){REPL.REPLBackend})() at ./task.jl:259
└ @ Revise ~/.julia/packages/Revise/ZblJU/src/lowered.jl:43
[...]
("/home/pfitzseb/Documents/julia-1.1.0/share/julia/base/mpfr.jl", 693)

Complete error message is here.

Moved macros give wrong location info

If a method definition occurs inside a macro, or if one uses whereis(lineinfo, method) and lineinfo corresponds to lines added by a macro, CodeTracking relies on the information in the lineinfo annotation to know anything about the macro. If the macro definition has moved due to edits, this info is going to be wrong.

`@code_expr` doesn't seem to work in vscode

I saw a note that @code_expr may fail in some cases. One particularly frustrating one is that it doesn't seem to work inside VSCode "Julia REPL" processes.

It's pretty easy to just Meta.parse, so it's not a big deal

definition throws error on (Pair{A, B})(a, b)

The method is defined as

@eval function (P::Type{Pair{A, B}})(@nospecialize(a), @nospecialize(b)) where {A, B}
    @inline
    return $(Expr(:new, :P, :(convert(A, a)), :(convert(B, b))))
end
(CodeTracking) julia> method = @which Pair{Int, Int}(1, 1)
Pair{A, B}(a, b) where {A, B}
     @ Base ~/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/base/Base.jl:138

(CodeTracking) julia> definition(String, method)
ERROR: AssertionError: isexpr(wheretyp, :<:)
Stacktrace:
 [1] is_func_expr(ex::Any, meth::Method)
   @ CodeTracking /media/rag/NVME/Code/CodeTracking.jl/src/utils.jl:103
 [2] definition(#unused#::Type{String}, method::Method)
   @ CodeTracking /media/rag/NVME/Code/CodeTracking.jl/src/CodeTracking.jl:261
 [3] top-level scope
   @ REPL[208]:1

It seems like the problem is the fname is :(P::Type{Pair{A, B}}) at

fname = callex.args[1]
but is :Type at
if !(isa(fname, Symbol) && is_gensym(fname)) && !isexpr(fname, :$)

strip_gensym(meth.name) is :Pair.

Should PkgFiles be exported? Or should pkgfiles be merged with PkgFiles ?

Right now there exists both PkgFiles and pkgfiles.

pkgfiles seems to be a more convient way to construct a PkgFiles object,
since it has a method that takes a Module, and one that takes string of the module's name.

The fact that PkgFiles does not have these methods suggests it's constructors are ment to be internal, and thus it should not be exported.
If this is not true then I fail to see the value in distinguishing pkgfiles from PkgFiles
and I would thus sugggest that these methods should be added and pkgfiles should be removed.

(I also think either way they should both be in a file called /src/pkgfiles.jl, because data.jl is not informative)

Is it possible to use this in a non interactive way?

This:

using InteractiveUtils
using Revise
using CodeTracking

function func()
return 123
end

println(@code_string func())
println(@code_expr func())

outputs:

function func()
    return 123
end
nothing

if I execute it using julia script.jl

How can I make this work?

Internal error when trying to find string definition of `Base.get_preferences`

julia> m = which(Base.get_preferences, Tuple{Base.UUID})
get_preferences(uuid::Union{Nothing, Base.UUID}) in Base at loading.jl:2006

julia> CodeTracking.definition(String, m)
ERROR: MethodError: no method matching +(::Nothing, ::Int64)
Closest candidates are:
  +(::Any, ::Any, ::Any, ::Any...) at operators.jl:591
  +(::T, ::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at int.jl:87
  +(::T, ::Integer) where T<:AbstractChar at char.jl:237
  ...
Stacktrace:
 [1] definition(#unused#::Type{String}, method::Method)
   @ CodeTracking ~/.julia/packages/CodeTracking/rwfOQ/src/CodeTracking.jl:230
 [2] top-level scope
   @ REPL[4]:1

an option to force parsing if required

It seems that Revise has some heuristics to bail out of some of its get_def calls,
if it looks like it is going to be too much work.
https://github.com/timholy/Revise.jl/blob/d160813bed054d80fe29854a3178fb726b3e808b/src/Revise.jl#L813
A sensible thing for Revise to do.

but when i call definition, I wish I could tell it:
that "No, I really do want the definition, no matter how much it costs",

This seems fiddly to reproduce.
Sometimes it shows up when I do

using CodeTracking
using Revise: Revise  # Revise must be loaded for CodeTracking to work

definition(@which(1.0+2))

and some times it doesn't and that just works.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.