Progress meter for long-running operations in Julia
Within julia, execute
using Pkg; Pkg.add("ProgressMeter")This works for functions that process things in loops or with map/pmap/reduce:
using Distributed
using ProgressMeter
@showprogress dt=1 desc="Computing..." for i in 1:50
sleep(0.1)
end
@showprogress pmap(1:10) do x
sleep(0.1)
x^2
end
@showprogress reduce(1:10) do x, y
sleep(0.1)
x + y
endThe first incantation will use a minimum update interval of 1 second, and show the ETA and final duration. If your computation runs so quickly that it never needs to show progress, no extraneous output will be displayed.
The @showprogress macro wraps a for loop, comprehension, @distributed for loop, or
map/pmap/reduce as long as the object being iterated over implements the length
method and will handle continue correctly.
using Distributed
using ProgressMeter
@showprogress @distributed for i in 1:10
sleep(0.1)
end
result = @showprogress desc="Computing..." @distributed (+) for i in 1:10
sleep(0.1)
i^2
endIn the case of a @distributed for loop without a reducer, an @sync is implied.
You can also control progress updates and reports manually:
function my_long_running_function(filenames::Array)
n = length(filenames)
p = Progress(n; dt=1.0) # minimum update interval: 1 second
for f in filenames
# Here's where you do all the hard, slow work
next!(p)
end
endFor tasks such as reading file data where the progress increment varies between iterations,
you can use update!:
using ProgressMeter
function readFileLines(fileName::String)
file = open(fileName,"r")
seekend(file)
fileSize = position(file)
seekstart(file)
p = Progress(fileSize; dt=1.0) # minimum update interval: 1 second
while !eof(file)
line = readline(file)
# Here's where you do all the hard, slow work
update!(p, position(file))
end
endThe core methods Progress(), ProgressThresh(), ProgressUnknown(), and their updaters
are also thread-safe, so can be used with Threads.@threads, Threads.@spawn etc.:
using ProgressMeter
p = Progress(10)
Threads.@threads for i in 1:10
sleep(2*rand())
next!(p)
end
finish!(p)and the @showprogress macro also works
using ProgressMeter
@showprogress Threads.@threads for i in 1:10
sleep(2*rand())
endusing ProgressMeter
n = 10
p = Progress(n)
tasks = Vector{Task}(undef, n)
for i in 1:n
tasks[i] = Threads.@spawn begin
sleep(2*rand())
next!(p)
end
end
wait.(tasks)
finish!(p)Optionally, a description string can be specified which will be prepended to the output,
and a progress meter M characters long can be shown. E.g.
p = Progress(n; desc="Computing initial pass...")will yield
Computing initial pass...53%|███████████████████████████ | ETA: 0:09:02
in a manner similar to python-progressbar.
Also, other properties can be modified through keywords. The glyphs used in the bar may be
specified by passing a BarGlyphs object as the keyword argument barglyphs. The BarGlyphs
constructor can either take 5 characters as arguments or a single 5 character string. E.g.
p = Progress(n; dt=0.5, barglyphs=BarGlyphs("[=> ]"), barlen=50, color=:yellow)will yield
Progress: 53%[==========================> ] ETA: 0:09:02
It is possible to give a vector of characters that acts like a transition between the empty character and the fully filled character. For example, definining the progress bar as:
p = Progress(n; dt=0.5,
barglyphs=BarGlyphs('|','█', ['▁' ,'▂' ,'▃' ,'▄' ,'▅' ,'▆', '▇'],' ','|',),
barlen=10)might show the progress bar as:
Progress: 34%|███▃ | ETA: 0:00:02
where the last bar is not yet fully filled.
Some tasks only terminate when some criterion is satisfied, for
example to achieve convergence within a specified tolerance. In such
circumstances, you can use the ProgressThresh type:
prog = ProgressThresh(1e-5; desc="Minimizing:")
for val in exp10.(range(2, stop=-6, length=20))
update!(prog, val)
sleep(0.1)
endSome tasks only terminate when some non-deterministic criterion is satisfied. In such
circumstances, you can use the ProgressUnknown type:
prog = ProgressUnknown(desc="Titles read:")
for val in ["a" , "b", "c", "d"]
next!(prog)
if val == "c"
finish!(prog)
break
end
sleep(0.1)
endThis will display the number of calls to next! until finish! is called.
If your counter does not monotonically increases, you can also set the counter by hand.
prog = ProgressUnknown(desc="Total length of characters read:")
total_length_characters = 0
for val in ["aaa" , "bb", "c", "d"]
global total_length_characters += length(val)
update!(prog, total_length_characters)
if val == "c"
finish!(prog)
break
end
sleep(0.5)
endAlternatively, you can display a "spinning ball" symbol
by passing spinner=true to the ProgressUnknown constructor.
prog = ProgressUnknown(desc="Working hard:", spinner=true)
while true
next!(prog)
rand(1:2*10^8) == 1 && break
end
finish!(prog)By default, finish! changes the spinner to a ✓, but you can
use a different character by passing a spinner keyword
to finish!, e.g. passing spinner='✗' on a failure condition:
let found=false
prog = ProgressUnknown(desc="Searching for the Answer:", spinner=true)
for tries in 1:10^8
next!(prog)
if rand(1:2*10^8) == 42
found=true
break
end
end
finish!(prog, spinner = found ? '✓' : '✗')
endIn fact, you can completely customize the spinner character
by passing a string (or array of characters) to animate as a spinner
argument to next!:
prog = ProgressUnknown(desc="Burning the midnight oil:", spinner=true)
while true
next!(prog, spinner="🌑🌒🌓🌔🌕🌖🌗🌘")
rand(1:10^8) == 0xB00 && break
end
finish!(prog)(Other interesting-looking spinners include "⌜⌝⌟⌞", "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏", "🕐🕑🕒🕓🕔🕕🕖🕗🕘🕙🕚🕛", "▖▘▝▗'", and "▁▂▃▄▅▆▇█".)
You can also print and update information related to the computation by using
the showvalues keyword. The following example displays the iteration counter
and the value of a dummy variable x below the progress meter:
x,n = 1,10
p = Progress(n)
for iter in 1:10
x *= 2
sleep(0.5)
next!(p; showvalues = [("iteration count",iter), ("x",x)])
endIn the above example, the data passed to showvalues is evaluated even if the progress bar is not updated.
To avoid this unnecessary computation and reduce the overhead,
you can alternatively pass a zero-argument function as a callback to the showvalues keyword.
x,n = 1,10
p = Progress(n)
generate_showvalues(iter, x) = () -> [("iteration count",iter), ("x",x)]
for iter in 1:10
x *= 2
sleep(0.5)
# unlike `showvalues=generate_showvalues(iter, x)()`, this version only evaluate the function when necessary
next!(p; showvalues = generate_showvalues(iter, x))
endYou can include an average per-iteration duration in your progress meter
by setting the optional keyword argument showspeed=true
when constructing a Progress, ProgressUnknown, or ProgressThresh.
x,n = 1,10
p = Progress(n; showspeed=true)
for iter in 1:10
x *= 2
sleep(0.5)
next!(p; showvalues = [(:iter,iter), (:x,x)])
endwill yield something like:
Progress: XX%|███████████████████████████ | ETA: XX:YY:ZZ (12.34 s/it)
instead of
Progress: XX%|███████████████████████████ | ETA: XX:YY:ZZ
In addition to the showspeed optional keyword argument,
all the progress meters also support the optional enabled keyword argument.
You can use this to conditionally disable a progress bar in cases where you want less verbose output
or are using another progress bar to track progress in looping over a function that itself uses a progress bar.
function my_awesome_slow_loop(n::Integer; show_progress=true)
p = Progress(n; enabled=show_progress)
for i in 1:n
sleep(0.1)
next!(p)
end
end
const SHOW_PROGRESS_BARS = parse(Bool, get(ENV, "PROGRESS_BARS", "true"))
m = 100
# let environment variable disable outer loop progress bar
p = Progress(m; enabled=SHOW_PROGRESS_BARS)
for i in 1:m
# disable inner loop progress bar since we are tracking progress in the outer loop
my_awesome_slow_loop(i; show_progress=false)
next!(p)
endJupyter notebooks/lab does not allow one to overwrite only parts of the output of cell.
In releases up through 1.2, progress bars are printed repeatedly to the output.
Starting with release xx, by default Jupyter clears the output of a cell, but this will
remove all output from the cell. You can restore previous behavior by calling
ProgressMeter.ijulia_behavior(:append). You can enable it again by calling ProgressMeter.ijulia_behavior(:clear),
which will also disable the warning message.
For remote parallelization, when multiple processes or tasks are being used for a computation,
the workers should communicate back to a single task for displaying the progress bar. This
can be accomplished with a RemoteChannel:
using ProgressMeter
using Distributed
n_steps = 20
p = Progress(n_steps)
channel = RemoteChannel(() -> Channel{Bool}(), 1)
# introduce a long-running dummy task to all workers
@everywhere long_task() = sum([ 1/x for x in 1:100_000_000 ])
@time long_task() # a single execution is about 0.3 seconds
@sync begin # start two tasks which will be synced in the very end
# the first task updates the progress bar
@async while take!(channel)
next!(p)
end
# the second task does the computation
@async begin
@distributed (+) for i in 1:n_steps
long_task()
put!(channel, true) # trigger a progress bar update
i^2
end
put!(channel, false) # this tells the printing task to finish
end
endHere, returning some number i^2 and reducing it somehow (+)
is necessary to make the distribution happen.
More control over the progress bar in a map function can be achieved with the progress_map
and progress_pmap functions. The keyword argument progress can be used to supply a custom progress meter.
p = Progress(10, barglyphs=BarGlyphs("[=> ]"))
progress_map(1:10, progress=p) do x
sleep(0.1)
x^2
endIt possible to disable the progress meter when the use is optional.
x, n = 1, 10
p = Progress(n; enabled = false)
for iter in 1:10
x *= 2
sleep(0.5)
next!(p)
endIn cases where the output is text output such as CI or in an HPC scheduler, the helper function
is_logging can be used to disable automatically.
is_logging(io) = isa(io, Base.TTY) == false || (get(ENV, "CI", nothing) == "true")
p = Progress(n; output = stderr, enabled = !is_logging(stderr))To add support for other functions, ProgressMeter.ncalls must be defined,
where ncalls_map, ncalls_broadcast, ncalls_broadcast! or ncalls_reduce can help
For example, with tmap from ThreadTools.jl:
using ThreadTools, ProgressMeter
ProgressMeter.ncalls(::typeof(tmap), ::Function, args...) = ProgressMeter.ncalls_map(args...)
ProgressMeter.ncalls(::typeof(tmap), ::Function, ::Int, args...) = ProgressMeter.ncalls_map(args...)
@showprogress tmap(abs2, 1:10^5)
@showprogress tmap(abs2, 4, 1:10^5)When developing or debugging ProgressMeter it is convenient to redirect the output to another terminal window such that it does not interfer with the Julia REPL window you are using.
On Linux/macOS you can find the file name corresponding to the other terminal by using the
tty command. This file can be opened
and passed as the output keyword argument to the
Progress/ProgressThresh/ProgressUnknown constructors.
Run tty from the other terminal window (the window where we want output to show up):
$ tty
/dev/pts/3
From the Julia REPL, open the file for writing, wrap in IOContext (to enable color), and
pass to the Progress constructor:
io = open("/dev/pts/3", "w")
ioc = IOContext(io, :color => true)
prog = Progress(10; output = ioc)Output from prog will now print in the other terminal window when executing update!,
next!, etc.
Thanks to Alan Bahm, Andrew Burroughs, and Jim Garrison for major enhancements to this package.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![femtocleaner[bot] avatar](https://avatars.githubusercontent.com/in/4123?v=4 "femtocleaner[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent