Git Product home page Git Product logo

jade.jl's Introduction

Documentation Build Status Coverage
Build Status Codecov branch

JADE

JADE is a Julia DOASA Environment. It is a flexible version of DOASA, written in Julia, which utilises stochastic dual dynamic programming to find optimal hydro release policies in electricity markets.

The Electricity Authority hosts a repository of JADE input files. These can be accessed from the EMI website. Any input files in the JADE Github repository are for testing purposes only.

Documentation

A detailed description of the inputs and outputs to JADE and how to use JADE can be found in the JADE user documentation, and the API documentation.

Copyright

JADE has been developed by the Electric Power Optimization Centre at the University of Auckland. JADE is licensed under the Mozilla Public License Version 2.0.

jade.jl's People

Contributors

odow avatar adow031 avatar

Stargazers

 avatar zuoj avatar Zeyu Mao avatar  avatar  avatar Oliver avatar Lea Kapelevich avatar Penghui Guo avatar  avatar

Watchers

 avatar  avatar

Forkers

odow bugn200

jade.jl's Issues

Identify public API surface

We currently export some functions

JADE.jl/src/JADE.jl

Lines 37 to 46 in b705880

export JADEdata,
JADEmodel,
optimize_policy!,
define_JADE_model,
define_JADE_solve_options,
create_JADE_model,
define_JADE_simulation,
simulate,
load_model_parameters,
load_simulation_parameters

the docs list a bunch more

https://github.com/EPOC-NZ/JADE.jl/blob/main/docs/src/api.md

We should clarify what is and what isn't public API, and then tidy up the naming conventions so that we don't have snake_case almost everywhere, but then have gettimeseries or the horrific getinflows_for_historical.

Data file parsing is custom, consider using DataFrames

x-ref #12 (comment)

We essentially have a customized CSV reader:

JADE.jl/src/data.jl

Lines 10 to 67 in b705880

"""
parsefile(f::Function, file::String, trim::Bool = false)
This function reads in a data file and extracts each of the non-comment / non-empty rows,
splits into into a vector using ',' as the delimiter, and then applies some custom processing `f`.
### Required Arugments
`f` custom function that processes a line of the data file.
`file` path of the file.
### Keyword Arguments
`trim` if `true` trailing commas will be discarded.
"""
function parsefile(f::Function, file::String, trim::Bool = false)
open(file, "r") do io
while !eof(io)
items, skip_line = getitems(io, trim)
skip_line && continue
f(items)
end
end
end
"""
getitems(io::IOStream, trim::Bool)
This function takes an `IOStream` for a CSV file and reading the first line, decarding empty / comment rows, and
otherwise splitting into into a vector using ',' as the delimiter. This vector is returned.
### Required Arugments
`io` `IOStream` object for a CSV file.
`trim` if `true` trailing commas will be discarded.
"""
function getitems(io::IOStream, trim::Bool)
line = strip(readline(io))
if (line == "" || first(line) == '%') # comment char
return String[], true
end
# drop trailling comments
line = split(line, '%')[1]
items = strip.(split(line, ","))
if trim
while length(items) > 0 && items[end] == ""
deleteat!(items, length(items))
end
end
if length(items) == 0 || maximum(length.(items)) == 0
return String[], true
end
return items, false
end

evaluate whether this is the best solution.

Historical simulation is untested and broken

Fixed a couple of things locally that I'll make a PR for in a bit. But:

function test_case_1_simulate()
    optimizer = MOI.OptimizerWithAttributes(HiGHS.Optimizer, MOI.Silent() => true)
    data = define_JADE_model("test1");
    data.use_terminal_mwvs = true
    solve_options = define_JADE_solve_options("test1");
    model = create_JADE_model(data, optimizer);
    optimize_policy!(model, solve_options; print_level = 0);

    simulation = define_JADE_simulation("test1");
    simulation.sim_type = :historical
    @test_throws(
        ErrorException("Invalid settings found. See REPL for details."),
        JADE.simulate(model, simulation),
    )
    simulation.sim_years = [2008]
    @test_throws(
        ErrorException("Invalid settings found. See REPL for details."),
        JADE.simulate(model, simulation),
    )
    simulation.replications = 1
    results = JADE.simulate(model, simulation)
end


ERROR: KeyError: key 45 not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:482 [inlined]
  [2] getindex
    @ ~/.julia/packages/SDDP/OoTfR/src/user_interface.jl:516 [inlined]
  [3] _simulate(model::SDDP.PolicyGraph{Int64}, variables::Vector{Symbol}; sampling_scheme::SDDP.Historical{Int64, Dict{Symbol, Float64}}, custom_recorders::Dict{Symbol, Function}, duality_handler::Nothing, skip_undefined_variables::Bool, incoming_state::Dict{Symbol, Float64})
    @ SDDP ~/.julia/packages/SDDP/OoTfR/src/algorithm.jl:1099
  [4] #208
    @ ~/.julia/packages/SDDP/OoTfR/src/plugins/parallel_schemes.jl:50 [inlined]
  [5] iterate
    @ ./generator.jl:47 [inlined]
  [6] _collect(c::UnitRange{Int64}, itr::Base.Generator{UnitRange{Int64}, SDDP.var"#208#209"{Base.Iterators.Pairs{Symbol, Any, NTuple{5, Symbol}, NamedTuple{(:sampling_scheme, :custom_recorders, :duality_handler, :skip_undefined_variables, :incoming_state), Tuple{SDDP.Historical{Int64, Dict{Symbol, Float64}}, Dict{Symbol, Function}, Nothing, Bool, Dict{Symbol, Float64}}}}, SDDP.PolicyGraph{Int64}, Vector{Symbol}}}, #unused#::Base.EltypeUnknown, isz::Base.HasShape{1})
    @ Base ./array.jl:695
  [7] collect_similar
    @ ./array.jl:606 [inlined]
  [8] map
    @ ./abstractarray.jl:2294 [inlined]
  [9] #_simulate#207
    @ ~/.julia/packages/SDDP/OoTfR/src/plugins/parallel_schemes.jl:49 [inlined]
 [10] #simulate#82
    @ ~/.julia/packages/SDDP/OoTfR/src/algorithm.jl:1274 [inlined]
 [11] simulate(JADEmodel::JADE.JADEModel, parameters::JADE.JADESimulation)
    @ JADE ~/.julia/dev/JADE/src/simulate.jl:279
 [12] top-level scope
    @ REPL[41]:1

Audit time.jl

Once upon a time, I remember writing this: https://github.com/EPOC-NZ/JADE.jl/blob/main/src/time.jl.

It's a little too cute use of the Julia type system, hiding complexity that makes the code in other parts harder to understand. I think at some point Tony or Lea found a bug in it.

I wonder if we could simplify things, and/or use Dates.jl instead.

inputs for a 2023 run

Hi Team,
I have started to play with the model and am trying to make a 2023 run.
How is the weekly split between peak/offpeak/shoulder defined ("hours_per_block.csv") ?
Is there a 2023 input dataset available somewhere?
Regards,
Gislain

Remove DataFrames.jl

Only usage is

JADE.jl/src/time.jl

Lines 117 to 129 in b705880

"""
For a data frame containing columns named WEEK and YEAR, return only the subset
of data that falls within a relevant time range.
"""
function cleandata(data::DataFrame, t1::TimePoint, t2::TimePoint)
if !(:WEEK in names(data) && :YEAR in names(data))
error("Data should contain columns named WEEK and YEAR.")
else
relevant =
map((x) -> between(t1, t2, x), map(TimePoint, data[:, :YEAR], data[:, :WEEK]))
return data[relevant.==true, :]
end
end

and that method is never called.

See also #8

Remove InSampleMonteCarlo2

Now available in SDDP v0.4.7

struct InSampleMonteCarlo2 <: SDDP.AbstractSamplingScheme
max_depth::Int
initial_stage::Int
terminate_on_cycle::Bool
terminate_on_dummy_leaf::Bool
rollout_limit::Function
end
"""
function InSampleMonteCarlo2(;
max_depth::Int = 0,
initial_stage::Int = 1,
terminate_on_cycle::Bool = false,
terminate_on_dummy_leaf::Bool = true,
rollout_limit::Function = i -> typemax(Int),
)
This function creates an instance of the `InSampleMonteCarlo2` `sampling_scheme`.
This is used for Monte Carlo simulations that may start from an arbitrary stage
(`initial_stage`).
### Keyword Arguments
`max_depth` is the maximum number of stages to simulate.
`initial_stage` is the stage number of the initial stage.
`terminate_on_cycle` if `true` the simulation will stop if it returns to a previously visited node.
`terminate_on_dummy_leaf` if `true` the simulation will may stop if it the probabilities of the noises don't sum to 1.
`rollout_limit` is a function that allows the number of simulated stages to vary with the training iteration number.
"""
function InSampleMonteCarlo2(;
max_depth::Int = 0,
initial_stage::Int = 1,
terminate_on_cycle::Bool = false,
terminate_on_dummy_leaf::Bool = true,
rollout_limit::Function = i -> typemax(Int),
)
if !terminate_on_cycle && !terminate_on_dummy_leaf && max_depth == 0
error(
"terminate_on_cycle and terminate_on_dummy_leaf cannot both be " *
"false when max_depth=0.",
)
end
new_rollout = let i = 0
() -> (i += 1; rollout_limit(i))
end
return InSampleMonteCarlo2(
max_depth,
initial_stage,
terminate_on_cycle,
terminate_on_dummy_leaf,
new_rollout,
)
end
"""
function SDDP.sample_scenario(
graph::SDDP.PolicyGraph{T},
sampling_scheme::InSampleMonteCarlo2,
) where {T}
This function is a version of the SDDP.jl `SDDP.sample_scenario` function which
uses the `InSampleMonteCarlo2` `sampling_scheme`. This enables a simulation to start
from an arbitrary stage (`initial_stage`).
### Required Arguments
`graph` is an SDDP.jl `PolicyGraph`.
`sampling_scheme` is an instance of `InSampleMonteCarlo2` sampling scheme.
"""
function SDDP.sample_scenario(
graph::SDDP.PolicyGraph{T},
sampling_scheme::InSampleMonteCarlo2,
) where {T}
max_depth = min(sampling_scheme.max_depth, sampling_scheme.rollout_limit())
# Storage for our scenario. Each tuple is (node_index, noise.term).
scenario_path = Tuple{T,Any}[]
# We only use visited_nodes if terminate_on_cycle=true. Just initialize
# anyway.
visited_nodes = Set{T}()
node_index = sampling_scheme.initial_stage
while true
node = graph[node_index]
noise_terms = node.noise_terms
children = node.children
noise = SDDP.sample_noise(noise_terms)
push!(scenario_path, (node_index, noise))
# Termination conditions:
if length(children) == 0
# 1. Our node has no children, i.e., we are at a leaf node.
return scenario_path, false
elseif sampling_scheme.terminate_on_cycle && node_index in visited_nodes
# 2. terminate_on_cycle = true and we have detected a cycle.
return scenario_path, true
elseif 0 < sampling_scheme.max_depth <= length(scenario_path)
# 3. max_depth > 0 and we have explored max_depth number of nodes.
return scenario_path, false
elseif sampling_scheme.terminate_on_dummy_leaf &&
rand() < 1 - sum(child.probability for child in children)
# 4. we sample a "dummy" leaf node in the next step due to the
# probability of the child nodes summing to less than one.
return scenario_path, false
end
# We only need to store a list of visited nodes if we want to terminate
# due to the presence of a cycle.
if sampling_scheme.terminate_on_cycle
push!(visited_nodes, node_index)
end
# Sample a new node to transition to.
node_index = SDDP.sample_noise(children)::T
end
# Throw an error because we should never end up here.
return error("Internal SDDP error: something went wrong sampling a scenario.")
end

x-ref odow/SDDP.jl#535

Excel file

Hi,
The documentation mentions an excel file "runJADE.xlsm". Where is this file available?
Regards,
Gislain

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.