gher-uliege / diva-workshops Goto Github PK

View Code? Open in Web Editor NEW

15.0 6.0 9.0 126.36 MB

Code, data and instructions for the Diva workshops

Home Page: https://gher-uliege.github.io/Diva-Workshops/

Jupyter Notebook 97.41% Julia 0.12% HTML 2.44% Shell 0.03%

diva spatial-analysis data-analysis interpolation oceanography ocean-sciences seadatanet seadatacloud

diva-workshops's People

Contributors

Stargazers

Watchers

Forkers

swatelet malek-belgacem msdevana celinednl stanleesocca yanchunhefork thomas-moore-creative mr-nobody-dey marie59

diva-workshops's Issues

Analysis notebook

In the previous version, we start making analysis with notebook 10-processing-quality-check.ipynb which is about quality check.

I wonder if we should change the order, i.e., present QC based on the analysis later in the course.

Broken links to download data

Some notebooks (or at least 20-Turtles-Density-map.ipynb) still contains URLs related to B2drop, which don't seem to work anymore.

Solution: replace with DoX link (easy)

Future: let's hope DoX will be maintained because it is used now in several of our projects to stored:

data files (so we can download them within a Julia session)
small files to perform unit tests

Alternative grid for interpolation (non lat-lon)

Hi all,

I am working with datasets in polar regions and would like to interpolate to a grid which is different from the lat-lon grid. I see that in the notebook the grid coordinates are defined as
lonr = lon_beg:dx:lon_end
Is there a way, instead, to give in input the grid points in another coordinate system? The locations would always be expressed in lat and lon, but they would not be organised in a regular lat-lon grid. It would be something similar to interpolate to scattered points.

Thank you very much in advance for your help.
Francesca

Warning: "Preconditioned conjugate gradients method did not converge" in notebook 90-full-analysis

Hi,
I am using DIVAnd for the first time and I have been working on an analysis based on the notebook 90-full-analysis, I got the following warning in output:

@time dbinfo = diva3d((lonr,latr,depthr,TS), (obslon,obslat,obsdepth,obstime), obsval, len, epsilon2, filename2,varname, bathname=bath_name, mask = mask, fitcorrlen = false, niter_e = 2, );

Output:
[ Info: Creating netCDF file /~/data/julia/fromjulia/gam_sal_processed.nc
[ Info: Time step 1 / 4
[ Info: scaled correlation length (min,max) in dimension 1: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 2: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 3: (25.0, 25.0)
[ Info: number of windows: 6
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
[ Info: number of windows: 6
[ Info: Time step 2 / 4
[ Info: scaled correlation length (min,max) in dimension 1: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 2: (100000.0, 100000.0)
[ Info: scaled correlation length (min,max) in dimension 3: (25.0, 25.0)
[ Info: number of windows: 6
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
┌ Warning: Preconditioned conjugate gradients method did not converge
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_solve.jl:83
[ Info: number of windows: 6
[ Info: Time step 3 / 4
┌ Warning: no data at 3
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/diva.jl:432

I'd like to understand the reason and how to solve this warning.
The resulting netcdf file looks good but I'd prefer to correct this before going further.

I hope you can help me to better understand how it works.

Cheers,
Javiera

Problem to write the observations to the netCDF

Hello,

in 90-full-analysis, the command to write the obs DIVAnd.saveobs(filename,(obslon,obslat,obsdepth,obstime),obsid); returns the following error.

The checkobs((obslon,obslat,obsdepth,obstime),obsval,obsid); seems to provide a correct output:

              minimum and maximum of obs. dimension 1: (11.716666221618652, 20.11)
              minimum and maximum of obs. dimension 2: (39.0, 45.755)
              minimum and maximum of obs. dimension 3: (0.0, 2508.516357421875)
              minimum and maximum of obs. dimension 4: (1911-08-17T09:24:00, 2015-02-10T10:43:00)
                          minimum and maximum of data: (25.0, 39.700001)

Any hint?

Julia Version 1.0.0 (2018-08-08)
DIVAnd v2.1.1+ [~/.julia/dev/DIVAnd]
NCDatasets v0.5.1+ [~/.julia/dev/NCDatasets]

BoundsError: attempt to access 1805275-element Array{Float64,1} at index [Base.LogicalIndex(Bool[true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, 
......
true, true, true, true, true, true, true, true])]

Stacktrace:
 [1] throw_boundserror(::Array{Float64,1}, ::Tuple{Base.LogicalIndex{Int64,BitArray{1}}}) at ./abstractarray.jl:484
 [2] checkbounds at ./abstractarray.jl:449 [inlined]
 [3] _getindex at ./multidimensional.jl:588 [inlined]
 [4] getindex at ./abstractarray.jl:905 [inlined]
 [5] #356 at ./none:0 [inlined]
 [6] iterate at ./generator.jl:47 [inlined]
 [7] collect(::Base.Generator{Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}},getfield(DIVAnd, Symbol("##356#357")){BitArray{1}}}) at ./array.jl:619
 [8] #saveobs#355(::Type, ::DateTime, ::BitArray{1}, ::Symbol, ::Function, ::String, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{String,1}) at /home/ctroupin/.julia/dev/DIVAnd/src/DIVAnd_save.jl:389
 [9] saveobs(::String, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{String,1}) at /home/ctroupin/.julia/dev/DIVAnd/src/DIVAnd_save.jl:387
 [10] top-level scope at In[38]:1

PosDefException in 13-processing-parameter-optimization

With imeth = 1,
this

bestfactorl[imeth+1],bestfactore[imeth+1], cvval,cvvalues, x2Ddata,y2Ddata,cvinter,xi2D,yi2D = DIVAnd_cv(mask,(pm,pn),(xi,yi),(obslon,obslat),obsval.-mean(obsval),len,epsilon2*rdiag,2,3,imeth);

gives a PosDefException error (see below).
Any hint?

PosDefException: matrix is not positive definite; Cholesky factorization failed.

Stacktrace:
 [1] #cholesky!#6 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1284 [inlined]
 [2] #cholesky! at ./none:0 [inlined]
 [3] #cholesky#8(::Float64, ::Bool, ::Nothing, ::typeof(LinearAlgebra.cholesky), ::SuiteSparse.CHOLMOD.Sparse{Float64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1322
 [4] #cholesky#9 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1315 [inlined]
 [5] cholesky at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SuiteSparse/src/cholmod.jl:1434 [inlined]
 [6] factorize!(::CovarIS{Float64,SparseArrays.SparseMatrixCSC{Float64,Int64}}) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/special_matrices.jl:94
 [7] DIVAnd_factorize!(::DIVAnd.DIVAnd_struct{Float64,Int64,2,SparseArrays.SparseMatrixCSC{Float64,Int64}}) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAnd_factorize.jl:37
 [8] #DIVAndrun#267(::Tuple{}, ::Bool, ::Bool, ::Float64, ::Int64, ::Int64, ::Tuple{}, ::Symbol, ::Array{Any,1}, ::Array{Float64,2}, ::Array{Int64,1}, ::Int64, ::typeof(DIVAnd_pc_none), ::DIVAnd.var"#269#271", ::Array{Float64,2}, ::Array{Float64,1}, ::Int64, ::Bool, ::Array{Any,1}, ::Float64, ::Tuple{}, ::Tuple{}, ::Int64, ::Tuple{}, ::Tuple{}, ::Array{Float64,1}, ::Array{Float64,1}, ::typeof(DIVAndrun), ::Type, ::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64}, ::Array{Float64,1}) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAndrun.jl:97
 [9] (::DIVAnd.var"#kw##DIVAndrun")(::NamedTuple{(:alpha, :alphabc),Tuple{Array{Int64,1},Int64}}, ::typeof(DIVAndrun), ::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64}, ::Array{Float64,1}) at ./none:0
 [10] #DIVAnd_cv#308(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(DIVAnd_cv), ::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Int64, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAnd_cv.jl:310
 [11] DIVAnd_cv(::BitArray{2}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,2},Array{Float64,2}}, ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Int64, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64) at /home/ctroupin/.julia/packages/DIVAnd/pXHkM/src/DIVAnd_cv.jl:73
 [12] top-level scope at ./In[12]:5

Getting rid of data files?

Most of the datafiles are stored in dox, it makes sense not to store datafiles, especially the netCDF, within github.

I'm checking if all of them are there.

fithorzlen fails in 12-correlation-length

@time lenxy,infoxy = fithorzlen(x,v,z; distfun = DIVAnd.distfun_m, searchz = z -> (z/4+10));

gives

TypeError: in keyword argument searchz, expected Float64, got var"#5#6"

Stacktrace:
 [1] (::DIVAnd.var"#kw##fithorzlen")(::NamedTuple{(:distfun, :searchz),Tuple{typeof(DIVAnd.distfun_m),var"#5#6"}}, ::typeof(fithorzlen), ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Array{Float64,1}) at ./none:0
 [2] top-level scope at util.jl:155
 [3] top-level scope at In[11]:1

Integrate 5-AdvancedTopics/time-aggregation.ipynb into 18-defining-time-periods.ipynb

These notebooks are quite similar:

https://github.com/gher-ulg/Diva-Workshops/blob/master/notebooks/5-AdvancedTopics/18-defining-time-periods.ipynb

https://github.com/gher-ulg/Diva-Workshops/blob/master/notebooks/5-AdvancedTopics/time-aggregation.ipynb

If I am not mistaken, @ctroupin will discuss them on Monday PM. Would it be useful to combine them?

Notebook clean-up

Are there any notebooks which are no longer be used (replaced e.g. by other functionalities)? If yes, I would propose to delete them.

notebook 15-example-analysis: ArgumentError: invalid base 10 digit 'C'

Julia version: 1.4.0
DIVAnd version: [efc8151c] DIVAnd v2.5.1

I ran the notebooks successfully, until the last command:

divadoxml(filename,varname,project,cdilist,xmlfilename,ignore_errors = ignore_errors)

which returns this error:

ArgumentError: invalid base 10 digit 'C' in "Cruise:WOD05_BG000003"

Stacktrace:
 [1] tryparse_internal(::Type{Int64}, ::SubString{String}, ::Int64, ::Int64, ::Int64, ::Bool) at ./parse.jl:132
 [2] parse(::Type{Int64}, ::SubString{String}; base::Nothing) at ./parse.jl:238
 [3] parse at ./parse.jl:238 [inlined]
 [4] get_originators_from_obsid(::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Base.Iterators.Flatten{Array{Array{String,1},1}}; ignore_errors::Bool) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:328
 [5] getoriginators(::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String; ignore_errors::Bool) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:300
 [6] gettemplatevars(::Array{String,1}, ::String, ::String, ::String; errname::String, WMSlayername::Array{String,1}, previewindex::Int64, basemap::String, additionalcontacts::Array{Any,1}, ignore_errors::Bool) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:827
 [7] divadoxml(::Array{String,1}, ::String, ::String, ::String, ::String; ignore_errors::Bool, previewindex::Int64, basemap::String, additionalvars::Dict{String,Any}, additionalcontacts::Array{Any,1}, WMSlayername::Array{String,1}) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:1015
 [8] divadoxml(::String, ::String, ::String, ::String, ::String; WMSlayername::String, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:ignore_errors,),Tuple{Bool}}}) at /home/ctroupin/.julia/packages/DIVAnd/rJCtr/src/SDNMetadata.jl:1044
 [9] top-level scope at In[34]:1

File download failure

The CI is failing (in all O.S.), at least one reason is that the file download (for example bathymetry) cannot be completed.

Origin of the problem

I guess it's because DoX URL:

This works:

julia> download("https://dox.uliege.be/index.php/s/k0f7FxA7l5FIgu9/download")
"/tmp/jl_onMfMLiNcN"

This fails:

julia> download("https://dox.ulg.ac.be/index.php/s/k0f7FxA7l5FIgu9/download")
ERROR: RequestError: Could not resolve host: dox.ulg.ac.be while requesting https://dox.ulg.ac.be/index.php/s/k0f7FxA7l5FIgu9/download
Stacktrace:
...

Writing a huge number of observations to a netCDF

So I'm trying to write all the Atlantic observations inside the netCDF I've created with DIVAnd:

@show length(obslat);
length(obslat) = 31850848
DIVAnd.saveobs(filename,(obslon,obslat,obsdepth,obstime),obsid);

and the last command results in my browser to be killed.

Probably this part has to be somehow enhanced.

"conversion to pointer not defined for StepRangeLen..." in notebook "90-full-analysis".

The cell

@time dbinfo = diva3d((lonr,latr,depthr,TS),
    (obslon,obslat,obsdepth,obstime), obsval,
    len, epsilon2,
    filename,varname,
    bathname=bathname,
    plotres = plotres,
    mask = mask_edit,
    fitcorrlen = false,
    niter_e = 2,
    ncvarattrib = ncvarattrib,
    ncglobalattrib = ncglobalattrib,
    surfextend = true
    );

yields this issue:

conversion to pointer not defined for StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}

Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] unsafe_convert(#unused#::Type{Ptr{Float64}}, a::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ Base ./pointer.jl:67
  [3] unsafe_convert(#unused#::Type{Ptr{Nothing}}, a::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ Base ./pointer.jl:66
  [4] nc_put_vara(ncid::Int32, varid::Int32, startp::Vector{Float64}, countp::Vector{Int64}, op::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/netcdf_c.jl:796
  [5] nc_put_var(ncid::Int32, varid::Int32, data::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/netcdf_c.jl:712
  [6] setindex!
    @ ~/.julia/packages/NCDatasets/c8XyT/src/variable.jl:342 [inlined]
  [7] setindex!(v::NCDatasets.CFVariable{Float64, 1, NCDatasets.Variable{Float64, 1, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), NTuple{6, Nothing}}}, data::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, indexes::Colon)
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/cfvariable.jl:632
  [8] ncfile(ds::NCDataset{Nothing}, filename::String, xyi::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Vector{Float64}, Vector{DateTime}}, varname::String; ncvarattrib::OrderedDict{String, String}, ncglobalattrib::OrderedDict{String, String}, thresholds::Vector{Tuple{String, Float64}}, deflatelevel::Int64, chunksizes::Vector{Int64}, type_save::Type{Float32}, timeorigin::DateTime, checksum::Symbol, saveindex::Tuple{Colon, Colon, UnitRange{Int64}}, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:climatology_bounds, :relerr), Tuple{Matrix{DateTime}, Bool}}})
    @ DIVAnd ~/ULiege/Tools/DIVAnd.jl/src/DIVAnd_save.jl:322
  [9] (::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), Nothing, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{Vector{Int64}}}, Vector{Float64}, Int64})(ds::NCDataset{Nothing})
    @ DIVAnd ~/ULiege/Tools/DIVAnd.jl/src/diva.jl:376
 [10] NCDataset(::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), Nothing, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{Vector{Int64}}}, Vector{Float64}, Int64}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:220
 [11] NCDataset(::Function, ::String, ::Vararg{String})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:218
 [12] diva3d(xi::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Vector{Float64}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{Vector{Int64}}}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, value::Vector{Float64}, len::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Float64, filename::String, varname::String; datadir::String, bathname::String, bathisglobal::Bool, plotres::typeof(plotres), timeorigin::DateTime, moddim::Vector{Float64}, zlevel::Symbol, ncvarattrib::OrderedDict{String, String}, ncglobalattrib::OrderedDict{String, String}, transform::Tuple{DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}}, distfun::typeof(DIVAnd.distfun_m), mask::BitArray{3}, background::Nothing, background_epsilon2_factor::Nothing, background_lenz::Nothing, background_len::Nothing, background_lenz_factor::Int64, filterbackground::Int64, fitcorrlen::Bool, fithorzcorrlen::Bool, fitvertcorrlen::Bool, fithorz_param::Dict{Any, Any}, fitvert_param::Dict{Any, Any}, memtofit::Int64, overlapfactor::Float64, niter_e::Int64, minfield::Float64, maxfield::Float64, surfextend::Bool, velocity::Tuple{}, stat_per_timeslice::Bool, error_thresholds::Vector{Tuple{String, Float64}}, divamethod::typeof(DIVAndgo), kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DIVAnd ~/ULiege/Tools/DIVAnd.jl/src/diva.jl:375
 [13] top-level scope
    @ ./timing.jl:220 [inlined]
 [14] top-level scope
    @ ./In[41]:0
 [15] eval
    @ ./boot.jl:373 [inlined]
 [16] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1196

Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4

Errors in the notebook 14b-errormaps-demo

The CI reported an error in notebook 14b-errormaps-demo so I tried to run it and got some issues:

In cell [15] (as per GitHub):

ploterr(aerrora)
title("Error using errormap version $method");

gives an error

PyError ($(Expr(:escape, :(ccall(#= /home/ctroupin/.julia/packages/PyCall/ygXW2/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'ValueError'>
ValueError('too many values to unpack (expected 2)')

In cell [16]:

aerror,method = DIVAnd_errormap(mask,(pm,pn),(xi,yi),(obslon,obslat),obsval.-mean(obsval),len,epsilon2,
    s;
    method = :precise,
    Bscale = false)
ploterr(aerror)
title("Error using automatic version $method");

gives the same error. In fact when I try to access the variable aerror I get this:

julia> aerror
'Y': ASCII/Unicode U+0059 (category Lu: Letter, uppercase)

Solutions

I could make it work with those changes below though I'm not sure why it failed?

Add an index [1] to the array:

ploterr(aerrora[1])
title("Error using errormap version $method");

Replace :precise by "precise" in the function call:

aerror,method = DIVAnd_errormap(mask,(pm,pn),(xi,yi),(obslon,obslat),obsval.-mean(obsval),len,epsilon2,
    s;
    method = "precise",
    Bscale = false)

Remove NaN values

Hi !

After I make an analysis, I got some warnings in output:
dbinfo = diva3d((lonr,latr,depthr,TS), (lon,lat,depth,obstime), value, (lenx,leny,lenz2), 0.1 * eps1 ./ diagR, filename,varname, bathname = bathname, bathisglobal = bathisglobal, plotres = plotres2, timeorigin = timeorigin, transform = Anam.loglin(slog), minfield = 0., memtofit = 100, solver = :direct, ) # Add used observations in the netcfd file DIVAnd.saveobs(filename,(lon,lat,depth,obstime),ids,used = dbinfo[:used])

Output:
┌ Info: starting DIVA computations for Autumn
└ @ Main In[20]:12
┌ Info: 2020-01-29T11:35:36.05
└ @ Main In[20]:13
┌ Info: Will write results in /home/george/my_project_dir/DIN_Sel5_LCH_LCV_propuse_Autumn//Water_body_dissolved_inorganic_nitrogen_(DIN)_Autumn.4Danl.nc
└ @ Main In[20]:34
┌ Info: Creating netCDF file
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/diva.jl:297
┌ Info: Time step 1 / 4
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/diva.jl:333
┌ Warning: Observations equal to NaN: 3
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/DIVAnd_obs.jl:45
┌ Info: number of windows: 11
└ @ DIVAnd /home/george/.julia/packages/DIVAnd/BGj1J/src/DIVAndgo.jl:89

So, I was wondering, if it's possible to remove all NaN values from dbinfo ( which is a dictionary) ?
Because, the resulting netcdf file, it's almost full with NaN values.

Cheers !
George

Figures for readme

Horizontal correlation length fitting

In the previous version we were using

fithorzlen(x,v,z, nmean = 500, distbin = collect(0.:0.1:6))

to estimate the correlation length but the optional arguments nmean and distbin are not available anymore in fit.jl, so what would be the equivalent call to fithorzlen?

Or we leave it as:

fithorzlen(x,v,z)

Same for

lenz,infoz = fitvertlen(x,v,z, nmean = 500, distbin = collect([0.:50:400; 500:100:600]))

TEST ISSUE :) Cannot open the notebooks with mac OS

Ideally:

configuration: versions of the tools, OS
packages
copy of the message

Removing notebook 05-DIVAnd-overview.ipynb

I propose we get rid of notebook 05-DIVAnd-overview.ipynb because it doesn't really need anything on the table (part of the content could be added to the README if needed).

I know this will cause a problem in the numbering of the notebooks (no number 5 anymore), but the numbering is already a bit messed up in some of the directories).

Use notebook extensions

Add to the doc/readme how to use the extensions:

https://github.com/ipython-contrib/jupyter_contrib_nbextensions
https://github.com/Jupyter-contrib/jupyter_nbextensions_configurator

so (for ex.) sections are automatically numbered