go-hep / hep Goto Github PK

From @sbinet on December 15, 2016 9:47

Copied from original issue: go-hep/fwk#42

From @sbinet on September 13, 2014 8:49

make sure consistency is upheld in face of double-put:

store := ctx.Store()
data := 42
// store data
err := store.Put("my-data", &data)

// oops, another time, under the same key
err = store.Put("my-data", &data)

otherwise, this could wreak havoc in the sync'ing of achan-s.

Copied from original issue: go-hep/fwk#39

right now, root-srv will fail to plot foo.root/dir1/dir2/h1.

From @sbinet on December 12, 2016 16:23

add a helper package to read/write "archives" of YODA files, so one can read/write a complete collection of hbook values (H1D, H2D, Scatter2D, ...) from/to a single YODA file.

Copied from original issue: go-hep/hbook#26

From @sbinet on August 18, 2014 14:48

do we need a virtual (r/w) filesystem ?

• check out possible candidates: https://github.com/sourcegraph/rwvfs

Copied from original issue: go-hep/fwk#13

From @sbinet on August 15, 2014 9:18

blog.golang.org has a nice write-up on contexts:

• http://blog.golang.org/context
• http://godoc.org/code.google.com/p/go.net/context

especially the part about hierarchical contexts.

would this be a better fit for fwk.Ctxt:

• to handle updates to a collection ?
• to handle aborts ? (e.g. abort a task because of a timeout, abort run because no more events in input file, etc...)

Copied from original issue: go-hep/fwk#11

From @sbinet on August 19, 2014 7:2

investigate whether usage of reflect on a Task.Process code would be possible to automagically extract the data flow.

Copied from original issue: go-hep/fwk#15

From @sbinet on December 15, 2016 9:47

Copied from original issue: go-hep/fwk#43

From @sbinet on December 7, 2016 16:9

Copied from original issue: go-hep/hbook#10

From @sbinet on August 21, 2014 11:56

using job configurations from issue go-hep/fwk#25 , it should be possible to submit jobs to a web server with simple GET/POST queries.

Copied from original issue: go-hep/fwk#26

From @sbinet on October 20, 2015 11:48

to provide an NlnN combinatorics (instead of O(N^3)) fastjet should implement Delaunay triangulations and voronoi tesselation (as is done in the C++ version of fastjet via the CGAL library)

https://en.wikipedia.org/wiki/Delaunay_triangulation
https://en.wikipedia.org/wiki/Voronoi_diagram
https://github.com/sbinet-staging/poly2tri.golang/blob/master/testbed/main.go

Copied from original issue: go-hep/fastjet#1

From @sbinet on August 20, 2014 14:55

people might cringe a bit at configuring a fwk job in go.
even if the github.com/go-hep/fwk/job.Job interface can be considered quite nice.

also, providing a "Turing-complete" language as a configuration language introduces problems of its own (one has to debug the program which creates the program one wants to run.)
c.f. all the issues with e.g. the Athena/Gaudi python joboptions.

Using a declarative language like hcl might fit the bill.

Copied from original issue: go-hep/fwk#24

From @sbinet on July 7, 2016 15:41

we may need to support (a subset of?) SQL queries.

parsers & SQL generators

• https://github.com/influxdata/influxdb/tree/master/influxql
• https://github.com/benbjohnson/sql-parser/blob/master/parser.go
• https://github.com/knq/xo
• http://0value.com/A-PEG-parser-generator-for-Go

ROW-wise / COL-wise

• http://db.csail.mit.edu/pubs/abadi-column-stores.pdf

Copied from original issue: go-hep/csvutil#1

From @sbinet on August 22, 2014 16:19

gaudi had issues with initialization of components (especially cycles in services/tools: ToolA -> ToolB -> ToolC -> ... -> ToolA)

the lack of resource dependencies declaration (component A needs component B) prevented to efficiently discover such issues.
it also prevented to initialize components in parallel.

investigate whether fwk couldn't do better here.

Copied from original issue: go-hep/fwk#28

From @sbinet on December 15, 2016 9:50

Copied from original issue: go-hep/pawgo#4

From @sbinet on December 15, 2016 9:48

Copied from original issue: go-hep/fwk#44

From @sbinet on August 19, 2014 13:34

Copied from original issue: go-hep/fwk#19

From @sbinet on December 9, 2016 18:5

Copied from original issue: go-hep/hbook#24

From @sbinet on December 7, 2016 16:9

Copied from original issue: go-hep/hbook#11

From @sbinet on December 15, 2016 9:49

Copied from original issue: go-hep/pawgo#3

From @sbinet on September 16, 2014 9:44

• define a format friendlier to concurrent I/O
• inspiration from Linux/BSD VFS ?
  • ZFS
  • Ext4
  • BTRFS
• inspiration from HDF5 ?

Requirements:

• allow streaming from http ?
• optimize for main usage: write once, read many
• optimize for read-speed
• use cases: user analysis, bulk reconstruction
• MPI I/O compliant ?

BTRFS

http://git.kernel.org/cgit/linux/kernel/git/mason/btrfs-progs.git/tree/
https://btrfs.wiki.kernel.org/index.php/Btrfs_design
http://www.cs.ucdavis.edu/~pandey/Teaching/ECS150/Lects/07fs.pdf

Ext4

https://ext4.wiki.kernel.org/index.php/Main_Page
http://foss.in/2006/cfp/slides/ext4-foss.pdf
https://ols.fedoraproject.org/OLS/Reprints-2008/kumar-reprint.pdf
https://github.com/torvalds/linux/tree/master/fs/ext4
http://atrey.karlin.mff.cuni.cz/~jack/papers/lk2009-ext4-btrfs.pdf

ZFS

http://maczfs.googlecode.com/files/ZFSOnDiskFormat.pdf
https://blogs.oracle.com/bonwick/en_US/entry/zfs_block_allocation
https://blogs.oracle.com/bonwick/entry/space_maps
https://github.com/zfsonlinux/zfs/

XFS

http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf

HDF5

http://www.hdfgroup.org/HDF5/doc/H5.format.html

ROOT

https://github.com/root-mirror/root/tree/master/io/doc/TFile
http://root.cern.ch/root/htmldoc/TFile.html
ftp://root.cern.ch/root/doc/11InputOutput.pdf

A4

https://github.com/a4/a4
http://arxiv.org/abs/1208.1600

blosc

http://www.blosc.org/
http://www.blosc.org/blosc-in-depth.html

FUSE

http://bazil.org/fuse/

LTFS

http://en.m.wikipedia.org/wiki/Linear_Tape_File_System

XZ

http://tukaani.org/xz/xz-file-format.txt

ACQt

https://indico.in2p3.fr/event/10875/session/7/material/0/3.pdf

Compression

Brotli

http://www.ietf.org/id/draft-alakuijala-brotli-07.txt
https://github.com/google/brotli/

Copied from original issue: go-hep/rio#1

From @sbinet on September 16, 2014 13:12

• investigate whether encoding/decoding of types could be stored into rio files
• assuming the proposal for Go plugins API goes in, it would be possible to dynamically load code. => just write the package name ?
• fwd/bwd compat of types. write the AST or some serialized version of the data type for versionning (instead of just the package name + data type name) (AKA, schema evolution)

Copied from original issue: go-hep/rio#2

From @sbinet on December 15, 2016 9:49

Copied from original issue: go-hep/pawgo#2

From @sbinet on November 16, 2016 9:15

right now, opening a CSV connection is done like so:

func foo() {
	// Open the CSV file as a database table.
	db, err := csvdriver.Conn{
		File:    "../data/iris.csv",
		Comment: '#',
		Comma:   ',',
	}.Open()
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	// Start a database transaction.
	tx, err := db.Begin()
	if err != nil {
		log.Fatal(err)
	}
	defer tx.Commit()

	// Define a SQL query that we will execute against the CSV file.
	query := "SELECT var3, var4, var5 FROM csv WHERE var5 = \"Iris-versicolor\""

	// etc...
}

this makes a query rather cumbersome to read.

it should be possible to open a CSV connection with an optional Columns []string field:

// Open the CSV file as a database table.
db, err := csvdriver.Conn{
	File:    "../data/iris.csv",
	Comment: '#',
	Comma:   ',',
	Columns: []string{"sepal_length", "sepal_width", "petal_length", "petal_width", "species"},
}.Open()

such that the query reads:

// Define a SQL query that we will execute against the CSV file.
query := `SELECT petal_length, petal_width, species FROM csv WHERE species = "Iris-versicolor"`

What do you think @dwhitena ?

Copied from original issue: go-hep/csvutil#2

From @sbinet on September 12, 2014 18:27

detect and prevent components from applying r/w updates of items in the data-store

Copied from original issue: go-hep/fwk#38

From @sbinet on August 13, 2014 6:45

fwk should support the ability to abort the processing of an event when it reaches a given timeout

Copied from original issue: go-hep/fwk#5

to reduce the amount of dependencies, consider using "gonum/graph/formats/dot" instead of "awalterschulze/gographviz".

From @sbinet on August 14, 2014 19:21

investigate usage of sync.Pool

• usage in event-based banks/arenas
• interplay with I/O
• interplay with garbage collection

Copied from original issue: go-hep/fwk#9

From @sbinet on December 7, 2016 16:33

Copied from original issue: go-hep/hbook#16

Scanner does not properly seal user from implementation details of Tree.
if the underlying file pointer is not at the correct position (ie pos=0), Scanner will fail.

see: c58ed3e.

right now, when a user uploads a file, root-srv gobbles it up all in memory and keeps it there (until the user session expires).

• introduce a local mode where the file is directly open from the disk
• when remote or on appengine, store the file on some temporary storage (a tmp directory keyed on the user session name)

it should be possible to combine multiple rootio.Trees (with the same layout) from multiple ROOT files into one rootio.Tree.

right, now, to get at a rootio.Tree, one does:

func main() {
	f, err := rootio.Open("testdata/chain.1.root")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	obj, err := f.Get("tree")
	if err != nil {
		log.Fatal(err)
	}
	tree := obj.(rootio.Tree)
	fmt.Printf("entries= %v\n", tree.Entries())
}

the rootio.Tree interface is:

// Tree is a collection of branches of data.
type Tree interface {
	Named
	Entries() int64
	TotBytes() int64
	ZipBytes() int64
	Branch(name string) Branch
	Branches() []Branch
	Leaves() []Leaf

	getFile() *File
	loadEntry(i int64) error
}

it should be possible to have a type that implements the rootio.Tree interface but logically concatenates multiple rootio.Trees together.
Something like the io.MultiReader function: https://godoc.org/io#MultiReader

Initially, one could just require the user to pass already created/retrieved rootio.Tree to a, e.g., Chain function:

// Chain returns a new Tree that is the logical concatenation of all the input Trees.
func Chain(trees ...Tree) Tree { ... }

one should think about resource ownership:

• who should close the ROOT file containing a Tree when that Tree has been completely read ?
• how to integrate the chain type with the Scanners ?
• is it better to have a Chain function that takes a tree name and a list of file names ? (so resource ownership is clearer). it's less composable...
• perhaps introduce a ChainScanner that opens and closes files as needed when each Tree has been consumed ?

this would allow to use rootio.Tree from hbook/ntup.Ntuple (and use hbook/ntup.Ntuple.Scan() and friends)

From @sbinet on December 7, 2016 16:33

Copied from original issue: go-hep/hbook#15

From @sbinet on September 16, 2014 13:12

• investigate whether encoding/decoding of types could be stored into rio files
• assuming the proposal for Go plugins API goes in, it would be possible to dynamically load code. => just write the package name ?
• fwd/bwd compat of types. write the AST or some serialized version of the data type for versionning (instead of just the package name + data type name) (AKA, schema evolution)

Copied from original issue: go-hep/rio#2

From @ebusato on October 12, 2016 15:10

Hi,

I'd like to retrieve the position on the x axis of the y maximum of the histogram within a certain range provided by the user. Right now you have the Max() method that returns the y maximum. What do you think would be the best option to implement what I need :

1. Change the Max() method by adding two parameters to specify the range and one return value which is the maximum position on the x axis
2. Create a new method and let Max() untouched in order to not impact clients.

?

Let me know and I'll send a PR.

Emmanuel

Copied from original issue: go-hep/hbook#8

From @sbinet on August 14, 2014 19:17

investigate feasibility of a multi-process, multi-node fwk.Task process.

• rpc
• golang circuit

Copied from original issue: go-hep/fwk#8

From @sbinet on August 13, 2014 6:46

fwk should (perhaps) support task-level timeout (and then abort event) when the task takes too much time to process.

Copied from original issue: go-hep/fwk#6

From @sbinet on December 7, 2016 16:11

add support for reading/writing from/to AIDA files

Copied from original issue: go-hep/hbook#12

From @sbinet on September 11, 2014 13:27

concrete components may have properties.
should we provide a way to expose/enforce/attach documentation for them ?
godoc is useless for them.

Copied from original issue: go-hep/fwk#37

From @sbinet on September 10, 2014 15:32

it should be possible to dump a fwk.App configuration into:

a human-readable format
a machine-readable format (serialization/gob/json/...)
for later re-use, comparison or regression-tests purposes.

it should also work for user-defined types. (currently, the JSON based format only works for builtins)

Copied from original issue: go-hep/fwk#34

From @sbinet on December 15, 2016 9:49

Copied from original issue: go-hep/pawgo#1

From @sbinet on December 9, 2016 18:5

Copied from original issue: go-hep/hbook#25

@aphecetche wants them

From @sbinet on December 7, 2016 16:35

Copied from original issue: go-hep/hbook#17

From @sbinet on August 19, 2014 6:58

boltdb has a nice interface for transactions (updates and read-only):
https://github.com/boltdb/bolt
https://github.com/boltdb/bolt#transactions

• evaluate API impact on fwk
• evaluate performances impact

Copied from original issue: go-hep/fwk#14