sciprogcentre / dataforge-core Goto Github PK

Code:

    val x = emptyList<Double>()
    val y = emptyList<Double>()
    val plot = Plotly.plot2D {
        trace(x, y) {
            name = "for a single trace in graph its name would be hidden"
        }
        layout {
            title = "Graph name"
            xaxis {
                title = "x axis"
            }
            yaxis {
                title = "y axis"
            }
        }
    }

Error:

Exception in thread "main" java.lang.IllegalArgumentException: Can't create list value from empty list
	at hep.dataforge.values.ListValue.<init>(Value.kt:171)
	at hep.dataforge.values.Value$Companion.of(Value.kt:59)
	at hep.dataforge.meta.MutableValueDelegate$transform$1.invoke(Delegates.kt:187)
	at hep.dataforge.meta.MutableValueDelegate$transform$1.invoke(Delegates.kt:169)
	at hep.dataforge.meta.ReadWriteDelegateWrapper.setValue(Delegates.kt:371)
	at scientifik.plotly.models.Trace.setX(Trace.kt)
	at scientifik.plotly.models.Trace$Companion.build(Trace.kt:54)
	at scientifik.plotly.Plot2D.trace(Plot2D.kt:26)
	at scientifik.plotly.Plot2DKt.trace(Plot2D.kt:46)
	at StaticPlotKt$main$plot$1.invoke(staticPlot.kt:16)
	at StaticPlotKt$main$plot$1.invoke(staticPlot.kt)
	at scientifik.plotly.Plotly.plot2D(Plotly.kt:15)
	at StaticPlotKt.main(staticPlot.kt:15)
	at StaticPlotKt.main(staticPlot.kt)

Originally posted by @Zelenyy in SciProgCentre/plotly.kt#11 (comment)

Consider redesign scheme to use inline class

Add output helper that consists only of output side effect (does not transform data) and has IOPlugin as a receiver.

Also add logger handler for input and output data. Logger helper could also be done via non-transforming action.

Add krangl bindings and custom io via IOFormat

We do not expect custom properties for them. It will allow to add extensions like boolean for better descriptor builders.

Allow enums in meta values to be resolved by custom field instead of names. It could be done, by introducing an interface with one field and inheriting enums from those interfaces.

Buffer API and implementation for file and streaming data.

It is probably possible to add values accessors as TaskBuilder extensions so one could define those in the root of task builder, automatically add it to descriptor and provide a type-safe way to access.

Need to remove custom modifications and re-deploy io artifact.

https://github.com/Him188/yamlkt

Move update method from extensions to the base class or create Config child with overrideable method to make it possible to batch process tree changes instead of doing them one by one.

https://github.com/mipt-npm/dataforge-core/blob/5265c0e5abe2dd26a64fc90f91ff862eaa30b669/dataforge-io/src/commonMain/kotlin/hep/dataforge/io/Binary.kt#L34

uses UInt while Binary.size is ULong. The length probably should be the same.

The Scheme class allows wrapping mutable and read-only dynamic Meta trees and adding a statically typed layer to them.
It is used quite extensively in Plotly.kt to allow observation of properties and dynamic changes.

Currently, Scheme is mutable by default and if used on read-only meta, it creates a mutable layer on top of the read-only base. The idea is to automatically generate a read-only version of mutable Scheme via compiler plugin and use it when SchemeSpec.read operation is used. The same plugin could be used to generate a descriptor from Scheme (#35).

The result could be probably achieved with KSP.

Probably need to add a cycle check to meta graph.

• Descriptor API

• Descriptor generation from meta

• Description generation from reflection

• Code generation of Specifications from descriptor

Reference issue: https://youtrack.jetbrains.com/issue/KT-37503

Describe and implement rules with which one Meta (sample) could "match" another Meta (template) without being equal.

The simplest idea is that sample must have all values in template and those values should be equal. This simple conventions works like in operator so A in B + B in A guarantees equality. Though it won't be possible to introduce more complex range rules.

And maybe descriptors from schema.

https://github.com/mipt-npm/dataforge-core/blob/dev/dataforge-meta/src/commonMain/kotlin/hep/dataforge/descriptors/ItemDescriptor.kt

Implement a meta transformation, applicable both for static meta and Configuration with a set of rules. For Configuration it should create dynamic meta, mirroring all changes.

Meta validation for specifications could look like this:

val i by int{it >0 && it <10}

Make Config open and allow to add override rules to Config::replaceItem or those rules to constructor. Rules could be used to validate incomming values and/or mutate other values.

Either add default meta to Specific itself with additional set of extensions duplicating ones form Configuration or add a base to configuration and remove Styled.

• strip indices in SNS
• enum serialization names

The idea is that workspace could automatically substitute or decorate task dependencies in order to achieve better flexibility.

We need to think a lot about design, since it could break predictability on different workspaces.

to is getting confused with pairs and brings a lot of mistakes. Add new syntax and deprecate to.

Change Java->Kotlin etc

Wrapper for Row with statically typed delegates. According to #38

Dataforge output had an unnecessary dependency on dataforge-io, which forced to use kotlinx-io build, which in turn required dev repository. It is fixed in dev, but io still requires dev repository to be loaded. It must be shown in the documentation.

Table implementation for SQL-based databases.

Proposal add function for work with non-uniform data, similar pipe and join . As first example, for two types:

Should look like this:

val task = task("myTask"){
  val dependency by otherTask // adds depdendency
  model{
    ...
  }
  transform{
    val node: DataNode<R>  = dependency()
  }
}

If read Tagless Envelope from ByteArray (see code), wrong calculation offset for data location.

    fun load(name: String): Envelope? {
        val entry = zipFile.getEntry(name)
        if (entry == null || entry.isDirectory) {
            return null
        }
        val bytes = zipFile.getInputStream(entry).readAllBytes()
        val binary = bytes.asBinary()
        val text = bytes.inputStream().bufferedReader().readText() // Смотрю содержимое в дебагерре
        return formatPeeker(binary)?.let {
            val partialEnvelope = binary.read {
                it.run { readPartial() }
            }
            val offset: Int = partialEnvelope.dataOffset.toInt()
            val size: Int = partialEnvelope.dataSize?.toInt() ?: (binary.size.toInt() - offset)
            val cut_bytes = bytes.sliceArray(offset..(offset+size-1))
            val text1 = cut_bytes.inputStream().bufferedReader().readText() // Смотрю содержимое в дебагерре
            SimpleEnvelope(partialEnvelope.meta, cut_bytes.asBinary())
        }
    }

Resolve name conflicts in case several plugins export tasks with the same name.

Data and DataNode to file and back.

Interpret the Schema as Meta and serialize/deserialize it.

For example join and joinByGroup work as reduce action.

Pipe is analog of map.

Either wait for compiler-plugins and generate two versions of specifications or separate meta and config in Specified and use first for read and second for write.