Git Product home page Git Product logo

twiddles's Introduction

Twiddles

A twiddle list is a list of one or more values, potentially of differing types, that supports incremental creation and supports conversion to case classes that are "shape compatible" with the constituent types of the twiddle list.

Twiddle lists are useful in the creation of protocols (e.g., decoders, encoders, codecs), where a protocol for a complex type is built from simpler constituent protocols. This technique was first popularized by parser combinators with syntax like lparen ~ expr ~ rparen. In contrast to type driven derivation schemes, where protocols are implicitly determined by the constituent types of a data constructor, twiddle lists keep the focus on the protocol.

This library provides the ability to work with twiddle lists for arbitrary types and provides a single API that works for both Scala 3 and Scala 2. On Scala 3, twiddle lists are represented as generic tuples -- e.g., F[Int *: String *: Boolean *: EmptyTuple] or equivalently F[(Int, String, Boolean)]. On Scala 2, twiddle lists are represented as Shapeless heterogeneous lists. The org.typelevel.twiddles package provides type aliases that allow for source compatibility (*: is aliased to shapeles.:: and EmptyTuple is aliased to shapeless.HNil).

Getting Started

Artifacts are published for Scala 2.12, 2.13, and 3 and all platforms (JVM, Scala.js, and Scala Native).

libraryDependencies += "org.typelevel" %%% "twiddles-core" % "0.7.2" // check Releases for the latest version
// Enable twiddle syntax for arbitrary types
import org.typelevel.twiddles.syntax._

case class Foo(x: Int, y: String)

val a = Option(42)
// a: Option[Int] = Some(value = 42)
val b = Option("Hi")
// b: Option[String] = Some(value = "Hi")

val foo = (a *: b).to[Foo]
// foo: Option[Foo] = Some(value = Foo(x = 42, y = "Hi"))

In this example, a *: b creates an Option[Int *: String *: EmptyTuple]. We then convert that value to an Option[Foo] via .to[Foo].

The *: operation comes from the imported twiddle syntax and is similar to the Scala 3 built-in tuple cons operation, but works on applied type constructors. The expression a *: b *: c builds an F[A *: B *: C *: EmptyTuple] from an F[A], F[B], and F[C]. The *: operation requires that the type constructor F has a cats.InvariantSemigroupal instance.

The to operation also comes from the imported twiddle syntax. Calling .to[X] on an F[T] for some twiddle list T results in an F[X] provided that T is shape compatible with X. In the most common case where X is a case class, shape compatibility is defined as T having the same types in the same order as the parameters of X. The to operation requires that the type constructor F has a cats.Invariant instance.

Invariant semigroupals are much more general than (covariant) functors, which means twiddle list support works for a wide variety of data types. For instance, contravariant functors are invariant semigroupals allowing us to use twiddle list syntax to incrementally build instances:

val fooOrdering = (summon[Ordering[Int]] *: summon[Ordering[String]]).to[Foo]
// fooOrdering: Ordering[Foo] = scala.math.Ordering$$anon$1@14de39e3

Library Usage

When designing a library that uses twiddle lists, the TwiddleSyntax trait can be mixed in to the companion object of a type constructor. This has the effect of providing twiddle syntax without requiring users of the library to import org.typelevel.twiddles.syntax._ at each call site.

import org.typelevel.twiddles.TwiddleSyntax
import cats.Applicative

trait Json
trait Decoder[A] {
  def decode(j: Json): Option[A]
}
object Decoder extends TwiddleSyntax[Decoder] {
  implicit val applicative: Applicative[Decoder] = new Applicative[Decoder] {
    def pure[A](a: A): Decoder[A] = _ => Some(a)
    def ap[A, B](ff: Decoder[A => B])(fa: Decoder[A]): Decoder[B] = j =>
      for {
        f <- ff.decode(j)
        a <- fa.decode(j)
      } yield f(a)
  }
}

val int: Decoder[Int] = _ => ???
// int: Decoder[Int] = repl.MdocSession$MdocApp0$$Lambda$9134/0x0000000802278000@3992fe1c
val string: Decoder[String] = _ => ???
// string: Decoder[String] = repl.MdocSession$MdocApp0$$Lambda$9135/0x0000000802278448@626b82b2

case class Foo(x: Int, y: String)
val fooDecoder = (int *: string).to[Foo]
// fooDecoder: Decoder[Foo] = repl.MdocSession$$anon$8$$Lambda$9138/0x0000000802279308@63c9be6e

In this example, the Decoder type has an Applicative instance defined in its companion object (and Applicative extends InvariantSemigroupal), and the companion object extends TwiddleSyntax. The latter enables use of *: and to with Decoder values without adding explicit imports (that is, there's no need to import org.typelevel.twiddles.syntax._ at call sites).

Etymology

The term "twiddle list" was first coined by Rob Norris in the Skunk library, where a twiddle list was defined as a left nested tuple. For example, a 4 element twiddle list consisting of an Int, String, Boolean, and Double was represented as (((Int, String), Boolean), Double).

This library uses a different encoding -- twiddle lists are encoded as tuples on Scala 3 and Shapeless heterogeneous lists on Scala 2. The previous 4 element twiddle list is represented as Int *: String *: Boolean *: Double *: EmptyTuple.

We adopt the name "twiddle list" to refer to the general technique of incremental construction of complex protocols.

twiddles's People

Contributors

buntec avatar francesconero avatar mpilquist avatar typelevel-steward[bot] avatar valencik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

twiddles's Issues

Compilation hangs on Scala 2 when specifying types for fairly long twiddles

Hello!

Thanks again @mpilquist for the 0.6.2 Skunk release. Unfortunately we found another blocker trying to complete our migration. This time there seems to be a problem when the compiler is forced to check a subtype relation for a Twiddle (in Skunk, this happens often, for example when passing data to queries or commands after inferring the Encoder type from the sql macro).

Here's a minimal example that takes about 30 seconds to compile on a fairly recent laptop:

object Debug {
  import org.typelevel.twiddles._
  
  val inferred =
    1 *: 1 *: 1 *: 1 *: 1 *: 1 *: 1 *: 1 *:
    1 *: 1 *: 1 *: 1 *: 1 *: 1 *: 1 *: 1 *:
    1 *: 1 *: 1 *: 1 *: 1 *: 1 *: 1 *: 1 *:
    1 *: 1 *: 1 *: 1 *: 1 *: 1 *:
        EmptyTuple
  
  type Expected = 
    Int *: Int *: Int *: Int *: Int *: Int *: Int *: Int *:
    Int *: Int *: Int *: Int *: Int *: Int *: Int *: Int *:
    Int *: Int *: Int *: Int *: Int *: Int *: Int *: Int *:
    Int *: Int *: Int *: Int *: Int *: Int *:
        EmptyTuple
  
  inferred: Expected
}

Removing the line inferred: Expected reduces the compile time to under a second, indicating that the issue isn't with the construction of the Twiddle or the Expected type per se.

Anything more than 30 elements in a twiddle, and the compilation times explode.

Here is the compilation profile for the snippet:

*** Cumulative timers for phases
#total compile time           : 1 spans, ()35254.291ms
  parser                      : 1 spans, ()2.018ms (0.0%)
  bm4-parser                  : 1 spans, ()11.805ms (0.0%)
  bm4-parser2                 : 1 spans, ()2.683ms (0.0%)
  kind-projector              : 1 spans, ()3.806ms (0.0%)
  namer                       : 1 spans, ()213.77ms (0.6%)
  packageobjects              : 1 spans, ()14.133ms (0.0%)
  typer                       : 1 spans, ()34539.101ms (98.0%) <------ typechecking is the problem
  bm4-typer                   : 1 spans, ()4.953ms (0.0%)
  superaccessors              : 1 spans, ()0.93ms (0.0%)
  extmethods                  : 1 spans, ()0.186ms (0.0%)
  pickler                     : 1 spans, ()0.299ms (0.0%)
  xsbt-api                    : 1 spans, ()3.484ms (0.0%)
  xsbt-dependency             : 1 spans, ()15.01ms (0.0%)
  refchecks                   : 1 spans, ()12.612ms (0.0%)
  patmat                      : 1 spans, ()0.202ms (0.0%)
  uncurry                     : 1 spans, ()1.76ms (0.0%)
  fields                      : 1 spans, ()0.377ms (0.0%)
  tailcalls                   : 1 spans, ()0.192ms (0.0%)
  specialize                  : 1 spans, ()33.64ms (0.1%)
  explicitouter               : 1 spans, ()0.511ms (0.0%)
  erasure                     : 1 spans, ()4.071ms (0.0%)
  posterasure                 : 1 spans, ()0.405ms (0.0%)
  lambdalift                  : 1 spans, ()0.598ms (0.0%)
  constructors                : 1 spans, ()1.067ms (0.0%)
  flatten                     : 1 spans, ()0.182ms (0.0%)
  mixin                       : 1 spans, ()0.431ms (0.0%)
  cleanup                     : 1 spans, ()0.862ms (0.0%)
  delambdafy                  : 1 spans, ()0.332ms (0.0%)
  jvm                         : 1 spans, ()27.799ms (0.1%)
  scalacenter-profiling       : 1 spans, ()267.273ms (0.8%)
  xsbt-analyzer               : 1 spans, ()7.995ms (0.0%)

and details for the typer phase:

*** Cumulative statistics at phase typer
  #plausibly compatible       : 30 (100.0%)
  #typed                      : 30 (100.0%)
#class symbols                : 10294
#typechecked selections       : 159
#created tree nodes           : 1996
#created scopes               : 750
  #implicit inscope hits      : 30 (100.0%)
#implicit searches            : 30
  #plausibly compatible       : 30 (100.0%)
  #matching                   : 30 (100.0%)
  #typed                      : 30 (100.0%)
  #found                      : 30 (100.0%)
  #implicit inscope hits      : 30 (100.0%)
time spent in scope population: 122 spans, ()0.012ms
#findMember ops               : 2181
  of which not found          : 631 (28.9%)
  of which multiple overloaded: 2 (0.1%)
  of which in implicit        : 1069 (49.0%)
#subtype ops                  : 1236
  of which in implicit        : 510 (41.3%)
#base type seqs               : 204
avg base type seq length      : 4.2
  of which for compound types : 30 (14.7%)
  of which for typerefs       : 173 (84.8%)
#type symbols                 : 19269
#unique types                 : 3221
#typechecked identifiers      : 189
  #found                      : 30 (100.0%)
#symbols                      : 32117
time spent typechecking       : 1 spans, ()34539.089ms
time spent in lubs            : 0 spans, ()0.0ms (0.0%) aggregate, 0.0ms (0.0%) specific
time spent in <:<             : 1236 spans, ()34447.786ms (99.7%) aggregate, 34447.678ms (99.7%) specific <----- subtyping check (?)
time spent in findmember      : 2181 spans, ()0.935ms (0.0%) aggregate, 0.935ms (0.0%) specific
time spent in findmembers     : 0 spans, ()0.0ms (0.0%) aggregate, 0.0ms (0.0%) specific
time spent in asSeenFrom      : 4797 spans, ()25.053ms (0.1%) aggregate, 24.847ms (0.1%) specific
time spent in baseTypeSeq     : 203 spans, ()1.032ms (0.0%) aggregate, 0.714ms (0.0%) specific
time spent in baseClasses     : 93 spans, ()2.786ms (0.0%) aggregate, 2.393ms (0.0%) specific
time classfilereading         : 114 spans, ()55.341ms (0.2%)
time spent in failed          : 0 spans, ()0.0ms (0.0%)
  failed apply                : 0 spans, ()0.0ms (0.0%)
  failed op=                  : 0 spans, ()0.0ms (0.0%)
time spent ref scanning       : 0 spans, ()0.0ms (0.0%)
time spent in implicits       : 30 spans, ()60.276ms (0.2%)
  successful in scope         : 30 spans, ()49.516ms (0.1%)
  failed in scope             : 0 spans, ()0.0ms (0.0%)
  successful of type          : 0 spans, ()0.0ms (0.0%)
  failed of type              : 0 spans, ()0.0ms (0.0%)
  assembling parts            : 0 spans, ()0.0ms (0.0%)
  matchesPT                   : 60 spans, ()0.224ms (0.0%)
time spent in macroExpand     : 0 spans, ()0.0ms (0.0%)
#sametype ops                 : 536870914
#all lubs/glbs                : 60
#typechecked applications     : 91
  #matching                   : 30 (100.0%)

Interestingly, I found that specifying the type variances in the type alias *: for Scala 2 exactly as done in Shapeless (so type *:[+A, +B <: Tuple] = ::[A, B]) eliminates this problem. This observation is intriguing since, in theory, specifying variances in a type alias should be functionally equivalent to not specifying them.

I'll open a PR shortly with this solution, even though I'll admit I'm not 100% sure of why it seems to solve the problem. In the meantime any insights you might have would be incredibly valuable.

Thanks again!

Rewrite twiddle syntax on Scala 3 as extension methods

This was partially done in #9 but I was unable to get all examples compiling. The goal is to remove these implicit conversions:

implicit def toTwiddleOpCons[B <: Tuple](fb: F[B]): TwiddleOpCons[F, B] = new TwiddleOpCons(
fb
)
implicit def toTwiddleOpTwo[B](fb: F[B]): TwiddleOpTwo[F, B] = new TwiddleOpTwo(fb)
and the associated AnyVals.

See #11 for a potential starting point.

Compilation hangs on Scala 2 using fairly long twiddles

Hello!

When I tried upgrading Skunk to 0.6.0, which uses this library, I discovered that compilation would hang, whereas with the previous version (using the old twiddle lists), it was fairly fast.

Digging in a bit, it seemed to be caused by some queries that selected a fair amount of columns (20 or so). I isolated a minimum example to demonstrate the problem: this snippet currently makes the compilation hang using Scala 2.13.12

import org.typelevel.twiddles._

1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
EmptyTuple

Scastie

I suspect the implicit conversion hlistToTuple in TwiddleCompat is causing unnecessary failed implicit searches exhibiting a quadratic complexity somewhere (even though the exact mechanism is not clear to me, it seems to be constantly trying to build a Tupler).

In fact, removing the implicit from scope makes it compile again.

import org.typelevel.twiddles.{hlistToTuple => _, _}

1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
1 *: 1 *: 1 *: 1 *:
EmptyTuple

Scastie

An approach I tried which seems to preserve the tuple conversion and not make the compiler hang is to define an implicit conversion for every tuple arity, mirroring the already existing tupleNtoHlist functions.

E.g.:

implicit def hlistToTuple2[Z <: Tuple, A, B](z: Z)(implicit
    tupler: Tupler.Aux[Z, (A, B)]
): (A, B) = tupler(z)

// and so on up to 22

This seems to help the compiler know the implicit is useless to build an HList. Is it because the output type is more explicit? I'm not sure.

I was planning on opening a PR to address this issue, as it is currently blocking our upgrade to the latest version of Skunk at work. We could keep using the legacy twiddles, but they would cause quite a lot of deprecation warnings. I just wanted to check which approach would be best to take first.

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.