Comments (2)
I can't repro with the following further minimized blob. So it's possibly something in the JobTest
code.
import com.spotify.scio.bigquery.{BigQueryType, TableRow}
import magnolify.bigquery._
object Test {
def main(args: Array[String]): Unit = {
val bqt = TableRowType[TestType]
val r1 = TestType("hello", Some("world".getBytes))
val tr1: TableRow = TestType.toTableRow(r1)
println(tr1)
val r1a = bqt(tr1)
println((r1a, r1a.bytes.map(new String(_))))
val r2 = TestType("hello", None)
val tr2: TableRow = TestType.toTableRow(r2)
println(tr2)
val r2a = bqt(tr2)
println((r2a, r2a.bytes.map(new String(_))))
}
@BigQueryType.toTable
case class TestType(id: String, bytes: Option[Array[Byte]])
}
GenericData{classInfo=[f], {id=hello, bytes=d29ybGQ=}}
(TestType(hello,Some([B@7d82bbc7)),Some(world))
GenericData{classInfo=[f], {id=hello}}
(TestType(hello,None),None)
from magnolify.
Turns out this has nothing to do with either magnolify or scio.
TableRow
here is a pure Java type using Jackson for ser/de, so the bytes field needs to be plain byte array with Some()
.
new TableRow().set("id", "test-id").set("bytes", Some(Array(1.toByte, 9.toByte)))
from magnolify.
Related Issues (20)
- Set cats and scalacheck dependencies as provided
- Neo4j Record support
- Support schema annotations for magnolify-parquet HOT 1
- Build with java 17
- magnolify.bigtable.ByteStringComparator is built with Java 11 since 0.4.7 release
- Publish docs as a GH site
- Enum implicit taken instead of typeclass derivation HOT 2
- Set avro.java.String property in magnolify-avro schema derivation? HOT 1
- Add support of Joda-time types to all magnolify modules
- Support ByteBuffer derivations? HOT 1
- Neo4J documentation
- Rework AvroCompat for ParquetType HOT 1
- Replace deprecated usage of `FileDescriptor.Syntax` in ProtobufType
- Remove package object inheritance
- ParquetType throws error writing Optional empty list HOT 1
- Enable dictionary encoding by default for enum ParquetFields
- Support Map schema types in Parquet HOT 1
- Handling non-Enums in EnumType[] type class in scala3
- Regression in bytes read safety in avro HOT 1
- How to get a `Map` instance from Guava Funnel? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from magnolify.