Where can I find sample code for using ORC column encryption when writing ORC files with spark dataSource about orc HOT 8 CLOSED

Yep. That's the test case which I wrote in the Spark community. :)
Thanks, @guiyanakuang . I'll close this issue because it seems that @liujinhui1994 found the answer here.

from orc.

Did you check Apache ORC webpage about Spark Configuration, @liujinhui1994 ? The encryption option should be provided in the same way during writing and reading. So, we prefer to put it to the table properties.

• https://orc.apache.org/docs/spark-config.html

CREATE TABLE encrypted (
  ssn STRING,
  email STRING,
  name STRING
)
USING ORC
OPTIONS (
  hadoop.security.key.provider.path "kms://http@localhost:9600/kms",
  orc.key.provider "hadoop",
  orc.encrypt "pii:ssn,email",
  orc.mask "nullify:ssn;sha256:email"
)

from orc.

I checked here. Only saw how the table is configured. There is no way to see the datasource. Because my task will not use the table.
Does the above configuration work the way dataSource does?

@dongjoon-hyun

from orc.

@liujinhui1994 Using a custom dataSource to read and write orc files?
Maybe this test case can help you.
https://github.com/apache/orc/blob/main/java/core/src/test/org/apache/orc/impl/TestEncryption.java

from orc.

Is it possible to do something like parquet encryption.
The following way to pass?
JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());
jsc.hadoopConfiguration().set("","")

@guiyanakuang

from orc.

https://spark.apache.org/docs/latest/sql-data-sources-parquet.html

from orc.

I think it's similar, but I haven't practiced using it this way
https://github.com/apache/spark/blob/c55b9fd1e014fac979b1e42f5a880e7b63286a54/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcEncryptionSuite.scala#L27-L60

from orc.

ok i'll try it thanks @guiyanakuang

from orc.