Comments (8)
Yep. That's the test case which I wrote in the Spark community. :)
Thanks, @guiyanakuang . I'll close this issue because it seems that @liujinhui1994 found the answer here.
from orc.
Did you check Apache ORC webpage about Spark Configuration, @liujinhui1994 ? The encryption option should be provided in the same way during writing and reading. So, we prefer to put it to the table properties.
CREATE TABLE encrypted (
ssn STRING,
email STRING,
name STRING
)
USING ORC
OPTIONS (
hadoop.security.key.provider.path "kms://http@localhost:9600/kms",
orc.key.provider "hadoop",
orc.encrypt "pii:ssn,email",
orc.mask "nullify:ssn;sha256:email"
)
from orc.
I checked here. Only saw how the table is configured. There is no way to see the datasource. Because my task will not use the table.
Does the above configuration work the way dataSource does?
from orc.
@liujinhui1994 Using a custom dataSource to read and write orc files?
Maybe this test case can help you.
https://github.com/apache/orc/blob/main/java/core/src/test/org/apache/orc/impl/TestEncryption.java
from orc.
Is it possible to do something like parquet encryption.
The following way to pass?
JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());
jsc.hadoopConfiguration().set("","")
from orc.
https://spark.apache.org/docs/latest/sql-data-sources-parquet.html
from orc.
I think it's similar, but I haven't practiced using it this way
https://github.com/apache/spark/blob/c55b9fd1e014fac979b1e42f5a880e7b63286a54/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcEncryptionSuite.scala#L27-L60
from orc.
ok i'll try it thanks @guiyanakuang
from orc.
Related Issues (20)
- ORC-1696: Fix ClassCastException when reading avro decimal type in bechmark HOT 1
- ORC-1696: Fix ClassCastException when reading avro decimal type in bechmark HOT 1
- ORC-1696: Fix ClassCastException when reading avro decimal type in bechmark HOT 1
- ORC-1699: Fix SparkBenchmark in Parquet format according to SPARK-40918 HOT 1
- orc-tools unknown subcommand "Merge" HOT 3
- Release ORC 2.0.1 HOT 1
- Buffer size too small. size HOT 5
- [CPP] Flushing without writing after registering a timeseries will cause a core dump.
- ORC-1721: Upgrade `aircompressor` to 0.27 HOT 1
- ORC-1721: Upgrade `aircompressor` to 0.27 HOT 1
- Release ORC 1.9.4 HOT 1
- [Java] Different semantic of lengths for CHAR(n) with C++ HOT 1
- ORC-1540: Remove MacOS 11 from GitHub Action CI and docs HOT 1
- ORC-1540: Remove MacOS 11 from GitHub Action CI and docs HOT 1
- ORC-1540: Remove MacOS 11 from GitHub Action CI and docs HOT 1
- ORC-1619: Add `MacOS 14` to GitHub Action HOT 1
- ORC-1738: [C++] Fix wrong Int128 maximum value HOT 1
- ORC-1738: [C++] Fix wrong Int128 maximum value HOT 1
- ORC-1738: [C++] Fix wrong Int128 maximum value HOT 1
- How does hive.exec.orc.default.buffer.size affect the file size? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from orc.