Comments (13)
Thank you. I using a compiled version of the tool from main branch:
wna@Wojciechs-MBP: greenmask % ./greenmask --version
greenmask version a2ecc2cdae5af10e9443ccdab61987f31bd3ade6 2024-08-20T11:49:00Z
I have used random numeric transformer as advised:
transformation:
- schema: public
name: backlog_entry_value
transformers:
- name: RandomNumeric
params:
column: "value"
min: 0.0
max: 10.0
decimal: 2
and received following error when trying to restore:
2024-08-24T12:15:57+02:00 WRN unable to restore table error="error post streaming handling: error from postgres connection: numeric field overflow A field with precision 10, scale 2 must round to an absolute value less than 10^8. (code 22003)"
I converted the dump from directory format to human readable SQL and found:
--
-- Data for Name: "backlog_entry_value"; Type: TABLE DATA; Schema: "public"; Owner: "wna"
--
COPY "public"."backlog_entry_value" ("unique_id", "created", "value", "author_name", "author_email", "backlog_entry_id") FROM stdin
dc673f92-404d-4873-ba36-779da5947748 2020-05-25 08:22:27.179153 -55142080548242225025.74 <redacted_username> <redacted_email> 33cd6668-6dd7-4059-95a1-e5ae03027c04
value is not kept in <min, max> bounds. This seems to be a bug.
from greenmask.
I confirm RandomNumeric transformer keeps value within defined bounds. Thank you
from greenmask.
Hi! Thank you for reporting. You've set the transformation in the wrong object (in storage) instead of dump
. It should be this way
dump:
pg_dump_options:
dbname: host=localhost user=wna dbname=bireporting
load-via-partition-root: true
transformation:
- schema: public
name: backlog_entry_value
transformers:
- name: RandomFloat
params:
column: "value"
min: 0.0
max: 10.0
precision: 2
storage:
type: directory
directory:
path: /Users/wna/Developer/
Try it
from greenmask.
However, I agree that it would be helpful to report this issue. I'll consider how to fix it.
from greenmask.
Thank you! Such a silly mistake.
Does this mean numeric(precision,scale)
type is not supported?
2024-08-24T11:20:11+02:00 ERR ValidationWarning={"hash":"","meta":{"AllowedTypes":["float4","float8"],"ColumnName":"value","ParameterName":"column","SchemaName":"public","TableName":"backlog_entry_value","TransformerName":"RandomFloat","TypeName":"numeric(10,2)"},"msg":"unsupported column type","severity":"error"}
2024-08-24T11:20:11+02:00 FTL cannot make a backup error="context error: fatal validation error"
from greenmask.
Yes, it does but only the v0.2b1 version or the current development version in the main. For v0.2b1 version and current main you can use RandomNumeric transformer that has wider configuration options such as decimal
.
For the latest version 1.14 you can try to override the type of the table in the config using the next
transformation:
- schema: public
name: backlog_entry_value
columns_type_override:
value: "numeric"
transformers:
- name: RandomFloat
params:
column: "value"
min: 0.0
max: 10.0
precision: 2
The details are here
from greenmask.
@wnatrifork FYI ^
from greenmask.
I've added a fix in MR #177. So in a new release if the unknown option was provided it will be reported.
from greenmask.
@wnatrifork The MR #177 is merged. I hope this will fix this implicit issue
from greenmask.
Thank you for reporting. I will check shortly
from greenmask.
@wnatrifork
That's definitely a bug. It will be fixed today/tomorrow.
from greenmask.
@wnatrifork MR #178 should close your issue
from greenmask.
@wnatrifork try now and report please
from greenmask.
Related Issues (20)
- feat: Database subset. Stage2 HOT 1
- Delete old dumps except for the last X HOT 1
- Data Generator Request: RandomCity HOT 4
- feat: Subset stage 3: Circular dependencies subset
- backlog 2024 Q3
- feat: Transformers collection and python library
- Container image HOT 5
- feat: PostgreSQL 17 support
- Unable to use `on-conflict-do-nothing` flag due to other missing flags that cannot be supplied HOT 6
- feat: Subset stage 4: Multi cycles in the same SCC
- feat: Subset stage 5: Virtual foreign keys
- fix: Make a new transformer verbose
- feat: Add pgzip support HOT 2
- feat: Implement siphash as hashing function
- backlog 2024 Q4
- feat: Implement transformation conveyor
- bug: restore command with -C -c has unexpected behaviour
- fix: refreash transaction state when restoring data HOT 2
- Converting dump into SQL results in missing semicolon and newline after `FROM stdin` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from greenmask.