Git Product home page Git Product logo

Comments (13)

wnatrifork avatar wnatrifork commented on August 27, 2024 1

Thank you. I using a compiled version of the tool from main branch:

wna@Wojciechs-MBP: greenmask % ./greenmask --version
greenmask version a2ecc2cdae5af10e9443ccdab61987f31bd3ade6 2024-08-20T11:49:00Z

I have used random numeric transformer as advised:

  transformation:
    - schema: public
      name: backlog_entry_value
      transformers:
        - name: RandomNumeric
          params:
            column: "value"
            min: 0.0
            max: 10.0
            decimal: 2

and received following error when trying to restore:

2024-08-24T12:15:57+02:00 WRN unable to restore table error="error post streaming handling: error from postgres connection: numeric field overflow A field with precision 10, scale 2 must round to an absolute value less than 10^8. (code 22003)"

I converted the dump from directory format to human readable SQL and found:

--
-- Data for Name: "backlog_entry_value"; Type: TABLE DATA; Schema: "public"; Owner: "wna"
--

COPY "public"."backlog_entry_value" ("unique_id", "created", "value", "author_name", "author_email", "backlog_entry_id") FROM stdin
dc673f92-404d-4873-ba36-779da5947748	2020-05-25 08:22:27.179153	-55142080548242225025.74	<redacted_username>	<redacted_email>	33cd6668-6dd7-4059-95a1-e5ae03027c04

value is not kept in <min, max> bounds. This seems to be a bug.

from greenmask.

wnatrifork avatar wnatrifork commented on August 27, 2024 1

I confirm RandomNumeric transformer keeps value within defined bounds. Thank you

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

Hi! Thank you for reporting. You've set the transformation in the wrong object (in storage) instead of dump. It should be this way

dump:
  pg_dump_options:
    dbname: host=localhost user=wna dbname=bireporting
    load-via-partition-root: true
  transformation:
    - schema: public
      name: backlog_entry_value
      transformers:
        - name: RandomFloat
          params:
            column: "value"
            min: 0.0
            max: 10.0
            precision: 2
storage:
  type: directory
  directory:
    path: /Users/wna/Developer/

Try it

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

However, I agree that it would be helpful to report this issue. I'll consider how to fix it.

from greenmask.

wnatrifork avatar wnatrifork commented on August 27, 2024

Thank you! Such a silly mistake.

Does this mean numeric(precision,scale) type is not supported?

2024-08-24T11:20:11+02:00 ERR ValidationWarning={"hash":"","meta":{"AllowedTypes":["float4","float8"],"ColumnName":"value","ParameterName":"column","SchemaName":"public","TableName":"backlog_entry_value","TransformerName":"RandomFloat","TypeName":"numeric(10,2)"},"msg":"unsupported column type","severity":"error"}
2024-08-24T11:20:11+02:00 FTL cannot make a backup error="context error: fatal validation error"

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

Yes, it does but only the v0.2b1 version or the current development version in the main. For v0.2b1 version and current main you can use RandomNumeric transformer that has wider configuration options such as decimal.

For the latest version 1.14 you can try to override the type of the table in the config using the next

transformation:
    - schema: public
      name: backlog_entry_value
      columns_type_override:
        value: "numeric" 
      transformers:
        - name: RandomFloat
          params:
            column: "value"
            min: 0.0
            max: 10.0
            precision: 2

The details are here

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

@wnatrifork FYI ^

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

I've added a fix in MR #177. So in a new release if the unknown option was provided it will be reported.

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

@wnatrifork The MR #177 is merged. I hope this will fix this implicit issue

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

Thank you for reporting. I will check shortly

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

@wnatrifork
That's definitely a bug. It will be fixed today/tomorrow.

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

@wnatrifork MR #178 should close your issue

from greenmask.

wwoytenko avatar wwoytenko commented on August 27, 2024

@wnatrifork try now and report please

from greenmask.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.