elgentos / masquerade Goto Github PK
View Code? Open in Web Editor NEWFaker-driven, configuration-based, platform-agnostic, locale-compatible data faker tool
License: MIT License
Faker-driven, configuration-based, platform-agnostic, locale-compatible data faker tool
License: MIT License
Hi! First of all thank your for this tool!
I'm trying to build a new .phar file following your process but it doesn't seem to work. So I decided just to clone your repo and try to build the same .phar and I found differences between the phar uploaded here and the phar is being created when I use the build process.
Being more specific I found a difference in one of the depencencies: elgentos/parser.
composer.lock is pointing to 1.4.2 (5ef1c392c83d928bdb58778618c7811e24f82416) If you take a look at the FileAbstract
class inside your actual phar you will see the next code (I'm not able to find this code inside elgentos/parser repository
):
private function safePath(string $path): string
{
while (($newPath = str_replace('..', '', $path)) !== $path) {
$path = $newPath;
}
return $path;
}
But if you take a look at the referenced code in composer.lock FileAbstract you will notice that the code differs:
private function safePath(string $path): string
{
while (($newPath = str_replace(['..', '//'], ['', '/'], $path)) !== $path) {
$path = $newPath;
}
return str_replace(['..', '//'], ['', '/'], $path);
}
So the new builds can't find correctly the config folder (here is where it fails
Error message:
In Glob.php line 55:
RecursiveDirectoryIterator::__construct(phar:/blablabla/dist/masquerade.phar/src/config/magento2): failed to open dir: No such file or directory
I think that maybe a rebase broke this depencency, any idea why is this happening?
Thank you in advance.
Hello,
is skipping not existing tables possible? I can see #11 resolved non-existing columns issue, but it would be great not to fail when the specific table isn't present (i.e. with a cmd flag switch?)
Now that we've added a try/catch block we can add YAML definitions for tables that don't necessarily have to be present in an install.
In our projects, we include YAML files for all possible extensions we use. If a project doesn't have that table, it'll just skip it now.
I've put a few of those YAML files in the Wiki, see:
What would be the best way to share these? I don't think adding them to Masquerade itself would be wise, since that'll clutter stuff and maybe even introduce unexpected behavior. A separate repo maybe? Keep placing them in the wiki? Any other ideas?
Seed is useful to keep data same between multiple runs.
We now use phar-composer.phar
, but Box seems like a better solution; https://box-project.github.io/box2/
When updating a row in a table, and setting the firstname, lastname and email address, all three are completely random. This means the email address does not 'match' the name, or username, for example;
user_id firstname lastname email username password
1 Jasper van de Berg [email protected] Megan .3O[&rw/oR^6p
This is, for Magento, in itself not problematic. The only point where it does not match but it should, is where the order data differs from the quote data, which shouldn't occur in a natural environment.
Maybe we can find a way to make this more matching.
Maybe take some inspiration from Divante's implementation; https://github.com/DivanteLtd/anonymizer#how-to-use-anonymizer-with-magento-eav
Should be fairly the same as magento2
See #66 (comment)
I wrote up our Bitbucket usage here: https://github.com/getjohn/masquerade/wiki/Bitbucket-Pipelines-Usage
Feel free to add it to your wiki :)
After masquerade silently fails i tried verbose mode to get an idea what happens there, but it does not work for -vv
or -vvv
Box Requirements Checker
========================
> Using PHP 7.3.27
> PHP is using the following php.ini file:
/usr/local/etc/php/php.ini
> Checking Box requirements:
......
[OK] Your system is ready to run the application.
[Foreign key constraint checking is off - deletions will not affect linked tables]
._ _ _. _ _. _ .__. _| _
| | |(_|_>(_||_|(/_|(_|(_|(/_
|
by elgentos
v0.3.4
Updating table using the following data
array (
'processor' => 'Elgentos\\Masquerade\\DataProcessor\\RegularTableProcessor',
'configuration' =>
array (
'pk' => 'user_id',
'columns' =>
array (
'firstname' =>
array (
'formatter' => 'firstName',
),
'lastname' =>
array (
'formatter' => 'lastName',
),
'email' =>
array (
'formatter' => 'email',
'nullColumnBeforeRun' => true,
'unique' => true,
),
'username' =>
array (
'formatter' => 'firstName',
'nullColumnBeforeRun' => true,
'unique' => true,
),
'password' =>
array (
'formatter' => 'password',
),
),
'name' => 'admin_user',
'provider' =>
array (
),
),
)
Nullifying columns in batches for [admin_user]
ERROR:
Unable to display the estimated time if the maximum number of steps is not set.
is it possible to run this agains an mysqldump ? it would be great to masquerade an existing mysql backup instead of changing data on database itself.
Hello and thank you for sharing the code,
I've tried to install Masquerade by following the instructions described in README.md (line 61) :
wget https://github.com/elgentos/masquerade/releases/download/0.1.10/masquerade.phar
and i received a 404 error.
However, it works with release v 0.1.9
Best regards
Issue: I followed the documentation surrounding how to add custom config values, but masquerade
does not seem to find my custom configuration.
My ultimate goal is to completely override the admin.yaml
file, as I don't want to masquerade any values in the admin_user
table. But I'm first trying to get masquerade
to pickup my custom config files.
Steps to reproduce:
Create a config/admin.yaml
file with these contents:
admin:
admin_user:
pk: user_id
columns:
firstname:
formatter: firstName
lastname2:
formatter: lastName
Run masquerade groups
Expected results:
+----------+------------------+---------------------------------------+--------------------+---------------------+
| Platform | Group | Table | Column | Formatter |
+----------+------------------+---------------------------------------+--------------------+---------------------+
| magento2 | admin | admin_user | firstname | email |
| magento2 | admin | admin_user | lastname | lastName |
| magento2 | admin | admin_user | lastname2 | lastName |
| magento2 | admin | admin_user | email | email |
| magento2 | admin | admin_user | username | firstName |
| magento2 | admin | admin_user | password | password |
…
Actual results:
+----------+------------------+---------------------------------------+--------------------+---------------------+
| Platform | Group | Table | Column | Formatter |
+----------+------------------+---------------------------------------+--------------------+---------------------+
| magento2 | admin | admin_user | firstname | firstName |
| magento2 | admin | admin_user | lastname | lastName |
| magento2 | admin | admin_user | email | email |
| magento2 | admin | admin_user | username | firstName |
| magento2 | admin | admin_user | password | password |
…
Screenshot, verifying my setup:
0.1.13
wget https://github.com/elgentos/masquerade/releases/download/0.1.13/masquerade.phar
returns 647bytes
0.1.12
wget https://github.com/elgentos/masquerade/releases/download/0.1.12/masquerade.phar
returns 3.2MB
This is a terrific tool BTW 🥇
To for example set a field to NULL or some other fixed value.
hi, lately i'm finding this error on an attempted run of masquerade to anonymise a magento DB for sandboxing.
i have tried this with a few recent db dumps from the last few months, and they all fail, while i've been using it successfully for a year and a half.
results of a verbose run here:
$ php masquerade.phar run -vvv
Box Requirements Checker
========================
> Using PHP 7.4.19
> PHP is using the following php.ini file:
/etc/php.ini
> Checking Box requirements:
✔ The application requires the version "^7.2" or greater.
✔ The application requires the extension "zlib".
✔ The package "doctrine/dbal" requires the extension "pdo".
✔ The package "illuminate/database" requires the extension "json".
✔ The package "illuminate/support" requires the extension "json".
✔ The package "nesbot/carbon" requires the extension "json".
[OK] Your system is ready to run the application.
[Foreign key constraint checking is off - deletions will not affect linked tables]
PHP Warning: Wrong COM_STMT_PREPARE response size. Received 7 in phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php on line 67
PHP Fatal error: Uncaught Error: Call to a member function execute() on bool in phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/vendor/illuminate/database/Connection.php:463
Stack trace:
#0 phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/vendor/illuminate/database/Connection.php(662): Illuminate\Database\Connection->Illuminate\Database\{closure}('SET GLOBAL sql_...', Array)
#1 phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/vendor/illuminate/database/Connection.php(629): Illuminate\Database\Connection->runQueryCallback('SET GLOBAL sql_...', Array, Object(Closure))
#2 phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/vendor/illuminate/database/Connection.php(464): Illuminate\Database\Connection->run('SET GLOBAL sql_...', Array, Object(Closure))
#3 phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/src/Elgentos/Masquerade/Console/RunCommand.php(330): Illuminate\Database\Connection->statement('SET GLOBAL sql_...')
#4 phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/src/Elgent in phar:///var/www/firetoys-sand1/masquerade/masquerade.phar/vendor/illuminate/database/Connection.php on line 463
I have double checked the config.yaml a few times, and it all seems in order. the only results i can see anywhere else all point to laravel issues, and masquerade is the only laravel user on this server as far as i'm aware. has this been seen before?
far as i can tell, foreign constraint checking is ON, and i have reset it in mysql with SET GLOBAL FOREIGN_KEY_CHECKS = 1;
to no avail
On build/push
I wasn't sure whether this is in scope or not as I'm sure most merchants run third-party payment methods. However, considering that Paypal is shipped with Magento, I probably would have expected this to happen.
Thanks for the quick release yesterday.
Alas, downloading https://github.com/elgentos/masquerade/releases/latest/download/masquerade.phar
no longer works.
Actually, it would be nice if one could download the phar for a tagged release. E.g. https://github.com/elgentos/masquerade/releases/0.3.5/download/masquerade.phar or something like that.
To make it backward compatible with older/newer versions of the specified framework
Implement our own configuration file parser elgentos/parser instead of hassankhan/config
.
Sorry for all the issuing lately.
I'm getting the following error when anonymizing the customer_entity table:
Updating customer_entity
6480/107551 [=>--------------------------] 6%
In Connection.php line 664:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '[email protected]' for key 'CUSTOMER_ENTITY_EMAIL_WEBSIT
E_ID' (SQL: update `customer_entity` set `email` = [email protected], `prefix` = , `firstname` = Ramon, `middlename` = , `lastnam
e` = Rice, `suffix` = where `entity_id` = 6515)
Even though email
is set to unique, faker sometimes generates email addresses which have already been used in the same runtime. This probably happens because faker only has so many data that recurrence might happen.
Here are some ideas to fix this issue:
To prevent memory limit errors on huge databases
Won't work with UUID or string PK like slack identifier (i.e. U0103P9J5ME)
Current it default to branch: (20180711.080851) which is from 4 years old, unless you specify right tag.
Think this old branch should be deleted or renamed, so that composer doesn't think it's newest tag.
For example;
randomNumber($nbDigits = NULL, $strict = false) // 79907610
We should be able to set $nbDigits
and $strict
somehow in our yaml definition.
Instead of running everything in the same php thread, it'd be nice if the application runs a subprocess for each table or something like that.
Hi,
I try to anonymize the customer entity except several domain
So i create a yaml file under
src/config/magento2/customer.yaml
in my yaml file i try first with only one domain to be sure everything is working as expected like
customers:
customer_entity:
provider: # this sets options specific to the type of table
where: "`email` not LIKE '%@domaineA.com' "
I run my command like that :
./masquerade.phar run --group=customer --platform=magento2 --config=src/config/magento2/
but all customer entities are "rewriten" to fake one including the one with domainA.com in the email field.
What I've done wrong ? and is it available to add a multiple filter like we can do on sql
customers:
customer_entity:
provider: # this sets options specific to the type of table
where: "`email` not LIKE '%@domaineA.com' AND `email` not LIKE '%@domaineB.com'"
Thanks for your help and thumb up to @peterjaap for the slack help also :)
Faker has the option to add our own provider for defined or custom formatters; https://github.com/fzaninotto/Faker#faker-internals-understanding-providers
It would be great to implement some kind of pattern (event? decorator?) to add our own providers by using this.
magento/magento2-base 2.4.4 requires symfony/console ~4.4.0
After i have created the entity.yaml file and defined the columns to mask how do i build it to get sql?
Add identify command to find possible columns to anonimize.
It should;
firstname
, email
, etc)So we can make of all the goodies in there, like self-update.
Hello,
Is Masquerade compatible with PostgreSQL ? I am having many errors:
SQLSTATE[42704]: Undefined object: 7 ERROR: unrecognized configuration parameter "foreign_key_checks"
SQLSTATE[42704]: Undefined object: 7 ERROR: unrecognized configuration parameter "sql_mode"
SQLSTATE[42703]: Undefined column: 7 ERROR: column "column_key" does not exist
Example creditmemo.yaml:
creditmemo:
sales_creditmemo:
provider:
delete: true
Run: php bin/masquerade run --with-integrity
Result:
._ _ _. _ _. _ .__. _| _
| | |(_|_>(_||_|(/_|(_|(_|(/_
|
by elgentos
v0.3.0
PHP Warning: Invalid argument supplied for foreach() in /Users/tjitse/sites/masquerade/src/Elgentos/Masquerade/Console/RunCommand.php on line 129
Warning: Invalid argument supplied for foreach() in /Users/tjitse/sites/masquerade/src/Elgentos/Masquerade/Console/RunCommand.php on line 129
PHP Notice: Array to string conversion in /Users/tjitse/sites/masquerade/src/Elgentos/Masquerade/Console/SymfonyOutput.php on line 92
Notice: Array to string conversion in /Users/tjitse/sites/masquerade/src/Elgentos/Masquerade/Console/SymfonyOutput.php on line 92
Deleting records from Array table
PHP Notice: Array to string conversion in /Users/tjitse/sites/masquerade/src/Elgentos/Masquerade/Console/SymfonyOutput.php on line 92
Notice: Array to string conversion in /Users/tjitse/sites/masquerade/src/Elgentos/Masquerade/Console/SymfonyOutput.php on line 92
Records have been deleted from Array table
Updating [sales_creditmemo_comment]
0/0 [----->----------------------] 0% < 1 sec Complete
But for some reason it duplicates the to-be-anonimized record and anonimizes that one. Maybe because of the binary UUID's?
But not all tables. customer
, customer_address
, newsletter_recipient
, user
, etc seem to be unaffected. order_customer
and order_address
contain the duplicate records.
Hi
I have followed the installation instructions and "Download the phar file:
curl -L -o masquerade.phar https://github.com/elgentos/masquerade/releases/latest/download/masquerade.phar
"
When I then run
php masquerade.phar run --help
I get the error
PHP Warning: require(phar:///home/ubuntu/masquerade.phar/vendor/composer/../symfony/translation/Resources/functions.php): failed to open stream: phar error: "vendor/symfony/translation/Resources/functions.php" is not a file in phar "/home/ubuntu/masquerade.phar" in phar:///home/ubuntu/masquerade.phar/vendor/composer/autoload_real.php on line 69 PHP Fatal error: require(): Failed opening required 'phar:///home/ubuntu/masquerade.phar/vendor/composer/../symfony/translation/Resources/functions.php' (include_path='.:/usr/share/php') in phar:///home/ubuntu/masquerade.phar/vendor/composer/autoload_real.php on line 69
Am I missing a step?
I am trying to run it on Ubuntu 20.04 with php 7.3.26
Thanks
Rob
After closing #6 , the syntax has become a little bit unwieldy;
creditmemo:
sales_creditmemo:
columns:
customer_note:
formatter:
name: sentence
transaction_id:
formatter:
name: randomNumber
nbDigits: 8
A better option would be to also allow this;
creditmemo:
sales_creditmemo:
columns:
customer_note:
formatter: sentence
transaction_id:
formatter: randomNumber
So we can use the simplified syntax when not passing on any arguments to the formatter.
First at all, thanks for this wonderful project.
I am having an issue with a primary key having the zero value
(It's an existing database, don't ask me how it happens)
Given the following schema:
CREATE TABLE `my_table` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`label` varchar(255) COLLATE utf8mb3_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=3745 DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_unicode_ci
I insert a record with a primary key set to zero:
TRUNCATE my_table;
ALTER TABLE my_table CHANGE id id INT(10) UNSIGNED NOT NULL;
INSERT INTO my_table (id, label) VALUES (0, 'test 0');
INSERT INTO my_table (id, label) VALUES (1, 'test 1');
select * from my_table
If I run the tool with the config below, the entry 0 is not altered:
entry:
my_table:
columns:
label:
formatter: firstName
Other entry are changed
@johnorourke I just ran into this;
PHP Notice: Undefined index: columns in phar:///usr/bin/masquerade/src/Elgentos/Masquerade/DataProcessor/DefaultDataProcessorFactory.php on line 27
PHP Warning: Invalid argument supplied for foreach() in phar:///usr/bin/masquerade/src/Elgentos/Masquerade/DataProcessor/DefaultDataProcessorFactory.php on line 27
PHP Notice: Array to string conversion in phar:///usr/bin/masquerade/src/Elgentos/Masquerade/Console/SymfonyOutput.php on line 92
Deleting records from Array table
In Connection.php line 669:
SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error i
n your SQL syntax; check the manual that corresponds to your MySQL server v
ersion for the right syntax to use near 'as `main`' at line 1 (SQL: delete
from `swissup_checkoutfields_values` as `main`)
In PDOConnection.php line 72:
SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error i
n your SQL syntax; check the manual that corresponds to your MySQL server v
ersion for the right syntax to use near 'as `main`' at line 1
In PDOConnection.php line 67:
SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error i
n your SQL syntax; check the manual that corresponds to your MySQL server v
ersion for the right syntax to use near 'as `main`' at line 1
This is the yaml file I'm using; https://github.com/elgentos/masquerade/wiki/Swissup-YAML-file
Any idea?
The idea is that we will delete all of the older customer data (for example, delete customers that have been created more than 30 days ago), so that the DB dump will be a lot smaller + reducing Masquerade execution time. The remaining data should be anonymized so we can use it anymwhere.
Example config:
customer_grid_flat:
provider:
delete: true
where: "`created_at` < now() - interval 30 day"
columns:
name:
formatter: name
email:
formatter: email
unique: true
nullColumnBeforeRun: true
dob:
formatter: dateTimeThisCentury
optional: true
billing_full:
....
Currently, Masquerade just executes the delete, then it moves on to the next table, leaving the remaining records in the table anonymized. Very logical, but it would be nice to have the possibility to delete AND anonymize.
What would be the best place to implement this feature?
Only for;
vatId
)taxpayerIdentificationNumber
)For example, running the en_US locale on magento2 platform now gives an error for quote, customer & order groups.
Illuminate v6.20 no longer provides the array_get
and array_first
helper functions so the code won't run.
Pull request coming shortly!
Haven't tried it yet, but I was wondering if it's possible to run on separate dumps from a split database environment?
There would be 3 dumps:
In other words, does it rely on all the usual tables being present?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.