Hello friends

The creation of the contengency table calculation and of the matching table cannot really exceed 100'000 of records or it hit performance and memory problem.

100'000 of records is really too little for our linkages. Most of the files we are processing with linkage process would weight about 2Go and hit 10 millions of line.

the "impl. language" column generates an error when saving in "application service edit"
it seems that this column changes the locale of the application (it or eng)

La table de contingence, et l’indication des motifs acceptés est très utile (pour k variables, la log contient les motifs de longueur k sous forme 11101...011 pour indiquer quelles sont les relâchements de contrainte effectués pour respecter les seuils).
Disposer de cette indication en sortie dans les tables pourrait être utile, soit comme élément qualitatif sur l’appariement, soit pour affiner les paires retenues.

Hello,
when I add a block in the settings of an RL, step 2 no longer converges. I did the test with 3 different variables generating 3, 18 and 101 blocks respectively

I can't get the test to work with the Levenshtein method. (3 variables : name,surname and lastcode)
error in log console
"ERROR: one or more variables give inconsistent estimates. Please, check the variables in the model or try to reduce the search space."

.

Hi,

I'm running into the following problem: the first few times I try to create a new user, it doesn't work.

How to reproduce

• docker-compose rm and docker-compose up or any other way to start the db from scratch
• go to Settings > Gestione utenti > Aggiungi Utente
• fill out the form for a new user and click on Salva => nothing happens
• click again on Salva => nothing happens
• a third time => nothing happens
• the fourth time: it works

POST responses and logs

The first three times, the POST request returns the following result: [{"type":"ERROR","text":"Error: could not execute statement; SQL [n/a]; constraint [is2_users_pkey]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement","details":""}]

And the database logs the following errors:

ERROR: duplicate key value violates unique constraint "is2_users_pkey"
DETAIL: Key (id)=(1) already exists.
STATEMENT: insert into is2_users (email, name, password, role_id, surname) values ($1, $2, $3, $4, $5)
RETURNING *
ERROR: duplicate key value violates unique constraint "is2_users_pkey"
DETAIL: Key (id)=(2) already exists.
STATEMENT: insert into is2_users (email, name, password, role_id, surname) values ($1, $2, $3, $4, $5)
RETURNING *
ERROR: duplicate key value violates unique constraint "is2_users_pkey"
DETAIL: Key (id)=(3) already exists.
STATEMENT: insert into is2_users (email, name, password, role_id, surname) values ($1, $2, $3, $4, $5)

On the fourth try, a new user is successfully created with id = 4

Hello,

The schema-postgresql.sql from src/main/ressources had been removed in a previous commit. We made this file in toulouse so that hibernate could initialize automatically the database on startup
when the following properties are set :
spring.datasource.initialization-mode=always
spring.datasource.initialize=true
spring.jpa.hibernate.ddl-auto=update

This autoinitialization works well with postgres as the script won't do anything if the database is already created. That was very handy when we deployed the application on the cloud or if we want to patch the application.

The other and current option is that the user builds itself the database with the script is2_postgres.sql found in db . Also the current docker compose uses this file and it should works well.
Will our deployements on the cloud use docker image or do we have to add a call to this external script for database initialization ?

Note also that the schema-postgresql.sql is more complicated to update than the is2_postgres.sql

What option do we want ?

It as decided to have the application be responsible for the management of the database, at least for the initialisation.

A clear enhancement of that would be to use a tool for managing migration and version of the database.

Two mainly tools in the Java landscape:

• liquibase
• flyway

The files are quite large so I will have to give them the test case by another way than github.

Before that, I was thinking it is maybe just a usage of blocking variable problem. Could you tell me if the files have to be ordered on the blocking key before processing with RELAIS ?

Hello friends,

Few issues from the BPE reuse case. I will try to contribute a bit in the next weeks.

• columns are not in the same order for matching and possible_matching output tables. Same annoyance for all tables in general. Order by alphabetical name ?

hi,
Why does lowering the threshold of a variable lead to the non-convergence of step 2?
How is the window variable used in the parameters of a variable (and what is it for)?

Hello everybody,

For ou BPE use case, i did a deterministic record linkage and i would like to use the 2 residual outputs as input of another deterministic record linkage.
Tou when i want to select input table with "carica tabella" button, no tables are available in the menu. Could you explain me how it works or could you check as it is maybe a bug ?

Thank you very much and how to see you soon

no log in tomcat.

it is not possible to choose the encoding of the file to load (UTF-8 required). Is it possible to envisage an evolution which requires encoding?

when adding a threshold (Record Linkage / parameters / THRESHOLD MATCHING or UNMATCHING), if you press the enter key on the keyboard, it is not saved.

with only 2 variables (name and lastname) on test files, Fellegi Sunter process has the following error

org.renjin.eval.EvalException: Exception calling list(name = modelframe, address = org.renjin.sexp.ExternalPtr@1d9f570, numParameters = c(4L)) : variable lengths differ (found for 'V2')

On peut suspecter des problèmes dans cet algo.
En analysant quelques paires différentes entre les propositions de Rapsodie et de Relais.

Hello friends,

The residual result looks wrong.

Here is a test case :

app_sirene.zip

Hello friends,

The jarowinkler distance used in the linkage looks good for single word (especially because the 4 first letters are given a big weight) but could you tell me what is the distance you are commonly use to compare sentence (=set of words).

Thank you

Hello friends,

"add a delete all button" or "allow multiselection" / delete for the variable used in the linkage.

I think "Delete all" would be a great and more simple at first.

https://github.com/mecdcme/is2/blob/master/db/is2-postgres.sql

you can find the last version of the database schema and metadata.
Check if the tables is2_gsbpm_process,is2_business_service ,is2_workset and is2_step_runtime are different with your version.
Maybe it's better if you save your data and recreate the database.
Did you also download the latest version of the code?

----- Messaggio originale -----
It looks like it is in the gsbpm table when finding parent process; parent colmn is in int8 tough so i don't understand still

Hibernate: select gsbpmproce0_.id as id1_14_0_, gsbpmproce0_.descr as descr2_14_0_, gsbpmproce0_.name as name3_14_0_, gsbpmproce0_.active as active4_14_0_, gsbpmproce0_.parent as parent6_14_0_, gsbpmproce0_.order_code as order_co5_14_0_, gsbpmproce1_.id as id1_14_1_, gsbpmproce1_.descr as descr2_14_1_, gsbpmproce1_.name as name3_14_1_, gsbpmproce1_.active as active4_14_1_, gsbpmproce1_.parent as parent6_14_1_, gsbpmproce1_.order_code as order_co5_14_1_ from is2_gsbpm_process gsbpmproce0_ left outer join is2_gsbpm_process gsbpmproce1_ on gsbpmproce0_.parent=gsbpmproce1_.id where gsbpmproce0_.id=?

I cannot see what is wrong; i tried to set all id with bigint type but no sucess;

Here is the error message :

org.springframework.dao.InvalidDataAccessApiUsageException: Provided id of the wrong type for class it.istat.is2.workflow.domain.BusinessService. Expected: class java.lang.Long, got class java.lang.Integer; nested exception is java.lang.IllegalArgumentException: Provided id of the wrong type for class it.istat.is2.workflow.domain.BusinessService. Expected: class java.lang.Long, got class java.lang.Integer

During the Rome hackathon, Docker and Docker Compose files for IS2 were produced. They lives in the ESSNet forked repo.

But it seems this work was not integrated to the official IS2 repo on GitHub.

It will be nice to avoid diverging too much if we dont want this previous work to be spoiled.

in Record Linkage / settings / variables / bind role
the first time you open the window, the checkbox is selected but not active. it works fine afterwards.

When you want to set the role of variables to MatchingA or MatchingB, if you open close the control or try to select variable then unselect them, the control bugs totally; Also there will be duplicated record in the database

The "Fellegi Sunter" step doesn't seem to create the Fellegi Sunter table required for the last step "Matching Table"

if you create a new data process in a session that already contains one, the new one displays the information from the previous one (console log, status) and the reset and clean buttons do not work.

following a Deterministic Record Linkage on 2.5 million lines, the interface is not able to display the result:

however, downloading the csv is still possible but we have doubts about the accuracy of the data because the number of lines does not correspond to the query we do in the database:
drop table ttt;
create temporary table ttt as
select name, json_array_elements (content :: json) from is2.is2_dataset_column
where name = 'columnName' and id = integer_value