After running a ruby rep migration we noticed that the pkey sequences aren't being updated so that the next value is +1 or +1 step from the last in use.

This probably falls outside of the primary usages of rubyrep but, thought we would identify it for you.

rubyrep V1.1.2
Java Version: 1.6.0.20-b02

seams rubyrep is not capable to sync tables with columns of array data type bigint[].

can rubyrep handle array data types?

hello!

If I try to make simultaneousely conflicting update in both tables
rubyrep_left=> BEGIN ;
BEGIN
rubyrep_left=> UPDATE aa SET name='6' WHERE id=5;
UPDATE 1
rubyrep_left=> COMMIT ;

rubyrep_right=> BEGIN ;
BEGIN
rubyrep_right=> UPDATE aa SET name ='7' WHERE id=5;
UPDATE 1
rubyrep_right=>
rubyrep_right=> COMMIT ;
COMMIT

rubyrep crashes:

Verifying RubyRep tables
Checking for and removing rubyrep triggers from unconfigured tables
Verifying rubyrep triggers of configured tables
Starting replication

2011-06-29T18:16:39+04:00 Exception caught: replication run timed out
/Users/e/Downloads/rubyrep-1.2.0/lib/rubyrep/replication_runner.rb:125:in execute_once' /Users/e/Downloads/rubyrep-1.2.0/lib/rubyrep/replication_runner.rb:138:inexecute'
/Users/e/Downloads/rubyrep-1.2.0/lib/rubyrep/replication_runner.rb:158:in run' /Users/e/Downloads/rubyrep-1.2.0/lib/rubyrep/command_runner.rb:102:inrun'
./bin/rubyrep:7:in `(root)'

2011-06-29T18:52:20+04:00 Exception caught: replication run timed out
Exception caught: no connection to 'right' database

Hi!

I found bug in synchronization when in primary key is special character.

I have two databases and in both is table "wynikiwzleceniu".
Table wynikiwzleceniu have 3 colummns: idzlecenia(integer), idwyniku(integer), kod(varchar).
PRIMARY KEY ("idzlecenia", "idwyniku", "kod");
in that table i have only two records:
|idzlecenia|idwyniku|kod|

|1|1|CHOL|

|1|1|C-PEPT|

In both database records are identical. When i try synchronize that table i have error:
Exception caught: ActiveRecord::ActiveRecordError: ERROR: duplicate key value violates unique constraint "wynikiwzleceniu_pkey": insert into "wynikiwzleceniu"("idzlecenia", "idwyniku", "kod") values(1, 1, 'C-PEPT')

Problem disappear when i remove "-" from C-PEPT in database. Then table in both database looks that:
|idzlecenia|idwyniku|kod|

|1|1|CHOL|

|1|1|CPEPT|

Well i was try with another special character and for example with that combinations are problem:
C*PEPT, C+PEPT, C?PEPT, C=PEPT, C/PEPT, C!PEPT
i dont check all but here example where synchronization working good (recognize that records are identical and don't try insert):
C\PEPT, C[PEPT

i was found another important thing, problem is only when "C-PEPT" have one of that special char and the second record first letter i same ("C"). When i change first letter to another problem disappear. Also when second record have only one letter it working good. I give some examples:
ERROR:
|1|1|CHOL|

|1|1|C-PEPT|

GOOD:
|1|1|ACHOL|

|1|1|C-PEPT|

ERROR:
|1|1|ACHOL|

|1|1|A-PEPT|

GOOD:
|1|1|B|

|1|1|B-PEPT|

ERROR:
|1|1|BBB|

|1|1|B-PEPT|

i run command ./rubyrep sync -c replication.conf

Here my config file:

RR::Initializer::run do |config|
config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'FULL',
:username => 'sysdba',
:password => '****',
:host => '192.168.56.135',
:port => '5432',
:schema_search_path => 'sysdba',
:encoding => 'win1250'
}

config.right = {
:adapter => 'postgresql',
:database => 'U_SPEC',
:username => 'sysdba',
:password => '****',
:host => '192.168.56.2',
:port => '5432',
:schema_search_path => 'sysdba',
:encoding => 'win1250'
}
config.options[:database_connection_timeout] = 600
config.options[:rep_prefix] = 'ubocze'
config.options[:proxy_block_size] = 200
config.options[:row_buffer_size] = 200
config.options[:commit_frequency] = 200
config.options[:replication_interval] = 10

config.options[:logged_replication_events] = [
:ignored_changes,
:all_conflicts
]    

config.include_tables 'wynikiwzleceniu'

P.S. Sorry for my bad english

when using value 'infinity' for timestamp column rubyrep crashes on

Exception caught: ActiveRecord::ActiveRecordError: ERROR: timestamp out of range: ...., '292278994-08-16 18:00:00', ...)

which is probably max timestamp

I have rubyrep trying to replicate one-way from mysql (left) to postgres (right) on a fairly large database (about 120,000,000 rows) and i am getting some strange errors in my application log. My end-goal is that i want a pristine copy of my mysql data transposed into postgres and i want to switch to the bigger and better machine running postgres in as little downtime as possible.

Snippet of error:

ActiveRecord::StatementInvalid: Mysql2::Error: Duplicate entry '17486' for key 1: INSERT INTO interventions (care_plan_id, intervention_template_id, resolution, start_date, end_date, created_at, updated_at, agency_id) VALUES (...)

Here is what is in rr_pending_changes for id 17486:

mysql> select * from rr_pending_changes where id=17486
-> ;
+-------------------------------+-------------+----------------+-------------+---------------------+-------+
| change_table | change_key | change_new_key | change_type | change_time | id |
+-------------------------------+-------------+----------------+-------------+---------------------+-------+
| hospice_aide_task_activations | id|22598061 | NULL | I | 2011-09-26 23:21:57 | 17486 |
+-------------------------------+-------------+----------------+-------------+---------------------+-------+
1 row in set (0.00 sec)

So, i think that two users are updating two different tables simultaneously and the changes table is not updating the bigint(20) autoincrement column fast enough, i guess. Is this something i am going to have to deal with in mysql or is there something i can do to make this work so the users can update at the same time?

Also, here are the relevant bits of my .conf file:

config.options[:sync_conflict_handling] = :left_wins
config.options[:auto_key_limit] = 2
config.options[:right_record_handling] = :delete
config.options[:adjust_sequences] = false
config.options[:replication_conflict_handling] = :left_wins
config.options[:right_change_handling] = :ignore

I am still in the initial scan/sync (which has not finished) and based on previous experiments it takes about 10 hours to complete. will this issue perhaps go away once the sync is finished?

Followed the tutorial I got the following error while running replicate.
I have the latest rubyrep 1.1.2.
Exception caught: ActiveRecord::ActiveRecordError: ERROR: must be owner of database test: CREATE LANGUAGE plpgsql

Situation:

left  # create table reptest (id serial primary key, foo text );
right # create table reptest (id int primary key,    foo text );

Rubyrep errors out in a non-helpful way:

$ rubyrep replicate -c rep.conf
Verifying RubyRep tables
Checking for and removing rubyrep triggers from unconfigured tables
Verifying rubyrep triggers of configured tables
Exception caught: undefined method `[]' for nil:NilClass

I have tried on a variety of servers. (A variety of ways connecting to localhost, 127.0.0.1, )
root can connect to the DBs just fine from the mysql command line.

running mysql-server-5.0.77-4.el5_6.6. Trying to run multiple instances on a single server.

[root@mysql-test1 rubyrep-1.2.0]# ./rubyrep scan -c myrubyrep.conf Exception caught: no connection to 'left' database

120412 14:52:04 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/ramcache/mysql4406/mysql4406.sock' port:
3306 Source distribution
120412 14:53:05 [Warning] Aborted connection 2 to db: 'ts631_keys'
user: 'root' host: 'localhost' (Got an error reading communication
packets)
120412 14:56:05 [Warning] Aborted connection 3 to db: 'ts631_keys'
user: 'root' host: 'localhost' (Got an error reading communication
packets)
120412 14:57:30 [Warning] Aborted connection 4 to db: 'ts631_keys'
user: 'root' host: 'localhost' (Got an error reading communication
packets)

Cannot connect... Any ideas?

----------------------------------------my.cnf
[root@mysql-test1 rubyrep-1.2.0]# more /etc/my.cnf
[mysqld]

[mysqld_multi]

socket=/var/lib/mysql/mysql6.sock

mysqld = /usr/bin/mysqld_safe
mysqladmin = /usr/bin/mysqladmin
log = /var/log/mysqld_multi.log

[mysqld1]
datadir=/var/lib/mysql
pid-file=/var/lib/mysql/mysqld.pid
socket=/var/lib/mysql/mysql.sock
user=mysql

Default to using old password format for compatibility with mysql 3.x

clients (those using the mysqlclient10 compatibility package).

old_passwords=1
port=4406
log-error=/var/log/mysql1.log
log-warnings

Disabling symbolic-links is recommended to prevent assorted security risks;

to do so, uncomment this line:

symbolic-links=0

[mysqld2]
datadir=/ramcache/mysql4406
pid-file=/ramcache/mysql4406/mysqld.pid
socket=/ramcache/mysql4406/mysql4406.sock
user=mysql

Default to using old password format for compatibility with mysql 3.x

clients (those using the mysqlclient10 compatibility package).

old_passwords=1
port=3306
log-error=/var/log/mysqld4406.log
log-warnings

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
--------------------------------------myrubyrep.conf
[root@mysql-test1 rubyrep-1.2.0]# more myrubyrep.conf
RR::Initializer::run do |config|
config.left = {
:adapter => 'mysql', # or 'mysql'
:database => 'ts631_keys',
:username => 'root',
:password => 'eruces',
:host => 'mysql-test1.medna.com',
:port => 3306
}

config.right = {
:adapter => 'mysql',
:database => 'ts631_keys',
:username => 'root',
:password => 'eruces',
:host => 'localhost',
:port => 4406
}

#config.include_tables 'dept'
#config.include_tables /^e/ # regexp matching all tables starting with e
config.include_tables /./ # regexp matching all tables in the database

end

[root@mysql-test1 rubyrep-1.2.0]# mysqld_multi report
Reporting MySQL servers
MySQL server from group: mysqld1 is running
MySQL server from group: mysqld2 is running
-----------------------------firewall off, selinux off.

I know that this is a little contrived but I believe that I have found a bug.

I have two databases, left and right. I am replicating right to left with the following configuration

RR::Initializer::run do |config|
  config.left = {
    :adapter  => 'postgresql',
    :database => 'hands_development',
    :username => 'xxxxxxxx',
    :password => 'xxxxxxxx',
    :host     => '127.0.0.1',
    :encoding => 'utf8'
  }

  config.right = {
    :adapter  => 'postgresql',
    :database => 'hands_development',
    :username => 'xxxxxxxx',
    :password => 'xxxxxxxx',
    :host     => 'db.example.com',
    :encoding => 'utf8'
  }

  config.options[:sync_conflict_handling] = :right_wins
  config.options[:replication_conflict_handling] = :right_wins
  config.options[:commit_frequency] = 1000
  config.options[:replication_interval] = 1
  config.include_tables 'records'
end

I also have a little script that puts data into the records table on the right database. Replication works fine. I stopped everything and deleted the contents of the records table in the left database (=> delete from records;).

I then restarted replication which did nothing - ie it did not attempt to repopulate the rows I had deleted in left from right. I then started the script to add data to the right database.

What then happened was that as data was being added to the right and replicated to the left the rows that had been deleted from the left were now being deleted from the right. Ultimately the only records left in the two databases were the record inserted by the script from it's last run.

Given the right_wins policy I would have thought that this should not have happened.

If however I drop the left database and create it anew then replication proceeds as expected - ie all the data from right flows into the left.

I will continue to fiddle, I have some suspicions that it could be something to do with sequences.

When exactly are the trigger functions re-created? This is just from today and looks very bizzarre:

$ grep 'CREATE OR REPLACE FUNCTION "rr_reptest"' postgresql-Thu.log | awk '/^2009-08-07/{print $1 " " $2}'
2009-08-07 06:59:03.666
2009-08-07 07:06:33.993
2009-08-07 07:07:38.042
2009-08-07 08:20:29.359
2009-08-07 08:26:15.617
2009-08-07 08:34:32.982
2009-08-07 08:34:36.011
2009-08-07 08:35:29.027
2009-08-07 08:36:52.088
2009-08-07 08:37:21.111
2009-08-07 08:37:54.163
2009-08-07 08:52:47.806
2009-08-07 08:53:46.853
2009-08-07 08:54:14.872
2009-08-07 11:13:59.298
2009-08-07 11:14:10.303
2009-08-07 11:15:00.336
2009-08-07 12:02:58.548
2009-08-07 12:07:17.744
2009-08-07 12:30:23.755
2009-08-07 12:31:49.795
2009-08-07 12:33:59.930

It shows conflicting records, which are the same, and showing two different records, which are the same...

[root@db1 rubyrep]# ./rubyrep scan --detailed=full -c env.conf    
                                 env 100% .........................   4
--- 
:left: 
  value: Vodafone
  name: SMSC
--- 
:conflict: 
- value: "18"
  name: logPageSize
- value: "18"
  name: logPageSize
--- 
:conflict: 
- value: "24"
  name: messageThreadPageSize
- value: "24"
  name: messageThreadPageSize
--- 
:right: 
  value: Vodafone
  name: SMSC

If I try to do sync:

[root@db1 rubyrep]# ./rubyrep sync --detailed=full -c env.conf              
                                 env   1% --- 
:left: 
  value: Vodafone
  name: SMSC
Exception caught: ActiveRecord::ActiveRecordError: ERROR: duplicate key value violates unique constraint "env_pkey": insert into "env"("name", "value") values('SMSC', 'Vodafone')

Hi, i'm thinking one thing.Have the table for Log a rotate system ? Or a clean script ?
thank a lot
franz

Situation:

• left: record inserted
• right: insert failed for whatever reason, got logged to left.rr_logged_events
• left: record updated
• rigt: attempts update, does not check if the row got modified

Result: the row never gets picked up.

Casual re-sync on a large table/busy database is not an option.

Additionally, the attempt_update() function calls log_replication_outcome before actually trying to do the update -- this doesn't look right...

Setting up replication right now will break permissions for non-superusers (unless you "grant insert" on rr_pending_changes and "select,usage" on its sequence to PUBLIC -- not very secure).

Could be fixed by making the trigger functions SECURITY DEFINER.

This is a critical one... :-/

Hi, we are users and admirers of the solution database rubyrep. When performing an replication instalation, the code below is not generated immediately on their rubyrep triggers. Is there any way it is generated at install time?

        PERFORM ACTIVE FROM rr_running_flags;
        IF FOUND THEN
          RETURN NULL;
        END IF;

Thank's any way,

Emmanuel Ruiz Lopes

When trying to do a scan or sync of semi large table, I get something like this:

/rubyrep scan -c sync.conf
device_port_power 15% ...1894

then it just kicks me back, the sync will correctly sync the table over to the slave, but it seems to get hung up at that one point.

In a postgresql table, I have a UNIQUE KEY defined using two columns, one of which contains a NULL value. When I run scan, etc., I get:
Exception caught: undefined method `<=>' for nil:NilClass
I'm specifying the :key pref as instructed, pointing to both of the key columns.
It looks like the bad comparison is line 31 of table_scan_helper.rb (v1.1.2-14-g0fd7954). Has this been discussed elsewhere?

I am having problem with PostgreSQL rubyrep 1.1.2 (JRuby package). Columns containing numeric values are truncated on replicate e.g. if table A in left database sets A value to 0.5 it comes out in right database as 0.0. The only thing abnormal about my configuration is that I am using two different postgresql versions. The configuration is:

left: PostgreSQL 8.4.4
right: PostgreSQL 8.3.11

When setting up the replication, I took care to "alter database set connection limit 0", then pg_dump the replicated tables, start rubyrep and "set connection limit -1"; this took about 1 minute.

Yet rubyrep insists on "Executing initial table syncs" when I start the replication... which takes 15 minutes for just one of the tables (600k rows, pg_relation_size reports 100MB).

An option to turn the "initial syncs" off would be handy.

With a config file of

RR::Initializer::run do |config|

MASTER

config.left = {:adapter=>"mysql", :username=>"root", :password=>"", :host=>"localhost", :database=>"server_production"}

BRANCH

config.right = {:adapter=>"mysql", :username=>"root", :password=>"", :host=>"my.servers.name.co.uk", :database=>"server_production"}

config.options[:left_record_handling] = :insert
config.options[:right_record_handling] = :delete
config.options[:sync_conflict_handling] = :left_wins

config.include_tables /./ # regexp matching all tables in the database
config.exclude_tables /schema_migrations/
end

I get

RAILS_ENV=server_production bundle exec rubyrep --verbose sync -c /tmp/foo.delete.me.aj.txt

Exception caught: !!! Missing the mysql2 gem. Add it to your Gemfile: gem 'mysql2'

If I use 'mysql2' in the config file I get

Exception caught: No ConnectionExtender available for :mysql2

mysql2 is in my Gemfile.lock:

$ grep mysql2 Gemfile.lock |grep =
mysql2 (= 0.2.6)!
$ grep rubyrep Gemfile.lock |grep =
rubyrep (= 1.2.0)

There were some encoding problems, so replication of some rows has failed and got logged. I've fixed the problems and wanted to get the rows replicated (can't just UPDATE them).

insert into rr_pending_changes
    select change_table, change_key, null, left_change_type, event_time
        from rr_logged_events;

rubyrep has stopped replicating anything. deleting these rows from rr_pending_changes did not solve the issue; I had to restart rubyrep.

scan and sync commands are successfull and table is synchronised just fine.
Running replicate command starts successfully but does not replicate changes.

rr_logged_events reports:

activity: replication
left_change_type: update
right_change_type: no_change
description: source record for update vanished
long description:

Exception
./bin/../lib/rubyrep/replicators/two_way_replicator.rb:323:in replicate_difference' ./bin/../lib/rubyrep/replicators/two_way_replicator.rb:254:inattempt_update'
./bin/../lib/rubyrep/replicators/two_way_replicator.rb:340:in replicate_difference' ./bin/../lib/rubyrep/replicators/two_way_replicator.rb:254:inattempt_update'
./bin/../lib/rubyrep/replicators/two_way_replicator.rb:340:in replicate_difference' ./bin/../lib/rubyrep/replication_run.rb:56:inrun'
./bin/../lib/rubyrep/replication_run.rb:49:in loop' ./bin/../lib/rubyrep/replication_run.rb:49:inrun'
./bin/../lib/rubyrep/replication_runner.rb:117:in execute_once' ./bin/../lib/rubyrep/task_sweeper.rb:14:intimeout'
./bin/../lib/rubyrep/task_sweeper.rb:62:in timeout' ./bin/../lib/rubyrep/task_sweeper.rb:60:ininitialize'

diff_dump:

--- !ruby/object:RR::ReplicationDifference
type: :left
loaded: true
changes:
:right: !ruby/object:RR::LoggedChange
table: os_mdata_gates
new_key:
key_sep: '|'
type: :no_change
key:
employee_id: "111111"
rfid:
database: :right
:left: !ruby/object:RR::LoggedChange
table: os_mdata_gates
new_key:
employee_id: "111111"
rfid:
key_sep: '|'
type: :update
key:
employee_id: "111111"
rfid:
database: :left
last_changed_at: 2009-12-01 10:39:49.120825 +00:00
first_changed_at: 2009-12-01 10:39:49.120825 +00:00

Hi Arndt,
First thanks for this wonderful solution.
I checked it locally, and then I loaded it on my server.

I created 2 DB (postgres) there, one rubyleft and one rubyright.
I loaded the data in rubyleft and rubyright was just loaded with the schema(no data).
I then configure the redmine as following.

RR::Initializer::run do |config|
config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'rubyleft',
:username => 'username',
:password => 'password',
:host => '127.0.0.1'
}

config.right = {
:adapter => 'postgresql',
:database => 'rubyright',
:username => 'username',
:password => 'password',
:host => '127.0.0.1'
}

config.options[:sync_conflict_handling]=:left_wins

config.options[:replication_conflict_handling]=:left_wins

config.options[:auto_key_limit]=75
end

I then ran these commands to sync the data.

./rubyrep-1.0.9/rubyrep sync -c redmine.conf
It doesnt give me anything and comes back to shell command

the result is same with scanning too
./rubyrep-1.0.9/rubyrep scan -c redmine.conf

It also seems to be working with replication, but actually it doesnt.
./rubyrep-1.0.9/rubyrep replicate -c redmine.conf

It shows me that its working fine. and shows 'Starting Replication'

Can you please tell me what to check, as nothing is working but it shows it works.

thanks
Nirah

Situation:

• left: encoding=latin2, client_encoding=windows-1250
• right: encoding=latin2, client_encoding=latin2

Rubyrep does not set client_encoding, so it'll get the data from left as windows-1250 and feed it to right claiming it's latin2. Result: broken encoding after replication.

https://github.com/rubyrep/rubyrep/tags doesn't list 1.2.0 tag and downloads offers 1.1.2 while 1.2.0 is available from http://rubyforge.org/frs/?group_id=7932&release_id=45695

The following may get prepended to a trigger. I'm not sure the situation where this happens but I believe it is when a single table is being replicated.

Regardless the schema name needs to come before rr_running_flags, e.g. "myschema".rr_running_flags.

        BEGIN
        PERFORM ACTIVE FROM rr_running_flags;
        IF FOUND THEN
          RETURN NULL;
        END IF;

Hi,
I just created 2 new databases. I also created one similar table in both of them.
And I started the rubyrep replication.

It gave me the follwing error and stopped.
ActiveRecord::ActiveRecordError: ERROR: syntax error at end of input: CREATE LANGUAGE plpgsql

Please give me the solution for this.
There is no bulk data , nothing and it isnt working.

Thanks for your support in advance

Niraj

First, excuse my English.

In some transactions, where happen respectively:

1 Insert in master table;
2 Insert in detail table;
3 Update in master table;

For some reason, Rubyrep is inverting the order of id, in the rr_pending_changes table. Is getting that way:

3 Update in master table; (record doesn't exists yet, in the target database)
2 Insert detail table; (error, violation of foreign key, Detail table --> Master Table)
1 Insert in master table (ok, but without detail record)

Does anyone have any idea what might be?

Regards,
Tales Macêdo
Brazil

take the following table definition:

CREATE TABLE tags
(
  tgtable character varying(100) NOT NULL,
  tglang character varying(3) NOT NULL,
  tgtag character varying(255) NOT NULL,
  tgcountuser integer,
  tgcountentries integer,
  CONSTRAINT tags_pk PRIMARY KEY (tgtable, tglang, tgtag)
)
WITH (
  OIDS=FALSE
);

sync (and replication) fails with a "duplicate key" error:

% ./rubyrep sync -c myrubyrep.conf 
                                tags  46% ...........Exception caught: ActiveRecord::ActiveRecordError: ERROR: duplicate key value violates unique constraint "tags_pk": insert into "tags"("tgtable", "tglang", "tgtag", "tgcountuser", "tgcountentries") values('locations', 'de', 'abgedreht', NULL, 5)

and without surpize, a scan shows all rows to be different (even though both tables on the left and on the right side are identical):

./rubyrep scan -c myrubyrep.conf 
                                tags 100% ......................... 2228

When syncing a PostgreSQL database containing a bytea field, data is only copied up to the first null byte. Anything after is silently discarded.

Probably a C string copy function used in there somewhere.

rubyrep scan -c myrubyrep.conf

It seems column name is missing from the 'order by' clause.

substance Exception caught: ActiveRecord::ActiveRecordError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'limit 1000' at line 1: select id, name from substance order by limit 1000

mysql> desc substance
| Field | Type | Null | Key | Default | Extra |

| id | bigint(20) | NO | PRI | NULL | auto_increment |

| name | longtext | YES | | NULL | |

When trying to replicate or sync(when there is data) data between two tables to share authentication information, the following error occurs:

Verifying RubyRep tables
Checking for and removing rubyrep triggers from unconfigured tables
Verifying rubyrep triggers of configured tables
Exception caught: ActiveRecord::ActiveRecordError: ERROR: unterminated dollar-quoted string at or near "$change_trigger$
BEGIN

          IF (TG_OP = 'DELETE') THEN
            INSERT INTO "public".rr_pending_changes(change_table, change_key, change_type, change_time)
              SELECT 'cookie', 'cookie_id|' || OLD."cookie_id", 'D', now()":           CREATE OR REPLACE FUNCTION "rr_cookie"() RETURNS TRIGGER AS $change_trigger$
        BEGIN

          IF (TG_OP = 'DELETE') THEN
            INSERT INTO "public".rr_pending_changes(change_table, change_key, change_type, change_time)
              SELECT 'cookie', 'cookie_id|' || OLD."cookie_id", 'D', now();
          ELSIF (TG_OP = 'UPDATE') THEN
            INSERT INTO  "public".rr_pending_changes(change_table, change_key, change_new_key, change_type, change_time)
              SELECT 'cookie', 'cookie_id|' || OLD."cookie_id", 'cookie_id|' || NEW."cookie_id", 'U', now();
          ELSIF (TG_OP = 'INSERT') THEN
            INSERT INTO  "public".rr_pending_changes(change_table, change_key, change_type, change_time)
              SELECT 'cookie', 'cookie_id|' || NEW."cookie_id", 'I', now();
          END IF;
          RETURN NULL; -- result is ignored since this is an AFTER trigger
        END;
      $change_trigger$ LANGUAGE plpgsql;

Aditional Information:
Database: pgsql 8.4.4
OS: RHEL 5.5

Hi,

I'm trying to get rubyrep running on mac osx 10.6, under ruby 1.9.2-p290. The gem installs ok, and I can generate configuration files, but when I try to scan or sync, rubyrep simply outputs: Trace/BPT trap, and exits.

I suspect this is because I am using the pg adapter instead of postgres as rubyrep dictates. I suppose I was hoping they would both work. I'm told that the postgres is essentially deprecated in favour of pg, and I've found I can't even install postgres: gem install postgres gives:

extconf.rb:4:in `<main>': uninitialized constant Object::PLATFORM (NameError)

Should I be able to use rubyrep with pg? If not, should I try creating a fork to do that? If not, how do I get postgres gem installed and working together with pg?

It looks like a really great tool, by the way! I'm looking forward to using it once I get it running.

Cheers,
Mitch

Hi,

I am doing a test. when there is something around 700,000 rows in rr_pending_changes, when I run the replicate command it frequently gives me,

"java.lang.OutOfMemoryError: GC overhead limit exceeded"
and
"Exception caught: replication run timed out"

error, I also increased the HEAP size to prevent an error related to that.

what is the mechanism behind it, can the script select from that table row by row and replicated the changes?

Hi everyone, I'm getting this error on a particular table and on always the same amount of #percentage everytime I try to run $rubyrep sync -c.

$ rubyrep sync -c db_replication.conf
user_group_login 100% ......................... 0
user 18% ....Exception caught: incomplete multibyte character

Please let me know what else you need to know?

I just wanted to know why i'm getting this "Exception caught" error, for us to be able to resolve this issue.

thanks,
Mark Guadalupe

I got that as soon as I do a rubyrep replicate -c myrubyrep.conf

I checked connectivity to the databases, passwords, etc. Everything is working.

My environment is ruby enterprise 1.8.7, and the postgres gem is pg 0.13.2

The whole message is:

Exception caught: PG::Error: connection is closed: SELECT tablename
FROM pg_tables
WHERE schemaname IN ('"$user"','public')

While doing:

rubyrep scan -c myrubyrep.conf

I get:

Exception caught: PGError: ERROR:
character 0xe2809c of encoding "UTF8" has no equivalent in "LATIN1"
: select "apl_aplid", "apl_articu", "apl_codoem", "apl_mdoid", "apl_anio", "apl_
anio2", "apl_cilin", "apl_versio", "apl_memo" from "apl" where ("apl_aplid") > (
9304) order by "apl_aplid" limit 1000

Both databases are UTF8 encoding !

Is there a way to configure the ruby client no to convert to LATIN1 ?

Thanks !

Hi, I'm a user of rubyrep.

I've been testing the program on a local network, and works perfectly.

The problem is when I try to make a connection through a WAN.

I modified files postgreSQL to accept the necessary connections, I
disabled all firewallsbut still not working.

I tried to change the configuration files rubyrep and still not
working.

I would like to know how I configure it not because I'm doing wrong,
and this programwould be very useful for my final degree project.

I enclose a copy of my config file.

Thanks.

RR::Initializer::run do |config|
config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'prueba',
:schema_search_path => 'pixelcomk09',
:username => 'postgres',
:password => '********',
:host => '192.168.1.99',
:port => '5432'
}

config.right = {
:adapter => 'postgresql',
:database => 'prueba',
:schema_search_path => 'pixelcomk09',
:username => 'postgres',
:password => '********',
:host => '10.0.1.7',
:proxy_host => 'public_ip',
:proxy_port => '5432'
}

config.options[:adjust_sequences] = false
config.options[:rep_prefix] = 'rr1'
config.options[:sync_conflict_handling] = :left_wins
config.options[:replication_conflict_handling] = :left_wins
config.include_tables /./

end

rubyrep-1.1.2

How add multiple schema?

I use in config on both:
:schema_search_path => 'voip,voip2',

then I had:

Exception caught: table '_model' not found

once again:

Exception caught: table 'rr_running_flags' not found

tables,trigers etc. 'rr_running_flags' adds only to first schema. When I used only one schema voip or voip2 then is ok.

I have rubyrep installed and configured to replicate between two postgres databases. However, when attempting to start it, I am getting:

Exception caught: PGError: ERROR: syntax error at or near "," at character 115
QUERY: INSERT INTO "mdbadmin".rr_pending_changes(change_table, change_key, change_type, change_time) SELECT 'emevt_mib', , 'D', now()

Any suggestions?

I am getting the following error when I try to run rubyrep

C:\rubyrep>rubyrep -v
c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:149:in activate': can't activate activesupport (= 2.2.2, runtime), already a ctivated activesupport-2.3.2 (Gem::Exception) from c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:165:inactivate'
from c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:164:in each' from c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:164:inactivate'
from c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:165:in activate' from c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:164:ineach'
from c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:164:in activate' from c:/ruby/lib/ruby/site_ruby/1.8/rubygems.rb:49:ingem'
from c:/ruby/bin/rubyrep:18

Hi,

Today, I was trying to setup rubyrep on a new server, the scan and sync commands work without any problems, but the replicates does :

# rubyrep --verbose replicate -c /usr/local/etc/rubyrep.conf 
Verifying RubyRep tables
Checking for and removing rubyrep triggers from unconfigured tables
Verifying rubyrep triggers of configured tables
Starting replication
Exception caught: Thread#join: deadlock 0x800e5d348 - mutual join(0x802d3c598)
/usr/local/lib/ruby/gems/1.8/gems/rubyrep-1.0.8/bin/../lib/rubyrep/replication_runner.rb:89:in `join'
/usr/local/lib/ruby/gems/1.8/gems/rubyrep-1.0.8/bin/../lib/rubyrep/replication_runner.rb:89:in `pause_replication'
/usr/local/lib/ruby/gems/1.8/gems/rubyrep-1.0.8/bin/../lib/rubyrep/replication_runner.rb:141:in `execute'
/usr/local/lib/ruby/gems/1.8/gems/rubyrep-1.0.8/bin/../lib/rubyrep/replication_runner.rb:152:in `run'
/usr/local/lib/ruby/gems/1.8/gems/rubyrep-1.0.8/bin/../lib/rubyrep/command_runner.rb:102:in `run'
/usr/local/lib/ruby/gems/1.8/gems/rubyrep-1.0.8/bin/rubyrep:7
/usr/local/bin/rubyrep:19:in `load'
/usr/local/bin/rubyrep:19

It's running standard ruby installation on a 64 bit FreeBSD 7.2.

XP, JDK 1.6.0_14. rubyrep 1.0.2., Postgres 8.1

RubyRep compares only the table "keys", which I had to create. Before it complained about this missing table. Obviously it works generally, but only for this table (so when I add 1 entry to one of the databases, it finds that one record as a difference).

Here my Conf-File:
RR::Initializer::run do |config|
config.left = {
:adapter => 'postgresql',
:database => '#####',
:username => '####',
:password => '#####',
:host => 'localhost'
}

config.right = {
:adapter => 'postgresql',
:database => '#####',
:username => '#####',
:password => '#####',
:host => '10.8.0.1'
}

config.include_tables /./

end

Result from rubyrep:
C:\rubyrep>rubyrep scan -d keys -b -c myrubrep.conf

keys ......................... 1

:left:
key: dummykey

I'm getting the following error:

Exception caught: PGError: FEHLER: Syntaxfehler bei »from«
LINE 1: select from "shops" order by "id" limit 1000
^

I really can't think of what the problem is. I've set the encoding correctly.
.left and .right are both postgresql, encoding both UTF-8

Situation: replication is running, I'm stopping it, doing some changes, then issuing "rubyrep sync". I'd expect the rr_pending_changes table to be cleared -- could be quite large.

Some possible implementations:

1. run in the "Repeatable read" transaction mode to avoid losing rows.
2. select * from rr_pending_changes order by change_time asc limit :commit_frequency, then DELETE these rows.

Hi, We are Brazilian Rubyrep users.

When the record do not exists in the target database and an update is sent by Rubyrep, he does not insert the record in the target database. I would like that the record was inserted. Are there any way for do this ?

Excuse my English.

I've been running into an issue where rubyrep (running in jruby) runs out of memory and throws a java exception, no matter how much memory I give it (up to 2GB).

For me, it turned out there was 10 million rows in rr_pending_changes in the slave side of my replication that I wasn't expecting to be there. And while nuking those will probably fix my issue, in general it seems like rubyrep shouldn't have such limitations.

I think it comes down to logged_change.rb, the method update, where it does a "while cursor.next?" loop. It looks like it ends up loading the entire table into memory at once.

Is there any reason there couldn't be a user definable limit here? Say load 1000, process those, and then load the next 1000?
It seems like it already does that on the query itself and the way it loops over the results a row at a time, but it defeats the purpose of doing that if you store an array with an element from every row, you still end up running out of memory.

I don't think I understand what's going on here well enough to submit a patch. Maybe you could just make the while loop break after 1000 iterations and things would just work, but I'm afraid it would break something I don't even know about.

Hi, We'd like to customize the rubyrep source code. How can we build a debugging environment? What are the tools used? Do you have any tips or example to follow?

Thanks any way

substance Exception caught: ActiveRecord::ActiveRecordError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'limit 1000' at line 1: select id, name from substance order by limit 1000

mysql> desc substance
-> ;
+-------+------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| name | longtext | YES | | NULL | |
+-------+------------+------+-----+---------+----------------+