proemion / repmgr-agent Goto Github PK
View Code? Open in Web Editor NEWDeployment agent for repmgr
License: GNU General Public License v3.0
Deployment agent for repmgr
License: GNU General Public License v3.0
Today while we did https://github.com/Proemion/it-internal/issues/593 we've noticed that pgdeploy is not working properly.
Those errors appeared within the normal switchover commands:
DETAIL: promoting server "frapspdb_n0" (ID: 1) using pg_promote()
NOTICE: waiting up to 60 seconds (parameter "promote_check_timeout") for promotion to complete
NOTICE: STANDBY PROMOTE successful
DETAIL: server "frapspdb_n0" (ID: 1) was successfully promoted to primary
INFO: executing notification command for event "standby_promote"
DETAIL: command is:
/usr/share/repmgr-agent/pgdeploy -c proemion -p port=5432 -P
Exception in thread 10.10.2.64:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/share/repmgr-agent/pgdeploy", line 110, in safe_deploy
return self.deploy(remote_host)
File "/usr/share/repmgr-agent/pgdeploy", line 131, in deploy
with shell.open(cluster_config, "wb") as pgbouncer:
File "/usr/lib/python3/dist-packages/spur/ssh.py", line 234, in open
sftp_file = SftpFile(sftp, sftp.open(name, mode), mode)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 327, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 730, in _request
return self._read_response(num)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 781, in _read_response
self._convert_status(msg)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 807, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
Exception in thread 172.21.70.153:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/share/repmgr-agent/pgdeploy", line 110, in safe_deploy
return self.deploy(remote_host)
File "/usr/share/repmgr-agent/pgdeploy", line 131, in deploy
with shell.open(cluster_config, "wb") as pgbouncer:
File "/usr/lib/python3/dist-packages/spur/ssh.py", line 234, in open
sftp_file = SftpFile(sftp, sftp.open(name, mode), mode)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 327, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 730, in _request
return self._read_response(num)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 781, in _read_response
self._convert_status(msg)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 807, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
Exception in thread 172.21.70.152:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/share/repmgr-agent/pgdeploy", line 110, in safe_deploy
return self.deploy(remote_host)
File "/usr/share/repmgr-agent/pgdeploy", line 131, in deploy
with shell.open(cluster_config, "wb") as pgbouncer:
File "/usr/lib/python3/dist-packages/spur/ssh.py", line 234, in open
sftp_file = SftpFile(sftp, sftp.open(name, mode), mode)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 327, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 730, in _request
return self._read_response(num)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 781, in _read_response
self._convert_status(msg)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 807, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
INFO: local node 2 can attach to rejoin target node 1
DETAIL: local node's recovery point: 69DE/61000028; rejoin target node's fork point: 69DE/610000A0
NOTICE: setting node 2's upstream to node 1
WARNING: unable to ping "host=192.168.145.22 user=postgres port=5432 application_name=frapspdb_n1"
DETAIL: PQping() returned "PQPING_NO_RESPONSE"
NOTICE: starting server using "sudo systemctl start [email protected]"
NOTICE: replication slot "repmgr_slot_1" deleted on node 2
WARNING: 1 inactive replication slots detected
DETAIL: inactive replication slots:
- repmgr_slot_3 (physical)
HINT: these replication slots may need to be removed manually
NOTICE: NODE REJOIN successful
DETAIL: node 2 is now attached to node 1
INFO: executing notification command for event "standby_switchover"
DETAIL: command is:
/usr/share/repmgr-agent/pgdeploy -c proemion -p port=5432 -P
Exception in thread 10.10.2.64:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/share/repmgr-agent/pgdeploy", line 110, in safe_deploy
return self.deploy(remote_host)
File "/usr/share/repmgr-agent/pgdeploy", line 131, in deploy
with shell.open(cluster_config, "wb") as pgbouncer:
File "/usr/lib/python3/dist-packages/spur/ssh.py", line 234, in open
sftp_file = SftpFile(sftp, sftp.open(name, mode), mode)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 327, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 730, in _request
return self._read_response(num)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 781, in _read_response
self._convert_status(msg)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 807, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
Exception in thread 172.21.70.152:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/share/repmgr-agent/pgdeploy", line 110, in safe_deploy
return self.deploy(remote_host)
File "/usr/share/repmgr-agent/pgdeploy", line 131, in deploy
with shell.open(cluster_config, "wb") as pgbouncer:
File "/usr/lib/python3/dist-packages/spur/ssh.py", line 234, in open
sftp_file = SftpFile(sftp, sftp.open(name, mode), mode)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 327, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 730, in _request
return self._read_response(num)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 781, in _read_response
self._convert_status(msg)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 807, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
Exception in thread 172.21.70.153:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/share/repmgr-agent/pgdeploy", line 110, in safe_deploy
return self.deploy(remote_host)
File "/usr/share/repmgr-agent/pgdeploy", line 131, in deploy
with shell.open(cluster_config, "wb") as pgbouncer:
File "/usr/lib/python3/dist-packages/spur/ssh.py", line 234, in open
sftp_file = SftpFile(sftp, sftp.open(name, mode), mode)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 327, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 730, in _request
return self._read_response(num)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 781, in _read_response
self._convert_status(msg)
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 807, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
After a pgdeploy run I had to restart pgbouncer manually to get a connection on port 6432.
Currently there is no way to use a single pgbouncer to connect to multiple database instances, because the individual pgdeploy instances will clobber each other's userlist.txt files. We need to implement partial updating using a section marked by special comments to support this.
In past we had run 3 PSQL instances on one server. That's why we needed a differentation of repmgr table. They were called repmgr.claas, repmgr.linde, repmgr.proemion,
In a single PSQL instance environment the DB is called main and the repmgr table is only called repmgr.
See output below:
[10:02]postgres@dev-psql-linde-n1:/root# /usr/share/repmgr-agent/pgdeploy -c main -p port=5432 -P
Traceback (most recent call last):
File "/usr/share/repmgr-agent/pgdeploy", line 174, in <module>
sys.exit(main())
File "/usr/share/repmgr-agent/pgdeploy", line 170, in main
updater = PgBouncerUpdater(options.cluster, options.pgconnstring, options.timeout, options.public_net)
File "/usr/share/repmgr-agent/pgdeploy", line 48, in __init__
"""ORDER BY id""" % self.cluster_name
File "/usr/lib/python2.7/dist-packages/psycopg2/extras.py", line 144, in execute
return super(DictCursor, self).execute(query, vars)
psycopg2.ProgrammingError: relation "repmgr_main.repl_nodes" does not exist
LINE 1: SELECT id, name, type, conninfo FROM repmgr_main.repl_nodes ...
when restoring a database from backups, a way to stop all the pgbouncers would be helpful to get rid of client connections easily.
Sometimes (we don't know why) reload seems not to work.
Maybe it's a upstart/systemd issue... don't know.
https://github.com/Proemion/repmgr-agent/blob/master/v2/pgdeploy#L143
Am I right, that restart would kill active connections?
related to: https://github.com/Proemion/psp-release-voodoo/issues/2203
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.