Comments (2)
Maybe you can check https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication
from meilisync.
I already had the flags enabled from this article, but data wasn't replicated.
Here is a docker compose I'm using to test:
version: '3'
services:
meilisync:
platform: linux/x86_64
image: long2ice/meilisync
volumes:
- ./meilisync-config.yml:/meilisync/config.yml
restart: always
depends_on:
- meilisearch
meilisearch:
image: getmeili/meilisearch:v1.3
ports:
- "7700:7700"
environment:
- MEILI_ENV=development
- MEILI_API_KEY=some-key
Sync config:
debug: true
progress:
type: file
meilisearch:
api_url: "http://meilisearch:7700"
api_key: "API_KEY"
source:
type: postgres
host: DB_IP
port: 5432
database: inro
user: replication_user
password: "PASSWORD"
sync:
- table: PG_SCHEMA_NAME.table_name
index: table_name
full: true
Logs from containers:
infrastructure-meilisearch-1 | [2023-09-19T09:12:16Z INFO actix_server::builder] starting 6 workers
infrastructure-meilisearch-1 | [2023-09-19T09:12:16Z INFO actix_server::server] Actix runtime found; starting in Actix runtime
infrastructure-meilisync-1 | 2023-09-19 09:12:26.540 | DEBUG | meilisync.main:_:33 - plugins=None progress=Progress(type=<ProgressType.file: 'file'>) debug=True source=Source(type=<SourceType.postgres: 'postgres'>, database='inro', password='', port=5432, host='', user='replication_user') meilisearch=MeiliSearch(api_url='http://meilisearch:7700', api_key='iwBx-', insert_size=None, insert_interval=None) sync=[Sync(plugins=None, table='items', pk='id', full=True, index='', fields=None), Sync(plugins=None, table='manifest', pk='id', full=True, index='manifest', fields=None)] sentry=None
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO actix_web::middleware::logger] 172.18.0.3 "POST /indexes/l/documents?primaryKey=id HTTP/1.1" 202 142 "-" "python-httpx/0.23.3" 0.007829
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO actix_web::middleware::logger] 172.18.0.3 "POST /indexes/l/documents?primaryKey=id HTTP/1.1" 202 142 "-" "python-httpx/0.23.3" 0.008936
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisync-1 | 2023-09-19 09:12:27.746 | INFO | meilisync.main:_:82 - Full data sync for table "items" done! 1378 documents added.
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO actix_web::middleware::logger] 172.18.0.3 "POST /indexes//documents?primaryKey=id HTTP/1.1" 202 143 "-" "python-httpx/0.23.3" 0.004431
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisync-1 | 2023-09-19 09:12:27.804 | INFO | meilisync.main:_:82 - Full data sync for table "manifest" done! 409 documents added.
I'm confused by the last log, it states that X documents was added, but indexes are empty:
from meilisync.
Related Issues (20)
- Postgres source ignores changes HOT 1
- Mongo ObjectId fields issue
- Postgres `get_full_data` is slow for large datasets HOT 1
- Not Fetching Data when specifying field attribute in config.yml HOT 2
- JSON data from MySQL is synced as strings in Meilisearch
- SQLite support HOT 1
- Meilisync Not Resyncing After Deleting Documents HOT 1
- ERROR argument after ** must be a mapping, not NoneType HOT 2
- TypeError: 'async for' requires an object with __aiter__ method, got coroutine HOT 21
- Lost Connection During Query / Attribute Error: `NoneType` object has no attribute `write`
- RDS DB replication slot not released by meilisync HOT 16
- ERROR: ModuleNotFoundError: No module named 'motor' HOT 1
- import psycopg2
- TypeError: 'NoneType' object is not subscriptable (ret = None) HOT 6
- Meilisync loses event when both create and update occur for the same primary key in an interval HOT 2
- Batch MySQL insert only synching the first occurence HOT 3
- TypeError: 'async for' requires an object with __aiter__ method, got list HOT 7
- Add non-root user to the docker image
- TypeError: Object of type Decimal is not JSON serializable HOT 2
- required argument is not an integer HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from meilisync.