Git Product home page Git Product logo

Comments (4)

mouday avatar mouday commented on August 19, 2024

具体报错细节为:

Traceback (most recent call last):
  File "/Users/qmp/.virtualenvs/qmpspider/lib/python2.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/Users/qmp/.virtualenvs/qmpspider/lib/python2.7/site-packages/mysqlsmom/mysqlsmom.py", line 321, in do_one_task
    query = MyModel.raw(task["stream"]["sql"].replace("?", "%s"), (last_start_time,)).dicts().iterator()
AttributeError: 'RawQuery' object has no attribute 'iterator'

from mysqlsmom.

mouday avatar mouday commented on August 19, 2024

我的配置文件 cron_config.py

# coding=utf-8

STREAM = "CRON"

# 修改数据库连接
CONNECTION = {
    'host': '127.0.0.1',
    'port': 3306,
    'user': 'root',
    'passwd': '123456'
}

# redis存储上次同步时间等信息
REDIS = {
    "host": "127.0.0.1",
    "port": 6379,
    "db": 0,
    # "password": "password",  # 不需要密码则注释或删掉该行
}

# 一次同步 BULK_SIZE 条数据到elasticsearch,不设置该配置项默认为1
BULK_SIZE = 1

# 修改elasticsearch节点
NODES = [{"host": "127.0.0.1", "port": 9200}]

TASKS = [
    {
        "stream": {
            "database": "demo",  # 在此数据库执行sql语句
            "sql": "select id, name from student where update_time >= ?",  # 将该sql语句选中的数据同步到 elasticsearch
            "seconds": 10,  # 每隔 seconds 秒同步一次,
            "init_time": "2018-08-15 18:05:47"  # 只有第一次同步会加载
        },
        "jobs": [
            {
                "pipeline": [
                    {"set_id": {"field": "id"}}  # 默认设置 id字段的值 为elasticsearch中的文档id
                ],
                "dest": {
                    "es": {
                        "action": "upsert",
                        "index": "demo",   # 设置 index
                        "type": "student"          # 设置 type
                    }
                }
            }
        ]
    }
]

# CUSTOM_ROW_HANDLERS = "./my_handlers.py"
# CUSTOM_ROW_FILTERS = "./my_filters.py"

from mysqlsmom.

m358807551 avatar m358807551 commented on August 19, 2024

新建个 py27 的虚拟环境安装吧。

旧环境里 的 peewee 版本跟 mysqlsmom 用到的 peewee 版本不兼容;

from mysqlsmom.

mouday avatar mouday commented on August 19, 2024

好的大佬

from mysqlsmom.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.