aliyun / aliyun-log-python-sdk Goto Github PK

View Code? Open in Web Editor NEW

161.0 21.0 117.0 1.44 MB

Use python to manage, produce and consume data with Aliyun Log Service.

Home Page: http://aliyun-log-python-sdk.readthedocs.io/

License: MIT License

Python 82.18% Jupyter Notebook 17.82%

aliyun-log-python-sdk's People

Contributors

Stargazers

Watchers

Forkers

funnykun y100143239 zzxwill twz915 fengrk huisezhiyin mk8310 bailaohe ggbg wjo1212 vapao biepianwohaha kutim 007gzs liketic dashuangshuang gezidan plcherry yonzhan2 songroger crhan sinaeftekhari wusir66666 xuxiaoahang2018 raohongjun cdwhlong qingreee1 simon-cj jiangming1 linzuyan duxinn abinghu huhaibin cthesky goclis hhr66 hiddenbomb ii0 wibud plpcm pythonohtyp phpsxg oyanrayring jiangsiwei qq7618qq k3v1n888 lewvan37 hiroaster gaoaozi focus-hj winlin cockcrow qiangshenqi hqcfly fortuneblod auqf jimmyzhoujcc saorisakura jasonlittledragon baishaofan mengskysama chensanle luyanfcp yzy-666 jasonshao sona201 deathef liyoudao6 guhanjie yuanqj zhangtiantianx coderkaka lleiyyang lawrenceforyou panawala yuyfbj adongfan kaireg chenfh5 yonghua-sun iamorchid lialzm sushaohua kucharzyk-sebastian oawang bypass007 zhagyilig lichengseu lucatisfun nodejh haoruilee harry-1989 zrz2233 xiaofan-linux neverafraid1 z0xc linrunqi08 aliyun-sls chaojiyou fanzhonghao

aliyun-log-python-sdk's Issues

support to get context of specific log

Just as the feature shown in web

support gzip compression

support gzip
for PullLogs, dynamically support it rather than depending on API's parameter.

cannot get logtail config for plug-in

e.g. add config for mysql, fail to get log tail config with below error:
{"errorCode": "InvalidInput", "errorMessage": "input json string missed necessary fields: input_type and/or logType", "requestId": ""}

windows pip install UnicodeDecodeError: 'gbk' codec can't decode byte

long_description = open('README.md').read()
UnicodeDecodeError: 'gbk' codec can't decode byte 0xad in position 847: illegal multibyte sequence

Provide a utility method "ping" or "verify"

it's used to veirfy if the AK is correct. CLI could call it by default.

list API passing size > 500 but will always get more than expected count

e.g.
list_logstore('project', size=501), will get 1000 rather than 501

API update logstore doesn't work

it reports error "shard_count" could only be modified by split & merge

package lz4 package

now, it support lz4, but there's no logservice_lz4 package inside.

allow to create logtail config under non-existing logstore

it's possible to create a logtail config under a non-existing logstore, such kind of logstore won't be shown on web console.
expected:

don't allow user to create such logstore.

some proxy will filter "Date" in http header and break the request package

aliyun.log.logexception.LogException: {"errorCode": "MissingParameter", "errorMessage": "missing http extend header key : x-log-date", "requestId": "11111111"}

get_log will get duplicated data among big volume of log

when data volume is large, use stat query like "* | select ... limit 10000" will cause this issue.

support multi-process for consumer group

it's somehow IO bound, but will also consume CPU for protobuf and text encoding processing. need to support multi-process based consumer group

chinese charset is messy in pdf downloaded from readthedocs.io

cannot show Chinese characters well, need to fix it.

support more time format

right now, it's 2018-1-1 10:10:10, better to support more style for usability.

support time string with time zone

right now, it's undefined.

support more logtail cofnig

right now, it only supports:

common type

but it has more:

separator
simple
more..

fail to update logtailconfig via json file via CLI

>> aliyun log update_logtail_config --project_name=dlq-test-sls-project1 --config_detail="file://./test1.json" --client-name=beijing
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/aliyun", line 11, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aliyunlogcli/cli_core.py", line 148, in main
    ret = getattr(client, method_name)(**args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aliyun/log/logclient.py", line 1205, in update_logtail_config
    resource = "/configs/" + config_detail.config_name
AttributeError: 'SeperatorFileConfigDetail' object has no attribute 'config_name'

push to pypi

please push this package to pypi

return ?? when there's multi-bytes in project description

return ?? when there's multi-bytes in project description.

after check, it seems an issue in server side even setting "Accept-Character" as "utf-8"

support https connection

nice to allow to disable ssl self-signed check for some special case.

support adding tag info into pulled logs

right now, it's not rendered.

add external storage management API support

auto update consumer group if there's an existing one

when creating consumer group, if there's an existing one, it will check if the setting is the same and fail if they're different.
it will be better to update the setting directly

support concurrent API to pull logs to local file

res = client.pull_log('project1', 'logstore1', from_time="2018-1-1 10:10:10", to_time="2018-1-1 10:20:10", file_path="/data/dump_{}.data")
res.log_print()

QueuedLogHandler不支持twistd

/code/main.py文件：

def polling():
    reactor.callLater(5, polling)
    print 'polling...   '
    try:
        ''+1
    except:
        log.error("some error")


if __name__ == '__builtin__':
    reactor.callLater(0, polling) 
    application = service.Application('main')

启动命令：

cd /code && twistd --python=/code/main.py

OOM error when using consumer group to get large volume of logstore

consume large data, but got OOM error, logstore count > 50.

*** start to consume data...
No handlers could be found for logger "aliyun.log.consumer.tasks"
No handlers could be found for logger "aliyun.log.consumer.tasks"
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\aliyun\log\consumer\shard_worker.py", line
 56, in get_task_result
    return task_future.result()
  File "C:\Python27\lib\site-packages\concurrent\futures\_base.py", line 455, in
 result
    return self.__get_result()
  File "C:\Python27\lib\site-packages\concurrent\futures\thread.py", line 63, in
 run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Python27\lib\site-packages\aliyun\log\consumer\tasks.py", line 136, i
n consumer_fetch_task
    raise Exception(e1)
Exception
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\aliyun\log\consumer\shard_worker.py", line
 56, in get_task_result
    return task_future.result()
  File "C:\Python27\lib\site-packages\concurrent\futures\_base.py", line 455, in
 result
    return self.__get_result()
  File "C:\Python27\lib\site-packages\concurrent\futures\thread.py", line 63, in
 run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Python27\lib\site-packages\aliyun\log\consumer\tasks.py", line 136, i
n consumer_fetch_task
    raise Exception(e1)
Exception
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\aliyun\log\consumer\shard_worker.py", line
 56, in get_task_result
    return task_future.result()
  File "C:\Python27\lib\site-packages\concurrent\futures\_base.py", line 455, in
 result
    return self.__get_result()
  File "C:\Python27\lib\site-packages\concurrent\futures\thread.py", line 63, in
 run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Python27\lib\site-packages\aliyun\log\consumer\tasks.py", line 135, i
n consumer_fetch_task
    logger.error(e1, exc_info=True)
  File "C:\Python27\lib\logging\__init__.py", line 1193, in error
    self._log(ERROR, msg, args, **kwargs)
  File "C:\Python27\lib\logging\__init__.py", line 1285, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, fun
c, extra)
  File "C:\Python27\lib\logging\__init__.py", line 1259, in makeRecord
    rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
  File "C:\Python27\lib\logging\__init__.py", line 271, in __init__
    self.filename = os.path.basename(pathname)
  File "C:\Python27\lib\ntpath.py", line 208, in basename
    return split(p)[1]
  File "C:\Python27\lib\ntpath.py", line 188, in split
    while head2 and head2[-1] in '/\\':
MemoryError

用python SDK获取的日志数量不对

我用这个脚本获取的日志数量很少只有大概1分钟的数量,不管我怎么选择写
From = int(time.time())-86400
To = int(time.time())
函数 sample_get_logs
总是获得很少日志数量,在web控制台上可以查到很多

allow to copy logstore settings to existing project

api logclient.copy_project expecting the target project doesn't exit. can we try to proceed if it exists?

support dashboard management

allow to list-crud dashboard

QueuedLogHandler uwsgi中会丢数据

uwsgi 中如果设置了master=true threading就有问题，设置master=false的话多进程停服务和重启服务好像有问题，现在先用 SimpleLogHandler 了， @wjo1212 有空帮忙看看，谢谢

https://stackoverflow.com/questions/32059634/python3-threading-with-uwsgi

support to update index to enable Chinese words segmentation

python2.6 decode error when calling list projects

TypeError: decode() takes no keyword arguments

allow to configure web-tracking of logstore

increase the restriction of log uploading

right now, it's 3MB, should be larger

support readable time format for get_histgram

support readable time string for get_log/get_cursor

as aliyun/aliyun-log-cli#11

when query contains new line with complex SQL, it might not return in get_log

e.g.

* |
            select ip,time, __time__ as sls_query_time,
                from_unixtime(MIN(to_unixtime(time))) AS min_data_time,
                to_unixtime(current_timestamp) as now,
                to_unixtime(current_timestamp) - MIN(to_unixtime(time)) as diff_time 
            group by ip,time, sls_query_time
            HAVING to_unixtime(current_timestamp) - MIN(to_unixtime(time)) > 90
            LIMIT 10000

x-log-progress doesn't exist when keys has multi-bytes and in Python3.5

    self.progress = Util.h_v_t(header, 'x-log-progress')
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aliyun/log/util.py", line 169, in h_v_t
    return header[key]
KeyError: 'X-Log-Progress'

Logging Handler支持特定格式识别: JSON

自动识别特定格式, 例如dict为JSON, KV等等

data = {'a':1, 'b': 2}
logger.error(data)

可以构建字段为a, b的域
问题1: 与现有冲突时, 处理策略是什么?
问题2: 将message索引构建为JSON的区别(更加灵活, 因为打log时, 格式是混合的)

其他情况:
2. KV格式, 例如字符串的, 自动提取Key

data = 'name=xiaoming home="nanjing"  score=12.5'
logger.info(data)

自动构建name, home, score字段, 能够识别双引号等

python2.6 support issue

protobuf latest version doesn't support py2.6 any more, but we could use older version in setup.py
other issue found:

Traceback (most recent call last):
  File "tests/sample.py", line 354, in <module>
    main()
  File "tests/sample.py", line 330, in main
    sample_pull_logs(client, project, logstore)
  File "tests/sample.py", line 53, in sample_pull_logs
    datetime.fromtimestamp(int(time.time() - 60)).strftime('%Y-%m-%d %H:%M:%S'))
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/aliyun/log/logclient.py", line 566, in get_cursor
    params = {'type': 'cursor', 'from': str(start_time) if start_time in ("begin", "end") else parse_timestamp(start_time)}
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/aliyun/log/util.py", line 215, in parse_timestamp
    return int((dt - datetime(1970, 1, 1)).total_seconds())
AttributeError: 'datetime.timedelta' object has no attribute 'total_seconds'

support logging handler

provide an Util for logging handler

support copy logstore

def copy_logstore(from_client, from_project, from_logstore, to_logstore, to_project=None, to_client=None):
    """
    copy logstore, index, logtail config to target logstore, machine group are not included yet.
    the target logstore will be crated if not existing

    :type from_client: LogClient
    :param from_client: logclient instance

    :type from_project: string
    :param from_project: project name

    :type from_logstore: string
    :param from_logstore: logstore name

    :type to_logstore: string
    :param to_logstore: logstore name

    :type to_project: string
    :param to_project: project name, copy to same project if not being specified, will try to create it if not being specified

    :type to_client: LogClient
    :param to_client: logclient instance, use it to operate on the "to_project" if being specified

    :type apply_machine_group: bool
    :param apply_machine_group: if apply config to same machine group resources, False by default.

    :return:
    """

support binary content read-write

right now, it's KV, need to support binary format

AttributeError: 'LogGroupList' object has no attribute 'LogGroups'

loggroup = res.get_loggroup_json_list()
File "/data/virtualpython/group_attr_analysis/lib/python2.7/site-packages/aliyun/log/pulllog_response.py", line 42, in get_loggroup_json_list
self._transfer_to_json()
File "/data/virtualpython/group_attr_analysis/lib/python2.7/site-packages/aliyun/log/pulllog_response.py", line 67, in _transfer_to_json
for logGroup in self.loggroup_list.LogGroups :
AttributeError: 'LogGroupList' object has no attribute 'LogGroups'

这个是什么原因导致的呢

    c1 = "i=c1, k1=v1,k2=v2 k3=v3"
    c2 = 'i=c2, k1=" v 1 ", k2="v 2" k3="~!@#=`;.>"'
    c3 = u'i=c3, k1=你好 k2=他们'.encode('utf8')
    c4 = u'i=c4, 姓名=小明 年龄=中文 '.encode('utf8')
    c5 = u'i=c5, 姓名="小明" 年龄="中文" '.encode('utf8')
    c6 = u'i=c6, 姓名=中文 年龄=中文'
    c7 = u'i=c7, 姓名="小明" 年龄=中文 '
    c8 = """i=c8, k1="hello
    world" k2="good
    morning"
    """
    c9 = "i=c9 k1:v1, k2=v2"

    data = [c1, c2, c3, c4, c5, c6, c7, c8, c9]

    for d in data:
        logger.info(d)

cannot copy project when shard count > 10

by default, you can only create logstore with shard <=10, but after splitting, the count might exceed 10, thus trying to copy such kind of logstore will fail.

code error when put wide-char in query string when get logs

    topic = ''
    From = int(time.time()) - 7600
    To = int(time.time())
    request = GetLogsRequest(project, logstore, From, To, topic, query="x测试*")
    response = client.get_logs(request)

it will complain encoding error