muma378 / moose Goto Github PK

links:

• http://www.ruanyifeng.com/blog/2016/07/yaml.html
• http://pyyaml.org/wiki/PyYAMLDocumentation

Config

[common]
root = /root/path/to/
base = /root/path/	# to remove
name = 				# custom defined
tasks = task_20717,task_20718

[task_20717]
title =
root  =  
subdirs = 

[task_20718]
title = 
subdirs = 

YAML

root: /root/path/to
base: /root/path
tasks:
  - 
    task_id: 20717
    title: 
    subdirs: 
  -
    task_id: 20718
    subdirs: 

add library pytz to setup.py

You don't have to specify -c options when running an action

import time

有一些action，例如 stats, review, proofread 需要查询该期的数据库，但是又不需要export的其他动作，是否有必有抽象成更高层的类，例如action.Query。

All scenes to upload

1. 最常见的情况：单个文件夹下所有文件上一期任务
2. 多个文件夹上一期任务
3. 一次性多个文件夹上多期任务
4. 一个文件夹下平均上到多期任务
5. 不上传文件但是生成到容器的索引
6. 找到所有文件，处理后上传:
   • 视频：抽帧
   • 图片：格式转换
   • 音频：pcm->wav, downsample, mono, 切割

Abstract config files

[upload]
root = 

How to locate files by blobname?

.files/
	2a/
		823446d2f806ae9f842156acc8b35d
	27/
		330c419a7d358098fedc4b09cdcb4b
		...
.index/
	20717
	20718
.config

2a/823446d2f806ae9f842156acc8b35d (md5 of file1)

$0/relpath/to/file1
# raw:intermediate1:intermediate2...
27330c419a7d358098fedc4b09cdcb4b:39dc4c419a7d358098fedc4b09cdcb4b
metadata(size, samplerate, channels...)

./index/20717

blob1: 2a823446d2f806ae9f842156acc8b35d  # 对应原始文件 or blob本地文件？
blob2: 27330c419a7d358098fedc4b09cdcb4b

.config

[upload]
0 = /local/path/to/root1
1 = /local/path/to/root2

• 工具 moose-admin update-index root: 当原始文件的路径移动后根据md5值更新relpath

Too many statements existed now, some of them should be removed and implemented in the subclassed apps not in the framework.

The first released stable version. Including the following features:

• ACTIONS
  • defines a general-purpose and highly abstract procedure;
  • defines actions including upload/export/review/operate;
  • adds stats to collect info in an action;
  • adds check to verify each data model;
  • adds unit-test for all actions;
  • docs for base flows and settings;
• MODELS
  • defines a data model covers the mostly used usage scenario:
• CORE
  • adds unit-test and covers 85% code lines;
  • reviews all commands;
• PROCESS
  • renames to toolbox;
  • moves download/stats/check in;
  • adds ffmpeg;
• CONNECTION
  • removes cifs and statement;
  • refactors fetch;
  • adds more queries in query;
• CONF
  • removes all not-used-yet settings;
• UTILS
  • adds docs for mostly used functions;
• DEMO
  • removes apps with highly similarity;
  • keeps at most 3 apps in different complexity;

Azure also supports login with SAS_URL. This way was ought to be used when the account belongs to customers. API reference is here.