Create a manager application which will provide scheduling on a specific cluster so that brender becomes multi-cluster.

Workers should create process (using fork) instead of threads, so that we can store the process
PID and kill it if necessary.

Using a "proper" m2m relation, as well as creating a mapping table (without id).

Hi,
I would like to create thumbnails for rendered images, including those saved using "File Output" nodes. Anybody knows if there is a simple way to get path of saved files after or during rendering process in Blender ‪ ... using a ‪python‬ script? or maybe other solution?
Thanks.

Implement API calls and use them in the dashboard. Figure out a basic interface to handle this.

Brender will stuck for some time when going to workers page if there are some offline, due to timeout connection when checking if worker is alive.

problem is on this line:

f = urllib.urlopen('http://' + worker.ip_address)

it should be simple to solve this by adding timeout option like so:

f = urllib.urlopen('http://' + worker.ip_address, timeout=10)

But server still wait for one minute to connect to worker. But if there a 10 or more workers?

moreover, the problem seems to be related to "create connection timeout" rather than with get data from remote computer, maybe it's a socket problem?
if so, it should work with this addition:

import socket
socket.setdefaulttimeout(10)

or

import urllib
urllib.socket.setdefaulttimeout(10)

but this is not working for tests I done.

what do you think? have you the same problem?

will type this up in a few

Currently we store them as config values, but in the future if would be good to store them in the manager database. This way it will be possible to have an API to to set them remotely from the server.

Of course if the manager is "private", it won't be possible to change those settings. This could be achieved by adding a PRIVATE_SETTINGS=True config value (by default the value will be False).

This needs to be discussed, but it might be beneficial to have a Settings table in the Manager db, where we define the STORAGE_PATH_WIN, STORAGE_PATH_LINUX and STORAGE_PATH_OSX constants. This way, when we pass over the data of a Task, files can actually be stored there.

This mean that in the task table we would drop the file_path_win, linux and osx, in favour of a single file_path field.

so i have been wondering this for awhile and now i'm going to spit it out so i can get it off my mind so here it goes

currently i have been wondering why we have a dashboard app and a server app
wouldn't it make more sense to make these into 1 app ?

that way the dashboard could talk to the database just a though and a question

i think it would make it easier on us if we go this route but that is just me

wondering what you guys think about this

Since more people are going to work on this, we are going to implement database migrations using Alembic. This is rather urgent.

Since we moved the worker functionality to the Manager, should this module be removed?

https://github.com/Nazg-Gul/bfarm

maybe we can use some ideas used in bfarm

when I try adding a shot, I get the error log at bottom.
The shot does get created in db, the jobs as well, but I believe it can not dispatch it to workers because it can't find the available workers. Of special interest ( and suspitious) is this :
PARAMS: [u'available']

where does the 'u' come from ?

p.s. I recreated the database from scratch.

127.0.0.1 - - [16/Oct/2013 09:35:10] "GET /shots/start/1 HTTP/1.0" 500 -
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/Library/Python/2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/Users/o/brender/server/modules/shots.py", line 63, in shot_start
start_jobs(shot.id)
File "/Users/o/brender/server/modules/jobs.py", line 153, in start_jobs
print(start_job(job.id))
File "/Users/o/brender/server/modules/jobs.py", line 118, in start_job
worker = Workers.get(Workers.status == 'available')
File "build/bdist.macosx-10.8-intel/egg/peewee.py", line 2325, in get
return sq.get()
File "build/bdist.macosx-10.8-intel/egg/peewee.py", line 1631, in get
% self.sql())
WorkersDoesNotExist: Instance matching query does not exist:
SQL: SELECT t1."id", t1."mac_address", t1."hostname", t1."status", t1."warning", t1."config", t1."system", t1."ip_address", t1."connection" FROM "workers" AS t1 WHERE (t1."status" = ?)
PARAMS: [u'available']

When a Worker connects to the Manager, does the count of total_workers change?

Currently it's not clear how we store frame thumbnails. Propose implementation here.

found this and thought i would share

http://pythonhosted.org/pulsar/overview.html

so i have a few questions for you guys @fsiddi @oenvoyage
how to add labels to issues?
how to add milestones?
best way to handle this situation
if i make commits on my fork and then you make changes whats the best way to handle merging?

i think with pull requests we should create a new branch for the feature/addition then merge it in i might be wrong
let me know what you guys think of this

also maybe we can start using this https://huboard.com/
on brender-farm.org like something like board.brender-farm.org
i think that huboard could help manage the issue and better assign things to developers

It restarts the first task and then goes ahead to other tasks.

Show details about the worker

Not sure how do solve this at the moment, since it depends on the filter function.

File "/Users/fsiddi/Developer/brender/brender/server/application/modules/jobs/__init__.py", line 322, in post
  TaskApi.delete_tasks(j)
File "/Users/fsiddi/Developer/brender/brender/server/application/modules/tasks/__init__.py", line 307, in delete_tasks
  manager = filter(lambda m : m.id == t.manager_id, Manager.query.all())[0]
IndexError: list index out of range

With proper settings to show more or less info is the manager is private or public.

I have been looking at http://semantic-ui.com/

and their UI looks amazing im wondering if you guys want to switch to this?

just a question cause i really like the way http://semantic-ui.com/ looks

let me know

Thanks

Kelvin

AngularJs is a framework 100% JavaScript to write web-applications that a rendered on client side. Application written with AngularJs are some times referred as single page web-application.

Differences from flask's render_template is that with angular all templates will be rendered on client side and AngularJs can directly query server for all data it needs. Web pages will be more dynamical and more flexible and you do not need a separate flask application to render html templates every time a client query a page. Data format for exchange is the same, json.

As result, there will be only one application on server side, which will be the server. And , as AngularJs is suitable for mobile browser to, means that you don't need to write a OS specific dashboard (as were mentioned in issues #22) ... there will be one dashboard suitable for all use-cases.

Brender is made by three individual applications. Should this be three individual packages or one big package?

for example I would like to have only one package and use is like this:

pip install brender
brender run server
brender run dashboard
brender run worker

and maybe having just a single configuration file for all three use cases.

On the other hands, why should you have to included server and dashboard in the worker package? when you need only the worker code? ... I'm a bit confusing. What are your thoughts about this?

It seems we are now loading the config 2 times...
once in brender.py:11-14
and second time in each module (for instance : worker/inti.py:15

I am not sure this is harmful, but it does a double output, and this harms my eyes :-)

[Info] XXXX Loading configuration from config.py
running server

• Running on http://localhost:9999/
• Restarting with reloader
  [Info] XXXX Loading configuration from config.py
  running server

When manager send "None" for total_workers field, servers answer 400.
Maybe we should send a boolean instead.

While on one hand it's nice to have brender.py server, brender.py dashboard and brender.py worker commands, it might be time to treat the three modules as separate applications and give them a unique manage.py. We can still keep a centralized config file.

Seems like @fsiddi broke Add Shot
commits that likely caused the breakage
e478b19, 9fe2349,
b56cd3a,
cefba4b
everything else seems to be working correctly

Traceback (most recent call last):
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/koder/virtualenvs/brender/brender/dashboard/dashboard.py", line 263, in shots_add
    shows=shows)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/templating.py", line 128, in render_template
    context, ctx.app)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/flask/templating.py", line 110, in _render
    rv = template.render(context)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/koder/virtualenvs/brender/brender/dashboard/templates/add_shot_modal.html", line 73, in top-level template code
    $('#brenderModalLabel').text('Add shot for {{shows[settings['active_show']]['name']}}');
  File "/Users/koder/virtualenvs/brender/lib/python2.7/site-packages/jinja2/environment.py", line 378, in getitem
    return obj[argument]
UndefinedError: dict object has no element Undefined

Using python manage.py runserver.

Start and Stop can't be executed on multiple shots at the moment (we collect multiple ids but the we make a query with all of them). We should either make a query per id or refactor the backend so that it would support multiple ids editing.

We could do it in a way similar to the worker, but that would lock us into using the manage.py, which is not good in the long run. Unless we implement similar code into a runserver.wsgi so that it can run with Apache as well.

Currently is still in use to define some config values in various apps. Would be good to make everything separated and get inter-app communication going only via HTTP.

something strange has happened ... @fsiddi please delete this duplicated issue

Hey, noob here. :)
Anyways, I've been doing a lot of research into render queues. And I found Brender, which looks awesome. Since I'm completely new to apache/php/mysql etc, I've been learning from scratch. So I finally got brender (the 0.5 version downloaded from the brender-farm.org site) installed on my server. Now I'm trying to get everything up and running :) In the manual, it said to start the server by typing "php brender_server" in the terminal on the server. So I did, and I got the error "Could not open input file: brender_server." So I started it by double-clicking the brender_server.php file. Which opened terminal and seemed to work. I've also created my clients on the web GUI, but I'm running into the same error in the terminal on those nodes. "Could not open input file brender_client." Double-clicking the corresponding brender-client.php opens terminal for less than a second before it closes. I don't think it's crashing, because no errors come up. (I'm on Ubuntu.)
Google hasn't been of any help, so I'm just wondering what exactly I'm missing here. If this isn't a good place to ask this kind of question, please refer me to some other site I could get an answer.

Thanks!

Use an interface similar to the one for Shots to set availability of workers. Possible statuses (not all available):

• available
• disabled
• busy
• rendering
• error

python logging for debug / info better than print statements

great explanation with examples
found here
http://victorlin.me/posts/2012/08/good-logging-practice-in-python/

Currently is broken. Would be enough to have a simple page with key info. Later we can add logs, stats, latest frame and so on.

Hello,

In the readme it says cd server/ & touch brender.sqlite

but when i start everything

i get this error

http://gyazo.com/3dfbb032dee05fecc7f4626a71f3d9cc

There are several points in the code where in order to catch an exception we use just the Except: statement, while we should be using the actual exception name instead.

Which, overall is a more "standard" of structuring a Flask app. This can also be considered an initial step for AngularJS support.

Computer 1 / Server

• Running Server
• Running Dashboard
• Running Worker

Computer 2 / Worker

• Running Worker

any ideas on how you want to get this started?

The log file can get really big quickly. Using chunk 1 after some frames its size can be 3Gb.

I understand that "shows" come to Brender from Attract... but why "SHOW"
This could have another name like Episode, Scene, Project, ShotGroup ... but show seems to be a bit confusional. This is valid at least for me.

@fsiddi please bring me to the light :) ... what a show should mean?

If workers have disconnected, and ta job is still running, currently it's not possible to stop it because:

  File "/shared/software/brender/venv/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/shared/software/brender/venv/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='192.168.4.205', port=5000): Max retries exceeded with url: /kill/25674 (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

Currently some empty migrations have been committed, simply a matter of removing them and making sure that things keep working on new and existing setups.

We want to check if a blend file exists and is really a blender file. Maybe have a look on filemagick library!!

http://pythonhosted.org/Flask-SQLAlchemy/

so on previous web applications i have built in python and pyramid

pyramid is just like flask just a lil different i used sql alchemy

i have been looking at how you guys are getting thing and it looks great im wondering if i can create a new branch and use sqlalchemy and have you guys talk a look at it im just throwing out options that will better the future of brender

the thing i like about sqlalchemy are these examples
http://pythonhosted.org/Flask-SQLAlchemy/quickstart.html#a-minimal-application
http://pythonhosted.org/Flask-SQLAlchemy/queries.html
http://flask.pocoo.org/docs/patterns/sqlalchemy/

i built this for example
https://github.com/cyprusglobe/pyramid_bs

the reason i think we should do this is you can

query the database for all workers lets say

Workers = Worker.query.all()
and you would get some thing like this
[<Worker u'132125336'>, <Worker u'12121231t'>]
which is each Worker Object

so if we do

for Worker in Workers:
    print Worker.id
    print Worker.ip
    print Worker.current_task

i think this would benefit us much better that way you just pass the querie over to the template

let me know what you guys think im currently porting over the current commit to a new branch and will submit it hopefully tonight or tommorrow

if you have any question please post below

GUIDELINES HERE

https://github.com/mitsuhiko/flask/wiki/Large-app-how-to