Add off-screen rendering support + Panda3D example about cefpython HOT 43 CLOSED

I'm not sure how to pass the void* buffer to python in the OnPaint method:

    virtual void OnPaint(CefRefPtr<CefBrowser> browser,
                       PaintElementType type,
                       const RectList& dirtyRects,
                       const void* buffer) {}

I want to create some example for Panda3D, if you have an idea what should such 
example do then let me know. Should we use wxPython? I've found some skeleton 
here:

http://www.panda3d.org/manual/index.php/Starting_Panda3D

i guess str/bytes type would work just fine, or array.array ( 
http://docs.python.org/2/library/array.html )

i did something similar with berkelium (similar to CEF) and other game engine, 
so idea was to render returned bitmap to texture and draw it on screen.
i can look into that since i would need to code it anyway, then it would serve 
as example too.

Can you create a Panda3D example using CefWindowInfo & CefRenderHandler api? I 
know it's not yet implemented, it will not compile, but it would be a great 
help to have it.

There are some issues with Panda3D, I couldn't find stable binaries to work 
with python 2.7, version 1.7.2 is bundled with python 2.6, so I'm using version 
1.8.0, but it is unstable and api has changed a lot, the most simple examples 
from their documentation do not work anymore in the 1.8.0 version.

It is also required to implement:

SendKeyEvent(KeyType type, const CefKeyInfo& keyInfo, int modifiers)
SendMouseClickEvent(int x, int y, MouseButtonType type, bool mouseUp, int 
clickCount)
SendMouseMoveEvent(int x, int y, bool mouseLeave)
SendMouseWheelEvent(int x, int y, int deltaX, int deltaY)
SendFocusEvent(bool setFocus)
SendCaptureLostEvent()

http://code.google.com/p/chromiumembedded/source/browse/trunk/cef1/include/cef_b
rowser.h#295

I will have hard time implementing off-screen rendering without an example 
Panda3D code. I haven't yet figured out how to display a buffer in a current 
window.

leave example to me, just expose required apis for starters and possibly give 
me test build :)

P.S. at least windows build runs on py2.7

The following methods from the Browser object should also be exposed:

bool GetSize(PaintElementType type, int& width, int& height)
void SetSize(PaintElementType type, int width, int height)
void Invalidate(const CefRect& dirtyRect)
bool GetImage(PaintElementType type, int width, int height, void* buffer)

I am certain void* buffer should be passed as str. Lucky for us - CEF returns 
BGRA buffer, and same format is used by panda3d natively.

Also i forgot to mention that 1.8 build of panda is actually stable release. I 
have no idea why site claims it is unstable, they do this for very latest 
release. Actual unstable packages there are dev packages (auto-builds of latest 
code repository snapshot) so use 1.8 without fear.

Im further investigating whats the best way to draw returned pixel data.

Current test app: http://pastebox.it/file/12sZ15HSgFUI8a2s0FJF8OUmZqk/dzr4SQ.py
My most handy p3d module: 
http://pastebox.it/file/KyOwUJQMh54BxwM-RpNST00rhwY/p3d.py

I think that an int pointer to the buffer should be returned, for performance 
reasons, we can load a texture using that pointer like this:

    Texture.setRamMipmapPointerFromInt(buffer.GetIntPointer(), 0, width*height*4)

buffer.GetString() might also get implemented if there is need for that.

Thanks for the example, though I've already managed to display the texture, but 
in a different way:

    tex = Texture()
    tex.setup2dTexture(512, 512, Texture.CMOff, Texture.FRgba)
    tex.setRamMipmapPointerFromInt(bufferIntPointer, 0, w*h*4)

    cm = CardMaker("browser2d")
    cm.setFrame(-0.75,0.75,-0.75,0.75)
    card = render2d.attachNewNode(cm.generate())
    card.setTexture(tex)

I still need example code for intercepting mouse move, mouse click, mouse 
wheel, keyboard and captureLost events.

I've had to refactor much code to be able support offscreen rendering, as the 
current design was flawed, limitations described in Issue 31 were also fixed. 
There was no way to uniquely identify an offscreen browser, I've created a CEF 
patch for this (issue 811 in the CEF tracker, 
http://code.google.com/p/chromiumembedded/issues/detail?id=811).

libRocket integration code that does input handling: 
https://www.panda3d.org/reference/1.8.0/cxx/rocketInputHandler.cxx_source.php

Thats how its set up in python:
self.r = rocket.RocketRegion.make('gui_screen', base.win) 
self.r.setActive(1)
self.ih = rocket.RocketInputHandler() 
base.mouseWatcher.attachNewNode(self.ih)
self.r.setInputHandler(self.ih)

There is a binary preview with offscreen rendering:
http://code.google.com/p/cefpython/downloads/detail?name=cefpython1_v0.51_previe
w1_windows_32bit.zip

See panda3d.py > ClientHandler > OnPaint().

I'm trying to make this work, the buffer length is variable, sometimes it is 
38513 bytes, the other time it is 1920000.

The buffer is a PaintBuffer class with 2 methods:
GetIntPointer()
GetString()

I've found your topic at panda3d forums:
https://www.panda3d.org/forums/viewtopic.php?t=15164

Should we add some other method to PaintBuffer that returns a texture with full 
dimensions? This could be written in C++ so it will be faster. Or some flipping 
to RGBA if required?

Great news! Guess we are steps away from rendering to texture eh?

OnPaint() reminded me my other little project i was working on. I did implement 
something similar using berkelium lib which is based on chromium just like CEF. 
As a result - interface is same. This is my code of painting on texture: 
http://pastebox.it/file/fAGxcpGbXvu8fhDsaO6lcBEXgHk/8EEvDA.cpp
It handles scrolling (by copying pixel rows that are already present in 
pixelbuffer) and replaces changed rects with new ones supplied in custom buffer.
Here we get to variable length of buffer. Do you know for sure what exactly 
that buffer is? Maybe it is set of dirty rects just like in berkelium? Presence 
of dirtyRects variable indicates so to me.

Idea of doing pixel hardlifting in c++ is great. I would suggest some kind of 
interface where user would be able to supply pixelbuffer in OnPaint() method, 
and c++ code would then copy dirty rects to that buffer.

In case of panda3d it so happens that internal pixel color format is same as in 
CEF, however it may not always be the case, so i assume it would be useful to 
be able to have cefpython flip pixel format to required one.

And in case my c++ code proves useful - do not hesitate and include it in 
cefpython under w/e license :)

And one final thing: it might be a tiny bit of overkill for cefpython, but for 
some reason panda3d texture coords start from bottom-left corner, not usual 
top-left corner (writing start of the buffer draws to the end of texture). It 
would be great if this "flipping-over" image could be done in c++ too, that 
would require running loop(s) backwards.

In the console output you will see that dirtyRects are always the same 
irrespectively of the buffer length:

OnPaint()
paintElementType: 0
dirtyRects: [[0, 0, 800, 600]]
buffer.GetIntPointer(): 265355264
width, height = 800, 600
len(buffer.GetString()): 1920000
OnPaint()
paintElementType: 0
dirtyRects: [[0, 0, 800, 600]]
buffer.GetIntPointer(): 265355264
width, height = 800, 600
len(buffer.GetString()): 38513

You can view example implementations of OnPaint in cefclient osrenderer.cpp & 
osrplugin.cpp:

http://code.google.com/p/chromiumembedded/source/browse/branches/1271/cef1/tests
/cefclient/osrenderer.cpp?r=962#157

http://code.google.com/p/chromiumembedded/source/browse/branches/1271/cef1/tests
/cefclient/osrplugin.cpp?r=962#282

There are new methods in the [Browser]: GetSize(), SetSize(), Invalidate().

In preview 2 I will also addd GetImage(), maybe this one will give us texture 
in full dimension?

  ///
  // Get the raw image data contained in the specified element without
  // performing validation. The specified |width| and |height| dimensions must
  // match the current element size. On Windows |buffer| must be width*height*4
  // bytes in size and represents a BGRA image with an upper-left origin. This
  // method should only be called on the UI thread.
  ///
  /*--cef()--*/
  virtual bool GetImage(PaintElementType type, int width, int height,
                        void* buffer) =0;

How do I do the flipping from top-left corner to bottom-left corner?

To copy rect rows for bottom-left-based texture copying should go backwards 
(actually i was mistaken saying loop goes backwards - it does not):
    // copy new rects
    for(unsigned int i = 0; i < numCopyRects; i++)
    {
        Rect cr = copyRects[i];
        int wid = cr.width();
        int hig = cr.height();
        int top = cr.top() - srcRect.top();
        int left = cr.left() - srcRect.left();

        for(int y = 0; y < hig; y++)
        {
            //was: unsigned int tb = ((cr.top() + y) * tw + cr.left()) * 4;
            unsigned int tb = ((hig - (cr.top() + y)) * tw + cr.left()) * 4;
            memcpy(&texd[tb], &src[(left + (y + top) * srcRect.width()) * 4], wid * 4);
        }
    }

And in words:
suppose we have 50x50 top-left based rect (what CEF gives to us).
we want to copy a row of top-left rect to bottom-left rect (panda3d texture).
pointer in our buffer to that row will be &buffer[ row * width * pix_size ].
pointer in target bottom-left based texture buffer would be &buffer[ (height - 
row) * width * pix_size ].
doing it so we would fill destination buffer from bottom-up making image to 
appear properly.

The expected ram image size is 1920000 (800*600*4) only when I set format to 
Texture.FLuminanceAlpha, when I set it to FRgba/FRgba4/FRgba8/FRgba32 then it 
is 3840000.

buffer.GetString() returning string of length 38513 was a bug, I was casting 
void* to char*, then to string, but it cut the buffer when first null byte was 
encountered, the fix was to provide the length of the string.

Now I've tried with format=Texture.FLuminanceAlpha and I got it working, but 
the picture is flipped upside-down, attaching screenshot.

Regarding the code from comment#18: we need to copy the whole buffer, as there 
is no way to pass dirty rects to panda (or is there a way?).

It works! Flipping picture:

    for y in range(0, height):
        tb = length - ((y+1)*width*4)
        memcpy(&dest[tb], &src[y*width*4], width*4)

How can I make the 2d texture appear at an angle, so that it looks 3d?

Can you provide the code to convert it from BGRA to RGBA?

A more efficient way would be to return a pointer to the buffer, but this code 
seem not to work, the browser goes black:

    self.tex.setRamMipmapPointerFromInt(buffer.GetIntPointer(), 0, width*height*4)

Off-screen rendering preview 2:
http://code.google.com/p/cefpython/downloads/detail?name=cefpython1_v0.51_previe
w2_windows_32bit.zip

Great progress! Thanks for another preview!

Converting BGRA to RGBA is just a matter of switching byte places like:
BGRA = 0x11223344
print 'BGRA: %08X' % BGRA
RGBA = BGRA & 0x00FF00FF    # null bytes that are different
print 'RGBA: %08X' % RGBA
RGBA |= ((BGRA >> 24) << 8) & 0xFFFFFFFF    # B
print 'RGBA: %08X' % RGBA
RGBA |= ((BGRA >> 8) << 24) & 0xFFFFFFFF    # R
print 'RGBA: %08X' % RGBA

output:
BGRA: 11223344
RGBA: 00220044
RGBA: 00221144
RGBA: 33221144

Ill be able to play with this stuff after few days when this holiday thing dies 
down. Cant wait actually :)

So I need to do a loop and shift bits for each pixel separately? Or is there a 
faster way?

The code in comment #25 does not work, the valid code to convert bgra to rgba 
is:

    rgba = (bgra & 0x00ff0000) >> 16
       | (bgra & 0xff00ff00)
       | (bgra & 0x000000ff) << 16;

Everything is almost ready, the only missing thing is an example of using 
librocket to forward mouse/key events to the CEF browser.

See [Changelog] for a list of changes made in version 0.51.

Off-screen rendering preview 3:
http://code.google.com/p/cefpython/downloads/detail?name=cefpython1_v0.51_previe
w3_windows_32bit.zip

I just noticed that the browser is black-white. The format was set to 
Texture.FLuminance, the fix is to call:

self.texture.setComponentType(Texture.TUnsignedByte)
self.texture.setFormat(Texture.FRgba4)

Version 0.51 released, off-screen rendering is ready, see changelog for this 
release:
http://code.google.com/p/cefpython/wiki/Changelog

Download:
http://code.google.com/p/cefpython/downloads/detail?name=cefpython1_v0.51_window
s_32bit.zip

amazing, thank you so much :) i started working on keyboard input for demo, 
will let you know when its done

demo with keyboard input working: 
http://pastebox.it/file/v80YV1oH9Ow_u22fMRh_Wdz3bfs/T28b7g.py/VVBrf3B-xUGR_2AURA
nWCZqKMas/

P.S. add textarea to cefsimple.html

Is calling SendFocusEvent() required to have keyboard focus?

  self.browser.SendFocusEvent(True)

It is not mandatory. Without it its still possible to enter text into text 
area, however text area does not have that yellow border which is added by 
webkit to focused element. So in a sense element is still focused but no visual 
feedback is provided.

In onButtonUp() you use KT_KEYDOWN?

What is this condition for?

  ..and key != 'backspace'"

KEYDOWN is a bug yeah..
and backspace should not be processed because it causes deletion of two chars 
instead of one.

It deleted two chars because you used KEYDOWN instead of KEYUP?

uhh you are right, i did not notice that. so proper way is to use 
cefpython.KEYTYPE_KEYUP and removing `and key != 'backspace'`

Thank you for the keyboard handlers code, works great, commited in revision 
80a6a5daec26.

Do you know how to get rid of the blurriness when the window is resized?

Not sure about that rly.

By the way - i see you do common mistake in python. Declarations of members 
right under class like here:
class World(DirectObject):
    browser = None

They declare static members of the class so in this case they are totally not 
needed. Even if self.browser is set in __init__ - World.browser is still None :)

That is not a mistake. Yes, this also declares static members of the class.

Project will move to Github. Find this issue at the new address (soon): 
https://github.com/cztomczak/cefpython/issues/36