google / forma Goto Github PK

An efficient vector-graphics renderer

License: Apache License 2.0

Rust 98.11% HTML 1.89%

forma's Introduction

A (thoroughly) parallelized experimental Rust vector-graphics renderer with both a software (CPU) and hardware (GPU) back-end having the following goals, in this order:

Portability; supporting Fuchsia, Linux, macOS, Windows, Android & iOS.
Performance; making use of compute-focused pipeline that is highly parallelized both at the instruction-level and the thread-level.
Simplicity; implementing an easy-to-understand 4-stage pipeline.
Size; minimizing the number of dependencies and focusing on vector-graphics only.

It relies on Rust's SIMD auto-vectorization/intrinsics and Rayon to have good performance on the CPU, while using WebGPU (wgpu) to take advantage of the GPU.

Getting started

Add the following to your Cargo.toml dependencies:

forma = { version = "0.1.0", package = "forma-render" }

4-stage Pipeline

1. Curve flattening	2. Line segment rasterization	3. Sorting	4. Painting
Bézier curves	line segments	pixel segments	sorted pixel segments, old tiles
⬇️⬇️⬇️	⬇️⬇️⬇️	⬇️⬇️⬇️	⬇️⬇️⬇️
line segments	pixel segments	sorted pixel segments	freshly painted tiles

Implementation Highlights ✨

Here are a few implementation highlights that make forma stand out from commonly used vector renderers.

Curvature-aware flattening

All higher cubic Béziers are approximated by quadratic ones, then, in parallel, flattened to line segments according to their curvature. This technique was developed by Raph Levien.

Cheap translations and rotations

Translations and rotations can be rendered without having to re-flatten the curves, all the while maintaining full quality.

Parallel pixel grid intersection

Line segments are transformed into pixel segments by intersecting them with the pixel grid. We developed a simple method that performs this computation in O(1) and which is run in parallel.

Efficient sorting

We ported crumsort to Rust and parallelized it with Rayon, delivering improved performance over its pdqsort implementation for 64-bit random data. Scattering pixel segments with a sort was inspired from Allan MacKinnon's work on Spinel.

Update only the tiles that change (currently CPU-only)

We implemented a fail-fast per-tile optimizer that tries to skip the painting step entirely. A similar approach could also be tested on the GPU.

Animation as it appears on the screen	Updated tiles only

You can run the demo above with:

cargo run --release -p demo -- spaceship

Similar Projects

forma draws heavy inspiration from the following projects:

Spinel, with a Vulkan 1.2 back-end
vello, with a wgpu back-end

Example

You can use the included demo example to render a few examples, one of which is a non-compliant & incomplete SVG renderer:

cargo run --release -p demo -- svg assets/svgs/paris-30k.svg

It renders enormous SVGs at interactive framerates, even on CPU: (compare to your web browser)

(Currently) Missing Pieces 🧩

Since this project is work-in-progress, breakage in the API, while not drastic, is expected. The performance on the GPU back-end is also expected to improve especially on mobile where performance is known to be poor and where the CPU back-end is currently advised instead.

Other than that:

Automated layer ordering
Strokes
More color spaces for blends & gradients
Faster GPU sorter
Use of f16 for great mobile GPU performance

Note

This is not an officially supported Google product.

forma's People

Contributors

Stargazers

Watchers

forma's Issues

Is "Update only the tiles that change (currently CPU-only)" really desired for the GPU?

There is "Update only the tiles that change (currently CPU-only)" in the README. That suggests that it is being considered that this should be ported to the GPU and such a port is planned. I question this because it goes against my entire mental model of GPU performance.

Supporting this idea is the convention for modern rendering software to just repaint everything. Every 3D game redraws the whole world onto a new frame, throwing away the previous one. Desktop environments and windowing systems redraw the whole screen to support modern effects and decorations. With their introduction of compositing, their equivalent of "update only the tiles that change" called "damage" is gone as every application gets their own buffer and modern software does not need to consider this anymore. I read how modern GPUs are actually faster when data is not reused for partial repaints. One form of reuse slowness is trying to draw something then discarding, our "update only the tiles that change" would be equivalent to discarding most of the screen. From Asahi Linux's GPU blog, it is stated that it is expensive to read framebuffer data back for reuse compared to rerendering and overwriting it. Then there is the problem of trying to render the next frame on the GPU right after the last one is finished without waiting for the CPU-GPU round-trip synchronization time. These all support the idea that GPUs like data pushed through without the latency of data dependencies, and reuse will only impair parallel processing to slow down the GPU with dependencies.

What might save this idea is battery consumption. Firefox goes through the trouble of using private APIs on macOS so that it reuses unchanged frame data. On the README however I see that the GIFs are about gaming so I don't know how much battery matters compared to Firefox's text use case.

So I would like people to teach me, what are your considerations, viewpoints, and experiences for GPU framebuffer reuse?

Vector-graphics Renderer

An efficient vector-graphics renderer

Find a solution for conflation artefacts.

forma currently does not deal with conflation artefacts that can be visible in some cases.

Ideally, we should find a solution to this that does not affect performance too much and that can be easily disabled.

SVG linearGradient `stop missing offset` error

<svg
  viewBox="0 0 10 10"
  xmlns="http://www.w3.org/2000/svg"
  xmlns:xlink="http://www.w3.org/1999/xlink">
  <defs>
    <linearGradient id="myGradient" gradientTransform="rotate(90)">
      <stop offset="0" stop-color="gold" />
      <stop offset=".58" stop-color="red" />
    </linearGradient>
  </defs>

  <circle cx="5" cy="5" r="4" fill="url('#myGradient')" />
</svg>

thread 'main' panicked at 'stop missing offset', demo/src/demos/svg.rs:839:3

Clean up shaders.

Currently, some shaders use templating, others don't, while also using different invocation styles.

We should:

organize shaders better with the use of imports
have more variables be configurable by templating (e.g. workgroup size)
use the same value for max invocations per dispatch and have one single method to deal with this limitation.

All floating point input should be checked to be finite.

Currently, the public API relies on all floating point values being finite. This is not guaranteed in Rust.

All these methods should be renamed to *_unchecked and made unsafe, and safe methods added that check the values and then call these methods instead.

Integration with Game Engine?

It seems like this crate is intended to do the rendering directly, but I'm curious about integrating it with a game engine like Bevy or Fyrox or even Godot. Do you have an example of integrating Forma with a game engine?

f16 support

f16 is required in order to achieve good performance on mobile GPUs. Current blocked on naga.

SVG demo holds CPU at 100% utilization indefinitely

System Information

Windows 10, OS build 19044.1586
Intel Core i7-7500U CPU @ 2.90 GHz
Intel HD Graphics 620 (Integrated), 8 GB VRAM
16 GB RAM, 2133 MHz

Description

Running cargo run --release -p demo -- svg assets/svgs/paris-30k.svg successfully compiles and displays the expected SVG map, however, all 4 logical processors are pegged at 100%, indefinitely, until I terminate the program.

Add more color spaces

Blending and gradients should have the flexibility of color spaces. This means:

coming up with an API for this
adding sRGB as an alternative to linear

Geometry outside the render target should have minimal impact.

Add a small WASM demo.

Affine transforms can't scale up

If you use anything outside -1 to 1, it fails to convert the affine transform. This almost seems intentional, but I can't find the justification for it anywhere and it seems unnecessary too, as all other renderers I've tried don't have any such limitation.

forma/forma/src/math/transform.rs

Lines 117 to 125 in fcef3e2

 GeomPresTransformError::ExceededScalingFactor { x: true, y: false } => { 

 write!(f, "exceeded scaling factor on the X axis (-1.0 to 1.0)") 

 } 

 GeomPresTransformError::ExceededScalingFactor { x: false, y: true } => { 

 write!(f, "exceeded scaling factor on the Y axis (-1.0 to 1.0)") 

 } 

 GeomPresTransformError::ExceededScalingFactor { x: true, y: true } => { 

 write!(f, "exceeded scaling factor on both axis (-1.0 to 1.0)") 

 }

GeomPresTransform should accept more than just affine transforms

Perspective transforms should be possible on layers as well as paths as long as these don't scale up. Currently, GeomPresTransform only accepts affine transforms, but there is no reason why we shouldn't accept perspective transforms as well.

Every public item in the API should have documentation and examples.

Add a fuzz harness.

Bug showing up in the circles demo

I just ran across forma and decided to see how well it performs by running the circles demo with larger and larger numbers of circles. At 100,000 circles, things get weird. The CPU renderer worked perfectly, but the GPU (both high and low) had weird tearing artifacts (if you have someplace I can upload a video to, I can work to capture one for you so you can see what I'm seeing).

The commands I used were the following:

cargo run --release -p demo -- c circles 100000
cargo run --release -p demo -- l circles 100000
cargo run --release -p demo -- h circles 100000

I'm not a graphics person, so I'm not sure if I'm doing something wrong, I'm just trying to use forma as an easy-to-use 2D vector drawing library. If I'm supposed to be doing something different, please let me know.

Meta

lsb_release -a:
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

uname -a:
Linux Anvil 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

rustc -vV:
rustc 1.66.0 (69f9c33d7 2022-12-12)
binary: rustc
commit-hash: 69f9c33d71c871fc16ac445211281c6e7a340943
commit-date: 2022-12-12
host: x86_64-unknown-linux-gnu
release: 1.66.0
LLVM version: 15.0.2

cargo -vV:
cargo 1.66.0 (d65d197ad 2022-11-15)
release: 1.66.0
commit-hash: d65d197ad5c6c09234369f219f943e291d4f04b9
commit-date: 2022-11-15
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.15.0 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q)
os: Ubuntu 22.04 (jammy) [64-bit]

rustc +beta -vV:
rustc 1.67.0-beta.2 (352eb59a4 2022-12-13)
binary: rustc
commit-hash: 352eb59a4c33abf739914422f2ad975925750146
commit-date: 2022-12-13
host: x86_64-unknown-linux-gnu
release: 1.67.0-beta.2
LLVM version: 15.0.6

cargo +beta -vV:
cargo 1.67.0-beta.2 (f6e737b1e 2022-12-02)
release: 1.67.0-beta.2
commit-hash: f6e737b1e3386adb89333bf06a01f68a91ac5306
commit-date: 2022-12-02
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.15.0 vendored)
libcurl: 7.86.0-DEV (sys:0.4.59+curl-7.86.0 vendored ssl:OpenSSL/1.1.1q)
os: Ubuntu 22.04 (jammy) [64-bit]

rustc +nightly -vV:
rustc 1.68.0-nightly (935dc0721 2022-12-19)
binary: rustc
commit-hash: 935dc07218b4bf6e20231e44eb9263b612fd649b
commit-date: 2022-12-19
host: x86_64-unknown-linux-gnu
release: 1.68.0-nightly
LLVM version: 15.0.6

cargo +nightly -vV:
cargo 1.68.0-nightly (c994a4a63 2022-12-18)
release: 1.68.0-nightly
commit-hash: c994a4a638370bc7e0ffcbb0e2865afdfa7d4415
commit-date: 2022-12-18
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.15.0 vendored)
libcurl: 7.86.0-DEV (sys:0.4.59+curl-7.86.0 vendored ssl:OpenSSL/1.1.1q)
os: Ubuntu 22.04 (jammy) [64-bit]

gcc -v:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04) 

g++ -v:
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04) 

clang -v:
Ubuntu clang version 14.0.0-1ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64

clang++ -v:
Ubuntu clang version 14.0.0-1ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64

$ nvidia-smi
Tue Dec 20 16:09:55 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A400...  Off  | 00000000:01:00.0  On |                  N/A |
| N/A   49C    P8    16W /  N/A |    388MiB /  8192MiB |     10%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2553      G   /usr/lib/xorg/Xorg                192MiB |
|    0   N/A  N/A      2687      G   /usr/bin/gnome-shell               51MiB |
|    0   N/A  N/A    198605      G   ...1/usr/lib/firefox/firefox      118MiB |
+-----------------------------------------------------------------------------+

Add strokes

Huge memory usage for GPU when using textures

Hey, so I had some fun porting text rendering to Forma using parley:

However, when using the GPU, the memory usage is quite high (e.g. for this demo, in release mode, ~400MB on my Mac). When I then start resizing the window, it quickly grows to ~600MB and beyond.

This doesn't happen when using a CPU runner. If I render text without emoji, then I also don't see this behaviour with the GPU runner either.

The way emoji are rendered is by rendering the glyph to a small image (16x16 right now) and then rendering this image with a 16x16 Path.

So only if I'm rendering emoji images on the GPU do I see huge memory consumption (and increasing memory consumption).

Do I need to perform some sort of cleanup when rendering images using Forma with the GPU renderer?

CI os matrix?

I read that the projects wants to target/support multiple OS.

But looking at CI.yml, it only seems to perform CI on ubuntu-latest.

How do you ensure it continues to work on those other OS?

Could an old.matrix be added in the CI GitHub Action workflow to test on the platforms GH Actions support (macos-latest at least?)?

Failed to render (truly) enormous svg file

Feel free to close if this is out of scope, but I have a large (15MB) PDF file that I am trying to view, and I attempted to use forma to do it. I converted it to svg using mutool (preserving text as text), and the result was a 189 MB SVG file.

cargo run --release -p demo -- svg huge_file.svg

caused my fans to kick in and a black box to appear, but no render output. I was able to run the spaceship demo without issues before (on a 2019 MBP 16" with an AMD Radeon Pro 5300M GPU (as well as integrated graphics)).

Happy to attach a profiler if that helps you in any way! Unfortunately I can't share the PDF or SVG file directly (it is secured), but if rendering gigantic SVGs is a goal of the project, I'd like to help.

Improve texture memory footprint on GPU.

forma currently uses rgba16float in multiple places. This was an easy choice early when prototyping when the main driver was to understand how to move the CPU pipeline to the GPU in order to get similar performance characteristics. The issue of high memory usage was brought up in #23.

I propose investigating what would be better choices for texture formats for both the output texture and the atlas texture.

SVG linearGradient `stop missing gradient start tag` error

<svg
  viewBox="0 0 10 10"
  xmlns="http://www.w3.org/2000/svg"
  xmlns:xlink="http://www.w3.org/1999/xlink">
  <defs>
    <linearGradient id="myGradient" gradientTransform="rotate(90)">
      <stop offset="0%" stop-color="gold" />
      <stop offset="58%" stop-color="red" />
    </linearGradient>
  </defs>

  <circle cx="5" cy="5" r="4" fill="url('#myGradient')" />
</svg>

thread 'main' panicked at 'stop missing gradient start tag', demo/src/demos/svg.rs:847:49

thread 'main' panicked at 'failed to find an appropriate adapter', demo/src/runner.rs:87:23

Hi, just testing it on Linux ppc64le (IBM PowerPC 64-bit Little-Endian) with an old graphics card and I get this error message, which has probably more to do with wgpu-rs but might be useful to have as reference:

target/release/demo cpu circles
thread 'main' panicked at 'failed to find an appropriate adapter', demo/src/runner.rs:87:23
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The backtrace is not informative, just says that this happened inside main() and the rest is boilerplate, but I’ll add it if you request it.

I’ve just tried wgpu-rs in case the problem was there, and the examples run although with an error message:

wgpu$ cargo run --release --example skybox
   Compiling wgpu v0.14.0 (/tmp/ramdisk/wgpu/wgpu)
    Finished release [optimized] target(s) in 14.02s
     Running `/tmp/ramdisk/wgpu/target/release/examples/skybox`
[2022-12-16T17:23:35Z ERROR wgpu_hal::vulkan::instance] enumerate_adapters: Initialization of an object has failed
Using AMD TAHITI (LLVM 12.0.1, DRM 2.50, 5.19.16_1) (Gl)
Avg frame time 11.352801ms

This and other examples run, although examples/hello does not:

wgpu$ cargo run --release --example hello
   Compiling wgpu v0.14.0 (/tmp/ramdisk/wgpu/wgpu)
    Finished release [optimized] target(s) in 5.66s
     Running `/tmp/ramdisk/wgpu/target/release/examples/hello`
[2022-12-16T17:23:05Z ERROR wgpu_hal::vulkan::instance] enumerate_adapters: Initialization of an object has failed
[2022-12-16T17:23:05Z ERROR wgpu_hal::vulkan::instance] enumerate_adapters: Initialization of an object has failed

I was surprised because I thought that using the cpu device the graphic card would not matter.

Add a data structure that would automate layer order allocation

We should provide a data structure on top of the Composition that handles Order allocation automatically. This means orders optional and having this data structure decide the precise order of all the layers.

	GeomPresTransformError::ExceededScalingFactor { x: true, y: false } => {
	write!(f, "exceeded scaling factor on the X axis (-1.0 to 1.0)")
	}
	GeomPresTransformError::ExceededScalingFactor { x: false, y: true } => {
	write!(f, "exceeded scaling factor on the Y axis (-1.0 to 1.0)")
	}
	GeomPresTransformError::ExceededScalingFactor { x: true, y: true } => {
	write!(f, "exceeded scaling factor on both axis (-1.0 to 1.0)")
	}