I'm having issues combining File Open dialogs and timers in the druid-shell hello example.

Currently File Open dialogs block while holding a mutable borrow on the WinHandler until the dialog comes back with a result path. If a timer event triggers while the dialog is open, it panics with "already borrowed".

#95 mentions a possible solution. Are there other solutions to consider?

As a workaround for now I don't block when opening the dialog, immediately returning an Err(Error:Null) in get_file_dialog_path and I have a closure that runs with the actual result but I'm not sure where to send it. If a solution is decided upon I can take a shot at a PR.

You can provoke a panic in the master branch by running the sample example, selecting File Open from the menu, then doing a dynamic monitor change (for example, unplugging an external monitor). This is because Windows requires the wndproc to be reentrant, and our implementation uses a (panicking) RefCell::borrow_mut() for safety.

There's a Reddit thread which goes into some more detail, including pointers to how winit handles this. There's also some discussion on the Zulip.

The upshot is that we want to refactor the way we handle interior mutability. In particular, druid-shell should handle it, so the WinHandler can be called mutably and not need its own interior mutability. There's also followup work to make the file dialog not hold long-lived borrows, but that's significantly lower priority.

This came up because we need a good way to plumb additional resources, particularly a factory for creating text layouts, to user methods.

This is a followup to concerns raised in #87.

There might be a second issue here (my mac is in the shop because the butterfly keyboard got a crumb in it, so I can't test), which is whether NSCursor::set leaks globally into other app windows. If so, we should probably change it so there's a single rect that covers the entire window (or, to be more precise, the area of the NSView that serves as the root of the druid widget hiearchy, when thinking about VST guest use cases). I think that change can be made at the shell level without affecting event flow or API, if needed.

@cmyr observed that NSCursor.set can clobber the global cursor.

The Apple docs are confusing. If you set up these tracking areas, and it drills down to Cocoa calling your view's cursorUpdate method, of which their suggested impl (see Listing 6-4) is to just call NSCursor::set. So it's unclear if this is really what's causing the clobbering of the global cursor, or whether it might be something else such as a grab that's not being let go.

There's also the performance issue of if we should call the set_cursor every time, or only when the cursor has actually changed. This is subtle, because of the need to track when the mouse has left the window on Windows. See discussion in #87 for details.

Investigate the clobbering of the global cursor on mac, and consider the performance improvement.

I regularly want to play around with some drawing code, and it's annoying to have to bring up a new sample project each time. It would be nice if there was an example I could just tweak as needed.

I'll probably do this at some point; if someone else wants to first, they should feel free!

As of #42 the window is now "key" when it is created, meaning that it can respond to mouse move events, but on startup from the command line, there are two visual phases with a transition. The first phase shows the window but behind the terminal (at least on my machine), and the second phase pops it to the front. This is not appealing.

For most apps (ie .app bundles) the activation is handled by Launch Services, not by the app binary itself. Colin and I experimented with various things, changing the order of activation, splitting makeKeyAndOrderFront into its two separate components, etc., but weren't able to figure out the Right Way to do this.

This would be a great starter issue for somebody with low level Cocoa knowledge.

As part of #156 I included a new method to WinHandler, got_focus(). This should be called whenever focus changes to a different window. This is implemented in mac but needs to be implemented for windows.

Something that is clearly needed for Runebender is the ability to set custom cursor images, so that we can have a pen cursor when we're in the pen tool, etc.

It would be good to

• Run CI:
• cargo build && cargo build --release for pull requests
• cargo test
• Add Github integration

Best is probably go to with Cirrus CI since it supports windows.

Environment and widget state

This is an attempt to sketch out a mechanism for controlling widget display properties (like colors or text size) in a way that allows support for theming, accessibility, and resumable/interruptable animations.

Faux dynamic typing with enums

At the core of this proposal is the idea of using enums to store homogenous, per-widget data.

This involves using a custom enum that has members for each concrete type that is used while drawing:

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Value {
    Point(Point),
    Size(Size),
    Rect(Rect),
    Color(Color),
    Float(f64),
}

Each of these concrete types implements Into<Value>, and TryFrom<Value> (we may wish to use a custom trait in place of TryFrom, but it works for prototyping):

let point = Point { x: 101.0, y: 42. };
let value: Value = point.into();
assert_eq!(Value::Point(Point { x: 101.0, y: 42. }), point);
let from_value: Point = value.try_into().expect("value of wrong concrete type");

To use these values, we put them in a map, and access them with a special getter method:

pub struct Variables {
    store: HashMap<String, Value>,
}

impl Variables {
    pub fn get<E, V: TryFrom<Value, Error = String>>(&self, key: &str) -> Result<V, String> {
        match self.store.get(key)
            .ok_or_else(format!("No entry for {}", key))
            .and_then(|v| v.try_into())
    }
}

This is the 'safe' version. In practice, this is probably unnecessarily cautious, and we can have get panic if the value is missing or of the incorrect type. The reason for this is that keys and their types should not change at runtime.

How we can use this

For Custom state

When a widget is instantiated, it has an opportunity to register custom keys and values that it will use when drawing:

trait Widget {
    fn init_state(&self) -> Option<Variables> {}
}

struct BouncingBall;

impl Widget for BouncingBall {
    fn init_state(&self) -> Option<Variables> {
        let mut vars = Variables::new();
        vars.set("ball.ypos", 0.0);
        Some(vars)
    }
    
    fn paint(&mut self, paint_ctx: &mut PaintCtx, geom: &Geometry) {
        let x = geom.min_x();
        let y = geom.min_y() + paint_ctx.state.get("ball.ypos");
        let circ = Circle::new((x, y), 40.);
        let fill_color = paint_ctx.render_ctx.sold_brush("0xFF_AA_DD_FF");
        paint_ctx.fill(circ, &fill_color, FillRule::NonZero);
    }   
}

The main motivation for this pattern is to allow drawing code to be animation aware.

With animations:

In the animate branch I've been experimenting with a more automated animation API. With this API animations are represented as a start and an end state, an animation curve, and a duration. The framework handles interpolating between the provided values; on each animation tick, the widget gets a callback with the current state for those values.

// on key down we create a new animation that includes a color and a float
fn key(&mut self, _event: &KeyEvent, ctx: &mut HandlerCtx) -> bool {
    let anim = Animation::with_duration(2.0)
        .adding_component("a_float", AnimationCurve::OutElastic, 1.0, 350.0)
        .adding_component(
            "a_color",
            AnimationCurve::Linear,
            0xFF_00_00_FF,
            0x00_00_FF_FF,
        );
    ctx.animate(anim);
    true
}

// on each animation tick, we receive the active animation, and can access the interpolated values
// this works just like the `Variables` dictionary above.
fn animate(&mut self, anim: &Animation, ctx: &mut HandlerCtx) {
    self.0 = anim.current_value("a_float");
    self.1 = anim.current_value("a_color");
    ctx.request_anim_frame();
}

This works okay, but introduces an annoying step: when you get the callback, you have to stash the updated values somewhere, request a frame, and use them when drawing.

Combining with custom widget state:

If, however, we combine this with the custom widget state discussed above, we can make everything just work; the keys in the animation can refer to keys declared as part of the widget's state, and then while animating we can just provide the correct interpolated values in the state that gets passed to the draw call.

In this world, then, if we want to animate a color change of the window background on a key press, we can just do:

impl Widget for BouncingBall {
    fn init_state(&self) -> Option<Variables> {
        Some(variables!["bg_color", Color::Red])
    }

    fn paint(&mut self, paint_ctx: &mut PaintCtx, geom: &Geometry) {
        let bg_color = paint_ctx.state.get("bg_color");
        paint_ctx.clear(&bg_color);
    }
    
    fn key(&mut self, _event: &KeyEvent, ctx: &mut HandlerCtx) -> bool {
        ctx.animate(Animation::with_duration(1.0)
            .adding_component("bg_color", AnimationCurve::Linear, Color::PURPLE)
            );
        true
    }
}

As an added bonus, we can use this to make our animations begin from their current position; if an animation is added while another is active, we can interpolate from the existing animation's values instead of starting from the beginning or end.

Environment

Another place where this dynamic Value + Variables pattern shines is with the concept of an 'environment'. The environment is a way of passing scoped state up the widget tree. An obvious use for this is with theming; we would like things like color schemes to be customizable by the application author in an easy way, while still providing reasonable defaults.

A simple approach to this is to use a Variables struct with a known default set of keys, which built-in widgets will use for their styling:

impl Widget for Button {
    fn paint(&mut self, paint_ctx: &mut PaintCtx, geom: &Geometry) {
        let bg_color = match (paint_ctx.is_active(), paint_ctx.is_hot()) {
            (true, true) => paint_ctx.env().get(colors::CONTROL_BACKGROUND_ACTIVE),
            (false, true) => paint_ctx.env().get(colors::CONTROL_BACKGROUND_HOVER),
                _ => paint_ctx.env().get(colors::CONTROL_BACKGROUND),
        };
        let brush = paint_ctx.render_ctx.solid_brush(bg_color);
        paint_ctx.render_ctx.fill(geom, &brush, FillRule::NonZero);
    }
}

And then if an application author wanted to have a custom theme, they could do something like,

let mut state = UiState::new();
// ...
state.env().set(colors::CONTROL_BACKGROUND, Color::DEAD_SALMON);
state.env().set(colors::CONTROL_BACKGROUND_ACTIVE, Color::BRIGHT_PINK);

These custom colors would then be used by the built-in button widget, as well as any other widgets in the tree that used this mechanism and referenced those keys.

Custom environment keys

New widgets can also participate in the environment, by adding their own keys and values when they are instantiated. This means that a custom widget can have its attributes overridden via the same mechanism as the default widgets.

Further thoughts:

Typed keys, if we want?

For even more type safety, we have the option of using typed keys. Essentially this looks like,

pub struct Key<T> {
    key: &'static str,
    value_type: PhantomData<T>,
    }
    
    impl<T> Key<T> {
        pub const fn new(key: &'static str) -> Self {
        Key {
        key,
        value_type: PhantomData,
    }
}

impl Variables {
    pub fn get<V: TryFrom<Value, Error = String>>(&self, key: Key<V>) -> V {
        let value = match self.store.get(*&key.key) {
        Some(v) => v,
        None => panic!("No Variables key '{}'", key.key),
        };
        value.into_inner_unchecked()
    }
}

// Example:

let vars = make_variables();
let point_key = Key::<Point>::new("my_point_key");
let my_point = vars.get(point_key);
// panics if this value is not a point

This is currently implemented in my experimental branches, and offers some nice guarantees; for default theme attributes these keys are pre-baked, so we can guarantee that the types are correct, and there's no possibility of accidentally having the incorrect inferred return value; you have to be explicit about what you expect when you create the key. This also encourages the practice of declaring all keys as consts, which is good practice.

Keys referencing Keys

It may be that for the widget state, we want the ability to include a key that references a value in the environment itself. This would be important for animations. As a concrete example, lets say we want a button background to animate between the dynamic colors::CONTROL_BACKGROUND and colors::CONTROL_BACKGROUND_ACTIVE colors. In the buttons paint call, we want to have access to a 'bg_color' state variable that will represent the current color.

If we want the default for this value to be whatever is currently declared in the theme, we might want to do something like:

fn init_state(&self) -> Option<Variables> {
    let mut vars = Variables::new();
    vars.set("bg_color", colors::CONTROL_BACKGROUND);
    Some(vars)
}

Here the value for this key isn't itself a color, but is instead another key that resolves to a color.

Arbitrary types in Value

If we wish, we can include a Custom variant in Value, which can store anything that can be dyn Any. We can provide a custom macro that writes the correct TryFrom / Into implementations for a custom type. I've prototyped this and have it compiling, but I want to hold back until we have a concrete use case.

State Magic

If we get ambitious, there is some fancy macro stuff we could do make init_state unnecessary. I could imagine code that looked something like,

#[druid(widget)]
Button {
    #[druid(state, init = colors::CONTROL_BACKGROUND)]
    bg_color: Color,
    #[druid(state, init = colors::BUTTON_BORDER)]
    border_color: Color,
    #[druid(state, init = botton::BORDER_WIDTH)]
    border_width: f64,
    label: Label,
}

It is not too difficult to imagine getting from here to something that provides some of the qualities of SwiftUI (for instance, changing a state variable automatically triggers an invalidation).

Conclusion

These are preliminary thoughts, but I wanted to get a rought cut of this down so we could discuss before I dig too much more into the implementation. If there's anything I can clarify, please let me know, otherwise feedback is welcome!

I just updated one of my projects building on druid and now I get this error, which i'm not entirely sure how to fix.

error[E0277]: `*const objc::runtime::Class` cannot be sent between threads safely
   --> /Users/james/.cargo/git/checkouts/druid-1a6b6c4d20db75fe/cfbde68/druid-shell/src/mac/mod.rs:190:1
    |
190 | / lazy_static! {
191 | |     static ref VIEW_CLASS: ViewClass = unsafe {
192 | |         let mut decl = ClassDecl::new("DruidView", class!(NSView)).expect("View class defined");
193 | |         decl.add_ivar::<*mut c_void>("viewState");
...   |
282 | |     };
283 | | }
    | |_^ `*const objc::runtime::Class` cannot be sent between threads safely
    |
    = help: within `mac::ViewClass`, the trait `std::marker::Send` is not implemented for `*const objc::runtime::Class`
    = note: required because it appears within the type `mac::ViewClass`
    = note: required because of the requirements on the impl of `std::marker::Sync` for `spin::once::Once<mac::ViewClass>`
    = note: required because it appears within the type `lazy_static::lazy::Lazy<mac::ViewClass>`
    = note: shared static variables must have a type that implements `Sync`
    = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

error: aborting due to previous error

   Compiling druid-win-shell v0.1.1
error[E0308]: mismatched types
   --> /home/vi/.cargo/registry/src/github.com-1ecc6299db9ec823/druid-win-shell-0.1.1/src/window.rs:809:48
    |
809 |         SetWindowLongPtrW(hwnd, GWLP_USERDATA, wndproc_ptr as LONG_PTR);
    |                                                ^^^^^^^^^^^^^^^^^^^^^^^ expected i32, found isize

This particular error does not occur for --target x86_64-pc-windows-gnu.

Adding as i32 resolves the error and produces an exe (not runnable with Wine although due to d2d1 not having been implemented enough).

See #165 (comment) for context. This should allow us to write more idiomatic Rust code by avoiding mutability where it isn't necessary.

@raphlinus Hope that summarizes the intention. If there's anything I missed, please feel free to add. 😅

I'm playing around with building druid on linux and got a small sample running based on the gtk fork from bobtwinkles/druid.

The fork hasn't been updated in a 5 months and seems to need some changes to work with the current state in the main druid repo.

What's the plan on moving this forward? Is gtk the way to go and should focus be directed to updating the fork to work with the latest code base, or is there another approach you were thinking about.

I noticed in #90 that @Dmitry-Borodin was also interested in working on linux. Maybe we can collaborate?

As a quality-of-life improvement, we'd like PaintCtx to deref into a Piet render context, so you don't have to write stuff like paint_ctx.render_ctx.draw_text() all the time.

Basically, the work is to add a Deref and DerefMut, and then to clean up all the uses in the paint methods.

I would like some object at a higher level than a window that can serve as a general application coordinator. This will be responsible for handling top-level lifecycle and system events, and managing windows.

A basic question (with a macOS bias): if I have no windows open, who handles a File > Open event?

So after looking over the code, I have an idea of where to start with platform independence. I'd like to start from the inside and work our way out, first by changing the signature for the Widget trait and then simply following the types. Before progress on this can be made, there are some questions to answer:

• What should the signature of paint be? Currently it is fn paint(&mut self, paint_ctx: &mut PaintCtx, geom: &Geometry). paint_ctx is simply a wrapper around the rendering context (both direct2d and directwrite) plus the information of whether the widget is hovered (a.k.a hot) or pressed (a.k.a active). Swapping the inner render context should be a straight 1 to 1 swap with piet RenderContext, but is this the right place for the hot and active bools to live?
• Should LayoutCtx include a RenderContext? The LayoutCtx currently takes a directwrite::Factory so that it can do things like layout text to get its size. If we pass in a RenderContext we'll be able to get this info. We can also pass in a TextLayoutBuilder or TextLayout which will also get us that information. However, we then need to parameterize LayoutCtx on one of these traits which will force us to bubble up a generic to LayoutCtx and HandlerCtx which is unfortunate.

event, unlike the other methods, takes the event before the context, which sticks out a bit. It should use the same order as the others for consistency

Hi Raph,

I thought i'd have a go at making druid work with wasm, since piet appears to support it (although curiously with the web feature rather than selecting by target architecture?).

druid-shell brings up 8 errors when i try and do it there. my gut feeling is that this is probably not the place to make the change, since the errors are for Window, WindowBuilder etc.
Trying to make the changes in in druid itself is definitely less good, with rustc giving up after 200 errors.

I know you've said it's probably a bit early to be worrying about too many backends, but given the logistical differences with this one, i thought it would be a good test of druid's design.

I'm willing to continue with the work as i'm able, but i think i'm going to need some guidance on the best way to go about it to get any further.

i tried to build project (at master branch) with cargo [1.36.0] build but got this error

<...>
   Compiling piet-common v0.0.4
   Compiling druid-shell v0.3.0 (<...>/druid/druid-shell)
error[E0432]: unresolved import `crate::platform`
  --> druid-shell/src/window.rs:21:5
   |
21 | use crate::platform;
   |     ^^^^^^^^^^^^^^^ no `platform` in the root

error[E0432]: unresolved import `platform`
  --> druid-shell/src/lib.rs:50:9
   |
50 | pub use platform::application;
   |         ^^^^^^^^ use of undeclared type or module `platform`

error[E0432]: unresolved import `platform`
  --> druid-shell/src/lib.rs:51:9
   |
51 | pub use platform::dialog;
   |         ^^^^^^^^ use of undeclared type or module `platform`

error[E0432]: unresolved import `platform`
  --> druid-shell/src/lib.rs:52:9
   |
52 | pub use platform::menu;
   |         ^^^^^^^^ use of undeclared type or module `platform`

error[E0432]: unresolved import `platform`
  --> druid-shell/src/lib.rs:53:9
   |
53 | pub use platform::util;
   |         ^^^^^^^^ use of undeclared type or module `platform`

error[E0432]: unresolved import `platform`
  --> druid-shell/src/lib.rs:54:9
   |
54 | pub use platform::win_main as runloop; // TODO: rename to "runloop"
   |         ^^^^^^^^ use of undeclared type or module `platform`

error[E0432]: unresolved import `platform`
  --> druid-shell/src/lib.rs:55:9
   |
55 | pub use platform::WindowBuilder;
   |         ^^^^^^^^ use of undeclared type or module `platform`

warning: unused `#[macro_use]` import
  --> druid-shell/src/lib.rs:28:1
   |
28 | #[macro_use]
   | ^^^^^^^^^^^^
   |
   = note: #[warn(unused_imports)] on by default

error: aborting due to 7 previous errors

For more information about this error, try `rustc --explain E0432`.
error: Could not compile `druid-shell`.

Not sure how seriously to take this, but it came up in #149, and we should know what's going on here.

I tested a bit on Windows as well (but see comment inline about key event matching which I had to bodge). The good news is that it seems pretty robust; I wasn't able to trigger any crashes. But there are signs it isn't quite right yet, I get this logged:

2019-09-14 13:52:18,273 ERROR [druid_shell::windows] dropped message 0x2, hwnd=0x220034, wparam=0x0, lparam=0x0

I did a modest (not incredibly deep) pass through the rest and it seems pretty good to me. I like that it's simpler than my PR (exposing less of the internal event dispatching) while also adding the Command abstraction. Basically I'm comfortable with this going in soon, seems like the major issue is stability around window lifetimes. Do we want to debug and fix that in this PR, or get the API in place and treat it as a separate bug?

With #11 drawing will be cross platform. The question is then "what next?". I propose we next try to make WinHandler cross platform. WinHandler is the trait that defines how to respond to window events like painting, size changes and keyboard/mouse events.

Here's a list of items that need to be addressed:

• WinHandler::connect takes a WindowHandle which has a weak reference to a WindowState which is highly Windows specific as it has a reference to HWND and WndProc. connect stores the WindowHandle on the state's LayoutContext. The handle is used for the following: invalidating widgets, closing the window, file dialog, converting physical pixels to px units, getting the current dpi. Perhaps this can all be abstracted away in a trait that is made concrete in a similar way that we do with piet-common
• The various keyboard handlers take modifier keys which can be different on different platforms. We could solve this the same way web APIs do: https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/getModifierState
• Realted to the first issue: WinHandler::mouse, WinHandler::mouse_move and WinHandler::size all use the windows specific WindowHandle that is stored on the UiState's LayoutContext
• WinHandler::destroy calls the windows specific win_main::request_quit()

Thoughts on how to abstract WindowHandle and how to handle modifier keys?

This could even just be a newtype around u32 for now, but we will need one eventually and it makes sense to put something in place now.

crossplatform

https://github.com/ggez/ggez
https://ggez.rs/

the relation with gfx-rs:
https://gfx-rs.github.io/2019/06/12/anniversary-5.html

See docs for info: mem::uninitialized is deprecated, and should be replaced by mem::MaybeUninit.

a while ago I thought why is there no none object-oriented UI framework that I can find
then I heard of IMGUI then someone said you can't use IMGUI because you need UI to be data-oriented then I thought why can't IMGUI be data-oriented then I read this https://uiink.com/articles/data-driven-immediate-mode-ui/
edit: I'm rewatching your talk and noticed you mentioned IMGUI before so I added that it should be data-driven IMGUI and also I have no proof of this but I wouldn't assume that IMGUI has to repaint the whole world

There was preliminary work in this direction, but I think that's been canned. Not sure if this was for any structural reason, but if not it would be good to have xi-trace support.

we're currently dumping a bunch of stuff to stderr, but to properly support application use we should be using (I suspect) the log crate.

This would be a builder style fn, so that you could write:

    AppLauncher::with_window(main_window)
        .print_logs(true)
        .launch(State::default())
        .expect("launch failed");
}

And then when we start the application we could setup a very barebones log handler that just calls eprintln. (@LiHRaM had something like this in an early draft of #135, which we might be able to reuse?)

make a list of widgets to be implemented

We have an informal policy not to commit code that prints warnings; it would be nice to make this an official policy, by baking it into CI.

If anyone wants to dig into this, that would be very welcome!

This issue collects the rough edges I've run against while adapting the synthesizer.

• There's some confusion with coordinate types, the old code uses f32 as that's most natural for Windows, yet kurbo tends to use f64. Some of this can be made smoother by adding more into conversions in kurbo taking f32 as well. I'm a little uncertain what we should use as the default. One possibility is the Coord associated type from piet. Basically, there's no compute-time cost for using f64, but it does potentially use more memory if not needed.

• We should probably merge linebender/piet#37.

• I am also more inclined to do linebender/piet#34. All the error possibilities are annoying, and not very useful.

• I missed having a circle shape. This belongs in kurbo, but I'm faking it by using the "circle" from piet-test's picture_1.

These should be using native menus wherever possible.

this is currently hard-coded.

I think Label and DynLabel should also get moved into their own file.

This will make it more useful, and cover up some current papercuts like the fact that application windows will not have useful names until we wire up more app bundle stuff.

As we work toward the "longer term future" as described in #16, a pasteboard (the Mac word) / clipboard (the Windows word) would very nice to have!

One possibility is to rely on the Clipboard crate. In fact, it seems like it's not totally necessary for this functionality to be inside druid-shell or even druid? To my understanding, the winit project declared clipboard functionality out of scope for this reason.

In the longer term, I would expect a clipboard that supports data types beyond text, or a non-global pasteboard, will be important for apps like Runebender. But maybe that's running before we can walk.

As mentioned in #134, we need some sort of enabled / disabled state for the button widget. It also makes sense for other widgets: slider, checkbox (I'll have a PR for that soon), textbox, scrollbar, and I'm sure other widgets to come.

To kick things off, here's @raphlinus's comment:

...what I was thinking (and this is borrowed to some extent from SwiftUI) is that the enabled state becomes part of env (probably with specific API surface for it; doesn't have to go through the typed key/value mechanism) and widgets respond to it. There are definitely questions though.

This sounds pretty straightforward to implement. I'm not sure what the questions are, though!

This is an event that is sent to widgets when the application first runs.

We likely want a fairly full suite of lifecycle events, but for the time being I think we should just add them as needed. This is an obvious candidate.

This is a continuation of a stream of work started in #52 and #76. That kicked off a refactor where we're passing down a mutable reference to a WinCtx in window handler functions, rather than having shell and window handler share a reference to a WindowHandle and use interior mutability to access it.

We've migrated some functions to WinCtx (invalidation, text factory, and setting cursor), but others remain in WindowHandle. It'd be nice to migrate most of the rest, or maybe even all of them and remove WindowHandle entirely.

Important remaining methods on WindowHandle, with notes on how to handle them:

• close. Shouldn't provide any special difficulty, just needs per-platform implementation.

• hi-dpi methods. These also shouldn't be difficult, but I'd like to change the conventions a little. Right now, they're expressed (in a Windows-centric way) as a dpi value, with a nominal value of 96.0. I'd rather use a dpi scaling factor, with a nominal value of 1.0.

• file dialogs. These are trickier. It would be possible to just move the existing API over, but the idea of doing a blocking call to create a modal dialog from inside the window handler creates problems. A better API would be to request a file dialog (as a method on WinCtx), and pass in a closure which should be called on completion of the dialog. The implementation would then wait until control passes up from the window handler, and then do the blocking call from inside the shell, specifically, while not holding a mutable borrow on WindowState. This would let handlers run (eg for painting the window) while the modal dialog is up. It would also adapt much more easily to a non-modal dialog, which might be better UX.

• idle handle (for external events). This is probably more of a convenience, as the idle handle will continue to be implemented with a shared reference and interior mutability.

• show. This doesn't need to be called by the window handler, only by whoever created the window. Possibly we split this up so the object returned by WindowBuilder::build() is different than the one internal to the window handler. This could get slightly more interesting when we have multiple windows. Also, it's possible we simplify the API and make the need to this method go away, all windows are implicitly shown right after WindowBuilder::build (we might need to dig into whether there are useful things that should be done after build but before show).

This issue presents the roadmap for druid-shell development over the next few months, and a bit of vision of the longer term plans. A goal of this issue is to clearly define what is in scope and out of scope for this crate.

Current state

Right now, druid-shell is druid-win-shell. It provides window creation, basic mouse and keyboard input, and a small amount of native platform UI services: file dialogs and menus. The drawing surface for a window is (as of recently) piet, which wraps the Direct2D context.

For historical context, druid-win-shell used to be xi-win-shell, which was factored out of xi-win at some point. The original xi-win codebase was monolithic and not factored into layers.

There is basic support for high-dpi, but not dynamic per window dpi switching (the APIs for this are tricky and only exist in recent Windows 10 versions). Getting high-dpi right is a goal.

One particular feature that is perhaps over-developed compared with the maturity of the rest of the crate is smooth window resize. There is a sophisticated hybrid presentation scheme that switches between a swap chain for steady state and drawing to the redirection surface during live resize. There's more discussion of this in rust-windowing/winit#786, one of the issues blocking the use of winit, about which more later.

Near future

We want to move druid to platform independence, and most of the Windows dependencies are in the shell layer. Much of the immediate work is tracked in #13.

A major choice is whether to use winit or to do our own window creation. For Windows at least, I'd like to preserve the existing window creation logic so as not to regress smooth resize, and also so that we can do quantitative measurements. For other platforms, I'm open to the idea of winit, but I'm also open to doing our own. We may have two window creation back-end (probably controlled by feature) for a while until all this sorts out.

Aside from platform independence, some of the features coming up include:

• Incremental present. One of the goals of druid is high performance, and an important piece of that is to only draw what's actually changed on the screen. An important but special case is blinking the cursor in a text editor. On Windows, we want to use IDXGISwapChain1::Present1 for this, and in general provide plumbing for incremental present. Note that incremental present is probably going to be tricky on other platforms; it's not clear that macOS has native support at all (see chromium#474299 for one way to fake this).

• Low latency present. Right now, when animating, we block in the present call until the next frame is available. During that blocking, we can't take input, so it adds to the overall latency. DXGI 1.3 has latency wait objects that can decrease latency.

• IME. The keyboard handling is primitive, and there are already issues like xi-editor/xi-win#67. It would be great to have a cross-platform abstraction for IME. Note that this is a hard problem, but also one I've investigated a bit.

• A gfx-hal context. The primary 2D drawing interface is piet, but I'd like to support 3D apps as well. It seems to me sensible to standardize on gfx-hal for this.

I'd like to focus on Windows, macOS, and Linux for the near term. I think it will also be interesting to target web, one of the reasons there is a web back-end for piet.

Longer term future

Longer term, the scope of druid-shell is to provide an abstraction for the interface between applications and operating system services for UI. I can see this including:

• Access to preferences. When the OS provides a preferences mechanism, it's better to use that than inventing our own. This also includes being sensitive to system-wide preferences, including locale.

• Accessibility. Getting accessibility right requires hooks to the native platform, not least of which is respecting preferences (see above). This probably also includes features such as speech.

• Paste buffer. This is particularly important for xi-win, but is generally useful.

• Drag and drop. Another basic and nontrivial UI function.

• More platforms. In addition to the platforms listed above, Android and iOS are very interesting. There is a longer tail of platforms, including Fuchsia, but it's likely serious work on those shouldn't start until there is a body of applications people want to run. A personal goal of mine is Nintendo Switch; it's likely that this work can't be completely open source, but I'd like as much of at as possible to be.

What to do about winit?

Long term, it would be good to merge the window creation and input with winit, so there is one common create for these services. Right now, I'm reluctant to do so, as it would regress performance (and performance is a major goal of druid), and we can probably be nimbler in experimenting and getting changes landed faster.

It's also going to be interesting to see how much the scope of winit increases. There are issues for drag and drop and IME (see also rust-windowing/winit#753), but apparently not preferences, accessibility, speech, or pasteboard. There's a bit of discusssion of native widgets for menus and file dialog but not movement.

Non-druid layers

The broader goal of building cross-platform UI is deliberately factored into smaller crates, in large part to enable a mix-and-match approach. I can see people wanting to experiment with other UI approaches than druid. As druid-shell matures, it will probably become an appealing layer. I want to support, that but not at the cost of bringing in too much premature generality. The main goal of druid-shell is to support the needs of druid.

Things not in scope

The scope of druid-shell includes presenting a drawing surface to the higher layers, but nothing about what's drawn in that surface. So widgets, layout, etc., are not in scope. This boundary may become fuzzy if we want to support embedded native widgets, but this is not foreseen as a major use case. In particular, on macOS and iOS, I see building the UI with druid widgets, rather than native Cocoa.

I've investigated the use of the compositor (see discussion of planeshift in pcwalton's 2019 plans blog. This is a very complex tradeoff, and for druid I'm resolving it in the direction of not integrating with the compositor. It certainly is true, though, that the compositor is the lowest power solution for cursor blinking and scrolling on most current systems. Part of my thinking is that I want to enable hardware overlays, as these are lower latency and lower power on forward-looking systems (DirectFlip). See servo/webrender#3115 for deeper discussion. (I can see why Servo is going this direction, and thus druid-shell likely won't be a good solution for Servo).

Similarly, I'm not planning on exposing the scroll functionality in Present1. This seems to be a fairly narrow Windows feature (though it could be faked on macOS/iOS using the compositor).

If the OS does not directly support menus (Linux doesn't, macOS and Windows do), then it is not in scope for druid-shell to polyfill that. Rather, it reports the lack of menu support to the higher level (druid), and it is up to that level to implement its own menu widgets if not available.

A great many OS services not directly tied to the UI are out of scope, including audio, MIDI, networking, etc. Speech is an interesting question.

It is a goal of druid-shell to be a cross platform abstraction, but that abstraction is sometimes leaky. Instead of trying to be heroic, the general approach is to provide access to platform-specific APIs, so that applications can call in directly (we can use any and #cfg as appropriate to gate this access).

Note: not all these decisions are set in stone, and we can reconsider them based on the needs of apps built on top. But I see deliberately restricting scope (at least for now) as a good way to maintain momentum.

Discussion is welcome.

This will be regularly needed, no reason not to hang it off of the type.

The is_printable fn on KeyCode is not very useful, because it alone does not tell you whether a key event will have printable text.

As an example: ctrl + e on macOS uses the 'printable' KeyE keycode, but sends the unprintable ascii value 0x5. similarly with ctrl + u, ctrl + k, and others.

First, I hope the title isn't confusing. I wasn't quite sure, what the right description of this might be.

There's a small problem when building this on mac.
A lot of people will have libffi installed via homebrew, this causes a little problem.

An Error message pops up, stating that pkg-config is not able to locate libffi

this can be fixed fairly easily using this command here:

export PKG_CONFIG_PATH=/usr/local/opt/libffi/lib/pkgconfig

Now pkg-config will find the ffi library and one is ready to have a good time playing around with druid and the examples!

Discussion:
Add a troubleshooting section to the readme

Currently text measurement is only possible once a handle to the RenderCtx is available, which is only in the paint fn. To do layout correctly, we need to be able to measure text in the layout fn. Not sure exactly how this should be plumbed, but it is currently preventing us from properly implementing most text-focused widgets.

This issue is a continuation of the review discussion for #69, which was a very basic scroll container, and describes some refinements that would be nice.

The easiest is probably scroll bars. Right now, there's no visual indication of scroll position. It would be pretty straightforward to draw those on top of the content, and not all that hard to wire up the click-drag gesture. I'd consider this "help wanted."

Of course, the scroll bars need to be subject to styling. This is a much deeper question, which we might get back to in a while. In the meantime, I think it makes sense to start to develop a default theme (at first hardcoded), and focus on making that not-so-terrible.

There's a cluster of issues around dynamics, listed in the review as "momentum and bounce." These require animation frames, which is currently not wired. I'd like to do something similar the "request_anim_frame" mechanism in old druid, but in any case that needs to happen before we can add things to this widget.

I think momentum is handled by the system on macOS. It seems to deliver a series of events to the app. We can get more metadata, including the momentum phase (this would allow us to turn off momentum by suppressing events), but I'm not sure how important this is. I believe on Windows momentum is not considered idiomatic. (For reference, see Handling Trackpad Events in the Cocoa docs, as the full range of trackpad gestures is extremely rich)

Bounce is legit, though, and I think can just be configured at widget creation time. It requires animation frames.

I'd also like to do smoothing, but this is something of a deeper question. This breaks down into two sub-issues. First, sometimes you get a discrete scrolling bump (for example from the scroll wheel, but also potentially from the content) and want to animate that out. Second, even more or less continuous sources of scroll deltas (trackpads) are noisy, with quantization both in time and value. For the latter, I think doing smoothing can improve the overall feel of the app. I'd like to do something very similar to the midi smoothing in synthesizer.io, but no doubt this will require some fine-tuning of parameters to make sure it feels good. (As a sub-issue, I'd also like druid-shell to have a more precise, jitter suppressed timebase for animation events, right now it just measures wall clock at event processing time - this should probably also be tracked by an issue because I'm not sure when I'm going to get to it).

Related to the above, the content window should also be able to request scrolling, for example in a text editor. I think an appropriate mechanism for this is to have an Action that's propagated up in the event flow. Similarly, the scroll offset can be propagated down as part of the environment. (Digression - in the design I'm currently contemplating, this will cause a flurry of update calls for all the descendants of the scroll container whenever the content is scrolled. I'm thinking maybe we have a way for widgets to indicate their interest in particular data sources)

I need a little convincing that we need "principal axis scrolling." In my experimentation, many of the problems with this on xi-mac come from poor scroll dynamics, where (for example) on a momentum flick the scrolling continues without being clamped by the app. Setting principal axis scrolling fixes this, by effectively quantizing the angle of the scroll, but I'm not sure it's the absolute best solution. That said, I just checked VS Code and Sublime Text on Windows, and it is the default scroll behavior there. I think I'd be happiest with it as an option but not the default (like bounce).

As discussed in this comment; the goal with keyboard event handling is to report the unmodified text value for a printable key, when a printable key is pressed.

was watching this video and was wondering why
you didn't mention amethyst as an option it uses an ECS too
and even though it's a work in progress as far as I know
it's closer to completion than this

ps: I'm aware this is not the place to say this but IDK how else to ask this
also this post might seem rude but I don't mean it to

That is, the red 'traffic light' button.

This is an invalid access, which suggests we're trying to send a message to an object that has been deallocated. My first hunch would be that the window is getting deallocated and is receiving a method from the view, or vice-versa. A first step to debug this would be to try the 'Zombies' instrument in Instruments.app.

This issue is to track a potential improvement for later. The current state of computing the interval passed to animations is "good enough".

Currently, the code for computing the animation interval is based on the elapsed wall clock time between calls to the window paint method. This should be pretty close to the reciprocal of the frame rate, but is potentially subject to jitter, especially if there are handlers in the event loop that take a nontrivial amount of time to run.

The comment on the Widget::anim method reads:

The interval argument is the time in nanoseconds between frames, for the purpose of computing animations. When framerate is steady, it should be exactly the reciprocal of the refresh rate of the monitor. If we are skipping frames, its cumulative sum should approximately track the passage of wall clock time and otherwise should be chosen to optimize for smoothness of animations.

The ideal heuristic for computing this interval is a deep topic, ideally the subject of a blog post when I figure it out.

(This comment will be blown away by the merge of the muggle branch, which is one reason I'm capturing it in an issue.)

My current thinking is that a good definition for the interval is the best guess of the presentation time for the next frame relative to the previous one. When this guess is accurate, the animation will be perceived as smooth. As stated in the above quote, when we're not skipping frames, it's not really necessary to "guess," it should be possible to get an accurate value from the graphics adapter.

What to do when we are skipping frames? One solution is to query presentation statistics (possibly GetFrameStatistics method on Windows or CVDisplayLink on mac) and do some kind of moving average of past frame-to-frame intervals.

It might be possible to estimate the likelihood of a dropped frame by looking at the wall-clock time and/or presentation statistics at the time the interval is computed. Figuring out a good heuristic will no doubt require some empirical measurement.

Another intriguing possibility to explore is to deliberately throttle the frame rate to make it consistent. For example, if it takes about 17ms to display a frame, it might better to present consistently at 30fps than to do it mostly at 60fps but with periodic frame drops. On Windows, this can be done fairly easily by setting SyncInterval to a value other than 1 in the IDXGISwapChain::Present method.

Likely more research is needed to figure out the absolute best way to do this, which again is a reason I'm filing an issue rather than just digging into it.

Applications should be able to schedule timers on the runloop that will call them back with a given Action or Event.

Druid should be able to open and manage multiple windows, and share state between them as appropriate.