As in any programming scenario, users often want to write a loop that just iterates over elements of a collection. To do this with the Remote Op abstraction, one can write this:

            const auto arraySize = array.Size();
            UiaUint elementIndex{ 0 };

            operationScope.For(
                []() {} /* initialize */,
                [&]() { return (elementIndex < arraySize); } /* condition */,
                [&]() { elementIndex += 1; } /* modification */,
                [&]() /* body */
                {
                    const auto element = array.GetAt(elementIndex);
                    /* Now after all this ceremony do something with element */
                });

We could add a helper to make this less verbose and far more readable:

operationScope.ForEach(array, [&](auto element)
{
    /* Do something with element! */
});

Under the hood, this could do the same thing as the verbose code to set up a for loop. (Or it could even generate some more efficient bytecode in the future...)

Currently, UiaOperationAbstraction.h requires header consumers to pre-#include headers such as winrt/Windows.Foundation.Collections.h to consuume the UIA operation abstraction library.

Address this inconvenience by self-containing the header for easier consumption.

To make development of more complex Remote Operations easier, the abstraction library could benefit from struct-like constructs.

For instance, we could add a tuple-like type that would be strongly typed (on the level of providing local and wrapper values -- just like in case of other UIA abstraction types) that could serve as a bridge between the current types and a struct with named fields/values.

Alternatively, we could work with UiaStringMap that contains UiaVariants but frankly that solution sounds slightly less convenient to me...

Because we cannot construct a UiaEnum from an AutomationRemoteAnyObject, UiaEnum currently cannot be included as part of a UiaTuple type.

It would probably be a good "quality of life" improvement to have a default value for this parameter (probably false -- as in don't use the cached value), as that would simplify code that is primarily executed in a Remote Operation, where this value is irrelevant.

The below code does not compile due to multiple operators matching right-hand side values:

auto operationScope = UiaOperationScope::StartNew();

// Give this operation a remote context.
UiaElement displayElement{ focusedElement };
operationScope.BindInput(displayElement);

// Ask for the control type of the imported element (which is represented as a `UiaEnum`).
auto controlType = displayElement.GetControlType(false /* useCachedApi */);

// Compare against different control type representations.
auto equalToAbstractionValue = (controlType == controlType); // Does not compile [1]
auto equalToComEnum = (controlType == UIA_TextControlTypeId); // Does not compile [2]
auto equalToWinRtEnum = (controlType == winrt::Microsoft::UI::UIAutomation::AutomationControlType::Text);

auto notEqualToAbstractionValue = (controlType != controlType); // Does not compile [1]
auto notEqualToComEnum = (controlType != UIA_TextControlTypeId); // Does not compile [2]
auto notEqualToWinRtEnum = (controlType != winrt::Microsoft::UI::UIAutomation::AutomationControlType::Text);

// Return the result.
operationScope.BindResult(equalToAbstractionValue, equalToComEnum, equalToWinRtEnum, notEqualToAbstractionValue, notEqualToComEnum, notEqualToWinRtEnum);
operationScope.Resolve();

// Ensure the correct results have been returned.
Assert::IsTrue(static_cast<bool>(equalToAbstractionValue));
Assert::IsTrue(static_cast<bool>(equalToComEnum));
Assert::IsTrue(static_cast<bool>(equalToWinRtEnum));
Assert::IsFalse(static_cast<bool>(notEqualToAbstractionValue));
Assert::IsFalse(static_cast<bool>(notEqualToComEnum));
Assert::IsFalse(static_cast<bool>(notEqualToWinRtEnum));

Looks like this is a conflict between conversion operators (to local and remote types) and comparison operators (defined for the types).

It sounds like we could reduce our comparison operators to 2 types:

1. UiaEnum,
2. T (for both COM and WinRT enum, since the compiler sees possible conversions from COM values [which are raw numbers] to WinRT enum values).

Plus, we could remove the comparisons against rvalue COM enums (that is already covered by other operators as far as I can tell).

Just cloned the project, acquired the latest sdk.. built the project with vs2019 Set the console demo to start and..

: 'Could not find Windows Runtime type 'Windows.UI.UIAutomation.Core.CoreAutomationRemoteOperation'.'

Any suggestions?

UiaInt's copy constructor works differently from its assignment operator, in that it simply creates an "alias" for the same remote object, rather than creating a new "deep" copy. Not only should the copy ctor always work the same way as the assignment operator, but this is definitely unexpected behavior for consumers.

While a workaround exists, we should still fix this.

Currently, it's not possible to build UiaOperationAbstraction with the /permissive- flag.

It seems that most of the errors are related to "two-phase name lookup".

We should fix these issues and then enable /permissive- for this project.

The Builder's AutomationRemoteOperationResultSet wraps the platform Result object and exposes the operation results through its own set of methods. We should also let callers obtain the underlying platform object, which has some additional rich information that can e.g. help debugging.

Somewhat related to #45 in that it's about exposing additional information that the Builder holds internally to callers to ease their debugging scenarios.

As the title mentions, the two types cannot be made elements of other collections. This is because they do not support necessary conversions and construction methods such as:

1. Construction out of the local type (such as std::shared_ptr<std::vector<T>> in case of UiaArray),
2. Conversion to that local type,
3. Asking about and casting to the remote types that backs the collection types.

This means that types like UiaStringMap<UiaArray<UiaElement>> (that -- for instance -- could list children of parents) cannot be compiled today.

This issue tracks fixing the compilation (and any other) problems with the collections to enable scenarios around more "structural" type nesting.

It has been suggested that a potentially good quality of life improvement for UiaOperationAbstraction would be to add a version of BindInput that accepts multiple arguments at once, so that binding multiple inputs could be done in a single line.

Something along the lines of...:

template<class... Args>
void BindInput(Args&... args)
{
    (BindInputSingle(args), ...);
}

// Where BindInputSingle is what BindInput is today

From Adam Hodson:

If I put together a remote operation using the winRT builder api, is there a way to get back the generated byte code for the series of calls strung together? The presented slide on diagnostics functionality assumes you're using byte code for the error location so wouldn't be that helpful if you used the builder api?

Yes, exposing the actual bytecode generated by the builder out to the client would certainly be useful. We should add this.

We could perhaps have both a "current bytecode" exposed off of the AutomationRemoteOperation itself (the bytecode that has been built up thus far), as well as an "executed bytecode" property on the ResultSet (the bytecode that was ultimately sent for execution).

IUIAutomationElement exposes an API that allows retrieving metadata information about element properties:

https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nf-uiautomationclient-iuiautomationelement7-getcurrentmetadatavalue

This issue tracks adding dedicated GetMetadataValue helpers in the WinRT Builder and the C++ Abstraction library.

MalformedBytecode can occur when:

• The size of the bytecode is above the maximum size
• The bytecode version isn't supported by the provider-side UIA
• An unsupported opcode is used
• An opcode is used improperly (e.g. it's at the end of the bytecode stream even though it should take arguments)
• Generally any part of the initial remote operation state (e.g. bytecode, imports, requested results, etc.) fails to be read on the remote side

Currently UiaOperationAbstraction throws E_FAIL from Resolve when MalformedBytecode is encountered. It might be good to throw a more informative error, or provide a mechanism to detect when the error is due to MalformedBytecode.

@zhayuli

When writing remote operation loop, consumers will frequently end up with a line such as this in the body of the loop:

                operationScope.If(ancestorElement.IsNull(), [&]()
                {
                    operationScope.Break();
                });

It could be helpful to add a helper that lets consumers write:

                operationScope.BreakIf(ancestorElement.IsNull());

This would be a little bit less verbose.

The same thing could be said for Continue...

We would like to be able to automatically build/validate the project as part of an Azure Pipelines build.

Since we're currently targeting an Insider Preview SDK version, this might be tricky to set up.

At the very least, we should add this once the project targets a stable version of the SDK.

After cloning the repository, installing missing NuGet packages and executing: msbuild UIAutomation.sln /p:Configuration=Release,Platform=x86 I got:

Microsoft (R) Build Engine 16.11.2+f32259642 dla platformy .NET Framework
Copyright (C) Microsoft Corporation. Wszelkie prawa zastrzeżone.

Projekty w tym rozwiązaniu są kompilowane pojedynczo. Aby umożliwić kompilację r
ównoległą, dodaj przełącznik "-m".
Kompilacja rozpoczęła się 2022-02-16 20:52:06.
Projekt "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\UIAutomatio
n.sln" w węźle 1 (domyślne elementy docelowe).
ValidateSolutionConfiguration:
  Tworzenie konfiguracji rozwiązania "Release|x86".
Projekt "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\UIAutomatio
n.sln" (1) kompiluje "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomatio
n\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj" (2) w węźle 1 (d
omyślne elementy docelowe).
InitializeBuildStatus:
  Modyfikowanie "Release\Microsof.7D645239.tlog\unsuccessfulbuild".
Midl:
  C:\Program Files (x86)\Windows Kits\10\bin\10.0.22000.0\x64\midl.exe /metadat
  a_dir "C:\Program Files (x86)\Windows Kits\10\References\10.0.22000.0\Windows
  .Foundation.FoundationContract\4.0.0.0" /winrt /W1 /nologo /char signed /env
  win32 /winmd "Release\Microsoft.UI.UIAutomation.winmd" /h "Microsoft.UI.UIAut
  omation_h.h" /tlb "Release\Microsoft.UI.UIAutomation.tlb" /target "NT60" /nom
  idl /struct_by_ref Microsoft.UI.UIAutomation.idl
  MIDLRT Processing .\Microsoft.UI.UIAutomation.idl
  Microsoft.UI.UIAutomation.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\Windows.Foundation.idl
  Windows.Foundation.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\inspectable.idl
  inspectable.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \shared\wtypes.idl
  wtypes.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \shared\wtypesbase.idl
  wtypesbase.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \shared\basetsd.h
  basetsd.h
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \shared\guiddef.h
  guiddef.h
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \um\unknwn.idl
  unknwn.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\hstring.idl
  hstring.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\AsyncInfo.idl
  AsyncInfo.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\EventToken.idl
  EventToken.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\windowscontracts.idl
  windowscontracts.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\IVectorChangedEventArgs.idl
  IVectorChangedEventArgs.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \um\oaidl.idl
  oaidl.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \um\objidl.idl
  objidl.idl
  MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
  \winrt\Windows.UI.UIAutomation.idl
  Windows.UI.UIAutomation.idl
.\Microsoft.UI.UIAutomation.idl(9): error MIDL2025: syntax error : expecting .
near "runtimeclass" [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation
\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(9): error MIDL2009: undefined symbol : unsealed
.AutomationRemoteObject [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutoma
tion\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(15): error MIDL2025: syntax error : expecting t
he keyword "interface" near "void" [c:\Users\Lukasz\Microsoft-UI-UIAutomation\s
rc\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(15): error MIDL2026: cannot recover from earlie
r syntax errors; aborting compilation [c:\Users\Lukasz\Microsoft-UI-UIAutomatio
n\src\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
Kompilowanie projektu "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomati
on\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj" wykonane (domyś
lne elementy docelowe) - NIEPOWODZENIE.

Kompilowanie projektu "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomati
on\UIAutomation.sln" wykonane (domyślne elementy docelowe) - NIEPOWODZENIE.


Kompilacja NIE POWIODŁA SIĘ.

"c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\UIAutomation.sln" (
domyślny element docelowy) (1)->
"c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\Microsoft.UI.UIAuto
mation\Microsoft.UI.UIAutomation.vcxproj" (domyślny element docelowy) (2)->
(element docelowy Midl) ->
  .\Microsoft.UI.UIAutomation.idl(9): error MIDL2025: syntax error : expecting
. near "runtimeclass" [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomati
on\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
  .\Microsoft.UI.UIAutomation.idl(9): error MIDL2009: undefined symbol : unseal
ed.AutomationRemoteObject [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAuto
mation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
  .\Microsoft.UI.UIAutomation.idl(15): error MIDL2025: syntax error : expecting
 the keyword "interface" near "void" [c:\Users\Lukasz\Microsoft-UI-UIAutomation
\src\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
  .\Microsoft.UI.UIAutomation.idl(15): error MIDL2026: cannot recover from earl
ier syntax errors; aborting compilation [c:\Users\Lukasz\Microsoft-UI-UIAutomat
ion\src\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxpro
j]

    Ostrzeżenia: 0
    Liczba błędów: 4

Czas, który upłynął: 00:00:02.18

I do not really understand why the version of Windows in use affects the result of the midl compiler. If this is impossible to fix or it is not worth fixing given Windows 7 age I would appreciate a clear note in the readme as to what versions of Windows are supported when building.

In the following snippet...

void Fn(UiaElement original)
{
  UiaElement newElement = original;
}

The newElement will end up aliasing the same remote object. More precisely, it'll refer to the same operand register as original (on the lowest-level of Remote Ops bytecode).

Meanwhile, in the following...

void Fn(UiaElement original)
{
  UiaElement newElement{ nullptr };
  newElement = original;
}

A Set instruction is emitted in the bytecode that assigns the original element to a new operand register in the bytecode.

This means that two things that typically should work the same (the copy ctor and assignment operator) end up with subtly different behavior.

This issue tracks making a decision about this -- whether to try to make any changes to make these two operations work the same way or whether to document this rough edge as expected/by design.

UIA provides IUIAutomationElement::BuildUpdatedCache to request a new UIA element with fresh, current property values and pattern objects. Currently, there is no equivalent of that in the Remote Operation abstraction library.

This issue tracks adding appropriate support for it.

This can be done by creating a custom provider that can be loaded by the ModernApp test utility. That method involves registering appxes on the user's system, though, so we could also go the route of implementing a Win32 app to act as the provider.

Original post at Q&A: https://docs.microsoft.com/en-us/answers/questions/929263/uiautomation-treewalker-misbehaving-in-microsoft-o.html

After getting a child element inside Picture Manager, using the TreeWalker to walk the UIA tree is behaving erratically.

I am running Windows 10 Pro 10.0.19044, and Microsoft Office 2010 Picture Manager.

How to reproduce (using inspect.exe tool in UI Automation mode):

1. Open up an image in Picture Manager
2. Inspect the image element: Inspect tools' TreeWalker "Navigate to Previous sibling" and "Navigate to Next sibling" buttons are unavailable
3. Inspect the toolbar under the image (where the Next and Previous buttons are): Inspect tools' next and previous sibling buttons are unavailable
4. Inspect the image element, click on TreeWalker "Parent" button, then click twice on either the "Navigate to First Child" button or the "Navigate to Last Child" button. Now the sibling buttons are available, and clicking through them gets both the image element and the toolbar element.
   Note: using the inspect.exe tool is only a demonstration, the same issue exists using the framework directly.

Expected behaviour:

1. TreeWalking to previous and next sibling should be available right away.
2. After getting the parent element, getting the original element shouldn't take two steps.

I tried to create a reproducible sample with C#, but the code freezes altogether. Open up a new Notepad window, and Microsoft Office 2010 Picture Manager window, and open a random image. Using System, System.Windows.Automation, System.Runtime.InteropServices:

 AutomationElement rootElement = AutomationElement.RootElement;
 AutomationElement npEl = rootElement.FindFirst(TreeScope.Children, new PropertyCondition(AutomationElement.NameProperty, "Untitled - Notepad"));
 npEl.SetFocus();
 Console.WriteLine("Found and activated Notepad");
 AutomationElement documentEl = npEl.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Document));
 Console.WriteLine("Document element name: " + documentEl.Current.Name);
    
 AutomationElement pmEl = rootElement.FindFirst(TreeScope.Children, new PropertyCondition(AutomationElement.NameProperty, "Microsoft Office 2010"));
 pmEl.SetFocus();
 Console.WriteLine("Found and activated window");
 AutomationElement imageEl = pmEl.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Image));
 Console.WriteLine("Image element name: " + imageEl.Current.Name);
 Console.ReadLine();

This stops indefinitely at line 11.

TextPattern2 GetCaretRange throw exception by receiving the result E_NOTIMPL from the porvider.

I expect to return null text range.

There is no explicit reason to avoid UiaTuple make move assignments.
But UiaTuple cannot just get the inherent function from UiaTypeBase, so if we want to make a move assignment for UiaTuple, we may need to define the move assignment operator and move constructor for UiaTuple explicitly.

There are a lot of problems with custom patterns in c# (WPF).

1. IUIAutomationRegistrar.RegisterPattern() gets imported incorrectly.
   The c++ signature is

 HRESULT RegisterPattern(
   [in]  const UIAutomationPatternInfo *pattern,
   [out] PATTERNID                     *pPatternId,
   [out] PROPERTYID                    *pPatternAvailablePropertyId,
   [in]  UINT                          propertyIdCount,
   [out] PROPERTYID                    *pPropertyIds,
   [in]  UINT                          eventIdCount,
   [out] EVENTID                       *pEventIds
 );

but when imported into c#, the PROPERTYID arrays are wrong:

void RegisterPattern([In] ref UIAutomationPatternInfo pattern,
    out int pPatternId, out int pPatternAvailablePropertyId,
    [In] uint propertyIdCount, out int pPropertyIds,
    [In] uint eventIdCount, out int pEventIds);

See https://docs.microsoft.com/en-us/answers/questions/889043/how-can-i-34fix34-a-mis-imported-com-interface-cwp.html

2. I got around (1) by doing the RegisterPattern() in a c++ DLL that I included in my project, but even when I call GetCurrentPattern() on the registered pattern ID (from my c# automated UI test harness), it returns a null, and I've got a bunch of logging in the c# application being tested, and as far as I can tell the call to GetCurrentPattern() failed inside the test harness (no communication made to the application being tested)

3. Even if I can get GetCurrentPattern() to work in the test harness, the GetPattern() override in the application being tested has a single parameter which is an enum PatternInterface so it's not really able to pass my custom pattern ID.

Should I create these as separate issues? Or is UIAutomation/c# not a supported combination?

This specific code snippet:

UiaElement element{ ... };
auto parent = element.GetParentElement(cacheRequest);
operationScope.BindResult(parent);
operationScope.Resolve();

should return a parent with a cached requested data or no element at all. However, today, it fails if the parent could not be obtained due to not existing or existing across process/connection boundary. The failures come from assuming that the parent is always found and therefore that caching happens on a non-null element.

This is unintended as the caller cannot control whether the parent exists or not and cannot prevent the caching from happening. We should fix that to continue executing the operation under the assumption that if the parent has not been found, no attempts to cache its data should be made (and the operation should continue on under the assumption that parent may or may not have been found -- just like it is for non-caching calls to the same interface method).

Types in UiaOperationAbstraction can often be converted to their local and remote variants implicitly. For example, passing a UiaString to a function that takes a wil::shared_bstr will perform the conversion automatically due to the UiaString method operator wil::shared_bstr() const;. These conversions throw if you try to convert a remote abstraction instance into its local variant, or a local abstraction instance into its remote variant. This can result in accidental and often confusing bugs.

It's worth having discussion on whether we want to move towards more explicit conversions to avoid these subtle bugs. This would be a breaking change for UiaOperationAbstraction consumers.

The UIA platform supports indexing remote strings, but the UiaOperationAbstraction helper library doesn't currently expose this.

UiaAbstracion wrappers need to be implemented for the new Extensibility features. This includes CallExtension, IsExtensionTarget, IsExtensionTarget, IsExtensionSupported, UiaEmpty, UiaByteArray, new UiaConnectionBoundObject to be implemented which has their corresponding counter winrt wrappers already implemented.

We should add support for arithmetic operators to wrapper library for types at lease like UiaInt, UiaUint and UiaDouble. Right now, we cannot do 'a + b' when both 'a' and 'b' are types like UiaInt.

This change aims to avoid common situations where UiaRect or UiaPoint need to be created before they can be given (often conditionally) values to store.

In such cases, the two types should be initialized to an empty rectangle and a (0, 0) point respectively. However, right now, that has to be done explicitly by providing RECT and POINT of the values.

We can improve it by creating default constructors for the two and using the "empty" values there.

The caller doesn't get their own copy, but rather a direct pointer to the internal data. This makes it easy to mis-use the returned pointer, such as assuming ownership or modifying the contents.

The below helper function implemented in the UiaOperationAbstraction.cpp file aims to convert a win32 RECT type to a Windows::Foundation::Rect, the returned Windows::Foundation::Rect is expected to represent same rectangle area of the RECT passed into this helper function.

winrt::Windows::Foundation::Rect ConvertRect(RECT rect)

But the order of parameters passed into constructor of Windows::Foundation::Rect is wrong, the order are supposed to be (left, top, height, width), but the parameters passed in are (height, width, left, top). This results in the returned rectangle with wrong position, e.g. the top left position is always at (0, 0).

Adjusting the order of input parameters passed into the constructor should be able to fix this issue.

The UIA platform supports accessing coordinates and dimensions of rectangles, but the UiaOperationAbstraction helper library doesn't currently expose this.

It could be useful for various clients to have an async version of Execute that doesn't block the calling thread.

One way to do it in this undocked helper libraries is to dispatch the work to a thread pool thread (co_await winrt::resume_background();) and do the work there.

Depending on feedback/input, the UIA platform might consider adding a "native" way to do async calls (without burning a threadpool thread) -- if it turns out that doing this in the wrapper only is insufficient for some user scenarios.

I am working with a UI tree element from Infragistics and I've encountered that the nodes within this tree have the IsOffscreen property set to true, which must be incorrect, since the nodes are clearly visible on screen.

The attached screenshot should illustrate the situation quite well:

I've been through Microsoft's API description of the IsOffscreen-property and I checked against all what it says, concerning the scrolling viewport visibility, the partial visibility, the collapsed- and expanded visibility, etc. and none of those things apply to my UI tree.

And since I keep stumbling across such issues within the UIAutomation framework from time to time, I'd guess it's simply a bug within the framework.

Best regards,
Markus

Currently, there is no corresponding function in the wrapper for IUIAutomationElement::GetCurrentMetadataValue, which means it is not convenient to get metadata through a remote operation. It would be nice to have something like:
UiaVariant UiaElement::GetMetadataValue(UiaPropertyId propertyId, UiaMetadata metadataId)

If takes one more step, to properly use this newly added GetMetadataValue, we may need to cast the returned UiaVariant to some other UiaEnum types. For example: UiaSayAsInterpretAs sayAsInterpretAs = variant.AsType<UiaSayAsInterpretAs>()
However, the current UiaEnum implementation does not have c_anyCast, which means we cannot make such a cast. So, it would be nice to have this casting issue resolved.

Currently, the solution only has x86 and x64 targets. We should configure ARM targets, as well.

Since we use exceptions for breaking/continuing in loops in the local case, we need to make sure TryCatch catches and rethrows those exceptions.

We also need to rethrow the success exception from UiaOperationDelegator::AbortOperationWithHresult.

The Builder API has a comprehensive test suite implemented in closed-source code. There's no reason these tests couldn't be open-sourced and included in the (now-growing) functional test suite in this repo.

I'm a dev on the MSVC compiler team and discovered a problem with the implementation of UiaBool while attempting to fix an overload resolution bug in the compiler.

Here's a reduced sample based on a subset of UiaBool that illustrates the problem: https://godbolt.org/z/57nMx5

For the test of a UiaBool with an implicit conversion to bool MSVC currently (but incorrectly) calls UiaBool::operator bool(). All other compilers, and MSVC with my fix, instead call UiaTypeBase::operator LocalT*, which will always yield 'true' as it's returning the non-null address of the variant member.

The main problem here is that the operator bool() overload will be a worse conversion than the base LocalT* conversion if the UiaBool object is not const (it involves a qualification conversion for the implicit object parameter, while the LocalT* conversion does not). This is because operator bool() is declared const, but operator LocalT*() is not.

There is a fundamental ambiguity here where there are implicit conversion sequences to both bool and BOOL*. One potential fix for this would be to make UiaBool::operator BOOL*() explicit.

To simplify code built with the UIA operation abstraction library, we should consider simplifying construction of the types provided by the library.

Specifically, types whose default values are implied or types that in other languages are often default-constructible could be good candidates, such as UiaString that right now has to be constructed as UiaString{ L"" } while the C++ equivalent of std::wstring can be constructed without providing arguments.

Some source files in the Remote Operation builder and abstraction libraries were auto-generated using a tool.

There's no reason the tool itself shouldn't be part of the project.

UIA returns AutomationRemoteOperationStatus::InstructionLimitExceeded when the number of instructions executed in the remote process exceeds 10000. However, currently the wrapper just transforms this into E_FAIL in AutomationRemoteOperationResultSet::OperationStatus. In UiaOperationScope::Resolve, this E_FAIL is plumbed through to the client.

We should provide a way to determine whether the remote operation exited with InstructionLimitExceeded. I'm listing three potential ways here, although more are certainly possible:

• Create a custom HRESULT using the customer bit and return this instead of E_FAIL.
• Create a custom exception which UiaOperationScope::Resolve throws on InstructionLimitExceeded.
• Add support to AutomationRemoteOperationResultSet to return whether InstructionLimitExceeded occurred, and provide an alternative to UiaOperationScope::Resolve which returns an AutomationRemoteOperationResultSet.

Newer versions of UIA allow UIA clients to Stringify values of some types.

This issue tracks adding support for the instruction to the bytecode and to the UIA operation abstraction library.

I don't think this would be possible in the wrappers. We'd need a new instruction, or potentially a modification to NewString which accepts a character. I think the former is the way to go, since modifying existing instructions doesn't really fit into our versioning model.

There's definitely also an argument that this isn't necessary, since someone could write their bytecode to deal in single-character strings, rather than actual character types. It's worth some discussion.

Thanks @mavangur for bringing this up.

UiaOperationScope has a useful utility method: ForEach which executes the given lambda on each element in the given array, whether running locally or remotely.

However, this method fetches each element from the array and passes it to the visitor function as const, which means it is impossible to modify the element from within the visitor function if the visitor function accepts the element by reference.

And because none of the Uia types have methods marked as const, it is therefore impossible to do much at all in the visitor function. For example,

scope.ForEach(myArray, [](auto& element){
 auto element.GetName();
});

Results in an error like the following:

build\x86\UIARemote\UIARemote.cpp(367): error C2662: 'UiaOperationAbstraction::UiaString UiaOperationAbstraction::UiaElement::GetName(bool)': cannot convert 'this' pointer from 'const ItemWrapperType' to 'UiaOperationAbstraction::UiaElement &'
        with
        [
            ItemWrapperType=UiaOperationAbstraction::UiaElement
        ]
build\x86\UIARemote\UIARemote.cpp(367): note: Conversion loses qualifiers
build\x86\microsoft-ui-uiautomation\include\UiaOperationAbstraction\UiaTypeAbstraction.g.h(2065): note: see declaration of 'UiaOperationAbstraction::UiaElement::GetName'
build\x86\microsoft-ui-uiautomation\include\UiaOperationAbstraction/UiaOperationAbstraction.h(2379): note: see reference to function template instantiation 'auto _remoteable_getTextContent::<lambda_c4afe82d3e528f79f41517a2c2e0fcb4>::operator ()<const ItemWrapperType>(const ItemWrapperType &) const' being compiled
        with
        [
            ItemWrapperType=UiaOperationAbstraction::UiaElement
        ]

A workaround is to have the lambda accept the element by value rather than reference:

scope.ForEach(myArray, [](auto element){
 auto element.GetName();
});

As I believe that copying the Uia* types is remotely cheap and ends up using the same underlying object anyway.

But personally I do prefer to code my functions taking Uia* types by reference as it just feels more symantically correct, and removes any guessing about whether copying is in deed cheep or not.

I'm also really not sure why scope.forEach deliberately fetches the element from the array as a const variable? I don't think the concept of const really makes any sense for these types, especially when none of the methods on them are marked as const.

Removing const here would be a one line change. I can open a pr to do this, but I first wanted to better understand why it was made const in the first place.