Hi,

I´m trying to get audio capture working on a ReSpeaker Core V2 (http://wiki.seeedstudio.com/ReSpeaker_Core_v2.0/) on Linux. But when I start my sample on linux I get an Error "Read failed"... at first I thought "plughw:0,0" would be wrong, but using anything other than that device name results in error that (more or less) clearly reads like "unknown device". So I think the problem is somewhere else..

Perhaps this will help (result from arecord -l):
**** List of CAPTURE Hardware Devices ****
card 0: seeed8micvoicec [seeed-8mic-voicecard], device 0: 100b0000.i2s1-ac108-pcm0 ac108-pcm0-0 []
Subdevices: 1/1
Subdevice #0: subdevice #0

When I use "hw0,0" as device name I get the following error:
Unhandled exception. System.ArgumentException: Hardware params set channels failed.
at Microsoft.Psi.Audio.LinuxAudioInterop.Open(String name, Mode mode, Int32 rate, Int32 channels, Format format, Access access)
at Microsoft.Psi.Audio.AudioCapture.Start(Action`1 notifyCompletionTime)
at Microsoft.Psi.Executive.PipelineElement.b__44_0()
at Microsoft.Psi.Scheduling.Scheduler.ExecuteAndRelease(SynchronizationLock synchronizationObject, Action action, SchedulerContext context)
at Microsoft.Psi.Scheduling.Scheduler.Run(Object workItem)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Thanks for your help

Carl

The issue is that there is no guarantee that final callback will be handled in upstream to downstream order, thus it’s possible for such components chained together to finalize and then see upstream messages generated. This can cause unexpected behavior as a pipeline is shutting down. More perniciously, the Parallel operator uses dynamic subpipelines internally which causes this (sub)pipeline shutdown behavior to potentially occur throughout the life of a running application.

Components registering to be called with Pipeline.RegisterPipelineFinalHandler(…) expect to be called only once no further upstream messages are forthcoming. Source components are stopped, scheduled messages throughout the system are drained, and then the final callbacks are invoked. However, components are free to post further messages as a result of these callbacks. For example,Join and the RelativeTimeWindow (exposed by the Window(…) operator) do this. Chaining these components together may cause issues due to the finalization ordering between them. Normally this issue manifests at pipeline shutdown, but Parallel and its use of subpipelines may exacerbate this into a steady state issue.

(This issue is currently being investigated)

In the build instructions, you specify adding ".NET desktop development workload, as well as the .NET Framework 4.7 targeting pack" however, there is no mention of adding .NET Core (available here) which was required for me to successfully build. I believe checking that the ".NET Core cross-platform development" option in the installer would also be sufficient, but I did not test that.

Here is an example to illustrate my problem:

            using (var pipeline = Pipeline.Create())
            {
                var sameTime = DateTime.Now;
                var fakeList = new List<(int, DateTime)>();
                fakeList.Add((0, sameTime));
                fakeList.Add((1, sameTime));

                // Create a generator
                var gen = Generators.Sequence(pipeline, fakeList);
                gen.Do(x => Console.WriteLine("Hello World"));

                pipeline.Run();
                Console.WriteLine("Done!");
                Console.ReadLine();
            }

The expected output would be output of "Hello World" twice before the pipeline shuts down. However, if you run this code, it will output "Hello World" once and nothing else. The program will keep running without any outputs forever.

The problem is that the two elements have the same originating time and the generator's emitter has thrown a Attempted to post a message without strictly increasing originating time or that is out of order in wall-clock time exception and the pipeline has start shutting down but fail to shutdown the generator completely.

After diving deeper into the code of Generator{T}.cs and Generators.cs, I hypothesize this is the sequence of errors that causes the pipeline to hang.

1. Generator{T}.cs line 68 throws the InvalidOperationException because the two elements have the same originating time.
2. Generator.cs line 108 never returns and the this.nextMessageTime is never updated.
3. The exception is caught by the pipeline and start shutting all the components down with the final originating time to be AFTER the last good message in the generator.
4. The pipeline's DeactivateComponents method calls the Stop method of the generator.cs (line 76).
5. this.nextMessageTime will be the old time and is before the this.finalMessageTime and won't completely stop the component, thinking there are more messages from it.
6. The pipeline thinks the generator is still catching up and generating old messages and wait for it.
7. It waits forever......

I can think of some fixes like catching the error on this level or additional checks in the Stop method. I was wondering whether y'all had faced similar problems and if this bug is already fixed in an internal release.

My actual use case is when converting ROS Messages to Psi Store, sometimes the messages were published so fast they have the exact same publish time. We used a generator to output all the messages and they stop working suddenly.

Edit:
I just read that this issue was briefly mentioned in the wiki where generators should have strictly increasing time. I think it will be helpful if this exception is surfaced to the user, especially if they using the nuget version where they cannot trace the pipeline and component state.

In Debug mode, PsiStudio will sometimes throw the following exception:

IOException: Cannot locate resource 'themes/images/resize_grip.png'.

It is safe to ignore the exception (click continue and PsiStudio still works.) but it will be nice if its handled, or the reference to image taken away.

I would like to request a new feature: a config file and primary entry point / Main driver that reads the config file. The goal of the config file is to have a single file (or heirarchy of files) that defines the configuration of a particular multimodal system setup. I think a config file will be useful for being able to run different experiments without needing to have different versions of the code. I also think it will increase wider-spread adoption of psi for people to make use of existing components. Perhaps most importantly, the config file would allow different configurations of psi without requiring Visual Studio. I imagine, for example, a robotics researcher who wants to bring together ASR & NLU (which psi has/soon will have) for their robot, but doesn't want to get into the code--they just want the NLU output to be passed to their robot component.

This might require more discussion and brainstorming, but here's a starting point:

Ideas for config file contents

• something that sets up the pipeline (with params!) / setup / takedown
• input buffers, output buffers
• what gets put into the store, when
• something that can start external processes (e.g., a python process + interop)
• parameter for filename (i.e., store file) to autoload in the studio for visualization

Ideas for psi main driver classs

• parameter pointing to config file
• parameters for overriding config file parameters
• Main driver uses the config file to setup and instantiate the psi components

References

• InproTK's config system uses the Sphinx4 config system, which is XML-based (I don't recommend using XML!) that has component definitions and allows components to be instantiated and shared across the entire system. Components have provisions for being able to set prameters from the config files
• The RASA pipeline is really nice and I think worth emulating as the format is easy to read

I think these are somewhat related but not sure if its something unique to my machine's setup. I'm trying to record and live preview a Kinect 2 Stream using the following code:

            using (var pipeline = Pipeline.Create(true))
            {
                var store = Store.Create(pipeline, "testStore", @"C:\Data\Stores");
                var kinectSensor = new KinectSensor(pipeline, new KinectSensorConfiguration() { OutputBodies = true, OutputColor = true });

                var numGen = Generators.Range(pipeline, 0, 100, TimeSpan.FromSeconds(1));
                numGen.Write("test", store, largeMessages: true);
                numGen.Do(m => Console.WriteLine(m));

                kinectSensor.Bodies.Write("Body", store, largeMessages:true);
                kinectSensor.ColorImage.Write("Image", store, largeMessages:true);

                pipeline.Run();
            }

The first thing I notice is that PsiStore is being written in 32kb chunks instead of the large message formats (262MBs I think?)

If I close the application and then open the store in PsiStudio, it takes a while to repair the store, but I can still visualize it.

However, when I try to visualize it LIVE when it is running, I get the following exception from the store generating application.

System.AggregateException
  HResult=0x80131500
  Message=Pipeline 'default' was terminated because of one or more unexpected errors
  Source=Microsoft.Psi
  StackTrace:
   at Microsoft.Psi.Pipeline.ThrowIfError() in C:\Users\xiangzht\source\repos\psi\Sources\Runtime\Microsoft.Psi\Executive\Pipeline.cs:line 1265
   at Microsoft.Psi.Pipeline.Run(ReplayDescriptor descriptor) in C:\Users\xiangzht\source\repos\psi\Sources\Runtime\Microsoft.Psi\Executive\Pipeline.cs:line 472
   at Microsoft.Psi.Pipeline.Run(TimeInterval replayInterval, Boolean useOriginatingTime, Boolean enforceReplayClock, Single replaySpeedFactor) in C:\Users\xiangzht\source\repos\psi\Sources\Runtime\Microsoft.Psi\Executive\Pipeline.cs:line 484
   at SimpleTest.Program.Main(String[] args) in C:\Users\xiangzht\source\repos\psi\Playground\SimpleTest\Program.cs:line 29

Inner Exception 1:
IOException: Not enough memory resources are available to process this command.

The memory also went from about 200MB all the way to 1GB before crashing. PsiStudio also showed this message:

I feel like this is a known issue and I seen it before? 😜 Let me know how I could help with debugging this issue.

I am following the steps mentioned here to try out live visualization. Running the application produces the following unhandled exception :

System.BadImageFormatException
HResult=0x8007000B
Message=Could not load file or assembly 'Microsoft.Psi.Visualization.Client.Windows, Version=0.3.16.5, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.
Source=
StackTrace:

Note that I have run PsiStudio once before this. I am using Microsoft.Psi.Visualization.Windows 0.3.16.5-beta from nuget and targeting .NET 4.7.

In the tutorial at https://microsoft.github.io/psi/tutorials/, the sixth code box that creates an audio pipeline for voice activity detection, won't compile. The tutorial should direct the user to add the Microsoft.Psi.Audio.Windows NuGet package.
However, this package is not compatible with a .NET Core Console Application (which is what the tutorial directs the reader to create at the beginning); it requires .NET 4.7.
So, to make this example work, the user has to create a new Console App with .NET 4.7, install the Microsoft.Psi.Runtime, Microsoft.Psi.Audio.Windows, and Microsoft.Psi.Speech.Windows NuGet packages. Then they have to add using Microsoft.Psi.Audio and using Microsoft.Psi.Speech to their C# code file, and then try out the code.

It might be worth explaining more about how the code picks a microphone to listen to and whether that detector should output anything at all while it runs if it doesn't hear anything. (maybe just test instructions would help).

Hi,

The latest release mentioned the addition of 3rd party visualizer support in PsiStudio. The release note included a small paragraph on how to add them. I was wondering if there is any documentation, guides or examples on how to create and use them? Any pointers on where to start will be helpful.

Hello!

I'm using the Mpeg4Writer to write videos captured from a webcam. Playing back the written video in multiple media players indicates that the duration information in the encoded file is incorrect and seeking capability is missing.

System and recording details
OS : Windows 10 v 1709 Build 16299.431
Camera : Tested on both Logitech Brio and Logitech C920 (recording at 1920x1080, 30fps)
Psi packages version : v0.3.16.5-beta through nuget

Steps to reproduce

1. Start with the code in the WebcamWithAudioSample
2. Add the following after line 70 in Program.cs that creates the webcam:

var writerConfiguration = Mpeg4WriterConfiguration.Default;
writerConfiguration.ContainsAudio = false;
Mpeg4Writer vidWriter = new Mpeg4Writer(pipeline, Path.Combine(pathToStore, "video.mp4"), writerConfiguration);
webcam.Out.PipeTo(vidWriter.ImageIn, DeliveryPolicy.Unlimited);

Playback results
I've tried playing back the resulting .mp4 (real duration 5sec) in Movies&TV, Windows Media Player, and VLC. Here are the screenshots of each:

Movies&TV:

Windows Media Player:

VLC:

I am running Ubuntu 16.04.3 on a Dell XPS 13 2015

I am trying to use the AudioCapture component and I am getting the following error.

Unhandled Exception: System.DllNotFoundException: Unable to load shared library 'asound' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libasound: cannot open shared object file: No such file or directory
   at Microsoft.Psi.Audio.LinuxAudioInterop.Open(Void** handle, String name, Int32 capture, Int32 mode)
   at Microsoft.Psi.Audio.LinuxAudioInterop.Open(String name, Mode mode, Int32 rate, Int32 channels, Format format, Access access)
   at Microsoft.Psi.Audio.AudioCapture.OnPipelineStart()
   at Microsoft.Psi.Scheduling.Scheduler.ExecuteAndRelease(SynchronizationLock synchronizationObject, Action action)
   at Microsoft.Psi.Scheduling.Scheduler.Run(Object workItem)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

I made sure that ALSA was installed and up to date. How do I get this library?

I'm trying to run the simple app in Linux. I followed the build instructions but one thing that I was unsure about is whether I need nugget or not. The README page gives the impression that it is optional, "You can build applications based on Platform for Situated Intelligence either by leveraging nuget packages, or by cloning and building the code. Below are instructions:"

Can you confirm whether nuget is required or not?

Also, if nugget is required, which packages do I need from nuget.org

Hi, I am just getting started with \psi and going over the 'BriefIntroduction' Tutorial.

On step 5 it says: 'Kinect for Windows SDK is required to run PsiStudio'

Does it actually require kinect device or only SDK? Would it work with a regular computer camera/microphon?

thank you
-Svetlana

I am following the Offline Visualization tutorial and trying to run PsiStudio.

When I build and run the PsiStudio project, I get the following error :

Details :

System.TypeInitializationException
HResult=0x80131534
Message=The type initializer for 'Microsoft.Psi.PsiStudio.PsiStudioContext' threw an exception.
Source=Microsoft.Psi.PsiStudio
StackTrace:
at Microsoft.Psi.PsiStudio.PsiStudioContext.get_Instance() in C:\src\Microsoft\psi\Sources\Tools\PsiStudio\Microsoft.Psi.PsiStudio\PsiStudioContext.cs:line 71
at Microsoft.Psi.PsiStudio.MainWindow..ctor() in C:\src\Microsoft\psi\Sources\Tools\PsiStudio\Microsoft.Psi.PsiStudio\MainWindow.xaml.cs:line 24

Inner Exception 1:
BadImageFormatException: Could not load file or assembly 'Microsoft.Kinect, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Reference assemblies should not be loaded for execution. They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)

Inner Exception 2:
BadImageFormatException: Cannot load a reference assembly for execution.

I have installed the required and optional pre-requisites mentioned here.

Looks like Kinect libraries are required to run PsiStudio. If so, what version ?

Using Ubuntu 18.04. I'm trying to build Sources/Runtime/Microsoft.Psi as part of one of the provided samples and am getting an error because my "ilasm" tool isn't generating the debug file when compiling the provided MemoryAccess.il library (it is generating the MemoryAcess.dll file, though). The code that compiles the library file can be found at the bottom of:
Sources/Runtime/Microsoft.Psi/Microsoft.Psi.csproj

Is there something obvious that I'm missing here?

Hi I had to figure this one out from the hard way os I thought it will be nice to share. In order to connect to a node other than master or the other than dev machine on psi/Ros, we need to add its IP to RosDns, otherwise it will try to connect master node with same port for RequestTopic connection. This is mainly due to resolve code in RosDns :

let resolve rosMaster (uri: string) = let host = if uri.StartsWith("http://") then hostRegex.Match(uri).Groups.[1].Value else uri match Map.tryFind host known with | Some found -> uri.Replace(host, found) | None -> uri.Replace(host, rosMaster) // default

I think it will be nice to document this somewhere in the tutorials. Otherwise it takes sometime to debug it, especially if you are not very familiar with F# like me.

When creating a PsiStore in Linux, the application fails and throws the following error:

Unhandled Exception: System.ObjectDisposedException: Cannot access a closed file.
   at System.IO.FileStream.Flush(Boolean flushToDisk)
   at System.IO.FileStream.get_SafeFileHandle()
   at System.IO.MemoryMappedFiles.MemoryMappedView.CreateView(SafeMemoryMappedFileHandle memMappedFileHandle, MemoryMappedFileAccess access, Int64 requestedOffset, Int64 requestedSize)
   at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewAccessor(Int64 offset, Int64 size, MemoryMappedFileAccess access)
   at Microsoft.Psi.Persistence.InfiniteFileWriter.CreateNewExtent() in /home/xiangzht/Dev/psi/psi/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileWriter.cs:line 294
   at Microsoft.Psi.Persistence.InfiniteFileWriter..ctor(String path, String fileName, Int32 extentSize, Boolean append) in /home/xiangzht/Dev/psi/psi/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileWriter.cs:line 52
   at Microsoft.Psi.Persistence.StoreWriter..ctor(String name, String path, Boolean createSubdirectory, Boolean append) in /home/xiangzht/Dev/psi/psi/Sources/Runtime/Microsoft.Psi/Persistence/StoreWriter.cs:line 89
   at Microsoft.Psi.Data.Exporter..ctor(Pipeline pipeline, String name, String path, Boolean createSubdirectory, KnownSerializers serializers) in /home/xiangzht/Dev/psi/psi/Sources/Runtime/Microsoft.Psi/Data/Exporter.cs:line 44
   at Microsoft.Psi.Store.Create(Pipeline pipeline, String name, String rootPath, Boolean createSubdirectory, KnownSerializers serializers) in /home/xiangzht/Dev/psi/psi/Sources/Runtime/Microsoft.Psi/Data/Store.cs:line 42
   at Microsoft.Psi.Samples.LinuxSpeechSample.Program.RunAzureSpeech() in /home/xiangzht/Dev/psi/psi/Samples/LinuxSpeechSample/Program.cs:line 88
   at Microsoft.Psi.Samples.LinuxSpeechSample.Program.Main() in /home/xiangzht/Dev/psi/psi/Samples/LinuxSpeechSample/Program.cs:line 49

The problem seems to be in Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileWriter.cs line 275. In Linux, the file is closed upon exiting the scope.

using (var file = File.Open(this.extentName, FileMode.Create, FileAccess.ReadWrite, FileShare.Read))
{
    newMMF = MemoryMappedFile.CreateFromFile(file, null, this.extentSize, MemoryMappedFileAccess.ReadWrite, HandleInheritability.Inheritable, false);
}

My current workaround is to drop the using and do the following instead.

var file = File.Open(this.extentName, FileMode.Create, FileAccess.ReadWrite, FileShare.Read);
newMMF = MemoryMappedFile.CreateFromFile(file, null, this.extentSize, MemoryMappedFileAccess.ReadWrite, HandleInheritability.Inheritable, false);

I'm not sure about the repercussions about not closing the file.

This is just a small documentation update that should probably be added.
To be able to run the brief introduction example: 5.1 Offline Visualization.
this one --> https://github.com/microsoft/psi/wiki/Brief-Introduction#51-offline-visualization
These Nuget references where missing for me when I was following this introduction:

• Microsoft.Psi.Audio
• Microsoft.Psi.Audio.Windows
• Microsoft.Psi.Imaging
• Microsoft.Psi.Speech

edit:
I just found out that this depends on what kind of project is chosen when creating a project.
In the documentation it states that you can take a .netcore console or a .netframework console project. I went for the .netcore one.
But apparently at least in my case these dependent Nuget references weren't added when adding the ones that are in the documentation.

And also because I went for the .netcore project I had to replace <TargetFramework>netcoreapp3.0</TargetFramework> with <TargetFramework>net472</TargetFramework> in the ".cspjoj" file (project file).

edit 2:
I also did some testing right now to see if everything works when you use a .netframework console project.
I found out that indeed the dependent Nuget references are added automaticly when I added the the references specified in the documentation.
But I still had to change something else to be able to run the code. I had to go to my project properties and under build I had to change the "Platform Target" to check "Optimize code" only. (and uncheck "Prefer 32-bit")

Is the Kinect v1 supported? I've been using the Kinect v2 with psi, running the sample, and that all works fine. Since we have a few extra v1s lying around, I was hoping to run things with those. Before I devote too much time into this effort, I was wondering if the Kinect v1 is even likely to work with psi.

When I attempt to use Studio to open the stores saved out while running the Kinect sample, I cannot visualize any of the saved streams. Any 2D data streams (i.e., video) open a visualization panel with no data, and attempting to plot any of the 1D streams causes the program to crash.

Just to confirm that I'm doing this correctly, here's the process:

-Run the Kinect sample for some length of time
-Stop the program and close the Psi Studio window
-Run Psi Studio
-Open the catalog file of the store that was created when running the Kinect sample
-Select "plot" (resulting in a crash) or "visualize" (resulting in an empty visualization panel) on any of the streams in the store

We got an old machine running 64 bit Windows 7 (updated to most recent patch) and wanted to try out /psi on it. I installed Visual Studio 2019 on it and created a new package according to the first example in the tutorial. The package targets .NET Core 2.1 and is a console application. We added the default code:

using Microsoft.Psi;

static void Main(string[] args)
{
    using (var p = Pipeline.Create())
    {
        var timer = Timers.Timer(p, TimeSpan.FromSeconds(0.1));
        timer.Do(t => Console.WriteLine(t));
        p.Run();
    }
}

The package was built successfully but when it start running, it threw an exception:System.EntryPointNotFoundException: 'Unable to find an entry point named 'GetSystemTimePreciseAsFileTime' in DLL 'kernel32.dll'.' at the line using (var p = Pipeline.Create()). A quick online search seemed to suggest that it's due to GetSystemTimePreciseAsFileTime doesn't exist in Windows 7. Could you validate whether this is true and the only way to solve it is through upgrading it to Window 10?

I also tried changing the platform target to x64 in Build and it still threw the same exception.

Exception details:

System.EntryPointNotFoundException
  HResult=0x80131523
  Message=Unable to find an entry point named 'GetSystemTimePreciseAsFileTime' in DLL 'kernel32.dll'.
  Source=Microsoft.Psi
  StackTrace:
   at Microsoft.Psi.Platform.Windows.HighResolutionTime.GetSystemTimePreciseAsFileTime(Int64& systemTimeAsFileTime)
   at Microsoft.Psi.Platform.Windows.HighResolutionTime.SystemTime()
   at Microsoft.Psi.Clock..ctor(DateTime virtualNow, Single replaySpeedFactor)
   at Microsoft.Psi.Scheduling.Scheduler..ctor(Func`2 errorHandler, Int32 threadCount, Boolean allowSchedulingOnExternalThreads, String name, Clock clock)
   at Microsoft.Psi.Pipeline..ctor(String name, DeliveryPolicy deliveryPolicy, Int32 threadCount, Boolean allowSchedulingOnExternalThreads)
   at Microsoft.Psi.Pipeline.Create(String name, DeliveryPolicy deliveryPolicy, Int32 threadCount, Boolean allowSchedulingOnExternalThreads)
   at PsiExample1.Program.Main(String[] args) in C:\Users\xiangzht\source\repos\PsiExample1\PsiExample1\Program.cs:line 10

I'm trying to use the Intel Realsense cameras in Psi. I downloaded the SDK 2.0 and added the environment variable. The building process is successful. When I changed the WebcamWithAudioSample to a RealsenseSample, I got the following exception:

System.BadImageFormatException: Could not load file or assembly 'Microsoft.Psi.RealSense_Interop.Windows.x64, Version=0.10.16.1, Culture=neutral, PublicKeyToken=null'. An attempt was made to load a program with an incorrect format.
File name: 'Microsoft.Psi.RealSense_Interop.Windows.x64, Version=0.10.16.1, Culture=neutral, PublicKeyToken=null'
   at Microsoft.Psi.RealSense.Windows.RealSenseSensor.Start(Action`1 notifyCompletionTime)
   at Microsoft.Psi.Executive.PipelineElement.<Activate>b__47_0() in C:\Users\thisiswys\Downloads\psi-master\Sources\Runtime\Microsoft.Psi\Executive\PipelineElement.cs:line 259
   at Microsoft.Psi.Executive.PipelineElement.<>c__DisplayClass42_0.<TrackStateObjectOnContext>b__0() in C:\Users\thisiswys\Downloads\psi-master\Sources\Runtime\Microsoft.Psi\Executive\PipelineElement.cs:line 143
   at Microsoft.Psi.Scheduling.Scheduler.ExecuteAndRelease(SynchronizationLock synchronizationObject, Action action, SchedulerContext context) in C:\Users\thisiswys\Downloads\psi-master\Sources\Runtime\Microsoft.Psi\Scheduling\Scheduler.cs:line 525

Here are the lines I changed in the sample:

I know that BadImageFormatException is always related to a mismatch of the target platform and #7 has encountered a similar problem. I analyzed my Microsoft.Psi.RealSense_Interop.Windows.x64.dll and believe that this file is in x64 format. And I ran the sample in x64 mode as follows:

but still I'm getting the exception. What could be the problem?

Hi, I am not able to subscribe image messages on psi by using RosMessageTypes.Sensor.Image definition. I am not 100% sure whether the problem is on the psi side it might be on the publisher side as well. However I have minimum control on the publisher and I was able to fix the problem on psi by using a custom message definition as follows :

private RosMessage.MessageDef depthImageMessageDef = RosMessage.CreateMessageDef(

      `"sensor_msgs/Image",
       "060021388200f6f0f447d0fcd9c64743",
       new[]
       {
                    Tuple.Create("header", RosMessage.RosFieldDef.NewStructDef(Standard.Header.Def.Fields)),
                    Tuple.Create("height", RosMessage.RosFieldDef.UInt32Def),
                    Tuple.Create("width", RosMessage.RosFieldDef.UInt32Def),
                    Tuple.Create("encoding", RosMessage.RosFieldDef.StringDef),
                    Tuple.Create("is_bigendian", RosMessage.RosFieldDef.BoolDef),
                    Tuple.Create("step", RosMessage.RosFieldDef.UInt32Def),
                    Tuple.Create("data", RosMessage.RosFieldDef.NewVariableArrayDef(RosMessage.RosFieldDef.UInt8Def)),
       });`

The only difference I see is that on uint32 fields are defined as int and data field is defined as byte array at Microsoft.Ros.RosMessageTypes.Sensor.Image.

By the way, I am able to subscribe other types of messages such as : RosMessageTypes.Geometry.Twist.Def and RosMessageTypes.Sensor.Range.Def

Do not rely on the originating times of messages from the SystemSpeechRecognizer and MicrosoftSpeechRecognizer components to be precise with respect to the input audio stream.

Workaround:
If such precision is required, align the bytes in the StreamingSpeechRecognitionResult.Audio property of the output message with the raw input audio to locate the corresponding utterance within the input audio stream.

I am attempting to install this on Ubuntu 16.04.

After cloning psi, installing .NET core and VS Code, I tried to run build.sh but there was a missing ilasm binary necessary for building.

I tried this: https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md which, after compilation, generates the ilasm binary, but if I try to run it by pointing my PATH env var to it, I get segfaults.

(This is Casey Kennington, btw)

I'm trying to install the new package (0.4.216.2-beta), but it is unable to resolve a dependency. Output of trying to install is this:

Attempting to gather dependency information for package 'Microsoft.Psi.Visualization.Windows.0.4.216.2-beta' with respect to project 'KioskMain', targeting '.NETFramework,Version=v4.7'
Gathering dependency information took 469.6 ms
Attempting to resolve dependencies for package 'Microsoft.Psi.Visualization.Windows.0.4.216.2-beta' with DependencyBehavior 'Lowest'
Unable to resolve dependency 'Microsoft.Psi.Data'. Source(s) used: 'nuget.org', 'Microsoft Visual Studio Offline Packages'.
Time Elapsed: 00:00:00.4755751
========== Finished ==========

I should also note that I had no problem installing any other update.

Sometimes the speech recognition results emitted by the SystemSpeechRecognizer and MicrosoftSpeechRecognizer components have incorrect originating timestamps which may be slightly off when compared against the input audio stream (i.e. the results appear to be shifted in time). This may be related to the way in which the originating times of the recognition results are estimated from the underlying recognition engine's internal audio offset position.

I've been trying to get PsiStudio up and running and have run into the same issue as issue #5 . The Psi.Sln built without issue however, after I installed the Kinect SDK, I still can't seem to get past the issue #5 and get the same exceptions. I'm trying to Debug Microsoft.Psi.PsiStudio for Any CPU. Here are the exceptions:

1. BadImageFormatException: Could not load file or assembly 'Microsoft.Kinect, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Reference assemblies should not be loaded for execution. They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)
2. BadImageFormatException: Cannot load a reference assembly for execution.

Looks like the Microsoft.Kinect dll is found in the C:\Users\LaptopName.nuget\packages\microsoft.kinect\2.0.1410.19000\lib\net45 location.

Should Psi point to the Kinect install location C:\Program Files\Microsoft SDKs\Kinect\v2.0_1409\Assemblies location? They look like the exact same dll only in different locations.

Any help is appreciated!

I see there is a window operator but that is waiting for the window to be filled, or at least it seems that if I do Average(TimeSpan.FromSeconds(1)) it will cause the producer to emit every second instead of retaining the inbound frequency.

This is useful to computer rms, loudness and smoothing signal without loosing frequency. Another application would be to create signal filters to smooth curves.

A common thing here would be to provide lowpass, gates, compressors, kalman, 1 euro operators

The Kinect face tracking + speech-to-text sample throws an uncaught exception on the first frame when running. I determined that the error is due to KinectFaceDetector.Faces emitting a list with zero items under a Select operation, in these two lines:

var mouthOpenAsFloat = kinectFaceDetector.Faces.Select((List<Microsoft.Psi.Kinect.Face.KinectFace> list) => (line 85)
var landmarks = kinectFaceDetector.Faces.Select((List<Microsoft.Psi.Kinect.Face.KinectFace> list) => (line 116)

In both cases, the code block inside the Select crashes due to an empty list. I correct it by adding list.Count > 0 tests in both blocks; however, I'm not sure if this is the intended operation of the Select operator or if it should not have fired at all.

Interop works beautifully when the psi app is running in Windows (and I can interop with Linux modules in other languages) using zmq. However, I'd like to setup an app on a Linux box and it would be really helpful to have interop capabilities.

Replaying data from a data store has generally been very useful, and this is the first time I have had a type that was not able to be deserialized. It looks like the error is related to an enum nested inside of the class. What is the best way to handle this? Just move the enum to outside the class? Do I need to define a deserializer? Or maybe I need to store with primitive types and reconstruct the type after deserializing?

System.AggregateException
HResult=0x80131500
Message=Pipeline 'default' was terminated because of one or more unexpected errors (Failed to create a deserializer for type System.Collections.Generic.EnumEqualityComparer1[[EmotionRepresentation.EmotionCategory+Category, EmotionRepresentation, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 because no type was registered for this name and the source type System.Collections.Generic.EnumEqualityComparer1[[EmotionRepresentation.EmotionCategory+Category, EmotionRepresentation, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 could not be found. Add a reference to the assembly containing this type, or register an alternate type for this name.)
Source=Microsoft.Psi
StackTrace:
at Microsoft.Psi.Pipeline.ThrowIfError()
at MainCore.ReasonerTest.Main(String[] args) in C:\linux\repos\temp\psiAssistiveAgent\MainCore\ReasonerTest.cs:line 44

Inner Exception 1:
SerializationException: Failed to create a deserializer for type System.Collections.Generic.EnumEqualityComparer1[[EmotionRepresentation.EmotionCategory+Category, EmotionRepresentation, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 because no type was registered for this name and the source type System.Collections.Generic.EnumEqualityComparer1[[EmotionRepresentation.EmotionCategory+Category, EmotionRepresentation, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 could not be found. Add a reference to the assembly containing this type, or register an alternate type for this name.

I just downloaded the latest version of Psi and trying out the Azure Kinect Component. On my Ubuntu machine, it refused to build and throwing the following error. I built it using the build.sh script that came with Psi.

Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
/home/xiangzht/.nuget/packages/microsoft.azure.kinect.sensor/1.3.0/build/netstandard2.0/Microsoft.Azure.Kinect.Sensor.targets(4,5): error : Azure Kinect only supports the x86/x64 platform ('AnyCPU' not supported) [/home/xiangzht/Dev/Psi/psi/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.csproj]

Build FAILED.

/home/xiangzht/.nuget/packages/microsoft.azure.kinect.sensor/1.3.0/build/netstandard2.0/Microsoft.Azure.Kinect.Sensor.targets(4,5): error : Azure Kinect only supports the x86/x64 platform ('AnyCPU' not supported) [/home/xiangzht/Dev/Psi/psi/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:00.49

I tried and successfully run it on my Windows Partition, so it seemed to be a Linux issue. I'm trying to debug it but any pointers will be helpful (Including if it won't work on Ubuntu)

The following code block generates a "generic GDI+ error":

using (Pipeline pipeline = Pipeline.Create("PsiNucStorage"))
            {
                KinectSensorConfiguration kinectSensorConfig = new KinectSensorConfiguration();
                kinectSensorConfig.OutputColor = true;
                kinectSensorConfig.OutputDepth = true;
                KinectSensor kinectSensor = new KinectSensor(pipeline, kinectSensorConfig);

                Exporter store = Store.Create(pipeline, "psiKinect", @".\data");

                Emitter<Shared<EncodedImage>> rgbImageStream = kinectSensor.ColorImage.Resize(100, 100).EncodeJpeg(100, DeliveryPolicy.LatestMessage).Out;
                Store.Write(rgbImageStream, "rgbImages", store, true, DeliveryPolicy.LatestMessage);

                Emitter<Shared<EncodedImage>> depthImageStream = kinectSensor.DepthImage.Resize(100, 100).EncodePng(DeliveryPolicy.LatestMessage).Out;
                Store.Write(depthImageStream, "depthImages", store, true, DeliveryPolicy.LatestMessage);

                Console.WriteLine("Starting data collect");
                pipeline.Run();
            }

The culprit seems to be the resize operation on the depth image stream - removing that stage resolves the issue.

Consecutive messages posted on the same stream with identical originating time timestamps are ambiguous when the stream is joined with another stream. In this case the result of the join operation is timing-dependent and not reproducible.

Workaround: Ensure the originating time on consecutive messages is incremented by at least 1 tick.

Hello!

I was wondering if there's any existing plans for Mac support for Media Capture and Imaging? It seems like the Linux Imaging component uses SkiaSharp for encoding/decoding, so perhaps that could be used on MacOS as well, but it might be worthwhile to go the CoreImage route there?

Similarly, the FFMpeg Native Reader used in the Linux component could also be used, but it might be better long term to use AVFoundation for media capture, because it would also pave the path for capturing from sensors on iOS, for writing mobile applications in the future.

Also, I suspect using the native approach would result in more CPU and/or memory efficient capture.

1. I was wondering if the dev team has any thoughts about this?
2. If there are no plans, and I write one for my needs, would it be more appropriate to submit a pull request or release as a separate component?

I'm trying to do an AudioCapture in a Moby Container (Azure IoT Edge) running on Raspbian GNU/Linux 10 (buster).

The following packages are installed (in the Container and on the host):

• alsa-utils
• libasound2
• libasound2-dev

Below is the Exception I get:

System.DllNotFoundException: Unable to load shared library 'asound' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libasound: cannot open shared object file: No such file or directory
   at Microsoft.Psi.Audio.LinuxAudioInterop.Open(Void** handle, String name, Int32 capture, Int32 mode)
   at Microsoft.Psi.Audio.LinuxAudioInterop.Open(String name, Mode mode, Int32 rate, Int32 channels, Format format, Access access)
   at Microsoft.Psi.Audio.AudioCapture.Start(Action`1 notifyCompletionTime)
   at Microsoft.Psi.Executive.PipelineElement.<Activate>b__46_0()
   at Microsoft.Psi.Scheduling.Scheduler.ExecuteAndRelease(SynchronizationLock synchronizationObject, Action action, SchedulerContext context)
   --- End of inner exception stack trace ---
   at Microsoft.Psi.Pipeline.ThrowIfError()
   at Microsoft.Psi.Pipeline.Run(ReplayDescriptor descriptor)

Here's the code I'm using:

            using (var pipeline = Pipeline.Create())
            {
                // Windows Version
                //var audioCapture = new AudioCapture(pipeline, new AudioCaptureConfiguration
                //{
                //    DropOutOfOrderPackets = true,
                //    OutputFormat = WaveFormat.Create(WaveFormatTag.WAVE_FORMAT_PCM, _rate, 16, _channels, 2, 88200),
                //});
                var audioCapture = new AudioCapture(pipeline, new AudioCaptureConfiguration
                {
                    Format = WaveFormat.Create(WaveFormatTag.WAVE_FORMAT_PCM, _rate, 16, _channels, 2, 88200)
                });

                audioCapture.Do(async (buffer, envelope) => await ProcessAudioMessage(buffer, envelope));
                pipeline.Run();
            }

I've tested the solution on a Windows machine, there it works without any problem.

I remember seeing a demo of the Diagnostics, but I cannot remember how to recreate it. And I can't find any documentation on this yet. I have found the API documentation for things in the Diagnostics namespace, but that hasn't helped me in figuring out how to use it.

I'm currently not interested in visualizing this data in PsiStudio. I want to have a component that can read the diagnostics data. I will then have a component with a websocket allowing me to access this realtime data through a web interface. I am primarily interested in verifying that all components are operational. But once I have the connection all working, I'll probably have other diagnostics information available via the web interface.

Does /psi work well on Mac OS X? What are the limitations?

When Join is used with a match window specified via a RelativeTimeInterval that has open endpoints (e.g. the endpoint itself is not included in the interval), may produce incorrect results (the interpolator incorrectly assumes closed endpoints).

Hi, I am a member of Northwestern University QRG. We believed that we got the version from github, setup all paths correctly and built all projects successfully. However, when we started the WebcamWithAudioSample in Visual studio 2017 with x64 under Debug mode, we got the exception "System.Runtime.InteropServices.COMException: 'Retrieving the COM class factory for component with CLSID {8F2D3209-E9CF-49A0-BCCB-18264AC9F676} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).'"
on the 22 line of VisualizationClient.cs file: "this.visualizationService = (IRemoteVisualizationService)new VisualizationService();". We assume that we followed all instructions online and installed the right version of .Net 4.7. How can we solve this problem?

It seems that it is not possible to connect a single ConsumerProducer component to multiple Consumer components. I cannot find anything about doing this in the docs. Is there any reason why this should not be possible? Are there plans to make this possible?

I am trying to connect a webcam through "rtsp protocol". I found in the "webcam sample", It uses the camera integrated into the computer,.Should I use the "Mediacapture" component or any other components to find another webcam? If it is the "Mediacapture" component, which place I can modify to find the rtsp address?
Thank you very much for answering!

I'm playing around rosturtlesim code and I wanted to change subscription to another datatype. I assume I dont have to write a new message type for an existing message type in ros (i.e., sesors.range) but how about the event handler method signature? How should PoseUpdate(IEnumerable<Tuple<string, RosMessage.RosFieldVal>> position) should change in that case?

I am setting up NetMQ/ZeroMQ to communicate between psi and python. I have some structured data going to python and getting parsed in python just fine. For example, the message payload of { 'help' : '1' } is easily read into a python dictionary.

The other direction has been harder. I'm trying to respond with the following message (full message just for completeness):
{'message': {'say': 'hello'}, 'originatingTime': '2020-04-02T23:24:30.010592'}
In psi, I wasn't able to setup a NetMQSource<Dictionary<string,string>> to emit a message with this payload. I changed the NetMQSource type to dynamic, and it emits fine, and I can see it is an ExpandoObject (which is expected based on my reading of JsonFormat.cs). But now I'm not sure what to do with that. Maybe I'm just not familiar with how to use ExpandoObject. Is there any way to get this to a Dictionary?

When building I see that there are dependencies on older sdk versions

The solution has a lot of dependencies, could this be simplified with a solution for

• all
• runtime (cross platform)
• additional modules (speech and all other things like audio and so on)
• platform specific modules

That could improve the experience for contributors and code exploration.

We have found that in Microsoft.Psi.Calibration the ProjectTo3D component and corresponding stream operator, as well as the underlying DepthExtensions.ProjectToCameraSpace() method that they rely on produce incorrect results. The fix requires changing hypothesisPoint.Z to hypothesisPoint.X on line 77 in DepthExtensions.cs, and changing the last parameter in the call to IntersectLineWithDepthMesh from line 34 from 0.1 to 0.001.

Also, the Point3D visualizer in PsiStudio displays the points at incorrect coordinates. The fix requires changing line 63 in Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/PointsVisual.cs from

this.spheres[i].Transform = new TranslateTransform3D(-point.Z, point.X, point.Y);

to

this.spheres[i].Transform = new TranslateTransform3D(point.X, point.Y, point.Z); 

We plan to address this issue in the next release.

We are trying to use Psi with some input of some cameras with the fisheye lens. We attempted to use OpenCV or Matlab to do the calibrations and integrate the code into the system. Before that, we found some related classes and methods in Psi under the namespace Microsoft.Psi.Calibration. But we can't find enough documents on how to use them.

Can you give some examples of camera calibration? Thanks!