fileonq / imaging.heif Goto Github PK

Description

In both .NET 5 and .NET 6 benchmarks the ToSpan has extra allocations that don't make sense. This API wraps the native pointer from the encoders project and it shouldn't be creating any additional allocations. See benchmark table below.

The benchmarks have been ran on different machines the important value is the memory allocation

Description

When a community member submits a pull request, they will most likely fork this repository and then submit the PR. Right now the benchmarks fail to checkout code correctly from the forked repo. This is due to how the ref is piped from the first build to the next. It assumes that it is retrieving the ref from the origin repository and doesn't take into account forked repositories.

Root Cause

The benchmark.yml build checks if it is a new PR or been requested to run via the check-command job. It will then store the current git ref and pass that value to the next job benchmark-build. This job will try and checkout the code and compile, but it assumes that it is checking out the origin repo. If the PR is a forked repository then the checkout command will fail.

Fetching the repository
  "C:\Program Files\Git\bin\git.exe" -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/libjpeg-turbo*:refs/remotes/origin/libjpeg-turbo* +refs/tags/libjpeg-turbo*:refs/tags/libjpeg-turbo*
  The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 1
  Waiting 14 seconds before trying again
  "C:\Program Files\Git\bin\git.exe" -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/libjpeg-turbo*:refs/remotes/origin/libjpeg-turbo* +refs/tags/libjpeg-turbo*:refs/tags/libjpeg-turbo*
  The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 1
  Waiting 13 seconds before trying again
  "C:\Program Files\Git\bin\git.exe" -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/libjpeg-turbo*:refs/remotes/origin/libjpeg-turbo* +refs/tags/libjpeg-turbo*:refs/tags/libjpeg-turbo*
  Error: The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 1

Proposed Changes

Update the benchmark.yml to work with forked repositories

Description

Currently the benchmarks project is only targeting .net5. This needs to support 4.8, .net5, and .net6.

Description

When using the generated NuGet FileOnQ.Imaging.Heif.1.0.0-pr.20 I am unable to run the Save() code as it fails with a VC++ Runtime Library error. See screenshot below

When I run the build locally it works without issue, it only fails when using the generated artifacts from the build.

• GitHub Action - build.yml from the build branch
• GitHub Action Result
• FileOnQ.Imaging.Heif.1.0.0-pr.20.zip

Local Steps

The build.yml follows a similar set of instructions I am doing locally.

1. Ensure build tools are all installed
2. build FileOnQ.Imaging.Heif this will also compile all native binaries
3. Update NuGets for all ConsoleApp targets
4. Run console app

The console app will save correctly. If you swap the NuGet out for the generated one it will fail on the save.

Screenshot

Additional Information

I am working on putting together getting started docs to make is easier to get up and running. This hasn't been merged into the main branch and can be found in the docs branch.

• building.md

Description

Add ability to resize the primary image or thumbnail using the libheif native resize api.

New APIs

IImage

void Resize(int width, int height); // New API
void Resize(int widthAndHeight); // New API

Usage

The code below would generate a 200x200 square thumbnail. This use case is valuable for heic images that don't have an embedded thumbnail.

using (var image = new HeifImage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
  primary.Resize(200, 200);
  primary.Save("Output.jpeg", 90);
}

Description

When decoding and writing images to jpeg libheif performs a conversion from the native colorspace and chroma to the expected YUV (colorspace) and 420 (chroma). This operation takes about .5s - 1.2s in debug mode. If we detect this prior to attempting a decode and keep it in the same format we may be able to shave time off our image processing. This will require updating our jpeg encoder to handle a variety of colorspace/chromas and have a fallback strategy

Description

This project is a complex mixture of .NET Framework, .NET and native C/C++. We should add a developer guide that will explain the different areas of the project which will help contributors understand how to use the solution and make changes.

For example the unit test project automatically builds a nupkg which allows debugging. This developer guide will mention it is a best practice to do a clean and rebuild on the unit test project so you can step into the apis and debug.

Description

The build process compiles libjpeg-turbo in debug mode instead of release mode. This generates a file that is roughly 600kb larger than if it was built in release mode. It is a best practice to try and compile upstream libraries in release mode as the code is optimized.

During the initial proof of concept the JPEG encoder that users libjpeg-turbo was failing to save. We were seeing errors when writing to disk. After investigation it was determined that we are going to have to use debug mode for now and try and fix this later

Sample Code

The snippet below is from the original proof of concept and will change in the future

public HeifImage(string file)
{
	heifContext = LibHeifContext.heif_context_alloc();
	var error = LibHeifContext.heif_context_read_from_file(heifContext, file, IntPtr.Zero);
	if (error.Code != LibHeifContext.ErrorCode.Ok)
		throw new Exception(Marshal.PtrToStringAnsi(error.Message));

	LibHeifContext.ImageHandle* imageHandle;
	var imageError = LibHeifContext.heif_context_get_primary_image_handle(heifContext, &imageHandle);

	var numberOfThumbnails = LibHeifContext.heif_image_handle_get_number_of_thumbnails(imageHandle);
	if (numberOfThumbnails > 0)
	{
		var itemIds = new uint[numberOfThumbnails];
		fixed (uint* ptr = itemIds)
		{
			LibHeifContext.heif_image_handle_get_list_of_thumbnail_IDs(imageHandle, ptr, numberOfThumbnails);
		}

		// no idea why this is failing
		LibHeifContext.ImageHandle* thumbHandle;
		var thumbError = LibHeifContext.heif_image_handle_get_thumbnail(imageHandle, itemIds[0], &thumbHandle);
		if (thumbError.Code != LibHeifContext.ErrorCode.Ok)
			throw new Exception(Marshal.PtrToStringAnsi(thumbError.Message));

		Encode(thumbHandle);
	}
	else
	{
		Encode(imageHandle);
	}

	void Encode(LibHeifContext.ImageHandle* handle)
	{
		var hasAlpha = LibHeifContext.heif_image_handle_has_alpha_channel(handle) == 1;
		var encoder = LibEncoder.encoder_jpeg_init(90);
		var options = LibHeifContext.heif_decoding_options_alloc();
		LibEncoder.encoder_update_decoding_options(encoder, handle, options);

		var bitDepth = LibHeifContext.heif_image_handle_get_luma_bits_per_pixel(handle);
		if (bitDepth < 0)
		{
			LibHeifContext.heif_decoding_options_free(options);
			LibHeifContext.heif_image_handle_release(handle);
			throw new Exception("Input image has undefined bit-dept");
		}

		LibHeifContext.Image* outputImage;
		var decodeError = LibHeifContext.heif_decode_image(
			handle,
			&outputImage,
			LibEncoder.encoder_colorspace(encoder, hasAlpha),
			LibEncoder.encoder_chroma(encoder, hasAlpha, bitDepth),
			options);

		LibHeifContext.heif_decoding_options_free(options);

		if (decodeError.Code != LibHeifContext.ErrorCode.Ok)
		{
			LibHeifContext.heif_image_handle_release(handle);
			throw new Exception(Marshal.PtrToStringAnsi(decodeError.Message));
		}

		if ((IntPtr)outputImage != IntPtr.Zero)
		{
			bool saved = LibEncoder.encode(encoder, handle, outputImage, "output.jpeg");
			if (!saved)
				throw new Exception("Unable to save");
		}

		LibEncoder.encoder_free(encoder);
	}
}

The libjpeg-turbo code fails on LibEncoder.encode(encoder, handle, outputImage, "output.jpeg") when the library is built under release mode but works when the library is built under debug mode

Difficulty: [Hard]

Description

Add thumbnail count API to determine how many thumbnail images if any are embedded in the heic image

New APIs

IHeifImage

// Gets the number of thumbnails embedded in the heif file
// this is typically 0 or 1
int GetThumbnailCount(); // New API

// Gets an array of all available thumbnail ids, if 0 
// then the image doesn't have any thumbnails
int[] GetThumbnailIds(); // New API

// Gets the thumbnail at the specified ID
IImage Thumbnail(int id); // New API

Usage

The code snippet below will write all thumbnails to disk

using (var image = new HeifImage("MyImage.heic"))
{
  int[] thumbnailIds = image.GetThumbnailIds();
  for (int i = 0; i , thumbnailIds.Length; i++)
  {
    using (var thumbnail = image.GetThumbnail(thumbnailIds[i]))
    {
        thumbnail.Write($"output_{i}.jpeg", 90);
    }
  }
}

Description

Prior to releasing the 1.0.0 version of FileOnQ.Imaging.Heif we need to complete all items below

• #58
• #59
• #7
• #11
• #15
• #63
• #9
• #21
• #64
• #60
• #13
• #61
• #22
• #28
• #31
• #32
• #38
• #40
• #2
• #37
• #43
• #46
• #65
• #68

Description

Add a benchmarking project that validates the performance of the application. This should have standard benchmarks for all documented API workflows.

Description

Update all public APIs in FileOnQ.Imaging.Heif to have proper xml docs. Verify the docs show up using the NuGet package.

Description

Add documentation for minimum versions of Visual Studio and dotnet

Goals

• Add minimum Visual Studio version
• Add minimum dotnet version
• Add globals.json to force dotnet to specific version version

Description

Add a baseline benchmark to each printed table. This will show the existing benchmark compared to the PR benchmark side-by-side. The data is saved in the repository, so we just need to extract it and merge it with the existing table

Description

Add new GitHub Actions to release final NuGets based on git tagging strategy

Description

The benchmark.yml hooks into issue comments which are used in both PRs and Issues, this build is only useful from a PR. We need to update the build to detect if it is running on an issue and stop it from processing. See build log below

https://github.com/FileOnQ/Imaging.Heif/runs/4795761106?check_suite_focus=true

Description

Add support for latest build in .NET 6 and update all .NET 5 builds to latest stable patch.

Tasks

• Update all projects and sample code to compile with .NET 6
• Update all projects and sample code to compile with latest patch in .NET 5
• Update NuGet packages to ship .NET 5, .NET 6 and .NET 4.8
• Add build to compile and test with .NET48, .NET5, and .NET 6

Description

There is a very small memory leak with .NET 6 when writing an image to storage. See benchmarking results below

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.19043.1415 (21H1/May2021Update)
AMD Ryzen 9 3950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  Job-KBXXVN : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT

Runtime=.NET 6.0  InvocationCount=1  LaunchCount=1
UnrollFactor=1

|             Method |     Mean |    Error |   StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
|------------------- |---------:|---------:|---------:|------:|------:|------:|----------:|------------------------:|-------------------:|
|    Thumbnail_Write | 65.39 ms | 0.928 ms | 0.868 ms |     - |     - |     - |     832 B |             5,982,865 B |               32 B |
|  Thumbnail_ToArray | 63.64 ms | 0.366 ms | 0.324 ms |     - |     - |     - |  66,888 B |             5,982,157 B |                  - |
|   Thumbnail_ToSpan | 63.72 ms | 0.403 ms | 0.377 ms |     - |     - |     - |     600 B |             5,981,901 B |                  - |
| Thumbnail_ToStream | 63.90 ms | 0.388 ms | 0.363 ms |     - |     - |     - |  66,952 B |             5,981,901 B |                  - |

Description

Update the NuGet package properties to have all of the correct data for the resulting NuGet package. This includes description, summary, tags, etc.

Description

We added the Interop API which updates the dll search directory for .NET Framework to match behaviors in .NET5+. This means we no longer need to have explicit x86 vs x64 interop classes. They all reference the exact same assembly name. Let's remove the duplicate file but keep the native interop wrapper. See code snippet example below.

internal static IntPtr InitJpegEncoder(int quality)
{
	switch (RuntimeInformation.ProcessArchitecture)
	{
		case Architecture.X64:
		case Architecture.X86:
			return native.encoder_jpeg_init(quality);
		default:
			throw new NotSupportedException($"Current platform ({RuntimeInformation.ProcessArchitecture}) is not supported");
	}
}

Description

In #32 and #31 we document the development side of building the automation for release tagging. We now need to create a readme file that documents our release process which includes docs on the following

• Dev Builds and their purpose
• Release Builds
• How to trigger a release build
• Version Schema

Description

It was discovered that we are seeing a memory leak in FileOnQ.Imaging.Heif when targeting .NET 4.8. Both the thumbnail (234B) and primary image (234B) are impacted. Below is the benchmarks.

Description

Add memory buffer APIs that can return an array or a stream

New APIs

IImage

byte[] ToArray(int quality = 90)
Stream AsStream(int quality = 90)
ReadOnlySpan<byte> AsReadOnlySpan(int quality = 90)

Usage - ToArray()

byte[] buffer;
using (var image = new HeifImage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
  buffer = primary.ToArray();
}

// do something with buffer

Usage - AsStream

Stream stream;
using (var image = new Heifimage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
  stream = primary.AsStream();
}

// do something with stream

// don't forget to dispose of your stream
stream?.Dispose();

Usage - AsReadOnlySpan

This one is a little tricky as Span<T> gives us access to native memory. If we allow C# to operate on native memory we should ensure it happens within the IDisposable using block otherwise there will be a memory leak.

ReadOnlySpan<byte> span;
using (var image = new HeifImage("MyImage.heic"))
using (var primary = image.PrimaryImage())
{
  ReadOnlySpan<byte> span = primary.AsReadOnlySpan();

  // do something with span
}

I noticed the redistributable links on README all point to the arm download.

Description

Compile project for use on iOS devices

Description

Compile project for use on Windows ARM64

Description

Add the entire .github folder to the solution so it is accessible from within Visual Studio. As new files are added we will need to add it to the solution manually

Description

During the build process it runs several chocolatey install steps to ensure that all C++ build dependencies are installed. If this step fails it doesn't automatically fail the build.

choco install meson
choco install ninja
choco install nasm

We need to update this script to fail if there is an issue downloading deps

Difficulty: [Easy]

Description

Compile project for use on Android devices

Description

After implementing automated benchmarking it documented there are large memory leaks with the current implementation. The data is available in #20 as well as the table copied below

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.17763.2114 (1809/October2018Update/Redstone5)
Intel Xeon Platinum 8171M CPU 2.60GHz, 1 CPU, 2 logical and 2 physical cores
.NET SDK=5.0.400
 [Host]     : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT
 Job-VLDWLB : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT

Runtime=.NET 5.0  InvocationCount=1  LaunchCount=1  
UnrollFactor=1  

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.17763.2114 (1809/October2018Update/Redstone5)
Intel Xeon Platinum 8171M CPU 2.60GHz, 1 CPU, 2 logical and 2 physical cores
.NET SDK=5.0.400
 [Host]     : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT
 Job-DXULZI : .NET 5.0.9 (5.0.921.35908), X64 RyuJIT

Runtime=.NET 5.0  InvocationCount=1  LaunchCount=1  
UnrollFactor=1  

We need to ensure all native memory is freed so these result in 0 native memory leaked

Description

Update build to sign all FileOnQ generated assemblies from this project.

• FileOnQ.Imaging.Heif
• FileOnQ.Imaging.Heif.Encoders

Both the PR build and the Main build will require code signing of our assemblies.

Description

Benchmarks only support .NET 5 and need to print data for all supported platforms. In #62 we identified a memory leak in .NET Framework that was pre-existing and it is tracked in #65. With proper benchmarking metrics this could have been caught much earlier in the release cycle

Task

• Add benchmarking for .NET Framework 4.8
• Add benchmarking for .NET 6

Description

Add native code debugging to simplify debugging between the interop of managed code and native code. This will allow a developer to step into any of the c++ code or 3rd party libraries.

Tasks

• Enable native code debugging on sample apps
• Update all 3rd party libraries to compile on the selected target
  • Excludes any libs from vcpkg

Description

The x265 dependency is only used for encoding and at this point FileOnQ.Imaging.Heif is only interested in decoding. That library is licensed under GPL and is not compatible with LGPL. In the future if we want to add support for encoding, we will need to revisit the license and how we can include it.

Task

Remove x265 from the library and docs

Description

The build system requires visual studio 2019 enterprise but should work with pro and community. Let's update the build process to fall back to other versions if enterprise is not installed

Updated Description

By: @ahoefling

The libjpef-turbo dependency is being compiled in debug mode which uses the Visual C Runtime, this makes the entire library dependent on debug binaries that are NOT included in the Visual C++ Redistributable package. We need to ensure all projects in this library are compiled in release mode.

Original Post

The Library is throwing a DLL not found error. This appears to be due to SetDLLDirectory not working correctly. We should switch over to using AddDLLDirectory instead.

Description

When using multiple FileOnQ imaging libraries in the same downstream project there are naming conflicts with build targets for net48. This is because both FileOnQ.Imaging.Heif and FileOnQ.Imaging.Raw use the same build <Target> name of CopyNativeDlls.

The conflict prevents native assemblies from being copied to the bin directory correctly. This will create a silent error as the build will not fail and the application will have a runtime error.

Proposed Solution

Make the build target names unique

Description

Compile project for use in Web Assembly (WASM) projects. We should test this with Blazor and Uno Platform WASM

Description

Add automated release notes to be generated as part of the git release tag build from #31. Using github-release-notes we will add all merged pull requests and links to the release notes. At this point in time we aren't as organized with issues so we only care about merged pull requests

Description

Currently the library only contains a console application for testing. This is great for developers. However to support non-developer testers, we need to have two WinForms apps for testing purposes. One targeting .NET 4.8 and another targeting .NET 5.0.

Tasks

• Create .NET 4.8 App
• Create .NET 5.0 App

Description

The windows binaries depend on the Visual C++ Redistributable being installed and the hosted GitHub Runner's come pre-installed with a full Visual Studio development environment. That environment has the redistributable installed as well as debug binaries. When running integration tests we may get false positives as the binaries will execute without issues but when used in a downstream project on another machine without visual studio it will fail.

This was first identified as part of #37

Solution

We need to implement an integration build action that will run on a clean VM of windows and complete the following steps

1. Install latest redistributable
2. Run tests

Description

The DLLs generated are not marked as deterministic when viewing in the NuGet Package Explorer. We need to update the build so they are deterministic.

https://nuget.info/packages/FileOnQ.Imaging.Heif/1.0.0