microsoft / winget-cli-restsource Goto Github PK
View Code? Open in Web Editor NEWThis project aims to provide a reference implementation for creating a REST based package source for the winget client.
License: MIT License
This project aims to provide a reference implementation for creating a REST based package source for the winget client.
License: MIT License
Windows Package Manager 1.2 Schema
Related to:
No response
// Create feed options for inclusion search: -1 so we can get all matches in inclusion, then filter down.
FeedOptions feedOptions = new FeedOptions
{
ResponseContinuationTokenLimitInKb = CosmosConnectionConstants.ResponseContinuationTokenLimitInKb,
EnableCrossPartitionQuery = true,
MaxItemCount = AllElements,
RequestContinuation = null,
};
if (manifestSearchRequest.FetchAllManifests || (manifestSearchRequest.Inclusions == null && manifestSearchRequest.Query == null))
{
IQueryable<CosmosPackageManifest> query = this.cosmosDatabase.GetIQueryable<CosmosPackageManifest>(feedOptions);
IDocumentQuery<CosmosPackageManifest> documentQuery = query.AsDocumentQuery();
ApiDataPage<CosmosPackageManifest> apiDataDocument = await this.cosmosDatabase.GetByDocumentQuery<CosmosPackageManifest>(documentQuery);
manifests.AddRange(apiDataDocument.Items);
}
Filter only searches will result in attempting to load the entire cosmos db database into azure function memory. Winget client uses filter only search for a variety of user command line input.
This implementation should be revised to not attempt to load the entire cosmos db database into memory.
The maximum number of installer nodes should be increased. Firefox has 231. Firefox has 291.
Increase the maximum number of installer nodes to 256 512 1024.
This needs to be implemented in the:
In the released Microsoft.Wnget.Source module is contained in a.ZIP file with an internal folder name "Winget.RestSource-WInget.PowerShell.Source. But that is NOt the name of the module and this makes it difficult to automate the download and installation.
The file downloaded is WinGet.RestSource-Winget.PowerShell.Source.zip
and inside that is a folder WinGet.RestSource-Winget.PowerShell.Source
. But the module name is Microsoft.Winget.Source
. In PowerShell, the module name (ie in the .PSD1 and .PSM1 files must match their parent folder name to facilitate autoloading.
For any module you folks release, have the .ZIP file, and the folder inside that Zip file be consistent with the module names in the PSD1/PSM1 files.
Download the released zip and look at the names inside the file
I would like the file name to be consistent with the folder name inside the zip file and that consistent with the module.
Given the module "Microsoft.WInget.Source|" the zip file should be Microsoft.WinGet.Source.zip and contain a folder named Microsoft.WinGet
see above
PSH [DB:\]: winget --info
Windows Package Manager (Preview) v1.2.3131-preview
Copyright (c) Microsoft Corporation. All rights reserved.
Windows: Windows.Desktop v10.0.22000.318
Package: Microsoft.DesktopAppInstaller v1.17.3131.0
Logs: %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir
Links
---------------------------------------------------------------------------
Privacy Statement https://aka.ms/winget-privacy
Licence Agreement https://aka.ms/winget-license
Third Party Notices https://aka.ms/winget-3rdPartyNotice
Homepage https://aka.ms/winget
Windows Store Terms https://www.microsoft.com/en-us/storedocs/terms-of-sale
Some REST sources may wish to provide custom behaviors based on input from the Windows Package Manager client.
Initial thinking is a max length of 1024 characters.
The client would ideally support a command-line parameter as well as a setting to specify the value to be sent to the REST source.
A separate Issue will be created in the winget-cli repository to support the client work for this feature.
The header name is: "Windows-Package-Manager".
Some packages require some form of "entitlement" to install a particular package. When a user is interacting with a source, some apps may require purchase or parental consent. The canonical example is the Microsoft Store. If the user is not authenticated or does not posses an entitlement for the App, the user should be provided a Purchase URL so they can acquire the entitlement. In the case of the Microsoft Store this could be a deep link to the store where the user can authenticate, purchase, or request parental consent.
Client Issue - microsoft/winget-cli#1345
[Placeholder]
Related to:
Note:
This is pending a specification documentation.
The Microsoft.WinGetSource PowerShell module currently connects using the AZ PowerShell module to validate resources exist and to retrieve connection information.
The module should also support connecting to the URLs regardless of the infrastructure the REST source is deployed to. This issue will add the ability to target the URL to manage manifests.
The user would need to be able to provide the URL, and the secret(s) necessary to work with the management REST endpoints.
No response
The current reference implementation does not support Normalized package name and publisher search, which is used for correlating packages with ARP entries. The support for this should be added for better correlation of packages.
The Rest schema only supports search for a single KeyWord in Request Match, but NormalizedPackagenameAndPublisher field requires two keywords, one for PackageName and another one for Publisher.
Another alternative would be to deprecate NormalizedPackagenameAndPublisher in favor of two fields NormalizedPackageName and NormalizedPublisher. A new match type can also be added to support normalized search as well instead of these.
I would like to be able to leverage certificate-based authentication for my REST source.
Related to:
No response
We should have a high level description of the project in the README including some steps for deploying the project.
Three configurations are supported in the New-WinGetSource.
Currently, only the Demo configuration uses the free tier of CosmosDB.
Note: Azure restricts a single instance of the CosmostDB free tier per Azure Subscription.
This would extend selecting the CosmostDB free tier to Basic, and Enhanced configurations.
No response
The Azure front door resources provide customers the ability to enable firewall services, caching, and other scaling behaviors.
The New-WinGetSource should be able to support configurations using the Azure front door services.
No response
Arch: x86
InstallerType: MSStore
ProductId: 9N0CTDPL6ZC3
PackageFamilyName: 10011IRXON.atBlue_96p6fcx5kk12j
A rest source needs to be able to return a store installer type with a product id.
Create code owners file
No response
We should figure out a maximum line length that we'd like to adhere to and standardize that in the repo. Ideally this would utilize tooling to refactor the code appropriately, but it may just have to involve manual changes.
We should also document this in the contributing section.
Several different forms of authentication have been proposed for accessing private sources. This issue is a "top level" issue to clarify which forms of authorization are being considered.
We will also need to expose the supported authentication mechanisms via the GetInformation endpoint.
No response
microsoft/winget-cli#970 Some packages are installed based on the location of the installer on the drive. This can be an undesired behavior, and may be further complicated by packages with long paths like Python as discovered in microsoft/winget-pkgs#13076.
Add files for community profile.
First off thank you to the team working on this project and making it open source for community contributions very early into it's development. With the announcement that the windows store ecosystem will be sunset in the first quarter of 2023 I'm sure there are many organizations out there relieved to see a project like this in the works to serve as an alternative.
Initial deployment of the private repository was a little tricky, but I fully understand this is a work in progress and the process/documentation will improve over time. Once the API was deployed everything seems to work very well out of the box (and can definitely see the appeal of how easy it is to pull software down onto machines with Winget).
Now were trying to imagine how this private repository could potentially be used as a replacement for the Windows Store LOB distribution channels. A key feature of the store LOB distribution is the ability for publishers to have a single catalogue of software packages, but control individual package assignments to their partners. For example if the publishers catalog has 3 software packages in it total, all 3 packages might be assigned/available to one connected partner, but another partner may only have access to 1 or 2 of those titles out of the catalogue.
Testing with the private repository we can see its simple enough to control access to the api endpoint itself and the underlying storage for each package installer. Where there is uncertainty right now is simply what is the best practice/recommended way to try and achieve this (or if this is a bad idea to try entirely?). We have been working with @mjfusa from the app consult team at Microsoft to try and plan our impending transition, and it was suggested to post our situation here as this team has the most expertise with private repositories and would have a better idea of any development plans in the coming year.
Currently were able to think of 2 (potential?) approaches to take and trying to work out if either would be manageable.
Each partner deploys their own private API endpoint and controls access to their users how they see fit. The publisher maintains a separate storage account for each partner, and a master storage account containing all of their package installation media. To assign a package to a specific partner, the package is synced from the master storage to the partners storage account and a manifest is pushed to the partners private repository with reference to the installer location in their specific storage location.
pros
cons
The publisher creates a single private API endpoint (all partners have access), storage accounts for each partner, and a master storage account containing all installers. To assign a package to a specific partner, the installer is synced to that partners storage account, and a unique package manifest is added to the repository with reference to it.
pros
cons
Is there something we are totally looking over right now that could be done at the individual manifest level, or somewhere else to easier control access? For example maybe having redundant installation locations in a manifest which are partner specific?
Would be grateful for any suggestions or advice you can provide in this area. Unfortunately at the moment things are largely undocumented so just want to try and ensure we are not going down the wrong path in the coming months. As the deadline for the store going away gets closer, the urgency to find an alternative will definitely keep ramping up.
Thank you again!
I would like to be able to search by publisher.
No response
On france region, when i try to use New-WinGetSource.ps1 Cosmos DB creation fail always.
I found that issue is on cosmosdb.json ARM template. The parameter "name" is not correct and must be "cosmosName" instead.
When, changed, New-ARMParameterObject.ps1 must be changed also to have the good name
use the following cmd line
New-WinGetSource -Name "contoso" -ResourceGroup "WinGetPrivateSource" -Region "francecentral" -ImplementationPerformance "Demo" -ShowConnectionInstructions
Cosmos DB ARM template validation is OK and must be created without error
Cosmos DB ARM template validation fail
Windows Package Manager REST Source Reference Implementation 1.1.20211101
Related to:
.
.
.
.
Related to:
Schema will be modified to support the installation of portable / stand-alone executables.
Inclusion of an installer type (portable) validation that will limit the number of Command elements to a single entry.
Note:
Waiting for specification to be published at WinGet-CLI to finalize requirements.
I noticed an error about missing dll on the Azure function app when I published from visual studio. The current azure function targets .NET 5.0. After switching to .NET Core 3.1 and re-publishing it, the function app worked fine.
Need to investigate if azure function app is behaving as expected with .NET 5.0.
when Cloning the repo "Git failed with fatal error. the remote end hung unexpectedly" Clone succeeded, but checkout failed. the remote end hung up unexpectedly
Use clone from Repo on VS 19 or VS 22
The repo is to be cloned and loaded into a solution within Visual Studio.
No solution, error message shows when Cloning the repo "Git failed with fatal error. the remote end hung unexpectedly" Clone succeeded, but checkout failed. the remote end hung up unexpectedly
Windows 10 10.0.19043 build 19043
Windows 11 10.0.22000 build 22000
Visual studio v19 & V22 (tested with)
Today we have no choice on function part, that use Windows application.
Could it be possible to specify a parameter to use linux function app instead of Windows one
Have an set of ARM template for app plan and azure functions implemeting Linux way in WinGet.RestSource.Infrastructure
In New-WinGetSource.ps1, give the possibility to give a parameter specifying Linux or Windows
In New-ARMParameterObject.ps1 , based on parameter given, create and fill the correct json file
in New-ARMObjects.ps1, based on the parameter, use the correct ARM template and the previously created JSon file with parameters
Azure Resource Manager (ARM) templates are currently provided in the release artifacts.
This work item is to enable a user to create or export ARM parameter files to customize the deployment configuration.
Examples include:
Max IO - for CosmosDB
Azure SKU selection - for Azure resources
Resource name customization
The New-WinGetSource cmdlet should also be extended to support using these ARM parameter files.
No response
The build pipeline should automatically publish the artifacts to the Release draft, and should include the SHA256 values for the assets.
No response
The manifest schema restriction was relaxed from 2000 characters to 2048 for the HTTP max length.
This change needs to be made to the REST API as well.
This is the PR from winget-cli:
microsoft/winget-cli#928
"WEBSITE_LOAD_CERTIFICATES": "[parameters('website_load_certificates')]",
is settable by a parameter to the arm template. The setting is not actually used by the reference though, so we should remove non-meaningful options from the sample to minimize complexity.
At main readme.md, under subheading "Automatically create a rest source" the link to powershell New-WinGetSource
is broken.
https://github.com/microsoft/winget-cli-restsource#automatically-create-a-rest-source
As im checking this out there does not seem any information on hosting your own local repository of packages. This information could be usefull for companies or users who have a lot of their own builds and might even use their own private applications.
I am also asking this because there is option to add aditional sources of packages to winget command line app. So i went on the hunt for that info right away.
thanks for looking at my issue.
To maintain consistency with semantic validation of yaml manifests, installer type must be a required field.
When running New-WinGetSource I receive the following error:
Write-Error: C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\New-WinGetSource.ps1:157
Line |
157 | New-ARMObjects -ARMObjects $ARMObjects -RestSourcePath $RestS …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Failed to create Azure object.
Name Value
---- -----
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName winget2test
ParameterPath C:\projects\Parameters\applicationinsights.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\applicationinsights.json
ObjectType AppInsight
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName winget2test
ParameterPath C:\projects\Parameters\keyvault.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\keyvault.json
ObjectType Keyvault
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName winget2test
ParameterPath C:\projects\Parameters\storageaccount.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\storageaccount.json
ObjectType StorageAccount
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName winget2test
ParameterPath C:\projects\Parameters\asp.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\asp.json
ObjectType asp
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName winget2test
ParameterPath C:\projects\Parameters\cosmosdb.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\cosmosdb.json
ObjectType CosmosDBAccount
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName WinGet
ParameterPath C:\projects\Parameters\cosmosdb-sql.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\cosmosdb-sql.json
ObjectType CosmosDBDatabase
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName Manifests
ParameterPath C:\projects\Parameters\cosmosdb-sql-container.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\cosmosdb-sql-container.json
ObjectType CosmosDBContainer
Parameters {contentVersion, Parameters, $Schema}
Error
ObjectName winget2test
ParameterPath C:\projects\Parameters\azurefunction.json
TemplatePath C:\Users\dahov\Downloads\WinGet.RestSource-Winget.PowerShell.Source\WinGet.RestSource-Winget.PowerShell.Source\Library\ARMTemplate\azurefunction.json
ObjectType Function
Following instructions here. Running below command (some values changed for security):
New-WinGetSource -Name winget2test -ResourceGroup rg-vnet01-wus2 -SubscriptionName "Test" -Region westus2
More meaningful error should be provided or resources should be created as expected.
Non-descript error received.
Windows Package Manager v1.1.12986
Copyright (c) Microsoft Corporation. All rights reserved.
Windows: Windows.Desktop v10.0.22000.318
Package: Microsoft.DesktopAppInstaller v1.16.12986.0
The Intune integration with the Windows Package Manager will utilize the REST API for package "search".
The source will likely be restricted to certificate-based authentication so the initial implementation will not support anonymous connections.
This issue is just being created in the spirit of transparency to show other work the Windows Package Manager team is doing.
Related to:
No response
As a part of #34 I mentioned having the release artifacts published so the deploy process can be made even more simple. It has a dependency on the .zip file produced when building the project.
Create a build pipeline and publish a release 😊.
We want to be able to support a REST source being hosted on infrastructure other than just Azure.
This would not impact the reference implementation, but additional improvements need to be made to have the PowerShell cmdlets communicate with any REST implementation. Note: the New-WinGetSource will continue to support only Azure, but the other cmdlets will be using the "management" REST endpoints and not connecting with Azure resources directly.
No response
Hello everyone. I wanted to let you all know as we enter into the holidays, there will be periods of less activity. Many of the Microsoft employees working on the project have substantial amounts of paid time off subject to loss if they are not taken this year. Many of the areas of focus during this time are in support of engineering improvements to help make the team more efficient when troubleshooting issues, or spending time "on-call" to support various aspects of the services and integration between GitHub and Azure DevOps.
NUGET repo Error occurred while restoring NuGet packages: Could not find a part of the path.
after manually downloading the ZIP as there is an issue with cloning. when restoring the NuGET i get the following:
NuGet repo Error occurred while restoring NuGet packages: Could not find a part of the path.
Restore should work
error
NUGET repo Error occurred while restoring NuGet packages: Could not find a part of the path.
Windows 10/11
Visual studio 19 / 22
This was probably an expected ask... will there be support for client authentication/authorization?
Win32 allows max protocol length of 2048. Do not limit to 39 characters.
I just hit upon something in the REST api that I believe we should consider changing at the schema/interface level. Some of the APIs return different result types depending on the count of items returned, for example:
return packages.Items.Count switch
{
0 => new NoContentResult(),
1 => new ApiObjectResult(new ApiResponse<Package>(packages.Items.First(), packages.ContinuationToken)),
_ => new ApiObjectResult(new ApiResponse<List<Package>>(packages.Items.ToList(), packages.ContinuationToken)),
};
Either No Content (null), a single object, or an array. For example:
This returns an array: .../api/packages/Microsoft.PowerToys/versions
{"Data":[{...},{...},,…],"ContinuationToken":"…"}
This returns a single object: .../api/packages/Zwift.Zwift/versions
{"Data":{...}}
This returns 204 (null): .../api/packages/Foo.Bar/versions
If the query returns more than one result, I get an object where Data is an array (and there’s a continuationToken property), but if there’s only one result, then Data is an object. This is true for the packages, packageManifests, locales, installers and versions APIs.
IMO this is a bad practice and requires clients to try to parse the result object twice in a catch block to successfully deserialize it. Instead we should always consistently return the same object type, potentially with only one item in the array, and a null continuationToken.
Call above APIs
Response type should be consistent regardless of # of results
Response type varies based on number of results
N/A
There are open compliance tasks that need to be reviewed for your winget-cli-restsource repo.
To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your microsoft GitHub organization.
Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/winget-cli-restsource/compliance
You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.
If you no longer need this repository, it might be quickest to delete the repo, too.
More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]
FYI: current admins at Microsoft include @msftrubengu, @hackean-msft, @JohnMcPMS, @kewike-msft, @mapill-msft, @ranm-msft, @denelon, @zachcarp, @luiscalv, @KevinLaMS, @jamespik, @ashpatil-msft, @yao-msft
Several endpoints (versions, installers, locales) allow for continuation tokens but do not honor them. These functions also do not honor max item limits on return pages.
The Microsoft Store requires Microsoft Account (MSA) authentication to acquire previously purchased Apps. The Microsoft Store source is implemented via the REST API.
Running winget install Paint.Net
is the canonical example.
The source GET information API should provide the support for MSA authentication to prompt users for an ACK to use MSA authentication for acquiring paid applications. Once this has been accepted by the user, the client should pass an authentication token to the store source on subsequent requests.
Note: This may also be a viable authentication mode for other private sources. This is not a duplicate of #37 looking at other authentication mechanisms.
The headers in the .cs files should correctly reflect the MIT license for this project.
The Windows Package Manager Manifest Creator generates multi-file YAML manifests for individual versions of a package. This enhancement would allow the cmdlets to look at the child folders to generate a single (JSON) manifest with multiple versions per package when targeting the "package" directory.
This would also work when targeting a logical "publisher" folder to generate multiple (JSON) manifests with multiple versions per package.
No response
Search endpoint implementation reruns entire search query and then tries to match indexes across different query results. If the backend was updated or the the data order changed in any way, the caller may see broken results when using continuation tokens. Additionally, this approach is cost inefficient and relatively slow.
A more common implementation pattern for continuation tokens would have been to cache/stay consistent with the results of the initial search query and just paginate the results across each continuation request from a client.
The YAML schema updates to v1.1 are tracked in client issue:
microsoft/winget-cli#1243
Implement the changes to support v1.1.0.
Additional enhancements to support the new store source
"Markets"
Get Information - requires Market
Get Package - accepts Market
Search - Filter on Market
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.