thunderstore-io / thunderstore-cli Goto Github PK

Currently the configuration system is rather empty compared to what it could be.

Ideally most (if not all) options that can be configured via the thunderstore.tm could also be provided via environment variables or CLI arguments. Understandably some complex datatypes (such as build mappings) might be difficult to support nicely via environment variables or CLI arguments, so some thought is required in how this is done in practice.

Moreover, the user-scoped configuration file which should hold cross-project configurations (such as auth tokens) is currently entirely implemented.

This issue exist as a placeholder for more detailed specifications, mostly as a reminder that the configuration system still needs work.

Automate the build & release process via GitHub pipelines

After staring a game with tcli run, the tcli process will orphan the game process upon termination.

Expected Behavior

After running a game with tcli run, terminating the tcli process (e.g. Ctrl+C, SIGTERM/SIGKILL) should also terminate the game process.

Current Behavior

The game process remains running as an orphan process after the tcli run process is terminated.

Steps to Reproduce

1. Start a managed game with tcli e.g. tcli.exe run {game identifier}
2. Verify both the tcli and game process are running:

$ ps o pid,ppid,pgid,sid,start,time,cmd --forest
   PID    PPID    PGID    SID  STARTED     TIME CMD
142557  142556  142557  142557 00:17:29 00:00:00 -bash
142913  142557  142913  142557 00:34:16 00:00:00  \_ tcli run -- -name ModdedTest ..
142929  142913  142913  142557 00:34:17 00:00:02      \_ /home/valheim/valheim/server/valheim_server.x86_64 -name ModdedTest ...

3. Terminate the tcli process (e.g. pressing Ctrl+C in a shell or sending SIGTERM/SIGKILL with kill)
4. Verify the tcli process is terminated, but the game process is running and orphaned:

$ ps o pid,ppid,pgid,sid,start,time,cmd --forest
    PID    PPID    PGID     SID  STARTED     TIME CMD
 142929       1  142913  142557 00:34:17 00:02:45 /home/valheim/valheim/server/valheim_server.x86_64 -name ModdedTest ...

Workaround

Manually terminate the game process with kill e.g. kill 142929 in example above.

Context (Environment)

Running tcli 0.2.0 with tcli-bepinex-installer 0.1.0 (compiled with an older glibc version) on Debian GNU/Linux 11 (bullseye)

Hey, I made some minor changes to the docs to clear some things up about the toml schema:

https://github.com/Lordfirespeed/thunderstore-cli/wiki

View my changes commit-by-commit here

You can pull my changes locally, then update this repository:

git clone https://github.com/thunderstore-io/thunderstore-cli.wiki.git
cd thunderstore-cli.wiki
git remote add downstream https://github.com/Lordfirespeed/thunderstore-cli.wiki.git
git pull downstream master
git push origin master

Implement releases via winget

https://docs.microsoft.com/en-us/windows/package-manager/winget/

As a kind of stopgap for monorepo support, in order to make it easy for developers to target their own packages when uploading them in the correct order, TCLI would automatically grab the latest version from Thunderstore and use that version in the generated manifest. Suggested by @xiaoxiao921, or iDeathHD#0151 on Discord.

These lines in the project file make building and running from source on Linux a pain, could we possibly move these to a specific configuration target like Release or move them entirely to being a part of the publish command in the Github Action? #40 also has to work around these options on the command line, so it doesn't make much sense that they are a part of the global configuration.

Full stack:
`~/tcli/tcli-0.2.3-linux-x64$ ./tcli install --profile 2024-06-05 valheim NOI_or_Die-NOI_or_Die-2.0.5

Downloading main package: NOI_or_Die-NOI_or_Die

Unhandled exception. Newtonsoft.Json.JsonReaderException: JSON integer 3909573219 is too large or small for an Int32. Path '[59].versions[0].file_size', line 1, position 2921397.
at Newtonsoft.Json.JsonTextReader.ParseReadNumber(ReadType readType, Char firstChar, Int32 initialPosition)
at Newtonsoft.Json.JsonTextReader.ReadNumberValue(ReadType readType)
at Newtonsoft.Json.JsonTextReader.ReadAsInt32()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at ThunderstoreCLI.Utils.ModDependencyTree.Generate(Config config, HttpClient http, PackageManifestV1 root, String sourceCommunity, Boolean useExactVersions)
at ThunderstoreCLI.Commands.InstallCommand.InstallZip(Config config, HttpClient http, GameDefinition game, ModProfile profile, String zipPath, String backupNamespace, String sourceCommunity, Boolean isModpack)
at ThunderstoreCLI.Commands.InstallCommand.InstallFromRepository(Config config, HttpClient http, GameDefinition game, ModProfile profile, Match packageMatch)
at ThunderstoreCLI.Commands.InstallCommand.Run(Config config)
at ThunderstoreCLI.ModManagementOptions.Execute()
at ThunderstoreCLI.Program.HandleParsed(BaseOptions parsed)
at ThunderstoreCLI.Program.Main(String[] args)
Aborted (core dumped)`

Trying to publish a mod with the categories ["mods", "serverside", "clientside", "tools"] and the communities ["lethal-company"] and getting the error

ERROR: Unexpected response from the server while publishing package:
Status code: 400 BadRequest
{"categories":{"1":["Object not found"],"2":["Object not found"]}}
Exiting

This happens on 0.2.1, 0.2.2 and 0.2.3. Not sure if i am just using the incorrect categories or if there is something that is going wrong. I tried looking through the code but i am unfamiliar with where i would find anything.

Can only upload readme file it seems currently

Automate testing our builds with virustotal before releasing as to not accidentally publish releases that are flagged as malware.

Projects should be able to specify other project files as dependencies. This would greatly simplify some cases where packages need several subpackages that should be uploaded but are part of the same project eg a in-house/in-repo dependency or metapackages (like R2API is becoming).

When publishing, dependencies should be published before their dependents if the version number that exists in the repository is lower than the one that exists locally.

TCLI gives you a URL after successfully uploading but this link doesn't include the community subdomain. Uploading to the 'test' community on thunderstore.dev gave me this:
https://thunderstore.dev/package/download/northstar/NorthstarLauncher/1.5.1/
instead of:
https://test.thunderstore.dev/package/download/northstar/NorthstarLauncher/1.5.1/

Terminal output:

Using tcli 0.1.4

Currently the SpinnerException is thrown in a way that hides the actual task errors that occurred. Make changes to ensure no errors are hidden, or that at least relevant debug information is available if need be.

Context: #34 (comment)

Currently we're setting many fields to defaults that read weirdly in potential error messages. For example, if an author name isn't defined, tcli will use AuthorName, and the user will see errors about author named AuthorName not being found.

It could be more clear if defaults that should be set were more verbose, e.g. UnsetAuthorPlaceholder or such.

TCLI should be extremely extensible in order to allow communities to expand and utilize it to their needs, and so that if needed, a community can add whatever functionality is needed without having to resort to PRing TCLI itself.

Some of the things that allow for extension:

• Mod managers, which allow for custom install/run logic for different loaders
• Subcommands, which would allow new/niche functionality to be added without feature creep on the core functionality
• Package build logic? Unsure if this would allow for anything more than what pre/post build events would allow
• Possibly just include C# modding libraries like MonoMod/HarmonyX in addition to the other methods? This would allow for basically everything, but I'm unsure if that should be "officially supported" or not, since this could lead to "I was relying on that internal functionality!" hell

Other than that basic functionality, I haven't really decided at all how these should be implemented/what they should be required to provide, so I'm very much open to suggestion here

Implement releases as .msi packages

TCLI should be able to run set commands before or after building the ZIP, like running dotnet build -c Release before building to ensure the ZIP is built from the current source code

Implement releases via Scoop

https://scoop.sh/

PR #33 added an .editorconfig file which suppresses warnings about reflection usage, as our JSON handling required the use of reflection.

Assuming #37 is done, we should be able to remove the warning suppression, or at least re-evaluate it's necessity.

I'm trying to move to tcli to build and publish my package. The generated thunderstore.toml file when I run tcli init contains an inline table for declaring dependencies like this:

dependencies = { AuthorName-PackageName = "0.0.1" }

Inline tables do not allow new lines so declaring my dependent mods would result in a super long line which is horrible for version controlling:

dependencies = { Atopy-QuickPing = "1.5.3", Azumatt-AAA_Crafting = "1.0.5", Azumatt-AzuClock = "1.0.0", ComfyMods-ComfyQuickSlots = "1.0.9", denikson-BepInExPack_Valheim = "5.4.1901", Digitalroot-Digitalroots_Slope_Combat_Assistance = "2.0.2", Goldenrevolver-Quick_Stack_Store_Sort_Trash_Restock = "1.2.0", MathiasDecrock-Snap_Points_Made_Easy = "1.3.2", OdinPlus-OdinHorse = "1.1.0", OdinPlus-OdinsCraftyBoxes = "1.0.6", Smoothbrain-Farming = "2.1.3", Smoothbrain-Lumberjacking = "1.0.3", Smoothbrain-Mining = "1.1.3", Smoothbrain-Ranching = "1.1.1", Smoothbrain-Sailing = "1.1.4", thedefside-BetterUI_Reforged = "2.4.0" }

Making a proper table out of the inline table does not work though. AFAIK it should look like this:

[dependencies]
Atopy-QuickPing = "1.5.3"
Azumatt-AAA_Crafting = "1.0.5"
Azumatt-AzuClock = "1.0.0"
ComfyMods-ComfyQuickSlots = "1.0.9"
denikson-BepInExPack_Valheim = "5.4.1901"
Digitalroot-Digitalroots_Slope_Combat_Assistance = "2.0.2"
Goldenrevolver-Quick_Stack_Store_Sort_Trash_Restock = "1.2.0"
MathiasDecrock-Snap_Points_Made_Easy = "1.3.2"
OdinPlus-OdinHorse = "1.1.0"
OdinPlus-OdinsCraftyBoxes = "1.0.6"
Smoothbrain-Farming = "2.1.3"
Smoothbrain-Lumberjacking = "1.0.3"
Smoothbrain-Mining = "1.1.3"
Smoothbrain-Ranching = "1.1.1"
Smoothbrain-Sailing = "1.1.4"
thedefside-BetterUI_Reforged = "2.4.0"

But this results in a manifest file that looks like this:

So either I'm doing something wrong or this functionality is bugged.

Might be an Ubuntu-specific issue? Haven't reproduced yet, will need some testing

https://github.com/DanielHeEGG/SmartEjectors/runs/4999301884?check_suite_focus=true

Following the Example Linux Valheim Server Setup Workflow guide, if I use a custom --tcli-directory with tcli run, mods are not loaded.

Workaround: Create a symlink from ~/.config/ThunderstoreCLI to your ${TCLI_DIRECTORY}.

Related: #94

Just realised that the latest commit (providing support for community categories) didn't ever get a release.

It would be great if a tag could get created for that so the release workflow gets triggered!

Implement releases via Chocolatey

https://chocolatey.org/

I get the following error when I try and publish a package. TCLI v0.2.1

ERROR: Unexpected response from the server while publishing package: Status code: 400 BadRequest {"categories":{"2":["Object not found"],"4":["Object not found"]}} Exiting

This is the content of the thunderstore.toml file.

  icon = "./icon.png"
  outdir = "./build"
  readme = "./README.md"
  
  [[build.copy]]
  source = "./dist"
  target = ""
  
  [config]
  schemaVersion = "0.0.1"
  
  [package]
  containsNsfwContent = false
  description = "Make the Wisp do a better job."
  name = "Better_Wisps"
  namespace = "Digitalroot"
  versionNumber = "1.0.12"
  websiteUrl = "https://github.com/Digitalroot-Valheim/Digitalroot.Valheim.BetterWisps"
  
  [package.dependencies]
  ValheimModding-Jotunn = "2.11.5"
  denikson-BepInExPack_Valheim = "5.4.2105"
  
  [publish]
  categories = ["server-side", "client-side", "mistlands-update", "tweaks", "gear"]
  communities = ["valheim"]
  repository = "https://thunderstore.io/"

The two causing the error are "mistlands-update" and "gear". Both should be usable and are listed at https://thunderstore.io/api/experimental/community/valheim/category/

Removing them allowed the file to be published as expected. I am not using TCLI to build the package, I am providing TCLI a pre-built zip file.

Tracking issue for TCLI's mod installation capabilities

All of this is on the develop branch until further notice for quick iteration

Currently, this is the checklist of things needed to make this even remotely workable as a shipped product:

• A reorganization of the Options/Configs structure, expects a thunderstore.toml file to always be present (done with f7012ee and b484ed8)
• Location for profiles/configs, currently %APPDATA%/ThunderstoreCLI/ on Windows (I believe) and ~/.config/ThunderstoreCLI/ on Linux
• General structure for game/profile configuration (https://github.com/thunderstore-io/thunderstore-cli/blob/develop/ThunderstoreCLI/Game/GameDefinition.cs and https://github.com/thunderstore-io/thunderstore-cli/blob/develop/ThunderstoreCLI/Game/ModProfile.cs)
• "Mod manager" interface contract (https://github.com/thunderstore-io/thunderstore-cli/blob/develop/ThunderstoreCLI/Game/ModManager.cs)
• install subcommand interface and implementation
• Example mod manager implementation for BepInEx
• Mod installation from repository (will most likely require manifest v1 to be workable with current structure, otherwise will need rewrites, therefore thunderstore-io/Thunderstore#93 is blocking here)
  • Automatic game recognition, ideally would use a new Thunderstore API instead of being hardcoded like r2mm/tmm
• Plugin system for additional mod managers, see #58
• run subcommand interface and implementation

Only really considering human readable and JSON at the moment, but all commands should be able to take inputs from and give outputs in all formats with a --format <format name> switch

There's some TOML fields that aren't being exposed by the CLI, this makes using tcli on CI a little more difficult since you have to use sed to change these values after the file is generated.

• Package description
• Communities
• website url
• dependencies

Pre-validate the required package metadata files before uploading:

• README.md
• manifest.json
• icon.png

There's an API endpoint available for validating each of these, see https://thunderstore.io/api/docs/ under experimental/submission/validate

Allowing the user to do something like

versionNumber = { file = "SourceFile.cs", regex = "const string VERSION = \\\"(.+)\\\";" }

So that a version number would only need to be updated in one place

Is this something I should add? The exact way this would look can change, but would this sort of option be wanted?

.NET 6 supports new JSON parsers which are generated at compile time instead of handled via reflection.

https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-source-generator/

Replace our JSON handling with these.

Context in PR description and comments here: #33 (comment)

When no dependencies are specified in the thunderstore.toml file, I get this:

ERROR: Unexpected response from the server while publishing package:
Status code: 400 BadRequest
{"file":["manifest.json dependencies 0: No matching package found for reference: AuthorName-PackageName-0.0.1"]}

Implement releases via Squirrel

https://github.com/Squirrel