google / googet Goto Github PK

GooGet package manager

License: Apache License 2.0

Go 99.07% PowerShell 0.93%

googet's Introduction

GooGet

GooGet (Googet's Obviously Only a Goofy Experimental Title) is a modular package repository solution primarily designed for Windows.

GooGet is shipped with the official Google Cloud Platform Windows images and is used to maintain the guest environment.

Build

To build and package googet run

go run goopack/goopack.go googet.goospec

This will result in googet.exe and googet.x86_64.VERSION.goo which can be installed on a machine with the googet install command (assuming googet is already installed).

To install on a fresh machine copy both googet.exe and the googet package over and run:

googet -root 'c:/ProgramData/GooGet' install googet googet.x86_64.VERSION.goo

Conf file

GooGet has the ability to use a conf file to change a few of the default settings. Place a file named googet.conf in the googet root, which by default is C:\ProgramData\GooGet and configurable by the -root flag.

proxyserver: http://address_to_proxy:port
archs: [noarch, x86_64]
cachelife: 10m

Repo file

GooGet has the ability to use a repo file to change some repo specific settings. The file is named your-repo-name.repo by default. It can be used to set the source URL and indicate that the client should pass authorization headers in requests to the source URL.

URL: https://foo.com/googet/bar
  useoauth: true

Google Cloud Storage as a back-end

Googet supports using Google Cloud Storage as its server.

set GOOREPO=%TEMP%\googet-repo
set REPONAME=my_repo
mkdir %GOOREPO%\%REPONAME%
mkdir %GOOREPO%\packages
go run goopack/goopack.go googet.goospec
copy *.goo %GOOREPO%\packages
go run server\gooserve.go -root %GOOREPO% -save_index %GOOREPO%\%REPONAME%\index
gsutil mb --project my-project my-googet-server
gsutil rsync -r %GOOREPO% gs://my-googet-server
googet addrepo gcs gs://my-googet-server

rem This command should print 'gcs: gs://my-googet-server'
googet listrepos

rem This command should list the googet package and any other packages in your repo
googet available -sources gs://my-googet-server/

Note that you must regenerate the index and re-upload it to your bucket each time you add or change packges.

googet's People

Contributors

Stargazers

Watchers

googet's Issues

googet installation failed without internet access

We are using Daisy workflow control with Google cloud build to create Windows images in Google cloud.
After we removed Internet access from our project, cloud build failed with error as blow:
Installing GooGet and GooGet packages.
ERROR: 2023/11/01 22:49:05.756131 client.go:104: error reading repo "https://packages.cloud.google.com/yuck/repos/google-compute-engine-stable": Get "https://packages.cloud.google.com/yuck/repos/google-compute-engine-stable/index.gz": net/http: TLS handshake timeout
ERROR: 2023/11/01 22:49:05.768357 googet_install.go:125: Can't resolve version for package "googet": no versions of package googet found in any repo
GooGet install failed from "D:\ProgramData\GooGet\googet.exe", retrying...
ERROR: 2023/11/01 22:49:36.223287 client.go:104: error reading repo "https://packages.cloud.google.com/yuck/repos/google-compute-engine-stable": Get "https://packages.cloud.google.com/yuck/repos/google-compute-engine-stable/index.gz": net/http: TLS handshake timeout
ERROR: 2023/11/01 22:49:36.242315 googet_install.go:125: Can't resolve version for package "googet": no versions of package googet found in any repo
'C:\ProgramData\GooGet\googet.exe' is not recognized as an internal or external command,
operable program or batch file.

We would like to be able to extract googet packages into local filesystem so that we can install googet offline.

Googet install will fail if driver is already installed

Observed behavior:

Sideload latest vioscsi driver with DISM
VERBOSE: Successfully added driver C:\builder\components\drivers\win81\vioscsi.inf

Install Googet package with vioscsi driver
Running C:\ProgramData\GooGet\googet.exe with arguments -root C:\ProgramData\GooGet -noconfirm install google-compute-engine-driver-vioscsi.
googet_install.go:165: Error installing google-compute-engine-driver-vioscsi.x86_64.16.1.2@11: command exited with error code 1

Expected behavior:

Googet will not error if driver is already installed

googet download ignores return codes

googet's download library, specifically the packageHTTP function, makes no attempt to validate that it received an expected return code (eg 200) from the server prior to saving the body of the response to disk. This can lead to goo files that contain the body of a failure page rather than the goo binary itself. Although the checksum verification may detect a problem in the resulting file before googet tries to use it, the failure mode of a checksum mismatch obscures the underlying problem with the network connection from the user. Ideally, a bad return code should never get written to disk, a more specific error message should be generated, and a corresponding exit code should be returned.

Windows - Installation scripts not successfully executing

I am having an issue with googet on a single server where it is not executing the installation script defined in the pkgspec file. As far as I can tell, it's not successfully calling powershell. If the install script is executed manually, it runs successfully.

I can repeat the issue with the following command (installation package does not matter, experience is the same regardless of package):

googet.exe -verbose -root c:\temp\jcl\Googet install -sources https://packages.cloud.google.com/yuck/repos/google-compute-engine-stable google-osconfig-agent

It will extract and copy all of the files and then log:

INFO : 2021/06/29 15:37:12.722922 system_windows.go:76: Running install command: "packaging/googet/install.ps1"
ERROR: 2021/06/29 15:37:12.729706 googet_install.go:165: Error installing google-osconfig-agent.x86_64.20210607.00.0+win@1: open NUL: The system cannot find the file specified.

Again, powershell works by hand, the installation script is in the cache folder where it was extracted.

I pulled down the latest googet sources and liberally sprinkled some logging statements around the install.go and goolib.go files to see if paths, filenames, etc were being passed successfully. Here's a truncated excerpt from logging statements output from a modified googet version:

INFO : 2021/06/30 12:26:32.621186 system_windows.go:76: Running install command: "packaging/googet/install.ps1"
INFO : 2021/06/30 12:26:32.640718 goolib.go:59: JCL - Cleaned File Path: "c:\\temp\\jcl\\Googet\\cache\\google-osconfig-agent.x86_64.20210607.00.0+win@1\\packaging\\googet\\install.ps1"
INFO : 2021/06/30 12:26:32.656348 goolib.go:61: JCL - scriptInterpreter: "powershell"
INFO : 2021/06/30 12:26:32.668047 goolib.go:70: JCL - Powershell args: ["-ExecutionPolicy" "Bypass" "-NonInteractive" "-NoProfile" "-Command" "c:\\temp\\jcl\\Googet\\cache\\google-osconfig-agent.x86_64.20210607.00.0+win@1\\packaging\\googet\\install.ps1"]
INFO : 2021/06/30 12:26:32.682688 goolib.go:91: JCL - run c: "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe -ExecutionPolicy Bypass -NonInteractive -NoProfile -Command c:\\temp\\jcl\\Googet\\cache\\google-osconfig-agent.x86_64.20210607.00.0+win@1\\packaging\\googet\\install.ps1"
INFO : 2021/06/30 12:26:32.698977 goolib.go:94: JCL - run ok: false
INFO : 2021/06/30 12:26:32.702705 goolib.go:95: JCL - run err: "open NUL: The system cannot find the file specified."
INFO : 2021/06/30 12:26:33.529111 install.go:157: JCL - installPkg: dst: "c:\\temp\\jcl\\Googet\\cache\\[email protected]", rs.PackageSpec: "google-osconfig-agent.x86_64.20210607.00.0+win@1", dbOnly: false
ERROR: 2021/06/30 12:26:33.534309 googet_install.go:165: Error installing google-osconfig-agent.x86_64.20210607.00.0+win@1: open NUL: The system cannot find the file specified.

I strongly suspect that the issue is within the OS and/or its configuration but it only appears that googet and other google services installed on the OS are impacted by this (e.g., OSConfig Agent service is not able to run powershell to determine installed updates). It appears to successfully hand execution to os/exec func Run which then generates the error. I am somewhat at a loss as to how to troubleshoot this further. Any direction you can give would be greatly appreciated.

Architecture: x86_64
KernelRelease: 10.0.17763.1999
KernelVersion: 10.0.17763.1999 (WinBuild.160101.0800)
LongName: Microsoft Windows Server 2019 Datacenter
ShortName: windows
Version: 10.0.17763

$PSVersionTable
Name                           Value
----                           -----
PSVersion                      5.1.17763.1971
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17763.1971
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Caches get mixed up if two repo URLs use the same final path component.

Here's the culprit:

googet/client/client.go

Line 151 in b08ef8d

cf := filepath.Join(cacheDir, filepath.Base(p)+".rs")

Example:
googet addrepo first http://myserver.com/googet/myrepo
googet addrepo second gs://my-bucket/googet/myrepo

This will result in googet using C:\ProgramData\googet\cache\myrepo.rs as the cache for both repos. googet available will list both repos but list the set of packages from whichever repo was read most recently for both of them.

The client has a unique name for each repository ('first' and 'second' in the example above). It should use that instead of the final path component.

Note that the fix is easy and I'm glad to submit a PR. However, the rollout could be tricky due to the potential of mixed-up caches. Doing something like giving the cache file a new extension (.rs2 or something) seems like the best way to avoid that. Please let me know if I should:

a) Please submit a PR!
b) No PR necessary, we'll fix it.
c) Too risky -- Let's leave it as-is and maybe put a warning in the docs about it.

Upgrading in-use files breaks if %TEMP% is not on C:

When updating a package, the client.RemoveOrRename() function first attempts to delete a file, and if that fails (e.g. if it is in use) it instead moves it to a temp file returned by ioutil.TempFile().

This fails if the system has been configured to point %TEMP% / %TMP% to a different drive to the source file.

For example:

googet_install.go:165: Error installing googet.x86_64.1.0.0@1234: rename \\?\C:\ProgramData\GooGet\googet.exe \\?\D:\TEMP\921630347: The system cannot move the file to a different disk drive.

In this case, it is trying to move C:\ProgramData\GooGet\googet.exe to D:\TEMP\921630347 which is unsupported by os.Rename().

This is a known issue, closed as wontfix:
golang/go#13766

Speculation as to how Golang is meant to succeed as a general purpose programming language if it can't even move files across drives in Windows is left as an exercise to the reader.

My suggestion for how to fix is to instead rename the file in-place (in the same directory), and add an entry to the following MULTI_SZ registry value to instruct Windows to delete the file on next reboot:

HKLM\System\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations

There's another alternative that involves directly calling MoveFileEx() with the MOVEFILE_DELAY_UNTIL_REBOOT flag, which avoids having to manipulate a MULTI_SZ structure in the registry.

It goes roughly like this:

(1) Attempt to delete file, which fails
(2) Rename file in-place, to avoid renaming across drives
(3) Rename file to tmp dir with MoveFileEx() syscall with the MOVEFILE_DELAY_UNTIL_REBOOT flag set

Googet on x86 displays AddressWidth exception

Observed Behavior
Built Googet using build.sh on Debian 9, set environment variable GOARCH=386
Saved executable to GCS bucket and DL to local Windows 7 Enterprise x86

.\googet -root 'C:\ProgramData\Googet' installed
ERROR: Logging before logger.Init.
ERROR: 2019/03/07 05:45:37.165601 system_windows.go:179: Error getting AddressWidth: Exception occurred. (Invalid query
)
No packages installed.

Expected Behavior
No error messages displayed.

Message is displayed for all googet commands

Add --extract to googet download params

Allows the end user to download and extract the contents of a Googet package to a local filesystem. Would also enable offline installations.

Running goopack on Windows requires backslashes in package files

Running goopack on Windows does not recognize forward slashes in the files clause of the goospec. Replacing forward slashes with backslashes makes it work. This is due to the glob function returning paths with backslashes (it being Windows).

Repro case:

PS C:\src> mkdir a; mkdir b; $null > a/x; $null > b/y

    Directory: C:\src


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         2/2/2021  11:45 PM                a
d-----         2/2/2021  11:45 PM                b


PS C:\src> @"
>> {
>>   "name": "goopack-test",
>>   "version": "0.0.0",
>>   "arch": "noarch",
>>   "files": {
>>     "a/x": "<ProgramFiles>/GoopackTest/x",
>>     "b/y": "<ProgramFiles>/GoopackTest/y"
>>   },
>>   "sources": [{
>>     "include": [
>>       "a/x",
>>       "b/y"
>>     ]
>>   }]
>> }
>> "@ | Out-File -encoding ASCII goopack-test.goospec
PS C:\src> goopack goopack-test.goospec
2021/02/02 23:59:18 requested files [a/x b/y] not in package
PS C:\src>

Changing the files to use backslashes makes it work:

PS C:\src> @"
>> {
>>   "name": "goopack-test",
>>   "version": "0.0.0",
>>   "arch": "noarch",
>>   "files": {
>>     "a\\x": "<ProgramFiles>/GoopackTest/x",
>>     "b\\y": "<ProgramFiles>/GoopackTest/y"
>>   },
>>   "sources": [{
>>     "include": [
>>       "a/x",
>>       "b/y"
>>     ]
>>   }]
>> }
>> "@ | Out-File -encoding ASCII goopack-test.goospec
PS C:\src> goopack goopack-test.goospec
PS C:\src> dir *.goo


    Directory: C:\src


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----         2/3/2021  12:03 AM            319 goopack-test.noarch.0.0.0.goo                                          


PS C:\src>

Another minor issue is that there is no indication why the forward slashes produced an error. It would be great to have a debug mode where the globbed file set is printed out.
/cc @adjackura