Git Product home page Git Product logo

qusal's Introduction

Dear reader,

You've found me.

I use separate PGP subkeys for different purposes:

  • Code signing key fingerprint: DF3834875B65758713D92E91A475969DE4E371E3
  • Email key fingerprint: CCDD547A4AD5E5A2EA6F093496A515DC1EB9622F

qusal's People

Contributors

ben-grande avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

qusal's Issues

fedora-minimal misses python3-urllib3

Commitment

I confirm that I have read the following resources:

Software version

81bf77c

Brief summary

Followed BOOTSTRAP.md to install dom0 requirements in order defined there.
fedora-minimal fails with python3-urrllib3 missing

Steps to reproduce

deploy dom0, debian-minimal, fedora-minimal, sys-cacher and then mgmt
fedora-minimal fails while mgmt succeeds (later install missing dep of the former)

Expected behavior

BOOTSTRAP defined steps should work as described.
Just deployed clean on freshly received nv41 for testing clean.

Actual behavior

Fails at fedora-minimal. mgmt deploys the missing dep?

Updating qusal components fails

Commitment

I confirm that I have read the following resources:

Software version

Here is my current update script.

Brief summary

Here is my update script while imperfect (i see qusal hardcoded... Should be rewritten to abstract, half baked I know, should probably show all commits and maybe even changes related for review prior of acceptance by end-user; as of now its middle ground)

$ cat ./qusal-dom0_update.sh 
qube="qusal" # qube name where you downloaded the repository
file="/home/user/qusal" # path to the repository in the qube

echo "-----"
echo "Legend:"
echo " locality (qube/dom0): action performed"
echo "-----"

echo "dom0: Cleaning local dirs..."
rm -rf "/home/user/QubesIncoming/${qube}"
mkdir -p "/home/user/QubesIncoming/${qube}"

echo "$qube: Fetching updates from GitHub..."
qvm-run --pass-io qusal "git -C /home/user/qusal fetch"

echo "$qube: Getting the hash of the current HEAD..."
current_head=$(qvm-run --pass-io qusal "git -C /home/user/qusal rev-parse HEAD")

echo "$qube: Getting the hash of the upstream HEAD..."
upstream_head=$(qvm-run --pass-io qusal "git -C /home/user/qusal rev-parse @{u}")

if [ "$current_head" == "$upstream_head" ]; then
    echo "$qube: Your qusal repository is already up to date."
else
    echo "$qube: Showing commit logs between current version and upstream..."
    qvm-run --pass-io qusal "git -C /home/user/qusal log --oneline ${current_head}..${upstream_head}"

    read -p "Review the commit logs above. If you are sure you want to update to the upstream version, press Y: " -n 1 -r
    echo    # (optional) move to a new line
   if [[ $REPLY =~ ^[Yy]$ ]]
    then
        echo "$qube: Updating qusal code from GitHub..."
        qvm-run --pass-io qusal "git -C /home/user/qusal merge @{u}"
    else
        echo "Update aborted by user."
        exit 1
    fi
fi

echo "dom0: Overwriting qusal's code with $qube's current codebase..."
qvm-run --pass-io --localcmd="UPDATES_MAX_FILES=10000 \
  /usr/libexec/qubes/qfile-dom0-unpacker user \
  ~/QubesIncoming/${qube}" \
  "${qube}" /usr/lib/qubes/qfile-agent "${file}"

echo "dom0: Deploying qusal salt recipes to dom0 salt dirs..."
/home/user/QubesIncoming/qusal/qusal/scripts/setup.sh

echo "dom0: Shutting down $qube qube..."
qvm-shutdown --wait qusal

echo "Done!"

It's output:

(130)$ ./qusal-dom0_update.sh 
-----
Legend:
 locality (qube/dom0): action performed
-----
dom0: Cleaning local dirs...
qusal: Fetching updates from GitHub...
From https://github.com/ben-grande/qusal
   ec7f62f..5722a25  main       -> origin/main
Fetching submodule salt/dotfiles
From https://github.com/ben-grande/dotfiles
   d3222aa..84bae87  main       -> origin/main
qusal: Getting the hash of the current HEAD...
qusal: Getting the hash of the upstream HEAD...
qusal: Showing commit logs between current version and upstream...
5722a25 fix: discover non-root username at runtime
e84959b fix: update fedora mirror list with upstream
760fdd9 doc: cacher documentation duplicates sections
bfd7b22 fix: incorrect path to repo rewriter service
234afc3 doc: update cacher table of contents
1ede2e1 fix: allow update check to work on cacher clients
a6f7d23 doc: wrong cacher header position
648bdad fix: remove updatevm tag after DomU uninstallation
7ec71cc feat: bump Mirage Firewall version
69745df fix: update dotfiles module
Review the commit logs above. If you are sure you want to update to the upstream version, press Y: Y
qusal: Updating qusal code from GitHub...
Updating ec7f62f..5722a25
Fast-forward
 salt/dom0/dotfiles.sls                             |   8 +-
 salt/dom0/files/autostart/xprofile.desktop         |   4 +-
 salt/dom0/gui-user.jinja                           |   9 +
 salt/dom0/kde.sls                                  |  12 +-
 salt/dom0/screenshot.sls                           |   8 +-
 salt/dotfiles                                      |   2 +-
 salt/sys-cacher/README.md                          |  48 +--
 salt/sys-cacher/deinit.top                         |   6 +-
 .../sys-cacher/files/client/bin/apt-cacher-ng-repo | 328 +++++++++++++++++----
 .../systemd/qubes-apt-cacher-ng-repo.service       |  15 +
 salt/sys-cacher/files/server/conf/acng.conf        |   2 +-
 .../sys-cacher/files/server/mirrors/fedora_mirrors |  46 ++-
 salt/sys-cacher/install-client.sls                 |  30 +-
 salt/sys-cacher/uninstall-client.sls               |   8 +-
 salt/sys-cacher/uninstall-client.top               |   2 +-
 salt/sys-mirage-firewall/create.sls                |   4 +-
 salt/utils/macros/sync-appmenus.sls                |   5 +-
 17 files changed, 415 insertions(+), 122 deletions(-)
 create mode 100644 salt/dom0/gui-user.jinja
 create mode 100644 salt/sys-cacher/files/client/systemd/qubes-apt-cacher-ng-repo.service
dom0: Overwriting qusal's code with qusal's current codebase...

dom0: Deploying qusal salt recipes to dom0 salt dirs...
dom0: Shutting down qusal qube...
Done!

So we understand that sys-cacher and dom0 were updated between said used versions.

Steps to reproduce

Look above.

Now. sys-cacher updated properly doing

$ sudo qubesctl top.enable sys-cacher browser
sudo qubesctl --targets=tpl-browser,sys-cacher-browser,tpl-sys-cacher,sys-cacher state.apply
sudo qubesctl top.disable sys-cacher browser
sudo qubesctl state.apply sys-cacher.appmenus,sys-cacher.tag
sudo qubesctl --skip-dom0 --targets="$(qvm-ls --no-spinner --raw-list --tags updatevm-sys-cacher | tr "\n" ",")" state.apply sys-cacher.install-client
local:
    ----------
    sys-cacher.top:
        ----------
        status:
            enabled
    browser.top:
        ----------
        status:
            enabled
local:
----------
          ID: debian-12-minimal-template-installed
    Function: qvm.template_installed
        Name: debian-12-minimal
      Result: True
     Comment: Template debian-12-minimal version 4.2.0 already installed
     Started: 18:43:33.954340
    Duration: 476.514 ms
     Changes:   
----------
          ID: dvm-debian-12-minimal-absent
    Function: qvm.absent
        Name: debian-minimal-dvm
      Result: True
     Comment: [SKIP] The VM with the name 'debian-minimal-dvm' is already missing. None
     Started: 18:43:34.430978
    Duration: 376.019 ms
     Changes:   
----------
          ID: dvm-debian-12-minimal-absent
    Function: qvm.absent
        Name: debian-12-minimal-dvm
      Result: True
     Comment: [SKIP] The VM with the name 'debian-12-minimal-dvm' is already missing. None
     Started: 18:43:34.807439
    Duration: 278.915 ms
     Changes:   
----------
          ID: debian-12-minimal
    Function: qvm.vm
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'debian-12-minimal' already exists.
              
              ====== ['prefs'] ======
              [SKIP] audiovm            : None
              [SKIP] label              : black
              [SKIP] include_in_backups : False
              [SKIP] maxmem             : 600
              [SKIP] memory             : 300
              [SKIP] vcpus              : 1
              
              ====== ['features'] ======
              [SKIP] Feature already in desired state: SET 'menu-items' = qubes-open-file-manager.desktop qubes-run-terminal.desktop qubes-start.desktop
              [SKIP] Feature already in desired state: SET 'default-menu-items' = qubes-open-file-manager.desktop qubes-run-terminal.desktop qubes-start.desktop
     Started: 18:43:35.087021
    Duration: 395.299 ms
     Changes:   
----------
          ID: dvm-debian-minimal
    Function: qvm.vm
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'dvm-debian-minimal' already exists.
              
              ====== ['prefs'] ======
              [SKIP] audiovm            : None
              [SKIP] template_for_dispvms: True
              [SKIP] label              : red
              [SKIP] include_in_backups : False
              [SKIP] maxmem             : 400
              [SKIP] memory             : 300
              [SKIP] template           : debian-12-minimal
              [SKIP] vcpus              : 1
              
              ====== ['features'] ======
              [SKIP] Feature already in desired state: ENABLE 'appmenus-dispvm' = Enabled
              [SKIP] Feature already in desired state: SET 'menu-items' = qubes-open-file-manager.desktop qubes-run-terminal.desktop qubes-start.desktop
     Started: 18:43:35.482830
    Duration: 379.892 ms
     Changes:   
----------
          ID: debian-minimal-set-default_template
    Function: cmd.run
        Name: qubes-prefs default_template debian-12-minimal
      Result: True
     Comment: Command "qubes-prefs default_template debian-12-minimal" run
     Started: 18:43:35.875445
    Duration: 459.336 ms
     Changes:   
              ----------
              pid:
                  44121
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: tpl-browser-clone
    Function: qvm.clone
        Name: tpl-browser
      Result: True
     Comment: [SKIP] A VM with the name 'tpl-browser' already exists. None
     Started: 18:43:36.335046
    Duration: 309.819 ms
     Changes:   
----------
          ID: tpl-browser
    Function: qvm.vm
      Result: True
     Comment: ====== ['prefs'] ======
              [SKIP] audiovm            : None
              [SKIP] maxmem             : 2000
              [SKIP] memory             : 300
              
              ====== ['features'] ======
              [SKIP] Feature already in desired state: SET 'default-menu-items' = firefox-esr.desktop chromium.desktop google-chrome.desktop qubes-run-terminal.desktop qubes-start.desktop
              [SKIP] Feature already in desired state: SET 'menu-items' = firefox-esr.desktop chromium.desktop google-chrome.desktop qubes-run-terminal.desktop qubes-start.desktop
     Started: 18:43:36.645616
    Duration: 119.326 ms
     Changes:   
----------
          ID: dvm-browser
    Function: qvm.vm
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'dvm-browser' already exists.
              
              ====== ['prefs'] ======
              [SKIP] audiovm            : *default*
              [SKIP] template_for_dispvms: True
              [SKIP] label              : red
              [SKIP] include_in_backups : False
              [SKIP] maxmem             : 2000
              [SKIP] memory             : 300
              [SKIP] vcpus              : 1
              
              ====== ['features'] ======
              [SKIP] Feature already in desired state: ENABLE 'appmenus-dispvm' = Enabled
              [SKIP] Feature already in desired state: ENABLE 'service.qubes-ctap-proxy' = Enabled
              [SKIP] Feature already in desired state: DISABLE 'service.tracker' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.evolution-data-server' = Disabled
              [SKIP] Feature already in desired state: SET 'menu-items' = firefox-esr.desktop chromium.desktop google-chrome.desktop qubes-run-terminal.desktop qubes-start.desktop
     Started: 18:43:36.765129
    Duration: 362.039 ms
     Changes:   
----------
          ID: tpl-sys-cacher-clone
    Function: qvm.clone
        Name: tpl-sys-cacher
      Result: True
     Comment: [SKIP] A VM with the name 'tpl-sys-cacher' already exists. None
     Started: 18:43:37.127488
    Duration: 298.403 ms
     Changes:   
----------
          ID: tpl-sys-cacher
    Function: qvm.vm
      Result: True
     Comment: ====== ['prefs'] ======
              [SKIP] autostart          : False
              [SKIP] audiovm            : None
              [SKIP] include_in_backups : False
              [SKIP] maxmem             : 500
              [SKIP] memory             : 300
              [SKIP] vcpus              : 1
              
              ====== ['features'] ======
              [SKIP] Feature already in desired state: DISABLE 'service.cups' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.cups-browsed' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.tracker' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.evolution-data-server' = Disabled
              [SKIP] Feature already in desired state: SET 'menu-items' = cacher-browser.desktop qubes-run-terminal.desktop qubes-start.desktop
              [SKIP] Feature already in desired state: SET 'default-menu-items' = cacher-browser.desktop qubes-run-terminal.desktop qubes-start.desktop
     Started: 18:43:37.426503
    Duration: 109.341 ms
     Changes:   
----------
          ID: sys-cacher
    Function: qvm.vm
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'sys-cacher' already exists.
              
              ====== ['prefs'] ======
              [SKIP] autostart          : False
              [SKIP] audiovm            : None
              [SKIP] label              : gray
              [SKIP] include_in_backups : True
              [SKIP] maxmem             : 0
              [SKIP] memory             : 500
              [SKIP] provides_network   : True
              [SKIP] template           : tpl-sys-cacher
              [SKIP] vcpus              : 1
              
              ====== ['features'] ======
              [SKIP] Feature already in desired state: ENABLE 'servicevm' = Enabled
              [SKIP] Feature already in desired state: ENABLE 'service.crond' = Enabled
              [SKIP] Feature already in desired state: DISABLE 'service.cups' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.cups-browsed' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.tinyproxy' = Disabled
              [SKIP] Feature already in desired state: SET 'menu-items' = cacher-browser.desktop qubes-run-terminal.desktop qubes-start.desktop
     Started: 18:43:37.536063
    Duration: 903.523 ms
     Changes:   
              ----------
              qvm.features:
                  ----------
                  qvm.features:
                      ----------
                      service.meminfo-writer:
                          ----------
                          new:
                          old:
                              None
----------
          ID: sys-cacher-browser
    Function: qvm.vm
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'sys-cacher-browser' already exists.
              
              ====== ['prefs'] ======
              [SKIP] autostart          : False
              [SKIP] audiovm            : None
              [SKIP] label              : gray
              [SKIP] include_in_backups : False
              [SKIP] maxmem             : 500
              [SKIP] memory             : 300
              [SKIP] netvm              : None
              [SKIP] template           : tpl-browser
              [SKIP] vcpus              : 1
              
              ====== ['features'] ======
              [SKIP] Feature already in desired state: DISABLE 'service.cups' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.cups-browsed' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.tracker' = Disabled
              [SKIP] Feature already in desired state: DISABLE 'service.evolution-data-server' = Disabled
              [SKIP] Feature already in desired state: SET 'menu-items' = cacher-browser.desktop qubes-run-terminal.desktop qubes-start.desktop
     Started: 18:43:38.440090
    Duration: 402.462 ms
     Changes:   
----------
          ID: sys-cacher-absent-rpc-policy
    Function: file.absent
        Name: /etc/qubes/policy.d/75-sys-cacher.policy
      Result: True
     Comment: File /etc/qubes/policy.d/75-sys-cacher.policy is not present
     Started: 18:43:38.857606
    Duration: 0.847 ms
     Changes:   
----------
          ID: sys-cacher-set-rpc-policy
    Function: file.managed
        Name: /etc/qubes/policy.d/45-sys-cacher.policy
      Result: True
     Comment: File /etc/qubes/policy.d/45-sys-cacher.policy is in the correct state
     Started: 18:43:38.858552
    Duration: 16.063 ms
     Changes:   
----------
          ID: sys-cacher-extend-volume
    Function: cmd.run
        Name: qvm-volume extend sys-cacher:private 20Gi
      Result: True
     Comment: Command "qvm-volume extend sys-cacher:private 20Gi" run
     Started: 18:43:38.874837
    Duration: 711.622 ms
     Changes:   
              ----------
              pid:
                  44357
              retcode:
                  0
              stderr:
              stdout:

Summary for local
-------------
Succeeded: 16 (changed=3)
Failed:     0
-------------
Total states run:     16
Total run time:    5.599 s
tpl-sys-cacher: OK
tpl-browser: OK
sys-cacher: OK
sys-cacher-browser: OK
local:
    ----------
    disabled:
        - sys-cacher.top
        - browser.top
    unchanged:
        - sys-cacher.top
        - browser.top
local:
----------
          ID: tpl-browser-start
    Function: qvm.start
        Name: tpl-browser
      Result: True
     Comment: qvm.state running
              Running None
     Started: 18:46:24.381322
    Duration: 9103.121 ms
     Changes:   
----------
          ID: tpl-browser-sync-appmenus
    Function: cmd.run
        Name: qvm-sync-appmenus tpl-browser
      Result: True
     Comment: Command "qvm-sync-appmenus tpl-browser" run
     Started: 18:46:33.487436
    Duration: 3805.185 ms
     Changes:   
              ----------
              pid:
                  60803
              retcode:
                  0
              stderr:
                  tpl-browser: Updating qubes-run-terminal
                  tpl-browser: Updating thunar-settings
                  tpl-browser: Updating qubes-open-file-manager
                  tpl-browser: Updating chromium
                  tpl-browser: Updating debian-xterm
                  tpl-browser: Updating vim
                  tpl-browser: Updating qvm-open-in-dvm
                  tpl-browser: Updating debian-uxterm
                  tpl-browser: Updating thunar
                  tpl-browser: Updating thunar-bulk-rename
                  tpl-browser: Creating appmenus
                  dvm-browser: Creating appmenus
                  sys-cacher-browser: Creating appmenus
                  sys-syncthing-browser: Creating appmenus
              stdout:
----------
          ID: tpl-browser-shutdown
    Function: qvm.shutdown
        Name: tpl-browser
      Result: True
     Comment: qvm.state halted
              Halted None
     Started: 18:46:37.294296
    Duration: 14362.81 ms
     Changes:   
----------
          ID: tpl-sys-cacher-start
    Function: qvm.start
        Name: tpl-sys-cacher
      Result: True
     Comment: qvm.state running
              Running None
     Started: 18:46:51.657806
    Duration: 7903.502 ms
     Changes:   
----------
          ID: tpl-sys-cacher-sync-appmenus
    Function: cmd.run
        Name: qvm-sync-appmenus tpl-sys-cacher
      Result: True
     Comment: Command "qvm-sync-appmenus tpl-sys-cacher" run
     Started: 18:46:59.562564
    Duration: 1708.123 ms
     Changes:   
              ----------
              pid:
                  62980
              retcode:
                  0
              stderr:
                  tpl-sys-cacher: Updating qubes-run-terminal
                  tpl-sys-cacher: Failed to get icon for qubes-run-terminal: No icon received
                  tpl-sys-cacher: Updating cacher-browser-general
                  tpl-sys-cacher: Failed to get icon for cacher-browser-general: No icon received
                  tpl-sys-cacher: Updating qubes-open-file-manager
                  tpl-sys-cacher: Failed to get icon for qubes-open-file-manager: No icon received
                  tpl-sys-cacher: Updating debian-xterm
                  tpl-sys-cacher: Failed to get icon for debian-xterm: No icon received
                  tpl-sys-cacher: Updating vim
                  tpl-sys-cacher: Failed to get icon for vim: No icon received
                  tpl-sys-cacher: Updating cacher-browser
                  tpl-sys-cacher: Failed to get icon for cacher-browser: No icon received
                  tpl-sys-cacher: Updating qvm-open-in-dvm
                  tpl-sys-cacher: Updating debian-uxterm
                  tpl-sys-cacher: Failed to get icon for debian-uxterm: No icon received
                  tpl-sys-cacher: Creating appmenus
                  sys-cacher: Creating appmenus
              stdout:
----------
          ID: tpl-sys-cacher-shutdown
    Function: qvm.shutdown
        Name: tpl-sys-cacher
      Result: True
     Comment: qvm.state halted
              Halted None
     Started: 18:47:01.272864
    Duration: 17185.429 ms
     Changes:   
----------
          ID: sys-cacher-browser-start
    Function: qvm.start
        Name: sys-cacher-browser
      Result: True
     Comment: qvm.state running
              Running None
     Started: 18:47:18.458921
    Duration: 8639.711 ms
     Changes:   
----------
          ID: sys-cacher-browser-sync-appmenus
    Function: cmd.run
        Name: qvm-sync-appmenus sys-cacher-browser
      Result: True
     Comment: Command "qvm-sync-appmenus sys-cacher-browser" run
     Started: 18:47:27.099941
    Duration: 1504.698 ms
     Changes:   
              ----------
              pid:
                  65134
              retcode:
                  0
              stderr:
                  sys-cacher-browser: Updating cacher-browser
                  sys-cacher-browser: Creating appmenus
              stdout:
----------
          ID: sys-cacher-browser-shutdown
    Function: qvm.shutdown
        Name: sys-cacher-browser
      Result: True
     Comment: qvm.state halted
              Halted None
     Started: 18:47:28.606212
    Duration: 11305.691 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-deb-11-heads
    Function: qvm.tags
        Name: deb-11-heads
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:39.912467
    Duration: 511.224 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-deb-12-xfce-spotify
    Function: qvm.tags
        Name: deb-12-xfce-spotify
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:40.423867
    Duration: 480.065 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-deb12_no_gnat-heads-docker
    Function: qvm.tags
        Name: deb12_no_gnat-heads-docker
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,restore_incomplete,updatevm-sys-cacher None
     Started: 18:47:40.904126
    Duration: 470.261 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-deb12_with_gnat-heads-docker
    Function: qvm.tags
        Name: deb12_with_gnat-heads-docker
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,restore_incomplete,updatevm-sys-cacher None
     Started: 18:47:41.374544
    Duration: 464.405 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-debian-11
    Function: qvm.tags
        Name: debian-11
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:41.839125
    Duration: 508.531 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-debian-12-extrepo-heads
    Function: qvm.tags
        Name: debian-12-extrepo-heads
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,restore_incomplete,updatevm-sys-cacher None
     Started: 18:47:42.347840
    Duration: 461.917 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-debian-12-minimal
    Function: qvm.tags
        Name: debian-12-minimal
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:42.809961
    Duration: 508.813 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-debian-12-xfce
    Function: qvm.tags
        Name: debian-12-xfce
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:43.318956
    Duration: 455.483 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-debian-12-xfce-vpn
    Function: qvm.tags
        Name: debian-12-xfce-vpn
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:43.774589
    Duration: 410.651 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-debian-luban3d
    Function: qvm.tags
        Name: debian-luban3d
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,audiovm-dom0,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:44.185390
    Duration: 464.602 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-f37-builder
    Function: qvm.tags
        Name: f37-builder
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:44.650240
    Duration: 473.729 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-fedora-37
    Function: qvm.tags
        Name: fedora-37
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:45.124141
    Duration: 492.338 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-fedora-37-xfce-printer
    Function: qvm.tags
        Name: fedora-37-xfce-printer
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:45.616632
    Duration: 449.277 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-fedora-38-xfce
    Function: qvm.tags
        Name: fedora-38-xfce
      Result: True
     Comment: [SKIP] All requested tags already set: audiovm-disp-sys-audio,created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:46.066028
    Duration: 364.792 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-fedora-39
    Function: qvm.tags
        Name: fedora-39
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:46.430940
    Duration: 452.162 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-fedora-39-minimal
    Function: qvm.tags
        Name: fedora-39-minimal
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:46.883202
    Duration: 454.85 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-browser
    Function: qvm.tags
        Name: tpl-browser
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:47.338151
    Duration: 441.128 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-dev
    Function: qvm.tags
        Name: tpl-dev
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:47.779391
    Duration: 403.484 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-fetcher
    Function: qvm.tags
        Name: tpl-fetcher
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:48.183049
    Duration: 416.162 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-media
    Function: qvm.tags
        Name: tpl-media
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:48.599356
    Duration: 491.642 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-mgmt
    Function: qvm.tags
        Name: tpl-mgmt
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:49.091170
    Duration: 389.864 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-qubes-builder
    Function: qvm.tags
        Name: tpl-qubes-builder
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:49.481173
    Duration: 339.315 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-reader
    Function: qvm.tags
        Name: tpl-reader
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:49.820616
    Duration: 386.967 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-sys-audio
    Function: qvm.tags
        Name: tpl-sys-audio
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:50.207741
    Duration: 379.25 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-sys-cacher
    Function: qvm.tags
        Name: tpl-sys-cacher
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:50.587131
    Duration: 442.62 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-sys-git
    Function: qvm.tags
        Name: tpl-sys-git
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:51.029880
    Duration: 444.179 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-sys-pgp
    Function: qvm.tags
        Name: tpl-sys-pgp
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:51.474202
    Duration: 474.598 ms
     Changes:   
----------
          ID: sys-cacher-tag-for-tpl-sys-syncthing
    Function: qvm.tags
        Name: tpl-sys-syncthing
      Result: True
     Comment: [SKIP] All requested tags already set: created-by-dom0,guivm-dom0,updatevm-sys-cacher None
     Started: 18:47:51.948966
    Duration: 464.37 ms
     Changes:   

Summary for local
-------------
Succeeded: 37 (changed=3)
Failed:     0
-------------
Total states run:     37
Total run time:   88.015 s
deb12_with_gnat-heads-docker: OK
deb-12-xfce-spotify: OK
deb-11-heads: OK
deb12_no_gnat-heads-docker: OK
debian-11: OK
debian-12-extrepo-heads: OK
debian-12-minimal: OK
debian-12-xfce: OK
debian-12-xfce-vpn: OK
fedora-37: OK
f37-builder: OK
debian-luban3d: OK
fedora-38-xfce: OK
fedora-39-minimal: OK
fedora-37-xfce-printer: OK
fedora-39: OK
tpl-fetcher: OK
tpl-browser: OK
tpl-media: OK
tpl-dev: OK
tpl-mgmt: OK
tpl-qubes-builder: OK
tpl-reader: OK
tpl-sys-audio: OK
tpl-sys-cacher: OK
tpl-sys-git: OK
tpl-sys-syncthing: OK
tpl-sys-pgp: OK
wyng-wdcloud: OK
heads-tests-deb12: OK

But dom0 failed

[user@dom0 ~]
$ sudo qubesctl top.enable dom0
sudo qubesctl state.apply
sudo qubesctl top.disable dom0
local:
    ----------
    dom0.top:
        ----------
        status:
            enabled
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
    output = template.render(**decoded_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/jinja2/environment.py", line 1291, in render
    self.environment.handle_exception()
  File "/usr/lib/python3.11/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 53, in top-level template code
  File "/usr/lib/python3.11/site-packages/jinja2/sandbox.py", line 326, in getattr
    value = getattr(obj, attribute)
            ^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'gui_user' is undefined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/utils/templates.py", line 218, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/utils/templates.py", line 482, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'gui_user' is undefined; line 53

---
[...]
    - require:
      - cmd: {{ slsdotpath }}-disable-lightdm

"{{ slsdotpath }}-activity-notifier":
  file.managed:
    - name: {{ gui_user.gui_user_home }}/.config/autostart-scripts/kde-activity-changed-notifier    <======================
    - source: salt://{{ slsdotpath }}/files/autostart-scripts/kde-activity-changed-notifier
    - user: {{ gui_user.gui_user }}
    - group: {{ gui_user.gui_user }}
    - mode: '0755'
    - makedirs: True
[...]
---
[CRITICAL] Rendering SLS 'base:dom0.kde' failed: Jinja variable 'gui_user' is undefined; line 53

---
[...]
    - require:
      - cmd: {{ slsdotpath }}-disable-lightdm

"{{ slsdotpath }}-activity-notifier":
  file.managed:
    - name: {{ gui_user.gui_user_home }}/.config/autostart-scripts/kde-activity-changed-notifier    <======================
    - source: salt://{{ slsdotpath }}/files/autostart-scripts/kde-activity-changed-notifier
    - user: {{ gui_user.gui_user }}
    - group: {{ gui_user.gui_user }}
    - mode: '0755'
    - makedirs: True
[...]
---
local:
    Data failed to compile:
----------
    Rendering SLS 'base:dom0.kde' failed: Jinja variable 'gui_user' is undefined; line 53

---
[...]
    - require:
      - cmd: {{ slsdotpath }}-disable-lightdm

"{{ slsdotpath }}-activity-notifier":
  file.managed:
    - name: {{ gui_user.gui_user_home }}/.config/autostart-scripts/kde-activity-changed-notifier    <======================
    - source: salt://{{ slsdotpath }}/files/autostart-scripts/kde-activity-changed-notifier
    - user: {{ gui_user.gui_user }}
    - group: {{ gui_user.gui_user }}
    - mode: '0755'
    - makedirs: True
[...]
---
DOM0 configuration failed, not continuing
local:
    ----------
    disabled:
        - dom0.top
    unchanged:
        - dom0.top
[user@dom0 ~]

Expected behavior

Success updating components.

Actual behavior

dom0 salt deployment on update failed.

sys-cacher : uninstallation doesn't work

Commitment

I confirm that I have read the following resources:

Software version

af27eb1

Brief summary

Still having issues deploying sys-cacher correctly (tem[lates still hit tinyproxy which fails translating http->https).
Consequently, uninstalling was attempted and fails.

Steps to reproduce

[user@dom0 ~/QubesIncoming/heads-tests-deb12-nix/qusal(main)]
$ qubesctl top.enable sys-cacher.deinit
local:
    ----------
    sys-cacher.deinit.top:
        ----------
        status:
            enabled
[user@dom0 ~/QubesIncoming/heads-tests-deb12-nix/qusal(main)]
$ qubesctl --templates state.apply
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
[CRITICAL] Rendering SLS 'base:sys-cacher.untag' failed: could not find expected ':'
local:
    Data failed to compile:
----------
    Rendering SLS 'base:sys-cacher.untag' failed: could not find expected ':'
DOM0 configuration failed, not continuing
[user@dom0 ~/QubesIncoming/heads-tests-deb12-nix/qusal(main)]
(1)$ qubesctl state.apply sys-cacher.remove-policy
local:
----------
          ID: sys-cacher-absent-rpc-policy
    Function: file.absent
        Name: /etc/qubes/policy.d/75-sys-cacher.policy
      Result: True
     Comment: Removed file /etc/qubes/policy.d/75-sys-cacher.policy
     Started: 15:03:52.522713
    Duration: 26.476 ms
     Changes:   
              ----------
              removed:
                  /etc/qubes/policy.d/75-sys-cacher.policy

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  26.476 ms
[user@dom0 ~/QubesIncoming/heads-tests-deb12-nix/qusal(main)]
$ qubesctl state.apply sys-cacher.untag
[CRITICAL] Rendering SLS 'base:sys-cacher.untag' failed: could not find expected ':'
local:
    Data failed to compile:
----------
    Rendering SLS 'base:sys-cacher.untag' failed: could not find expected ':'
DOM0 configuration failed, not continuing

Expected behavior

applying deinstallation steps from README should work as described

Actual behavior

seems like syntax errors are present and fails from untag.

apt cacher fedora updates fail with zchunk errors on fresh install

Commitment

I confirm that I have read the following resources:

Software version

Tested up to commit b2c9479

Brief summary

Two separate attempts at a fresh install using qusal failed when running sudo qubesctl --targets=tpl-mgmt state.apply when iterating through the mgmt installation steps. The script fails at the install-salt-deps stage due to sys-cacher zchunk errors.

Steps to reproduce

Perform a fresh install using qusal and proceed through the bootstrap guide up to mgmt install.

Expected behavior

Installation should proceed as expected with sys-cacher handling fedora updates. Alternatively all fedora templates should not be using apt cacher

Actual behavior

After installing sys-cacher all attempts to update fedora vms fail with zchunk errors

sys-cacher deployed on qos install with sys-whonix as default proxy doesn't work

Commitment

I confirm that I have read the following resources:

Software version

Master

Brief summary

Sys-cacher still doesn't work for me on fresh install

But just saw unman/shaker@1cda4ca

(which @unman clearly states not using whonix per choice and prefers sys-tor which he hasn't yet published)

Steps to reproduce

Deploy q4.2 with sys-whonix as proxy for updates and then qusal

Expected behavior

sys-cacher working out of the box on possible QOS installable options

Actual behavior

Updates still attempted on tinyproxy by error observed. Orders of policies seem wrong.

qvm-template list is incomplete when the updatevm has sys-cacher configured

Software version

Possibly every since sys-cacher.install-client is being called in sys-pihole.install in R4.2.

Brief summary

Listing templates from Dom0 does not work if using certain qubes for certain functionality that should support it, as sys-pihole is being set as the updatevm.

Steps to reproduce

Install sys-pihole, configure it to be the updatevm and make sure it is using sys-cacher as the updates proxy. Make sure that sys-cacher netvm is set to sys-pihole.

Notice the template list is incomplete. Comment the proxy line in /etc/dnf/dnf.conf and try again and see that the list is complete.

Expected behavior

Complete list of templates available.

Actual behavior

Incomplete list of templates via qvm-template list, only showing installed templates.

dom0 calls sys-pihole via qvm-template, which calls sys-cacher via qubes.UpdatesProxy, which then calls sys-pihole again as the netvm.

Possible solution

There are two solutions:

  1. do not cache updates from sys-pihole as it is probably the netvm of the sys-cacher
  2. create a separate updatevm

The first option disadvantage is having a slower install as packages are fetched through the network if they are not cached.

The second option disadvantage is that the updatevm is never powered off automatically after being used, leaving it hanging around while setting the updatevm to be the same as the default_netvm is good as it does not require one more qube to be powered on, besides that there are no security benefits in having a separate updater qube for dom0 as it does not trust the DomU anyway.

Although there are two solutions to the problem, it doesn't answer clearly why the problem occurs? What happens in sys-pihole that when using sys-cacher, the fetching of the template list does not work, while updating dom0 does work?

upstream Qubes VM kernel

https://github.com/ben-grande/qusal/blob/main/salt/kicksecure-minimal/install.sls

"{{ slsdotpath }}-installed":
  pkg.installed:
    - refresh: True
    - install_recommends: False
    - skip_suggestions: True
    - pkgs:
      - kicksecure-qubes-cli
      - lkrg-dkms
      - hardened-kernel
      - tirdad
      - linux-image-amd64
      - linux-headers-amd64
      - grub2
      - qubes-kernel-vm-support

Installing

  - linux-image-amd64
  - linux-headers-amd64
  - grub2
  - qubes-kernel-vm-support

should not be up to Kicksecure. Basically I am suggesting to contribute towards QubesOS/qubes-issues#5212.

This is problematic because it derivatives from Kicksecure default kernel, which is Qubes default kernel configuration.

This is unmaintainable, because if there are Qubes VM kernel specific crashes, issues, then these would be reported by users to Kicksecure where I would have no possibility to debug these except saying "please reproduce this in Qubes Debian and report the bug to Qubes".

  1. Add a PR for https://github.com/QubesOS/qubes-meta-packages to add a meta package pulling all of these dependencies.

  2. Add a PR for Qubes Debian Template builder installing the kernel packages by default.

  3. Other changes towards making "flipping the switch to VM kernel" more easy. Last PR would be actually changing the default.

Don't be discouraged by QubesOS/qubes-doc#1342 not being merged for a long time. That apparently depends on Qubes internal processes, maintainer availability. It's easier to land Qubes source code pull requests than Qubes documentation pull requests.

do not enforce removal of `/etc/apt/sources.list`

"{{ slsdotpath }}-remove-debian-default-sources.list":
  file.absent:
    - require:
      - pkg: "{{ slsdotpath }}-installed"
    - name: /etc/apt/sources.list

Deletion of /etc/apt/sources.list makes sense when building Kicksecure while a default Debian sources list has been used but this should not be enforced going forward.

The user should be free to use /etc/apt/sources.list with custom repositories without having that file surprisingly deleted. That should be a "do once" action only.

And also only be removed if the user did not previously customized it to avoid data loss (loss of repositories).

qvm-screenshot tool missing region selection that was available in original version

Commitment

I confirm that I have read the following resources:

Current problem (if any)

an option to qvm-screenshot is missing so that keyboard shortcut + mouse interaction permits to select region on the screen to be the only selected input to be copied to destination qube as screenshot, as opposed to pass whole dom0 screenshot of qubes whole window and have to edit that screenshot to crop to arrive to the same result, incurring unecesary steps to arrive to the same result.

Proposed solution

Offer an option to make --region screenshot possible as upstream original qvm-screenshot tool to assign shortcut and use mouse to draw rectangle on screen to not have to use dom0 fullscreen or the whole qube window as a screenshot

The value to a user, and who that user might be

devels: have only relevant part of the screen be captured to quickly post on github issue
documentarist: be able to only select part of a screen to focus from macro <-> micro perspectives to help end users follow what is important in created doc
ux designer: create canvas based on actual UX and desired UX by just changing actual to desired GUI for changes and quicly iterate
Me: I have to do edit windows capture manually, which makes me loose precious time which could be done at the source otherwise.

Cacher doesn't deinit from non-templates

Software version

Any.

Brief summary

Easy deinit of sys-cacher can't be done on non-templates as Qubes Salt pillars have not implemented ability to detect tags. This makes it harder for non-templates to be uninstalled in the top format.

Steps to reproduce

Try to deinit non-templates.

Expected behavior

Uninstall from all qubes.

Actual behavior

Uninstall only in templates

As of now, you can uninstall from any qube by deleting the tag, removing the updates-proxy-setup service and running the uninstall-client state.

Proposed solution

Gonzalo Bulnes did qvm-tags-in-pillar, it makes sense for this formula to be upstreamed to Qubes OS rather than pulled to Qusal, but it depends on how much time it will take to be included in Qubes stable repos.

It can make sense for tags to be added to qubesctl, then the state solution will be easier to write than to discover which qubes have a tag and then call qubesctl on the command line, which can lead to typos.

Comment of mine on Gonzalo project asking to upstream it.

Improve MPV performances

Commitment

I confirm that I have read the following resources:

Current problem (if any)

The current installation of MPV in disp-media is not optimized. Playing video uses too many resources because it's not configured correctly.

Proposed solution

As explained in this post from QubesOS Forum, creating /etc/mpv/mpv.conf and setting two properties drastically improves performances. That's all needed.
There are probably other settings that can help (de-interlacing?), I still have some flickering sometimes but it's way better than the current state.
I also have sound delay. Setting a -300ms delay fixes it, but I don't really understand where it comes from.
Finally, should we increase max memory and vcpu settings to have more power? Can it help?

The value to a user, and who that user might be

Better performances when playing video with MPV in disp-media.

Remove Mirage Firewall tarball for a better git object compression

Current problem (if any)

Mirage Firewall formula has a tarball, it is easier to hold the tarball and bring it with the formula to dom0 them having to download it over the internet an install it in Dom0.

Unfortunately it is a large object and greatly increases the git database.

Proposed solution

Download Mirage-Firewall over the internet and install it in Dom0.

The value to a user, and who that user might be

Users and developes can get updated versions without having to waiting for me to build it locally. It also decreases the git database size.

.

sys-cacher should handle addition of new repositories definitions automatically with inotify

Commitment

I confirm that I have read the following resources:

Current problem (if any)

When deploying sys-cacher, dom0 checking mechanisms are nullified as of now. Also some repositories lists are currently not taken into consideration (extrepos not touched per sys-cacher deployment as of now)

Normal, since qubes are responsible to check for updates, and can't, since URLs of parent templates are modified to talk only over apt-cacher. Checks called from dom0 to qubes silently fail and no parent templates are reported as having updates.

Proposed solution

There is multiple possible solutions here, all of which have drawbacks and upsides.

  • inotifywatch script checking to see if running in template/standalone/qube and change urls dynamically for all files under OS flavor repo lists

  • current instructions instruct users to apply fixes from dom0 which enables tag, enables service in qube/dispvm

    • This is so that the firewall policies are respected generally, making the user responsible to know what he's doing.
      • inotify/conditional application if qube/template above would not require such manual intervention since the logic would be deployed by salt script under template on which qubes depend.

This is a case study advocating for #31 :)

The value to a user, and who that user might be

Be able to deploy new repositories/software for testing over qubes, deploy repositories in templates without needing to rewrite URLs manually, have template update notifications as normally expected from dom0 widget

Make sys-cacher apply fix for URLs dynamically

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Proposed solution

The value to a user, and who that user might be

As if today, sys-cacher is static and doesn't apply changes to repositories deployed after cacher deployment.

The ideal would be to have fixes for url automatically applied. Suggestion applied upstream bit not yet applied: unman/shaker#17

.

Does Qubes plan to deprecate `salt`?

Quote QubesOS/qubes-issues#8413 (comment) @marmarek

This is standard output from salt, not specific to qubesctl (which is just a wrapper around salt). If anything, that would be a feature request to salt. But since we are moving to own updater in place of salt, I don't think such change is worth doing.

Asked there. The outcome of that seems crucial for the future of this project.

User and group errors in sys-git dom0 salt

Commitment

I confirm that I have read the following resources:

Brief summary

When attempting to run sudo qubesctl state.apply sys-git.install-client in dom0 per the documentation, errors related to the placement of dotfiles occur because user user is not available Group user is not available as, I believe, dom0 isn't always simply named user by the user.

Steps to reproduce

Follow instructions to "Dom0 Update with Git". Run sudo qubesctl state.apply sys-git.install-client per instructions.

Expected behavior

Process completes without errors.

Actual behavior

[ERROR   ] User user is not available Group user is not available
[ERROR   ] {'/home/user/.config/git/template/hooks': {'directory': 'new'}}
[ERROR   ] {'/home/user/.config/git/shell': {'directory': 'new'}}
[ERROR   ] {'/home/user/.local/bin': {'directory': 'new'}}
[ERROR   ] {'/home/user/.config/git/shell': {'directory': 'new'}}
[ERROR   ] {'/home/user/.local/bin': {'directory': 'new'}}
[ERROR   ] User user is not available Group user is not available
[ERROR   ] {'/home/user/.local/bin': {'directory': 'new'}}
[ERROR   ] User user is not available Group user is not available
local:
----------
          ID: dotfiles-copy-git-home
    Function: file.recurse
        Name: /home/user
      Result: False
     Comment: User user is not available Group user is not available
     Started: 19:01:25.136938
    Duration: 27.153 ms
     Changes:   
----------
          ID: dotfiles-fix-executables-git-template-dir-home
    Function: file.directory
        Name: /home/user/.config/git/template/hooks
      Result: False
     Comment: No directory to create /home/user/.config/git/template/hooks in
     Started: 19:01:25.164330
    Duration: 2.333 ms
     Changes:   
              ----------
              /home/user/.config/git/template/hooks:
                  ----------
                  directory:
                      new
----------
          ID: dotfiles-fix-executables-git-shell-dir-home
    Function: file.directory
        Name: /home/user/.config/git/shell
      Result: False
     Comment: No directory to create /home/user/.config/git/shell in
     Started: 19:01:25.167138
    Duration: 4.096 ms
     Changes:   
              ----------
              /home/user/.config/git/shell:
                  ----------
                  directory:
                      new
----------
          ID: dotfiles-fix-executables-git-bin-dir-home
    Function: file.directory
        Name: /home/user/.local/bin
      Result: False
     Comment: No directory to create /home/user/.local/bin in
     Started: 19:01:25.174142
    Duration: 2.355 ms
     Changes:   
              ----------
              /home/user/.local/bin:
                  ----------
                  directory:
                      new

----------
          ID: dotfiles-fix-executables-git-shell-dir-skel
    Function: file.directory
        Name: /home/user/.config/git/shell
      Result: False
     Comment: No directory to create /home/user/.config/git/shell in
     Started: 19:01:29.038202
    Duration: 1.826 ms
     Changes:   
              ----------
              /home/user/.config/git/shell:
                  ----------
                  directory:
                      new
----------
          ID: dotfiles-fix-executables-git-bin-dir-skel
    Function: file.directory
        Name: /home/user/.local/bin
      Result: False
     Comment: No directory to create /home/user/.local/bin in
     Started: 19:01:29.040329
    Duration: 1.771 ms
     Changes:   
              ----------
              /home/user/.local/bin:
                  ----------
                  directory:
                      new
----------
          ID: dotfiles-copy-sh-home
    Function: file.recurse
        Name: /home/user/
      Result: False
     Comment: User user is not available Group user is not available
     Started: 19:01:29.042306
    Duration: 4.555 ms
     Changes:   
----------
          ID: dotfiles-fix-executables-sh-dir-home
    Function: file.directory
        Name: /home/user/.local/bin
      Result: False
     Comment: No directory to create /home/user/.local/bin in
     Started: 19:01:29.047105
    Duration: 1.876 ms
     Changes:   
              ----------
              /home/user/.local/bin:
                  ----------
                  directory:
                      new
----------
          ID: dotfiles-copy-x11-home
    Function: file.recurse
        Name: /home/user/
      Result: False
     Comment: User user is not available Group user is not available
     Started: 19:01:34.358719
    Duration: 6.51 ms
     Changes: 

.

Make dotfiles installation optional

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Some files from creator's personal Dotfiles are required during some installation. As stated in the design guidlines, Qusal sould "provides a minimal modular isolated environment", and "should not focus on a specific Qubes OS user base". Currently we need to fork all Dotfiles which are personal settings.

Proposed solution

If some of these files are required for the installation to succeed, put them in this project. Otherwise, define a way to make dotfiles installation optionnal, by opting-in or opting-out.

The value to a user, and who that user might be

Make the installation process more generic without any personal preferences.

Use a single update state

Current problem (if any)

The salt module pkg.uptodate is called on every installation state. The problem occurs when a state references multiple installation states, thus pkg.uptodate is called several times.

Note that pkg.installed is also called several times, but that is harder to fix because the package names vary while to update, there is no argument list.

Proposed solution

A global state that is called by the include in the installation states, thus referencing a single pkg.uptodate.

The value to a user, and who that user might be

Faster installation of the states for both users and developers. Especially important on Whonix qubes due to Tor network latency and not using the cacher.

Add recipe to deploy qubes-video-companion for screensharing/videoconferencing

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Sharing screens/webcam for people doing videoconferencing or live screensharing is complicated under Qubes.
https://github.com/QubesOS/qubes-video-companion resolved that.

But its not installed by default. qusal offers the possibility to do it easily and do the right thing creating the proper qubes to do ti efficiently.

Proposed solution

Another out of the box inexperience.

The value to a user, and who that user might be

User: ability to use camera easily
Documentarists: now they can use qubes-screenshot merged into dom0 deployment. But creating screencast videos is still convulated.
Developers: Easily share workflow leading to an error in github issues, screeencasting directly a single window content for recording, with/without sound.

Add Tailscale formula

Current problem (if any)

Remote management on restricted network is difficult:

  • Tunneling SSH is difficult and requires a public managed node;
  • Opening ports on the router and allowing traffic to flow to the self-hosted VPN; and
  • Creating a hidden service with Tor or I2P is slow and can have credentials authentication (Onion Authentication) but easier than all the other self-hosted methods;

Proposed solution

Add tailscale.

Adding tailscale to a qube can be a security concern. Your OpenID provider can authenticate to your machine, but your can also use your own OpenID instance. Using Tailscale does not open ports to the internet, only your configured nodes can access it, so less dangerous than opening ports on your router in this metric.

The installation will of course be optional and restricted to the qubes you want to have it.

The value to a user, and who that user might be

Users can remotely manage remote qubes and non-qubes more easily without having to setup their tunnel, be it self-hosted VPN, VPS with SSH Tunnel or Hidden service with Onion Authentication.

.

Add qubes-qvm-screenshot-tool dom0 salt recipe deployment

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Dom0 screenshot and setting preferences shortcuts is not so straightforward for end users. One amazing project was put under contrib packages, but bootstrapping first package installation is also not straightforward for end users.

Proposed solution

It would be really helpful to offer dom0 installation of https://github.com/QubesOS-contrib/qubes-qvm-screenshot-tool with a dom0 salt recipe, binding xfce alt-printscreen and printscreen keyboard shortcuts to both KDE and Xfce.

The value to a user, and who that user might be

Sharing a windows content or whole desktop through copying it directly to proper domu is already properly resolved by https://github.com/QubesOS-contrib/qubes-qvm-screenshot-tool. On system reinstallation, it should be as easy to deploy as qusal makes it for other tools.

Of course this is my preference. Not sure why this is not the default yet and in contrib repo, but that tool is available and superior to the default. Trying it is adopting it, with space for improvement upstream for it to eventually become the default.

Evaluate Salt's module `archive` for signature verification

Current problem (if any)

PGP verification is done with cmd.run due to to unavailability of options in gpg.verify and archive.extracted.

In Salt 3007.0, new options to these modules have been added regarding PGP signature verification, most notably:

  • You can trust only certain keys to be able to verify a file, the verification can occur by any key or enforce that all keys needs to be used to verify a file (signed_by_any, signed_by_all.
  • The archive.extracted can now also choose the gnupghome and keyring. The
  • The signature verification can occur on the checksum file source_hash_sig

Proposed solution

Evaluate if it is worth the change.

Sequoia has the GNUPG interface through chameleon, but gpg.verify does not appear to have a way to specify the gpg binary, in any way, chameleon is not available in Debian yet, it is on Fedora though.

When we migrate to Sequoia completely instead of GNUPG, unless Salt supports Sequoia, we are back to cmd.run as the module for signature verification.

The value to a user, and who that user might be

Cleaner Salt output, less workarounds with cmd.run. Potentially a more tested program.

.

Create an easier Qusal installation procedure for users to report errors depending on installed version

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Trust is everything.
Instructions in global README.md leaves the user to find its own way to pass ben-grande public key to dom0. Instructions could be better, and lead the user to different sources to verify the public key(eg https://keys.openpgp.org/search?q=00C64E14F51F9E56).

One way to do this is to have it available in a seperate git repo, also signed with the public key.

Github will then show verified commits (no more unverified yellow, scary commit traces on repo).

Proposed solution

  • Publish public key to seperate ben-grande repo.
  • Sign that commit with said public key
  • Link main README.md to repo containing public key as well as where the author also published the public key to other public keyservers which propagated it elsewhere.
  • Maybe even add that verification to qusal dom0 helper

Additionally (unrelated), maybe add in helper script the following output to help user (and ad to issue template):

  • Have the dom0 git fetcher script output:
    • git commit of fetched to dom0 git repo
    • have the helper script call setup script to reduce user error of salt recipes deployed
    • have the helper script verify the last commit against imported and checked public key

The value to a user, and who that user might be

  • Easy way for the user to have project's public key inside of dom0.
  • Easy way for user to validate commit domu to dom0 qusal repo
  • Easy way for the user to validate that deployed qusal to dom0 is latest versin (reduce user errors)

Disp-sys-audio not configured to start automatically on boot

Commitment

I confirm that I have read the following resources:

Software version

Latest

Brief summary

All qubes setuped to depend on sys-audio but sys-audio not setup to boot automatically nor any qube starting it

Steps to reproduce

Deploy sys-audio

Expected behavior

Have pci device removed from dom0 and pass-through sys-audio on boot as well as all Bluetooth controller (but not possible by default. Notes should be added in sys-audio notes)

Actual behavior

Audio still on dom0 unless sys-audio manually started. More troubleshooting needed, will comment later

Improve documentation regarding Salt

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Salt can be difficult at first for users unfamiliar with it. The goal here might not be to write a full tutorial, but writing some basic info could help better understanding how it works, how to use it and some specific explanations for this project.

Proposed solution

Add a Salt document in the docs folder with quick introduction, resources, tips and tricks and explanations about how it's used in this project.
Also, update all qubesctl commands to add sudo as it's required to run the command.

The value to a user, and who that user might be

Salt being the main tool here, it helps the user better understand the project and thus facilitate contributions.

sys-cacher misses important acng.conf improvements from upstream : Fedora still can't update with sys-cacher deployed

Commitment

I confirm that I have read the following resources:

Current problem (if any)

upstream: https://github.com/unman/shaker/commits/main/cacher/acng.conf
downstream: https://github.com/unman/shaker/blob/main/cacher/acng.conf

Comparison of files:

diff -u <(curl --silent https://raw.githubusercontent.com/ben-grande/qusal/main/salt/sys-cacher/files/server/conf/acng.conf) <(curl --silent https://raw.githubusercontent.com/unman/shaker/main/cacher/acng.conf
--- /dev/fd/63	2024-04-19 16:41:32.935677832 -0400
+++ /dev/fd/62	2024-04-19 16:41:32.936677832 -0400
@@ -1,8 +1,3 @@
-# SPDX-FileCopyrightText: 2022 - 2024 unman <[email protected]>
-# SPDX-FileCopyrightText: 2023 - 2024 Benjamin Grande M. S. <[email protected]>
-#
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
 #
 # IMPORTANT NOTE:
 #
@@ -16,20 +11,14 @@
 # software package downloads. It's supposed to be in a directory specified by
 # the -c option of apt-cacher-ng, see apt-cacher-ng(8) for details.
 # RULES:
-# - letter case in variable names does not matter
-# - names and values are separated by colon or equals sign
-# - for boolean variables, zero means false, non-zero means true
-# - "default value" means built-in (!) defaults, i.e. something which the
-#   program uses if the option is not set here or in other config files.
-#   That value might be explicitly mentioned in the description. Where it is
-#   not, there is no reason to assume any of the examples to be the default
-#   value! In doubt, use acngtool to query the value of the particular variable.
+# Letter case in variable names does not matter, names and values should be
+# separated with colons. For boolean variables, zero number is considered false,
+# non-zero considered true. If a default value is not explicitly mentioned in
+# the description, the commented value assignments mostly represent the default
+# values of the particular variables.
 
 # Storage directory for downloaded data and related maintenance activity.
 #
-# Note: When the value for CacheDir is changed, change the file
-# /lib/systemd/system/apt-cacher-ng.service too
-#
 CacheDir: /var/cache/apt-cacher-ng
 
 # Log file directory, can be set empty to disable logging
@@ -52,9 +41,7 @@
 # local interface. DNS resolution is performed using getaddrinfo(3) for all
 # available protocols (IPv4, IPv6, ...). Using a protocol specific format will
 # create binding(s) only on protocol specific socket(s), e.g. 0.0.0.0 will
-# listen only to IPv4. The endpoint can also be specified as host:port (or
-# [ipv6-address]:port) which allows binding on non-standard ports (Port
-# directive is ignored in this case).
+# listen only to IPv4.
 #
 # Default: listens on all interfaces and protocols
 #
@@ -72,32 +59,18 @@
 # In this example, some backends files might be generated during package
 # installation using information collected on the system.
 # Examples:
-#Remap-debrep: file:deb_mirror*.gz /debian ; file:backends_debian # Debian Archives
-Remap-alxrep: file:archlx_mirrors /archlinux # ; file:backend_archlx # Arch Linux
-Remap-alxrep: file:archlx_mirrors /archlinux # ; file:backend_archlx # Arch Linux
+#Remap-debrep: https://deb.debian.org http://deb.debian.org  file:deb_mirrors.gz /debian ; file:backends_debian # Debian Archives
+#Remap-alxrep: file:archlx_mirrors /archlinux # ; file:backend_archlx # Arch Linux
+Remap-alxrep: file:archlx_mirrors /archlinux 
+Remap-debrep: https://deb.debian.org http://deb.debian.org  file:deb_mirrors.gz /debian 
+Remap-fedora: file:fedora_mirrors # Fedora Linux
+Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu # Ubuntu Archives
 Remap-cygwin: file:cygwin_mirrors /cygwin # ; file:backends_cygwin # incomplete, please create this file or specify preferred mirrors here
-Remap-debrep: https://deb.debian.org http://deb.debian.org  file:deb_mirrors.gz /debian
+#Remap-sfnet:  file:sfnet_mirrors # ; file:backends_sfnet # incomplete, please create this file or specify preferred mirrors here
 Remap-epel:   file:epel_mirrors # Fedora EPEL
-Remap-fedora: file:fedora_mirrors # Fedora Linux
-Remap-fedora: file:fedora_mirrors # Fedora Linux
-Remap-gentoo: file:gentoo_mirrors.gz /gentoo ; file:backends_gentoo # Gentoo Archives
-Remap-klxrep: file:kali_mirrors /kali ; file:backends_kali # Kali Linux Archives
-Remap-secdeb: security.debian.org security.debian.org/debian-security deb.debian.org/debian-security /debian-security cdn-fastly.deb.debian.org/debian-security ; deb.debian.org/debian-security security.debian.org cdn-fastly.deb.debian.org/debian-security
-Remap-sfnet:  file:sfnet_mirrors # ; file:backends_sfnet # incomplete, please create this file or specify preferred mirrors here
 Remap-slrep:  file:sl_mirrors # Scientific Linux
-Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu # Ubuntu Archives
-# Qusal external repositories
-Remap-dockerrep: https://download.docker.com http://download.docker.com
-Remap-googlerep: https://dl.google.com http://dl.google.com
-Remap-hashicorprep: https://apt.releases.hashicorp.com http://apt.releases.hashicorp.com
-Remap-kicksecuredebrep: https://deb.kicksecure.com http://deb.kicksecure.com
-Remap-launchpadrep: https://ppa.launchpad.net http://ppa.launchpad.net
-Remap-opentofurep: https://packages.opentofu.org http://packages.opentofu.org
-Remap-qubesdebrep: https://deb.qubes-os.org http://deb.qubes-os.org
-Remap-qubesyumrep: https://yum.qubes-os.org http://yum.qubes-os.org
-Remap-signalrep: https://updates.signal.org http://updates.signal.org
-Remap-syncthingrep: https://apt.syncthing.net http://apt.syncthing.net
-Remap-whonixdebrep: https://deb.whonix.org http://deb.whonix.org
+Remap-gentoo: file:gentoo_mirrors.gz /gentoo ; file:backends_gentoo # Gentoo Archives
+Remap-secdeb: security.debian.org ; security.debian.org deb.debian.org/debian-security
 
 # Virtual page accessible in a web browser to see statistics and status
 # information, i.e. under http://localhost:3142/acng-report.html
@@ -110,17 +83,17 @@
 ReportPage: acng-report.html
 
 # Socket file for accessing through local UNIX socket instead of TCP/IP. Can be
-# used with inetd (via bridge tool in.acng from apt-cacher-ng package), is also
-# used internally for administrative purposes.
+# used with inetd (via bridge tool in.acng from apt-cacher-ng package).
 #
-# Default: /run/apt-cacher-ng/socket
+# Default: not set, UNIX socket bridge is disabled.
 #
-# SocketPath: /var/run/apt-cacher-ng/socket
+# SocketPath:/var/run/apt-cacher-ng/socket
 
 # If set to 1, makes log files be written to disk on every new line. Default
 # is 0, buffers are flushed after the client disconnects. Technically,
 # it's a convenience alias for the Debug option, see below for details.
 #
+# UnbufferLogs: 0
 UnbufferLogs: 1
 
 # Enables extended client information in log entries. When set to 0, only
@@ -155,18 +128,6 @@
 #
 ExThreshold: 4
 
-# If set to true, the removal (i.e. response status 404) of remote
-# volatile/index files is considered a hint to consider the local cached
-# versions irrelevant and also expire them just like package files. This adds
-# some risk of removing too much cache contents in cases where a middlebox
-# reports bogus 404 codes.
-#
-# If false (0), a less sloppy algorithm is used to invalidate certain keyfiles
-# first, which might subsequently expire the cache contents but much later or
-# maybe never unless the administrator intervenes.
-#
-FollowIndexFileRemoval: 1
-
 # If the expiration is run daily, it sometimes does not make much sense to do
 # it because the expected changes (i.e. removal of expired files) don't justify
 # the extra processing time or additional downloads for expiration operation
@@ -231,32 +192,21 @@
 # is refused when this value is reached (below zero = unlimited).
 # MaxConThreads: -1
 #
-# Timeout for a forced disconnect in cases where a client connection is about
-# to be closed but remote refuses to confirm the disconnect request. Setting
-# this to a lower value mitigates the effects of resource starvation in case of
-# a DOS attack but increases the risk of failing to flush the remaining portion
-# of data.
-# DisconnectTimeout: 15
-
-# By default, if a remote suddenly reconnects, ACNG tries at least two times to
-# redownload from the same or different location (if known).
-# DlMaxRetries: 2
-
 # Pigeonholing files (like static vs. volatile contents) is done by (extended)
 # regular expressions.
 #
 # The following patterns are available for the purposes detailed, where
 # the latter takes precedence over the former:
-# - <PFilePattern> for static data that doesn't change silently on the server.
-# - <VFilePattern> for volatile data that may change like every hour. Files
+# - «PFilePattern» for static data that doesn't change silently on the server.
+# - «VFilePattern» for volatile data that may change like every hour. Files
 #   that match both PFilePattern and VfilePattern will be treated as volatile.
 # - Static data with file names that match VFilePattern may be overriden being
 #   treated as volatile by making it match the special static data pattern,
-#   <SPfilePattern>.
-# - <SVfilePattern> or the "special volatile data" pattern is for the
+#   «SPfilePattern».
+# - «SVfilePattern» or the "special volatile data" pattern is for the
 #   convenience of specifying any exceptions to matches with SPfilePattern,
 #   for cases where data must still be treated as volatile.
-# - <WfilePattern> specifies a "whitelist pattern" for the regular expiration
+# - «WfilePattern» specifies a "whitelist pattern" for the regular expiration
 #   job, telling it to keep the files even if they are not referenced by
 #   others, like crypto signatures with which clients begin their downloads.
 #
@@ -269,8 +219,9 @@
 #
 # To see examples of the expected syntax, run: apt-cacher-ng -p debug=1
 #
-PfilePatternEx: .*yaml.gz$|.*fedora.*arch=x86_64$|.*f[0-9]+&arch=x86_64
-VfilePatternEx: .*fedora.*updateinfo.*xml.zck$|^/\?release=[0-9]+&arch=.*|.*/RPM-GPG-KEY.*|.*\?repo=fedora|.*pkg.tar.zst.sig
+PfilePatternEx: .*yaml.gz$|.*fedora.*arch=x86_64$|.*f37&arch=x86_64|.*f38&arch=x86_64|.*f39&arch=x86_64
+# VfilePatternEx:
+VfilePatternEx: .*fedora.*updateinfo.*xml.zck$|^/\?release=[0-9]+&arch=.*|.*/RPM-GPG-KEY.*|.*\?repo=fedora|.*pkg.tar.zst.sig|.*archlinux.*sha256sums.txt|.*archlinux/iso.*tar.gz.sig
 # SPfilePatternEx:
 # SVfilePatternEx:
 # WfilePatternEx:
@@ -326,13 +277,7 @@
 
 # Network timeout for outgoing connections, in seconds.
 #
-# NetworkTimeout: 40
-
-# Fast fallback timeout, in seconds. This is the time to wait before
-# alternative target addresses for a client connection are tried, which can be
-# usefull for quick fallback to IPv4 in case of whacky IPv6 configuration.
-#
-# FastTimeout = 4
+# NetworkTimeout: 60
 
 # Sometimes it makes sense to not store the data in cache and just return the
 # package data to client while it comes in. The following DontCache* parameters
@@ -358,7 +303,9 @@
 # details.
 #
 # Example:
-DontCache: .*fedora.*updates.*updateinfo.xml.zck .*fedora.*updates.*repomd.xml
+# DontCache: .*.local.university.int
+DontCache: .*fedora.*updates.*updateinfo.xml.zck  .*fedora.*updates.*repomd.xml
+#DontCache: .*fedora.*updates.*updateinfo.xml.zck
 
 # Default permission set of freshly created files and directories, as octal
 # numbers (see chmod(1) for details).
@@ -443,6 +390,7 @@
 # This restriction can be disabled by specifying a list of allowed ports or 0
 # for any port.
 #
+# AllowUserPorts: 80 443
 AllowUserPorts: 80 443
 
 # Normally the HTTP redirection responses are forwarded to the original caller
@@ -460,15 +408,12 @@
 
 # There some broken HTTP servers and proxy servers in the wild which don't
 # support the If-Range header correctly and return incorrect data when the
-# contents of a (volatile) file changed. This also applies to incomplete
-# resumed downloads.  Setting VfileUseRangeOps to 0 disables Range-based
-# requests (using purely If-Modified-Since and requesting the complete file
-# instead, if changed). Setting it to a negative value removes even this check
-# and means fetching the whole file from the beginning.
+# contents of a (volatile) file changed. Setting VfileUseRangeOps to zero
+# disables Range-based requests while retrieving volatile files, using
+# If-Modified-Since and requesting the complete file instead. Setting it to
+# a negative value removes even If-Modified-Since headers.
 #
 # VfileUseRangeOps: 1
-#
-# Syncthing server: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1053833
 VfileUseRangeOps: 0
 
 # Allow data pass-through mode for certain hosts when requested by the client
@@ -481,17 +426,20 @@
 #
 # Default: ^(bugs\.debian\.org|changelogs\.ubuntu\.com):443$
 # PassThroughPattern: ^(bugs\.debian\.org|changelogs\.ubuntu\.com):443$
+#PassThroughPattern: ^codecs\.fedoraproject\.org:443$|mirrors.rpmfusion.org:443
 PassThroughPattern: ^codecs\.fedoraproject\.org:443$
 
-# Interval an overaged local cache item (i.e. active file descriptor) can be
-# considered broken so that a new forced download can be started. Such
-# situation can happen when a very slow clients keeps a hot cache item active
-# for extended amounts of time so that even the remote freshness checks
-# intervals might become overrun.
-#
-# Default time is based on the value of FreshIndexMaxAge with a safety factor.
+# It's possible that an evil client requests a volatile file but does not
+# retrieve the response and keeps the connection effectively stuck over
+# many hours, blocking the particular file for other download attempts (which
+# leads to not reporting file changes on server side to other users). The work
+# around is the use of alternative file descriptors inside of apt-cacher-ng,
+# however this might cost some extra download traffic due to worse cache usage.
+# The ResponseFreezeDetectTime value specifies when a file descriptor in the
+# mentioned state is to be considered defect and will require special handling.
+# Default time is 500 seconds.
 #
-# ResponseFreezeDetectTime: 60
+# ResponseFreezeDetectTime: 500
 
 # Keep outgoing connections alive and reuse them for later downloads from
 # the same server as long as possible.
@@ -598,10 +546,3 @@
 # Set to zero to disable this feature completely. Default: one megabyte
 #
 # ReserveSpace: 1048576
-
-# PermitCacheControl will allow users to specify a few hints for processing
-# of a request, for example bypassing the local cache (see
-# https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control for
-# no-cache, no-store).
-#
-# PermitCacheControl: no-cache, no-store

Proposed solution

Keep track of upstream changes, and also do PR there so that the outcome (sys-cacher) works on QubesOS properly.

  • Fedora still problematic
  • mirror list need updating
  • advancements in regexes needs to be shared back upsteam
  • changes of cache behavior upstream needs to follow downstream

Upstream attempted to move this forward:

Discussions on sys-cacher are happening on QOS forum https://forum.qubes-os.org/t/apt-cacher-ng-and-fedora-cannot-prepare-internal-mirrorlist-status-code-403/22852 and elsewhere.

The value to a user, and who that user might be

Default template by default (next next next) on Q4.2.1 is still Fedora, and Fedora doensn't work out of the box.

`dom0.port-forward` installed by default in `dom0` Salt scripts but used only by `sys-syncthing`

Commitment

I confirm that I have read the following resources:

Current problem (if any)

dom0.port-forward script is installed by default in the init.sls's dom0 Salt installs, but seems to be currently used only by sys-syncthing, thus not necessary as a default install.

Proposed solution

Remove the port-forward line from init.sls. This is already present in syncthing's create script.
I also propose to install it in the fetcher salt scripts, because it can be necessary to port-forward when torrenting.

The value to a user, and who that user might be

Install only what's necessary when necessary.

Other jinja parsing errors: dom0 recipe

Commitment

I confirm that I have read the following resources:

Software version

4ac0ec9

Brief summary

dom0 recipe fails to apply

Steps to reproduce

See below

Expected behavior

Works

Actual behavior

Doesn't work:

[user@dom0 ~/QubesIncoming/heads-tests-deb12-nix/qusal(main)]
(123)$ git rev-parse HEAD 
4ac0ec9613ebc10b7e32ae4a6081b5ec45dd1bef
[user@dom0 ~/QubesIncoming/heads-tests-deb12-nix/qusal(main)]
$ sudo qubesctl top.enable dom0
sudo qubesctl state.apply
sudo qubesctl top.disable dom0
local:
    ----------
    dom0.top:
        ----------
        status:
            enabled
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
    output = template.render(**decoded_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/jinja2/environment.py", line 1291, in render
    self.environment.handle_exception()
  File "/usr/lib/python3.11/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 19, in top-level template code
  File "/usr/lib/python3.11/site-packages/jinja2/sandbox.py", line 391, in call
    if not __self.is_safe_callable(__obj):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/jinja2/sandbox.py", line 275, in is_safe_callable
    getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'load' is undefined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/utils/templates.py", line 218, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/utils/templates.py", line 482, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'load' is undefined; line 19

---
[...]
  - qubes-vm-update-if-stale: 4
  - qubes-vm-update-max-concurrency: 4
  - qubes-vm-update-restart-system: 1
  - qubes-vm-update-restart-other: 0
{%- endload %}
{{ load(defaults) }}    <======================

{% endif -%}
---
[CRITICAL] Rendering SLS 'base:dom0.update-settings' failed: Jinja variable 'load' is undefined; line 19

---
[...]
  - qubes-vm-update-if-stale: 4
  - qubes-vm-update-max-concurrency: 4
  - qubes-vm-update-restart-system: 1
  - qubes-vm-update-restart-other: 0
{%- endload %}
{{ load(defaults) }}    <======================

{% endif -%}
---
local:
    Data failed to compile:
----------
    Rendering SLS 'base:dom0.update-settings' failed: Jinja variable 'load' is undefined; line 19

---
[...]
  - qubes-vm-update-if-stale: 4
  - qubes-vm-update-max-concurrency: 4
  - qubes-vm-update-restart-system: 1
  - qubes-vm-update-restart-other: 0
{%- endload %}
{{ load(defaults) }}    <======================

{% endif -%}
---
DOM0 configuration failed, not continuing
local:
    ----------
    disabled:
        - dom0.top
    unchanged:
        - dom0.top
[user@dom0 ~/QubesIncoming/heads-tests-deb12-nix/qusal(main)]

documentation of a newbs journey to getting sys-audio/salt in general to work.

I am a complete newb to salt and only slightly less of a newb to linux in general. The problem with qubes is that once you have bought into the design philosophy of anonymity through isolation, no other OS even comes close to fulfilling its promise. So, if one doesn't wish to compromise ones own usage envelope one becomes a compelled to soldier on with an OS with a learning curve so steep that I often feel when I take a step forward I must also take two steps back..

Truly, Qubes is the eve-online of operating systems. But just because something is difficult doesn't make it not worth doing,

I have been trying to get audio to work for sometime so that I can get my bluetooth speakers/headset to work. I'd like to set up a Kodi qube, since starting to use qubes about 4 months ago I have needed transfer any content with sound to my windows laptop, and in doing so defeating the purpose of using qubes altogether. Ideally I'd like to stop using windows outside of virtual instancing, if at all.

After a lot of failure I found Ben's guide last night and thought it looked so simple even I'd be able to do it; sadly however, this was not to be the case. After emailing Ben for help he suggested I document my journey here so that others might learn from my learning experience too. As I run into difficulty, Ben will explain how to overcome each issue so that ultimately there is a roadmap that other Qube newbies (Quewbies?) can subsequently follow.

Beyond Ben' the other salt resources I have looked at are qubes salt beginners guide by leo, and Salt Configuration Management by Gonzalos.

For the ease of subsequent readers, I will state the CLI command given and then in a quotation box state the output spat out by Dom0.

qvm-port-forward QUBESRPC error

Hey there, thanks for the inspiration and contribution!
It helped me to understand saltstack better and getting my fresh 4.2 up and ready in no time!
However i stumbled into some trouble when using the qvm-port-forward command.
Using qvm-port-forward -a add -q sys-syncthing -n tcp -p 22000
throws me the following error:

info sys-sycthing: adding input rule daddr 10.137.0.56
Unknown command "QUBESRPC qubes.VMShell dom0" [while processing "QUBESRPC qubes.VMShell dom0"]

Is there just a policy missing and how would look such a policy that allows to use the port forwarding command from dom0?
How do i debug this or where do i have to look if the nessecary rules got created and how to remove this if the command isnt working but in case has created some rules....?
I couldnt find anything related to port forwarding in the "sys-net" qube

Sys-cacher depends on browser.install-common which is not found

Commitment

I confirm that I have read the following resources:

Software version

Latest

Brief summary

sys-cacher depends on browser which depends on browser-installed-chromium which depdends on browser.install-common which is not found

Steps to reproduce

Deploy sys-cacher after dom0, debian-minimal and fedors-minimal

Expected behavior

sys-cacher deployed successfully

Actual behavior

Half baked deployment requiring deinit to revert. browser is installed but doesn't talk to cacher through 8082 and updates cannot be downloaded. Also default uodatevm not changed in global config.

Create a script to sync from domU to dom0

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Syncing the project from the domU to dom0 can be achieved in multiple ways. The qvm-run command written in the project's README is the simplest as it doesn't require any package installation like git in dom0, but is a bit long to type each time.

Proposed solution

Create a script, for example copy-from-domU.sh that can be executed each time we want to sync the project. We will need to define in it the qube and file variables once.

The value to a user, and who that user might be

The command will still need to be executed manually the first time to move the project to dom0, but then the script makes it faster, and no need to install any package in dom0.

dom0 rsync state

Can the rsync client state be applied to dom0, in the event a user wished to backup the /etc/ directory in dom0 to the rsync server?

mgmt qube can't be used with fedora due to fedora major upgrades using new python versions

Software version

From my research of qubes-posts, affected fedora-32, fedora-33 and now currently affects fedora-39.

Brief summary

When Fedora releases a new version, there is history of them breaking salt-ssh due to new Python versions, meaning that to target a fedora-39, you need to have the management qube based on fedora-39.

Currently, the mgmt formula creates the mgmt qube based on debian-12-minimal, it works great for fedora-38, but not for fedora-39.

Steps to reproduce

Install the mgmt formula.
Try to target a fedora-39 qube.

Expected behavior

Salt works.

Actual behavior

Salt breaks due to incompatible python version and salt-ssh importing it. See linked upstream issue above.

It breaks every formula that depends on fedora templates, qubes-builder, and the template installation itself via the fedora or fedora-minimal formula, as the install.sls will want to update the package list.

  • Debian's Python: 3.11.2
  • Fedora's Python: 3.12.0

Solution

Requires fedora-39-minimal and fedora-39 (full) to be installed. The global preferencemanagement_dispvm has to be set to fedora-39, then we can create a management qube based on fedora-39-minimal called dvm-mgmt-fedora or make the fedora type itself the default mgmt template dvm-mgmt, so far, this hasn't caused problems for targetting Debian with Salt as far as I could find from issues and posts.

Setting the management_dispvm to the fedora-39 (full) is inevitable during installation, as the minimal version doesn't yet have the necessary salt packages and can't be targetted without getting a working fedora version, in this case, the bloat one.

After installation, the management_dispvm can be set again to dvm-mgmt-fedora, thus having a lightweight management stack for lower resource usage, lower attack surface and faster boot time.

Leaving the global preference with the debian based qube dvm-mgmt and the fedora qubes have the qvm-prefs management_dispvm set to dvm-mgmt-fedora is not an option, we would need to query all the fedora templates and set their prefs set.

Changing the default template of the dvm-mgmt to Fedora seems to makes sense because from the available templates, it has the newest Python versions and it is required on new Fedora major upgrades while the contrary is not true, debian-11 could target debian-12. Using separate management templates can make sense in the future in case other templates starts having a default Python version newer than Fedora.

Can't use split-gpg2 on Fedora 39 client and Debian 12 server

Software version

R4.2, Fedora 39, Debian 12.

$ gpg-agent --version

Fedora:

gpg-agent (GnuPG) 2.4.4
libgcrypt 1.10.2-unknown

Debian:

gpg-agent (GnuPG) 2.2.40
libgcrypt 1.10.1

Brief summary

Steps to reproduce

On the Qrexec policy, set the target of qubes.Gpg2 to a fedora based qube that has split-gpg2 installed.

On the client:

$ gpg -bsau KEYFPR README.md

Expected behavior

File signed successfully.

Actual behavior

Can't sign files when using Fedora client and Debian server.

gpg: WARNING: server 'gpg-agent' is older than us (2.2.40 < 2.4.4)
gpg: Note: Outdated servers may lack important security fixes.
gpg: Note: Use the command "gpgconf --kill all" to restart them.
gpg: problem with fast path key listing: IPC parameter error - ignored
gpg: skipped "KEYFPR": Unusable secret key
gpg: signing failed: Unusable secret key

It appears as just a warning, but after watching the debug log on the server split-gpg2, it is actually an error. When tested with a fedora based server, it works.

Resolution

Necessary to switch the origin template of tpl-sys-pgp to fedora-minimal instead of debian-minimal, therefore any recent or old client versions can work with sys-pgp.

For anyone that has already created tpl-sys-pgp, just set the template of sys-pgp to any other template, delete tpl-sys-pgp and run the installation steps of sys-pgp.

Install `qubes-core-agent-passwordless-root` by default in minimal templates

Commitment

I confirm that I have read the following resources:

Current problem (if any)

Minimal templates does not have qubes-core-agent-passwordless-root installed. This makes more difficult to update them when necessary (install new packages in templates, temporary add configurations in a dispVM for testing purpose, etc).

Is there a specific reason why it has not been done?

Proposed solution

As explained in the official QubesOS documentation, install qubes-core-agent-passwordless-root by default in debian-minimal and fedora-minimal.

The value to a user, and who that user might be

Easier control of VMs by having sudo privileges.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.