Git Product home page Git Product logo

iricigor / psaptgetupdate Goto Github PK

View Code? Open in Web Editor NEW
9.0 2.0 1.0 802 KB

Provide functionality for PowerShellGet inspired by Linux command apt-get update. You can 100 times faster search for online modules, scripts, updatable modules, etc.

License: MIT License

PowerShell 100.00%
powershellget apt-get apt-get-update package-management local-caching powershell-gallery linux-command fast update find

psaptgetupdate's Introduction

PowerShell's apt-get update

This module provides functionality for PowerShellGet inspired by Linux command apt-get update. You can 100 times faster search for online modules, scripts, updatable modules, etc.

This is only proof-of-concept work, though it is (almost) fully functional.

Explanation

Currently PowerShell and PowerShellGet are having following two issues\feature requests which are related to each other:

  • Finding information about modules and scripts is very slow. It is running over the Internet and last about two seconds. This implementation speeds search commands 100 times, downto 20 ms!
  • When user types command which they do not have installed on the system, we can instruct them how to install it in some cases.

Core of the module is local index (cache) of all modules, scripts and commands which is just simply downloaded from the repository. Generated index file is prepared (generated and zipped) on a dedicated cloud VM, but command for that is also included in the module. For verification, it can be also executed locally.

It would be good to implement this functionality in PowerShellGet and PowerShell Gallery directly.

For more details, see (illustrated) design page.

Speed comparison

Actual speed will depend on your local system, namely of a disk and processor speed. Results below are from medium class personal computer. On low end machines, results will not be more than 2-3 times slower, which is still 20-50 times faster than already existing commands.

If further speed improvement is needed, simple indexing can increase speed a few times.

Results for 10 repetitions (in seconds)

10x executed Find-Command Find-Script Find-Module
old commands 22.9751159 23.0243 23.4820757
new commands 0.2507481 0.188659 0.1713624

How testing was done

Testing was done executing commands similar to the ones below. Test was repeated 5 times and middle result was recorded.

Measure-Command {1..10 | % {Find-Command 'Read-Credential' -Repository 'PSGallery'}} | Select TotalSeconds
Measure-Command {1..10 | % {Find-CommandFromCache 'Read-Credential'}} | Select TotalSeconds

Error handling

Module is also updating default CommandNotFoundAction error handler. As it is running fast, if it can, it will tell you how to add requested command! Just like in some Linux distros.

Error Handling

How to install

You can install this module in a couple of ways listed below.

  • From PowerShell Gallery (preferred way):

Module can be installed from PS Gallery using the command

Install-Module psaptgetupdate -Scope CurrentUser
  • Clone repository:

If you want to see the entire GitHub repository, just clone it and import module afterwards.

git clone https://github.com/iricigor/psaptgetupdate.git
Import-Module .\psaptgetupdate.psd1 -Force

Commands

This module provides following commands:

Cache Management

  • Update-PSRepositoryCache (alias PSAptGetUpdate)- Downloads index file and expands it to local cache, equivalent to apt-get update Linux command.
  • New-PSRepositoryCache - Generates zipped index file and uploads it to storage account. This is running as a scheduled task on dedicated server. Standard users do not need to run it.

Search and Update Operations

  • Find-ModuleFromCache - finds modules in local cache
  • Find-ScriptFromCache - finds scripts in local cache
  • Find-CommandFromCache - finds command in local cache
  • Update-ModuleFromCache - finds updatable modules in local cache, it accepts wildcard '*'; for POC it supports -WhatIf simulation only

Examples

Update-Index

Update-PSRepositoryCache

Creates local repository cache. Internet connection is required. Should be run first, before other examples. Command runs for about 4-5 seconds. If executed with -Verbose switch, you may actually see how old is server version of index file.

Find-Command

Find-CommandFromCache 'Get-Folder' | Select -First 3
ModuleName                Name       Version         Repository
----------                ----       -------         ----------
VMware.VimAutomation.Core Get-Folder 11.0.0.10336080 PSGallery
WFTools                   Get-Folder 0.1.58          PSGallery
PSFolderSize              Get-Folder 1.6.3           PSGallery

This command runs in about 20 milliseconds, which is about 100 times faster than standard Find-Command.

Update all modules

If called without any names, this command will update all modules in the system. In this POC, actual update is not implemented, so you need to run it with -Verbose or -WhatIf to see actual updatable modules.

Please note from verbose output that commandlet in this mode is processing dozens of modules per second! Standard commandlet Update-Module takes about 15 second to check for update of one module.

Update-ModuleFromCache -Verbose
VERBOSE: 31-Oct-18 10:47:50 PM Update-ModuleFromCache starting
VERBOSE: 10:47:50 PM Reading list of all modules from the system
VERBOSE: 10:48:12 PM checking module ClipboardText for updatable version
VERBOSE: 10:48:12 PM Performing action Update to version 0.1.7 on target Module 'ClipboardText' version 0.1.1
VERBOSE: Performing the operation "Update to version 0.1.7" on target "Module 'ClipboardText' version 0.1.1".
VERBOSE: 10:48:12 PM checking module EasyAzureFunction for updatable version
VERBOSE: 10:48:12 PM Performing action Update to version 0.7.1 on target Module 'EasyAzureFunction' version 0.6
VERBOSE: Performing the operation "Update to version 0.7.1" on target "Module 'EasyAzureFunction' version 0.6".
VERBOSE: 10:48:12 PM checking module fifa2018 for updatable version
VERBOSE: 10:48:12 PM Performing action Update to version 0.2.45 on target Module 'fifa2018' version 0.1.11
VERBOSE: Performing the operation "Update to version 0.2.45" on target "Module 'fifa2018' version 0.1.11".
VERBOSE: 10:48:12 PM checking module Plaster for updatable version
VERBOSE: 10:48:12 PM checking module platyPS for updatable version
VERBOSE: 10:48:12 PM Performing action Update to version 0.12.0 on target Module 'platyPS' version 0.11.1
VERBOSE: Performing the operation "Update to version 0.12.0" on target "Module 'platyPS' version 0.11.1".

External Links

Build status

Each commit or PR to master is checked on Azure DevOps Pipelines on two build systems:

  1. Ubuntu Linux v.16.04 LTS host running PowerShell (Core) v.6.1 Windows Build Status
  2. Windows host running Windows PowerShell v.5.1 Windows Build Status

Support

You can chat about this commandlet via Skype (no Skype ID required), by clicking a link below.

chat on Skype

Contributing

If you find any problems, feel free to open a new issue.

GitHub open issues GitHub closed issues

If you want to contribute, please fork the code and make a new PR after!

GitHub GitHub top language

Module icon

Module icon represents faster update symbol in PowerShell similar colors.

Fast Update icon

psaptgetupdate's People

Contributors

iricigor avatar trucido avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

trucido

psaptgetupdate's Issues

Update-ModuleFromCache to return strings only

Add parameter to the command Update-ModuleFromCache that will return only list of module names that can be updated. Then, you can use something like

Update-ModuleFromCache -NamesOnly | Update-Module

Publish version 1.0

Checklist:

  • commit code with updated number
  • publish to PS Gallery
  • create release in GitHub

Add module name hashing

This gives good distribution and runs on average below 2ms

function Hash ([string[]]$Name) {
    foreach ($N1 in $Name) {
        $Sum = $N1.ToCharArray() | % {[byte]$_} | Measure-Object -Sum
        [char]([int]($Sum.Sum % 26)+64)
    }
}

add speed comparison graph from Excel

Use current speed results from home/home, and also laptop/office
Use inverse values (i.e. operations per second), so that my module has bigger columns

Add help system

Synopsis and Docs folder

This should give nice output

gcm -module psaptgetupdate | % {help $_ -full}

Checklist:

  • draft md files in Docs folder
  • populate all missing fields
  • add online version for commands
  • add online xml help
  • add documentation test

add icon

combine fast forward and refresh

file locked handling

if running two psaptgetupdate processes in parallel, you can get locked files issues in:

  • downloading file
  • extracting file

There should be some error handling around it.

Ensure module/script version is OK when creating index

Cannot compare "@{Major=4; Minor=4; Build=2; Revision=-1; MajorRevision=-1; MinorRevision=-1}" to "4.4.2" because the objects are not the same type or the object "@{Major=4; Minor=4; Build=2; Revision=-1; MajorRevision=-1; MinorRevision=-1}" does not implement "IComparable".

Find-CommandFromCache returns all the commands which start with given string

from @edyoung

Hi Igor, thanks very much for showing how much this can help. There's a lot of potential! We'll need to give it a bit of thought to see how best to incorporate in PSGet - things like what to do with multiple repositories. I noticed that Find-CommandFromCache get-azvm returns all the commands which start with get-azvm rather than just 'get-azvm', which is likely an easy change.

age test throws an error on first run

Get-Item : Cannot find path 'C:\Users\xxx\AppData\Local\PSGalleryIndex\Commands.Cache' because it does not exist.
At C:\Users\xxx\Documents\WindowsPowerShell\Modules\psaptgetupdate\1.0\Public\Update-PSRepositoryCache.ps1:20 char:37
+         $Age = [int](((Get-Date) - (Get-Item $IP.Commands).LastWriteT ...
+                                     ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Users\xxx...\Commands.Cache:String) [Get-Item], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand

double cache update

If user executes psaptgetupdate when module is not loaded, module will be automatically loaded (default ps behaviour). But, this loading will trigger psaptgetupdate also, so actually command will run twice. Not a big issue, as it runs for 5 seconds, but still reconsidering the logic is needed

Add Linux build and Publish test results

Checklist

  • Setup Linux build
  • List all the issues from the build
  • Add publish tests task
  • Fix all the issues
  • Update readme with new build status
  • Enable CI/PR integration for new build

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.