robotpy / robotpy-installer Goto Github PK

RobotPy installer program

License: MIT License

Python 98.74% Shell 1.26%

robotpy-installer's Introduction

RobotPy: Python for FRC cRIO Robot Controller

Author:	Peter Johnson, FRC Team 294
Author:	Dustin Spicuzza, FRC Team 2423/1418
Copyright:	Copyright © 2010, Peter Johnson, Ross Light, Dustin Spicuzza

About RobotPy

RobotPy is a distribution of Python intended to be used for the FIRST Robotics Competition. Teams can use this to write their robot code in Python, a powerful dynamic programming language.

Features

Python is simple to learn and easy to maintain.
RobotPy lets you reload code without restarting.
RobotPy provides access to the WPILib class library.
You don't need to use WindRiver (unless you're rebuilding RobotPy itself).

Installation for most people

Download the binary releases from our FIRSTForge site, unpack them, and run install.py. Download from: http://firstforge.wpi.edu/sf/projects/robotpy

Installation from Source using our build VM

Using the provided build virtual machine is the easiest way to build RobotPy from source. See build-vm/README.txt for instructions.

Installation from Source (Advanced Users Only)

Step 1: Install Python

Python 2.7.5 is known to work. Ensure that you add your python installation directory to your PATH variable.

Note: This step is required for SIP installation

Step 2: Install SIP v4.15.3:

The cmake build process currently checks for exact versions of SIP, as the generated code changes from release to release.

http://www.riverbankcomputing.com/static/Docs/sip4/installation.html

Note: Installation of SIP on Windows requires Visual Studio or MinGW installed, see the SIP build instructions for more details.

Step 3: Build the source tree

TODO: Update these instructions for Windows too?

The RobotPy build process uses cmake to build the RobotPy distribution. frcmake is a wrapper around cmake that sets the build environment up to build for the cRio.

RobotPy currently uses the GCC 4.8 compiler distributed at the following URL to build the binary distribution. Wind River may be used, but we haven't built it with Wind River, so you may run into trouble using it with frcmake.

http://firstforge.wpi.edu/sf/projects/c--11_toochain

Create a directory (the 'binary directory') outside of the RobotPy source tree, and run this:

$ frcmake /path/to/robotpy

Once you've done this, you can build the code:

$ make

To copy the build output to the 'dist' directory, run this command from your binary directory:

$ make install

To build a binary release, just run the following command and a zip file will be created in your build directory.

$ make package

Step 4: Robot Installation

From the 'dist' directory inside of your binary directory, run install.py

Alternatively, you can FTP the contents of the 'robot' directory to your cRio directly. However, install.py is easier.

Development tools

See the utilities directory for useful development tools that may make your RobotPy development experience easier.

The pyfrc python package is a recommended development package that provides unit testing and other capabilities for your robot code.

https://github.com/robotpy/pyfrc

Technical Overview

RobotPy is a packaging of a patched Python 3.2 interpreter (found in the RobotPy/Python subdirectory of the source code). All access to the WPILib is generated by a SIP interface, which is found in Packages/wpilib/sip/. When the robot is started, it initializes the Python interpreter and runs the file py/boot.py. From there, all responsibility is given to the boot.py script, which is referred to as the bootloader.

If boot.py ever exits (due to an exception, for example), the C++ code exits. The default boot.py simply exits on any user exception. If this happens, you can reboot easily via NetConsole by simply typing "reboot" followed by hitting the enter key. This is how code reloads are performed. As boot.py is written in Python, this behavior can be customized as desired.

Major Differences from standard Python

Several Python modules with large and/or incompatible dependencies removed, namely: curses, dbm, gdbm, tkinter, nis, ossaudiodev, resource, spwd, syslog, termios, audioop, bz2, crypt, grp, ssl, pwd, and mmap.

Licensing

A brief overview of licensing terms:

Python: PSF License
RobotPy (all other support code): MIT License

If you redistribute RobotPy and add other libraries, please include their licensing information here.

RobotPy

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

robotpy-installer's People

Contributors

Stargazers

Watchers

Forkers

czhao39 stmobo erikboesen auscompgeek crispybacon1999 shu-matic thetriplev connectionmaster martinrioux otherjon cwstryker

robotpy-installer's Issues

[BUG]: installer should delete all packages before deploy

Problem description

I did an upgrade, and..

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
robotpy-cscore 2024.0.0b4 requires pyntcore==2024.0.0b4, but you have pyntcore 2024.1.1.0 which is incompatible.
robotpy-cscore 2024.0.0b4 requires robotpy-wpinet==2024.0.0b4, but you have robotpy-wpinet 2024.1.1.0 which is incompatible.
robotpy-cscore 2024.0.0b4 requires robotpy-wpiutil==2024.0.0b4, but you have robotpy-wpiutil 2024.1.1.0 which is incompatible.
Successfully installed pyntcore-2024.1.1.0 robotpy robotpy-cli-2024.0.0 robotpy-hal-2024.1.1.0 robotpy-wpilib-utilities-2024.0.0 robotpy-wpimath-2024.1.1.0 robotpy-wpinet-2024.1.1.0 robotpy-wpiutil-2024.1.1.0 wpilib-2024.1.1.0

The problem is that my new project was just robotpy, and not cscore... so I'm left in a mostly broken state (well, if I use those packages anyways).

I think when the deploy process detects that things are out of sync, it should just delete all of the existing packages (except pip) and install them fresh (or maybe it's a setting? or it asks the user). The user should have done a sync before hand anyways, so it shouldn't fail?

The risk is that if it deletes everything, the user could be left with an unusable system if the install doesn't succeed, but I don't think there's any reasonable way we can avoid that given the disk space constraints on the rio.

Operating System

RoboRIO

Installed Python Packages

No response

Reproducible example code

No response

Move caches to single directory

It's a bit obnoxious to have three different files trash your desktop.

Distribute roboRIO wheels instead of ipks

The more I think about this, the better it sounds. Would solve #41 , make CI cross-compiles significantly easier to accomplish, and probably provide a more consistent user experience if done right.

The only real catch here is that we would need a way to know that a binary opkg requirement was needed for some particular package -- for example, cscore requires OpenCV, and scipy requires fortran/blas. I imagine the best way to do this is to publish some list (similar to the way Packages works) that lets robotpy-installer know about these dependencies and let it handle it automatically.

https://www.tortall.net/~robotpy/wheels/2020/roborio/ contains wheels for most of our binary wheels.
Creat a wheel building repository via the crossenv container, for things like scipy et al
Upgrade robotpy-installer to download the wheels via tortall
Remove existing opkg management features and hide them behind a different command
Add migration code to robotpy-installer to remove existing ipks and use wheels
Update documentation sites

Add test suite

Should be easy enough to create a fake ssh server using twisted or paramiko and run that on Travis...

When running an install, raise error that file cannot be found

To prevent problems with people running the installer from different directories.

[BUG]: sync should ask the user before downgrading packages

Problem description

We write pyproject.toml by default, so the user might not notice that it is even there. When they update robotpy manually, then do a sync, it will downgrade them which is not likely what they want. We should ask before continuing.

Reference: https://www.chiefdelphi.com/t/robotpy-2024-1-1-available/448118/7?u=virtuald

Operating System

Windows, MacOS, Linux, RoboRIO

Installed Python Packages

No response

Reproducible example code

No response

Does not print stderr channel

download-pip refuses to download pure-Python wheels

Noticed this when downloading wpilib-controller.

Make a way to deploy to a RoboRIO 2 SD card

Haven't given this a ton of thought yet. Should think about it?

Installer should use ETag so it doesn't download the opkg feeds over and over again...

The NI feed is particularly large, and could be annoying in a limited bandwidth environment.

Migrated from robotpy/robotpy-wpilib#167

Installation problems

New-ish mentor trying to migrate new-ish team to Python, so it's almost certainly my mistake, but hopefully you can point me in the right direction (and maybe there's some documentation to update).

From a brand-new Windows 11 VM, I install WPILib, Python (current 3.12 fails to install robotpy (errors with Path objects), so I'm using 3.11.6 now), Phoenix Tuner, and REV Hardware Client, then I upgrade pip and then run python3 -m pip install robotpy. (I get other problems trying the 2024.0.0b3 version, but that's another story.) Downloading and installing Python on the roboRIO works fine. But after I successfully download files with python3 -m robotpy_installer download robotpy, attempting to install them on the roboRIO yields a stack trace:

 [...]
16:07:40:691 INFO    : robotpy.installer   : -> RoboRIO 2 image version: 2023_v3.1
16:07:40:777 INFO    : robotpy.installer   : -> RoboRIO disk usage 499.6M/29.0G (2% full)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\robotpy_installer\__main__.py", line 4, in <module>
    main()
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\robotpy_installer\installer.py", line 748, in pip_install
    ssh.exec_cmd(" ".join(pip_args), check=True, print_output=True)
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\robotpy_installer\sshcontroller.py", line 94, in exec_cmd
    print(line, end="")
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 5-44: character maps to <undefined>

What am I doing wrong?

[BUG]: sync should run pip install in another console instead of os.execv

Problem description

It's a weird user experience, but our script has to exit. Printing out a message saying pip is doing its thing in another window will be better than what we have now.

Reference: https://www.chiefdelphi.com/t/robotpy-2024-1-1-available/448118/3?u=virtuald

Operating System

Windows

Installed Python Packages

No response

Reproducible example code

No response

robotpy-hal depends on new NI image

When installing certain libraries, need to somehow tie together wpilib releases and NI image versions. Right now it checks them independently.

I'm thinking maybe some metadata in pyproject.toml that can be extracted by the installer? Or maybe a special thing equivalent to python_requires.

[BUG]: deploy fails with socket.gaierror

Problem description

An error occurs that has socket.gaierror: [Errno 8] getaddrinfo failed in it.

I ran into this today on my team's laptop on Windows, but I've never ran into it on OSX/Linux. It kept selecting the *.local address, but didn't seem to have actually resolved it so it passes the DNS name to paramiko for the connection. When paramiko tries to use it, the connection fails for some reason.

If you look carefully at the _resolve_addr function, if none of the addresses are a SOCK_STREAM address then it will just return the original name. I'm not really sure why this is the case.

I don't have a Windows machine at home so I don't think I'll be able to diagnose this anytime soon. It would be great if someone else who has this issue could give the whole console output when running py robot.py deploy --debug.

Operating System

Windows

Installed Python Packages

No response

Reproducible example code

No response

[BUG]: Install fails if requirements.txt is saved as utf-16

Problem description

Looks like extra 0x00 shows up on the RIO side when the local requirements.txt isn't utf-8

PS C:\Users\chris\git\firstRobotPy> python -m robotpy_installer install -r requirements_rio.txt 
07:41:57:254 INFO    : robotpy.installer   : RobotPy Installer 2024.0.0b2
07:41:57:254 INFO    : robotpy.installer   : -> caching files at C:\Users\chris\wpilib\2024\robotpy
07:41:57:255 INFO    : robotpy.installer   : -> using existing config at 'C:\Users\chris\git\firstRobotPy\.installer_config'   
07:41:57:262 INFO    : robotpy.installer   : Finding robot for team 1736
07:41:57:312 INFO    : robotpy.installer   : -> Robot is at 10.111.76.141
07:41:57:313 INFO    : robotpy.installer   : Connecting to robot via SSH at 10.111.76.141
07:41:57:482 INFO    : paramiko.transport  : Connected (version 2.0, client OpenSSH_8.3)
07:41:57:890 INFO    : paramiko.transport  : Auth banner: b'NI Linux Real-Time (run mode)\n\nLog in with your NI-Auth credentials.\n\n'
07:41:57:892 INFO    : paramiko.transport  : Authentication (password) successful!
07:41:58:020 INFO    : robotpy.installer   : -> RoboRIO 2 image version: 2024_v1.1
07:41:58:148 INFO    : robotpy.installer   : -> RoboRIO disk usage 534.6M/3.4G (16% full)
07:41:58:409 INFO    : paramiko.transport.sftp: [chan 3] Opened sftp connection (server version 3)
07:41:58:426 INFO    : paramiko.transport.sftp: [chan 3] sftp session closed.
ERROR: Invalid requirement: 'ÿþ#\x00 \x00T\x00h\x00i\x00r\x00d\x00-\x00p\x00a\x00r\x00t\x00y\x00 \x00p\x00y\x00t\x00h\x00o\x00n\x00 \x00l\x00i\x00b\x00r\x00a\x00r\x00i\x00e\x00s\x00' (from line 1 of http://localhost:44255/requirements/requirements_rio.txt)
Error: installing packages: Command '/home/admin/rpip --no-cache-dir --disable-pip-version-check install --no-index --root-user-action=ignore --find-links http://localhost:44255/pip_cache/ --upgrade --upgrade-strategy=eager --pre -r http://localhost:44255/requirements/requirements_rio.txt' returned non-zero error status 1

Operating System

Windows

Installed Python Packages

Package                  Version
------------------------ ----------------
anyio                    3.6.2
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
astroid                  2.15.4
asttokens                2.2.1
attrs                    23.1.0
autopep8                 2.0.2
backcall                 0.2.0
bcrypt                   4.0.1
beautifulsoup4           4.12.2
black                    22.10.0
bleach                   6.0.0
cachetools               5.3.1
certifi                  2022.12.7
cffi                     1.15.1
charset-normalizer       3.1.0
click                    8.1.3
colorama                 0.4.6
comm                     0.1.3
coverage                 7.3.3
cryptography             40.0.1
debugpy                  1.6.7
decorator                5.1.1
defusedxml               0.7.1
dill                     0.3.6
executing                1.2.0
fastjsonschema           2.16.3
fqdn                     1.5.1
google-api-core          2.11.1
google-api-python-client 2.90.0
google-auth              2.21.0
google-auth-httplib2     0.1.0
google-auth-oauthlib     1.0.0
googleapis-common-protos 1.59.1
gspread                  5.9.0
httplib2                 0.22.0
idna                     3.4
iniconfig                2.0.0
ipykernel                6.22.0
ipython                  8.12.0
ipython-genutils         0.2.0
ipywidgets               8.0.6
isoduration              20.11.0
isort                    5.12.0
jedi                     0.18.2
Jinja2                   3.1.2
jsonpointer              2.3
jsonschema               4.17.3
jupyter                  1.0.0
jupyter_client           8.2.0
jupyter-console          6.6.3
jupyter_core             5.3.0
jupyter-events           0.6.3
jupyter_server           2.5.0
jupyter_server_terminals 0.4.4
jupyterlab-pygments      0.2.2
jupyterlab-widgets       3.0.7
lazy-object-proxy        1.9.0
MarkupSafe               2.1.2
matplotlib-inline        0.1.6
mccabe                   0.7.0
mistune                  2.0.5
mypy-extensions          1.0.0
nbclassic                0.5.5
nbclient                 0.7.4
nbconvert                7.3.1
nbformat                 5.8.0
nest-asyncio             1.5.6
notebook                 6.5.4
notebook_shim            0.2.3
oauth2client             4.1.3
oauthlib                 3.2.2
packaging                23.1
pandocfilters            1.5.0
paramiko                 3.1.0
parso                    0.8.3
pathspec                 0.11.0
photonlibpy              2024.1.1b32
pickleshare              0.7.5
Pint                     0.20.1
pip                      23.3.1
platformdirs             3.1.1
pluggy                   1.0.0
prometheus-client        0.16.0
prompt-toolkit           3.0.38
protobuf                 4.23.3
psutil                   5.9.5
pure-eval                0.2.2
pyasn1                   0.5.0
pyasn1-modules           0.3.0
pycodestyle              2.10.0
pycparser                2.21
pyfrc                    2024.0.0b1
Pygments                 2.15.1
pylint                   2.17.3
PyNaCl                   1.5.0
pynetconsole             2.0.4
pyntcore                 2024.0.0b3.post1
pyparsing                3.1.0
pyrsistent               0.19.3
pytest                   7.3.0
pytest-reraise           2.1.2
python-dateutil          2.8.2
python-json-logger       2.0.7
pywin32                  306
pywinpty                 2.0.10
PyYAML                   6.0
pyzmq                    25.0.2
qtconsole                5.4.2
QtPy                     2.3.1
regex                    2022.9.13
requests                 2.28.2
requests-oauthlib        1.3.1
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
robotpy                  2024.0.0b3
robotpy-apriltag         2024.0.0b3.post1
robotpy-commands-v2      2024.0.0b3
robotpy-cscore           2024.0.0b3.post1
robotpy-hal              2024.0.0b3.post1
robotpy-halsim-ds-socket 2024.0.0b3.post1
robotpy-halsim-gui       2024.0.0b3.post1
robotpy-halsim-ws        2024.0.0b3.post1
robotpy-installer        2024.0.0b2
robotpy-navx             2024.0.0b1
robotpy-pathplannerlib   2023.3.4.1
robotpy-rev              2024.0.0b1.post1
robotpy-wpilib-utilities 2024.0.0b1
robotpy-wpimath          2024.0.0b3.post1
robotpy-wpinet           2024.0.0b3.post1
robotpy-wpiutil          2024.0.0b3.post1
rsa                      4.9
Send2Trash               1.8.0
setuptools               65.5.0
setuptools-scm           8.0.4
six                      1.16.0
sniffio                  1.3.0
soupsieve                2.4.1
stack-data               0.6.2
tba-api-v3client         3.8.0
terminado                0.17.1
tinycss2                 1.2.1
tomlkit                  0.11.8
tornado                  6.3.1
traitlets                5.9.0
typing_extensions        4.8.0
uri-template             1.2.0
uritemplate              4.1.1
urllib3                  1.26.15
wcwidth                  0.2.6
webcolors                1.13
webencodings             0.5.1
websocket-client         1.5.1
wheel                    0.36.2
widgetsnbextension       4.0.7
wpiformat                2022.30
wpilib                   2024.0.0b3.post1
wrapt                    1.15.0

Reproducible example code

make a requirements.txt encoded as utf-16

run python -m python -m robotpy_installer install -r requirements.txt

Change installation to use a HTTP server via SSH tunnel

One annoying problem when running installs is that we have this complex logic for copying files over , ensuring that pip/opkg find our packages, and then cleaning up the files (which we don't do currently).

Instead, when we SSH to the roborio, we should open up a custom HTTP server on the local host that serves the opkg cache and the pip cache, that way pip/opkg can install from a 'network' source instead of copying the files over. I'd be willing to bet this would be faster to install also.

To do this we would need to move to paramiko -- but binary wheels are now available for this for all major platforms (unlike in 2015), so I think this would be an excellent time to switch. Additionally, by moving to paramiko, we should be able to execute commands in a much simpler and streamlined way than what we currently do.

One thing to be careful of is to never open up an actual HTTP server on the user's machine (eg, it needs to be a virtual server). In some school environments, users won't have admin privileges, so the local firewall may block such things.

[BUG]: RobotFinder hangs program until connections time out

Problem description

Probably should migrate to asyncio or use non-blocking sockets + select.

When I run the deploy-info command (which executes pretty quickly) it just sits there waiting for the connections to finish timing out.

Operating System

Linux

Installed Python Packages

No response

Reproducible example code

No response

Didn’t support for VMX-pi?

Didn’t support for VMX-pi?
I'm tried to running robotpy-installer for VMX-pi,but it's not work!

pip3 issues

pip 9.x downloads wheels instead of tar.gz of pure python packages, which can be a problem if the wrong python is being used. There's a platform flag you can give to pip 9
Need to disable pip's caching, we don't need it for this

Migrated from robotpy/robotpy-wpilib#249

opkg support breaks on PEP 440 prerelease versions

e.g. it thinks 2019.0.0a5 > 2019.0.0.

deploy does not catch incorrect robot.py filename case

On Windows (and presumably macOS by default), deploy happily copies Robot.py to the robot, causing Python to complain about the nonexistent robot.py on the robot.

Should document how to remove a package

Seems like the installer can't do this right now. Seems like it would be useful to give it the ability...

python needs setcap to allow RT stuff

See:

Installer requires the Packages file to work correctly

This makes sense for non-python3 packages. However, so that we don't distribute extra crap with our release, we should make it so that a specified package can be installed even without Packages present. If the user uses 'download', then it will be present anyways. If they delete it.. their own fault.

Migrated from robotpy/robotpy-wpilib#170

Package source 2021 failed

Hi, when I follow the instruction to install package on RoboRIO, I got the error below:

Downloading https://www.tortall.net/~robotpy/feeds/2021/Packages
Traceback (most recent call last):
  File "/Users/liyuhao/miniforge3/bin/robotpy-installer", line 8, in <module>
    sys.exit(main())
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/robotpy_installer/installer.py", line 489, in download_python
    ctx.forward(opkg_download, packages=[_ROBOTPY_PYTHON_VERSION])
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/core.py", line 784, in forward
    return __self.invoke(__cmd, *args, **kwargs)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/robotpy_installer/installer.py", line 289, in opkg_download
    opkg.update_packages()
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/robotpy_installer/opkgrepo.py", line 48, in update_packages
    _urlretrieve(pkgurl, feed["db_fname"], True)
  File "/Users/liyuhao/miniforge3/lib/python3.9/site-packages/robotpy_installer/utils.py", line 70, in _urlretrieve
    with contextlib.closing(urllib.request.urlopen(req)) as rfp:
  File "/Users/liyuhao/miniforge3/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/liyuhao/miniforge3/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Users/liyuhao/miniforge3/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Users/liyuhao/miniforge3/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/Users/liyuhao/miniforge3/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Users/liyuhao/miniforge3/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

It turns out that https://www.tortall.net/~robotpy/feeds/2021/Packages is 404. Could anyone take a look at this problem?

Smarter roborio connection

Should try the USB IP by default and see if that works.

Support requirement markers in pyproject.toml

I'd like to be able to add foo==0.1; platform_machine != "roborio" sorts of requirements. Currently when we sync to the robot it ignores any markers if specified, so users cannot use requires to install non-robot things into their environment during sync.

deploy fails

We're getting an error preventing us from deploying. All of the latest packages should be installed.

source venv/bin/activate && python robot.py deploy
  16:51:44:288 INFO    : wpilib              : WPILib version 2020.2.2.8
  16:51:44:288 INFO    : wpilib              : HAL version 2020.2.2.0
  16:51:44:288 INFO    : wpilib              : Running with simulated HAL.
  16:51:44:288 INFO    : faulthandler        : registered SIGUSR2 for PID 13059
  16:51:44:289 INFO    : robotpy.installer   : Finding robot for team 6672
  16:51:44:299 INFO    : robotpy.installer   : -> Robot is at 10.66.72.2
  16:51:44:299 INFO    : robotpy.installer   : Connecting to robot via SSH at 10.66.72.2
  16:51:44:714 INFO    : paramiko.transport  : Connected (version 2.0, client OpenSSH_7.6)
  16:51:45:233 INFO    : paramiko.transport  : Auth banner: b'NI Linux Real-Time (run mode)\n\nLog in with your NI-Auth credentials.\n\n'
  16:51:45:233 INFO    : paramiko.transport  : Authentication (password) successful!
  WPILib version on robot is 2020.2.2.8
  Traceback (most recent call last):
    File "robot.py", line 21, in <module>
      run(Green)
    File "/home/winston/Code/Python/2020-Green/venv/lib/python3.8/site-packages/wpilib/_impl/main.py", line 186, in run
    retval = options.cmdobj.run(options, robot_class, **kwargs)
  File "/home/winston/Code/Python/2020-Green/venv/lib/python3.8/site-packages/pyfrc/mains/cli_deploy.py", line 258, in run
    controller.ssh_exec_commands(sshcmd, True)
  File "/home/winston/Code/Python/2020-Green/venv/lib/python3.8/site-packages/robotpy_installer/sshcontroller.py", line 57, in ssh_exec_commands
    for line in iter(stderr.readLine, ""):
AttributeError: 'ChannelStderrFile' object has no attribute 'readLine'

Make it easier to install 'robotpy official' packages

Knowing the distinction between pip and opkg is good for advanced users, but vaguely confusing for less advanced users. We should provide a smarter package installation experience that requires less knowledge to do it right.

Replace pip.main() in installer

The installer script uses the older version of the pip package. Specifically, the code uses the pip.main() function, which has been removed since pip version 10.

A quick workaround would be the callable-pip package, a wrapper around pip that allows you to use the pip.main() syntax that's currently there. The long-term solution would be to either using subprocess to call pip or using import statements.

Make command error detection not suck

Previously it was a giant bash command because of limitations in the prior ssh. We should segment it so it's easier to diagnose issues.

What if we distributed the installer as a standalone exe?

... instead of distributing a packaged RobotPy zipfile? The install docs would include that as part of your introduction to RobotPy. Could use pyinstaller for this.

One advantage of this approach is that then teams wouldn't need to install python on their non-development computers.

Probably would need a GUI version at some point.

robotpy installer not working

New linux 18.04 LTS
New install of pyfrc after some work
New image of roborio

Command works

python3 robotpy_installer download-robotpy

Command fails

python3 robotpy_installer install-robotpy_
12:29:04:414 INFO    : robotpy.installer   : Finding robot for team 1432
12:29:05:499 INFO    : robotpy.installer   : -> Robot is at roboRIO-1432-FRC.local
12:29:05:499 INFO    : robotpy.installer   : Connecting to robot via SSH at roboRIO-1432-FRC.local
12:29:07:639 INFO    : paramiko.transport  : Connected (version 2.0, client OpenSSH_7.6)
12:29:07:844 INFO    : paramiko.transport  : Auth banner: b'NI Linux Real-Time (run mode)\n\nLog in with your NI-Auth credentials.\n\n'
12:29:07:845 INFO    : paramiko.transport  : Authentication (password) successful!
IMAGEVERSION = "FRC_roboRIO_2020_v10"
Downloading http://localhost:33215/opkg_cache/python38_3.8.1-r0_cortexa9-vfpv3.ipk.
**12:29:08:665 ERROR   : robotpy.installer   : Command** (IV=$(grep IMAGEVERSION /etc/natinst/share/scs_imagemetadata.ini); echo $IV; [ "$IV" == 'IMAGEVERSION = "FRC_roboRIO_2020_v10"' ] || (echo '-> ERROR: installer requires RoboRIO image 2020_v10! Use --ignore-image-version to force install' && /bin/false)) && echo 'set -e
PACKAGES=()
DO_INSTALL=0
if ! opkg list-installed | grep -F "python38 - 3.8.1-r0"; then
    PACKAGES+=("http://localhost:33215/opkg_cache/python38_3.8.1-r0_cortexa9-vfpv3.ipk")
    DO_INSTALL=1
else
    echo "python38 already installed"
fi
if ! opkg list-installed | grep -F "python38-robotpy-wpiutil - 2020.2.2.0"; then
    PACKAGES+=("http://localhost:33215/opkg_cache/python38-robotpy-wpiutil_2020.2.2.0_cortexa9-vfpv3.ipk")
    DO_INSTALL=1
else
    echo "python38-robotpy-wpiutil already installed"
fi
if ! opkg list-installed | grep -F "python38-pyntcore - 2020.2.2.2"; then
    PACKAGES+=("http://localhost:33215/opkg_cache/python38-pyntcore_2020.2.2.2_cortexa9-vfpv3.ipk")
    DO_INSTALL=1
else
    echo "python38-pyntcore already installed"
fi
if ! opkg list-installed | grep -F "python38-robotpy-hal - 2020.2.2.0"; then
    PACKAGES+=("http://localhost:33215/opkg_cache/python38-robotpy-hal_2020.2.2.0_cortexa9-vfpv3.ipk")
    DO_INSTALL=1
else
    echo "python38-robotpy-hal already installed"
fi
if ! opkg list-installed | grep -F "python38-wpilib - 2020.2.2.7"; then
    PACKAGES+=("http://localhost:33215/opkg_cache/python38-wpilib_2020.2.2.7_cortexa9-vfpv3.ipk")
    DO_INSTALL=1
else
    echo "python38-wpilib already installed"
fi
if [ "${DO_INSTALL}" == "0" ]; then
    echo "No packages to install."
else
    opkg install  ${PACKAGES[@]}
fi' > install_opkg.sh && bash install_opkg.sh && rm install_opkg.sh && ([ -x /usr/local/bin/pip3 ] || exit 87) && /usr/local/bin/pip3 install --no-index --find-links=http://localhost:33215/pip_cache/ --pre --upgrade --no-deps pynetworktables robotpy-wpilib-utilities **returned non-zero error status 255**

Connect your roboRIO to the internet

SSH in, and copy the following to /etc/opkg/robotpy.conf:

src/gz robotpy http://www.tortall.net/~robotpy/feeds/2020
Run this:

opkg update
( this is where it fails with no public key error )
opkg install python38-wpilib netconsole-host

Support opkg package verification

The 2020 roborio supports this, so our opkg repo needs to do that at a minimum.

Detect SSL errors and suggest certifi upgrade

Reference: https://www.chiefdelphi.com/t/robotpy-module-not-found/399156/2

I'm guessing that's their problem, as I believe pip uses certifi for certificates instead of the system store.

Remove generic IPK management

Plan is to download python ipk directly from github releases.

Uses pip.main(), removed in pip 10.0.0

I get this exception whenever I try downloading robotpy while using pip versions >=10.0.0. The only way to solve it that I've found (though, admittedly, given that I'm at a competition I've had scant time to try other solutions) is uninstalling pip and reinstalling version 9.x with easy_install.

$ robotpy-installer download-robotpy
18:06:17:470 INFO    : robotpy.installer   : RobotPy Installer 2018.0.2
18:06:17:471 INFO    : robotpy.installer   : -> caching files at /private/tmp
Downloading https://www.tortall.net/~robotpy/feeds/2018/Packages
100%
Downloading http://download.ni.com/ni-linux-rt/feeds/2017/arm/ipk/cortexa9-vfpv3/Packages
100%
Traceback (most recent call last):
  File "/usr/local/bin/robotpy-installer", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/robotpy_installer/installer.py", line 1129, in main
    retval = options.cmdobj(options)
  File "/usr/local/lib/python3.6/site-packages/robotpy_installer/installer.py", line 825, in download_robotpy
    return self.download_pip(self._create_rpy_pip_options(options))
  File "/usr/local/lib/python3.6/site-packages/robotpy_installer/installer.py", line 1022, in download_pip
    return pip.main(pip_args)
AttributeError: module 'pip' has no attribute 'main'

Add a free space detector/output when doing file operations

Warn users when it's really low?

Installer prefers python36-robotpy-cscore 2017.0.0rc3 over 2017.0.0

From a quick glance at the installer source, it looks like the installer will just look at the last instance of the package declared in the Packages file?

SSH should not attempt publickey auth

I get a bunch of passphrase prompts from my SSH agent when I deploy/install to a roboRIO.

ipk installation fails when there's a cycle

Because each package is installed individually, it fails. I don't quite remember why it's done individually, but it's a problem.

Reference: https://www.chiefdelphi.com/forums/showpost.php?p=1638824&postcount=5

Installer not working on newly imaged RoboRIO.

Here is the output when I run python -m robotpy_installer install-robotpy in Windows 10 when connected to a newly imaged RoboRIO:

20:31:22:613 INFO : robotpy.installer : RobotPy Installer 2017.1.2
20:31:22:613 INFO : robotpy.installer : -> caching files at C:\Users\czhao\OneDrive\Documents\frc2017-1123
20:31:24:161 INFO : robotpy.installer : Copying over the opkg cache...
20:31:24:172 INFO : robotpy.installer : Looking up hostname 'roborio-1123-frc.local'...
20:31:25:371 INFO : robotpy.installer : Connecting to robot via SSH at roborio-1123-frc.local
Remote working directory is /home/admin
local:C:\Users\czhao\OneDrive\Documents\frc2017-1123\opkg_cache\install_opkg.sh => remote:/home/admin/opkg_cache/install_opkg.sh
Using username "admin".
NI Linux Real-Time (run mode)

Log in with your NI-Auth credentials.

20:31:31:316 INFO : robotpy.installer : Copying over the pip cache...
20:31:35:291 INFO : robotpy.installer : Running installation...
20:31:35:291 INFO : robotpy.installer : Done.
netconsole-host - 1.0
netconsole-host already installed, continuing...
python36 - 3.6.0-r1
python36 already installed, continuing...
Collecting wpilib
Could not find a version that satisfies the requirement wpilib (from versions: )
No matching distribution found for wpilib
20:31:42:236 ERROR : robotpy.installer : Command '['C:\Users\czhao\OneDrive\Documents\frc2017-1123\venv\lib\site-packages\robotpy_installer\win32\plink.exe', '-pw', '', '[email protected]', 'bash opkg_cache/install_opkg.sh && /usr/local/bin/pip3 install --no-index --find-links=pip_cache --pre --upgrade wpilib robotpy-hal-base robotpy-hal-roborio robotpy-wpilib-utilities']' returned non-zero exit status 1

I get the same error when I run python -m robotpy_installer install-pip wpilib.

Check to see if pip is installed

On the RoboRIO, otherwise user gets an error message they don't understand.

https://www.chiefdelphi.com/t/robotpy-installation-error-cannot-connect-to-robot-via-ssh/341641/6?u=virtuald

Make the upgrade path easier

As we get more and more packages, knowing to upgrade them all is more problematic. Need a simple way that ideally 'just works' and doesn't cause users to miss updates.

First thought was to inspect the local ipk/pip cache and download new versions of those, then on install do the same.

One thought is that pyfrc should probably check all robotpy installed versions when deploying to the robot? Make sure local matches remote, etc.

Or, what about creating a dependency file of some sort, and operating on that? Then users can check it in, and we could sync it both locally and on the remote.

Add `make-offline` command

It would be pretty straightforward to do this. Could turn it into a self-executing python zip.

download-pip can take forever

User reported downloading robotpy-wpilib-utilities and it took forever because pip was trying to build wheels

Check for appropriate RoboRIO image before installing

Should probably do this...

print installer version in logs

Add package synchronization mechanism for local/remote python environment

Idea is something like this:

User specifies their dependencies in pyproject.toml
When pyfrc deploys, it ensures your remote environment matches your environment in pyproject.toml
robotpy-installer looks in your pyproject.toml to know what version to download to your offline cache, and optionally to your local installation too
- call the command 'sync'?

Vendors are TBD. Probably makes sense to keep vendors in vendordep jsons and integrate that with this somehow... but maybe not.

Python vscode extension would need to know how to update pyproject.toml to upgrade to a specific wpilib wheel. Might just make sense to have that be an installer command?

installer should validate the packages file before replacing it

Otherwise will leave user in broken state

Migrated from robotpy/robotpy-wpilib#254

robotpy / robotpy-installer Goto Github PK

robotpy-installer's Introduction

RobotPy: Python for FRC cRIO Robot Controller

About RobotPy

Features

Installation for most people

Installation from Source using our build VM

Installation from Source (Advanced Users Only)

Development tools

Technical Overview

Major Differences from standard Python

Licensing

RobotPy

robotpy-installer's People

Contributors

Stargazers

Watchers

Forkers

robotpy-installer's Issues

Problem description

Operating System

Installed Python Packages

Reproducible example code

Problem description

Operating System

Installed Python Packages

Reproducible example code

Problem description

Operating System

Installed Python Packages

Reproducible example code

Problem description

Operating System

Installed Python Packages

Reproducible example code

Problem description

Operating System

Installed Python Packages

Reproducible example code

Problem description

Operating System

Installed Python Packages

Reproducible example code

Recommend Projects

Recommend Topics

Recommend Org