Git Product home page Git Product logo

studip-fuse's Introduction

Stud.IP FUSE driver

studip-fuse is a FUSE (file-system in user-space) driver that provides files from lectures in the course management tool Stud.IP on your computer.

studip-fuse uses the official Stud.IP REST API and authenticates via OAuth1, which will open a prompt in your browser on the first start. Password-based login via the standard Stud.IP login (using HTTP Basic Auth --login-method basic) or Shibboleth (--login-method shib) is also possible. All connections to the university servers transporting the login data are made via HTTPS and your credentials will not be copied or distributed in any other way.

Installation (on Ubuntu 18.04)

Install the debian packages python3 (providing binary python3 version ≥ 3.6), python3-pip (providing binary pip3 version ≥ 9) and fuse on your system via apt:

sudo apt install python3 python3-pip fuse

And install the python package studip-fuse from GitHub for your current user.

pip3 install git+https://github.com/N-Coder/studip-fuse --user

Now you can try mounting your Stud.IP files (optionally pointing --studip-url to the api.php endpoint of your Stud.IP instance and providing the appropriate --login-method):

mkdir Stud.IP
studip-fuse mueller123 ~/Stud.IP --pwfile=- --foreground

To mount the drive you can also use the standard mount tool if you installed studip-fuse globally:

$ sudo -i
# pip3 install git+https://github.com/N-Coder/studip-fuse
# mount -t fuse -o allow_other,uid=1000,gid=1000 "studip-fuse#mueller123" /home/user/Stud.IP

If you're running the driver in the background (i.e. by not passing the --foreground option), all status messages (e.g. warnings about an invalid password) will be sent to the system log running under /dev/log. You can unmount the folder and kill the driver the usual way:

# umount /home/user/Stud.IP # or, alternatively also with user rights:
$ fusermount -u ~/Stud.IP

To display file status information emblems and add an "Open on Stud.IP" option menu entry in Nautilus, run the following command to install the plug-in:

$ studip-fuse-install-nautilus-plugin
Checking requirements...
Installing studip-fuse Nautilus extension to /home/user/.local/share/nautilus-python/extensions...
Copying script source code to /home/user/.local/share/nautilus-python/extensions/studip_fuse_nautilus_plugin.py...
Done installing, please restart Nautilus to enable the plugin.

Supported Environments

This program has been tested on

  • Ubuntu 18.04 using Python 3.6.7 and pip 9.0.1
  • Fedora 29 using Python 3.7.2 and pip 18.1

using the Stud.IP instance of University of Passau (https://uni-passau.de/studip) and the Stud.IP Developer Instance (http://develop.studip.de/studip/).

Ubuntu 16.04 is no longer officially supported, because it only ships python 3.5.2, but aiohttp requires ≥ 3.5.3.

Login Methods

Currently, three different methods for logging in to Stud.IP are supported via --login-method:

`basic`
This can be used if your Stud.IP instance uses the built-in authentication system. Use this, if your login page looks like this and/or the Basic Auth dialog popping up when you open the `api.php` URL accepts your credentials.
`shib`
This can be used if your Stud.IP instance uses Shibboleth Single Sign-On. Use this, if your login page looks like this. Please note that this login method parses the HTML responses of the Shibboleth server that corresponds to `--shib-url`, so things might break.
`oauth`
This should be available for any Stud.IP instance, as long as you registered an OAuth application with the administrator of the instance and provide the appropriate `--oauth-client-key`, or your instance provided built-in client keys in studip_fuse/launcher/oauth_tokens.py.

Required Routes

The following routes need to be available from the API defined by --studip-url:

  • discovery,
  • user,
  • studip/settings,
  • studip/content_terms_of_use_list,
  • studip/file_system/folder_types,
  • extern/coursetypes,
  • semesters,
  • user/:user_id/courses,
  • course/:course_id/top_folder,
  • folder/:folder_id,
  • file/:file_ref_id,
  • file/:file_ref_id/download

The list is also checked at every startup, see REQUIRED_API_ENDPOINTS in studip_fuse/studipfs/api/session.py. If any of the routes is not available and a HTTP error 403 "route not activated" is returned, please contact the administrators of your Stud.IP instance.

Installation Options

Any of the following commands should work interchangeably for installing Stud.IP-FUSE

pip3 install --user git+https://github.com/N-Coder/studip-fuse
pip3 install --user --editable git+https://github.com/N-Coder/studip-fuse

or after a git clone of this repository and cding to that directory

pip3 install --user .
pip3 install --user --editable .
python3 ./setup.py install --user
python3 ./setup.py develop --user

If you think anything regarding your installation broke, try running pip3 uninstall studip-fuse until the package is no longer found and then reinstalling.

Command-line options

$ studip-fuse -h
usage: studip-fuse [-h] [-o O [O ...]] [-d] [-v] [--debug-aio] [--format FORMAT] [--cache-dir CACHE_DIR]
                   [--studip-url STUDIP_URL] [--login-method {shib,oauth,basic}] [--pwfile PWFILE] [--shib-url SHIB_URL]
                   [--oauth-client-key OAUTH_CLIENT_KEY] [--oauth-session-token OAUTH_SESSION_TOKEN] [--oauth-no-login]
                   [--oauth-no-browser] [--oauth-no-store] [-f] [-s] [--allow-other] [--allow-root] [--nonempty]
                   [--umask UMASK] [--uid UID] [--gid GID] [--default-permissions] [--debug-fuse] [-V]
                   user mount

studip-fuse is a FUSE (file-system in user-space) driver that provides files from lectures in the course management
tool Stud.IP on your computer.

positional arguments:
  user                  Stud.IP username
  mount                 path to mount point

optional arguments:
  -h, --help            show this help message and exit
  -o O [O ...]          FUSE-like options (default: None)
  -d, --debug           turn on all debugging options (default: False)
  -v, --debug-logging   turn on debug logging (default: False)
  --debug-aio           turn on asyncio debug logging (default: False)
  -V, --version         show program's version number and exit

Stud.IP Driver Options:
  --format FORMAT       format specifier for virtual paths
                        (default: {semester-lexical}/{course-class}/{course}/{course-type}/{short-path}/{file-name})
  --cache-dir CACHE_DIR, --cache CACHE_DIR
                        path to cache directory (default: /home/user/.cache/Stud.IP-Fuse)
  --studip-url STUDIP_URL, --studip STUDIP_URL
                        Stud.IP API URL (default: https://studip.uni-passau.de/studip/api.php/)

Authentication Options:
  --login-method {shib,oauth,basic}
                        method for logging in to Stud.IP session (default: oauth)
  --pwfile PWFILE       path to password file or '-' to read from stdin (for 'basic' and 'shib' auth)
                        (default: /home/user/.config/Stud.IP-Fuse/.studip-pw)
  --shib-url SHIB_URL, --sso SHIB_URL
                        Stud.IP SSO URL (default: https://studip.uni-passau.de/studip/index.php?again=yes&sso=shib)
  --oauth-client-key OAUTH_CLIENT_KEY
                        path to JSON file containing OAuth Client Key and Secret
                        (default: [internal key for given Stud.IP instance])
  --oauth-session-token OAUTH_SESSION_TOKEN
                        path to file where the session keys should be read from/stored to
                        (default: /home/user/.config/Stud.IP-Fuse/.studip-oauth-session)
  --oauth-no-login      disable interactive OAuth authentication when no valid session token is found (default: False)
  --oauth-no-browser    don't automatically open the browser during interactive OAuth authentication (default: False)
  --oauth-no-store      don't store the new session token obtained after logging in (default: False)

FUSE Options:
  -f, --foreground      run in foreground (default: False)
  -s, --nothreads       single threads for FUSE (default: False)
  --allow-other         allow access by all users (default: False)
  --allow-root          allow access by root (default: False)
  --nonempty            allow mounts over non-empty file/dir (default: False)
  --umask UMASK         set file permissions (octal) (default: None)
  --uid UID             set file owner (default: None)
  --gid GID             set file group (default: None)
  --default-permissions enable permission checking by kernel (default: False)
  --debug-fuse          enable FUSE debug mode (includes --foreground) (default: False)

Option format

Options can either be specified using --key=value or -o key=value, so the following to lines are identical regarding the option values:

studip-fuse mueller123 /home/user/Stud.IP --allow_root --uid=1000 --gid=1000
mount -t fuse -o allow_root,uid=1000,gid=1000 "studip-fuse#mueller123" /home/user/Stud.IP

Path formatting options

You can use the following values in the format string for the generated paths:

path
path to the file, relative to the root folder of the course
Example: Hauptordner/Skript/PDF-Versionen
short-path
path to the file, relative to the root folder of the course, stripped from common parts
Example: Skript/PDF-Versionen
semester
full semester name
Example: Wintersemester 2018-2019
semester-id
system-internal hexadecimal UUID of the semester
Example: 36bd96b432c1169978c1594d2251e629
semester-lexical
full semester name, allowing alphabetic sorting
Example: 2018 WS -19
semester-lexical-short
shortened semester name, allowing alphabetic sorting
Example: 2018WS
semester-short
shortened semester name
Example: WS 18-19
course
official name of the course, usually excluding its type
Example: Algorithmen und Datenstrukturen
course-abbrev
abbreviation of the course name, generated from its initials
Example: AuD
course-class
type of the course (teaching, community,...)
Example: Lehre
course-description
optional description given for the course
course-group
user-assigned (color-)group of the course on the Stud.IP overview page
Example: 7
course-id
system-internal hexadecimal UUID of the course
Example: eceaf9871792e0339797d1be91f9015d
course-location
room where the course is held
course-number
number assigned to the course in the course catalogue
Example: 5200
course-subtitle
optional subtitle assigned to the course
course-type
type of the course (lecture, exercise,...)
Example: Vorlesung
course-type-short
abbreviated type of the course, usually the letter appended to the course number in the course catalogue
Example: V
file-description
optional description given for the file
Example: Kapitel 1
file-downloads
number of times the file has been downloaded
Example: 3095
file-id
system-internal hexadecimal UUID of the file
Example: 8556e68de68b5e33d8d4572057431233
file-mime-type
file's mime-type detected by Stud.IP
Example: application-pdf
file-name
(base-)name of the file, including its extension
Example: A+D141.pdf
file-size
file size in bytes
Example: 3666701
file-storage
how the file is stored on the Stud.IP server
Example: disk
file-terms
terms on which the file might be used
Example: SELFMADE_NONPUB

You can combine these formatting options in any way you like, e.g.:

studip-fuse mueller123 ~/Stud.IP --format="{semester-lexical-short}/{course-abbrev} {type-abbrev}/{short-path}/{author} - {name}"

Not all combinations have been tested, if you encounter any problems with a (sensible) combination, please open a bug report. Please note that depending on your path format, generating folder listings could become very slow. For example using the format "{course}/{semester-lexical-short} {type-abbrev}/{short-path}/{name}" would require listing all your courses from all your semesters, which might take a while.

Further information on files

To get more information on the files in your Stud.IP folder, have a look at their xargs:

$ attr -l '~/Stud.IP/2014 SS/Lehre/Algorithmen und Datenstrukturen/Vorlesung/Skript/PDF-Versionen/A+D141.pdf'

The following keys are available:

"studip-fuse.known-tokens"
JSON-object with all the tokens mentioned above and their corresponding values for the respective file
"studip-fuse.json"
big JSON-object with *all* the information we got from the Stud.IP REST API
"studip-fuse.contents-status"
string indicating whether the contents of the file or folder are available yet:
pending, available, failed, unknown or unavailable
"studip-fuse.contents-excpetion"
exception that prevented the contents of the file or folder from being loaded
"studip-fuse.url"
absolute URL to the object in the Stud.IP web interface

Modes of Operation

This driver obeys the Unix philosophy of doing one thing well and working together with other programs. Advanced features, for which generic solutions already exists, haven't been implemented redundantly to keep the program simple.

For this reason, the following details are design-decisions and no bugs:

  • the whole mount is read-only, so no modification to the downloaded files is possible.
  • all information is cached in a static, but lazy way. This means that once you open a folder or file, its contents will be loaded once and stay the same throughout the whole lifetime of the program, even if they are changed online. To load the updated information, you need to restart the driver. Note: If a file didn't change online, its previously downloaded version may still be reused. Directories will always be loaded anew.
  • the driver needs to have a working internet connection to load new directories or files. Making already loaded files and folder persistently available when working offline is not guaranteed.
  • when mounting in background mode, problems with the Stud.IP API (e.g. wrong password) will only be detected after the program forks to background. This problem will be reported to the syslog and the program exits.

...but there are existing tools that fix this peculiarities:

  • to keep all files available for offline use, just use a standard synchronization tool like rsync (example config) or syncthing (with the driver running on your server).
  • to update information that has changed online, mount studip-fuse via autofs, so that it will be unmounted automatically once you don't need it anymore. Once you access the folder again, the driver will be started anew and load the new information.
  • to make local changes to the files, use overlayfs to make the read-only studip-fuse filesystem writeable by storing the changes in a separate location (example config). This also enables you to delete (i.e. hide) and rename files and folders, while renamed entities will still update their contents when they are changed online.
  • to wait for successful startup, check the file studip-status.txt in the user_data_dir, which will be appended once the driver completed starting up. See here on how to use use tail and grep for this.

Alternative Implementations

studip-fuse's People

Contributors

ein-stein-chen avatar n-coder avatar n-st avatar thomasleister avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

n-st zahedbri

studip-fuse's Issues

Doesn't create missing config directory

[user@pc ~]$ studip-fuse user ~/StudIP --foreground
Starting Studip-Fuse 3.1 release (installed from revision 83a639f) with FUSE 2.9 (libfuse.so.2) running via CPython 3.7.3 on Linux-5.0.13-arch1-1-ARCH-x86_64-with-arch
OAuth Session invalid, starting log-in flow
Go to the following address to log in via OAuth1: http://127.0.0.1:17548/request
main() function quit exceptionally
Traceback (most recent call last):
  File "/home/user/.local/venvs/studip-fuse/lib/python3.7/site-packages/studip_fuse/launcher/main.py", line 98, in main
    login_oauth_args(args)
  File "/home/user/.local/venvs/studip-fuse/lib/python3.7/site-packages/studip_fuse/launcher/main.py", line 66, in login_oauth_args
    with open(args.oauth_session_token, "wt") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/.config/Stud.IP-Fuse/.studip-oauth-session'

Creating the folder manually fixes the problem.

Run command when mount becomes available

Enhancement: Add an option to run a command when studip-fuse succeeds in mounting.

This would allow the user to correctly automate mounting the FUSE, rsyncing files to local storage, and unmounting the FUSE afterwards.

Allow mounting from fstab with `user` option

It would be useful to be able to add a studip-fuse mount to /etc/fstab and mount it on demand (through the noauto and user mount options):

studip-fuse#doe123    /home/johndoe/StudIP.fuse    fuse    noauto,user,ro,allow-other,uid=1000,gid=1000,pwfile=/home/johndoe/.config/studip-fuse/.studip-pw

Problem: Unfortunately, this is impossible at the moment: the user option "implies the options noexec, nosuid, and nodev" (see man 8 mount), which causes studip-fuse to abort:

studip-fuse: error: unrecognized arguments: --noexec

Solution: Since studip-fuse doesn't produce virtual files with executable, suid, or device attributes anyway, both the positive (--exec) and negative (--noexec) can be simply ignored without causing any problems.

studip-fuse fails to start: No module named 'studip_fuse.launcher.aioimpl'

studip-fuse fails to start after installing it in a virtualenv with the pip command from the README:

% python3 -m venv /home/nst/bin/py/studip-fuse
% source /home/nst/bin/py/studip-fuse/bin/activate
% pip3 install git+https://github.com/N-Coder/studip-api git+https://github.com/N-Coder/studip-fuse
…
% studip-fuse mueller123 /home/nst/StudIP.fuse --pwfile=- --foreground
Traceback (most recent call last):
  File "/home/nst/bin/py/studip-fuse/bin/studip-fuse", line 11, in <module>
    load_entry_point('studip-fuse==3.0', 'console_scripts', 'studip-fuse')()
  File "/home/nst/bin/py/studip-fuse/lib/python3.7/site-packages/pkg_resources/__init__.py", line 487, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/nst/bin/py/studip-fuse/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2728, in load_entry_point
    return ep.load()
  File "/home/nst/bin/py/studip-fuse/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2346, in load
    return self.resolve()
  File "/home/nst/bin/py/studip-fuse/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2352, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/nst/bin/py/studip-fuse/lib/python3.7/site-packages/studip_fuse/launcher/main.py", line 5, in <module>
    import studip_fuse.launcher.aioimpl.asyncio as aioimpl_asyncio
ModuleNotFoundError: No module named 'studip_fuse.launcher.aioimpl'

This is using Python 3.7.2 on Arch Linux.

Parse further information from Stud.IP

  • File creation date
  • parse Course list from "Farbgruppierung"
    • this requires further requests to get information for all courses
  • VirtualPath._known_tokens should also return file name and extension as separate fields
  • provide file extension / MIME-type if not specified
  • check length specified by StudIP

The optimal solution would be a generic method for generating listings of a certain group of objects and then querying further details about these objects (if they weren't included in the listing).

Can't install on ArchLinux: No matching distribution found for setuptools-meta

Full error below. Manually installing setuptools-meta with pip install --user git+https://github.com/noirbizarre/setuptools-meta.git allow the installation.

$ pip install --user git+https://github.com/N-Coder/studip-fuse
Collecting git+https://github.com/N-Coder/studip-fuse
  Cloning https://github.com/N-Coder/studip-fuse to /tmp/pip-req-build-06kzvvj_
  Running command git clone -q https://github.com/N-Coder/studip-fuse /tmp/pip-req-build-06kzvvj_
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-06kzvvj_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-06kzvvj_/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-06kzvvj_/pip-egg-info
         cwd: /tmp/pip-req-build-06kzvvj_/
    Complete output (33 lines):
    WARNING: The wheel package is not available.
    ERROR: Could not find a version that satisfies the requirement setuptools-meta (from versions: none)
    ERROR: No matching distribution found for setuptools-meta
    Traceback (most recent call last):
      File "/usr/lib/python3.8/site-packages/setuptools/installer.py", line 128, in fetch_build_egg
        subprocess.check_call(cmd)
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpzn_e62n9', '--quiet', '--find-links', 'git+https://github.com/noirbizarre/setuptools-meta.git#egg=setuptools-meta', 'setuptools-meta']' returned non-zero exit status 1.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-06kzvvj_/setup.py", line 100, in <module>
        setup(
      File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 143, in setup
        _install_setup_requires(attrs)
      File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 138, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 718, in fetch_build_eggs
        resolved_dists = pkg_resources.working_set.resolve(
      File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 781, in resolve
        dist = best[req.key] = env.best_match(
      File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1066, in best_match
        return self.obtain(req, installer)
      File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1078, in obtain
        return installer(requirement)
      File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/usr/lib/python3.8/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['/usr/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpzn_e62n9', '--quiet', '--find-links', 'git+https://github.com/noirbizarre/setuptools-meta.git#egg=setuptools-meta', 'setuptools-meta']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

'ascii' codec can't encode character '\xe4' in 'Universit\xe4t Passau' on Python 3.5.3

Starting Studip-Fuse 3.1 83a639f with FUSE 2.9 (libfuse.so.2) running via CPython 3.5.3 on Linux-4.9.0-9-amd64-x86_64-with-debian-9.9
Mounting at / (uid=0, gid=0, pid=0, python pid=29063)
Initializing asyncio event loop...
Opening StudIP session...
Logging in via oauth...
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 983, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 92: ordinal not in range(128)
Call stack:
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/finksim/src/studip-fuse/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 31, in start
    root_rp = stack.enter_context(session_context_manager(args, loop, future))
  File "/usr/lib/python3.5/contextlib.py", line 307, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/finksim/src/studip-fuse/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 145, in session_context
    yield loop.run_until_complete(task)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 315, in _wakeup
    self._step()
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/finksim/src/studip-fuse/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 133, in enter
    log.info("Logged in as %s on %s", args.user, await session.get_instance_name())
Message: 'Logged in as %s on %s'
Arguments: ('fink13', 'Universit\xe4t Passau Stud.IP v4.2.1.2 running at https://studip.uni-passau.de/studip/api.php/')
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 983, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 32: ordinal not in range(128)
Call stack:
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/finksim/src/studip-fuse/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 31, in start
    root_rp = stack.enter_context(session_context_manager(args, loop, future))
  File "/usr/lib/python3.5/contextlib.py", line 307, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/finksim/src/studip-fuse/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 145, in session_context
    yield loop.run_until_complete(task)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1424, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 315, in _wakeup
    self._step()
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/finksim/src/studip-fuse/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 133, in enter
    log.info("Logged in as %s on %s", args.user, await session.get_instance_name())
Message: 'Logged in as %s on %s'
Arguments: ('fink13', 'Universit\xe4t Passau Stud.IP v4.2.1.2 running at https://studip.uni-passau.de/studip/api.php/')
Loop and session ready, sending result back to main thread

Low-level fuse driver improvements

Currently, there seem to be 3 different fuse libraries for python:

Note:
GVfs, the GNOME virtual file system, also provides similar features to FUSE and exposes a lot more details about files and internal FS processes.
It also exposes the mounted virtual file systems to normal POSIX-applications via FUSE.
Using GObject-bindings, a backend could be programmed in python.
Unfortunately, it is by far not as open for extension as FUSE and also depends on a lot of GNOME-specific libraries.

TODO

  • Check which of the libraries would be suited best in this use-case.
  • Also, include the python low-level (e.g. SEGFAULT) faulthandler to log problems with the drivers which are outside of python's reach to stderr or the logfile.
  • Also check windows-compatibility (e.g. per this fusepy PR via WinFsp or this issue via Dokan)

Improved caching for offline operation

All of the following is required to allow offline operation

Cache Management

  • allow custom cache validation
    • cached results containing/wrapping a failed future shouldn't be kept in cache
  • failed Downloads should be dropped from cache
  • when restarting a failed Download, its previously successfully downloaded parts should be kept/reused and only the failed parts/chunks retried
  • add ttl-based cache clearing
  • make cache persistable / serializeable
  • use factory for StudIP API data model classes based on object UIDs, preventing duplicate objects
  • directly cache the results of the StudIP API Session getter functions
  • only persist the now unique instances of the StudIP API data model classes
  • reuse expired cache-value as fallback when new attempt fails too often or takes to long
  • implement circuit-breaker algorithm for checking for network connectivity and deciding whether (or how often) a function should be retried or the expired value should be reused

Task Management

  • not being able to login in when starting loop will always prevent startup when offline
  • session termination during runtime is not handled
  • store cached values on disk periodically and load then on program start
  • allow eager loading of certain directories (with lower priority than user/FUSE operations)
  • implement framework for lifecycle tasks
    1. startup tasks (e.g. restore model),
    2. low priority background tasks (e.g. reset semester selection, save model),
    3. required/dependency (cached) tasks (e.g. login) and
    4. shutdown tasks (e.g. reset semester selection, save model)

Future Work

  • cached functions and download parts should be retried immediately on temporary network failures
  • use ephemeral caching on all other functions (if really needed)
  • test fuse functionality, cache logic and API functions
  • allow user to switch between online and offline operation
    • provide two hidden directories that also contain the full directory tree, but using only cached values (/.cached/*) / only online values (/.online/*)
    • or make appindicator allowing to show and switch modes
  • make all caching and circuit-breaker related values (timeouts, thresholds,...) configurable
  • properly implement and tune HTTP timeout and status code handling

Python 3.5.3 compatibility

  File "/home/finksim/src/studip-fuse/studip_fuse/studipfs/api/aiointerface.py", line 4, in <module>
    from typing import Any, AsyncContextManager, AsyncGenerator, Callable, Coroutine, Dict, Generic, NamedTuple, Optional, TypeVar, Union
ImportError: cannot import name 'AsyncContextManager'

  File "/home/finksim/src/studip-fuse/studip_fuse/studipfs/api/aiointerface.py", line 4, in <module>
    from typing import Any, AsyncGenerator, Callable, Coroutine, Dict, Generic, NamedTuple, Optional, TypeVar, Union
ImportError: cannot import name 'AsyncGenerator'

  File "/home/finksim/src/studip-fuse/studip_fuse/launcher/aioimpl/asyncio/aiohttp_client.py", line 115, in DownloadState
    EMPTY = enum.auto()
AttributeError: module 'enum' has no attribute 'auto'

  File "/home/finksim/src/studip-fuse/studip_fuse/studipfs/api/session.py", line 6, in <module>
    from typing import AsyncGenerator, List, Mapping, Tuple
ImportError: cannot import name 'AsyncGenerator'

typing.AsyncGenerator is new in version 3.5.4
typing.AsyncContextManager is new in version 3.6
enum.auto is new in version 3.6

studip-fuse at Leibniz-Universität Hannover

I'm currently trying to use studip-fuse for the studip of the Leibniz-Universität Hannover and get problems during login. I'm trying to execute studip-fuse with theese arguments:
studip-fuse -f --studip-url https://studip.uni-hannover.de/ --login-method shib --sso "https://studip.uni-hannover.de/Shibboleth.sso/Login?target=https%3A%2F%2Fstudip.uni-hannover.de%2F%3Fsso%3Dshib%26again%3Dyes%26cancel_login%3D1&entityID=https%3A%2F%2Fsso.idm.uni-hannover.de%2Fidp%2Fshibboleth" --pwfile - myUsername /path/to/mountpoint/
Might the error occur, because the login dialog of our webSSO is different from that of the university in Passau?

Starting Studip-Fuse 3.2 release (installed from revision 1e2a7b0) with FUSE 2.9 (libfuse.so.2) running via CPython 3.8.2 on Linux-4.15.0-96-generic-x86_64-with-glibc2.10
Password: 
Going to mount at /home/nk/studIP (uid=0, gid=0, pid=-1238159824, python pid=16658)
Initializing asyncio event loop...
Opening StudIP session...
Logging in via shib...
Initiating shut down sequence...
Event loop closed
Uncaught exception:
Traceback (most recent call last):
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/refuse/high.py", line 862, in _wrapper
    return func(*args, **kwargs) or 0
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/refuse/high.py", line 1121, in init
    return self.operations('init', '/')
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/studip_fuse/studipfs/fuse_ops.py", line 104, in __call__
    ret = getattr(self, op)(path, *args)
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/studip_fuse/studipfs/fuse_ops.py", line 122, in init
    self.loop_stop_fn, self.loop_run_fn, self.root_rp = self.loop_future.result()
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 63, in future_context
    yield future
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 31, in start
    root_rp = stack.enter_context(session_context_manager(args, loop, future))
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/contextlib.py", line 425, in enter_context
    result = _cm_type.__enter__(cm)
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 145, in session_context
    yield loop.run_until_complete(task)
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/studip_fuse/launcher/aioimpl/asyncio/main_loop.py", line 126, in enter
    await http_client.shib_auth(start_url=args.shib_url, username=args.user, password=args.get_password())
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/studip_fuse/launcher/aioimpl/asyncio/aiohttp_client.py", line 92, in shib_auth
    resp.raise_for_status()
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 941, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='400', url=URL('https://sso.idm.uni-hannover.de/idp/profile/SAML2/Redirect/SSO?SAMLRequest=hZLRToMwgJ7EhfQeytjQ0QwS3C5cMh0Z6IU3ptAzaQIt9pSpby8fW92Y6HX/fv85X84Cedheoi1trXbw2gNa571tFLLTQ0x6o5jmKJEp3gIyW7E8vduwwP69gj6VlW6IkyKCsVKrpVbYt2ByMAdZwcNuE5Pa2g4ZpWh7ITuvV9KtuVL6AMYTQPNalqVuwNYeoqZHekCzbV4QZzWMIxU/gi8wqD0p2lg&f7q6TLSXDZwhOxDSQGVpnm%2BJs17F5PkhorivMJsG0V5AxOcAEFZlWFai9CfXpRhiiD2sFVqubEwCP/Bdf%2BYGfjEJWDhkzfREnOy8%2BI1UQqqXvy2VYwjZ659firrjYIxg8LTUESLI4umanYnNh/28s/1JOkn8E47fgbrhbtms7dj%2Bg16tMN7L6cNKm0WCTCaEJuOXnxeSfAI%3D&RelayState=ss:mem:b15895d7dcca0c937cb84186a4b45e3f69f99ba4d101afec03e22aabe950b9d2&SigAlg=http://www.w3.org/2001/04/xmldsig-more%23rsa-sha256&Signature=vxIhyAYle7zYYRo4RVX79IHYfNIksWKwI0%2BJBj1wf4ddGnVajd7U5WZd3mwxB7zH0NYQej9WORSki7TCDpwI2IaJRYYQ2cBJO%2BV/Io7akqMTIrXqc%2BA2eUnQrEr1OwCSgyyj2538RAuomuCUWnJcjRprG6vSzwwSDbZufTly9eJj8pwe9Y1H9KxFzJp46xTL3VXRkofgSxh6IsKE4/Ha0i/7lVoZ4IJItR5XOrzyFSnhBO/MF/%2B/Z6fwnz5EKpWqIDyUqaDkzOeO4t8IFzqUAoYFvEhGZzqfYymoJ5aqDPbsgZV67RhCU%2Bi/pulZLq%2BTYmKbYjTnMhH1l%2ByP1XroN2NiVKfdTsRinScF5W4EMLm1GKgdTFABkTVTTK%2BtFo/afuscUAW6GkfBm9BUaoDXm7jsPTw6CIXdxJck%29fx70LaTwFHLfX6JV3CmxtGVV%2BX/zzyGiEzmJgmHrcLyiw0LjPLu0HjGwcCm8/ftl6gqT48Z5HyNy4eGza3v3knaK6YCPcwf9NhwnYoC8ev1Y5vIqsCH8OmQWSnLZue821Upwd5uKAUA7ahicFdtL7TB9dBvbW9bFr0oH20mlMDk7ohlEsW6sRGpWkNaTsL5B8D5OCGByU1O131OoUd1q4sfiTEyHYesZu4Ia2SgI3SqErD0NbDlXMK5IaMgZqtjQ9hchEE%3D')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 237, in 'calling callback function'
  File "/home/nk/anaconda3/envs/studip/lib/python3.8/site-packages/refuse/high.py", line 888, in _wrapper
    self.__critical_exception = e
NameError: name 'self' is not defined
^CUnmounting from / (uid=0, gid=0, pid=0, python pid=16658)
Unmounting complete

The last two lines only appear after interrupting the program with ctrl-c.
I changed some characters in the URL of the clientResponseError, because I'm not sure if my login credencials are encoded in there.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.