This tool scrapes the public metadata sources from set of stratum0 and stratum1 servers. It grabs:

- cvmfs/info/v1/repositories.json 

And then for every repo it finds (that it's not told to ignore), it grabs:

- cvmfs/<repo>/.cvmfs_status.json
- cvmfs/<repo>/.cvmfspublished

pip install cvmfs-server-scraper

#!/usr/bin/env python3

import logging
from cvmfsscraper import scrape, scrape_server, set_log_level

# server = scrape_server("aws-eu-west1.stratum1.cvmfs.eessi-infra.org")

set_log_level(logging.DEBUG)

servers = scrape(
    stratum0_servers=[
        "stratum0.tld",
    ],
    stratum1_servers=[
        "stratum1-no.tld",
        "stratum1-au.tld",
    ],
    repos=[],
    ignore_repos=[],
)

# Note that the order of servers is undefined.
print(servers[0])

for repo in servers[0].repositories:
    print("Repo: " + repo.name )
    print("Root size: " + repo.root_size)
    print("Revision: " + repo.revision)
    print("Revision timestamp: " + repo.revision_timestamp)
    print("Last snapshot: " + str(repo.last_snapshot))

Note that if you are using a Stratum1 server with S3 as its backend, you need to set repos explicitly. This is because the S3 backend does not have a cvmfs/info/v1/repositories.json file. Also, the GeoAPI status will be NOT_FOUND for these servers.

# Data structure

## Server

A server object, representing a specific server that has been scraped.

````python
servers = scrape(...)
server_one = servers[0]

server.name

The name of the server, usually its fully qualified domain name.

server.geoapi_status

A GeoAPIstatus enum object. Defined in constants.py. The possible values are:

• OK (0: OK)
• LOCATION_ERROR (1: GeoApi gives wrong location)
• NO_RESPONSE (2: No response)
• NOT_FOUND (9: The server has no repository available so the GeoApi cannot be tested)
• NOT_YET_TESTED (99: The server has not yet been tested)

server.repositories

A list of repository objects, sorted by name. Empty if no repositores are scraped on the server.

server.ignored_repositories

List of repositories names that are to be ignored by the scraper.

server.forced_repositories

A list of repository names that the server is forced to scrape. If a repo name exists in both ignored_repositories and forced_repositories, it will be scraped.

A repository object, representing a single repository on a scraped server.

servers = scrape(...)
repo_one = servers[0].repositories[0]

repo_one.name

The fully qualified name of the repository.

repo_one.server

The server object to which the repository belongs.

repo_one.path

The path for the repository on the server. May differ from the name. To get a complete URL, one can do:

url = "http://" + repo_one.server.name + repo_one.path

These attributes are populated from cvmfs_status.json:

Information from .cvmfspublished is also provided. For explanations for these keys, please see CVMFS' official documentation. The field value in the table is the field key from .cvmfspublished.