Git Product home page Git Product logo

slate-catalog's Introduction

IMPORTANT: This repository has been deprecated in favor of the new incubator and stable application catalog repositories. See Adding the application to the catalog for more details.

SLATE Application Catalog

The SLATE Application Catalog is a Helm catalog focused on providing high-quality, secure packages for Scientific Computing applications.

How to use

We have designed the catalog to both be compatible with standard Helm clients, as well as the slate commandline interface and API used by the SLATE project. If you're already using SLATE, then there's nothing additional that you need to do, just use the SLATE client.

Using the SLATE Catalog with Helm

Simply add the SLATE repository to your Helm configuration:

[17:00]:~ $ helm repo add slate http://jenkins.slateci.io/catalog/stable/
"slate" has been added to your repositories

You can then test to see if an application is available:

[17:00]:~ $ helm search condor
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                                       
slate/htcondor	0.8.5        	8.6.13     	HTCondor distributed high-throughput computing ...

Differences from other Helm packages

Applications running within SLATE operate under a more restrictive environment than typical Kubernetes applications. For instance, SLATE apps are not allowed to create namespaces, roles, storage classes, or install Operators on the user's behalf. Users within the SLATE system are typically external users to a cluster administered by someone else.

Contributing

Repository layout

Example:

stable/
	my_app/
		my_app/
			(chart sources)
		images/
			my_app/
				Dockerfile
				image_nametag
				(other container image source files)
			a_helper_image/
				Dockerfile
				image_nametag
				(other container image source files)

As for the main kubernetes helm repository, there are two repositories: stable and incubator. Stable holds applications that are fully vetted while incubator holds those that are still under development. Each application is a subdirectory within one of those two directories.

Each application subdirectory must contain another subdirectory with the same name which contains the helm chart sources. This enables including other data besides chart sources for an application, however, because helm requires a chart source directory to have a name matching the chart name nested directories with the same names are unavoidable.

Besides the subdirectory for the chart sources, an application's directory can contain a subdirectory named 'images', which contains further subdirectories for any container image sources, one per image. The image source directories may have any names. Each image source directory should contain the Dockerfile which defines the image, any supporting files, and one special file named 'image_nametag' which contains the name and tag to be used for the image, like "my_app:latest".

How to rebuild the packages

To rebuild all packages from sources:

> mkdir build
> cd build
> cmake ..
> make

CMake 3 is required, so on some systems it may be necessary to replace cmake above with cmake3.

Known limitations

When a chart is added or a file is added to a chart the cmake scripts will not take notice until make rebuild_cache is run.

Incubator vs Stable

When a new application is packaged as a helm chart and is added to the SLATE catalog it is initially placed into the Incubator. It will stay in the incubator until it has been thouroughly tested to prove that the application works in SLATE the same way it is supposed to outside of SLATE. Once all parties involved agree that the application functions it will be moved to the Stable repository where it can be easily accessed from the portal and command line interface

Chart Version

Whenever a change is made to a chart in any way, from templates to documentation, the version value in Chart.yaml must be incremented so that Helm will update the helm charts used by the API Server

slate-catalog's People

Contributors

adamhgriffith-uofu avatar amblinalong avatar ataylor89 avatar benkulbertis avatar brianhlin avatar cbmckni avatar checkyourgages avatar cnweaver avatar davidorgill avatar edquist avatar gcarcassi avatar ike091 avatar ivukotic avatar jarofmy avatar jiahuikchen avatar jlstephen avatar jmcragun avatar jstidd avatar lincolnbryant avatar luxgriffith avatar mansalu avatar matyasselmeci avatar muali14 avatar pjorgen1 avatar rajends-umich avatar rynge avatar slateci-dockerhub avatar spelkington avatar sthapa avatar wookiee2187 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

slate-catalog's Issues

squid: Ephemeral request should not go up if `CacheDirOnHost: True`

In https://github.com/slateci/slate-catalog/blob/master/stable/osg-frontier-squid/osg-frontier-squid/templates/deployment.yaml#L52-L59

we add the Ephemeral requests and limit to be, approximately:
request = EphemeralRequestSize + CacheSize
limit = EphemeralRequestSize + CacheSize

However, if the value CacheDirOnHost is true, then we do not need to add the CacheSize to the EphemeralRequestSize

Something like

{{ if eq CacheDirOnHost "false" }}
  ephemeral-storage: {{ add .Values.SquidConf.RequestEphemeralSize  (floor (div (mul .Values.SquidConf.CacheSize 9537) 10000))  }}Mi
  ephemeral-storage: {{ add .Values.SquidConf.LimitEphemeralSize  (floor (div (mul .Values.SquidConf.CacheSize 9537) 10000))  }}Mi
{{ else }}
  ephemeral-storage: {{ .Values.SquidConf.RequestEphemeralSize  }}Mi
  ephemeral-storage: {{ .Values.SquidConf.LimitEphemeralSize  }}Mi

?

This is affecting my squid deployment at NET2 where the ephemeral disk size is rather small and we request CacheSize+EmphemeralDiskSize where we don't need to.

Incubator/FTS3 Dockerfile does not build

See below:

me@machine ~/repos/slate-catalog/incubator/fts3/images/fts3 (master)
$ docker build --file Dockerfile --tag fts3:local .
Sending build context to Docker daemon  56.32kB
Step 1/12 : FROM opensciencegrid/software-base:fresh
fresh: Pulling from opensciencegrid/software-base
2d473b07cdd5: Already exists 
da8b159483d0: Pull complete 
57d12b06dd6c: Pull complete 
5bcd57bcb6b8: Pull complete 
9fb64df4374b: Pull complete 
9f4fb1cb058d: Pull complete 
2dee4357d571: Pull complete 
ec2c372df97d: Pull complete 
9fe72bc286f7: Pull complete 
4f4fb700ef54: Pull complete 
Digest: sha256:86c58f50821ea3a753271b67b9ec543c71ac9a7b77705b996ca23ff3e7a1cd9d
Status: Downloaded newer image for opensciencegrid/software-base:fresh
 ---> af5de536acc0
Step 2/12 : LABEL description="File Transfer Service from CERN"
 ---> Running in e48c75ae0dcf
Removing intermediate container e48c75ae0dcf
 ---> 7dda42d268c4
Step 3/12 : LABEL [email protected]
 ---> Running in 5dda909d434f
Removing intermediate container 5dda909d434f
 ---> 3a428c840b48
Step 4/12 : ADD "http://fts-repo.web.cern.ch/fts-repo/fts3-prod-el7.repo" "/etc/yum.repos.d/"
Downloading [==================================================>]     128B/128B
 ---> e9a9ce3ecb0b
Step 5/12 : ADD "https://dmc-repo.web.cern.ch/dmc-repo/dmc-rc-el7.repo" "/etc/yum.repos.d/"
Downloading [==================================================>]     146B/146B
 ---> a750a597adfa
Step 6/12 : RUN yum install -y osg-ca-certs
 ---> Running in 4b4fab671b8b
Loaded plugins: fastestmirror, ovl, priorities
Determining fastest mirrors
 * base: mirrors.xmission.com
 * epel: mirror.lshiy.com
 * extras: mirror.rackspace.com
 * updates: mirror.rackspace.com
715 packages excluded due to repository priority protections
Package osg-ca-certs-1.106-1.osg35.el7.noarch already installed and latest version
Nothing to do
Removing intermediate container 4b4fab671b8b
 ---> 0264e1fe68fd
Step 7/12 : RUN yum install --disablerepo=osg --disablerepo=osg-testing -y gfal2-all gfal2-util fts-server fts-client fts-rest fts-monitoring fts-mysql fts-server-selinux fts-rest-selinux fts-monitoring-selinux fts-msg fts-infosys
 ---> Running in 5cbe376fd7db
Loaded plugins: fastestmirror, ovl, priorities
Loading mirror speeds from cached hostfile
 * base: mirrors.xmission.com
 * epel: mirror.lshiy.com
 * extras: mirror.rackspace.com
 * updates: mirror.rackspace.com
688 packages excluded due to repository priority protections
Package gfal2-util is obsoleted by python3-gfal2-util, trying to install python3-gfal2-util-1.7.1-1.el7.cern.noarch instead
Resolving Dependencies
--> Running transaction check
---> Package fts-client.x86_64 0:3.11.3-1.el7.cern will be installed
--> Processing Dependency: fts-libs(x86-64) = 3.11.3-1.el7.cern for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libzmq.so.5()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libvomsapi.so.1()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libprotobuf.so.8()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libpanic.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libgridsite.so.2()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libglobus_gsi_sysconfig.so.1()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libglobus_gsi_credential.so.1()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libglobus_common.so.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libgfal_transfer.so.2()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libgfal2.so.2()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_url_copy.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_server_lib.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_proxy.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_msg_ifce.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_msg_bus.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_db_generic.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_config.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_common.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libfts_cli_common.so.3()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libdirq.so.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_unit_test_framework-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_timer-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_regex-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_program_options-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_iostreams-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_filesystem-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libboost_chrono-mt.so.1.53.0()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libVoShares.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libVoNameCliTest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libUrlCopyProcess.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libUrlCopyCmd.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libUri.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libTransferStatusCliTest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libThreadPool.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libSubmitTransferCliTest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libSetCfgCli.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libServerConfigReader.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libServerConfig.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libSeConfig.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libRestSubmission.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libRestModifyJob.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libRestDeletion.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libRestBanning.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libResponseParser.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libPidTools.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libOptimizer.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libMsgBus.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libLogger.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libListTransferCliTest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libJobIdCliTest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libHttpRequest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libDnCliTest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libDaemonTools.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libConcurrentQueue.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libCliBaseTest.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libCfgParser.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libAutoInterruptThread.so()(64bit) for package: fts-client-3.11.3-1.el7.cern.x86_64
---> Package fts-infosys.x86_64 0:3.11.3-1.el7.cern will be installed
--> Processing Dependency: glue-schema for package: fts-infosys-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: bdii for package: fts-infosys-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libpugixml.so.1()(64bit) for package: fts-infosys-3.11.3-1.el7.cern.x86_64
---> Package fts-monitoring.noarch 0:3.11.0-1.el7 will be installed
--> Processing Dependency: python-django16 for package: fts-monitoring-3.11.0-1.el7.noarch
--> Processing Dependency: python-decorator for package: fts-monitoring-3.11.0-1.el7.noarch
--> Processing Dependency: mod_wsgi for package: fts-monitoring-3.11.0-1.el7.noarch
--> Processing Dependency: mod_ssl for package: fts-monitoring-3.11.0-1.el7.noarch
--> Processing Dependency: httpd for package: fts-monitoring-3.11.0-1.el7.noarch
--> Processing Dependency: MySQL-python for package: fts-monitoring-3.11.0-1.el7.noarch
---> Package fts-monitoring-selinux.noarch 0:3.11.0-1.el7 will be installed
---> Package fts-msg.x86_64 0:3.11.3-1.el7.cern will be installed
--> Processing Dependency: libactivemq-cpp.so.19()(64bit) for package: fts-msg-3.11.3-1.el7.cern.x86_64
---> Package fts-mysql.x86_64 0:3.11.3-1.el7.cern will be installed
--> Processing Dependency: soci-mysql(x86-64) for package: fts-mysql-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: fts-mysql-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libsoci_mysql.so.4.0()(64bit) for package: fts-mysql-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libsoci_core.so.4.0()(64bit) for package: fts-mysql-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: fts-mysql-3.11.3-1.el7.cern.x86_64
---> Package fts-rest.noarch 0:3.11.1-1.el7 will be installed
--> Processing Dependency: python-fts = 3.11.1-1.el7 for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-jwcrypto >= 0.6 for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: gridsite(x86-64) >= 1.7 for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-urllib3 for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-simplejson(x86-64) for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-requests for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-pylons for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-paste-deploy for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-oic for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-jwt for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: python-dirq for package: fts-rest-3.11.1-1.el7.noarch
--> Processing Dependency: gfal2-python(x86-64) for package: fts-rest-3.11.1-1.el7.noarch
---> Package fts-rest-selinux.noarch 0:3.11.1-1.el7 will be installed
---> Package fts-server.x86_64 0:3.11.3-1.el7.cern will be installed
--> Processing Dependency: gfal2-plugin-srm(x86-64) >= 2.20.0 for package: fts-server-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: gfal2-plugin-http(x86-64) >= 2.20.0 for package: fts-server-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: gfal2-plugin-gridftp(x86-64) >= 2.20.0 for package: fts-server-3.11.3-1.el7.cern.x86_64
---> Package fts-server-selinux.x86_64 0:3.11.3-1.el7.cern will be installed
--> Processing Dependency: selinux-policy >= 3.13.1-268.el7_9.2 for package: fts-server-selinux-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: /usr/sbin/semodule for package: fts-server-selinux-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: /usr/sbin/semodule for package: fts-server-selinux-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: /sbin/restorecon for package: fts-server-selinux-3.11.3-1.el7.cern.x86_64
--> Processing Dependency: /sbin/restorecon for package: fts-server-selinux-3.11.3-1.el7.cern.x86_64
---> Package gfal2-all.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: gfal2-plugin-xrootd(x86-64) = 2.20.5-1.osg35up.el7 for package: gfal2-all-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: gfal2-plugin-sftp(x86-64) = 2.20.5-1.osg35up.el7 for package: gfal2-all-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: gfal2-plugin-rfio(x86-64) = 2.20.5-1.osg35up.el7 for package: gfal2-all-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: gfal2-plugin-lfc(x86-64) = 2.20.5-1.osg35up.el7 for package: gfal2-all-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: gfal2-plugin-file(x86-64) = 2.20.5-1.osg35up.el7 for package: gfal2-all-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: gfal2-plugin-dcap(x86-64) = 2.20.5-1.osg35up.el7 for package: gfal2-all-2.20.5-1.osg35up.el7.x86_64
---> Package python3-gfal2-util.noarch 0:1.7.1-1.el7.cern will be installed
--> Processing Dependency: python(abi) = 3.6 for package: python3-gfal2-util-1.7.1-1.el7.cern.noarch
--> Processing Dependency: gfal2-util-scripts = 1.7.1-1.el7.cern for package: python3-gfal2-util-1.7.1-1.el7.cern.noarch
--> Processing Dependency: gfal2-python3 >= 1.11.0 for package: python3-gfal2-util-1.7.1-1.el7.cern.noarch
--> Processing Dependency: python3 for package: python3-gfal2-util-1.7.1-1.el7.cern.noarch
--> Running transaction check
---> Package MySQL-python.x86_64 0:1.2.5-1.el7 will be installed
---> Package activemq-cpp.x86_64 0:3.9.3-3.el7 will be installed
--> Processing Dependency: libapr-1.so.0()(64bit) for package: activemq-cpp-3.9.3-3.el7.x86_64
---> Package bdii.noarch 0:5.2.26-8.el7 will be installed
--> Processing Dependency: /usr/bin/mkpasswd for package: bdii-5.2.26-8.el7.noarch
--> Processing Dependency: initscripts for package: bdii-5.2.26-8.el7.noarch
--> Processing Dependency: logrotate for package: bdii-5.2.26-8.el7.noarch
--> Processing Dependency: openldap-clients for package: bdii-5.2.26-8.el7.noarch
--> Processing Dependency: openldap-servers for package: bdii-5.2.26-8.el7.noarch
--> Processing Dependency: policycoreutils-python for package: bdii-5.2.26-8.el7.noarch
--> Processing Dependency: policycoreutils-python for package: bdii-5.2.26-8.el7.noarch
---> Package boost-chrono.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-filesystem.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-iostreams.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-program-options.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-regex.x86_64 0:1.53.0-28.el7 will be installed
--> Processing Dependency: libicuuc.so.50()(64bit) for package: boost-regex-1.53.0-28.el7.x86_64
--> Processing Dependency: libicui18n.so.50()(64bit) for package: boost-regex-1.53.0-28.el7.x86_64
--> Processing Dependency: libicudata.so.50()(64bit) for package: boost-regex-1.53.0-28.el7.x86_64
---> Package boost-system.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-test.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-thread.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-timer.x86_64 0:1.53.0-28.el7 will be installed
---> Package fts-libs.x86_64 0:3.11.3-1.el7.cern will be installed
---> Package fts-tests.x86_64 0:3.11.3-1.el7.cern will be installed
--> Processing Dependency: gfal2-plugin-mock for package: fts-tests-3.11.3-1.el7.cern.x86_64
---> Package gfal2.x86_64 0:2.20.5-1.osg35up.el7 will be installed
---> Package gfal2-plugin-dcap.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: dcap-tunnel-gsi(x86-64) for package: gfal2-plugin-dcap-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libdcap.so.1()(64bit) for package: gfal2-plugin-dcap-2.20.5-1.osg35up.el7.x86_64
---> Package gfal2-plugin-file.x86_64 0:2.20.5-1.osg35up.el7 will be installed
---> Package gfal2-plugin-gridftp.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: libglobus_gssapi_gsi.so.4(globus_gssapi_gsi)(64bit) for package: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libglobus_gssapi_gsi.so.4()(64bit) for package: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libglobus_gss_assist.so.3()(64bit) for package: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libglobus_gass_copy.so.2()(64bit) for package: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libglobus_ftp_control.so.1()(64bit) for package: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libglobus_ftp_client.so.2()(64bit) for package: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64
---> Package gfal2-plugin-http.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: davix-libs >= 0.8.0 for package: gfal2-plugin-http-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libdavix_copy.so.0()(64bit) for package: gfal2-plugin-http-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libdavix.so.0()(64bit) for package: gfal2-plugin-http-2.20.5-1.osg35up.el7.x86_64
---> Package gfal2-plugin-lfc.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: liblfc.so.1()(64bit) for package: gfal2-plugin-lfc-2.20.5-1.osg35up.el7.x86_64
---> Package gfal2-plugin-rfio.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: dpm-libs(x86-64) for package: gfal2-plugin-rfio-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libdpm.so.1()(64bit) for package: gfal2-plugin-rfio-2.20.5-1.osg35up.el7.x86_64
---> Package gfal2-plugin-sftp.x86_64 0:2.20.5-1.osg35up.el7 will be installed
---> Package gfal2-plugin-srm.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: srm-ifce >= 1.23.1 for package: gfal2-plugin-srm-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libgfal_srm_ifce.so.1()(64bit) for package: gfal2-plugin-srm-2.20.5-1.osg35up.el7.x86_64
---> Package gfal2-plugin-xrootd.x86_64 0:2.20.5-1.osg35up.el7 will be installed
--> Processing Dependency: libXrdUtils.so.3()(64bit) for package: gfal2-plugin-xrootd-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libXrdPosix.so.3()(64bit) for package: gfal2-plugin-xrootd-2.20.5-1.osg35up.el7.x86_64
--> Processing Dependency: libXrdCl.so.3()(64bit) for package: gfal2-plugin-xrootd-2.20.5-1.osg35up.el7.x86_64
---> Package gfal2-util-scripts.noarch 0:1.7.1-1.el7.cern will be installed
---> Package globus-common.x86_64 0:18.13-1.el7 will be installed
--> Processing Dependency: libltdl.so.7()(64bit) for package: globus-common-18.13-1.el7.x86_64
---> Package globus-gsi-credential.x86_64 0:8.3-1.el7 will be installed
--> Processing Dependency: libglobus_gsi_callback.so.0()(64bit) for package: globus-gsi-credential-8.3-1.el7.x86_64
--> Processing Dependency: libglobus_gsi_cert_utils.so.0()(64bit) for package: globus-gsi-credential-8.3-1.el7.x86_64
--> Processing Dependency: libglobus_openssl_error.so.0()(64bit) for package: globus-gsi-credential-8.3-1.el7.x86_64
---> Package globus-gsi-sysconfig.x86_64 0:9.5-1.el7 will be installed
---> Package glue-schema.noarch 0:2.0.11-1.el7 will be installed
---> Package gridsite.x86_64 0:2.3.4-1.el7 will be installed
--> Processing Dependency: libcanl_c.so.2()(64bit) for package: gridsite-2.3.4-1.el7.x86_64
--> Processing Dependency: libgsoap.so.4()(64bit) for package: gridsite-2.3.4-1.el7.x86_64
---> Package gridsite-libs.x86_64 0:2.3.4-1.el7 will be installed
---> Package httpd.x86_64 0:2.4.6-97.el7.centos.5 will be installed
--> Processing Dependency: httpd-tools = 2.4.6-97.el7.centos.5 for package: httpd-2.4.6-97.el7.centos.5.x86_64
--> Processing Dependency: system-logos >= 7.92.1-1 for package: httpd-2.4.6-97.el7.centos.5.x86_64
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-97.el7.centos.5.x86_64
---> Package libdirq.x86_64 0:0.5-1.el7 will be installed
---> Package mariadb-libs.x86_64 1:5.5.68-1.el7 will be installed
---> Package mod_ssl.x86_64 1:2.4.6-97.el7.centos.5 will be installed
---> Package mod_wsgi.x86_64 0:3.4-18.el7 will be installed
---> Package policycoreutils.x86_64 0:2.5-34.el7 will be installed
--> Processing Dependency: libselinux-utils >= 2.5-14 for package: policycoreutils-2.5-34.el7.x86_64
---> Package protobuf.x86_64 0:2.5.0-8.el7 will be installed
---> Package pugixml.x86_64 0:1.8-1.el7 will be installed
---> Package python-decorator.noarch 0:3.4.0-3.el7 will be installed
---> Package python-dirq.noarch 0:1.8-1.el7 will be installed
---> Package python-fts.noarch 0:3.11.1-1.el7 will be installed
--> Processing Dependency: python-sqlalchemy for package: python-fts-3.11.1-1.el7.noarch
--> Processing Dependency: m2crypto for package: python-fts-3.11.1-1.el7.noarch
---> Package python-jwcrypto.noarch 0:0.6.0-1.el7.cern will be installed
--> Processing Dependency: python-cryptography >= 1.5 for package: python-jwcrypto-0.6.0-1.el7.cern.noarch
---> Package python-jwt.noarch 0:1.5.3-1.el7 will be installed
---> Package python-oic.noarch 0:0.15.1-1 will be installed
--> Processing Dependency: python-pyjwkest >= 1.3.6 for package: python-oic-0.15.1-1.noarch
--> Processing Dependency: python-pycryptodomex for package: python-oic-0.15.1-1.noarch
--> Processing Dependency: python-future for package: python-oic-0.15.1-1.noarch
---> Package python-paste-deploy.noarch 0:1.5.0-10.el7 will be installed
--> Processing Dependency: python-paste for package: python-paste-deploy-1.5.0-10.el7.noarch
---> Package python-pylons.noarch 0:1.0.1-2.el7 will be installed
--> Processing Dependency: python-beaker >= 1.3.1-5 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-formencode >= 1.2.1 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-mako >= 0.2.4 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-myghty >= 1.1 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-nose >= 0.10.4 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-paste-script >= 1.7.3 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-routes >= 1.12 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-tempita >= 0.2 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-weberror >= 0.10.1 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-webhelpers >= 0.6.4 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-webob >= 0.9.6.1 for package: python-pylons-1.0.1-2.el7.noarch
--> Processing Dependency: python-webtest >= 1.1 for package: python-pylons-1.0.1-2.el7.noarch
---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed
---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed
--> Processing Dependency: python-six for package: python-urllib3-1.10.2-7.el7.noarch
---> Package python2-django16.noarch 0:1.6.11.7-5.el7 will be installed
--> Processing Dependency: python-django16-bash-completion = 1.6.11.7-5.el7 for package: python2-django16-1.6.11.7-5.el7.noarch
---> Package python2-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python2-gfal2-1.12.0-1.el7.cern.x86_64
--> Processing Dependency: libboost_python-mt.so.1.53.0()(64bit) for package: python2-gfal2-1.12.0-1.el7.cern.x86_64
---> Package python2-simplejson.x86_64 0:3.11.1-1.el7 will be installed
---> Package python3.x86_64 0:3.6.8-18.el7 will be installed
--> Processing Dependency: python3-libs(x86-64) = 3.6.8-18.el7 for package: python3-3.6.8-18.el7.x86_64
--> Processing Dependency: python3-setuptools for package: python3-3.6.8-18.el7.x86_64
--> Processing Dependency: python3-pip for package: python3-3.6.8-18.el7.x86_64
--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-18.el7.x86_64
---> Package python3-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python3-gfal2-1.12.0-1.el7.cern.x86_64
--> Processing Dependency: libboost_python3.so.1.53.0()(64bit) for package: python3-gfal2-1.12.0-1.el7.cern.x86_64
---> Package selinux-policy.noarch 0:3.13.1-268.el7_9.2 will be installed
---> Package soci.x86_64 0:4.0.0-4.el7 will be installed
--> Processing Dependency: libboost_date_time-mt.so.1.53.0()(64bit) for package: soci-4.0.0-4.el7.x86_64
---> Package soci-mysql.x86_64 0:4.0.0-4.el7 will be installed
---> Package voms.x86_64 0:2.1.0-0.24.rc2.el7 will be installed
---> Package zeromq.x86_64 0:4.1.4-6.el7 will be installed
--> Processing Dependency: libpgm-5.2.so.0()(64bit) for package: zeromq-4.1.4-6.el7.x86_64
--> Processing Dependency: libsodium.so.23()(64bit) for package: zeromq-4.1.4-6.el7.x86_64
--> Running transaction check
---> Package apr.x86_64 0:1.4.8-7.el7 will be installed
---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
---> Package boost-date-time.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-python.x86_64 0:1.53.0-28.el7 will be installed
---> Package boost-python36.x86_64 0:1.53.0-30.el7 will be installed
---> Package canl-c.x86_64 0:2.1.8-1.el7 will be installed
--> Processing Dependency: libcares.so.2()(64bit) for package: canl-c-2.1.8-1.el7.x86_64
---> Package centos-logos.noarch 0:70.0.6-3.el7.centos will be installed
---> Package davix-libs.x86_64 0:0.8.2-1.el7.cern will be installed
---> Package dcap-libs.x86_64 0:2.47.12-15.el7 will be installed
---> Package dcap-tunnel-gsi.x86_64 0:2.47.12-15.el7 will be installed
---> Package dpm-libs.x86_64 0:1.13.0-1.el7 will be installed
--> Processing Dependency: lcgdm-libs(x86-64) = 1.13.0-1.el7 for package: dpm-libs-1.13.0-1.el7.x86_64
--> Processing Dependency: liblcgdm.so.1()(64bit) for package: dpm-libs-1.13.0-1.el7.x86_64
---> Package expect.x86_64 0:5.45-14.el7_1 will be installed
--> Processing Dependency: libtcl8.5.so()(64bit) for package: expect-5.45-14.el7_1.x86_64
---> Package gfal2-plugin-mock.x86_64 0:2.20.5-1.osg35up.el7 will be installed
---> Package globus-ftp-client.x86_64 0:9.8-1.el7 will be installed
--> Processing Dependency: globus-xio-popen-driver(x86-64) >= 2 for package: globus-ftp-client-9.8-1.el7.x86_64
--> Processing Dependency: libglobus_xio.so.0()(64bit) for package: globus-ftp-client-9.8-1.el7.x86_64
---> Package globus-ftp-control.x86_64 0:9.10-1.el7 will be installed
--> Processing Dependency: globus-io(x86-64) >= 11 for package: globus-ftp-control-9.10-1.el7.x86_64
--> Processing Dependency: globus-xio-gsi-driver(x86-64) >= 4 for package: globus-ftp-control-9.10-1.el7.x86_64
--> Processing Dependency: libglobus_gssapi_error.so.2()(64bit) for package: globus-ftp-control-9.10-1.el7.x86_64
--> Processing Dependency: libglobus_io.so.3()(64bit) for package: globus-ftp-control-9.10-1.el7.x86_64
---> Package globus-gass-copy.x86_64 0:10.12-1.el7 will be installed
--> Processing Dependency: libglobus_gass_transfer.so.2()(64bit) for package: globus-gass-copy-10.12-1.el7.x86_64
---> Package globus-gsi-callback.x86_64 0:6.2-1.el7 will be installed
---> Package globus-gsi-cert-utils.x86_64 0:10.10-1.el7 will be installed
--> Processing Dependency: libglobus_openssl.so.0()(64bit) for package: globus-gsi-cert-utils-10.10-1.el7.x86_64
---> Package globus-gsi-openssl-error.x86_64 0:4.4-1.el7 will be installed
---> Package globus-gss-assist.x86_64 0:12.7-1.el7 will be installed
--> Processing Dependency: libglobus_callout.so.0()(64bit) for package: globus-gss-assist-12.7-1.el7.x86_64
---> Package globus-gssapi-gsi.x86_64 0:14.20-1.el7 will be installed
--> Processing Dependency: libglobus_gsi_proxy_core.so.0(GLOBUS_GSI_PROXY_CORE_8)(64bit) for package: globus-gssapi-gsi-14.20-1.el7.x86_64
--> Processing Dependency: libglobus_gsi_proxy_core.so.0()(64bit) for package: globus-gssapi-gsi-14.20-1.el7.x86_64
---> Package gsoap.x86_64 0:2.8.16-12.el7 will be installed
---> Package httpd-tools.x86_64 0:2.4.6-97.el7.centos.5 will be installed
---> Package initscripts.x86_64 0:9.49.53-1.el7_9.1 will be installed
--> Processing Dependency: sysvinit-tools >= 2.87-5 for package: initscripts-9.49.53-1.el7_9.1.x86_64
--> Processing Dependency: iproute for package: initscripts-9.49.53-1.el7_9.1.x86_64
---> Package lfc-libs.x86_64 0:1.13.0-1.el7 will be installed
---> Package libicu.x86_64 0:50.2-4.el7_7 will be installed
---> Package libselinux-utils.x86_64 0:2.5-15.el7 will be installed
---> Package libsodium.x86_64 0:1.0.18-1.el7 will be installed
---> Package libtool-ltdl.x86_64 0:2.4.2-22.el7_3 will be installed
---> Package logrotate.x86_64 0:3.8.6-19.el7 will be installed
---> Package m2crypto.x86_64 0:0.21.1-17.el7 will be installed
---> Package openldap-clients.x86_64 0:2.4.44-25.el7_9 will be installed
---> Package openldap-servers.x86_64 0:2.4.44-25.el7_9 will be installed
--> Processing Dependency: systemd-sysv for package: openldap-servers-2.4.44-25.el7_9.x86_64
--> Processing Dependency: libwrap.so.0()(64bit) for package: openldap-servers-2.4.44-25.el7_9.x86_64
---> Package openpgm.x86_64 0:5.2.122-2.el7 will be installed
---> Package policycoreutils-python.x86_64 0:2.5-34.el7 will be installed
--> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libselinux-python for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libcgroup for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-34.el7.x86_64
--> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-34.el7.x86_64
---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed
---> Package python-django16-bash-completion.noarch 0:1.6.11.7-5.el7 will be installed
---> Package python-formencode.noarch 0:1.2.6-1.el7 will be installed
---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed
--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch
---> Package python-myghty.noarch 0:1.2-5.el7 will be installed
---> Package python-nose.noarch 0:1.3.7-1.el7 will be installed
---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed
--> Processing Dependency: pyOpenSSL for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch
---> Package python-paste-script.noarch 0:1.7.5-7.el7 will be installed
--> Processing Dependency: python-cheetah for package: python-paste-script-1.7.5-7.el7.noarch
--> Processing Dependency: python-cherrypy for package: python-paste-script-1.7.5-7.el7.noarch
---> Package python-pyjwkest.noarch 0:1.4.2-1 will be installed
---> Package python-routes.noarch 0:1.13-2.el7 will be installed
--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch
---> Package python-six.noarch 0:1.9.0-2.el7 will be installed
---> Package python-sqlalchemy.x86_64 0:0.9.8-2.el7 will be installed
---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed
---> Package python-weberror.noarch 0:0.10.3-8.el7 will be installed
--> Processing Dependency: python-pygments for package: python-weberror-0.10.3-8.el7.noarch
---> Package python-webhelpers.noarch 0:1.3-9.el7 will be installed
---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed
---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed
---> Package python2-cryptography.x86_64 0:1.7.2-2.el7 will be installed
--> Processing Dependency: python-pyasn1 >= 0.1.8 for package: python2-cryptography-1.7.2-2.el7.x86_64
--> Processing Dependency: python-idna >= 2.0 for package: python2-cryptography-1.7.2-2.el7.x86_64
--> Processing Dependency: python-cffi >= 1.4.1 for package: python2-cryptography-1.7.2-2.el7.x86_64
--> Processing Dependency: python-enum34 for package: python2-cryptography-1.7.2-2.el7.x86_64
---> Package python2-future.noarch 0:0.18.2-2.el7 will be installed
---> Package python2-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python2-gfal2-1.12.0-1.el7.cern.x86_64
---> Package python2-pycryptodomex.x86_64 0:3.9.7-1.el7 will be installed
--> Processing Dependency: libtomcrypt.so.0()(64bit) for package: python2-pycryptodomex-3.9.7-1.el7.x86_64
---> Package python3-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python3-gfal2-1.12.0-1.el7.cern.x86_64
---> Package python3-libs.x86_64 0:3.6.8-18.el7 will be installed
--> Processing Dependency: libtirpc.so.1()(64bit) for package: python3-libs-3.6.8-18.el7.x86_64
---> Package python3-pip.noarch 0:9.0.3-8.el7 will be installed
---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed
---> Package srm-ifce.x86_64 0:1.24.5-1.el7 will be installed
--> Processing Dependency: CGSI-gSOAP >= 1.3.10 for package: srm-ifce-1.24.5-1.el7.x86_64
--> Processing Dependency: libcgsi_plugin.so.1()(64bit) for package: srm-ifce-1.24.5-1.el7.x86_64
---> Package xrootd-client-libs.x86_64 1:5.4.2-1.osg35up.el7 will be installed
---> Package xrootd-libs.x86_64 1:5.4.2-1.osg35up.el7 will be installed
--> Processing Dependency: libXrdServer.so.3()(64bit) for package: 1:xrootd-libs-5.4.2-1.osg35up.el7.x86_64
--> Running transaction check
---> Package CGSI-gSOAP.x86_64 0:1.3.11-1.el7 will be installed
---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed
---> Package c-ares.x86_64 0:1.10.0-3.el7 will be installed
---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed
---> Package globus-callout.x86_64 0:4.3-1.el7 will be installed
---> Package globus-gass-transfer.x86_64 0:9.4-1.el7 will be installed
---> Package globus-gsi-proxy-core.x86_64 0:9.8-1.el7 will be installed
--> Processing Dependency: libglobus_proxy_ssl.so.1()(64bit) for package: globus-gsi-proxy-core-9.8-1.el7.x86_64
---> Package globus-gssapi-error.x86_64 0:6.3-1.el7 will be installed
---> Package globus-io.x86_64 0:12.4-1.el7 will be installed
---> Package globus-openssl-module.x86_64 0:5.2-1.el7 will be installed
---> Package globus-xio.x86_64 0:6.6-1.el7 will be installed
---> Package globus-xio-gsi-driver.x86_64 0:5.4-1.el7 will be installed
---> Package globus-xio-popen-driver.x86_64 0:4.1-5.el7 will be installed
---> Package iproute.x86_64 0:4.11.0-30.el7 will be installed
--> Processing Dependency: libmnl.so.0(LIBMNL_1.0)(64bit) for package: iproute-4.11.0-30.el7.x86_64
--> Processing Dependency: libxtables.so.10()(64bit) for package: iproute-4.11.0-30.el7.x86_64
--> Processing Dependency: libmnl.so.0()(64bit) for package: iproute-4.11.0-30.el7.x86_64
---> Package lcgdm-libs.x86_64 0:1.13.0-1.el7 will be installed
--> Processing Dependency: perl(DBI) for package: lcgdm-libs-1.13.0-1.el7.x86_64
--> Processing Dependency: perl(Env) for package: lcgdm-libs-1.13.0-1.el7.x86_64
---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed
---> Package libselinux-python.x86_64 0:2.5-15.el7 will be installed
---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
---> Package libtomcrypt.x86_64 0:1.17-26.el7 will be installed
--> Processing Dependency: libtommath >= 0.42.0 for package: libtomcrypt-1.17-26.el7.x86_64
--> Processing Dependency: libtommath.so.0()(64bit) for package: libtomcrypt-1.17-26.el7.x86_64
---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed
---> Package python-IPy.noarch 0:0.75-6.el7 will be installed
---> Package python-cffi.x86_64 0:1.6.0-5.el7 will be installed
--> Processing Dependency: python-pycparser for package: python-cffi-1.6.0-5.el7.x86_64
---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed
---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed
---> Package python-idna.noarch 0:2.4-1.el7 will be installed
---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed
---> Package python-pygments.noarch 0:1.4-10.el7 will be installed
--> Processing Dependency: python-imaging for package: python-pygments-1.4-10.el7.noarch
---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed
---> Package python2-cheetah.x86_64 0:2.4.4-6.el7 will be installed
--> Processing Dependency: python2-markdown for package: python2-cheetah-2.4.4-6.el7.x86_64
---> Package python2-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python2-gfal2-1.12.0-1.el7.cern.x86_64
---> Package python2-pyasn1.noarch 0:0.1.9-7.el7 will be installed
---> Package python3-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python3-gfal2-1.12.0-1.el7.cern.x86_64
---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed
---> Package systemd-sysv.x86_64 0:219-78.el7_9.5 will be installed
---> Package sysvinit-tools.x86_64 0:2.88-14.dsf.el7 will be installed
---> Package tcl.x86_64 1:8.5.13-8.el7 will be installed
---> Package tcp_wrappers-libs.x86_64 0:7.6-77.el7 will be installed
---> Package xrootd-server-libs.x86_64 1:5.4.2-1.osg35up.el7 will be installed
--> Processing Dependency: libmacaroons.so.0()(64bit) for package: 1:xrootd-server-libs-5.4.2-1.osg35up.el7.x86_64
--> Running transaction check
---> Package globus-gsi-proxy-ssl.x86_64 0:6.5-1.el7 will be installed
---> Package iptables.x86_64 0:1.4.21-35.el7 will be installed
--> Processing Dependency: libnfnetlink.so.0()(64bit) for package: iptables-1.4.21-35.el7.x86_64
--> Processing Dependency: libnetfilter_conntrack.so.3()(64bit) for package: iptables-1.4.21-35.el7.x86_64
---> Package libmacaroons.x86_64 0:0.3.0-2.el7 will be installed
---> Package libmnl.x86_64 0:1.0.3-7.el7 will be installed
---> Package libtommath.x86_64 0:0.42.0-6.el7 will be installed
---> Package perl-DBI.x86_64 0:1.627-4.el7 will be installed
--> Processing Dependency: perl(RPC::PlServer) >= 0.2001 for package: perl-DBI-1.627-4.el7.x86_64
--> Processing Dependency: perl(RPC::PlClient) >= 0.2000 for package: perl-DBI-1.627-4.el7.x86_64
---> Package perl-Env.noarch 0:1.04-2.el7 will be installed
---> Package python-pillow.x86_64 0:2.0.0-23.gitd1c6db8.el7_9 will be installed
--> Processing Dependency: libtiff.so.5(LIBTIFF_4.0)(64bit) for package: python-pillow-2.0.0-23.gitd1c6db8.el7_9.x86_64
--> Processing Dependency: libjpeg.so.62(LIBJPEG_6.2)(64bit) for package: python-pillow-2.0.0-23.gitd1c6db8.el7_9.x86_64
--> Processing Dependency: libwebp.so.4()(64bit) for package: python-pillow-2.0.0-23.gitd1c6db8.el7_9.x86_64
--> Processing Dependency: libtiff.so.5()(64bit) for package: python-pillow-2.0.0-23.gitd1c6db8.el7_9.x86_64
--> Processing Dependency: libjpeg.so.62()(64bit) for package: python-pillow-2.0.0-23.gitd1c6db8.el7_9.x86_64
--> Processing Dependency: libfreetype.so.6()(64bit) for package: python-pillow-2.0.0-23.gitd1c6db8.el7_9.x86_64
---> Package python-pycparser.noarch 0:2.14-1.el7 will be installed
--> Processing Dependency: python-ply for package: python-pycparser-2.14-1.el7.noarch
---> Package python2-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python2-gfal2-1.12.0-1.el7.cern.x86_64
---> Package python2-markdown.noarch 0:2.4.1-4.el7 will be installed
---> Package python3-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python3-gfal2-1.12.0-1.el7.cern.x86_64
--> Running transaction check
---> Package freetype.x86_64 0:2.8-14.el7_9.1 will be installed
--> Processing Dependency: libpng15.so.15(PNG15_0)(64bit) for package: freetype-2.8-14.el7_9.1.x86_64
--> Processing Dependency: libpng15.so.15()(64bit) for package: freetype-2.8-14.el7_9.1.x86_64
---> Package libjpeg-turbo.x86_64 0:1.2.90-8.el7 will be installed
---> Package libnetfilter_conntrack.x86_64 0:1.0.6-1.el7_3 will be installed
---> Package libnfnetlink.x86_64 0:1.0.1-4.el7 will be installed
---> Package libtiff.x86_64 0:4.0.3-35.el7 will be installed
--> Processing Dependency: libjbig.so.2.0()(64bit) for package: libtiff-4.0.3-35.el7.x86_64
---> Package libwebp.x86_64 0:0.3.0-10.el7_9 will be installed
---> Package perl-PlRPC.noarch 0:0.2020-14.el7 will be installed
--> Processing Dependency: perl(Net::Daemon) >= 0.13 for package: perl-PlRPC-0.2020-14.el7.noarch
--> Processing Dependency: perl(Net::Daemon::Test) for package: perl-PlRPC-0.2020-14.el7.noarch
--> Processing Dependency: perl(Net::Daemon::Log) for package: perl-PlRPC-0.2020-14.el7.noarch
---> Package python-ply.noarch 0:3.4-11.el7 will be installed
---> Package python2-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python2-gfal2-1.12.0-1.el7.cern.x86_64
---> Package python3-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python3-gfal2-1.12.0-1.el7.cern.x86_64
--> Running transaction check
---> Package jbigkit-libs.x86_64 0:2.0-11.el7 will be installed
---> Package libpng.x86_64 2:1.5.13-8.el7 will be installed
---> Package perl-Net-Daemon.noarch 0:0.48-5.el7 will be installed
---> Package python2-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python2-gfal2-1.12.0-1.el7.cern.x86_64
---> Package python3-gfal2.x86_64 0:1.12.0-1.el7.cern will be installed
--> Processing Dependency: gfal2-core >= 2.21.0 for package: python3-gfal2-1.12.0-1.el7.cern.x86_64
--> Finished Dependency Resolution
Error: Package: python3-gfal2-1.12.0-1.el7.cern.x86_64 (dmc-rc-el7)
           Requires: gfal2-core >= 2.21.0
           Installing: gfal2-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-core = 2.20.5-1.osg35up.el7
Error: Package: python2-gfal2-1.12.0-1.el7.cern.x86_64 (dmc-rc-el7)
           Requires: gfal2-core >= 2.21.0
           Installing: gfal2-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-core = 2.20.5-1.osg35up.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
The command '/bin/sh -c yum install --disablerepo=osg --disablerepo=osg-testing -y gfal2-all gfal2-util fts-server fts-client fts-rest fts-monitoring fts-mysql fts-server-selinux fts-rest-selinux fts-monitoring-selinux fts-msg fts-infosys' returned a non-zero code: 1

Adding @LincolnBryant as a watcher.

Minio needs improvement

  • Documentation should explain how to properly create a SLATE secret with both keys, and connect to values.

  • There are additional files that can be removed from the container image
    See: #153

  • We should modify the git clone in the minio dockerfile to point at release tags

  • There is additional/unwanted functionality in the chart that should be removed
    * Multiple ways to create buckets
    * Statefulset that deploys the client
    * TLS ???
    * Others ???

Hosted CE should use k8s-standard cert names

In the secret for the credentials (HostCredentials section in the yaml), we currently reference one file (host.key) in one secret for the cert and a second file (host.cert) in a second secret for the key.

Unfortunately, certificates produced by a tool like cert-manager (https://cert-manager.io/docs/usage/certificate/) appear to hardcode the more common names of tls.crt and tls.key.

Would it be possible to extend the chart to switch to these names for the files? I don't think tls.crt and tls.key are written-down standards but rather what everyone else appears to use.

Numeric instances are broken

today I wanted to have slate make a bunch of instances of something, but didn't have "replicas" templated in my deployment. so instead i did something like this:

for i in {1..30}; do sed "s/Instance: global/Instance: $i/" htcondor.yaml > htcondor-$i.yaml ; done

such that Instance is Instance: 1, Instance: 2, ...

but then I get an error:

Failed to install application htcondor: Failed to start application instance with helm:
Error: render error in "htcondor/templates/deployment.yaml": template: htcondor/templates/_helpers.tpl:16:16: executing "condor.fullname" at <$name>: wrong type for value; expected string; got float64

 system namespace: slate-system

quoting the number also doesn't work. putting a character with the number does, though.

Update nginx stable app ingress api version to networking.k8s.io/v1

With the primary production clusters moving to K8s v1.24.3 we need to update the API version for the NGINX application ingress. Specifically, in ./stable/nginx/nginx/templates/service.yml update the apiVersion of the Ingress:

---
{{ if .Values.Ingress.Enabled }}
apiVersion: extensions/v1beta1 --> networking.k8s.io/v1
kind: Ingress
...

Add Certificate object to HostedCE

We would like to add an optional Certificate object to the HostedCE chart.

On River (UChicago) and Tiger (UW-Madison) we have CertManager installed, and so we would like to replace the certificate from Let's Encrypt in the pod with a certificate issued by CertManager. This makes renewal a lot easier.

Add external-dns annotations in the osg-hosted-ce chart

Outside SLATE, we use external-dns for integration with DNS. Can we add the appropriate annotation to the Service generated for the CE?

Looks something like this:

...
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: hosted-ce1.svcs.opensciencegrid.org
...

Hosted CE generates invalid HTCondor config for non-default port

In the generated 99-instance.conf here:

https://github.com/slateci/slate-catalog/blob/master/stable/osg-hosted-ce/osg-hosted-ce/templates/configmap.yaml#L75

we copy in the full hostname, potentially including the destination port. This results in SSH trying to use a connection string as folllows:

 ssh -o "BatchMode yes" [email protected]:14022

However, this is not valid SSH syntax and results in an error message along the lines of:

Could not resolve hostname ce.example.com:14022: Name or service not known

Instead, one wants:

 ssh -o "BatchMode yes" [email protected]

given that a different part of the container setup already sets the default port in the ~/.ssh/config.

osg-frontier-squid: Make CPU request configurable

We should make the CPU configurable on OSG Frontier Squid. Let's make the default 2 CPUs?

I am not sure what Kubernetes will do when no explicit request. Does it default to using all CPUs, or just one?

Move FTS3 to Attic

The incubator application FTS3 no longer builds and is taking down the catalog build process in #587.

user@machine ~/repos/slate-catalog/incubator/fts3/images/fts3 (master)
$ docker build --tag bob:local --no-cache .
...
...
Error: Package: fts-server-3.12.0-1.el7.cern.x86_64 (fts3-prod-el7)
           Requires: gfal2-plugin-gridftp(x86-64) >= 2.21.0
           Installing: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-plugin-gridftp(x86-64) = 2.20.5-1.osg35up.el7
Error: Package: fts-server-3.12.0-1.el7.cern.x86_64 (fts3-prod-el7)
           Requires: gfal2-plugin-gridftp(x86-64) >= 2.21.0
           Available: gfal2-plugin-gridftp-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-plugin-gridftp(x86-64) = 2.20.5-1.osg35up.el7
Error: Package: fts-server-3.12.0-1.el7.cern.x86_64 (fts3-prod-el7)
           Requires: gfal2-plugin-srm(x86-64) >= 2.21.0
           Available: gfal2-plugin-srm-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-plugin-srm(x86-64) = 2.20.5-1.osg35up.el7
Error: Package: fts-server-3.12.0-1.el7.cern.x86_64 (fts3-prod-el7)
           Requires: gfal2-plugin-srm(x86-64) >= 2.21.0
           Installing: gfal2-plugin-srm-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-plugin-srm(x86-64) = 2.20.5-1.osg35up.el7
Error: Package: python3-gfal2-1.12.0-1.el7.cern.x86_64 (dmc-rc-el7)
           Requires: gfal2-core >= 2.21.0
           Installing: gfal2-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-core = 2.20.5-1.osg35up.el7
Error: Package: fts-server-3.12.0-1.el7.cern.x86_64 (fts3-prod-el7)
           Requires: gfal2-plugin-http(x86-64) >= 2.21.0
           Available: gfal2-plugin-http-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-plugin-http(x86-64) = 2.20.5-1.osg35up.el7
Error: Package: fts-server-3.12.0-1.el7.cern.x86_64 (fts3-prod-el7)
           Requires: gfal2-plugin-http(x86-64) >= 2.21.0
           Installing: gfal2-plugin-http-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2-plugin-http(x86-64) = 2.20.5-1.osg35up.el7
Error: Package: fts-monitoring-3.12.0-1.el7.cern.noarch (fts3-prod-el7)
           Requires: rh-python36-mod_wsgi
Error: Package: fts-rest-server-3.12.0-1.el7.cern.noarch (fts3-prod-el7)
           Requires: rh-python36-mod_wsgi
Error: Package: fts-server-3.12.0-1.el7.cern.x86_64 (fts3-prod-el7)
           Requires: gfal2(x86-64) >= 2.21.0
           Installing: gfal2-2.20.5-1.osg35up.el7.x86_64 (osg-upcoming-testing)
               gfal2(x86-64) = 2.20.5-1.osg35up.el7

@LincolnBryant has given the go-ahead to move this to the ./attic.

Normalize squid configuration against OSG upstream

Today we have a script that customizes squid.conf to do the following:
https://github.com/slateci/slate-catalog/blob/master/stable/osg-frontier-squid/osg-frontier-squid/templates/configmap.yaml#L11-L56

Customizing Squid.conf

What we do today

It first sets the IP range here:

acl NET_LOCAL src {{ .Values.SquidConf.IPRange }}

Then it sets the monitoring IP range:

 acl HOST_MONITOR src {{ .Values.SquidConf.MonitoringIPRange }}

Then it sets the Cache memory:

cache_mem {{ .Values.SquidConf.CacheMem }} MB

Then we configure the squid as appropriate depending on whether or not there is 1 worker or N workers:

    {{ if .Values.SquidConf.Workers }}
    workers {{ .Values.SquidConf.Workers }} # multiple worker case
    cache_dir ufs /var/cache/squid/squid${process_number} {{ .Values.SquidConf.CacheSize }} 16 256
    logformat awstats %>a kid${process_number} %un [%{%d/%b/%Y:%H:%M:%S}tl.%03tu %{%z}tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh %tr "%{X-Frontier-Id}>h %{cvmfs-info}>h" "%{Referer}>h" "%{User-Agent}>h"
    visible_hostname '`uname -n`'/${process_number}
    {{ else }} # single worker case
    cache_dir ufs /var/cache/squid {{ .Values.SquidConf.CacheSize }} 16 256
    {{ end }}

Then we use some cpu affinity business to pin the squid processes to particular cores:

    {{ if .Values.SquidConf.Cpu_Affinity_Map }}
    cpu_affinity_map {{ .Values.SquidConf.Cpu_Affinity_Map }}
    {{ end }}

Set the logfile rotation:

    {{ if .Values.SquidConf.Logfile_Rotate }}
    logfile_rotate {{ .Values.SquidConf.Logfile_Rotate }}
    {{ end }}

What we need to do

We need to change how we customize squid.conf to follow OSG's upstream image. They have made 3 configuration parameters first class citizens:

  1. squid.conf's acl NET_LOCAL src is set by the environment variable SQUID_IPRANGE. We should remove this customization, and instead populate the environment variable SQUID_IPRANGE in deployment.yaml
  2. Likewise, cache_mem should be set by populating the environment variable SQUID_CACHE_MEM
  3. Finally, cache_dir is set by an environment variable as well: SQUID_CACHE_DISK. This gets more complicated because we have two possible values for it depending on the mode in which squid is running (single worker or multiple workers)

For the configuration parameters that are not exposed as environment variables, we need to write some scripts as per https://github.com/opensciencegrid/docker-frontier-squid/blob/master/squid-customize.sh#L8-L19 that suggest making scipts that live in /etc/squid/customize.d.

We need to write a script that populates the following configuration parameters:

  1. Monitoring IP range, i.e., injects the following config: acl HOST_MONITOR src {{ .Values.SquidConf.MonitoringIPRange }}
  2. CPU affinity map, i.e.,
    {{ if .Values.SquidConf.Cpu_Affinity_Map }}
    cpu_affinity_map {{ .Values.SquidConf.Cpu_Affinity_Map }}
    {{ end }}
  1. Logfile rotation
    {{ if .Values.SquidConf.Logfile_Rotate }}
    logfile_rotate {{ .Values.SquidConf.Logfile_Rotate }}
    {{ end }}
  1. Modify cache dir and workers when we have a user request more than 1 squid worker, i.e., the configuration from here:
    {{ if .Values.SquidConf.Workers }}
    workers {{ .Values.SquidConf.Workers }} # multiple worker case
    cache_dir ufs /var/cache/squid/squid${process_number} {{ .Values.SquidConf.CacheSize }} 16 256
    logformat awstats %>a kid${process_number} %un [%{%d/%b/%Y:%H:%M:%S}tl.%03tu %{%z}tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh %tr "%{X-Frontier-Id}>h %{cvmfs-info}>h" "%{Referer}>h" "%{User-Agent}>h"
    visible_hostname '`uname -n`'/${process_number}
    {{ else }} # single worker case
    cache_dir ufs /var/cache/squid {{ .Values.SquidConf.CacheSize }} 16 256
    {{ end }}

I am not sure if this needs to be a pre-awk shell script (i.e., needs to be named customize.d/[0-4]*.sh) or post-awk (customize.d/[5-9]*.sh)

External TLS cert/key not auto-updating for Hosted CE

I noticed the following in the Kubernetes docs:

Note: A container using a Secret as a subPath volume mount will not receive Secret updates.

Unfortunately, subPath is exactly how we handle the mounting of hostcert.pem and hostkey.pem. This means Let's Encrypt updates from cert-manager of the Secret does not propagate into the pod.

I think potentially the best way to do this is to mount the secret in a separate directory, not use subPath, and then point the hosted CE configuration at these files.

frontier-squid is a deployment of size 1 and can not be scaled up

The deployment size is fixed at 1. In principle it could be made a variable and increased, however this would clash with use of ReadWriteOnce PVCs, as each squid pod would try to mount the same volume.

The scenario where you'd want to scale up the squid deployment (e.g. performance and robustness in a production environment with a heavy load on the squids) is also one where you'd care about cold caches for performance reasons and want to use persistent storage via a PVC instead of emptydirs.

The only way to do that is by using a statefulSet instead of a deployment, which provides volumeClaimTemplates so each member of the statefulSet can have its own PVC.

A statefulSet can use emptyDirs but as the converse is not possible (a deployment > 1 can not use PVCs) it seems that a statefulSet would be the only way to make the helm chart scalable. Aside from scalability this would also improve the resilience and availability of the squid service, e.g. on upgrade each member would be updated in a rolling manner instead of a brief outage of the one SPOF pod in the deployment.

Supposing that the deployment is changed to a statefulset, I think the impact on existing users (who use PVCs and a replica size of 1) would only be a one-time cache emptying when upgrading via Helm; the old PVC would be deleted and a new one with a slightly different name would be made and used by the new statefulSet squid.

However having multiple squids behind the same monitoring service might show inconsistent results. In principle this could be handled with separate services for the monitoring port of each individual statefulSet member. That being said I think the same issue exists with multi-worker squid instances? So it may not be a big deal.

Squid does not respect the memory request in multicore

In the UC squid, for example, we set:

  CacheMem: "32768"

When we are using multicore mode, that will do 2 incompatible things:

  1. Set the Kubernetes memory request to 32GB
  2. Give each subprocess 32GB

This means that the container will use N*CacheMem MB of RAM, which is N times larger than what we request!

Let's plan to let the user specify how much memory they want to use, total, and we don't leak the internal abstractions to the user.

frontier squid pods may not get scheduled due to nodeselector if PVC is used

It looks like .Values.SLATE.LocalStorage has multiple functions; it enables use of a PVC but also applies a nodeselector:
https://github.com/slateci/slate-catalog/blob/master/stable/osg-frontier-squid/osg-frontier-squid/templates/deployment.yaml#L28

On a standard cluster (at least on v1.21) it seems nodes do not have a storage: "local" label so by default the pod would not be able to run on any node out of the box. PVCs often use network storage anyway.

HostedCE needs cron for fetchcrl

The HostedCE container needs a crond running under supervisord, such that it can periodically run fetchcrl and keep certs up to date.

This needs to be added into the supervisor config:

[program:crond]
command=/usr/sbin/crond -n
autorestart=true

and we need a corresponding cronjob for running fetchcrl

need to update docs regarding hostNetworking for HostedCE

we added hostNetworking support back into the HostedCE

it would be nice to update the documentation in the values file saying:

  1. hostNetwork is a type of network you can use
  2. Hostname and RequestIP options are ignored in this case

normalize configuration of x509 certificates

Currently we don't create x509 certificates in a consistent way.

For example, the gridftp application creates a secret called x509-certificates and dumps them into a directory, where the container then picks them up and manipulates them appropriately. For example, the invocation for GridFTP looks something like this:

slate secret create <--flags> secretname --from-file=hostcert.pem --from-file=hostkey.pem

XCache creates a secret with named keys and paths, which changes the invocation of the slate secret command. So it's invocation looks like this:

slate secret create <--flags> secretname --from-file=usercert=usercert.pem --from-file=userkey=userkey.pem

We need to normalize these into ONE configuration that will work for SLATE, so the interface is consistent.

Logger Side Car Cannot Use Slate Secret

if [ -z $HTPASSWD ]; then
PASS=$(tr -dc 'a-f0-9' < /dev/urandom | head -c16)
echo "Your randomly generated logger credentials are"
echo "**********************************************"
echo "logger:$PASS"
echo "**********************************************"
HTPASSWD="$(openssl passwd -apr1 $(echo -n $PASS))"
fi
# maybe validate this
mkdir -p /etc/nginx/auth
echo "logger:$HTPASSWD" > /etc/nginx/auth/htpasswd

When using a slate secret, to set the logger web password, it gets added to the file system in plain text instead of hashed like the randomly generated one is. The webserver is expecting a hashed password so log in attempts with the correct password do not work.

I beleive a fix would look something like this:

if [ -z $HTPASSWD ]; then
      HTPASSWD=$(tr -dc 'a-f0-9' < /dev/urandom | head -c16)
      echo "Your randomly generated logger credentials are"
      echo "**********************************************"
      echo "logger:$HTPASSWD"
      echo "**********************************************"
fi

# maybe validate this
mkdir -p /etc/nginx/auth
echo "logger:$(openssl passwd -apr1 $(echo -n $HTPASSWD))" > /etc/nginx/auth/htpasswd

frontier-squid requests unnecessary ephemeral-storage when PVCs are used

When .Values.SLATE.LocalStorage (side note: that's a bit of a misnomer because PVCs are not necessarily local), a PVC is used for the squid data volume, which makes up most of the storage usage. However that doesn't stop it from requesting ephemeral storage larger than the squid data volume:

https://github.com/slateci/slate-catalog/blob/master/stable/osg-frontier-squid/osg-frontier-squid/templates/deployment.yaml#L71

That means the pod requests a big emptydir , large enough for the squid data cache, but it will not use it since the cache is actually on PVC instead. This wastes disk space and makes the squid pod harder to schedule.
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-emphemeralstorage-consumption

Version label prevents normal helm upgrades

The OSG folks are having issues doing upgrades using the Helm operator for Flux - specifically it seems to be a problem with having the Chart version in the label of our applications.

The errors are something like this:

ts=2021-04-26T22:48:56.5489564Z caller=release.go:357 component=release release=osgdev-chtc-itb-slurm-ce targetNamespace=osgdev resource=osgdev:helmrelease/chtc-itb-slurm-ce helmVersion=v3 error="upgrade failed: cannot patch \"osg-hosted-ce-chtc-itb-slurm\" with kind Deployment: Deployment.apps \"osg-hosted-ce-chtc-itb-slurm\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"chtc-itb-slurm\", \"chart\":\"osg-hosted-ce-3.8.2\", \"instance\":\"chtc-itb-slurm\", \"release\":\"osgdev-chtc-itb-slurm-ce\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable" action=upgrade

I think Helm cannot upgrade because we have the Chart version baked into the chart. Updating the Chart changes the value and breaks the selector, as I understand it.

Can we remove it?

frontier-squid storageClassName is hard coded to "nfs-provisioner"

https://github.com/slateci/slate-catalog/blob/master/stable/osg-frontier-squid/osg-frontier-squid/templates/pvc.yaml#L17

Each cluster may have different CSI provisioners so the storageClassName should be a configurable variable.
In fact, usually it shouldn't need to be set because the cluster's default storageClass should be sufficient. I think a PVC should only need to specify a storageClass if there are multiple types of storage available and it has a strong opinion on which type of storage it wants, so having storageClassName absent (unless defined by the user) may be a suitable default.

Note that the DefaultStorageClass admission controller is enabled by default.

Moreover, to support the use of statically provisioned volumes, it would be good for the helm chart to also support setting volumeName, with storageClassName: "" as described here:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reserving-a-persistentvolume
NB storageClassName: "" is different from not having a storageClassName.

Built index.yaml missing all chart versions

  • This was initially reported by Brian Lin with regards to the osg-hosted-ce application.
  • This bug is not present in the new GitHub action-driven catalogs.

Bug

The index.yaml files built by Jenkins for:

do not contain all available chart versions. For example, in index.yaml for nginx:

nginx:
  - apiVersion: v1
    appVersion: 1.15.9
    created: "2022-07-24T09:46:15.105994007Z"
    description: A simple nginx deployment which serves a static page
    digest: 2fbcb560c62b94ba34962297acf423c0c5aba06f4d1b00cba0c0628368f8c94e
    name: nginx
    urls:
    - nginx-1.2.0.tgz
    version: 1.2.0

This results in:

$ slate app versions nginx
Fetching application versions...
...
1.2.0

despite nginx versions all the way back to 1.0.0 being available on https://jenkins.slateci.io/catalog/stable/.

Proposed Solution

Update CMakeLists.txt used by Jenkins to include all available versions on the pages linked above in the built index.yaml files.

Clarify & Simplify logging options

We have 6 logging options:

Logfile_Rotate
MaxAccessLog
UseHostpathLogDir
LogToStdout
DisableLogging
CleanLog

Users probably want something more like

I want to give Squid N MB for logging.

Or even more ideally,

I want to keep Squid logs of X days/weeks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.