In this workshop we will configure and understand Knox authentication to use with Hadoop services in a cluster. For this workshop We need Kerberos enabled CDP-DC 7.X cluster with Knox installed.

• LAB 1:

  • Knox Overview
  • Knox Usecase
  • Knox Configs

• LAB 2:

  • Install Knox

• LAB 3:

  • Generating API and UI topologies in Knox CSD
    • Adding a known service to cdp-proxy
    • Adding a custom service parameter to a known service (assuming the service is already enabled)
    • Updating a custom service parameter
    • Removing a custom service parameter
    • Removing a known service from cdp-proxy
    • Adding a custom service in cdp-proxy
    • Adding a custom topology in the deployed Knox Gateway

• LAB 4:

  • Managing Knox shared providers in Cloudera Manager
    • SSO authentication provider
      • PAM to LDAP
    • API Authentication Provider
      • PAM to LDAP
    • Saving Aliases

• LAB 5:

  • Modifying an existing shared provider
    • Disabling a provider in an existing provider configuration
    • Modifying a provider in an existing provider configuration
    • Add a new provider in an existing provider configuration
    • Adding a new shared provider configuration

• LAB 6:

  • Replace knox ssl certificate
  • Knox SSO with keycloak

• LAB 7:

  • Troubleshooting
    • How authentication works and its logging
    • Error codes
    • How Knoxsso works, and its logging

• LAB 8:

  • Usecase
    • Configure knox kerberos topology using CM
    • Configure default sso provider using CM (Enable group lookup for cdp-proxy using identity-assertion as HadoopGroupProvider)
    • Configure default sso provider for (shiro authentication + group lookup) using CM
    • Access atlas-api using Knox in Non-Kerberos env (we recommnded kerberos env for knox else most of the service wont work and we dont have list of it)

• Knox can only be deployed in secure clusters with kerberos enabled
• Knox will need to be explicitly installed - it is NOT installed by default in secure clusters
• We will have two topologies
  • cdp-proxy for UI access
  • cdp-proxy-api for API access
• These two topologies are managed in CM
  • sets of topology specific checkboxes to include UIs and APIs within the topology and enable CM based discovery of them by Knox
  • Provider configuration to specify the authentication and authorization provider configuration for each topology is also managed within CM
• The same trusted proxy model used in CDP public cloud will be leveraged in DC
• Knox itself does nothing to block direct access to UIs and APIs but it is common for clusters to be surrounded by a firewall of one sort or another and access to the internal resources of the cluster to go through Knox. At the end of the day, direct access by external clients is a deployment decision for DC unlike CDP public cloud.
  • once inside the cluster, there is direct access to those resources for which you have line of sight
• We have Knox Homepage which provide a list of UI and API endpoints.

I] Fresh 7.x DC Cluster

1. Knox is not installed
2. Cluster is kerberized
3. Knox is installed
   • All UIs and APIs are selected for autodiscovery and proxying by Knox by default - admin may opt-out of those unwanted via checkboxes in Knox admin page
   • Additional config may be required to enable trusted proxy for certain services or put HS2 in HTTP mode, etc.
   • Any required config changes may require restart of one or more service
4. URLs for UIs and APIs will be made available as QuickLinks|Home Page|Client Configs
5. Gateway URLs will be able to be used for UI access
6. Gateway URLs for API access
   • HTTP Basic is the default authentication mechanism for proxied API access
   • By default it authenticates against PAM (again assuming local accounts due to a secure cluster)
   • Authentication is done at the gateway via the Shiro authentication provider
   • Upon successful authentication the request is dispatched to the backend service API - propagating the effective user via trusted proxy (kerberos+doAs)

Step 1: Setup CDP-DC cluster with latest version.(Use squadron or Ycloud)

Step 2: Enable Kerberos (using squadron one click script/ Ycloud option)

Step 3: Add Knox service Follow the documentation.

Doc: Installing Apache Knox!

• We need pass master secret key at the time of installation.
• The master secret is required to start the server. This secret is used to access secured artifacts by the gateway instance. By default, the keystores, trust stores, and credential stores are all protected with the master secret.
• It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessible by the user that the gateway is running as.

Once Knox is Installed.

• By default, Knox authentication for default topologies are set to PAM.

Step 4: We need to create a pam user and he should be part of admin group on all Knox Host.

$ vi /etc/pam.d/cdp-dc

#%PAM-1.0
auth sufficient pam_unix.so
auth sufficient pam_sss.so
account sufficient pam_unix.so
account sufficient pam_sss.so

$ chmod 444 /etc/shadow

# create a user who is part of admin group

$ useradd admin
$ usermod -a -G admin admin
$ id admin

Step 5: Use above username and password to login into Knox Homepage UI or Admin page UI.

In CDP DC there is no Cloudbreak who can deploy cdp-proxy and cdp-proxy-api topologies in Knox. To make it easier for end-users we introduced two new CM configurations where these topologies can be managed within CM:

• Knox Simplified Topology Management - cdp-proxy
• Knox Simplified Topology Management - cdp-proxy-api

Each of these configurations describes a Knox topology called cdp-proxy and cdp-proxy-api, respectively. End-users are allowed to add/remove/update services into/from these topologies as well as creating their own custom topology(ies) by editing the appropriate safety valve (see below).

It is very important that these topologies will be deployed by CM only, and only if Knox’s service auto-discovery feature is turned on using the Enable/Disable Service Auto-Discovery checkbox on CM UI:

As you can see, the Knox CSD comes with some pre-defined service parameters out-of-the-box (e.g. CM-API:httpclient.connectionTimeout=5m), but it does not mean CM-API will be added in cdp-proxy and/or cdp-proxy-api topologies.

If end-users want a known service (that is, the service is an officially supported Knox service with all the required service definition files) to be included in these topologies they need to enable that service on CM UI using the corresponding gateway_auto_discovery_[cdp-proxy|cdp-proxy-api]enabled$service checkbox.

For instance, enabling Atlas API and UI services in cdp-proxy end-users should check [gateway_auto_discovery_cdp_proxy_enabled_atlas[ and gateway_auto_discovery_cdp_proxy_enabled_atlas_ui:

If end-users want a custom service (that is, the service is not officially supported by the Knox team, but end-users have their own service definition files) in cdp-proxy and/or cdp-proxy-api topologies they simply need to add the service declaration in any of the above shown Knox Simplified Topology Management panels.

In the following samples, I’ll guide you on how to add/remove/update known and custom services to cdp-proxy.

In this sample, we are going to add ATLAS and ATLAS UI to cdp-proxy. You should simply enable gateway_auto_discovery_cdp_proxy_enabled_atlas and gateway_auto_discovery_cdp_proxy_enabled_atlas_ui checkboxes on Knox’s Configuration page in CM and save the changes. As a result, the 'Refresh needed' stale configuration indicator appears. You should click it and wait until the refresh process finished.

You can check if ATLAS and ATLAS-UI were added to cdp-proxy by hitting the following URL: (https://$KNOX_GATEWAY_HOST:PORT/$GATEWAY_PATH/admin/api/v1/topologies/cdp-proxy)

In this sample, we are going to add a custom service parameter with a custom value (myCustomServiceParameter=myValue) to ATLAS in cdp-proxy.

It is as simple as adding a new line in the Knox Simplified Topology Management - cdp-proxy panel in the following format: $SERVICE_NAME[:$PARAMETER_NAME=$PARAMETER_VALUE]. The 'url' and 'version' parameter names are preserved keywords to set the given service's URL and version. Valid declarations:

HIVE:url=http://localhost:123
HIVE:version:3.0.0
HIVE:test.pramameter.name=test.parameter.value

After you added the new ATLAS entry and saved the changes the 'Refresh needed' stale configuration indicator appears. You should click it and wait until the refresh process finished.

You can confirm if ATLAS in cdp-proxy got updated with the new service parameter the same way as described above. You can check by hitting the following URL:

(https://$KNOX_GATEWAY_HOST:PORT/$GATEWAY_PATH/admin/api/v1/topologies/cdp-proxy)

In this sample, we are going to update a previously entered service parameter - myCustomServiceParameter=myValue to myNewValue- from ATLAS in cdp-proxy. We simply change that entry, save our changes and refresh our cluster.

In this sample, we are going to remove a previously entered service parameter - myCustomServiceParameter=myNewValue - from ATLAS in cdp-proxy. We simply remove that entry, save our changes and refresh our cluster.

In this sample, we are going to remove the previously added ATLAS and ATLAS-UI services from cdp-proxy. We disable the gateway_auto_discovery_cdp_proxy_enabled_atlas and gateway_auto_discovery_cdp_proxy_enabled_atlas_ui checkboxes on Knox’s Configuration page in CM, save the changes and refresh the cluster.

In this sample, we are going to add a custom service (MY_SERVICE) in cdp-proxy with the following attributes:

• version - the service’s version. We will set it to 1.0.0
• url - the service URL. We will set it to https://sampleHost:1234
• service parameter - a sample service parameter. We will define a custom service parameter just like above for ATLAS

It is very important to note that adding a custom service only makes sense if the service definition files (service.xml and rewrite.xml) are existing in the KNOX_DATA_DIR/services folder.

To achieve the goals we need to add 3 new entries with the above-listed parameters in Knox Simplified Topology Management - cdp-proxy. Then we save the changes, refresh the cluster and check if the newly added custom service is available in cdp-proxy.

In this sample, we are going to add a custom service (MY_SERVICE) in custom-topology with the following attributes:

• providerConfigRef - a string representing a reference of an existing share-provider. We will use the pre-configured pam provider
• version - the service’s version. We will set it to 1.0.0
• url - the service URL. We will set it to https://sampleHost:1234
• service parameter - a sample service parameter. We will define a custom service parameter just like above for ATLAS

To achieve our goals we need to add a new entry in Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml:

Name = custom-topology
Value =providerConfigRef=pam#
MY_SERVICE:version=1.0.0#
MY_SERVICE:url=https://sampleHost:1234#
MY_SERVICE:myCustomServiceParameter=myValue
#Description = This is a custom topology with one service called MY_SERVICE

In addition to the above-described attributes, you can also define service-discovery related configuration in your custom descriptor:

• discoveryType - the type of service discovery. Valid values are: ClouderaManager, Ambari
• discoveryAddress - the service-discovery end-point
• discoveryUser - the user name which the service discovery feature uses to connect to the previously set address. This is optional, if not present Knox will try to fetch the discovery user using the previously saved cm.discovery.user Gateway level alias (CM generates it automatically in case service discovery is enabled).
• discoveryPasswordAlias - the password alias (on Gateway level) the service discovery uses to get the connection password using Knox’s alias service. Please note that the alias should be created in advance! This is optional, if not present Knox will try to fetch the discovery password using the previously saved cm.discovery.password Gateway level alias (CM generates it automatically in case service discovery is enabled).
• cluster - the cluster to be discovered

• Modifying the SSO authentication provider used by the UIs using the Knox SSO capabilities such as the Admin and Home Page UIs
• Modifying the API authentication provider used by pre-defined topologies such as admin, metadata or cdp-proxy-api
• Adding/modifying new/existing shared provider configurations
• Saving aliases using a new Knox Gateway command

With the newest version of CM a new Knox configuration has been added, called Knox Simplified Topology Management - SSO Authentication Provider, with the following initial configuration:

role=authentication
authentication.name=ShiroProvider
authentication.param.sessionTimeout=30
authentication.param.redirectToUrl=/${GATEWAY_PATH}/knoxsso/knoxauth/login.html
authentication.param.restrictedCookies=rememberme,WWW-Authenticate
authentication.param.urls./**=authcBasic
authentication.param.main.pamRealm=org.apache.knox.gateway.shirorealm.KnoxPamRealm
authentication.param.main.pamRealm.service=login

Every change here goes directly into knoxsso topology that affects manager, homepage and cdp-proxy topologies as they are using the federation provider.

In the following sample you will see how to change the PAM authentication (which comes OOTB with Knox) to LDAP authentication. It is as simple as replacing the PAM related configuration entries in the above ShiroProvider to LDAP related properties (e.g. with demo LDAP server configuration):

Knox first authenticates the end user, and then adds that user as a query parameter to the request (?doAs=USERNAME) to the backend. The backend then checks that the request is trusted (request IP and request user) and extracts the end user (USERNAME) from the query parameter. The backend service then does whatever is necessary as that backend user. Knox and the proxied services authenticate to each other via Kerberos.

role=authentication
authentication.name=ShiroProvider
authentication.param.sessionTimeout=30
authentication.param.redirectToUrl=/${GATEWAY_PATH}/knoxsso/knoxauth/login.html
authentication.param.restrictedCookies=rememberme,WWW-Authenticate
authentication.param.urls./**=authcBasic
authentication.param.main.ldapRealm=org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm
authentication.param.main.ldapContextFactory=org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory
authentication.param.main.ldapRealm.contextFactory=$ldapContextFactory
authentication.param.main.ldapRealm.contextFactory.authenticationMechanism=simple
authentication.param.main.ldapRealm.contextFactory.url=ldap://localhost:33389
authentication.param.main.ldapRealm.contextFactory.systemUsername=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org
authentication.param.main.ldapRealm.contextFactory.systemPassword=${ALIAS=knoxLdapSystemPassword}
authentication.param.main.ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org

# Start Demo LDAP
get the parcel location from Knox Node:

$ ps aux | grep knox
$ /opt/cloudera/parcels/CDH-7.2.1-1.cdh7.2.1.p0.4847773/lib/knox/bin/ldap.sh start

After you finished editing the properties you have to save the configuration changes. This will make the Refresh Needed stale configuration indicator appear. Once the cluster refresh finishes, all topologies that are configured to use Knox SSO will be authenticated by the configured LDAP server.

Similarly to the previous point, a new Knox configuration has been added, called ,Knox Simplified Topology Management - API Authentication Provider, with the following initial configuration:

role=authentication
authentication.name=ShiroProvider
authentication.param.sessionTimeout=30
authentication.param.urls./**=authcBasic
authentication.param.main.pamRealm=org.apache.knox.gateway.shirorealm.KnoxPamRealm
authentication.param.main.pamRealm.service=login

Every change here goes directly into admin, metadata and cdp-proxy-api topologies. Since the only difference between this initial configuration and the one Knox SSO uses is two additional parameters (redirectToUrl and restrictedCookies) I would not like to repeat the same steps on how to change PAM authentication to LDAP as they are exactly the same.

https://github.com/bhagadepravin/commands/blob/master/knox-cdp-dc.md

There is a new command available for the Knox Gateway role which allows end-users to save an alias=password pair to an arbitrary number of topologies on each host where an instance of the Knox Gateway is installed without the need of running the Knox CLI tool manually.

A new password-type input field is added, called save_alias_command_input_password. The format of an entry in this input field should be: topology_name_1[:topology_name_2:...:topology_name_N].alias_name=password

Sample: cdp-proxy-api:admin:metadata.knoxLdapSystemPassword=guest-password

After the end-user entered a meaningful and valid value and saved the configuration changes he/she can run the command from Knox’s action list: Actions/Save Alias

Tip: if you need to add a Gateway level alias, please use __gateway as topology name. For instance: __gateway.knoxLdapSystemPassword=admin-password

At the time of this document being written, the following shared provider configurations are deployed in CDP DC with Knox OOTB (pam and sso are available only if service auto-discovery is enabled fo Knox Gateway role): admin (used by the admin topology) homepage (used by the homepage topology) knoxsso (homepage, manager and cdp-proxy topologies are configured to use this one) manager (used by the manager topology) metadata (used by the metadata topology) pam (used by the cdp-proxy topology) pam (used by the cdp-proxy-api topology)

The following changes are allowed in any of these shared providers:

• disable a particular provider
• modify a particular provider
• add a new provider

All of these actions can be done via editing the Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml by implementing the following language elements:

• the key of a new entry should be like this: providerConfigs: providerConfig_1 [,providerConfig_2,..,providerConfig_3]
• the value should contain the following name/value pairs separated by a hash (#) character:
  • $role=webappsec|authentication|federation|identity-assertion|authorization|hostmap|ha
  • $role.name=ROLE_NAME (e.g. ShiroProvider)
  • $role.enabled=true|false (optional; defaults to 'true')
  • $role.param.param_1=value_1 (parameters are optional too).
  • $role.param_N.param1=value_N

In this sample you will see how to disable the authorization provider in the manager shared provider configuration. This particular authorization provider is set as follows (in its JSON descriptor):

{
         "role": "authorization",
         "name": "AclsAuthz",
         "enabled": "true",
         "params": {
            "knox.acl.mode": "OR",
            "knox.acl": "KNOX_ADMIN_USERS;KNOX_ADMIN_GROUPS;*"
         }
      }

To disable it the following should be done:

• 1. add the following entry in the above-referenced safety valve:
  • name = providerConfigs:manager
  • value = role=authorization#authorization.name=AclsAuthz#authorization.enabled=false#authorization.param.knox.acl=KNOX_ADMIN_USERS;KNOX_ADMIN_GROUPS;#authorization.param.knox.acl.mode=OR*
• 2. save your changes
• 3. refresh the cluster

As you can see only the enabled flag was changed.

In this sample you will see how to modify the previously disabled authorization provider in the manager shared provider configuration. The steps are the same as described in the section before but the value of the safety valve entry is different:

role=authorization#authorization.name=AclsAuthz#authorization.enabled=false#authorization.param.knox.acl=myTestUser;KNOX_ADMIN_GROUPS;#authorization.param.knox.acl.mode=OR*

With this change you are authorizing a user called myTestUser to login and execute administrative actions on the Knox Admin UI.

As of now, the manager shared provider configuration does not have the HA provider set. In this sample you will see how to add a new HA provider (this time only the ATLAS service will be configured for high availability) in the manager shared provider configuration with keeping all the changes from the previous sections.

As you might figured it out, the steps are the same again except for the value in the safety valve, which has to be set to this one:

role=authorization#authorization.name=AclsAuthz#authorization.enabled=false#authorization.param.knox.acl=myTestUser;KNOX_ADMIN_GROUPS;#authorization.param.knox.acl.mode=OR#role=ha#ha.name=HaProvider#ha.param.ATLAS=enabled=true;maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000*

It is possible that you add a brand new shared provider configuration too. In this sample you will see how to create testProviders with the following providers set:

• authentication: ShiroProvider / PAM
• identity-assertion: Default
• authorization: Ranger (XASecurePDPKnox)

The following should be added in the :

• 1. add the following entry in the above-referenced safety valve:
     • name = providerConfigs:testProviders
     • value = role=authentication#authentication.name=ShiroProvider#authentication.param.main.pamRealm=org.apache.knox.gateway.shirorealm.KnoxPamRealm#authentication.param.main.pamRealm.service=login#role=identity-assertion#identity-assertion.name=Default#role=authorization#authorization.name=XASecurePDPKnox
• 2. save your changes
• 3. refresh the cluster

Since all providers you add via this mechanism are enabled by default I shortened the value field by omitting the enabled flag.

curl -ik -u knoxui:knoxui "https://KNOX-HOSTNAME:8443/gateway/admin/api/v1/providerconfig/testProviders"

• We can also use Knox Admin-UI to create shared provider: https://KNOX-HOSTNAME:8443/gateway/manager/admin-ui/

Here We have Two Options,

1. Auto-TLS Disabled
2. Auto-TLS Enabled

# Check the existing Knox certificate:
$ openssl s_client -showcerts -connect localhost:8443	
$ ll /var/lib/knox/gateway/data/security/keystores/gateway.jks
$ /usr/java/jdk1.8.0_232-cloudera/bin/keytool -list -keystore /var/lib/knox/gateway/data/security/keystores/gateway.jks

ls -ltr /var/lib/knox/gateway/data/security/keystores
cd /var/lib/knox/gateway/data/security/keystores/
mkdir /tmp/backup
mv * /tmp/backup

cd /var/lib/knox/gateway/data/security/keystores/
/usr/java/jdk1.8.0_232-cloudera/bin/keytool -genkey -alias gateway-identity -keyalg RSA -keysize 1024 -dname "CN=c474-node4.supportlab.cloudera.com,OU=INTERNL,O=Hadoop,L=SELF,ST=TEST,C=US" -keypass Welcome -keystore gateway.jks -storepass Welcome -keypass Welcome
chown knox:knox /var/lib/knox/gateway/conf/gateway.jks

cd /opt/cloudera/parcels/CDH-*/lib/knox/bin
pwd
export KNOX_GATEWAY_CONF_DIR=/var/lib/knox/gateway/conf
export KNOX_GATEWAY_DATA_DIR=/var/lib/knox/gateway/data
export KNOX_GATEWAY_LOG_DIR=/var/log/knox/gateway
export KNOX_GATEWAY_LOG_OPTS="-Dlog4j.configuration=/var/lib/knox/gateway/conf/gateway-log4j.properties"
export KNOX_CLI_LOG_OPTS="-Dlog4j.configuration=/var/lib/knox/gateway/conf/knoxcli-log4j.properties"

./knoxcli.sh create-master --force
ls -ltr /var/lib/knox/gateway/data/security/master
chown knox:knox /var/lib/knox/gateway/data/security/master

ls -ltr /var/lib/knox/gateway/data/security/keystores

# Check the new Knox certificate: 
$ openssl s_client -showcerts -connect localhost:8443

By default SSL is enabled on Knox, if Auto-TLS is enabled certificates are replaced as per CM configurations:

Follow below steps to replace existing SSL certificate for Knox.

grep -a2 jks /var/lib/knox/gateway/conf/gateway-site.xml

/usr/java/jdk1.8.0_232-cloudera/bin/keytool -genkey -alias gateway-identity -keyalg RSA -keysize 1024 -dname "CN=pbhagade-boo-1.pbhagade-boo.root.hwx.site,OU=INTERNL,O=Hadoop,L=SELF,ST=TEST,C=US" -keypass Welcome -keystore gateway.jks -storepass Welcome -keypass Welcome

cp gateway.jks /var/lib/knox/gateway/conf/
ls -ltr /var/lib/knox/gateway/conf/gateway.jks
chown knox:knox /var/lib/knox/gateway/conf/gateway.jks

cd /var/lib/knox/gateway/data/security/keystores
mkdir backup
mv * backup/

#Create truststore

/usr/java/jdk1.8.0_232-cloudera/bin/keytool -import -file /opt/cloudera/CMCA/trust-store/cm-auto-in_cluster_ca_cert.pem -keystore /var/lib/knox/gateway/conf/truststore.jks -alias truststore -storepass Welcome

cd /opt/cloudera/parcels/CDH-7.2.1-1.cdh7.2.1.p0.4847773/lib/knox/bin

export KNOX_GATEWAY_CONF_DIR=/var/lib/knox/gateway/conf
export KNOX_GATEWAY_DATA_DIR=/var/lib/knox/gateway/data
export KNOX_GATEWAY_LOG_DIR=/var/log/knox/gateway
export KNOX_GATEWAY_LOG_OPTS="-Dlog4j.configuration=/var/lib/knox/gateway/conf/gateway-log4j.properties"
export KNOX_CLI_LOG_OPTS="-Dlog4j.configuration=/var/lib/knox/gateway/conf/knoxcli-log4j.properties"

./knoxcli.sh create-master --force


[root@pbhagade-boo-1 bin]# ./knoxcli.sh create-master --force
***************************************************************************************************
You have indicated that you would like to persist the master secret for this service instance.
Be aware that this is less secure than manually entering the secret on startup.
The persisted file will be encrypted and primarily protected through OS permissions.
***************************************************************************************************
Enter master secret:
Enter master secret again:
Master secret has been persisted to disk.
[root@pbhagade-boo-1 bin]#

ls -ltr /var/lib/knox/gateway/data/security/master
chown knox:knox /var/lib/knox/gateway/data/security/master

CM UI -> KNOX -> Configuration --> Security - TLS Certificate Alias (Optional)

gateway_tls_certificate_alias=gateway-identity

CM UI -> KNOX -> Configuration --> Knox Gateway TLS/SSL Server JKS Keystore File Location

CM UI -> KNOX -> Configuration --> Knox Gateway TLS/SSL Client Trust Store File

gateway.tls.keystore.path
gateway.httpclient.truststore.path

/opt/cloudera/parcels/CDH-7.2.1-1.cdh7.2.1.p0.4847773/lib/knox/bin/knoxcli.sh create-alias cm.discovery.user --value admin
/opt/cloudera/parcels/CDH-7.2.1-1.cdh7.2.1.p0.4847773/lib/knox/bin/knoxcli.sh create-alias cm.discovery.password --value admin
/opt/cloudera/parcels/CDH-7.2.1-1.cdh7.2.1.p0.4847773/lib/knox/bin/knoxcli.sh create-alias gatewaykeystorepassword --value Welcome
/opt/cloudera/parcels/CDH-7.2.1-1.cdh7.2.1.p0.4847773/lib/knox/bin/knoxcli.sh create-alias gatewaytruststorepassword --value Welcome

Check if the log file for any issues:

tailf /var/log/knox/gateway/gateway.log

Use openssl cmd to verify the certificate:

openssl s_client -connect localhost:8443

Before you start Make sure you have Docker installed.

yum install yum-utils device-mapper-persistent-data lvm2 -y
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce -y
# or
# yum install docker -y

systemctl start docker
systemctl status docker
systemctl enable docker

Start Keycloak From a terminal start Keycloak with the following command:

$ docker run -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin -e DB_VENDOR=H2 -p 8080:8080 --name keycloak jboss/keycloak

Wait until you see

12:40:52,608 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990

This will start Keycloak exposed on the local port 8080. It will also create an initial admin user with username admin and password admin.

Login to the admin console Go to the Keycloak Admin Console http://hostname:8080/auth/admin and login with the username and password you created earlier.

Ref: https://www.keycloak.org/getting-started/getting-started-docker

Crtl+C
$ docker start keycloak

Create a new realm

• 1. Create SAML client in keycloak.
• 2. Sample config

• 3. Create user account under the realm and set password.
• 4. Get the IDP SSO metadata using keycloak url :

# curl -ik https://keycloak-FQDN:port/auth/realms/KeycloakRealmName/protocol/saml/descriptor -o idp.xml

curl -ik -u admin:admin http://pbhagade-boo-1.pbhagade-boo.root.hwx.site:8080/auth/realms/workshop/protocol/saml/descriptor  -o idp.xml

• 5. Copy idp.xml file to the knox host. Remove HTTP header from idp.xml file
• 6. With the newest version of CM a new Knox configuration has been added, called Knox Simplified Topology Management - SSO Authentication Provider, with the following initial configuration:

Tip: You can also use Knox Admin UI or manually create custom saml topology.

role=authentication
authentication.name=ShiroProvider
authentication.enabled=false
role=federation
federation.name=pac4j
federation.param.clientName=SAML2Client
federation.param.pac4j.callbackUrl=https://pbhagade-boo-1.pbhagade-boo.root.hwx.site:8443/gateway/knoxsso/api/v1/websso
federation.param.saml.identityProviderMetadataPath=/etc/knox/conf/idp.xml
federation.param.saml.serviceProviderEntityId=Knox-saml-workshop
authentication.param.remove=main.pamRealm
authentication.param.remove=main.pamRealm.service

• 7. Setup any service for SSO authentication and verify the SSO redirection and authentication.
• 8. Sample config knoxsso.xml

<topology>
    <generated>true</generated>
    <gateway>
        <provider>
            <role>webappsec</role>
            <name>WebAppSec</name>
            <enabled>true</enabled>
            <param>
                <name>xframe.options.enabled</name>
                <value>true</value>
            </param>
        </provider>
        <provider>
            <role>authentication</role>
            <name>ShiroProvider</name>
            <enabled>false</enabled>
            <param>
                <name>main.pamRealm</name>
                <value>org.apache.knox.gateway.shirorealm.KnoxPamRealm</value>
            </param>
            <param>
                <name>main.pamRealm.service</name>
                <value>login</value>
            </param>
            <param>
                <name>redirectToUrl</name>
                <value>/${GATEWAY_PATH}/knoxsso/knoxauth/login.html</value>
            </param>
            <param>
                <name>restrictedCookies</name>
                <value>rememberme,WWW-Authenticate</value>
            </param>
            <param>
                <name>sessionTimeout</name>
                <value>30</value>
            </param>
            <param>
                <name>urls./**</name>
                <value>authcBasic</value>
            </param>
        </provider>
        <provider>
            <role>federation</role>
            <name>pac4j</name>
            <enabled>true</enabled>
            <param>
                <name>clientName</name>
                <value>SAML2Client</value>
            </param>
            <param>
                <name>pac4j.callbackUrl</name>
                <value>https://pbhagade-boo-1.pbhagade-boo.root.hwx.site:8443/gateway/knoxsso/api/v1/websso</value>
            </param>
            <param>
                <name>saml.identityProviderMetadataPath</name>
                <value>/etc/knox/conf/idp.xml</value>
            </param>
            <param>
                <name>saml.serviceProviderEntityId</name>
                <value>knox-pravin</value>
            </param>
        </provider>
        <provider>
            <role>identity-assertion</role>
            <name>Default</name>
            <enabled>true</enabled>
        </provider>
    </gateway>

    <service>
        <role>KNOXSSO</role>
        <param>
            <name>knoxsso.token.ttl</name>
            <value>86400000</value>
        </param>
    </service>
    <application>
        <name>knoxauth</name>
    </application>
</topology>

Note

(Changes required for properties pac4j.callbackUrl, saml.identityProviderMetadataPath and saml.serviceProviderEntityId)

When things aren’t working the first thing you need to do is examine the diagnostic logs. Depending upon how you are running the gateway these diagnostic logs will be output to different locations.

Apache Knox Gateway is a reverse proxy that authenticates and provides a single access point for REST and HTTP interactions with the CDP Data Center clusters.

There are two types of providers supported in Knox for establishing a user’s identity:

1. Authentication Providers
2. Federation Providers

Authentication providers directly accept a user’s credentials and validates them against some particular user store.

Federation providers, on the other hand, validate a token that has been issued for the user by a trusted Identity Provider (IdP).

The current release of Knox ships with an authentication provider based on the Apache Shiro project and is initially configured for BASIC authentication against an LDAP store. This has been specifically tested against Apache Directory Server and Active Directory.

<gateway>
    <provider>
        <role>authentication</role>
        <name>ShiroProvider</name>
        <enabled>true</enabled>
        <param>
            <name>main.ldapRealm</name>
            <value>org.apache.shiro.realm.ldap.JndiLdapRealm</value>
        </param>
        <param>
            <name>main.ldapRealm.userDnTemplate</name>
            <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
        </param>
        <param>
            <name>main.ldapRealm.contextFactory.url</name>
            <value>ldap://localhost:33389</value>
        </param>
        <param>
            <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
            <value>simple</value>
        </param>
        <param>
            <name>urls./**</name>
            <value>authcBasic</value>
        </param>
    </provider>

https://knox.apache.org/books/knox-1-1-0/user-guide.html#Example+provider+configuration+to+use+advanced+LDAP+authentication

Test sample WEBHDFS curl cmd:

curl -iku knoxui:knoxui https://localhost:8443/gateway/cdp-proxy-api/webhdfs/v1/?op=LISTSTATUS

If necessary you can enable additional logging by editing the Knox Gateway Logging Threshold. Changing the Knox Gateway Logging Threshold value from ERROR to DEBUG will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.

Test sample WEBHDFS curl cmd and observe the logging with INFO later with DEBUG enabled.

curl -iku knoxui:knoxui https://localhost:8443/gateway/cdp-proxy-api/webhdfs/v1/?op=LISTSTATUS

tail -f /var/log/knox/gateway/gateway.log /var/log/knox/gateway/gateway-audit.log

Test sample WEBHDFS curl cmd with incorrect password and observe the logging.

curl -iku knoxui:knoxui https://localhost:8443/gateway/cdp-proxy-api/webhdfs/v1/?op=LISTSTATUS

tail -f /var/log/knox/gateway/gateway.log /var/log/knox/gateway/gateway-audit.log

• 401 (authentication error)
• 403 (forbidded error)
• 404 (NPE,incorrect configs)
• 413 (Header issue)
• 500 (server side issue)

Below describes what should happen:

1. Service A is configured to use KnoxSSO (knox sso url, public key, cookie name)
2. Try to access http://serviceA.abc.com
3. Browser redirect from serviceA.abc.com to knoxsso URL
4. KnoxSSO prompts for login information or redirects to backend that prompts for login (ie: okta)
5. If successful login, Knox creates a signed JWT hadoop-jwt cookie (signed with Knox private key)
6. KnoxSSO tries to redirect back to Service A via browser
7. Service A tries to read hadoop-jwt cookie and verifies with KnoxSSO public key
8. If Service A is able to read hadoop-jwt cookie, then Service A says you are logged in

• Misconfiguration with Service A - Knox SSO is not configured for Service A
  • Missing/misconfigured jetty servlet

• Either Knox SSO is not running at that URL or there is an issue with Knox SSO
• Check Knox gateway.log and backend logs (like LDAP, Okta)

• Knox SSO has a few checks to make sure it is not an open redirect.
  • The redirecting.jsp page will say there is an error.
• Check gateway.log and it will have a message about configuring the whitelist
• Check that gateway-site.xml gateway.dispatch.whitelist.services contains knoxauth
  • BUG-114475 - Ambari <2.7.3, knoxauth is not set in gateway.dispatch.whitelist.services
• gateway-site.xml has gateway.dispatch.whitelist which is the default redirect whitelist config
  • KNOX-1577 details some issues with default whitelist and not escaping the / parameters
  • Requires gateway.dispatch.whitelist.services to be set correctly
• knoxsso.redirect.whitelist.regex is a topology level config that overrides the redirect whitelist setting in gateway-site.xml
  • knoxsso.redirect.whitelist.regex is set in the KnoxSSO topology

If you think you set the whitelist correctly and its still not working, checking Knox gateway.log since there will be a message about the host it tried to redirect to and the whitelist that didn’t match. This should give clues as to what the problem is.

• Cookie could be set to secure but Service A is not secure (not HTTPS)
  • knoxsso.cookie.secure.only in KnoxSSO topology
• The cookie could have the wrong domain.
  • Browser will not present the cookie to Service A if the domain doesn’t match
• Both Knox and Service A need to share a common part of the domain - knoxsso.cookie.domain.suffix in KnoxSSO topology
• Service A is looking for a different cookie name than hadoop-jwt
  • Either of the following:
    • Configure Service A to look for hadoop-jwt cookie
    • knoxsso.cookie.name in KnoxSSO topology
• Service A can’t verify the hadoop-jwt cookie
  • check the public key configured in Service A matches the Knox SSO public key expected
  • Check the logs for Service A about not able to verify cookie

In most cases, if any of the above are a problem, you will end up in an “endless” redirect loop since Service A will not think you are authenticated but Knox SSO is working and sending you back to Service A

• Service A may not be able to confirm you are a user
  • Check Service A logs about user not found
• Make sure that user you tried to log in with is configured for access to Service A If this is a problem, you will end up in a redirect loop potentially as well.

https://knox.apache.org/books/knox-1-2-0/user-guide.html#KnoxSSO+Setup+and+Configuration https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-knox-sso/content/knox_sso.html https://knox.apache.org/books/knox-1-2-0/dev-guide.html#KnoxSSO+Integration

Goto -> CM -> Knox -> Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml

name = providerConfigs:kerberos1
value = role=authentication#authentication.name=HadoopAuth#authentication.param.config.prefix=hadoop.auth.config#authentication.param.hadoop.auth.config.signature.secret=knox-signature-secret#authentication.param.hadoop.auth.config.type=kerberos#authentication.param.hadoop.auth.config.simple.anonymous.allowed=false#authentication.param.hadoop.auth.config.token.validity=1800#authentication.param.hadoop.auth.config.cookie.domain=root.hwx.site#authentication.param.hadoop.auth.config.cookie.path=/gateway/kerberos1#authentication.param.hadoop.auth.config.kerberos.principal=HTTP/[email protected]#authentication.param.hadoop.auth.config.kerberos.keytab=/home/knox.keytab#authentication.param.hadoop.auth.config.kerberos.name.rules=DEFAULT#role=identity-assertion#identity-assertion.name=Default#role=authorization#authorization.name=XASecurePDPKnox



name = kerberos1
value = providerConfigRef=kerberos#discoveryType=ClouderaManager#discoveryAddress=https://knox-workshop-1.knox-workshop.root.hwx.site:7183/#cluster=Cluster 1#discoveryUser=admin#discoveryPasswordAlias=admin#WEBHDFS

• Note: you need modify above values like principal,domain,keytab

Restart Knox and test.

Check knox gateway.log for any issues

Goto -> CM -> Knox -> Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml

name = providerConfigs:sso

value = role=federation#federation.name=SSOCookieProvider#federation.param.sso.authentication.provider.url=https://KNOX-HOSTNAME:8443/gateway/knoxsso/api/v1/websso#role=identity-assertion#identity-assertion.name=HadoopGroupProvider#identity-assertion.enabled=true#identity-assertion.param.CENTRAL_GROUP_CONFIG_PREFIX=gateway.group.config#role=authorization#authorization.name=XASecurePDPKnox#authorization.enabled=true

• Note: you need modify above values like #federation.params.sso.authentication.provider.url to your knox hostname

Restart Knox and test.

Check knox gateway.log for any issues

grep -a4 HadoopGroupProvider /var/lib/knox/gateway/conf/topologies/cdp-proxy.xml


tail -f /var/log/knox/gateway/gateway.log /var/log/knox/gateway/gateway-audit.log

Check if user can see groups:

cat /var/log/knox/gateway/gateway-audit.log  | grep user | grep identity-mapping

Goto -> CM -> Knox -> Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml

name = providerConfigs:sso

value = role=federation#federation.name=SSOCookieProvider#federation.enabled=false#role=authentication#authentication.name=ShiroProvider#authentication.param.sessionTimeout=30#authentication.param.main.ldapRealm=org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm#authentication.param.main.ldapContextFactory=org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory#authentication.param.main.ldapRealm.contextFactory=$ldapContextFactory#authentication.param.main.ldapRealm.contextFactory.url=ldap://localhost:33389#authentication.param.main.ldapRealm.contextFactory.authenticationMechanism=simple#authentication.param.main.ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org#authentication.param.main.ldapRealm.userSearchAttributeName=uid#authentication.param.main.ldapRealm.authorizationEnabled=true#authentication.param.main.ldapRealm.contextFactory.systemUsername=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org#authentication.param.main.ldapRealm.contextFactory.systemPassword=guest-password#authentication.param.main.ldapRealm.contextFactory.systemAuthenticationMechanism=simple#authentication.param.main.ldapRealm.userObjectClass=person#authentication.param.main.ldapRealm.searchBase=dc=hadoop,dc=apache,dc=org#authentication.param.main.ldapRealm.userSearchBase=dc=hadoop,dc=apache,dc=org#authentication.param.main.ldapRealm.groupSearchBase=dc=hadoop,dc=apache,dc=org#authentication.param.main.ldapRealm.groupObjectClass=groupofnames#authentication.param.main.ldapRealm.memberAttribute=member#authentication.param.main.ldapRealm.memberAttributeValueTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org#authentication.param.main.ldapRealm.groupIdAttribute=cn#authentication.param.urls./**=authcBasic##role=identity-assertion#identity-assertion.name=SwitchCase#identity-assertion.enabled=true#identity-assertion.param.principal.case=lower#identity-assertion.param.group.principal.case=lower

Above should look like:

role=federation
federation.name=SSOCookieProvider
federation.enabled=false
role=authentication
authentication.name=ShiroProvider
authentication.param.sessionTimeout=30
authentication.param.redirectToUrl=/${GATEWAY_PATH}/knoxsso/knoxauth/login.html
authentication.param.restrictedCookies=rememberme,WWW-Authenticate
authentication.param.urls./**=authcBasic
authentication.param.main.ldapRealm=org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm
authentication.param.main.ldapContextFactory=org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory
authentication.param.main.ldapRealm.contextFactory=$ldapContextFactory
authentication.param.main.ldapRealm.contextFactory.url=ldap://localhost:33389
authentication.param.main.ldapRealm.contextFactory.authenticationMechanism=simple
authentication.param.main.ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
authentication.param.main.ldapRealm.userSearchAttributeName=uid
authentication.param.main.ldapRealm.authorizationEnabled=true
authentication.param.main.ldapRealm.contextFactory.systemUsername=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org
authentication.param.main.ldapRealm.contextFactory.systemPassword=guest-password
authentication.param.main.ldapRealm.contextFactory.systemAuthenticationMechanism=simple
authentication.param.main.ldapRealm.userObjectClass=person
authentication.param.main.ldapRealm.searchBase=dc=hadoop,dc=apache,dc=org
authentication.param.main.ldapRealm.userSearchBase=dc=hadoop,dc=apache,dc=org
authentication.param.main.ldapRealm.groupSearchBase=dc=hadoop,dc=apache,dc=org
authentication.param.main.ldapRealm.groupObjectClass=groupofnames
authentication.param.main.ldapRealm.memberAttribute=member
authentication.param.main.ldapRealm.memberAttributeValueTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
authentication.param.main.ldapRealm.groupIdAttribute=cn
role=identity-assertion
identity-assertion.name=SwitchCase
identity-assertion.enabled=true
identity-assertion.param.principal.case=lower
identity-assertion.param.group.principal.case=lower

• Note: you need modify above values like ldap details

Restart Knox and test.

Check knox gateway.log for any issues

CDP deployment for Knox assumes secure clusters. Kerberos is required for the necessary trusted proxy doAs capability. There are a handful of services that would work with pseudo/simple auth from Knox but that is not certified in CDP by QE and I couldn't even tell you the definitive list of services that would work.

As trusted proxy doesnt work without kerberos, customer can leverage knox SSO. Followed below steps and was able to validate atlas API using cdp-proxy topology

• Configure atlas for SSO authentication:

CM UI > Atlas > Configurations (set below properties)

atlas.sso.knox.enabled (select check box)
atlas.sso.knox.providerurl : https://<knoxHost>:8443/gateway/knoxsso/api/v1/websso
atlas.sso.knox.publicKey : signing public key ( by default knox ssl public cert)

$JAVA_HOME/bin/keytool -printcert -sslserver <knox>:<port> -rfc

Copy PEM content without BEGIN and END lines to the text field.

• As knoxsso is configured to use cookies only with ssl enabled services, set the property in knox configs 'knoxsso.cookie.secure.only=false' if Atlas is non-ssl.

CM UI > Knox > Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml

Add below xml content to add 'knoxsso.cookie.secure.only'

<property>
<name>knoxsso</name>
<value>providerConfigRef=knoxsso#KNOXSSO:knoxsso.token.ttl=86400000#KNOXSSO:knoxsso.cookie.secure.only=false#app:knoxauth</value>
</property>

• Restart Atlas and Knox services
• Use below API to test atlas API with knox SSO authentication .

 curl  -s -c cookie.txt -b cookie.txt  -k -u USER:PASSWORD -L 'https://<knoxHost>:8443/gateway/cdp-proxy/atlas/api/atlas/admin/status'
{"Status":"ACTIVE"}