Use case --> FTL+Ignite+SharedNothing for OpenShift.
$ helm install fdsn600app ./helm --set cpType=openshift
Error: YAML parse error on BE Helm Chart/templates/persistent-volume-and-claim.yaml: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go value of type releaseutil.SimpleHead

With dry-run, found detailed error trace as described below -->
$ helm install fdsn600app . --debug --dry-run
install.go:159: [debug] Original chart version: ""
install.go:176: [debug] CHART PATH: /home/apps/be-tools/cloud/kubernetes/helm

Error: YAML parse error on BE Helm Chart/templates/persistent-volume-and-claim.yaml: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go value of type releaseutil.SimpleHead
helm.go:84: [debug] error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go value of type releaseutil.SimpleHead
YAML parse error on BE Helm Chart/templates/persistent-volume-and-claim.yaml
helm.sh/helm/v3/pkg/releaseutil.(*manifestFile).sort
/home/circleci/helm.sh/helm/pkg/releaseutil/manifest_sorter.go:146
helm.sh/helm/v3/pkg/releaseutil.SortManifests
/home/circleci/helm.sh/helm/pkg/releaseutil/manifest_sorter.go:106
helm.sh/helm/v3/pkg/action.(*Configuration).renderResources
/home/circleci/helm.sh/helm/pkg/action/action.go:159
helm.sh/helm/v3/pkg/action.(*Install).Run
/home/circleci/helm.sh/helm/pkg/action/install.go:238
main.runInstall
/home/circleci/helm.sh/helm/cmd/helm/install.go:229
main.newInstallCmd.func1
/home/circleci/helm.sh/helm/cmd/helm/install.go:117
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:842
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:950
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:83
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357

For the application deployment on kubernetes cluster with vendor PKS-GCP , it show error as "ImagePullBackOff" when private registry like google container registry is used for docker images.
Followed the steps from below page to configure service account but it does not work ,
https://github.com/TIBCOSoftware/be-tools/wiki/Setting%20up%20Google%20Container%20Registry

If google container registry is marked as public repo then it works fine.

Error:

Events:
Type Reason Age From Message

Normal Scheduled 44s default-scheduler Successfully assigned default/fdcache-test-becacheagent-0 to vm-8c8595a0-e04c-4280-7ee3-d583ebb8ae3c
Normal BackOff 19s (x2 over 42s) kubelet, vm-8c8595a0-e04c-4280-7ee3-d583ebb8ae3c Back-off pulling image "us.gcr.io/streambase-200318/fdcache:6.0"
Warning Failed 19s (x2 over 42s) kubelet, vm-8c8595a0-e04c-4280-7ee3-d583ebb8ae3c Error: ImagePullBackOff
Normal Pulling 5s (x3 over 43s) kubelet, vm-8c8595a0-e04c-4280-7ee3-d583ebb8ae3c Pulling image "us.gcr.io/streambase-200318/fdcache:6.0"
Warning Failed 5s (x3 over 43s) kubelet, vm-8c8595a0-e04c-4280-7ee3-d583ebb8ae3c Failed to pull image "us.gcr.io/streambase-200318/fdcache:6.0": rpc error: code = Unknown desc = Error response from daemon: pull access denied for us.gcr.io/streambase-200318/fdcache, repository does not exist or may require 'docker login': denied: Permission denied for "6.0" from request "/v2/streambase-200318/fdcache/manifests/6.0".
Warning Failed 5s (x3 over 43s) kubelet, vm-8c8595a0-e04c-4280-7ee3-d583ebb8ae3c Error: ErrImagePull

Few issues while building docker image from software installers failing on Mac(5.6.1):

• The default bash version on Mac is 3.2 and the scripts don't work until bash is upgraded to 4.x or higher.
• This issue may not be Mac specific. The docker build fails if ActiveSpaces install is not available. But its mentioned as optional in the documentation.
• This issue may not be Mac specific. As <BE_HOME>/cloud/docker is not shipped with installer, the build image fails on below line in Dockerfile:
  COPY --from=intermediate /opt/tibco/be/${BE_SHORT_VERSION}/cloud/docker /opt/tibco/be/${BE_SHORT_VERSION}/cloud/docker

When we pass external dockerfile (not from installation) to any script build_app_image.sh ,build_rms_image.sh, build_teagent_image.sh it always picks up dockerfile from installation ignoring what is passed to command using -d option.

This issue affects when we tried to test consul gvprovider using scripts available here as build_app_image script always picks dockerfile from installation which does not have any code related to consul.

Following error is occurred when non default dockerfile is passed:

~/be-tools/cloud/docker/frominstall$ ./build_app_image.sh -a /home/jgirase/work/BE/5.6.1/Consul/560/TestGV/ -r testgv:560 -l /home/jgirase/Installation/BE_560_V114GA/be/5.6/ -d /home/jgirase/be-tools/cloud/docker/Dockerfile_fromtar --gv-providers consul

INFO: VERSION : 5.6.0
INFO: APPLICATION DATA DIRECTORY : /home/jgirase/work/BE/5.6.1/Consul/560/TestGV/
INFO: DOCKERFILE : /home/jgirase/be-tools/cloud/docker/Dockerfile_fromtar
INFO: IMAGE REPO : testgv:560
INFO: CDD FILE NAME : default.cdd
INFO: EAR FILE NAME : TestGV_consul.ear

INFO: BE_HOME /home/jgirase/Installation/BE_560_V114GA/be/5.6/
BASEDIR : /home/jgirase/Installation/BE_560_V114GA
BE_DIR : be/5.6
BE_VERSION : 5.6
AS_DIR : as/2.4
AS_VERSION : 2.4
tar -C /home/jgirase/Installation/BE_560_V114GA -cf /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/be.tar tibcojre64 be/5.6/lib be/5.6/bin be/5.6/cloud/docker/lib be/5.6/teagent be/5.6/mm be/5.6/rms be/5.6/studio be/5.6/eclipse-platform be/5.6/examples/standard/WebStudio as/2.4/lib as/2.4/bin
tar -C /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -xf /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/be.tar
Replacing base directory in the files from /home/jgirase/Installation/BE_560_V114GA to /opt/tibco
find /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -name '.tra' -print0 | xargs -0 sed -i.bak 's//home/jgirase/Installation/BE_560_V114GA//opt/tibco/g'
find /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -name '.cdd' -print0 | xargs -0 sed -i.bak 's//home/jgirase/Installation/BE_560_V114GA//opt/tibco/g'
find /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -name '.tra' -print0 | xargs -0 sed -i.bak 's/tibco.env.CUSTOM_EXT_PREPEND_CP=./tibco.env.CUSTOM_EXT_PREPEND_CP=/opt/tibco/be/ext/g'
find /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -name 'be-teagent.props' -print0 | xargs -0 sed -i.bak 's//home/jgirase/Installation/BE_560_V114GA//opt/tibco/g'
find /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -name 'log4j.properties' -print0 | xargs -0 sed -i.bak 's//home/jgirase/Installation/BE_560_V114GA//opt/tibco/g'
rm /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087/be/5.6/bin/_annotations.idx
find /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -name '*.tra' -print0 | xargs -0 sed -i.bak '$a
'java.property.be.engine.jmx.connector.port=%jmx_port%''
tar -C /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/YHNwhIth1571379087 -cf /home/jgirase/be-tools/cloud/docker/frominstall/tmp_8238/be.tar as be tibcojre64
Done..
INFO: Building docker image for TIBCO BusinessEvents Version:5.6.0 and Image Repo:testgv:560 and Dockerfile:/home/jgirase/be-tools/cloud/docker/Dockerfile_fromtar
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/jgirase/Installation/BE_560_V114GA/be/5.6/cloud/docker/frominstall/home: no such file or directory
Docker build failed.

BE currently supports application monitoring.

1. InfluxDB with Dashboards done in Grafana or any other InfluxDB support Dashboarding tool.
2. Tibco Streaming with Dashboards done in Tibco LiveViewWeb.

The scope of this issue,

1. Configurative support for application metrics. This depends on the cdd configuration.
2. Client connection support i.e. cdd configs for InfluxDB and Tibco Streaming to be exposed via values.yml
3. InfluxDB+Grafana and TIBCO Streaming+LiveViewWeb in a production setup will most likely will always be managed/deployed externally/separately. However its good to have an oob support for these, specifically the ones which have public charts already available like InfluxDB, Grafana.

The docker image build script for RMS, build_rms_image.sh is failing with below error,

Step 25/58 : RUN perl -I. -Mbe_docker_install -e "be_docker_install::replaceRunbeVersionToken('$BE_PRODUCT_VERSION','./runbe.sh')"
---> Running in e8191f7813c4
Cannot open file ./runbe.sh for read at be_docker_install.pm line 419.
Removing intermediate container e8191f7813c4
The command '/bin/sh -c perl -I. -Mbe_docker_install -e "be_docker_install::replaceRunbeVersionToken('$BE_PRODUCT_VERSION','./runbe.sh')"' returned a non-zero code: 2
Docker build failed.

Its required to add a new line before updating discover url in be-engine.tra.

RUN echo >> be-engine.tra \
 && echo "java.property.be.engine.cluster.as.discover.url=%AS_DISCOVER_URL%" >> be-engine.tra  \
 && echo "java.property.be.engine.cluster.as.listen.url=%AS_LISTEN_URL%" >> be-engine.tra \
 && echo "java.property.be.engine.cluster.as.remote.listen.url=%AS_REMOTE_LISTEN_URL%" >> be-engine.tra \
 && echo "java.property.com.sun.management.jmxremote.rmi.port=%jmx_port%" >> be-engine.tra

Might need to update other platform Dockerfiles too (haven't tested)

The docker build scripts of app and RMS are working fine but the script for TEA is failing for the same set of installers.
I have used AS 2.4.1 and BE 5.6.1 installers [AS + BE Enterprise + Process + Views]. All the installers are kept in same folder.
Attached log for the failed build script for TEA.

buildTEAgent.txt

Below errors are observed when trying to use existing yaml scripts from kubernetes folder to run deployments on AWS,
error: unable to recognize "manifest.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

After updating apiversion to apps/v1, it shows below error,

error validating "manifest.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

We need to have Readme files for docker and gv support. End users should find github self sufficient to use. Not need us to redirect them to doc.tibco.com for basic "getting started/usage" stuff.

Given all the existing combinations today and more to be added eventually
we will have further set of configurations files to our already existing large set. Making this more difficult and painful to manage.

Helm Charts is a perfect way to handle . It will help create a single template set of configuration files with a resource provider containing reference properties. So for a cloud provider + deployment option, all we would need to do is replace resource provider properties with appropriate provider/deployment specific values.

This will not only cover our existing support for BE+AS2 but will cover our upcoming support in 6.0 for BE with FTL/Ignite/AS4/Cassandra.

Placed the latest TIBCO BE hotfix in the same installer location as TIBCO BE binaries and the hotfix is not getting extract/unzip and install into the final image.

The script build_teagent_image.sh for creating docker image for TEA agent from installers is failing with below error,

Step 25/45 : RUN rm -r /opt/tibco/be/${BE_SHORT_VERSION}/lib/ext/tpcl/aws && rm -r /opt/tibco/be/${BE_SHORT_VERSION}/lib/ext/tpcl/gwt && rm -r /opt/tibco/be/${BE_SHORT_VERSION}/lib/ext/tpcl/tomsawyer
---> Running in 202f2fef9c49
rm: cannot remove '/opt/tibco/be/5.6/lib/ext/tpcl/gwt': No such file or directory
Removing intermediate container 202f2fef9c49
The command '/bin/sh -c rm -r /opt/tibco/be/${BE_SHORT_VERSION}/lib/ext/tpcl/aws && rm -r /opt/tibco/be/${BE_SHORT_VERSION}/lib/ext/tpcl/gwt && rm -r /opt/tibco/be/${BE_SHORT_VERSION}/lib/ext/tpcl/tomsawyer' returned a non-zero code: 1
Docker build failed.

S2I toolkit is also available for windows and we also support creating images for Linux container from windows. So a similar script should be available to use on windows to be in sync with our existing docker folders.

The initial BE parameters such as unclustered, inmemory, store, cache, sharedNothing, etc are not consistent with the CDD. They should be equal in upper/lower case, i.e, the string should be same as seen in CDD.

With the configurations set in values.yaml for cluster provider, cache provider, store, etc. we do helm deployment and get the last statement of output as :
Your release is named fdsa600app. and deployed in openshift with FTL mode.

FTL is a cluster provider and when we talk about mode in context with BE its usually a persistence or OM options.
So if my setup is OpenShift as cloud platform, FTL as cluster provider and Store as OM then the statement should come as :
Your release is named fdsa600app. and deployed in OpenShift with FTL as cluster provider and mode as 'Store'.

With the current scripts, docker unit tests are invoked automatically after building the BE app image successfully. Providing an option to disable docker unit tests would help the user to skip tests if he/she wants.

The information that is listed out about supplied arguments are not consistent across platforms (win, linux). There are few parameters in linux which are missing and some information is not consistent with the way it is displayed on windows.
Look at the output of build app utility for win and linux as shown below :

C:\Share\BE_560_V80\be\5.6\cloud\docker\bin>build_app_image.bat -l E:\v80 -a E:\Work_560\temp-new\SN -r fdsn:v80
INFO: Supplied Arguments

INFO: Installers Location - E:\v80
INFO: Ear/Application Location - E:\Work_560\temp-new\SN
INFO: Image Repo - fdsn:v80

INFO: Dockerfile - Dockerfile.win
INFO: Installers Platform - win
INFO: BusinessEvents Hf - na
INFO: Addons - views,process,
INFO: ActiveSpaces version - 2.4.0
INFO: ActiveSpaces Hf - 002
INFO: CDD file name - fdstore.cdd
INFO: EAR file name - fdstore.ear

[apps@beqacentosz820 bin]$ ./build_app_image.sh -l /home/apps/Installers/BE/560/v82 -a /home/apps/Testing/temp560N/None -r fdc:v82
INFO:Supplied Arguments :

INFO:VERSION : 5.6.0
INFO:EDITION : enterprise
INFO:INSTALLER DIRECTORY : /home/apps/Installers/BE/560/v82
INFO:APPLOCATION Data DIRECTORY : /home/apps/Testing/temp560N/None
INFO:ADDONS : process,views
INFO:DOCKERFILE : Dockerfile
INFO:HF : na
INFO:AS-HF : 002
INFO:IMAGE VERSION :

It is observed that apiVersion: apps/v1beta1 is invalid for minikube inmemory, cache(persistence none/sharedall/sharedNothing) deployments
Selector is missing in spec section of deployments.

Issues observed in :
minikube version: 1.7.3
kubectl version : 1.17.3

Steps:
1.Build the application image using installers (docker/bin) folder with some GV
2.Run Consul server
3.Create GV/key-value pairs in consul server same as define in BE project
4.Run application image using below command,
docker run -d -e CONSUL_SERVER_URL=http://consul:8500 -e APP_CONFIG_PROFILE=default -e BE_APP_NAME=TestGV --net=mybridge --name=testgv testgv:560installer
5.Application does not read GV values from consul server.It also shows below error message in logs,
INFO: Reading GV values from Consul..
INFO: CONSUL_SERVER_URL = http://consul:8500
INFO: BE_APP_NAME = TestGV
INFO: APP_CONFIG_PROFILE = default
INFO: Reading GV values from Consul.. (TestGV/default/)
./gvproviders/consul/run.sh: line 28: /home/tibco/be/gvproviders/consul/consul: No such file or directory
./gvproviders/consul/run.sh: line 28: /home/tibco/be/gvproviders/consul/jq: No such file or directory

Basically it failed to download consul cli inside application image.

The script is working fine for BE 5.6.0 + AS 2.4.0 but its failing when either BE is 5.6.1 or AS 2.4.1 installers are used. Include these two versions for current release.

[apps@beqacentosz820 s2i]$ ./create_builder_image.sh -l /home/apps/Installers/BE/561/v24
ERROR: TIBCO BusinessEvents (5.6.0) is not present in the target directory. There should be only one.

[apps@beqacentosz820 s2i]$ ./create_builder_image.sh -l /home/apps/Installers/BE/560/GA
INFO:Supplied Arguments :

INFO:VERSION : 5.6.0
INFO:EDITION : enterprise
INFO:INSTALLER DIRECTORY : /home/apps/Installers/BE/560/GA
INFO:APPLICATION DATA DIRECTORY : na
INFO:ADDONS : process,views
INFO:DOCKERFILE : Dockerfile
INFO:BE-HF : na
INFO:AS-HF : na
INFO:IMAGE VERSION : na

argver: 5.6.0, asver: 2.2.0, asver:2.4.1

ERROR :BE Version :5.6.0 is not compatible with Activespace version 2.4.1.

Also, a proper error message should be thrown for incompatible versions.

The --gv-providers is not required in builder script and description of --repo is wrong, it should be related to base image.

[apps@beqacentosz820 s2i]$ ./create_builder_image.sh -h

Usage: create_builder_image.sh

[-l|--installers-location] : Location where TIBCO BusinessEvents and TIBCO Activespaces installers are located [required]

[-d|--docker-file] : Dockerfile to be used for generating image.(default Dockerfile) [optional]

[--gv-providers] : Names of GV providers to be included in the image. Supported value(s) - consul [optional]

[-r|--repo] : The app image Repository (example - fdc:latest) [required]

[-h|--help] : Print the usage of script [optional]

NOTE : supply long options with '='

The build script under frominstall folder for application image fails to create the image if BE_HOME does not have AS installed in it.

[apps@beqacentosz820 frominstall]$ ./build_app_image.sh -a /home/apps/Testing/temp561/fd -r fd:v70was

INFO: VERSION : 5.6.1
INFO: APPLICATION DATA DIRECTORY : /home/apps/Testing/temp561/fd
INFO: DOCKERFILE : Dockerfile_fromtar
INFO: IMAGE REPO : fd:v70was
INFO: CDD FILE NAME : fd.cdd
INFO: EAR FILE NAME : fd.ear

WARN: BE_HOME not provided, using current BE_HOME.
WARN: AS installation not found.
BASEDIR : /home/apps/Installations/BE/BE_561_v70WAS
BE_DIR : be/5.6
BE_VERSION : 5.6
tar -C /home/apps/Installations/BE/BE_561_v70WAS -cf /home/apps/Installations/BE/BE_561_v70WAS/be/5.6/cloud/docker/frominstall/tmp_22026/be.tar tibcojre64 be/5.6/lib be/5.6/bin be/5.6/teagent be/5.6/mm
tar -C /home/apps/Installations/BE/BE_561_v70WAS/be/5.6/cloud/docker/frominstall/tmp_22026/KieZcaeT1574058515 -xf /home/apps/Installations/BE/BE_561_v70WAS/be/5.6/cloud/docker/frominstall/tmp_22026/be.tar
Replacing base directory in the files from /home/apps/Installations/BE/BE_561_v70WAS to /opt/tibco
find /home/apps/Installations/BE/BE_561_v70WAS/be/5.6/cloud/docker/frominstall/tmp_22026/KieZcaeT1574058515 -name '.tra' -print0 | xargs -0 sed -i.bak 's//home/apps/Installations/BE/BE_561_v70WAS//opt/tibco/g'
find /home/apps/Installations/BE/BE_561_v70WAS/be/5.6/cloud/docker/frominstall/tmp_22026/KieZcaeT1574058515 -name '.cdd' -print0 | xargs -0 sed -i.bak 's//home/apps/Installations/BE/BE_561_v70WAS//opt/tibco/g'
sed: no input files
123
Creating BE archive failed

Issue:
Consul binaries not loaded when build the docker image with latest docker build scripts.
(https://github.com/TIBCOSoftware/be-tools/tree/v1.0).

That is related to a missing command when load the wget package in ubuntu image.
Without wget installed, it's not possible to load the binaries so it failed to load GVs from consul (script error reported on startup of the docker container when execute gvproviders/consul/run.sh (line 33 binaries missign).

To fix the issue update docker script /gvproviders/consul/setup.sh and add "apt-get update" gefore load the wget package.

Content:

#!/bin/bash

Copyright (c) 2019. TIBCO Software Inc.
This file is subject to the license terms contained in the license file that is distributed with this file.

echo "Setting up consul gv provider.."

cd /home/tibco/be/gvproviders/consul

Support: required to install wget package correctly
apt-get update
apt-get install -y wget
Download jq.
wget "https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64"
mv jq-linux64 jq
chmod +x jq
Download consul cli and extract it.
wget "https://releases.hashicorp.com/consul/1.6.1/consul_1.6.1_linux_amd64.zip"
unzip consul_1.6.1_linux_amd64.zip
rm consul_1.6.1_linux_amd64.zip

Note:
As a workaround for now, I suggested customer to download the files, unzip jq-linux64, rename the binary to "jq" and copy both files to gvproviders\consul before build the image.

Steps:
1.Define GV of all types in BE project and build application image using frominstall docker scripts.
2.Define key/value pairs for above GV in consul server, start application container.
3.It will read values from consul sever and same is debug out in logs
4.Now change the value in consul server like assign any random string to boolean type GV or assign string to integer type GV.
5.Restart application container
6.Log shows previous values for invalid value , it should show default value

GV values from Consul

tibco.clientVar.Group_1/TestBoolean=false
tibco.clientVar.Group_1/TestInteger=bob
tibco.clientVar.Group_1/TestPassword=tibco123
tibco.clientVar.Group_1/TestString=consulingroup
tibco.clientVar.TestBoolean=true
tibco.clientVar.TestInteger=8500
tibco.clientVar.TestPassword=123123567
tibco.clientVar.TestString=consultesting1234

2019 Nov 05 10:15:38:517 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestString GV value : consulingroup
2019 Nov 05 10:15:38:517 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestInteger GV value : 0
2019 Nov 05 10:15:38:517 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestBoolean GV value : false
2019 Nov 05 10:15:38:517 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestPassword GV value : tibco123

With 6.0 we are adding a new deployment configurations with FTL/Ignite/AS4/Cassandra.

Today our docker support is limited to only BE-AS2x. This need to be extended to include these new deployment configurations.

The script for creating application image placed under folder frominstall is failing with below error when default BE_HOME is used,

Command - [./build_app_image.sh -a /home/apps/Testing/temp561/None -r fdc:v67fiA]

Step 27/31 : RUN /home/tibco/be/gvproviders/setup.sh ${GVPROVIDERS}
---> Running in 819a24f42965
/bin/sh: 1: /home/tibco/be/gvproviders/setup.sh: Permission denied
The command '/bin/sh -c /home/tibco/be/gvproviders/setup.sh ${GVPROVIDERS}' returned a non-zero code: 126
Docker build failed.

The installers used in building application image are --> AS 240 + AS HF2 + BE Ent. + Process +Views. But while creating application image, it is not picking AS Hotfix installer. I have verified the same by running cache agent which shows AS version as base AS 2.4.0 version instead of AS HF2 version. Due to this issue, could not verify the case where cache fails to start with image built using AS 2.4.0 with HF2 hotfix.

E:\Work_Cloud\Git\be-tools\cloud\docker\bin>build_app_image.bat -l E:\My\TIBCO_Installers\Windows\560GA -a E:\Work_Cloud\temp560\temp-new\None -r fdc:560
INFO: Supplied Arguments
INFO: Installers Location - E:\My\TIBCO_Installers\Windows\560GA
INFO: Ear/Application Location - E:\Work_Cloud\temp560\temp-new\None
INFO: Image Repo - fdc:560
INFO: Dockerfile - Dockerfile.win
INFO: Installers Platform - win
INFO: BusinessEvents Hf - na
INFO: Addons - views,process,
INFO: ActiveSpaces version - 2.4.0
INFO: ActiveSpaces Hf - na
INFO: CDD file name - fdcache.cdd
INFO: EAR file name - fdcache.ear

The builder image created from the script create_builder_image.sh always set the image name as s2ibuilder:01 and new image overwrites the previous image instead of creating new one. So it is very difficult for end user to keep a track of images built with different BE and AS versions, especially if using respective Hotfix installers too.

Try to build image using frominstall using below command,
jgirase@jgirase-ubuntu-P51:~/be-tools/cloud/docker/frominstall$ ./build_app_image.sh -l /home/jgirase/Installation/BE_560_V114GA/be/5.6 -a /home/jgirase/work/BE/5.6.1/Consul/560/TestGV -r testgv:560install --gv-providers=consul

Error:
Step 17/29 : COPY gvproviders /home/tibco/be/gvproviders
---> ee4e99aa5e7f
Step 18/29 : WORKDIR /home/tibco/be
---> Running in 04c5d7c0bfc3
Removing intermediate container 04c5d7c0bfc3
---> 72abc7c24601
Step 19/29 : RUN sed "s/%%%BE_VERSION%%%/${BE_SHORT_VERSION}/" /opt/tibco/be/${BE_SHORT_VERSION}/cloud/docker/lib/runbe.sh > ./runbe.sh && chmod +x *.sh
---> Running in 28a3e2de47a4
sed: can't read /opt/tibco/be/5.6/cloud/docker/lib/runbe.sh: No such file or directory
The command '/bin/sh -c sed "s/%%%BE_VERSION%%%/${BE_SHORT_VERSION}/" /opt/tibco/be/${BE_SHORT_VERSION}/cloud/docker/lib/runbe.sh > ./runbe.sh && chmod +x *.sh' returned a non-zero code: 2
Docker build failed.

The script for creating base image,create_builder_image.sh is failing. It is looking for a file in wrong directory,
cp: cannot stat ‘/home/apps/Installers/BE/560/GA/be_docker_install.pm’: No such file or directory

It might be related to environment settings. It is not working on QA machines. I have tried on different OS. Below is the output of the build command,
[apps@beqacentosz820 s2i]$ ./create_builder_image.sh -l /home/apps/Installers/BE/560/GA -v 5.6.0 -i v01
INFO:Supplied Arguments :
INFO:VERSION : 5.6.0
INFO:EDITION : enterprise
INFO:TARGET DIRECTORY : /home/apps/Installers/BE/560/GA
INFO:ADDONS : na
INFO:DOCKERFILE : bin/Dockerfile
INFO:HF : na
INFO:AS-HF : na
INFO:IMAGE VERSION : v01
cp: cannot stat ‘/home/apps/Installers/BE/560/GA/be_docker_install.pm’: No such file or directory
Can't locate be_docker_install.pm in @inc (@inc contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).
BEGIN failed--compilation aborted.
rm: cannot remove ‘be_docker_install.pm’: No such file or directory
INFO:Copying Packages..
INFO:CURRENT_DIR : /home/apps/Testing/Git/be-tools/cloud/docker
./create_builder_image.sh: line 188: /home/apps/Installers/BE/560/GA/package_files.txt: No such file or directory
Can't open /home/apps/Installers/BE/560/GA/package_files.txt: No such file or directory.
Can't open /home/apps/Installers/BE/560/GA/package_files.txt: No such file or directory.
INFO:Building docker image for TIBCO BusinessEvents Version:5.6.0 and Image Version:v01 and Docker file:bin/Dockerfile
ERROR:Improper As version.Aborting.
Deleting temporary intermediate image..
"docker rmi" requires at least 1 argument.
See 'docker rmi --help'.

Usage: docker rmi [OPTIONS] IMAGE [IMAGE...]

Remove one or more images
Deleteting folder

Define global variable of type password in BE application and build image for the same using frominstall docker scripts.
Define key value pair for above GV in consul server and pass this server url to run container for above image.
When the value of password type GV is debug out in logs using below function , it shows password in plain text while the value set in GV editor is printed in encrypted format.
System.debugOut("######## TestPassword GV value : " + System.getGlobalVariableAsString("TestPassword","pass"))

Output:
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Inside startup rule function
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## TestString GV value : consultesting1234
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## TestInteger GV value : 8500
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## TestBoolean GV value : true
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## TestPassword GV value : 123123567
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestString GV value : TestGroup
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestInteger GV value : 200
2019 Nov 05 09:56:05:445 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestBoolean GV value : false
2019 Nov 05 09:56:05:446 GMT 0 be-engine Info [main] - [user] [inference-class] ######## Group_1/TestPassword GV value : #!haiAbrNG4tnINHX3zL8M8jgqDea9sN9Ze9PtmB+ZhbA=

The values.yaml file under helm contains separate global variables for Cassandra host and port while the Cassandra connection section in CDD does not have separate fields for host and port. There is a single field for server contact which is made up of hostname and port as hostname:port (default value --> localhost:9042).
So update the yaml file accordingly.

1. Reduce size of frominstallation docker image
2. Include folders specific to component liek TEA,RMS
3. Create common script for validation of installer

discovery and inference pods are giving waiting for space error in k8s cache mode deployments.

Observed discovery=[tibpgm] in individual pod logs.
Error is in generating docker image.

Cc: @vpatil-tibco @vivekshende

Issue:
In docker environment (Container created and used to start and stop) on each container restart all GVs and JVM props added again to existing be-engine.tra file.
Caused by this be-engine.tra file grows to 1GB on customer site.

Requred to update runbe.sh script - start BE agent with a backup file copied from orign be-engine.tra file first or overwrite existing settings for each property.

Reproduction:

1.) Create BE application image
2.) Create a docker container from BE image

e.g.:
docker container create "fd:latest"

3.) Check container ID and start/stop container several times

eg:
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7a8e92fea554 fd:latest "./runbe.sh" 2 hours ago Up About an hour 5555/tcp, 50000-50001/tcp quirky_khayyam

4.)
Start and top container several times:

e.g.:
docker container start 7a8e92fea554
docker container stop 7a8e92fea554
docker container start 7a8e92fea554
docker container stop 7a8e92fea554
docker container start 7a8e92fea554

5.)
Connect to container with terminal (/bin/sh or /bin/bash) and check be-engine.tra file
-> For each restart JVM props added (in sample no GVs configured)

e.g.: Commands and output:

docker exec -it 7a8e92fea554 /bin/sh

cd /opt/tibco/be/5.6/bin
tail be-engine.tra

...
Jars that should be skipped by Tomcat server's TLDJarScanner
tomcat.util.scan.DefaultJarScanner.jarsToSkip=xdata.jar,tibrvj.jar
java.property.java.rmi.server.hostname 172.17.0.2
java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetaspaceSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
java.property.java.rmi.server.hostname 172.17.0.2
java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetaspa java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetaspaceSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
java.property.java.rmi.server.hostname 172.17.0.2
java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetaspa java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetaspaceSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
java.property.java.rmi.server.hostname 172.17.0.2
java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetaspa java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetasp java.extended.properties=-server -Xms1024m -Xmx1024m -Xss2m -javaagent:/opt/tibco/be/5.6/lib/cep-base.jar -XX:MaxMetaspaceSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC

TIBCO BusinessEvents (BE) should support custom config management in addition to Consul something like BWCE (https://github.com/TIBCOSoftware/bw-samples/wiki/Custom-Config-Management)

It is observed that discoverynode is added for Persistence-None & Shared-All but not for Shared-Nothing. Is there any significance for it?

CC: @vpatil-tibco @CloudDevBe @vivekshende

The section has not been updated with the new changes.
The current text describes the command usage as :
./create_builder_image.sh -l -v -i
where
[-l|--installers-location] : Location where TIBCO BusinessEvents and other required installers are located [required]
[-d|--docker-file] : Dockerfile to be used for generating image.(default Dockerfile) [optional]"
For example:
./create_builder_image.sh -l /Users/test/BE_Installers

But currently it is :
[apps@beqacentos7 s2i]$ ./create_builder_image.sh -h

Usage: create_builder_image.sh

[-l|--installers-location] : Location where TIBCO BusinessEvents and other required installers are located [required]

[-d|--docker-file] : Dockerfile to be used for generating image.(default Dockerfile) [optional]

[--disable-tests] : Disables docker unit tests on created image. [optional]

[--gv-providers] : Names of GV providers to be included in the image. Supported value(s) - consul [optional]

[-r|--repo] : The builder image Repository (example - s2ibuilder:latest) [required]

[-h|--help] : Print the usage of script [optional]

NOTE : supply long options with '='

Hotfix not installed caused by a script issue when check for BE hotfix file. The filename (hotfix) does not include the ARG_EDITION (enterprise).

Filename: TIB_businessevents-hf_5.6.1_HF-004_linux26gl25_x86_64.zip

Script: s2i/create_builder_image.sh
Variable beHfCntd set to 0 (filename used for the check not match the real name)

BE_HF_REGEX="${BE_PRODUCT}-${ARG_EDITION}__HF"
...
beHfPckgs=$(find $ARG_INSTALLER_LOCATION -name "$BE_HF_REGEX$INSTALLER_PLATFORM")
beHfCnt=$(find $ARG_INSTALLER_LOCATION -name  "$BE_HF_REGEX*$INSTALLER_PLATFORM" | wc -l)

Note:
Hotfix still not installed correctly when set BE_HF_REGEX="${BE_PRODUCT}-*_HF" (TIBCOUniversal installer binary not found).

Woraround:
Add hotfix (zip file) to the root folder of the BE 5.x base installation. The TIBCO Universal Installer unzip that file and installs the hotfix with the base version.
Steps:

• Copy BE installer and hotfix to the Unix system (e.g. /home/tibco/work/installerFiles)

• Create tmp folder (e.g. /home/tibco/work/installerFiles)

• Unzip base version in that folder
    e.g.
    cd /home/tibco/work/installerFiles/tmp
    unzip ../TIB_businessevents-enterprise_5.6.1_linux26gl25_x86_64.zip

• Move hotfix to tmp folder
    e.g.
    mv ../TIB_businessevents-hf_5.6.1_HF-004_linux26gl25_x86_64.zip .
   

• Create new zip file (same name as base version)
    e.g.
    zip -r TIB_businessevents-enterprise_5.6.1_linux26gl25_x86_64.zip *
   

• Copy the new created zip file to your docker installfiles folder and create the image
  ./build_app_image.sh -l installerFiles -a appFiles -r imagename -d dockerfile

Kindly add executable permissions to all build images related shell scripts inside folder docker/frominstall same as docker/bin folder

The BE docker image built for windows container is of size 12 GB approx. which is large considering that we are building it from frominstall which is used for creating images of less size. The application image will take 1 GB more so there should be a way to create these images of less size.

Also reduce the engine startup time as it is taking longer time as compared with linux containers.
Attaching build image log for the reference.

(Migrated issue https://jira.tibco.com/browse/BE-26668 from Jira to GitHub)

Command run:
./build_app_image.sh -l /home/jgirase/Installers/BE/5.6.0/V114_GA/zips/ -a /home/jgirase/work/BE/5.6.1/Consul/560/TestGV -r testgv:560installer --gv-providers consul

Error:
Step 33/67 : RUN perl -I. -Mbe_docker_install -e "be_docker_install::install_be('$BE_PRODUCT_VERSION','enterprise','$BE_PRODUCT_ADDONS','$BE_PRODUCT_HOTFIX','$AS_PRODUCT_HOTFIX','$AS_VERSION')"
---> Running in 6a1770f4fc58

INFO:Installing ActiveSpaces 2.4.0...

INFO:Installing ActiveSpaces 2.4.0...DONE

INFO:Installing BusinessEvents 5.6.0...
Cannot open file /opt/tibco/be/5.6/bin/be-engine.tra for read at be_docker_install.pm line 484.
Removing intermediate container 6a1770f4fc58
The command '/bin/sh -c perl -I. -Mbe_docker_install -e "be_docker_install::install_be('$BE_PRODUCT_VERSION','enterprise','$BE_PRODUCT_ADDONS','$BE_PRODUCT_HOTFIX','$AS_PRODUCT_HOTFIX','$AS_VERSION')"' returned a non-zero code: 2
Docker build failed.
Deleting temporary intermediate image..

Copyright © 2019. TIBCO Software Inc.
This file is subject to the license terms contained in the license file that is distributed with this file.

Things to cover here,

1. Enhance/add more docker tests.
2. Add unit tests for Helm.
3. Plug unit tests with travis.

The information mentioned there suggests to use service type as LoadBalancer for Azure platform and NodePort for others but I have to use LoadBalancer in case of OpenShift too. So The information for type is quite incomplete.
Update the service type section :
#Required for all applications in all cloud providers
beservice:
name: beservice
type: NodePort #LoadBalancer for azure, NodePort

As part of 5.6, we combined multiple docker scripts into single script. However we missed adding "fromInstall" option to it. Which we included later a separate set.

Ideally we should simply have an argument to the base docker script to accept "fromInstall" option.

Command : ./build_app_image.sh -l /home/jgirase/Installers/BE/5.6.1/V59 -a /home/jgirase/work/BE/5.6.1/Consul/561/JMSChannel/ -r jmschannel:561 --gv-providers consul

Installer directory contains AS 2.4.1 linux zip and BE 5.6.1 windows zip
While building image it shows error message as below,
ERROR: TIBCO BusinessEvents HF is present but TIBCO BusinessEvents Base version is not present in the target directory.

This error message is not saying that linux installer for base version is missing. Even there is no hotfix present in installer directory as it stated.

The docker build script might fails if user already have base OS image tagged as latest, (ex, ubuntu:latest) and it is couple of months old. I have encountered such issue and it was not easy to debug.
So it is better to tag the base OS image with specific version. This fix include updating all the DockerFiles, for ex:
FROM ubuntu:latest as intermediate