Comments (23)
@aegershman this dashboard is part of our (https://www.outcoldsolutions.com) application Monitoring Docker: Metrics and Log Forwarding
from docker-splunk.
Also just run docker run -it -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunk/universalforwarder:latest
and got 999 as uid
splunk@5d5ee3e77e7b:/$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
splunk 1 0.1 0.0 18028 2928 pts/0 Ss+ 15:27 0:00 /bin/bash /sbin/entrypoint.sh start-service
splunk 139 3.3 1.2 392600 79208 ? Sl 15:27 0:00 splunkd -p 8089 start
splunk 140 0.0 0.2 80960 13156 ? Ss 15:27 0:00 [splunkd pid=139] splunkd -p 8089 start [process-runner]
splunk 257 0.0 0.0 4232 652 pts/0 S+ 15:27 0:00 tail -n 0 -f /opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
splunk 258 1.0 0.0 18204 3276 pts/1 Ss 15:28 0:00 bash
splunk 266 0.0 0.0 36636 2796 pts/1 R+ 15:28 0:00 ps aux
@kdthanvi could you verify again with latest
? I assume 7.1.0
behaves differently, because it is an old/legacy image.
from docker-splunk.
I believe the new 7.2.2 container published yesterday includes a fix for this.
from docker-splunk.
Just verified, seems like it is not only a UF problem but also Enterprise image as well.
Seems like the reason for that, because ansible uses sudo, see #36 (comment), and so launches splunk as root.
I exec in the wrong container. All worked as expected - I see splunk as a user for UF and Enterprise image.
from docker-splunk.
@kdthanvi Just verified UF and Enterprise images 7.2.0, the actual UID of the processes are 999, and I see Splunk as a user running processes
from docker-splunk.
(Sorry for off-topic, but @outcoldman could I ask what that dashboard view is in the screenshot you posted? Is that from within Splunk? How did you get that data?)
from docker-splunk.
@outcoldman Thank you for the great help.
I deleted all the images I had regarding splunk, I did clean docker run
. Looks like the image splunk/universalforwarder:latest
works as you said (with docker exec
also I got logged in as splunk user).
from docker-splunk.
When I use the latest image (with my production setup which is using kubernetes), it's NOT working.
Reason being :
Due to some constraints, I have kubernetes security policy set assecurityContext : runAsNonRoot : true
(this is the reason I wanted to run the splunkforwarder as non-root user).
And when I am running the splunk/universalforwarder:latest
image, I am getting the error Error: container has runAsNonRoot and image has non-numeric user (splunk), cannot verify user is non-root
.
I checked the uid in my local container with id -u splunk
& I can see it as 999
.
But somehow to k8s
it is seen as uid == nil
(explained here).
@outcoldman any way I can resolve/debug this ?
Is there any document you've, where I can build the image locally (with this git repo) ?
PS : I tried building it but I am not sure about the arguments here.
Please let me know.
Thanks!
from docker-splunk.
@kdthanvi I believe current image has issues with non-root user, because it uses sudo
access sometimes #36
I am not 100% sure about that
from docker-splunk.
I don't think #36 is relevant here, this seems to be an implementation detail with Kubernetes specifically. I found kubernetes/kubernetes#59819 which seems to point out that in order for the securityContext/runAsNonRoot to work, the Dockerfile itself has to be built with USER <id>
specified instead of USER <name>
.
@kdthanvi can you confirm with Kubernetes version you're running in production? Additionally, you should be able to build your own image using the Makefile in this repository (ex. make splunk-debian-9
). Also based on that github issue you linked, seems like you might be able to circumvent this by changing runAsNonRoot: true
to runAsUser: 999
. But that might not be applicable depending on your specific security requirements.
from docker-splunk.
Hi @nwang92 Thanks for the comment.
I thought the same when I read the error message, so I update the Dockerfile to have USER <id>
instead of USER <name>
.
The image runs well in my local (thanks to you, I could build successfully using make uf-debian-9
).
But when I am running in the kubernetes cluster, the pod isn't starting as it should(it's is CrashLoopBackOff
state) and the slpunk-uf container is giving me below error :
$ kubectl logs <pod-name> -c <container-name>
sudo: unable to change to root gid: Operation not permitted
sudo: unable to initialize policy plugin
Any idea on this ?
I am using kubernetes version v1.9.6
.
Also, about runAsUser : 999
this could help but there are more than 1 containers running in the same pod. So putting it use fixed user id for each container would not be a good way (I guess).
Also, as I mentioned there are security issues so I am not very sure if I could go for that.
Please let me know. Thanks!
from docker-splunk.
Please check this commit. This is how I changed/added to make the uf-debian-9
image run with USER <id>
.
from docker-splunk.
I checked the other parts of the uf & theentrypoint.sh
also has many sudo
tasks. So I tried to change the usermod -aG sudo ${SPLUNK_USER}
to usermod -aG sudo -u ${SPLUNK_USER_ID} ${SPLUNK_USER}
as well.
But then the docker run -it -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunkforwarder-debian-9
fails with too many permission denied
errors.
from docker-splunk.
@kdthanvi gotcha, yeah I'm guessing changing ${SPLUNK_USER}
to ${SPLUNK_USER_ID}
will likely fail downstream. I suppose the areas of concern are mainly in the entrypoint in this particular case? Or other areas where privilege escalation comes into play?
I don't have a k8s cluster readily available for that particular version, but I'm going to try building the image from your commit and seeing if I can get the basic docker run
command working.
from docker-splunk.
Hi @nwang92 just fyi, I did check the entrypoint and it's loaded with sudo
. I did try to change(removed sudo
and gave the splunk:9999
user permission for those directories) all those as well, but then I ended up facing many permission issues & No such file or directory
issues.
Here is the error that I am getting during docker build
.
Btw, with the commit I shared, the image builds successfully and even docker run
works well in my local. But the error happens in k8s cluster, due to the entrypoint's sudo
operations.
PS : Thanks for your support!
from docker-splunk.
Hey kdthanvi, out of curiosity, what are you using the UF for? Sidecar for pods running in k8s? or to actually monitor k8s?
from docker-splunk.
@matthewmodestino I am running splunk-uf as a sidecar container in my k8s pod for getting the application logs (from the app container, running inside the same pod).
from docker-splunk.
@kdthanvi I built the image from your fork (https://hub.docker.com/r/nwang92/uf-k8s-test/) and tried it out but I'm not able to reproduce the issue you're seeing, but it's possible that I don't have the securityContext defined correctly. For reference, here's my pod yaml:
apiVersion: v1
kind: Pod
metadata:
name: uf-seccontext
spec:
securityContext:
runAsNonRoot: true
containers:
- name: uf-seccontext
image: nwang92/uf-k8s-test
imagePullPolicy: Always
env:
- name: SPLUNK_START_ARGS
value: --accept-license
- name: SPLUNK_PASSWORD
value: helloworld!
securityContext:
runAsNonRoot: true
My pod seems to be running ok:
uf-seccontext 1/1 Running 0 8m
And the user/group ID's match what you've defined in that commit:
$ kubectl exec -it uf-seccontext bash
splunk@uf-seccontext:/$ id -u splunk
9999
splunk@uf-seccontext:/$ id -g splunk
9999
I am using k8s 1.8.4, so it's possible this may be a feature in a new version of the engine. But wondering if you have different securityContext definitions that I might have missing?
from docker-splunk.
Hi @nwang92 What we're doing is not in yaml, but in podsecuritypolicy.
In one line how we're managing is : podsecuritypolicy
-> role
-> rolebinding
-> serviceaccount
-> pod
So, maybe specifying this in yaml won't restrict the use of sudo
as done in podsecuritypolicy
(I am not sure about this though).
from docker-splunk.
Do you happen to know what your PodSecurityPolicy is set to? I tried creating a PodSecurityPolicy with runAsUser defined as below:
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: example
spec:
privileged: false # Don't allow privileged pods!
# The rest fills in some required fields.
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: MustRunAsNonRoot
fsGroup:
rule: RunAsAny
volumes:
- '*'
Using this security policy, I followed the examples from the docs and setup a user/role/role-binding:
kubectl create namespace psp-example
kubectl create serviceaccount -n psp-example fake-user
kubectl create rolebinding -n psp-example fake-editor --clusterrole=edit --serviceaccount=psp-example:fake-user
alias kubectl-admin='kubectl -n psp-example'
alias kubectl-user='kubectl --as=system:serviceaccount:psp-example:fake-user -n psp-example'
kubectl-admin create -f example-psp.yaml
kubectl-admin create role psp:unprivileged --verb=use --resource=podsecuritypolicy --resource-name=example
kubectl-admin create rolebinding fake-user:psp:unprivileged --role=psp:unprivileged --serviceaccount=psp-example:fake-user
But still unable to reproduce your exact issue, using the same pod yaml from above (nwang92/uf-k8s-test image).
$ kubectl-user create -f security-context.yml
pod/uf-seccontext created
$ kubectl-user get po
NAME READY STATUS RESTARTS AGE
uf-seccontext 1/1 Running 0 1m
from docker-splunk.
Is this issue good to close out?
from docker-splunk.
Any idea when this update will be released?
from docker-splunk.
Thanks @mikedickey!
from docker-splunk.
Related Issues (20)
- Failed to install the app 'Splunk add on for linux'
- Splunk 9.1.3 has a regression on the `SPLUNKD_SSL_ENABLE` variable HOT 3
- create-default does not work because the six module is not found. HOT 1
- Clients can't connect to splunk ds 9.2 and 9.2.0.1 HOT 1
- Index creation using default.yml is no longer working in splunk images > 9.0.7.
- libcrypto.so error on initial password hashing HOT 10
- Splunk 9.1.3 Startup Errors HOT 1
- Please include eu-stack utility into the docker image to allow performance troubleshooting within an indexer cluster
- Can't start latest/9.0.9 version HOT 2
- Python 3.7 EOL
- Launching a Forwarder licensed heavy container fails in 9.0.9, 9.1.4, and 9.2.1
- Can't download files from github inside of docker container
- Best Practices for Docker files
- UF Crashes on Container Restart (9.2 and 9.1) HOT 1
- 9.2.1 as splunk_deployment_server doesn't start HOT 2
- 9.2.1/9.1.4 hec token not created when providing env var SPLUNK_HEC_TOKEN HOT 2
- Add support for Java 17
- Installing JDK 11 inside Splunk container
- Splunk UF - Warning: Attempting to revert the SPLUNK_HOME ownership causes troubles with Ansible play
- Splunkbase app not installed when using SPLUNK_APPS_URL
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-splunk.