Comments (10)
stoader
commented on June 12, 2024
These log lines:
2020-08-12 20:26:35,686 INFO [org.apache.spark.banzaicloud.metrics.sink.PrometheusSink] - metricsNamespace=None, sparkAppName=Some(nokia-sps-decoder-Spark), sparkAppId=Some(application_1584132323547_997813), executorId=Some(6)
2020-08-12 20:26:35,687 INFO [org.apache.spark.banzaicloud.metrics.sink.PrometheusSink] - role=executor, job=application_1584132323547_997813
indicate that metrics are being pushed by the sink to Pushgateway so this doesn't seem to be a spark-metrics issue.
If you want to debug what payload spark-metrics sends to pushgateway than reconfigure your spark-mertics to send metrics to something like netcat (e.g. provide the address and port of a netcat as *.sink.prometheus.pushgateway-address and *.sink.prometheus.pushgateway-address-protocol=http
Also, worth checking Pushgateway logs as well.
from spark-metrics.
julianblack
commented on June 12, 2024
Thanks for responding. You're right when I use netcat from a Hadoop edge node, using HTTP protocol, I'm seeing all the executor metrics.
Unfortunately though, this doesn't help me troubleshoot why it wouldn't be working to a remote HTTPS pushgateway on the executors only.
There is no mention of the role=executor in the pushgateway logs, so it doesn't seem to be getting there.
Is it possible it could be an SSL certificate or connectivity issue?
Is there a way to print these types of exceptions if that is the case?
Does the plugin work with HTTPS?
Again, I really appreciate your input.
from spark-metrics.
stoader
commented on June 12, 2024
if there were SSL related errors it would be visible in the logs, also the driver metrics would affected as well.
If you capture the executor metrics with netcat and than push those to pushgateway using curl, through https, from one of the executor nodes, do those metrics make through?
from spark-metrics.
julianblack
commented on June 12, 2024
Hi @stoader , I just tested this and yes it works through curl after I paste the netcat output into a file.
Manual curl commands:
cat push_load_1.txt | curl --silent --data-binary @- "https://xxxx/xxx-pushgateway-qa/metrics/job/application_1584132323547_1018676/role/driver/app_name/nokia-sps-decoder-Spark/instance/xxxx"
cat push_load_2.txt | curl --silent --data-binary @- "https://xxxx/xxxx-pushgateway-qa/metrics/job/application_1584132323547_1018676/role/executor/app_name/nokia-sps-decoder-Spark/instance/xxxx/number/3"
Sample executor payload (push_load_2.txt):
Pushgateway UI:
from spark-metrics.
stoader
commented on June 12, 2024
did you run the curl command from the executor node or from your dev machine?
from spark-metrics.
julianblack
commented on June 12, 2024
From my dev machine, to verify the payload format. But I can also try it from a data node.
from spark-metrics.
julianblack
commented on June 12, 2024
Its also worth noting I spun up a different pushgateway instance, in a separate kubernertes cluster and seeing same behavior when plugin is pushing the data.
from spark-metrics.
julianblack
commented on June 12, 2024
@stoader just to cover all bases here. I am passing the jars above seen in the original post. Are these the correct dependencies?
(version wise).
I also noticed when I removed the truststore.jks from the driver, I didn't receive an error message, and the stats were not pushed to the pushgateway. (Same behavior I'm seeing on the executors.) Do you know why I wouldnt be seeing an error message? I think if we can figure the logging, it will give us more insight to the issue on the executors. (Potentially SSL-related).
Thanks,
from spark-metrics.
julianblack
commented on June 12, 2024
Alright scratch that last comment. I do see this error on the driver:
2020-08-17 15:48:45,581 INFO [org.apache.spark.banzaicloud.metrics.sink.PrometheusSink] - metricsNamespace=None, sparkAppName=Some(nokia-sps-decoder-Spark), sparkAppId=Some(application_1584132323547_1019351), executorId=Some(driver)
2020-08-17 15:48:45,581 INFO [org.apache.spark.banzaicloud.metrics.sink.PrometheusSink] - role=driver, job=application_1584132323547_1019351
2020-08-17 15:48:45,582 ERROR [org.apache.spark.util.Utils] - Uncaught exception in thread Thread-5
java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:248)
at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:255)
at sun.net.www.protocol.https.HttpsClient.createSocket(HttpsClient.java:405)
at sun.net.NetworkClient.doConnect(NetworkClient.java:162)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:305)
at io.prometheus.client.exporter.PushGateway.pushAdd(PushGateway.java:182)
at com.banzaicloud.spark.metrics.sink.PrometheusSink$Reporter.report(PrometheusSink.scala:98)
at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
at com.banzaicloud.spark.metrics.sink.PrometheusSink.report(PrometheusSink.scala:233)
at org.apache.spark.metrics.MetricsSystem$$anonfun$report$1.apply(MetricsSystem.scala:116)
at org.apache.spark.metrics.MetricsSystem$$anonfun$report$1.apply(MetricsSystem.scala:116)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.metrics.MetricsSystem.report(MetricsSystem.scala:116)
at org.apache.spark.SparkContext$$anonfun$stop$3.apply$mcV$sp(SparkContext.scala:1938)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1385)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1937)
at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:575)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1990)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
from spark-metrics.
julianblack
commented on June 12, 2024
@stoader I've solved the issue. It was an SSL issue on the executors. My extraJavaOptions were being overwritten at the application level. Though as seen above, the plugin logs the SSL issue on the driver, but not on the executors. It might help to have these logs for troubleshooting at the executor level. Anyways thanks for all the help! Awesome plugin to extract metrics.
from spark-metrics.
Related Issues (20)
- Metrics namespace HOT 11
- Pushgateway Read timed out HOT 2
- Want to understand that this spark matrics repo will work with prometheus in Hadoop cluster ? HOT 4
- Release Spark provided fix to maven HOT 2
- Metrics name pre-processing by custom Prometheus sink is working for only one component(driver/executor/applicationMaster) HOT 10
- Filter metrics HOT 10
- Metric Name RegEx Replacement doesn't work with JMX HOT 2
- Spark Metrics Stop Pushing After Pushgateway Restarts
- Configure sink to stop sending job as label/group-key HOT 3
- Metrics filter doesn't work
- Add remote_write to PrometheusSink
- Security Policy violation Binary Artifacts HOT 315
- Security Policy violation Branch Protection HOT 314
- Prometheus Sink is not working with SparkPi
- VictoriaMetrics HOT 2
- Repetitions of last metric value HOT 2
- Adding the ability to set custom labels on metrics
- com.banzaicloud:spark-metrics_2.12:3.1-1.0.0 version Not published to maven central HOT 3
- Only driver metrics visible on local
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-metrics.