killbill / killbill-platform Goto Github PK

De-register healthchecks/metrics, when the stop() method is called on the bundle/plugin.

Today we only have org.killbill.osgi.system.bundle.export.packages (part of OSGIConfig), and so when a package is missing the choice is to either recompile/release platform or copy all the the existing package and add the missing one to add it as a system property.

We want to cut that in multiple pieces that will allow for overrides more easily.

So the mock time is shared with the plugins.

We have this. But it turn out that working on this need more work and testing than I thought, so the PR will be closed and work will be allocated after killbill/killbill-oss-parent#552 completed.

Quote from the PR:

Wondering if we can better refactor this...

OSGI bundles can currently request a LogService which is provided by KillbillLogWriter. Our implementation simply delegates it to (slf4j) LoggerFactory (logback in practice).

LogService has been deprecated (https://docs.osgi.org/specification/osgi.cmpn/7.0.0/service.log.html) and bundles should now be using (osgi) LoggerFactory. I still think we should support it for now though, but the "main" service implementation should be a KillbillLoggerFactory which implements (osgi) LoggerFactory, returns a KillbillLogger, which then delegates to (slf4j) LoggerFactory.

KillbillLogWriter should essentially internally only rely on KillbillLoggerFactory / KillbillLogger (instead of relying on both slf4j and (osgi) LoggerFactory as you have refactored.

See slide 17 of https://www.slideshare.net/bjhargrave/the-new-osgi-logservice-14-and-integrating-with-slf4j which describes exactly our problem: we have bundles (e.g. Kill Bill plugins) directly relying on slf4j and pure osgi bundles relying on the OSGI log services (e.g. core Felix framework). We should feed everything into slf4j (option 1 of slide 18).

Maybe we can also re-use existing work... There might be some classes upstream (slf4j project? Felix?) which can already implement some of this, to be investigated (the (osgi) Logger has quite a few methods to implement...). The only tricky bit is the integration with LogEntriesManager, which provides an HTTP API (Server-sent events) to retrieve these logs (this is how the KPM UI in Kaui works). We should make sure this doesn't break.

As part of this, I recommend doing quite a bit of testing with a Kill Bill plugin but also some generic OSGI bundle (https://felix.apache.org/documentation/downloads.html), to verify the logs still flow correctly to the main slf4j system (and that the logback configuration applies to these, e.g. logs from bundles should respect the defined logback format from the logback.xml).

Note: the astute reader will notice that LogEntriesManager only receives OSGI logs, not slf4j ones (e.g. logs from Kill Bill plugins using slf4j directly). This is a long standing limitation (last I checked, there isn't a way to listen to slf4j logs).

BTW feel free to remove jruby references (support removed in 0.23.x).

See #5.

This is a follow up issue based on silence SSE logs issue. This problem showing up when:

1. killbill running with kpm and logger plugin installed.
2. SSE endpoint (http://127.0.0.1/plugins/killbill-osgi-logger) consumed by KAUI (or any SSE client)

Note that running killbill with jetty vs tomcat will produce slightly different error messages, and both tomcat and jetty throws different Exception in different times. But exception message always started by ServerRuntime$Responder - Error while closing the output stream in order to commit response on both of them.

When googling for specifics error (like java.lang.IllegalStateException: s=OPEN,api=BLOCKED,sc=false,e=null) some says it is jetty problem, and fixed long ago.

Last Update:

• Jan 10, 2023

• Jan 9, 2023.

Tomcat (8.5.82)

killbill and KAUI deployed as described in getting started guide.

NullPointerException: null

Full stacktrace here.

2022-12-14T03:47:11,311+0000 lvl='ERROR', log='ServerRuntime$Responder', th='http-nio-8080-exec-6', xff='', rId='', tok='', aRId='', tRId='', Error while closing the output stream in order to commit response.
java.lang.NullPointerException: null
        at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:214)
        .... 
        at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:302)
	at java.base/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:187)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:278)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247)
	at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:714)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:258)
        ....

ArrayIndexOutOfBoundsException

Full stacktrace here.

2022-12-14T03:29:03,306+0000 lvl='ERROR', log='ServerRuntime$Responder', th='http-nio-8080-exec-3', xff='', rId='', tok='', aRId='', tRId='', Error while closing the output stream in order to commit response.
java.lang.ArrayIndexOutOfBoundsException: Index 8 out of bounds for length 8
	at org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:317)
	at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1182)
        ....
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:302)
	at java.base/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:187)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:278)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247)
	at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:714)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:258)
        ....

InvalidMarkException: null

Full stacktrace here.

2022-12-14T04:05:38,653+0000 lvl='ERROR', log='ServerRuntime$Responder', th='http-nio-8080-exec-4', xff='', rId='', tok='', aRId='', tRId='', Error while closing the output stream in order to commit response.
java.nio.InvalidMarkException: null
	at java.base/java.nio.Buffer.reset(Buffer.java:399)
	at java.base/java.nio.ByteBuffer.reset(ByteBuffer.java:1125)
	at java.base/java.nio.ByteBuffer.reset(ByteBuffer.java:262)
        ....
at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:278)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247)
	at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:714)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363
        ....

Jetty

Killbill run using mvn jetty:run command, while KAUI deployed in tomcat following the getting started guide.

IllegalStateException (s=OPEN,api=BLOCKED,sc=false,e=null or s=CLOSING,api=BLOCKED,sc=false,e=null)

Full stacktrace here.

2022-12-14 12:17:10 [qtp367363986-348] ERROR o.g.j.server.ServerRuntime$Responder - Error while closing the output stream in order to commit response.
java.lang.IllegalStateException: s=OPEN,api=BLOCKED,sc=false,e=null
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:813)
	at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:302)
	at java.base/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:187)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:278)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247)
	at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389)
        ....
        

IllegalArgumentException: setContentLength(0) when already written 628

Full stacktrace here.

java.lang.IllegalArgumentException: setContentLength(0) when already written 628
	at org.eclipse.jetty.server.Response.setContentLength(Response.java:949)
	at javax.servlet.ServletResponseWrapper.setContentLength(ServletResponseWrapper.java:116)
	at javax.servlet.ServletResponseWrapper.setContentLength(ServletResponseWrapper.java:116)
	at org.glassfish.jersey.servlet.internal.ResponseWriter.writeResponseStatusAndHeaders(ResponseWriter.java:122)
	at org.killbill.billing.server.filters.KillbillMDCInsertingServletFilter$Adapter.writeResponseStatusAndHeaders(KillbillMDCInsertingServletFilter.java:71)
	at org.killbill.billing.server.filters.RequestDataFilter$Adapter.writeResponseStatusAndHeaders(RequestDataFilter.java:106)
	at org.glassfish.jersey.server.ServerRuntime$Responder$1.getOutputStream(ServerRuntime.java:632)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:171)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:276)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247)
	....

Originally reported here.

Now, AFAIR, this property is a fallback property. But I don't remember why it need to have extra / (that causing bug). Maybe try to comply with previous version?

Nevertheless, current/default state of configuration causing bug explained above, and it need to be fixed. Things that need to do:

1. After all, constructed URL https://oss.sonatype.org//content/ .... is wrong. It should be construct valid URL.
2. Removing extra / in default value seems make sense, but do check thoroughly it not break anything else.
3. Implements this comment.
4. While working at this: Use consistent naming URI vs URL.

/org/killbill/billing/server/version.properties conflicts with the one in Kill Bill, Kill Pay.

If would be nice to find them all and pass a productName to ProductInfo. This would fix the immediate issue and let us have a more refined update checker mechanism in the future.

mvn package complains: Could not find artifact com.palominolabs.metrics:metrics-guice:jar:3.2.1 in central (https://repo.maven.apache.org/maven2)

1. Enforce the Spotbugs checks by enabling the property check.fail-spotbugs
2. Remove the various check.spotbugs-exclude-filter-file properties and associated spotbugs-exclude.xml files
3. Remove the @SuppressFBWarnings annotations
4. Address the errors.

It would be nice to simplify deployments by automatically detecting service discoveries like Eureka (see https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html).

In class org.killbill.billing.osgi.BundleRegistry line 77 the method FileInstall.startBundle() is invoked, which returns false when the bundle fails to start for any reason.

The exception is not propagated, the result of the call is not checked and the Bundle is added to the registry despite the operation failing to start the bundle.

When trying to invoke BundleRegistry.stopAndUninstallBundle() on a bundle that has failed to start the bundle is not removed from the Registry forcing a restart in order to clear the bundle that has failed to stop.

Dear Team,
Does Killbill support chargeback in case subscription expiry date passed but cancellation date entered before expiry date.
For Example:
Product A Subscribed from 1-1-2020 to 29-02-2020. Monthly bill generated for both months (Jan and Feb). On 5th March, Cancellation request received for 20-02-2020.
What will be billing engine behavior?
I assume, billing engine will calculate a chargeback from 20-02-2020 to 29-02-2020. Is this assumption correct?

In KillbillActivator, register the HealthCheckRegistry as an OSGI service, so that plugins can access other plugins and Kill Bill healthchecks.

With #162, the log entries will contain the accountRecordId and userToken.

Add support for filtering the stream by userToken or accountId (e.g., via a query parameter). The plugin will need to call the RecordIdApi first to find the accountRecordId from the accountId.

Compilation error when compiling w/ Java 1.6.0_65:
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ killbill-platform-test ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 9 source files to /Users/kpostlewaite/repos/killbill-platform/platform-test/target/test-classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[23,20] package java.nio.file does not exist
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[24,20] package java.nio.file does not exist
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[25,20] package java.nio.file does not exist
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[26,20] package java.nio.file does not exist
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[27,20] package java.nio.file does not exist
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[49,25] cannot find symbol
symbol : class Path
location: class org.killbill.billing.beatrix.integration.osgi.TestJrubyNotificationPlugin
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[49,58] cannot find symbol
symbol : variable FileSystems
location: class org.killbill.billing.beatrix.integration.osgi.TestJrubyNotificationPlugin
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[49,48] cannot find symbol
symbol : variable Paths
location: class org.killbill.billing.beatrix.integration.osgi.TestJrubyNotificationPlugin
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[67,23] cannot find symbol
symbol : class NoSuchFileException
location: class org.killbill.billing.beatrix.integration.osgi.TestJrubyNotificationPlugin
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[76,27] cannot find symbol
symbol : variable Files
location: class org.killbill.billing.beatrix.integration.osgi.TestJrubyNotificationPlugin
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[105,33] cannot find symbol
symbol: variable Files
return Files.exists(MAGIC_FILE_PATH);
[ERROR] /Users/kpostlewaite/repos/killbill-platform/platform-test/src/test/java/org/killbill/billing/beatrix/integration/osgi/TestJrubyNotificationPlugin.java:[117,8] cannot find symbol
symbol : variable Files
location: class org.killbill.billing.beatrix.integration.osgi.TestJrubyNotificationPlugin
[INFO] 12 errors

Kaui will close periodically the SSE channel, which leads to confusing and verbose logs:

killbill_1  | Jan 24, 2020 1:59:11 AM org.apache.coyote.AbstractProcessor setErrorState
killbill_1  | INFO: An error occurred in processing while on a non-container thread. The connection will be closed immediately
killbill_1  | java.io.IOException: Broken pipe
killbill_1  | at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
killbill_1  | at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
killbill_1  | at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
killbill_1  | at sun.nio.ch.IOUtil.write(IOUtil.java:65)
killbill_1  | at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
killbill_1  | at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
killbill_1  | at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
killbill_1  | at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
killbill_1  | at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1259)
killbill_1  | at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670)
killbill_1  | at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:607)
killbill_1  | at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:597)
killbill_1  | at org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:581)
killbill_1  | at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:272)
killbill_1  | at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1511)
killbill_1  | at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:282)
killbill_1  | at org.apache.coyote.Response.action(Response.java:173)
killbill_1  | at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:317)
killbill_1  | at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284)
killbill_1  | at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
killbill_1  | at org.killbill.billing.plugin.core.resources.ServletSse.send(ServletSse.java:73)
killbill_1  | at org.jooby.Sse.lambda$send$3(Sse.java:1065)
killbill_1  | at org.jooby.funzy.Throwing.fn(Throwing.java:2423)
killbill_1  | at org.jooby.funzy.Throwing.access$400(Throwing.java:39)
killbill_1  | at org.jooby.funzy.Throwing$Supplier.get(Throwing.java:145)
killbill_1  | at org.jooby.funzy.Try.apply(Try.java:691)
killbill_1  | at org.jooby.Sse.send(Sse.java:1063)
killbill_1  | at org.jooby.Sse.access$000(Sse.java:425)
killbill_1  | at org.jooby.Sse$Event.send(Sse.java:590)
killbill_1  | at org.killbill.billing.osgi.bundles.logger.LogsSseHandler$1.run(LogsSseHandler.java:67)
killbill_1  | at org.killbill.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:48)
killbill_1  | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
killbill_1  | at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
killbill_1  | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
killbill_1  | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
killbill_1  | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
killbill_1  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
killbill_1  | at java.lang.Thread.run(Thread.java:748)

Dear Team,

Does Killbill support chargeback in case subscription expiry date passed but cancellation date entered before expiry date.
For Example:
Product A Subscribed from 1-1-2020 to 29-02-2020. Monthly bill generated for both months (Jan and Feb). On 5th March, Cancellation request received for 20-02-2020.
What will be billing engine behavior?
I assume, billing engine will calculate a chargeback from 20-02-2020 to 29-02-2020. Is this assumption correct?

We currently export a health check for our KB internal queues based on some HoltWinters metrics we compute.

We have had reports of companies complaining about false positives, where their deployment is reported as unhealthy while the situation is expected (e.g monthly bill run where entries spike up).

We would like to add a way to disable the check - and simply return true - based on some configuration.

If PluginNamingResolver created without specifying the pluginVersion, getPluginVersion() throw IllegalStateException. It is not correct, since we have a chances to analyzing version from stringContainsVersion.

IIRC, that assertion added when working with ArtifactAndVersionFinder, but as workaround, we can change that line in ArtifactAndVersionFinder with:

final String pluginNameAsArtifactId = pluginKey + "-plugin";

And then, we can remove PluginNamingResolver.of(pluginKey) static factory. And start to re-condider getPluginVersion() implementation. After all, none of operation in PluginNamingResolver should throw an exception, since even if we cannot found any version, it will return 0.0.0.

In KillbillLogWriter, MDC entries can be fetched from MDC.getCopyOfContextMap(). Change LogEntryJson to contain these: userToken, tenantRecordId, accountRecordId.

InfluxDB writer plugin:

https://github.com/iZettle/dropwizard-metrics-influxdb

Supports authString when sending metrics to influxdb.
Counterpart in killbill doesn't support it, thus authenticated access is not supported.

This is similar to what we do when we make API calls from KB to plugin (see https://github.com/killbill/killbill-platform/blob/master/osgi/src/main/java/org/killbill/billing/osgi/ContextClassLoaderHelper.java

There's random exception thrown when installing plugin.

2024-03-29T22:00:36,844+0000 lvl='ERROR', log='Bus', th='bus_events-th', xff='', rId='', tok='', aRId='', tRId='0', Exception thrown by subscriber method handleKillbillEvent(org.killbill.billing.notification.plugin.api.ExtBusEvent) on subscriber org.killbill.billing.osgi.OSGIListener@7cc8ae6c when dispatching event: DefaultBusExternalEvent{objectId=null, accountId=null, tenantId=null, objectType=SERVICE_BROADCAST, eventType=BROADCAST_SERVICE}
java.lang.IllegalStateException: Plugin <XXX>-plugin--7997d0 version null cannot be started because the version 1.0.1-7997d04 already exists in the registry (state = 32)
	at org.killbill.billing.osgi.BundleRegistry.installAndStartNewBundle(BundleRegistry.java:74)
	at org.killbill.billing.osgi.OSGIListener.handleKillbillEvent(OSGIListener.java:109)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

This might have something to do with calling various /nodesInfo endpoints (install, start, check at almost the same time) and there's race condition? Need more investigation when this actually happened.

Like we have the HTTP, GET http://127.0.0.1:8080/1.0/kb/bundles/{bundleId}, from OSGI API can we do this request ? thanks

The Joiner.join causes a lot of churn in young objects.

The typical way to register handlers from a plugin can be seen from the code snapshot below extracted from the analytics plugin:

dispatcher.registerEventHandlers(configHandler,
           new OSGIKillbillEventDispatcher.OSGIFrameworkEventHandler() {
               @Override
               public void started() {
                  // ...
                   dispatcher.registerEventHandlers(analyticsListener);
                  // ...
               }
           });

We are registering a lambda function (anonymous function) whose goal is to a receive platform event when ALL plugins have been started, and then register our handlers analyticsListener to start processing bus events. This sequence works well when we start the platform because the platform code sends a custom OSGI event.

However, this sequence does not work when we restart the plugin because such event is only called during platform initialization and not for a single plugin start/restart, and so restarting a plugin whose event registration matches the snippet above leads to killbill event handlers not being registered.

The KPM plugin code tries to guess the artifactId by convention (PluginNamingResolver.of(pluginKey).getPluginName()). However, if the convention isn't respected, the download fails. If the caller has specified pluginArtifactId, the plugin should trust the caller and use the specified pluginArtifactId instead of guessing. The code should also be made more robust so that it works even when one of pluginKey, artifactId, and version is not specified. This fix would need to be implemented carefully taking into consideration all the possible combinations of pluginKey, artifactId, and version.

More details in https://github.com/killbill/technical-support/issues/134

See https://github.com/brettwooldridge/HikariCP/wiki/FAQ:

Q: How do I properly shutdown the HikariCP DataSource?

A: Shutting down the DataSource is especially important in web application containers
where applications can be hot-deployed. Call the shutdown() or close() method on
the HikariDataSource instance. You can typically configure Spring or other IOC containers
to specify a "destroy" method.