coherence-community / oracle-bedrock Goto Github PK
View Code? Open in Web Editor NEWOracle Bedrock
License: Other
Oracle Bedrock
License: Other
We noticed that due to earlier cleanup / refactoring of the Container streams I/O is no longer output from JUnit tests when running in Container Mode.
This is due to the "default" scope using the new ContainerScope implementation, that of which captures and pipes output. This is incorrect behavior for the "default" scope, which should instead send I/O directly to the Platform Scope.
Add Average to oracle-tools for monitoring/JMX use
Oracle Tools fails to work on Windows due to timing and class-path issues
Currently we only allow capturing stdout for an ApplicationConsole. This is problematic when we need to test with applications specifically use other streams, like stderr. Furthermore we currently provide no facility to control or provide stdin for an application, thus again limiting the ability to write fully interactive tests.
To resolve this I propose we redesign/refactor how we use ApplicationConsoles. In particular I suggest the following interface:
interface ApplicationConsole
{
/**
* Obtains the BufferedReader that can be used to read content an Application sent to stdout.
*/
public BufferedReader getStdOutReader();
/**
* Obtains the BufferedReader that can be used to read content an Application sent to stderr.
*/
public BufferedReader getStdErrReader();
/**
* Obtains the PrintWriter that can be used to send input to the stdin of an Application.
*/
public PrintWriter getStdInWriter();
/**
* Closes the ApplicationConsole.
*/
public void close();
}
We've already fixed white space issues in the class path, but the system properties that are added to the command must also be sanitized if they have white spaces.
Now that we can remotely execute Callables on Applications we can introduce the ability to use this with deferred asserts. ie: assert that eventually executing some remove callable arrives in a required state.
When running Oracle Tools on JDK8 (b91), Coherence (any version using XSD) won't start due to improved/tightened JDK8 XSD loading permissions.
To work around this we should add the following default system property to the Coherence-based Schemas, at least until this is either fixed in JDK8 or Coherence (with a change of XSD loading security).
schema.setSystemProperty("javax.xml.accessExternalSchema", "file");
Document Coherence Member Schema and Coherence Member Builder
We should introduce a Coherence-based StorageMemberSchema to capture the standard settings for a Storage Enabled Members.
Once we've done this we should make the existing CoherenceMemberSchema inherit from the StorageMemberSchema and deprecate it.
We need to clean up the missing javadocs and ensure that we have clean builds on Java 6, 7 and 8
Currently the processes built by the NativeApplicationBuilder(s) will create (child) processes that are independent of the creating (parent) process. Should the parent process terminate (for what ever reason), the child processes will be let running (ie: orphaned). Often this is undesirable (and not the expected default behavior).
Leveraging the work in #51, it should be possible to introduce the concept of orphan-protection for native application builders.
Write the Oracle Tools Book
Consider a test that wants to set numerous System properties, and/or establish local JMX instrumentation for a test.
Instead of having @before test methods do this via calls to System.setProperty(...) etc, it would be nice if we could simply leverage an Application Schema.
eg: In the case of a Coherence-based test we could do this.
@Before
public void setup()
{
ClusterMemberSchema schema = new ClusterMemberSchema().setClusterPort(...);
LocalApplicationBuilder builder = new LocalApplicationBuilder();
LocalApplication application = builder.build(schema); //<-- will set the local application properties etc, from the schema. }
In the latest release of Java (on OSX), applications appear to run with awt enabled. This causes background testing applications to gain UI focus and thus stop developers working in the foreground on other applications. To resolve this we need to introduce the headless mode switch and set it "on" by default.
eg: -Djava.awt.headless=true
For platforms that don't correctly isolate Coherence applications, and/or those that don't respect standard network settings, we should provide a top-level Abstract class that we can base all our tests (especially for the Coherence Incubator).
This will ensure we always have the correct isolation.
We should introduce a Coherence-based ManagementMemberSchema to capture the standard settings for a Management Node.
When creating application processes we often would like to see diagnostic information about how the process. For example, the environment, settings, classpath etc, that the Oracle Tools established to establish an application.
To provide this capability, on a schema-by-schema basis we should add the ability to do this:
schema.setDiagnosticsEnabled(true);
Additionally, we should allow:
When running on some platforms (like Windows) copying all of the System Properties, including those defined at runtime by the Java Platform, from one process and passing them to another can be problematic. In some instances attempting to re-define the Java Platform defined system properties can cause applications (ie: Java) not to start. Hence we need to avoid providing this facility!
To solve this issue we're going to refactor the PropertiesBuilder.fromCurrentSystemProperties method to instead become PropertiesBuilder.fromCurrentNonStandardSystemProperties so that the PropertyBuilder that is created only contains those defined by the end-user.
It turns out we don't actually use this method anywhere anyway - apart from in tests. Anyone using this method outside of the tests are essentially in trouble anyway - as their applications won't be cross platform.
Randy suggested that we should move to version 4 from version 3
Currently we use the term "virtualization" to represent the concept of running a Java application "virtually" with in another application. Though correct in that we actually "virtualize" System properties, I/O, MBean servers etc, this however can be confusing when running such applications in a virtual appliance.
We should rename the internal classes to be a "Container" instead of "Virtualized". While this may cause confusion when discussing using applications in a container, it is closer to what is happening in a traditional sense (with the additional characteristic of isolating I/O streams and MBean servers which containers dont' traditionally do).
By introducing a specialized Junit Test Runner for Containers, we can do a lot more isolation and dramatically simplify test setup/cleanup.
eg: we can support annotations like
@SystemIsolation(NONE)
@SystemIsolation(FOR_EACH_TEST_METHOD)
@SystemIsolation(FOR_TEST_CLASS)
This could then be extended for Coherence as well.
NOTE: When we do this we can remove the requirement to inherit from the AbstractTest class (we could theoretically deprecate it)
Upgrade to use the latest version of Coherence (or a version pom)
When using an Iterator there's often the requirement to capture an individual value and then use it later.
For example: When using an AvailablePortIterator to configure a cluster, the first port chosen should be used for all other members in the cluster (so that a cluster is formed).
Instead of choosing the port "early", we often want to delay the choice until the last minute. To do this we need a way of "capturing" a choice and then using the choice in other places.
Capture<Integer> clusterPort = new Capture<Integer>(availablePortIterator);
The first time the clusterPort is used it won't have a value, in which case the availablePortIterator is used to determine a value. After that the first "captured" value will always be used.
When attempting to run Coherence in WKA mode (using system properties), both the WKA port ("tangosol.coherence.wka.port") and the Local Address Port ("tangosol.coherence.localport") need to be set to the same value.
Unfortunately the ClusterMemberSchema doesn't provide an out-of-the-box method to set the LocalPort which is somewhat annoying (as you have to use setSystemProperty).
We should hence;
i). add a "setLocalHostPort" method, and
ii). when a call to "setWellKnownAddressPort" is made, it should also call this method (to make it easy for the developer).
The license headers in all files contain the text:
It should be:
We need to fix this in:
Currently when we use a virtualized system (of System resources), the StdOut and StdErr streams are prefixed with [(Default Virtual System) XX:].
This is not really required and actually makes the output look weird. We can easily remove this (and line numbering).
To indicate that virtualization is occurring we should simply log a message when virtualization starts and stops (via logging).
Paul Makin found that when running Native / External Application-based Tests on Windows NT / XP, destroying the application may lead to tests hanging.
This appears to be caused by the order in which we close the i/o streams. To resolve the problem we must close the streams before interrupting the i/o threads and before destroying the underlying process.
When running a Java application internally when it is closed its JMX server is left running. This leaves the port in use but can also on rare occasions stop the JVM from shutting down. The port in use is not a big issue if using an AvailablePortIterator to assign port numbers but could be for repeated use of a schema in a test suite with a fixed JMX port. The intermittent problem with the JVM came to light occasionally when running tests for the .Net wrapper of Oracle Tools. Occasionally the test runner process in Visual Studio would not stop as the internal JVM still had live non-daemon threads associated to JMX running. Closing the JMX servers seemed to make this issue go away - although it was very intermittent to start with.
Closing the JMX server also allows test code to wait for an internal process to be closed. It is possible to register a listener on a JMX connection to a virtual process and be notified when that connection closes.
E.G you can do something like this...
JMXConnector jmxConnector = member.getDeferredJMXConnector().get();
final CountDownLatch memberDestroyedLatch = new CountDownLatch(1);
jmxConnector.addConnectionNotificationListener(new NotificationListener()
{
@Override
public void handleNotification(Notification notification, Object handback)
{
memberDestroyedLatch.countDown();
}
},
new NotificationFilter()
{
@Override
public boolean isNotificationEnabled(Notification notification)
{
return false == JMXConnectionNotification.OPENED.equals(notification.getType());
}
},
null);
member.destroy();
memberDestroyedLatch.await();
This is useful where your test needs to know the process has really stopped before it does something
When creating external/native Java processes, it would be nice if we could control/debug them.
To enable this we should add the ability to specify debug properties when using External/Native Java Application builders (they don't apply to Virtual/Container-based builders).
This allows a single jar to be generated, thus making adoption easier
Currently the application names produced by an application group builder start at 0. This is confusing, especially when being used with Coherence (and other cluster-based systems), where member names start at 1.
Fixing this will be a BREAKING CHANGE for anyone that assumes the name of an application starts at 0.
I noticed when running test on Windows, specifically using a Jenkins installation that was in "C:\Program Files\jenkins" that the application builders may not work as expected and in some circumstances, actually fail.
The proposed solution is to introduce the concept of a ClassPath class that can be used to represent and manipulate ClassPaths in a platform independent manner, instead of having class path parsing code all over the place.
Document Distributed Testing Best Practises
Upgrade to use Coherence 3.7.1.10
We have a setLogLevel method but should also have a setLog method.
When running the VirtualizationClassLoaderTests on Windows with Java 7 and 8, the order of the test method execution becomes significant. This is because some methods cause side-effects which then later methods either implicitly depend upon or don't expect. As the order of method execution for Junit (based on Java 7 and 8 method reflection) is somewhat random on Windows, this can cause tests to fail some what randomly.
We need to ensure that there's no implicit ordering or dependencies or side-effects in our test methods, particularly those testing ClassLoaders.
The Notified implementation can never be guaranteed to operate correctly when we consider the possibility of Operating System "notification". Furthermore the behavior and timing of notification can differ significantly across platforms, especially between Windows and Linux, worse when these platforms are virtualized.
Consequently the safest thing to do is remove this support - it's not used anywhere in the Coherence Incubator.
Often we would like the ability to remotely execute Callables/Runnables in Java Applications (whether they are executing in a container, native or remotely). While we can currently execute Callables when Java Applications are running in a container, we don't yet have the ability or infrastructure to do this in Natively (or eventually Remotely).
We've seen numerous times that failing to set a schema to inherit environment properties can cause application start and shutdown issues. In every case it seems that failing to inherit environment properties is the underlying cause (as without it new processes have no environment variables - which can make locating the Java executable/binary problematic).
Consequently we should make it inheriting the default, not clearing (as it is now).
NOTE: all existing functional tests are set to inherit, so it's a good reason to make it the default!
We should introduce a Coherence-based ProxyMemberSchema to capture the standard settings for a Proxy Node.
The current approach to "ensuring" a Deferred is to simply poll at regular intervals until a maximum timeout is reached.
While useful, this could be improved as continuous polling is often a waste of time. Instead we should allow different strategies to be specified, including:
By default we should use a Random Fibonacci Back-Off approach as this has proven to be more effective than polling and exponential back-off algorithms.
To implement this approach we should modify the Ensured class to resolve and use an Iterator that provides suitable times to wait for each attempt at resolving the underlying resource. Once this refactoring has occurred, we can define several Iterator implementations that provides Ensured's with the appropriate values. Those would include:
Currently we rely on JMX being enabled to provide cluster/member information that for the most part is used for integration testing (with assertThat).
Now that we have the ability to submit arbitrary remote executables, there's little reason to use/require JMX. Changing to use remote executables will reduce/eliminate the requirement for JMX setup / configuration / installation / virtualization, especially in a Container-based environment.
An ApplicationGroupController allows programmatic control of ApplicationGroups at runtime. Specific examples include;
a). the ability to perform automatic "Rolling Restarts" of (specific) Applications in an ApplicationGroup.
b). the ability to introduce Chaos Monkeys (see: http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html) into an Application Group.
c). systematically perform custom actions against Applications in an ApplicationGroup (eg: running a report).
We need to update to support the latest github API - without this we can't publish documentation
Once we have a custom Container runner we could introduce a specific Coherence-based runner that can setup/establish clusters using annotations.
This will make testing way simpler and more isolated through declarative configuration.
eg:
@cluster(name=RANDOM, port=RANDOM, ttl=0)
We can also extend this to pre-load cache data, record cache data after runs/exceptions and compare cluster state to that which is desired. ie: assert that the state of a cluster after running a test is as expected.
NOTE: When we do this we can remove the requirement to inherit from the AbstractCoherenceTest class (we could theoretically deprecate it)
We should provide a simplification of this:
getMBeanAttribute(new ObjectName("Coherence:type=Service,name=Federated[DistributedCache],nodeId=1"),
"StatusHA", String.class)
Document Application Group and Application Group Builders
Currently the Deferred#get() method does not permit returning null object references as null has the implied meaning "object not yet available". This is unfortunately as it doesn't let null be returned - when it is a valid object.
To resolve this we need to refactor the semantics so that:
We throw an UnresolvableInstanceException - when the Object can never be resolved (thus replacing ObjectNotAvailableException which is misleading)
We throw an InstanceUnavailableException - when the Object resolution failed, but may be retried in the future (thus replacing the return value of "null").
We really need the ability to kill an application in addition to simply destroying it. This is especially required for failure-based testing.
Note: we may need to use some operating system level calls to achieve this.
We should introduce an abstract a Coherence-based MemberSchema as the parent of all MemberSchemas. This will allow us to share common settings easily between MemberSchema.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.