eclipse-serializer / serializer Goto Github PK

Add Binary Handler für Collections$SetFromMap class

The data in this private class is marked as 'Transient,' which is why these data are not being correctly saved. We may need to add a new, specific binary handler.

When deleting a StorageFile its delete method ensures that the StorageFile is writable. The ensureWritable method internally opens a FileChannel if the NIO AFS is used. That FileChannel is not closed after deleting the file (in the app). The open file channel may prevent the OS from deleting the file and releasing disk space.

Is your feature request related to a problem? Please describe.

The creation of SBOMs is expected to be something that will be common practice, or sometimes even mandatory, when you deliver software. The US already has an "Executive Order on Improving the Nation’s Cybersecurity" with the SBOM requirement:
https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/ See Sec 4.E.vii

Describe the solution you'd like

Create a SBOM for every build of Eclipse Serializer. Brian Vermeer has a good article about this topic: https://snyk.io/blog/create-sboms-java-maven-gradle/

Describe alternatives you've considered

Having no SBOM ;)

Additional context

What is an SBOM?
A software bill of materials, often abbreviated as SBOM, is a list of all software components used in an application. The SBOM is made up of third-party open-source libraries, vendor-provided packages, and first-party artifacts built by the organization. You can see it as the full list of ingredients for your applications.

But be careful not to confuse an SBOM with Maven’s Bill Of Materials (BOM).

An SBOM is something you create next to your application so that any user or client has a uniform way to find out what your application is using under the hood.

An issue with module definitions came up when using the Serializer project within Eclipse Store.

Trying to load an existing storage with a missing PersistenceTypeDictionary.ptd causes an unclear exception.
Maybe the exception could be improved or the storage initialization could handle the case that storage files already exists but the PersistenceTypeDictionary is missing.

Is your feature request related to a problem? Please describe.

The current implementation doesn't support serialising Java records directly or as part of other objects.

Describe the solution you'd like

Java records should be serialisable for example following test should pass:

package com.example.application;

import org.eclipse.serializer.Serializer;
import org.junit.jupiter.api.Test;

public class BasicTestingSerializer {

    public record Person(String name, int age) {
    }

    static class Company {
        private String name;

        private Person ceo;

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public Person getCeo() {
            return ceo;
        }

        public void setCeo(Person ceo) {
            this.ceo = ceo;
        }
    }

    @Test
    public void example() {

        // create a company object
        Company company = new Company();
        company.setName("Acme Inc.");
        company.setCeo(new Person("John Doe", 42));
        
        // create a serializer which handles byte arrays
        Serializer<byte[]> serializer = Serializer.Bytes();

        // serialize a company
        byte[] data = serializer.serialize(company);

        // deserialize the data back to a company
        company = serializer.deserialize(data);
        if (company.getCeo().name().equals("John Doe")) {
            System.out.println("It works!");
        }
    }

}

Describe alternatives you've considered

Revert back to using POJOs

Additional context

Default JDK Serialisation eats records happily (as long as they implement Serializable)

Someone working on Eclipse Store reported a VM crash to the JDK but the issue is some nasty code in Eclipse Serializer:
https://bugs.openjdk.org/browse/JDK-8325173

Someone working on Eclipse Serializer needs to re-examine this code.

As discussed, we will update to Java 11 as a minimum requirement.
Within this step, we can build true JMS compatible modules without Maven hackery.

Environment Details

• Eclipse Serializer Version: 1.2.0
• JDK version: 21
• OS: Linux
• Used frameworks: Spring Boot, Memcached

Describe the bug

When spawning virtual threads they get pinned when using the serializer

2024-03-13T13:31:13.208+01:00 TRACE 32774 --- [xyz-local] [               ] o.e.s.p.binary.types.BinaryLoader        : Updating LoadItem OID=1000000000000000254, Type=30 java.lang.String
Thread[#168,ForkJoinPool-1-worker-1,5,CarrierThreads]
    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:183)
    java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
    java.base/java.lang.VirtualThread.park(VirtualThread.java:582)
    java.base/java.lang.System$2.parkVirtualThread(System.java:2643)
    java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
    java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:219)
    java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
    java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990)
    java.base/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)
    java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)
    ch.qos.logback.core.OutputStreamAppender.writeBytes(OutputStreamAppender.java:200)
    ch.qos.logback.core.OutputStreamAppender.writeOut(OutputStreamAppender.java:193)
    ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:237)
    ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
    ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:85)
    ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
    ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
    ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
Thread[#179,ForkJoinPool-1-worker-2,5,CarrierThreads]
    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:183)
    ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:426)
    ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:386)
    ch.qos.logback.classic.Logger.debug(Logger.java:499)
    org.eclipse.serializer.persistence.types.PersistenceTypeHandlerEnsurer$Default.logHandlerCreation(PersistenceTypeHandlerEnsurer.java:269)
    org.eclipse.serializer.persistence.types.PersistenceTypeHandlerEnsurer$Default.ensureTypeHandler(PersistenceTypeHandlerEnsurer.java:252)
    org.eclipse.serializer.persistence.types.PersistenceTypeHandlerProviderCreating.ensureTypeHandler(PersistenceTypeHandlerProviderCreating.java:178)
    org.eclipse.serializer.persistence.types.PersistenceTypeHandlerProviderCreating.provideTypeHandler(PersistenceTypeHandlerProviderCreating.java:86)
    java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
    java.base/java.lang.VirtualThread.park(VirtualThread.java:582)
    java.base/java.lang.System$2.parkVirtualThread(System.java:2643)
    java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
    java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:219)
    java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
    java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990)
    java.base/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)
    java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)
    ch.qos.logback.core.OutputStreamAppender.writeBytes(OutputStreamAppender.java:200)
    ch.qos.logback.core.OutputStreamAppender.writeOut(OutputStreamAppender.java:193)
    ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:237)
    ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
    ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:85)
    org.eclipse.serializer.persistence.types.PersistenceTypeHandlerManager$Default.internalEnsureTypeHandler(PersistenceTypeHandlerManager.java:608) <== monitors:1
    ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
    ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
    ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
    ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:426)
    ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:386)
    ch.qos.logback.classic.Logger.debug(Logger.java:487)
    org.eclipse.serializer.TypedSerializer$Default.deserialize(TypedSerializer.java:321) <== monitors:1

To Reproduce

Use the TypedSerializer in a VirtualThread

Expected behavior

The thread doesn't get pinned.

Additional context

We're seeing a lot of latency where the application is just waiting.

In our code, we have a lot of DTOs like this:

@lombok.Value
@lombok.Builder
class Entity {

  String requiredField;

  @lombok.Builder.Default
  String requiredFieldWithDefaultValue = "default value";

}

For the provided example, Lombok generates a private constructor, getters, and a builder with a default value for requiredFieldWithDefaultValue field.

Will such an object be correctly deserialized by this library? By "correctly" I mean:

1. Deserialization works with a private constructor.
2. If the requiredFieldWithDefaultValue field was added only in the second version of the class, deserializing content from the first version (without this field) will have "default value" as a default value for the field.

The create method in AbstractBinaryHandlerGenericImmutableCollections12 contains a dangerous hack that should be replace by a proper solution. Or more precisely: a necessary change to how Lazy references are linked to the clustered storage will make this hack work no more.

Some background:

The concept of the loading process and implementation of ALL other BinaryHandlers is the following:

• The loader sends a set of ObjectIds to the PersistenceSource (usually the storage layer).
• It receives a block of binary data back.
• The TypeHandlers know how to make sense of the data (entity headers, primitive values, reference ObjectIds)
• entity instances are created via PersistenceTypeHandler#create without references, just to have an instance to register for each of their ObjectIds.
• The loader uses PersistenceTypeHandler#iterateLoadableReferences to iterate all reference ObjectIds on the BINARY level (the raw block of binary data) and checks which ones are already accounted for and which ones have to be loaded
• This continues iteratively until all raw data objectIds have been resolved and all required raw data is available
• THEN, when it is guaranteed that all instances for occuring objectIds are present, PersistenceTypeHandler#updateState is called to fill in the missing references ("connect" the entities to each other) to the created shallow instances.

There code in the mentioned TypeHandler, however, abuses the current linking workaround Persister instance to execute loading directly from the create method in a type handler. This MIGHT work correctly for now and/or for simple cases (or maybe it even doesn't but an inconsistency caused by it has not been discovered), but it definitely breaks the technical concept, introduces unnecessary complexity and creates an accident waiting to happen should the technical concept change in the slightest way that does not take into account this hack.

Consider:
This hack starts a new loading process WHILE processing an ongoing loading process.
It completely bypasses all the already loaded data, the already resolved objectId, the already created instances.
This surely works in a trivial development test, but if the two (or potentially even more) concurrent and competing(!) loading processes yield different data, any kind of inconsistent data and/or JVM crashes can be the consequence.
In short: NEVER start a new loading process from within an ongoing loading process.

Therefore, this handler should be implemented in the proper way, like the other handlers are:

• #create only creates reference-less instances
• #iterateLoadableReferences iterates the reference's objectIds in the binary data
• #updateState fills in the missing references

Environment Details

• Eclipse Serializer Version: 1.2.0
• JDK version: 21.0.1
• OS: Win 10
• Used frameworks: -

Describe the bug

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
	at java.base/java.util.Objects.checkIndex(Objects.java:385)
	at org.eclipse.serializer.collections.lazy.LazyHashMap.calculateIndexPosition(LazyHashMap.java:545)
	at org.eclipse.serializer.collections.lazy.LazyHashMap$SegmentsSpliterator.<init>(LazyHashMap.java:1388)
	at org.eclipse.serializer.collections.lazy.LazyHashMap$ValueSpliterator.<init>(LazyHashMap.java:1654)
	at org.eclipse.serializer.collections.lazy.LazyHashMap$Values.spliterator(LazyHashMap.java:1170)
	at java.base/java.util.Collection.stream(Collection.java:750)
	at org.eclipse.serializer.collections.lazy.LazyCollection.stream(LazyCollection.java:75)

To Reproduce

new LazyHashMap<String, String>().values().stream().toList();

Expected behavior

No Exception, just an empty List as result.

Environment Details

• Eclipse Serializer Version: 1.2.0
• JDK version: 21.0.1
• OS: Win 10
• Used frameworks: -

Describe the bug

I am using a LazyHashSet to store a Wrapper Object, which is identified by identity. When I just use LazyHashSet.add(wrapper) it happens that the same Wrapper appears multiple times in the LazyHashSet and add returns true even tho the Wrapper is already in the set. I am currently not sure when exactly it happens, it is not always.
It seems to work correctly when I do a contains check before adding.

To Reproduce

Store Objects in the LazyHashSet with no overriden equals or hashCode Method. Add the same instance multiple times. The LazyHashSet is stored with eclipse-store, but currently contains only a few objects (< 30).

Expected behavior

One Object instance only appears once in the LazyHashSet.

We might want to consider using java.util.logging instead of slf4j.

java.util.logging is part of the JDK, so using it means having one less external dependency.

Description:
The current project is using an outdated version of the SLF4J API. This task involves upgrading the SLF4J API to the latest version (version 2 or higher).

The upgrade process will involve the following steps:

1. Identify the current version of SLF4J API being used in the project.
2. Research the changes and improvements in the latest version of SLF4J API.
3. Update the project's pom.xml file to use the latest version of SLF4J API.
4. Test the project to ensure that it still works as expected after the upgrade.
5. Resolve any issues or conflicts that arise due to the upgrade.
6. Document the changes made, the reasons for those changes, and any issues encountered during the upgrade process.

This task will help keep the project up-to-date with the latest improvements and features offered by the SLF4J API. It will also help ensure that the project benefits from any security patches or bug fixes included in the latest version of the SLF4J API.

Lazy collections implementations

• LazyArrayList
• LazyHashMap
• LazyHashSet

with specialized type handlers for optimized persistence.

Environment Details

• Eclipse Serializer Version: 1.2.0
• JDK version:21.0.1
• OS:Ubuntu 17
• Used frameworks: SpringBoot 2.7.8

Describe the bug

Maybe it is not bug, only some things which affect the performance. After adding the ‘this.itemCount = 0;’, the performance is higher than default. it seems that the default clean forge to set the itemCount to zero.

` protected void internalInitialize(final int hashLength)
{
synchronized(this.head)
{
this.itemCount = 0;
this.hashSlots = new Item[hashLength];
this.hashRange = hashLength - 1;

			// initializing/clearing item chain
			(this.tail = this.head).next = null;
			
			this.synchCreateStoringChunksBuffers();
		}
	}
	`

Could we do like this and it is safe? @hg-ms , thank you in advance!

To Reproduce

Use LazyArrayList as DataRoot, write and delete very frequently. and log the write/delete time cost, you will find the write/delete cost is relative high, maybe 10ms-200ms. I only used 2k threads to write/delete, I wrapper the LazyArrayList by a ReadWriteLock for concurrently operation.

Expected behavior

the write/delete cost will be small.

Screenshots

Additional context

This low priority.

The license expression in the file headers that I reviewed uses the deprecated GPL-2.0. It should be GPL-2.0-only.

What I see:

EPL-2.0 OR GPL-2.0 WITH Classpath-exception-2.0

What it should be:

EPL-2.0 OR GPL-2.0-only WITH Classpath-exception-2.0

Again... this is (very) low priority.

Environment Details

• Eclipse Serializer Version: 1.3.1
• JDK version: 21.0.2
• OS: alpine linux or Win 10
• Used frameworks: -

Describe the bug

After 1 hour of idle the LazyReferenceManager goes to 100% cpu usage. It directly goes back to normal when something gets loaded from the store.

To Reproduce

Use a LazyHashMap as a store for example. Store something in the LazyHashMap, use something from the LazyHashMap and than leave the app idle for 1 hour.

Expected behavior

LazyReferences get freed and the application stays at low cpu usage when idling.

htop from process

running in an alpine temurin pterodactly egg, but this also happens in my IDE on Win 10.

1147628 pteroda+  20   0   31.7g 316940  21956 R  99.9   0.2   1366:51 LazyReferenceMa                                                                                                                                                                                                  1147563 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:00.00 java
1147564 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:01.51 java
1147565 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:00.07 GC Thread#0
1147566 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:00.00 G1 Main Marker
1147567 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:00.02 G1 Conc#0
1147568 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:19.74 G1 Refine#0
1147569 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:03.10 G1 Service
1147570 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:58.98 VM Periodic Tas
1147571 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:03.06 VM Thread
1147572 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:00.00 Reference Handl
1147573 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:00.00 Finalizer
1147574 pteroda+  20   0   31.7g 316940  21956 S   0.0   0.2   0:00.00 Signal Dispatch

According to the Document: https://jdk.java.net/22/release-notes
There are some changes in unsafe.java

sun.misc.Unsafe shouldBeInitialized and ensureClassInitialized are removed (JDK-8316160)

core-libs

The shouldBeInitialized(Class) and ensureClassInitialized(Class) methods have been removed from sun.misc.Unsafe. These methods have been deprecated for removal since JDK 15. java.lang.invoke.MethodHandles.Lookup.ensureInitialized(Class) was added in Java 15 as a standard API to ensure that an accessible class is initialized.

The Eclipse Store uses this Method in class: XMemory.java

It is important to resolve this issue for the functionality of eclipse-serializer for java 22 and above

Readme states:

Build only the documentation PDF

mvn generate-resources -pl :documentation

But I get

mvn generate-resources -pl :documentation
[INFO] Scanning for projects...
[ERROR] [ERROR] Could not find the selected project in the reactor: :documentation @ 
[ERROR] Could not find the selected project in the reactor: :documentation -> [Help 1]

Also I could not find any PDF generated. Does the Readme need a correction?

Within the modules serializer-persistence and serializer-persistence-binary there are export clauses in the module-info.java file for internal packages.
This should be adapted by either moving out the classes that are not internal or rename the package so that there is no conflict in naming and usage.

Prevent data corruption by throwing an exception in case of enum ordinal changes by automatic legacy type mappings.

Is your feature request related to a problem? Please describe.

We'd like to use EclipseSerializer to serialize objects to a database to be (later) deserialized by a potentially different instance. Therefore we need the TypedSerializer but the problem is, that the output produced is HUGE and seems to include all class information the serializer knows about.
Example: A sample object is serialied to about 2.5kB of binary data using the normal Serializer but to 65kB with the TypedSerializer even when not many other classes have been serialized before and the cache seemed to only include around 150 known classes.

Describe the solution you'd like

I'd like to see a strategy to only include class information about classes effectively encountered in the serialized data and try to reduce the amount of extra data of TypedSerializer

This would open many new usecases for Eclipse Serializer

Environment Details

• Eclipse Serializer Version: 1.0.0
• JDK version: 17.0.7
• OS: Windows 10

Describe the bug

Hello!
I stumbled across your de/serialization framework, which looks great!
As we are just in the middle of transitioning away from hessian, I thought to give it a try. :)
We have created a little testsuite to check if serialization works for us and a few things seem to be not working, so I wanted to ask if these are bugs or if I need to configure something or if there is any other way around the issues.

1. exceptions: org.eclipse.serializer.persistence.exceptions.PersistenceExceptionTypeNotPersistable: Type not persistable: "class java.lang.Exception".
   Is there a way to enable exceptions?
2. EnumSet: java.lang.NullPointerException: Cannot load from object array because "this.this$0.universe" is null
3. BitSet: does not seem to be properly de/serialized (Expected :{135}, Actual :{})
4. ImmutableList: e.g. List.of(1) -> Expected :[1] Actual :[1, java.lang.Object@69d2e517] some strange object in the list?
5. ImmutableSet: like above, but with Set.of(1)
6. de/serializing an object that has transient fields which are initialized like e.g.
   private transient ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
   private transient WriteLock wLock = rwLock.writeLock();
   private transient ReadLock rLock = rwLock.readLock();
   the fields remain null. -> nullreference in later code
7. When trying to de/serialize records, I get
   Could not obtain access to "jdk.internal.misc.Unsafe", please start the VM with --add-exports java.base/jdk.internal.misc=ALL-UNNAMED

Regards,
Raphael

We should provide an additional serializer which transfers the type information as well.