Provides a tool to archive data in S3, and query data that is archived through S3 Select. Data is archived in json newline delimited files with two json object; properties and message. Properties provide metadata about the message that can be used with S3 Select to receive a specified subset of message from the archive. The message itself is stored HEX encoded and is decoded back to its original form when using getMessagesFromArchive() function.
- Clone this repository
- Run mvn clean install
- Add dependency to applicable project pom
<dependency>
<groupId>us.dot.faa.swim.tools</groupId>
<artifactId>swim-s3-archiver</artifactId>
<version>1.0</version>
</dependency>Import Required Dependancies
import us.dot.faa.swim.tools.s3archiver.S3Archiver;
import us.dot.faa.swim.tools.s3archiver.ArchivedMessageObject;
import us.dot.faa.swim.tools.s3archiver.MessageReciever; Create new S3Archiver
S3Archiver s3Archiver = new S3Archiver("BUCKET", "S3_SERVICE_URL", "S3_REGION", "S3_KEY", "S3_SECRET", "ARCHIVE_PREFIX", false);Set Upload Schedule (chron [Sec Min Hour Day])
s3Archiver.setUploadChronScheule("0 0/15 * *");Start S3 Archiver
s3Archiver.start(); Add Data for Archiving
HashMap<String, String> properties = new HashMap<String, String>();
properties.put("propertyName", propertyValue);
s3Archiver.archiveMessage("Data to Archive", properties);Query Archive for Specific Data in Archive Files
Hashtable<String,List<String>> filters = new Hashtable<>();
filters.put("Property_Key_1", Arrays.asList("Property_1_Value_A","Property_Value_1_B")); // multiple properties indicate OR
filters.put("Property_Key_2", Arrays.asList("Property_2_Value_A"));
s3Archiver.getMessagesFromArchive(s3Archiver.getCategoryPrefix(), Instant.now().minusSeconds(10 * 60 * 60), Instant.now(), filters, true, this);Recieve and Process Messages
@Override
public void recieveMessage(ArchivedMessageObject archivedMessage) {
System.out.println("Properties:");
System.out.println(archivedMessage.getProperties().toString());
System.out.println("Message:");
System.out.println(archivedMessage.getMessage());
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent