fr33r / noti Goto Github PK

• Need to have the ability to create a new Target resource and have it appended to the collection of Target resources.

• Need to have the ability to retrieve a single Target resource.

Overview

• It is common that a series of notifications being sent on different occasions have the same “content structure”. In other words, the content within the notifications may be nearly identical, except for a handful of parts. Instead of clients needing to send along the nearly identical content on each occasion, this scenario can be more efficiently tackled using templating.
  • This becomes particularly useful when notification content becomes large, an example being HTML content within email notifications.
• Clients could create templates ( ‘POST /templates/‘ ) and then create notifications using these templates by specifying the template, and the values to provide to the template during notification creation.
• Templates would consist of the content within the template, as well as template variable metadata.
  • The metadata could consist of the variable name, type, default values, list of acceptable values, etc.

Background

• In order to accommodate multiple incoming media types for representations, duplicative methods within resource classes (AudienceResource, TargetResource, etc.) are created. Examples include:

  • AudienceResource#createAndAppend_XML
  • AudienceResource#replace_XML
  • TargetResource#createAndAppend_XML
  • TargetResource#replace_XML
  • etc.

• It's worth noting that a fair amount of troubleshooting went into figuring out why 415 Unsupported Media Type responses were being received from response class methods that had the appropriate @Consumes/@Produces annotiations. It was discovered that it was due to the fact the representation classes were constrained to a single media type, and the instantiation of those representation classes must not have been possible.

Representation Organization

• Purposefully, there is deliberate separation between representations by media type. The api.representations package contains a sub-package per media type, such as json, xml, siren, etc.
• Major benefits come from this approach:
  • Minimal risk when new media types are supported, or existing media types are removed. The classes for each media type are isolated from one another.
  • Allows for representations of each media type to evolve independently.
    • It's frequent that representations are similar between media types, but there is no requirement of this. In fact, the richer the media type (especially hypermedia-capable media types), the more they will differ in content.
  • Each representation has a single concern/responsibility - to represent resource state via a particular media type.
    • Without the creation of representation classes per media type, representation classes take on the responsibility of serialization for every possible media type, which causes the classes to become bloated.

Proposed Solutions

• Pass in InputStream to resource classes, and perform the parsing on the InputStream.
  • Could potentially instantiate and invoke MessageBodyReaders?
• Create a resource class for each media type, instead of duplicating functions with postfixes such as _JSON, _XML, _Siren, etc.

• Resource representations are a "presentation layer" concern. Due to this, a more general abstraction should be returned from application services, which is then translated in the "presentation layer" (RESTful API layer) into the appropriate representation specified via HTTP content negotiation.

• There is a need to create and append new Audience resources to the collection of Audience resources.

• Or as Mike Amundsen calls it, "billboard" URI.
• Essentially, need the root node in the graph of application state.
• Could potentially include the following in the representation:
  • Link to notification collection.
  • Link to target collection.
  • Link to audience collection.
  • Action to create notification.
  • Action to create target.
  • Action to create audience.

...maybe?

• There is need to associate Target resources to Audience resources as members.
• The Target resource must exist before-hand.

Introduce All Representation Metadata

After running through RFC7231, I further clarified my understanding of what information is officially considered a "representation":

For the purposes of HTTP, a "representation" is information that is
intended to reflect a past, current, or desired state of a given
resource, in a format that can be readily communicated via the
protocol, and that consists of a set of representation metadata and a
potentially unbounded stream of representation data. - Section 3

More importantly, I was exposed to a more formal definition of "representation metadata". More specifically, the RFC explains that the following HTTP headers are used to communicate representation metadata:

As currently implemented, the resource metadata recorded for a representation already includes the media type and location (although as currently implemented, the "location" is always the request target URI - which could more abstractly be represented as the "content location" to communicate that a representation's location is not always the request target URI). I propose introducing representation encoding and language into the representation metadata set stored and utilized within the application for optimistic concurrency control and cache validation.

Representation Selection

As per the RFC definition of a representation (above), a representation "consists of a set of representation metadata and a potentially unbounded stream of representation data". In other words, the metadata is considered part of the representation itself. Due to this, these metadata fields (discussed above) are a necessity when selecting the correct representation during content negotiation.

Essentially, the representation location (used for representation identification), media type, encoding, and language serve as a composite key - two sets of representation metadata that differ in any one of these fields is considered a different representation altogether, as this metadata plays a role in defining the representation itself. To use an example, two representations communicating the same resource state but differ in target language audiences, are different representations of the current resource state. Similarly, a representation encoded using the gzip coding scheme is a different representation altogether than a representation encoded using the deflate coding scheme, even though they both communicate the same resource state.

Strong ETag Generation

Since tracking all of the representation metadata fields (discussed above) is pivotal for defining a representation, this metadata additionally has downstream impact on determining whether a representation's content has changed. When handling conditional requests, we must first be sure to complete proper representation selection to ensure that we are comparing the same representation.

RFC7232 provides insight on what a "strong validator" (strong ETag) is:

A "strong validator" is representation metadata that changes value
whenever a change occurs to the representation data that would be
observable in the payload body of a 200 (OK) response to GET. Section 2.1

There are a variety of strong validators used in practice. The best
are based on strict revision control, wherein each change to a
representation always results in a unique node name and revision
identifier being assigned before the representation is made
accessible to GET. A collision-resistant hash function applied to
the representation data is also sufficient if the data is available
prior to the response header fields being sent and the digest does
not need to be recalculated every time a validation request is
received. However, if a resource has distinct representations that
differ only in their metadata, such as might occur with content
negotiation over media types that happen to share the same data
format, then the origin server needs to incorporate additional
information in the validator to distinguish those representations. Section 2.1

Isolating ETag Generation From ETag Persistence

The application should be able to leverage the benefits of strong validators either with or without persistence of the validators. In other words, generating the validator should be decoupled from persisting the validator for optimization purposes. This allows the application to provide cache validation with or without the presence of an online persistence mechanism, which maintains the benefits of caching mechanisms even in the event validator storage is offline (perhaps due to software or hardware failure, data center fail over, etc.).This decoupling also better enables the diversification of storage mechanisms used, since there is no coupling between the generation of the validator and the means of which it is stored. This means that multiple methods of persistence can be utilized for application-defined optimizations (such as storing frequent validators in an in-memory persistence mechanism, while placing infrequent validators in a relational data store).

Isolating Resource State From Representation Metadata

It is tempting to embed timestamps recording when resource state was last updated, version numbers, etc. directly with the resource data itself. Besides implications induced due to violation separation of concerns (one concept is utilized for tracking changes to representations, the other for recording the current state of the resource; one is used primarily for caching purposes, while the other serves as the definition of state managed by the application; one is relatively disposable and serves mostly as an optimization, while the other is critical path in order for the application to fulfill is purpose), it also is abrasive in the effectiveness of the caching mechanisms achieved by introducing such metadata.

To provide the most effective caching mechanism, we should not assume that a change to resource state is a change to the representation of that state. Such an assumption should be considered a heavy-handed approach, as it will discard (recompute) strong validators when they could still be leveraged. Bear with me on this example:

Let's say we have a resource A with two representations, å and ∫. Resource A consists of given name and surname as its state. Representation å contains a field communicating the given name of resource A, but not surname. However, representation ∫ contains a field corresponding to resource A's surname, but does not include given name. We increment a version number or update a timestamp once either given name or surname of resource A changes.

If this timestamp or version number is utilized to calculate a strong validator for representations of resource A (such as representations å and ∫), than we will unnecessarily revoke validators when content of the representation doesn't change. In terms of the example, we will revoke the validators for both å and ∫ when just given name changes, even though the only validator that should recomputed is for representation å.

The solution is to not associate such timestamps or versioning schemes with resource data, but instead with representation data. This warrants isolating such data items from where resource data is stored and maintained, and instead maintain a separate storage for this representation data. Due to this, I encourage that the proposed additions of representation metadata be kept separate as currently implemented.

Description

When issuing the mvn site command to generate project and javadoc documentation, an issue is experienced when retrieving details about the project's dependencies. Here is an example of the console output encountered:

[INFO] Generating "Dependencies" report    --- maven-project-info-reports-plugin:2.9
[WARNING] The repository url 'http://packages.confluent.io/maven/' is invalid - Repository 'confluent' will be blacklisted.
[WARNING] The repository url 'https://flywaydb.org/repo' is invalid - Repository 'flyway-repo' will be blacklisted.
[WARNING] The repository url 's3://flyway-repo/release' is invalid - Repository 'flyway-repo-private' will be blacklisted.
[WARNING] The repository url 'http://repository.springsource.com/maven/bundles/external' is invalid - Repository 'spring-external' will be blacklisted.
[WARNING] The repository url 'http://nexus.codehaus.org/snapshots/' is invalid - Repository 'codehaus-snapshots' will be blacklisted.
[WARNING] The repository url 'https://github.com/tzolov/maven-repo/raw/master/' is invalid - Repository 'git-tzolov' will be blacklisted.
[ERROR] Unable to determine if resource antlr:antlr:jar:2.7.7:compile exists in http://maven.glassfish.org/content/groups/glassfish
[ERROR] Unable to determine if resource antlr:antlr:jar:2.7.7:compile exists in http://download.java.net/maven/2/
[ERROR] Unable to determine if resource antlr:antlr:jar:2.7.7:compile exists in http://oss.sonatype.org/content/repositories/jetty-snapshots
[ERROR] Unable to determine if resource antlr:antlr:jar:2.7.7:compile exists in http://download.java.net/maven/glassfish
[ERROR] Unable to determine if resource ch.qos.logback:logback-access:jar:1.2.3:compile exists in http://maven.glassfish.org/content/groups/glassfish
[ERROR] Unable to determine if resource ch.qos.logback:logback-access:jar:1.2.3:compile exists in http://download.java.net/maven/2/
[ERROR] Unable to determine if resource ch.qos.logback:logback-access:jar:1.2.3:compile exists in http://oss.sonatype.org/content/repositories/jetty-snapshots
[ERROR] Unable to determine if resource ch.qos.logback:logback-access:jar:1.2.3:compile exists in http://download.java.net/maven/glassfish
[ERROR] Unable to determine if resource ch.qos.logback:logback-classic:jar:1.2.3:compile exists in http://maven.glassfish.org/content/groups/glassfish
[ERROR] Unable to determine if resource ch.qos.logback:logback-classic:jar:1.2.3:compile exists in http://download.java.net/maven/2/
[ERROR] Unable to determine if resource ch.qos.logback:logback-classic:jar:1.2.3:compile exists in http://oss.sonatype.org/content/repositories/jetty-snapshots
[ERROR] Unable to determine if resource ch.qos.logback:logback-classic:jar:1.2.3:compile exists in http://download.java.net/maven/glassfish
[ERROR] Unable to determine if resource ch.qos.logback:logback-core:jar:1.2.3:compile exists in http://maven.glassfish.org/content/groups/glassfish
[ERROR] Unable to determine if resource ch.qos.logback:logback-core:jar:1.2.3:compile exists in http://download.java.net/maven/2/
[ERROR] Unable to determine if resource ch.qos.logback:logback-core:jar:1.2.3:compile exists in http://oss.sonatype.org/content/repositories/jetty-snapshots
[ERROR] Unable to determine if resource ch.qos.logback:logback-core:jar:1.2.3:compile exists in http://download.java.net/maven/glassfish
[ERROR] Unable to determine if resource com.101tec:zkclient:jar:0.10:compile exists in http://maven.glassfish.org/content/groups/glassfish
[ERROR] Unable to determine if resource com.101tec:zkclient:jar:0.10:compile exists in http://download.java.net/maven/2/

As a short-term mitigation, <dependencyLocationsEnabled>false</dependencyLocationsEnabled> is specified for maven-project-info-reports-plugin.

• Although already present for notifications, should also leverage for messages.
• Could enforce like so:
  • Desired status comes in via POST/PUT.
  • Apply all data changes specified (besides status).
  • Calculate status based on new state.
  • If there is a mismatch between calculated status and provided status, reject.
  • Otherwise, proceed.

• Need to have the ability to set the state of a Target resource.

• It would be an invaluable development asset to have curl configuration files for each request.
  • This would ultimately allow for quick and easy requests to be made to validate code changes and smoke test functionality.

• There is a need to completely delete an Audience resource.

Overview

• As currently implemented, notification messages are published to Kafka via SMSMessageQueue during the unit of work’s execution.
• It is desired that the messages are published after the unit of work has completed for the following reasons:
  • The database is changed at the conclusion of the unit of works execution. In other words, the database transaction is the database transaction is initiated, the changes applied, and the transaction committed until after all notification messages have been published. This causes unnecessary issues where the consumers of the messages in the SMS Queue act before the database transaction committed.
    • Instead, consumers should be confident that the information they receive upon consumption exists and is secondary, as consumption can be retried.
  • Even though the transaction is not initiated until after publishing has completed, the database connection is reserved for that unit of work throughout it’s lifecycle. This means that the connection is being held open longer than it needs to be, as the outcome of the interactions with the SMS Queue should not influence the creation of the notification.
    • For example, if there are issues with loading messages into the queue, those attempts can be retried.

• Need to have the ability to delete a Target resource.

• Currently, noti makes use of DriverManager.getConnection() directly.
• Ideally, we would leverage a connection pool.
• io.dropwizard.db contains types that allow for usage of connection pools.

For example:

• application/protobuf
• application/x-protobuf

How should I handle this?

• Include:
  • Noti
  • Noti Consumer
  • External APIs
  • Database
  • Kafka
  • Schema Registry
  • Load Balancer

• There is a need to remove members from an Audience resource.

• Can track the following metrics:
  • Open Connections.
  • Connection Duration.
• Introduce open tracing.

• There is a need to be able to retrieve a single Audience resource.

It would be beneficial to support another hypermedia-based representation format. HAL seems like a great candidate.

It should support:

• application/hal+json
• application/hal+xml

• To prevent old notifications from being sent in the event of a recovery of down streams, when the notification is potentially irrelevant.
  • Imagine getting a notification that your pizza arrived the next morning.

• Identified a database connection leak.
• Proposed solution is to have UnitOfWork extend AutoCloseable.

• Currently, the revision mechanism used to track changes to representations does not account for various states of the representation being changed external to a HTTP PUT request on the parent resource itself.
• For example, the revision is correctly incremented of a resource representation of an Audience resource when an HTTP PUT request is issued for that Audience resource. However, if a PUT request is issued for any of the sub-resource within that Audience resource, such as to change the state of one of its members, the representation revision would not be incremented, and thus, and new entity tag would not be generated.
• First guess: need to persist a dependency tree for these resource representations; traverse the tree and increment revision of all ancestor nodes.
• Worst case: fall back to utilizing content hashing; will need to utilize marshaling of concrete type.

There is a need to be able to trace through various aspects of the system to determine bottlenecks. It is also important to support tracing so that external clients that also utilize the Open Tracing standard can analyze traces across service boundaries.

• I've learned my lesson with phone numbers; better to get ahead with this one.
• When all said and down, a phone number should be represented like so (application/json for example):

{
  "countryCode" : "1",
  "number": "1231231234",
  "E164": "+1.1231231234"
}

• Currently, there no graceful handling of errors.
• Introduce Jersey 2.0 Exception Filters to intercept exceptions and return the appropriate HTTP responses.

• There is a need to define the state for a particular Audience resource.

• Look into potential benefits of introducing Identity Map.
  • Could cache object on read within transaction.
  • Could flush cache upon UnitOfWork#save() or UnifOfWork#undo() .
  • There are already spots where a read happens multiple times.

• The Spotify Docker Plugin looks promising.

...its time.

• At this current time, DriverManager.getConnection() is used to establish a connection to the MySQL storage. This solution does not optimize connections to the database.
• Instead, NOTI should leverage a connection pool to make more efficient use of database connections.

Overview

• When an incoming request is made, the Accept* headers are analyzed for proactive negotiation purposes.
  • The current design doesn’t gracefully handle an exact match for the Accept header when determining how to generate outgoing representations.

• Although it will take a relatively significant amount of involvement, there should be a health check determining whether Noti can successfully contact Kafka brokers.
• Based on my initial reading, it appears the following is the high level strategy:
  • Create a health topic.
  • Produce a message to this topic.
    • Could be an arbitrary string.
  • Consume the message.
    • If consumed successfully, mark as healthy.
    • If time limit expires, mark as unhealthy.

• The concept of a Tag was the first swing at providing a mechanism for grouping various Targets in a logical way.
• With the introduce of the Audience, there is no longer a need for the concept of Tag. Therefore it should be removed.