QoS guarantees can be in the form of messages/sec or similar metric. Ensures that the tenant's streams/topics are not starved. Amazon Kinesis guarantees: One shard provides a capacity of 1MB/sec data input and 2MB/sec data output. One shard can support up to 1000 PUT records per second.

Ability to consume Events in Streams via S3, HDFS and NFS. Once events are persisted Tier 2, they should be accessible in read-only mode via S3, HDFS and NFS heads by other services such as MapReduce Job. This must be enforced by platform.

Multiple replicas of the same data is avoided.

HDFS is still a powerful paradigm for batch analytics. This helps in analyzing messages using traditional Hadoop frameworks like MapRed, Hive..

Multi-instance: controller should register itself with zookeeper

Pravega provides a durable transaction log for high-speed writes and tailing reads of data 'Streams'. The events in the streaming storage layer will 'tier' to a long term deep storage (like HDFS) through configurable policies.

It is desirable for Pravega to be able to handle varying throughput of streams elegantly. Rather than statically defining certain number of Streaming Node to handle particular Streams, Pravega should be able to adjust it's surface based on the throughput that Stream is currently experiencing.

Sub-tasks:

• Preliminary design: #378
• Design complete:
• Implementation: PR #444

• XOR-compaction across geos
• Strong-consistency guarantees
• Access during outage
• Compliance
• Cross-protocol access
• ...etc

Ability to define ACLs/permissions on Stream level, which apply to both Tier 1 and Tier 2 storage systems

Pravega should have the ability to handle geo-replicated Streams with reads and writes enabled on all replicated sites. We refer to this configuration as Active-Active ( Contrasts to an Active-Passive configuration where Streams are actively written to from only one Primary site. Others are read-only.

This req is part of #21

Note: Active-Passive should be included in EAP2 releases, not Active-Active.

The sandbox must be a simple downloadable environment with minimal prerequisites to run (e.g. VirtualBox / VMWare OAV). It should have the following features:

• Self contained with Pravega functionality enabled
• Production grade stability
• Example configuration pre-configured with realistic environment
  Including existing accounts, buckets, topics, sample data…

HDFS is an attractive option because of it's popularity and penetration. Pravega should support Hadoop compatible interface.

Ability to choose one (of many possible) pre-created Replication Groups for defining the geo-replication policy for each stream seperately. As part of #21 feature

A vast majority of organizations with Data Pipelines in production today use Kafka as their primary messaging cluster. To enable them easily to migrate to Pravega, it is essential to give them a Kafka compatible API on top of Streaming Storage so that they don't have to recode the production producers/consumers.

Pravega should be 100% compatible with Kafka 0.10.X API for clients (Producers/Consumers - Data plane)
Drop in replacement for existing Kafka clusters.
Reuse developer familiarity with Kafka APIs.
Leverage existing tools like Kafka Connect, Schema Registry, Mirror Maker

Some APIs might not be supportable (like getMetrics()). Those should be documented.

Events persisted in the system should be reliable/fault-tolerant. We do not want to provide an option of non-guaranteed (best-effort) delivery. Expectation is that the Streaming Storage layer persists reliably to disk before the Producer gets ACKd.

When tiered to the long-term HDFS or compatible Storage engine protection mechanisms are leveraged.

Messages are always available after being ACKd and maintain ordering guarantees of Streaming API.
Have not heard cust need possibly-lossy persistence. Cust scenarios rarely tolerate losses.

Some requests implicitly assume they will not be interleaved with others.
IE: The server implementation assumes that a client will not issue multiple setupAppend requests concurrently from the same connection, as it may become confused if this happens.
Note that the client as-written should not do this, but the server should be better protected.

Multi-instance: controllers should take action once a pravega node/controller is down

Operator needs to be able to define the streams deployment surface (no. of streaming nodes).

Ability to define/configure number of Streaming Nodes that will be available to service tenants' requests.

Ex: Out of 100 nodes, operator needs 25 instances for Pravega

Currently we have Travis in trial version which allows 100 builds. After that we need to pay for it. We should have some plan for it. Either emccode is paying for it OR we are :)

Apart from metrics/performance of Streams, it is useful to display Producer/Consumer metrics like:

• How many Producers are producing to any particular Stream
• How many messages have they attempted to add to the stream (but possibly failed due to timeout...etc)
• How many Consumer Groups are pulling from a particular stream
• How many consumers exist in each of those Consumer Groups
• How many messages have been consumed by a particular consumer group (are they all caught up? or lagging by 2000 messages)
• (Possibly) health of Producers/Consumer instances

These metrics would help gain insights like: "Stream OilRefinerySensor124 has 15 producers. Producer 5 is probably dead because it hasn't produced messages in a while. Consumer Group C1 and C2 are all caught-up. Consumer Group C2 is lagging by 2000 events."

Multi-instance: controllers should implement heartbeat mechanism to indicate availability

We have received feedback that it would be useful to have Partitions queried and identified by strings instead of IDs.
Stream names need to be unique within a Tenant. Between tenants, Stream names don't have to be unique.

Streaming Service enables consumption/read of a Stream from multiple sites via the Streaming API, HDFS and NFS APIs

Ability to store and manage schemas for messages to be shared between Producers/Consumers - Schema Registry
Schemas are expected to be used in the following ways:

• Pravega will validate messages against schema when they are received from producers
• Pravega will add metadata to each message referencing the appropriate schema
• Consumers will be able to reference schema to deserialize messages
• Index and Search capabilities will use schema to select fields for indexing
• Analytics capabilities will use schema to facilitate selecting relevant fields and deserializing them
• Gracefully propagating schema changes as they evolve

Ability to logically group multiple Streams which share same policies/configurations (aka Stream Groups)

Auto scaling will allow streams segments to scale up and down based on events throughput rate. From consumer perspective, to avoid starving while auto scaling happens, user should be able to specify minimum parallelism when creating a stream

To demonstrate the capabilities and advantages of Pravega, we need to publish sample applications. These will be provided to early users to try out the service. Some examples could be reading a Twitter firehose into a Stream and a consumer which drains it into a Spark or Flink cluster.

It is expected that the Pravega will utilize Mesos for its resource management, thus providing a true converged infrastructure experience. Will probably imply a 'Streaming' Mesos framework needs to be implemented.

Multi-instance: pravega nodes should implement heartbeat mechanism to indicate availability

We need exponential backoff and reconnect logic to be built in. The current code does not implement this properly.
This can be done with the threadPool of the connection:
private void doConnect() {
Bootstrap b = ...;
b.connect().addListener((ChannelFuture f) -> {
if (!f.isSuccess()) {
long nextRetryDelay = nextRetryDelay(...);
f.channel().eventLoop().schedule(nextRetryDelay, ..., () ->
{ doConnect(); }
); // or you can give up at some point by just doing nothing.
}
});
}
But this still requires a custom event to reset the clock (because we don't want to until it is working), and a message up the stack to notify of any reconnects.
However realistically, we probably want to handle this at a higher layer, because segments can move between hosts.

It can first be made available first in private universe and then public.

It is desirable to have some configurable policies which dictate how/when events move between the two tiers. Could be time-based/memory utilization based - for example.

TBD: Spec out details with customers

Predictable lag (ceil) between event landing in Tier1 and the time event lands in Tier 2

Tier 2 consistency mechanisdm should ensure the message is 'visible' to all readers across geos
This will ensure remote-reader will always have the latest messages written from other sites

This does not include any work for distributed log. We need to do this for Pravega controller and streaming service nodes. Pravega artifacts should be available on Universe for any dcos users can install.

Pravega should be install-able on public clouds (via template-styled tools like CloudFormation)

To establish Pravega as a credible streaming platform, we will need to publish and document a well-defined set of APIs which will let apps provision, monitor, configure and manage Pravega.

This will be distinct from the Kafka API which also will be supported in the future releases.

User-defined metadata attached to streams, which provides further information or categorization of the the stream. Custom metadata is formatted as key-value pairs that are set when creating a stream. E.g.

• Client = Dell EMC
• Event = Dell EMC World 2017
• Billing ID = 123
• ..

To be competitive with existing messaging infrastructures today (like Kafka), we should be able to provide write and read latency (tail and catch up reads) which are comparable or preferably better.

TBD: Exact numbers, end-to-end versus broker latency...etc

There is a expectation of logical isolation of streams/events/long term data storage between the tenants.
Each tenant can have same stream with name "Foo". Multi tenancy must be supported both in both tier 1 and tier 2 layers. The events put by a tenant should be isolated from other tenants on the same host.

Multi-instance: Pravega nodes should register themselves with zookeeper

To ensure rogue tenants are not abusing the Pravega. Admin should have the ability to define soft quotas which based of aggregate throughput seen by the Tenant's streams. If the soft quota is breached, alerts/notifications will help the tenant react.

Currently they throw an error if you try to serialize them.

The number of Pravega Nodes needed in the system might vary depending on the number of tenants/streams/segments. Operator needs the ability to scale the streaming surface up/down to maintain QoS guarantees and quotas.

Ability to geo-replicate events in Streams in Tier 2 storage

1. Stream content with its segments
2. Stream configuration, metadata and acl as well #22