superstreamlabs / memphis Goto Github PK
View Code? Open in Web Editor NEWMemphis.dev is a highly scalable and effortless data streaming platform
Home Page: https://docs.memphis.dev
License: Other
Memphis.dev is a highly scalable and effortless data streaming platform
Home Page: https://docs.memphis.dev
License: Other
Hey,
Please add the default username and password (root/memphis) to the print the CP prints when deployed using docker-compose.
Thanks!
Try to edit your avatar and then call to refreshToken -> the old avatarId return.
Unique, robust, dynamic, and easy-to-use schema store for Protobufs, Avro, and JSON based data formats.
Will be available to manage via the GUI, SDKs, and REST and with dedicated resources embedded in the broker itself without the need to deploy dedicated tools/system and add extra dependencies and management overhead.
A schema defines how a message or event should look and in which format, creating a contract between producer and consumer that the broker will enforce or be aware of.
Hey,
At the moment, memphis doesn't allow producers with the same name at the same time. We need to allow this feature.
Error keeps showing up even after upgrading the consumer SDK and restart it.
Station is stuck with infinite loop
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Is your feature request related to a problem? Please describe.
No
Describe the solution you'd like
At the moment, the DLS feature is enabled by default for every station and cannot be disabled.
We want to add the ability to enable/disable DLS in a certain station.
Get 500 when I try to create a new station.
details for recovery:
factory name: melvis factory
user: root
Can you stop your annoying e-mail spam? I don't care about whatever the project this is.
Hey Dear Contributors,
In the latest steering meeting, users raised a few requests to enable a quick deployment over AWS/GCP/DO,
including Kubernetes cluster with three nodes, Memphis cluster, and ingress for production usage.
The purpose behind it is to make Memphis even easier to deploy. Some devs or users in general are a bit backed down regarding k8s. We would love to lower that entrance barrier.
Referring Material -
https://www.digitalocean.com/community/tutorials/how-to-use-terraform-with-digitalocean
https://registry.terraform.io/providers/digitalocean/digitalocean/latest/docs/resources/kubernetes_cluster
https://registry.terraform.io/providers/hashicorp/aws/latest
https://registry.terraform.io/providers/hashicorp/google/latest
Terraform files should be able to -
If anything is missing or should be added, please comment.
Thank you ๐
Describe the bug
See title
To Reproduce
nats pub stationName.final message
Expected behavior
the "SDk" should be "SDK"
Create a terraform file to automatically deploy best practiced EKS with Memphis cluster
Remove deployment frictions of Memphis Broker and have a faster, best-practiced deployment over the popular clouds
One-click installation of Memphis
From:
08-05-2022 01:46:46 PM - fatal: Memphis - You have to connect with application type user
To:
Memphis - Please use a user of type "Application" and not "Management"
Describe the bug
A certain station that already exists keep saving the old IP address of a certain producers/consumers which I used their names but from an entirely different host with a different IP address.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
IP should be updated
We need to replace the name of volumes with a meaningful name.
From '89d900b16b4e9cf527b6e6e10dae96ff266ebea390cb208392a335717f191e3c...' To 'Memphis_data' for example.
Enhanced user management, including permissions per station/features, groups, roles
To enable potential users and organizations with branched teams and different personas that require a higher level of security, compliance, and granularity. In most brokers, ACL (Access control list) mechanism will be found. Memphis will enable the same granularity with ease-of-use
Better security features and compliance
Create a terraform file to automatically deploy best practiced DO Kubernetes cluster with Memphis cluster
Remove deployment frictions of Memphis Broker and have a faster, best-practiced deployment over the popular clouds
One-click installation of Memphis
Create a terraform file to automatically deploy best practiced GKE with Memphis cluster
Remove deployment frictions of Memphis Broker and have a faster, best-practiced deployment over the popular clouds
One-click installation of Memphis
Different schemas often occur when working with multiple data sources / different services / different pipelines that produce data on the same topic, while the consumers on the other side expect specific structure.
Usually, that job is distributed across different teams and business units and requires expensive sync in time and effort.
Upstream changes are often impossible.
For example, AWS Lambda cannot ingest JSON-based payload with a boolean value, but rather should be converted into string before.
Ability to perform deep, inline transformation of events structure at the broker level while in transit.
Functions like: Concat / trim / regex / sum / and more...
Would be great to get comments about expected functions
Reduce logic from producers and consumers, increase data quality, reduce downtime and crashes, and enable complex streaming pipelines for in-app purposes.
Memphis's core value is to build streaming use cases faster, with a low amount of code.
This feature is a milestone - building a modular framework, easy-to-modify, adjust, and scale connectors framework
that will eliminate the need to create consumers/producers that only collect and push data from/to popular sources.
Based on multiple requests by our users asking to have ready-to-use connectors to pull and push data from and directly to Memphis or out. The number of connectors cannot lean on Memphis team resources and should be built to enable users to quickly implement their type of connector in case they can't find the connector they need in the Hub.
Build fully operational streaming pipelines in a low-code manner
At first login, the response from the server takes a while.
Enable connectivity between Debezium to Memphis to enable CDC use cases.
Multiple streaming pipelines require support in CDC for different databases. Debezium gathers the most popular ones like MySQL, Postgre, and more in a single framework
Faster creation of CDC-based streaming pipelines
Released on v1.1 and can be found on both the integration center and detailed here as well - https://memphis.dev/blog/part-1-integrating-debezium-server-and-memphis-dev-for-streaming-change-data-capture-cdc-events/
The "Getting Started" model for new users is not attracting enough usage and should be redesigned.
Help new users onboard Memphis
Increase engagement and familiarity with Memphis features.
"Factory" object not making sense and adding a layer of complexity for no good reason.
Should be removed.
The initial thought was to add an extra level of management. After engaging with users, it turned out it's not being used in a way it makes sense for it to stay.
Reduce complexity
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Is your feature request related to a problem? Please describe.
No.
Describe the solution you'd like
To be able to create stations with names like "demo-1" / "demo.1"
Describe alternatives you've considered
Like rabbitmq routing keys
When using Stations with the Memory storage type and with a single replica, if we kill the broker and start it again the Station(stream) will perish. although the DB still holds metadata as if the Station exists.
Expected behavior
We should carefully design a solution here,
we can update the DB in this case, or not allow the Memory storage type at all...
I was looking through the modules and noticed NATS. Isn't NATS by itself claiming to be a better queue to kafka, how does this library add to that?
consumer code
const memphis = require('memphis-dev');
(async function () {
try {
await memphis.connect({
host: 'localhost',
username: 'root',
connectionToken: 'memphis'
});
const consumer = await memphis.consumer({
stationName: 'benchmark',
consumerName: 'yaniv_cons1',
consumerGroup: ''
});
consumer.on('message', (message) => {
console.log(message.getData().toString());
message.ack();
});
consumer.on('error', (error) => {
console.log(error);
});
} catch (ex) {
console.log(ex);
memphis.close();
}
})();
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.