oneirocom / magick Goto Github PK

Magick is a cutting-edge toolkit for a new kind of AI builder. Make Magick with us!

License: Other

TypeScript 94.19% HTML 0.16% CSS 1.25% JavaScript 3.38% SCSS 0.66% Shell 0.07% Procfile 0.01% PLpgSQL 0.02% Dockerfile 0.16% Handlebars 0.11%

ai blueprints-visual-scripting fine-tuning gpt-3 gpt-3-prompts large-language-models no-code node-editor prompt-engineering text-generation

magick's Introduction

WARNING! This repository is undergoing a massive upgrade. Our readme is outdated and our local deploy process is being migrated. Check back soon for the new and improved Magick!!

Magick is a groundbreaking visual AIDE (Artificial Intelligence Development Environment) for no-code data pipelines and multimodal agents. Magick can connect to other services and comes with nodes and templates well-suited for intelligent agents, chatbots, complex reasoning systems and realistic characters.

🗝 Key Features

Realtime agents which can perform actions on their own, interact with users and other agents in different modalities with a unified memory and self
Social connectors to Discord, Twitter and Twilio -- Zoom, Google Meet, Reddit, Slack connectors will be available soon as plugins!
Search Google, Wikipedia and the Semantic Web
Many included powertools, including voice and image generation and vector search
Powerful graph-based IDE for complex data pipelines
Graphs can be embedded in subgraphs and shared for rapid community development.

🔮 Magick: Automate, Build, and Bring Your Ideas to Life with AI.

With Magick, you can unleash the power of AI without needing to know how to code everytbing in a command line interface. Using our visual node based environment, you can connect to popular services and explore a world of pre-built nodes and connectors to bring your vision to life.

▶️ Click the image above to watch the video

Core Concepts

Spells

A spell is a pipeline that describes data moving from one place to another, running through different processes we call "nodes", via wires we call "connections". In Magick, the collection of data, nodes, variables, and presets for each graph is known as a "spell". Spell is not a machine learning term.

We just like it. Spells can be imported and exported at any time. Spells in their raw form are JSON, a standard format that is easy to share.

Nodes

At the core, Magick is a system for taking in data, doing stuff to it, and then sending the final data out.

This "stuff" is called a "transformation", the data transforms from one thing into the next. The "stuff" that is happening to the data is a black box that takes something in, anything, and returns something out.

We call the black box where the transformstion takes place a "node". Nodes are the building blocks of Magick.

Creating Nodes

Nodes are created in the composer window of the "Spells" tab. You can right-click in the composer and add nodes from the context menu. You can also drag and drop nodes from the "Nodes" tab into the composer. Nodes can be dragged and dropped around the composer to re-arrange them.

Nodes can be deleted by right-clicking on them and selecting "Delete".

Node Types

Input Node
Prompt Template
Code Node: Lets you define both inputs and outputs.
Generator Node: Lets you define your own input sockets and then work with that data inside of the node.
Wait For All Node: Used to wait for different execution branches to complete before joining back into a single branch -- this is a good way to do several slow tasks at once.
...and lots more! (if you have suggestions for any, do let us know)

Inputs and Outputs

All nodes have some inputs and/or outputs, although they don't necessarily have to have both. Inputs and outputs are visually displayed as sockets. The color of the socket determines the type of data it can receive, with "gray" being the default untyped or "any" type.

Sockets

Data passed into sockets is available to the node. It can process that data, do something to it, and return the result to the output socket. Some nodes (like Generator node) let you define your own input sockets and then work with that data inside of the node. Some nodes (like the Code node) let you define both inputs and outputs.

Triggers

Triggers tell nodes to start asynchronous tasks. Some nodes can process data without needing a trigger, but most nodes need triggers. Triggers can be emitted from one socket out to more than one input, however the order of execution is not guaranteed. You can use the "Wait For All" node to wait for different execution branches to complete before joining back into a single branch -- this is a good way to do several slow tasks at once

⚙️ Installation

📚 Documentation / Guide

Prerequisites

Before you start, ensure you have the following software installed on your machine:

git: Version control system, required for cloning the repository.
- Download git
node.js 18+: JavaScript runtime, needed for running the application.
- Download Node.js
Docker: Enables you to run the project within containers for easier setup and distribution.
- Download Docker Desktop

Follow the respective installation guides to set up each piece of software.

Install pipx: A tool for installing and running Python applications in isolated environments.

On Unix and macOS: Open a terminal and run the following command:

python3 -m pip install --user pipx
python3 -m pipx ensurepath

On Windows: Open a command prompt and execute:

py -m pip install --user pipx
py -m pipx ensurepath

Install Poetry (Manages python packages)

pipx install poetry

Verify Installation

pipx --version

Once installed, proceed to set up Magick.

Setup

Clone and set up Magick

git clone https://github.com/Oneirocom/Magick
cd Magick
npm install
poetry install --no-root
npm run dev

Run Magick!

npm run dev

Note: Installation is automatic. Most Node projects require npm install - With Magick, dependencies will automatically be installed for you. Linux users may need to enter sudo password to install some dependencies.

Build

Build will take some time initially. When everything is ready, the client will be ready at localhost:4200

Please be aware Magick is under heavy development which may cause breaking changes.

Database

Magick installs postgres by default. If you want to use a different database, you can change the database connection in the .env file

Webhooks

Some connectors (like Github) require webhooks. In development, Magick uses ngrok to create a tunnel to your local machine. You will need to create an account and add your auth token to the .env file

Deploy your own Postgres database

To deploy your own database, we suggest using Supabase or another Postgres database. The current setup for events and documents requires the pgvector extension to be enabled.

The following documents should help you with setup:

Initialize a new database

Magick uses Feathers 5 for backend, which in turn uses Knex for making database queries. We will offer a better database configuration experience in the future. For now, you will need to manually configure the database connection in the .env file and then run the migration script.

cd apps/server
npm run migrate

Self signed certificates

Developing locally, it can be very helpful to have google chrome accept all self signed cetificates coming from localhost. To do this, simply paste the following snippet into chromes URL bar and enable the feature:

chrome://flags/#allow-insecure-localhost

Contributors

_Michael 💻 📦 💼	_{M̵̞̗̝̼̅̏̎͝Ȯ̴̝̻̊̃̋̀Õ̷̼͋N̸̩̿͜ ̶̜̠̹̼̩͒} 💻 📦	_alextitonis 💻	_{Daniel Adams} 💻	_{Anthony Burchell} 💻 📝	_jin 📝	_{Shivaditya Shivganesh} 💻
_khalildh 💻	_Pigloo 💻	_{avirtualfuture} 💻 📖	_{Alexandre Ferreira} 💻	_{Matei Anghel} 💻	_Trentalantar 📖	_{Lisa Mega Watts} 📖
_r2 📖	_{IRANKUNDA Fabrice} 💻	_{Kien Nguyen} 💻	_{Ben Moore} 💻	_Q 💻	_{thomageanderson} 💻	_Pietro 💻
_codingsh 💻	_{Ben Botwin} 💻 📦	_{K. S. Ernest (iFire) Lee} 💻	_{coffeeorgreentea} 💻 🚇 💼	_Knar 💻	_{Emmy NSABIMANA} 💻

Based on Thoth by Latitude Games

The original source code was made by the following contributors to Thoth, which is archived here

_Jakob
💻

_{Sean Gillespie}
💻

_Michael
💻

_{Preston Gull}
💻🎨

_{Nick Walton}
💻

_{Mitchell Gordon}
💻

_?/janus
💻

_{Alan Walton}
💻

Come Build With Us!

We are building Magick as a community in the open on Discord. If you are interested in contributing to the project, we'd love to have you.

Join us here: https://bit.ly/magickdiscordgh

magick's People

Contributors

Stargazers

Watchers

Forkers

msub2 developerfred lalalune rishav1707 sshivaditya2019 hozzjss darknexusdev khalildh thomageanderson davincidreams alexjorgef avirtualfuture tensornetics unicorndev99 atlasfoundation antpb belopot fabriceirankunda bizzyvinci joukosalonen abdoiiii cosmiccanvasexplorer mul1sh peponpylon blkluv mancingcrypto iheanyi1 reymlgz stackedq git19112019 darkdrag00nv2 0xpetra zerefwth pigloo qepri annias pizzooid antech426 abaso007 szir777 djmondoent alcamaster1290 carmosss blasbot fire liquidinfinity ai-innovation benbot ai-alebrijecircus-x vloss3 syaikhipin humandesign01 dglazkov mustamusti plurigrid mivanovitch assafei 00mjk samirmahmudzade fibelius gsspdev osbarcelos79 jfontestad cipherproxy connally82 jensinjames ihc523 ruofeidu irishwarlock89 coffeeorgreentea itsharex dwtcourses veryvanya max-bot1 mexicanamerican rjphnx kustomzone rossman22590 scottcertain tobiasweede ryvix jeffara xkkjiayou wanghaisheng emmyn5600 everblake cr-tech-repo imacoduh vital121 ccc0168 keyzf huangyingting brunoscaglione samk13 mrmetaverse codeaudit sdozono antonioevans rachfop geauxspecialist

magick's Issues

Avatar window is missing from available windows to open

Avatar window is open by default, but not available from the menu

Multimodal Graph

We want to be able to receive on input via REST API and then write into a Discord channel. This will require multi modality. If we have this as a standard interface then we can add plugins and connectors to it easily.

Weaviate Neural Search Demo

We use Weaviate for neural search, and have a search running. Demonstrate long term memory using Weaviate by recalling stored events from database, creating vector embeddings and storing them into a weaviate store, then recalling facts and relevant conversation based on input. This should all be done in a Magick spell or set of spells so that agents can automatically add to their long term memory and recall from it.

https://app.dework.xyz/m3/ai-43567/view/board-lblry0sg?taskId=551855dd-ae87-4b66-8f86-826ae12bff84

Requirements:

new weaviate nodes for storage and retrieval
Weaviate schema that mirrors events
Weaviate endpoints on server
retrieval nodes could be divided into different types of vector retrieval. And minimum a semantic search, and question answer search.

Move avatar window into plugin-avatar

We want the avatar and all avatar core code out.

Make a new plugin, duplicating the Fine Tune plugin and copying the routes everywhere
Move all avatar code into plugin window

Entity server crashes

Whenever an error occurs in a node on a graph, it crashes the whole entity server. We need much better error reporting and handling on the entity server so this doesn't happen.

Move ethereum to plugin abstraction

Move all ethereum nodes into @magickml/plugins-ethereum

Move values for ethereum key to a component that is injected into the agent manager

Save spell isn't working

When I go to save a spell, I am unable to, because it's not doing or saying anything

Python Integration

Note, this has been claimed by volit on Discord, who has until Jan 10 to report results. Anyone else may discuss with him to speed it up.
https://app.dework.xyz/m3/ai-43567/view/board-lblry0sg?taskId=02418aba-715b-4255-9524-87b9c2e7c1c2

Change comment from "undefined" to "right click to edit"

Right now the comment plugin makes a comment that is undefined. Annoying. Its should should "right click to edit"

Add blog contributors

Typing "Shift+F" will create a comment box on the graph even if you don't have it focused. Same with Shift+C

For example, naming a node "Agent Facts" will create a comment box as I'm typing

Add Contributors

Add alex as a contributor for past code contributions

Add "rename" option to edit menu

We should be able to rename a spell without having to "save as" and delete the other, kind of annoying

Async responses

Right now the whole graph has to complete before returning.

We need to be able to output and trigger responses asynchronously to the graph completing.

For one, this will enable multiple outputs.

Also, this will enable us to run memory tasks etc after responding, which should dramatically improve response time.

This is a prerequisite to multimodal graphs with inputs and outputs from other sources.

Replace JSON with stringified JSON in database, make default database local sqlite

We want to remove the docker dependency so that users can quickstart without it, using a local sqlite instance. However, our current models use JSON. We can dynamically set which we startup with, but only if the models use the same supported features.

Replace JSON with stringified JSON in models
Make sure calls are creating JSON from the returned text
Add default to sqlite in model and .env
Update README

Replace Google STT with Whisper

We tried this with C++, was hard.

A better solution is to run a server on banana.dev

We currently use Google Speech to Text to figure out what people are saying, but Whisper is better quality and can be run locally. A C++ version is here, which can be ported to WASM as well:https://github.com/ggerganov/whisper.cpp

https://app.dework.xyz/m3/ai-43567/view/board-lblry0sg?taskId=0e1e9ec5-a406-47bc-ac26-0b169dc48e79

Clean http/https handling

Currently, users are getting stuck starting up because of invalid self-signed certificates.

We should either default to http and unsigned (which may have ramifications for some of our services and pipelines) or solve our certificate stuff and just have everything clean and working so https works everywhere locally and http works when users are behind tls

Provide channel type to Agent from Discord

We want to present the channel ID and channel type (msg vs DM), this is mostly hooked up but not provided throught Discord adapter so it is undefined in the agent type.

Add import to main menu / move to top level menu

Right now you have to go to "open" to import, import should just be in the file menu

Connector simulator

Playtesting with the conversational VRM or with the playtest window for agents is a pain, as we have to add a bunch of raw config to the "data" dropdown, which is just a temporary hack to pass more complicated data out.

It would be awesome if we had a dropdown with preset data for different connectors (discord, twitter, etc) which would make it really easy to test each of them as the data comes through the pipe.

Jupyter Notebook Integration

Allow users to call a running Notebook from their Jupyter server by name (ideally validated via dropdown menu)

Bounty here
https://app.dework.xyz/m3/ai-43567?taskId=7b7d8230-9e9f-4add-a090-06781768cf30

Improve UI of inspector components

Currently all the components in the inspector are poorly spaced and clunky. Here is a rough design I did up that I think would improve it a lot. Eeach one of the data Control components could probably be improved and polished more. Ignore the coloring in this design and just use the global primary color for the borders, etc.

(feathers) make sure plugins only register once, even if hot reloaded client

It seems like we might have some double reloading issues from hotreloading. Please review logs and ensure that when saving a file in client or server, or when reloading the client, plugins are not hot reloaded twice.

the goal is to make sure that our hotreloading is robust to any strangeness that may have arison from recent changes.

Fix padding / margin on inspector

The inspector padding and margin are bad

If an input has a default value of 0 it's rendered empty

 
builder(node: MagickNode) {
    const out = new Rete.Output('output', 'output', numSocket)
    const dataInput = new Rete.Input('trigger', 'Trigger', triggerSocket, true)

    const _Min = new NumberControl({
      dataKey: '_Min',
      name: 'Value min',
      icon: 'moon',
    })

    const _Max = new NumberControl({
      dataKey: '_Max',
      name: 'Value max',
      icon: 'moon',
    })

    const name = new InputControl({
      dataKey: 'name',
      name: 'Name',
      icon: 'moon',
    })

   
    node.data._Max = 100
    node.data._Min = 0
    
    node.inspector.add(name).add(_Min).add(_Max)   
    return node.addOutput(out).addInput(dataInput)
  }

Fix autosave

Autosave is currently disabled. There is a tendency, though hard to reproduce, when the autosave will case a whole spell to be wiped. It is likely on a spell reload event when the spell is empty for a brief moment, and that is registered as a diff.

Electron app

Package app into stand alone electron for easy download and experimentation. Will internally use sqlLite. Implement with Capacitor.

Broken Link.

In readme.md file there is a broken link below screenshots. It is an all contributors badge which is not shown up there.

Screenshots:

(feathers) Ensure Discord bot closes and starts back up

Currently, Discord bot can hold its connection and receive messages twice. We need to make sure that agents die completely when reset, and that Discord connection is properly handled.

Agent should never respond twice, basically. If the root spell is saved, or the front end is reset, or anything happens that could trigger a reset (hot reload), we want to make sure there are no issues.

** UPDATE:

I have added some logic to isolate the double-agents. However, if the discord bot crashes, we need to make sure it dies and restarts successfully.

Variable nodes are easily corrupted

If I clone a string or prompt node it can break the whole app, destroy all my strings, etc

Move all search to plugin abstraction

Move node.display into socket plugin

moon: hmm weird bug we're noticing, where node.display is not a function, might be related to worker fixes
parzvial: Oh yeah. We would need to remove those. On server when worker runs node.display isn’t a function since it is running on the server.
I could move the display functionality into the socket plugin when the node received the socket message from the servers worker.

Add projects

Plugin system for connectors

Connectors currently are all different and loaded all together in the entity. We need a proposal and implementation of a module system for the connectors. A clean API that can be generalized to all our connectors. We should start with migrating over a single connector as a trial. Let's start with discord.

Alphabetize node menu

Currently, the node menu categories are in a state of disarray. It would be good to alphabetize it so it is easier to navigate.

Current dropdown:

Move search corpus to side window

If a component has a name, show it

Some components can be named - for example variables

We removed this functionality because it was changing the name of the actual component and breaking things.

What we want is to show the node name on the node, if it exists

This is the current

We want more like this:

Automatic sockets for inspector controls

Data controls should be able to create a socket on the component which it is registered. This will mean that all inspector components also have sockets that can be used to override values. This creates convenience and ease of use.

Persistent layout

Layouts right now reset after reloading. These could be stored in redux for persistence between sessions. Bonus for saving multiple layouts for recall later.

import/export project

A project for now will be considered all the spells in a person's workspace, since everything currently is local and personal. A project will be all the spells in the DB, along with a file spellbook.json or similar which is at the root. The spellbook will list the name of all the spells. All spells, plus this spellbook.json, will be zipped and downloaded.

For import, someone selects the spellbook.json and all the spells, and uploads them. We read the spellbook and then load all the spells in.

note: spellbook may not be needed for this. But there may be metadata we want to associate anyways and it may be good to set the pattern now.

Add self to contributors

Move agent window to drawer

The side drawer should be populated with functionality that pertains to a whole project, or whose UX would be really bad inside of a single window in the spell composer.

Create a test suite of Node tests to make Magick robust.

We should really have a handful of core tests that ensure essential functionality is working. Subspells. Spell runner. Core components.

This might be a single robust spell with subspells that runs on PR that tests almost everything. Though AI services are mocked. Interface makes this easy to do.

We should be able to ensure that all basic functionality for now is secure once main branch is stable. We can expand to more robust and granular tests after.

Speaker / sender not stored properly in db

Test contribution add

@all-contributors please add @michaelsharpe for code and platform

Delete does not get registered

The delete/backspace does not remove the nodes

Public variables

Public variable nodes. Should have different types. Could possibly just make our current variables exposable as public. A public variable can be inject into the spell. So either as data from outside or as sockets into a spell. Used to create sets of data for different entities to share the same brain/spell.

Need to consider whether public variables can be populated from the top level spell all the way down to subspells or not.

Replace all sequelize and pg calls with prisma
Replace postgres locally with prisma and sqlite
Add instructions and a community cluster for developing supabase

oneirocom / magick Goto Github PK

magick's Introduction

WARNING! This repository is undergoing a massive upgrade. Our readme is outdated and our local deploy process is being migrated. Check back soon for the new and improved Magick!!

🗝 Key Features

🔮 Magick: Automate, Build, and Bring Your Ideas to Life with AI.

Core Concepts

Spells

Nodes

Creating Nodes

Node Types

Inputs and Outputs

Sockets

Triggers

⚙️ Installation

📚 Documentation / Guide

Prerequisites

Setup

Run Magick!

Build

Database

Webhooks

Deploy your own Postgres database

Initialize a new database

Self signed certificates

Contributors

Based on Thoth by Latitude Games

Come Build With Us!

magick's People

Contributors

Stargazers

Watchers

Forkers

magick's Issues

Screenshots:

Recommend Projects

Recommend Topics

Recommend Org