ghostzero / tmi-cluster Goto Github PK

View Code? Open in Web Editor NEW

17.0 3.0 5.0 1.41 MB

TMI Cluster for Twitch Chatbots

Home Page: https://tmiphp.com/docs/tmi-cluster

License: MIT License

PHP 86.40% Blade 3.64% JavaScript 0.90% Vue 6.33% SCSS 2.74%

laravel laravel-package php7 tmi tmi-cluster php php8

tmi-cluster's Introduction

TMI Cluster for Twitch Chatbots

Introduction

TMI Cluster enables a highly scalable IRC client cluster for Twitch. TMI Cluster consists of multiple supervisors that can be deployed on multiple hosts. The core is inspired by Horizon, which handles the complex IRC process management. It is designed to work within the Laravel ecosystem.

The cluster stores its data in the database and has a Redis Command Queue to send IRC commands and receive messages.

Features

Supervisors can be deployed on multiple servers
Up-to-date Twitch IRC Client written in PHP 8
Scalable message input/output queue
Advanced cluster status dashboard
Channel management and invite screen

PHP Twitch Messaging Interface

The TMI Cluster is powered by the PHP Twitch Messaging Interface client to communicate with Twitch. It's a full featured, high performance Twitch IRC client written in PHP 8.

Official Documentation

You can view our official documentation here.

tmi-cluster's People

Contributors

Stargazers

Watchers

Forkers

fossabot hartl3y94 siberfx

tmi-cluster's Issues

Start supervisor with average supervisor scale

It would be helpful if when starting a supervisor the current scale size of the cluster is directly taken into account. For example, if the initial scale is 5, or 15, but currently 10 is needed on average.

This would scale new supervisors directly in the correct size.

Proof of Concept - Multi-Tenancy

Motivation

I have now received feedback from several people that users should be able to change the bot username.

ETA: The plan is to start this proof of concept in Q2/Q3 2022.

Task

In the first prove of concept I would suggest to implement it like this:

The chabot will still enter the chat as a default bot account (eg. bot_username).
Messages will be sent with the specified bot account (eg. streamer_username).
By default the TMI-Cluster will still continue to use the default bot account.
It is only possible to define one bot account per channel.

Background of POC

Adding another bot account as a listener of chat messages puts a considerable load on the TMI cluster in terms of resources. Therefore, I suggest that the chatbot continues to join the channels with the current default bot account so that this problem is migrated.

In other words: Users only recognize that, for example, bot_username is in the chat, but will reply with streamer_username.

Updated Documentation

Sending Messages via TMI-Cluster

You can send a message within the laravel ecosystem at any time, you don't even need to have the TMI cluster connected in the target chat. The messages are processed async via the TMI cluster queue. You can define the bot_id attribute within the ChannelManager to override the message sender.

use GhostZero\TmiCluster\Facades\TmiCluster;

TmiCluster::sendMessage('ghostzero', 'Hello World!');

Current Documentation: https://tmiphp.com/docs/tmi-cluster.html#sending-messages-via-tmi-cluster

Create a example cli project that uses TMI Cluster

To better onboard new people, we could create a cli project based on laravel-zero, for example.

Influx Scraper not working

My plan is to create a InfluxDB Dashboard for TMI-Cluster.

I tried to define some typings, but influx still does not scrape metrics, so i need to dig deeper.

array_merge(): Argument #2 must be of type array, stdClass given

2-12-20 14:36:09] [d7eded7e] TMI Cluster Client getting evacuated with statu                                                                                                                    s code 5...
Channel Distributor: 0 locked channels got unlocked for re-join.
TMI Cluster Client evacuated! Migrated: 0
Got exit signal with code 5
[2022-12-20 14:36:09] [a300d9ec] Register cleanup loop for every 403 sec.
[2022-12-20 14:36:10] [d7eded7e] Register cleanup loop for every 646 sec.
[2022-12-20 14:36:14] array_merge(): Argument #2 must be of type array, stdClass                                                                                                                     given
[2022-12-20 14:36:14] #0 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Support                                                                                                                    /Arr.php(31): array_merge()
#1 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Repositories/RedisChannelDist                                                                                                                    ributor.php(54): GhostZero\TmiCluster\Support\Arr::unique()
#2 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Repositories/RedisChannelDist                                                                                                                    ributor.php(70): GhostZero\TmiCluster\Repositories\RedisChannelDistributor->join                                                                                                                    Now()
#3 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Repositories/SupervisorReposi                                                                                                                    tory.php(61): GhostZero\TmiCluster\Repositories\RedisChannelDistributor->flushSt                                                                                                                    ale()
#4 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/AutoScale.php(79): GhostZero\                                                                                                                    TmiCluster\Repositories\SupervisorRepository->flushStale()
#5 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/AutoScale.php(64): GhostZero\                                                                                                                    TmiCluster\AutoScale->releaseStaleSupervisors()
#6 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Supervisor.php(107): GhostZer                                                                                                                    o\TmiCluster\AutoScale->scale()
#7 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Supervisor.php(78): GhostZero                                                                                                                    \TmiCluster\Supervisor->autoScale()
#8 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Supervisor.php(53): GhostZero                                                                                                                    \TmiCluster\Supervisor->loop()
#9 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Commands/TmiClusterCommand.ph                                                                                                                    p(90): GhostZero\TmiCluster\Supervisor->monitor()
#10 /var/www/waifu/vendor/ghostzero/tmi-cluster/src/Commands/TmiClusterCommand.p                                                                                                                    hp(57): GhostZero\TmiCluster\Commands\TmiClusterCommand->start()
#11 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Container/BoundMethod                                                                                                                    .php(36): GhostZero\TmiCluster\Commands\TmiClusterCommand->handle()
#12 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Container/Util.php(40                                                                                                                    ): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#13 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Container/BoundMethod                                                                                                                    .php(93): Illuminate\Container\Util::unwrapIfClosure()
#14 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Container/BoundMethod                                                                                                                    .php(37): Illuminate\Container\BoundMethod::callBoundMethod()
#15 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Container/Container.p                                                                                                                    hp(653): Illuminate\Container\BoundMethod::call()
#16 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Console/Command.php(1                                                                                                                    36): Illuminate\Container\Container->call()
#17 /var/www/waifu/vendor/symfony/console/Command/Command.php(298): Illuminate\C                                                                                                                    onsole\Command->execute()
#18 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Console/Command.php(1                                                                                                                    21): Symfony\Component\Console\Command\Command->run()
#19 /var/www/waifu/vendor/symfony/console/Application.php(1015): Illuminate\Cons                                                                                                                    ole\Command->run()
#20 /var/www/waifu/vendor/symfony/console/Application.php(299): Symfony\Componen                                                                                                                    t\Console\Application->doRunCommand()
#21 /var/www/waifu/vendor/symfony/console/Application.php(171): Symfony\Componen                                                                                                                    t\Console\Application->doRun()
#22 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Console/Application.p                                                                                                                    hp(94): Symfony\Component\Console\Application->run()
#23 /var/www/waifu/vendor/laravel/framework/src/Illuminate/Foundation/Console/Ke                                                                                                                    rnel.php(129): Illuminate\Console\Application->run()
#24 /var/www/waifu/artisan(37): Illuminate\Foundation\Console\Kernel->handle()
#25 {main}
[2022-12-20 14:36:29] Reconnecting 654 disconnected channels...
[2022-12-20 14:36:36] Scale out: 10
[2022-12-20 14:36:37] Scale out: 11
[2022-12-20 14:36:37] [edf2b530] Register cleanup loop for

Load balance channels per supervisor

Currently there is no load balancing at supervisor level. However, this would prevent too many channels on one process.

The join process also enters some channels twice. This must be prevented Then you can also switch off the auto-cleanup (which ensures that 24/7 channels are sometimes removed).

Improve the channel join

Currently there is a small issue, when a supervisor goes offline and all channels need to be migrated.

The PR is already work in progress.

Catch fatal errors, and migrate channels

Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388616 bytes)

[2022-10-18 19:39:38] [94ca50d9] Register cleanup loop for every 530 sec.
[2022-10-18 19:39:40] [94ca50d9] TMI Cluster Client getting evacuated with status code 5...
Channel Distributor: 0 locked channels got unlocked for re-join.
TMI Cluster Client evacuated! Migrated: 0
Got exit signal with code 5
[2022-10-18 19:39:46] Unable to acquire flush stale lock...
[2022-10-18 19:39:50] Reconnecting 93212 disconnected channels...
[2022-10-18 19:40:22] #0 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(719): Illuminate\Database\Connection->runQueryCallback('update `tmi_clu...', Array, Object(Closure))
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(576): Illuminate\Database\Connection->run('update `tmi_clu...', Array, Object(Closure))
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(509): Illuminate\Database\Connection->affectingStatement('update `tmi_clu...', Array)
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3281): Illuminate\Database\Connection->update('update `tmi_clu...', Array)
#4 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(999): Illuminate\Database\Query\Builder->update(Array)
#5 /var/www/html/vendor/ghostzero/tmi-cluster/src/Repositories/DatabaseChannelManager.php(61): Illuminate\Database\Eloquent\Builder->update(Array)
#6 /var/www/html/vendor/ghostzero/tmi-cluster/src/Repositories/RedisChannelDistributor.php(56): GhostZero\TmiCluster\Repositories\DatabaseChannelManager->acknowledged(Array)
#7 /var/www/html/vendor/ghostzero/tmi-cluster/src/Repositories/RedisChannelDistributor.php(70): GhostZero\TmiCluster\Repositories\RedisChannelDistributor->joinNow(Array, Array, false)
#8 /var/www/html/vendor/ghostzero/tmi-cluster/src/Repositories/SupervisorRepository.php(61): GhostZero\TmiCluster\Repositories\RedisChannelDistributor->flushStale(Array, Array)
#9 /var/www/html/vendor/ghostzero/tmi-cluster/src/AutoScale.php(78): GhostZero\TmiCluster\Repositories\SupervisorRepository->flushStale()
#10 /var/www/html/vendor/ghostzero/tmi-cluster/src/AutoScale.php(64): GhostZero\TmiCluster\AutoScale->releaseStaleSupervisors(Object(GhostZero\TmiCluster\Supervisor))
#11 /var/www/html/vendor/ghostzero/tmi-cluster/src/Supervisor.php(107): GhostZero\TmiCluster\AutoScale->scale(Object(GhostZero\TmiCluster\Supervisor))
#12 /var/www/html/vendor/ghostzero/tmi-cluster/src/Supervisor.php(78): GhostZero\TmiCluster\Supervisor->autoScale()
#13 /var/www/html/vendor/ghostzero/tmi-cluster/src/Supervisor.php(53): GhostZero\TmiCluster\Supervisor->loop()
#14 /var/www/html/vendor/ghostzero/tmi-cluster/src/Commands/TmiClusterCommand.php(90): GhostZero\TmiCluster\Supervisor->monitor()
#15 /var/www/html/vendor/ghostzero/tmi-cluster/src/Commands/TmiClusterCommand.php(57): GhostZero\TmiCluster\Commands\TmiClusterCommand->start(Object(GhostZero\TmiCluster\Supervisor))
#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): GhostZero\TmiCluster\Commands\TmiClusterCommand->handle()
#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\Container\Container->call(Array)
#22 /var/www/html/vendor/symfony/console/Command/Command.php(308): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#24 /var/www/html/vendor/symfony/console/Application.php(998): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /var/www/html/vendor/symfony/console/Application.php(299): Symfony\Component\Console\Application->doRunCommand(Object(GhostZero\TmiCluster\Commands\TmiClusterCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /var/www/html/vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Application.php(102): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#28 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#29 /var/www/html/artisan(37): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#30 {main}

In Arr.php line 565:
                                                                               
  Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388616 bytes)

Slow query

create index tmi_cluster_supervisors_deleted_at_index
    on tmi_cluster_supervisors (deleted_at);

Scaling notification

PHP 8.1 - Implement serialize() and unserialize()

TMI Cluster: 3.0.8
PHP: 8.1

LOGS

[2022-11-18 00:07:41] [51273e78] PHP Deprecated: GhostZero\Tmi\Events\Twitch\NoticeEvent implements the Serializable interface, which is deprecated. Implement __serialize() and __unserialize() instead (or in addition, if support for old PHP versions is necessary) in /home/forge/example.com/vendor/ghostzero/tmi/src/Events/Twitch/NoticeEvent.php on line 8
PHP Deprecated: GhostZero\Tmi\Events\Twitch\ModsEvent implements the Serializable interface, which is deprecated. Implement __serialize() and __unserialize() instead (or in addition, if support for old PHP versions is necessary) in /home/forge/example.com/vendor/ghostzero/tmi/src/Events/Twitch/ModsEvent.php on line 8

[2022-11-18 00:11:22] [51273e78] PHP Deprecated: GhostZero\Tmi\Events\Irc\PingEvent implements the Serializable interface, which is deprecated. Implement __serialize() and __unserialize() instead (or in addition, if support for old PHP versions is necessary) in /home/forge/example.com/vendor/ghostzero/tmi/src/Events/Irc/PingEvent.php on line 14

[2022-11-18 00:16:13] [51273e78] PHP Deprecated: GhostZero\Tmi\Events\Irc\PartEvent implements the Serializable interface, which is deprecated. Implement __serialize() and __unserialize() instead (or in addition, if support for old PHP versions is necessary) in /home/forge/example.com/vendor/ghostzero/tmi/src/Events/Irc/PartEvent.php on line 12

Poll dashboard instead of full page refresh

Currently the page refresh every 10 seconds. This leads to some UX fails and is not so fancy on big clusters.

Graceful shutdown

Currently the irc processes do not gracefully shutdown. This means, that all current connected channels will not be migrated to other instances within a low time-frame. This means, all channels will be only collected if our purge system is collecting stale processes and supervisors. But this will take ~30 seconds.

Install documentation

Currently, there is no documentation how to install and us the TMI cluster. This can be resolved with an getting started within the readme or wiki.

Explain how to install
Explain how to use supervisors (start, stop, supervisor evacuation)
Explain how to use the TMI Event's

ghostzero / tmi-cluster Goto Github PK

tmi-cluster's Introduction

TMI Cluster for Twitch Chatbots

Introduction

Features

PHP Twitch Messaging Interface

Official Documentation

tmi-cluster's People

Contributors

Stargazers

Watchers

Forkers

tmi-cluster's Issues

Motivation

Task

Background of POC

Updated Documentation

Sending Messages via TMI-Cluster

Recommend Projects

Recommend Topics

Recommend Org