sammyjo20 / laravel-haystack Goto Github PK

View Code? Open in Web Editor NEW

596.0 596.0 25.0 389 KB

⚡️ Supercharged job chains for Laravel

Home Page: https://docs.laravel-haystack.dev

License: MIT License

PHP 100.00%

chained-jobs database hacktoberfest jobs laravel php queue

laravel-haystack's Introduction

👋 Howdy. I'm Sam! 🤠

I build web apps for the financial industry and developer community. During the day I work at Plannr Technologies Ltd. We're building a revolutionary back-office system for the financial industry from the ground up

My Open Source Packages

🤠 Saloon

Saloon is tool to help you build better integrations with third party APIs. It provides you with an eloquent, standardised syntax that will scale with your team. With Saloon, you don't have to worry about the underlying HTTP Client like Guzzle and spend more time with the data that APIs give you.

You can learn more about Saloon by clicking here

👩‍🌾 Laravel Haystack

Laravel Haystack is a package that allows you to have a job chain stored in the database. Since all of the jobs in the chain are in the database, memory usage is low and you can delay jobs for a long time or have long running jobs without risking using all your memory. Laravel Haystack supports every queue connection/worker out of the box. (Database, Redis/Horizon, SQS).

You can learn more about Laravel Haystack by clicking here

🏇 Lasso

Lasso is your asset "wrangler" (see what I did there). It's built for Laravel, and it automatically compiles your Webpack/Laravel Mix assets and uploads them to a Filesystem of your choice. It will then download those assets on all your web servers.

You can learn more about Lasso on the official website: getlasso.dev

☸️ Wagonwheel

Wagonwheel is a simple Laravel package built for one sole purpose: To provide an online version of all your Laravel emails. It's fully customisable and really quick to implement.

You can learn more about Wagonwheel here.

My Holopins!

laravel-haystack's People

Contributors

Stargazers

Watchers

laravel-haystack's Issues

Feature - Add the ability to record attempts for long-waits

I've noticed a potential problem with the "longRelease" method. Since the job is serialized and stored in the queue, we don't record the number of attempts that have happened before its released again. I think this should be a relatively easy thing to fix.

Here's my idea for fixing it:

Add an "attempts" column to the haystack_bales table
When a job has been released using "longRelease", we increment the releases, similar how normal releasing works
When the job is loaded onto the queue next time, we manually set the attempts. We could do this by creating the PendingJob class first and then dispatching it.

Enhancement - Add a "paused" closure

It would be good if Haystack had a closure that was called every time a haystack was paused. This is useful if you want to update your own statuses based on paused state.

For example:

Haystack::build()
    ->addJob(new ProcessPodcast)
    ->paused(function ($data) {
        // Set $podcast->paused = true;
    })
    ->dispatch();

Append to haystack method does not use the global delay/queue/connection

I noticed a bug where if you tried using appendToHaystack, it won’t use the default delay/queue/connection. In order to fix this, I need to add this configuration to the haystack database instead of applying it to each job.

TypeError when retrieving array with getHaystackData()

Hi,

I'm saving an array with getHaystackData() in one job:

$this->setHaystackData('pendingTransactions', $pendingTransactions,  'array');

Then in another job in the chain, I retrieve the data like so:

$this->pendingTransactions = $this->getHaystackData('pendingTransactions');

That throws TypeError: json_decode(): Argument #1 ($json) must be of type string, resource given

Stack:

TypeError: json_decode(): Argument #1 ($json) must be of type string, resource given
#43 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(1233): json_decode
#42 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(1233): Illuminate\Database\Eloquent\Model::fromJson
#41 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(743): Illuminate\Database\Eloquent\Model::castAttribute
#40 /vendor/sammyjo20/laravel-haystack/src/Models/HaystackData.php(73): Sammyjo20\LaravelHaystack\Models\HaystackData::getValueAttribute
#39 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(631): Illuminate\Database\Eloquent\Model::mutateAttribute
#38 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(2031): Illuminate\Database\Eloquent\Model::transformModelValue
#37 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(462): Illuminate\Database\Eloquent\Model::getAttributeValue
#36 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php(441): Illuminate\Database\Eloquent\Model::getAttribute
#35 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(2120): Illuminate\Database\Eloquent\Model::__get
#34 /vendor/sammyjo20/laravel-haystack/src/Concerns/ManagesBales.php(342): Sammyjo20\LaravelHaystack\Models\Haystack::Sammyjo20\LaravelHaystack\Concerns\{closure}
#33 /vendor/laravel/framework/src/Illuminate/Collections/Collection.php(770): Illuminate\Support\Collection::mapWithKeys
#32 /vendor/laravel/framework/src/Illuminate/Database/Eloquent/Collection.php(379): Illuminate\Database\Eloquent\Collection::mapWithKeys
#31 /vendor/sammyjo20/laravel-haystack/src/Concerns/ManagesBales.php(343): Sammyjo20\LaravelHaystack\Models\Haystack::allData
#30 /vendor/sammyjo20/laravel-haystack/src/Concerns/ManagesBales.php(356): Sammyjo20\LaravelHaystack\Models\Haystack::conditionallyGetAllData
#29 /vendor/sammyjo20/laravel-haystack/src/Concerns/ManagesBales.php(187): Sammyjo20\LaravelHaystack\Models\Haystack::finish
#28 /vendor/sammyjo20/laravel-haystack/src/Concerns/ManagesBales.php(213): Sammyjo20\LaravelHaystack\Models\Haystack::fail
#27 /vendor/sammyjo20/laravel-haystack/src/JobEventListener.php(137): Sammyjo20\LaravelHaystack\JobEventListener::handleFailedJob
#26 /vendor/sammyjo20/laravel-haystack/src/LaravelHaystackServiceProvider.php(69): Sammyjo20\LaravelHaystack\LaravelHaystackServiceProvider::Sammyjo20\LaravelHaystack\{closure}
#25 /vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(421): Illuminate\Events\Dispatcher::Illuminate\Events\{closure}
#24 /vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(249): Illuminate\Events\Dispatcher::dispatch
#23 /vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(195): Illuminate\Queue\Jobs\Job::fail
#22 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(582): Illuminate\Queue\Worker::failJob
#21 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(528): Illuminate\Queue\Worker::markJobAsFailedIfWillExceedMaxAttempts
#20 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(452): Illuminate\Queue\Worker::handleJobException
#19 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(429): Illuminate\Queue\Worker::process
#18 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(375): Illuminate\Queue\Worker::runJob
#17 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(173): Illuminate\Queue\Worker::daemon
#16 /vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(150): Illuminate\Queue\Console\WorkCommand::runWorker
#15 /vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(134): Illuminate\Queue\Console\WorkCommand::handle
#14 /vendor/laravel/horizon/src/Console/WorkCommand.php(51): Laravel\Horizon\Console\WorkCommand::handle
#13 /vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
#12 /vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\Util::unwrapIfClosure
#11 /vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\BoundMethod::callBoundMethod
#10 /vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::call
#9 /vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\Container::call
#8 /vendor/laravel/framework/src/Illuminate/Console/Command.php(144): Illuminate\Console\Command::execute
#7 /vendor/symfony/console/Command/Command.php(308): Symfony\Component\Console\Command\Command::run
#6 /vendor/laravel/framework/src/Illuminate/Console/Command.php(126): Illuminate\Console\Command::run
#5 /vendor/symfony/console/Application.php(1002): Symfony\Component\Console\Application::doRunCommand
#4 /vendor/symfony/console/Application.php(299): Symfony\Component\Console\Application::doRun
#3 /vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application::run
#2 /vendor/laravel/framework/src/Illuminate/Console/Application.php(102): Illuminate\Console\Application::run
#1 /vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Foundation\Console\Kernel::handle

PHP 8.1, on latest Laravel and laravel-haystack, migrations were migrated.

Idea - Haystack "State"

I just thought of a great idea for a feature in Haystack. I could introduce a JSON column to the Haystack model for state that can be stored and passed down into every job. This could be useful if you're using haystack in a pipeline style where the next job relys on some data that is set in the state of a previous job.

// Job A:
$this->addHaystackData('name', 'Sam', 'laravel-cast')

$this->addHaystackData('data', ['foo' => 'bar'], 'array')

// Job B
$this->getHaystackData('name') // Sam

$this->getHaystackData('data') // ['foo' => 'bar']

Potential Bug - Exceptions Thrown Inside Jobs Don’t Finish The Haystack

I need to guarantee this is a bug, but it looks like we’re not finishing and running the “catch” method on the haystack if an exception is thrown inside of a job.

Feature - Initital delay

Currently, you can set a delay to happen on every job, however, there isn't the ability to have an initial delay. This could be a handy feature to have.

Laravel 11 Support

Hey @Sammyjo20 can you add support to Laravel 11?

Thanks!

HaystackBale serialization not working with PostgreSql

Laravel Version: v8.83.23
PHP Version: 8.1.12
Database Driver & Version: PostgreSQL 12.4

A serialized object in PHP that has null values before private/protected fields will fail to serialize because Postgres PDO serialization will stop on null values.
Example:
"O:1:"A":1:{s:8:"\0*\0field";N;}"

We can use base64_encode before inserting & base64_decode after fetching the serialized objects.

The laravel team had the same issue for batches feature
laravel/framework#36061

unserialize(): Error at offset 0 of 8760 bytes

when job implements ShouldBeEncrypted

Haystack pauses suddenly

i couldn't quite reproduce the issue yet ... but right now my db uses singlestore which has this form of ids

so after pushing 400 haystack_bales or so ... suddenly I found no jobs are queued and the haystack row is stale

also I found some queues inside haystack_bale with delay 0

my queue configuration is process_automatically is set to true

also when I run $haystack->dispatchNextJob() ... it process some jobs then stop

and inside my queue jobs ... no special thing just normal handle function and what was mentioned in the docs here

would appreciate if you can guide me or point me to a direction ... so far am suspecting this line could be the issue but not sure

my call to haystack looks like this

 Haystack::build()->withName("test")->addJob(x)->addJob(y)->withDelay(1)->allowsFailure()->dispatch();