Thanks for a very useful module!

Situation:
7000+ rows export on a high load machine

Problem:
QueuedJobService has a default memory limit of 256Mb. Their method runJob has a check isMemoryTooHigh. In our case, that limit was met. The idea behind the way QueuedJobService handles this situation seems to be to release memory and let the next iteration of ProcessJobQueueTask take up where we left it.

It looks like silverstripe-gridfieldqueuedexport can not handle this situation. In the end I have an export with only the records from the last restart. This is easy to spot because the column headers are missing in the first line.

Then the part I am not sure about:
It looks like the problem might be that silverstripe-gridfieldqueuedexport uses write mode and not append mode for file handling in GenerateCSVJob->getCSVWriter():

$csvWriter = Writer::createFromPath($this->getOutputPath(), 'w');

Maybe the module is not build to account for this situation. In that case I think it is worth mentioning it in the README.

https://app.travis-ci.com/github/silverstripe/recipe-kitchen-sink/jobs/537028781

SilverStripe\GridFieldQueuedExport\Tests\GenerateCSVJobTest::testGenerateExport()

Couldn't restore GridField

This will require a new client/lang folder, upstream master file on transifex, new source file, and update the PHP logic to include the necessary javascript files conditionally.

This message displays continuously after running the task
error-log.INFO: Next pending job could NOT be found or lock could NOT be obtained. [] []

exporting UDF submissions

GFQE version 2.8.1

Can you please note in the documentation that out of the box this only auto-applies to UserDefinedForm (pages), and that to apply this to the submissions gridfield of ElementForm requires:

DNADesign\ElementalUserForms\Model\ElementForm:
  extensions:
    - SilverStripe\GridfieldQueuedExport\Extensions\UserFormUseQueuedExportExtension

(Alternatively, would this be an acceptable addition to the module's config.yml?)

On a particular environment we encountered a permission-issue after a server-update. The permission is checked in the GridField_URLHandler to make sure the right user is accessing the export.

/src/Forms/GridFieldQueuedExporButton.php

    public function checkExport($gridField, $request = null)
    {
        $id = $request->param('ID');
        $job = QueuedJobDescriptor::get()->filter('Signature', $id)->first();

        
        if ((int)$job->RunAsID !== Security::getCurrentUser()->ID) {
            return Security::permissionFailure();
        }

The problem seems to be that the Security::getCurrentUser()->ID is returned as a sting, and the $job->RunAsID is casted into an INT.

Can I make a PR to change this into

        if ((int)$job->RunAsID !== (int)Security::getCurrentUser()->ID) {

Or is there some other voodoo going on, why my CurrentUser is returning a string?

silverstripe/framework: 4.8.0
silverstripe/gridfieldqueuedexport: 2.3.0

Hi there,
I am running silverstripe-gridfieldqueuedexport 2.1, SilverStripe 4.2 & PHP 7.1.22 and I came across a namespacing issue where the template for GridFieldQueuedExportButton cannot be found after clicking the export button.
None of the following templates could be found: SilverStripe\GridfieldQueuedExport\Forms\GridFieldQueuedExportButton in themes "Array ( [0] => silverstripe/admin:cms-forms [1] => $default ) " SSViewer.php:215

Although the template file is actually located under templates/SilverStripe/GridFieldQueuedExport/Forms/GridFieldQueuedExportButton.ss (note the difference of GridField vs Gridfield.

Looks like the namespacing needs to be updated to match GridField or I just changed the folder for the template to templates/SilverStripe/GridfieldQueuedExport/Forms/GridFieldQueuedExportButton.ss for the meantime.

Cheers,
Alex

User Story

Acceptance Criteria

Sub Tasks

• [ ]

• [ ]

• [ ]

On multi server environments sometime the error message "This export has already been downloaded. For security reasons each export can only be downloaded once." appears

This is because the queued job has completed, and the file is on one server, but not the server you are currently on, so the GridFieldQueuedExportButton.php file_exists() check will fail

The file will sync across, and refreshing the CMS will fix this, however this isn't obvious to the user

Pull requests

• SS3: #33
• SS4: #34
• CWP 1.x: silverstripe/cwp#167
• CWP 2.x: silverstripe/cwp#168

https://github.com/silverstripe/silverstripe-gridfieldqueuedexport/blob/2.3.0/src/Forms/GridFieldQueuedExportButton.php#L193

The title ($gridField->Title()) is more likely to be human-readable than the name - e.g. ModelAdmin generates a sanitised version of the FQCN of the DataObject.

https://github.com/silverstripe/silverstripe-gridfieldqueuedexport/blob/master/src/Jobs/GenerateCSVJob.php#L134-L153

mkdir is affected by the current user's mask, which can result in writing the .exports dir (and sub dir/files) with permissions other than the declared 0770 (I'm seeing 750).

https://github.com/silverstripe/silverstripe-gridfieldqueuedexport/blob/master/src/Forms/GridFieldQueuedExportButton.php#L249-L258

This precludes the users with group perms from being able to cleanup the csv dir/file before downloading the csv file.

• On the surface this doesn't affect Live as the warning is supressed and the file sent as normal, but the files aren't cleaned up.
• On Dev this prevents download of the csv due to both the warning and file contents being output.

This wont affect those who execute the export-job and download action as the same user. But as one is likely handled by cron, and the other apache, chances are that different users will be involved.

An easy fix here is to use umask(0) when making the dirs.
EDIT: turns out umask is unsafe in a multi-theaded environment and chmod should be used instead.

PR here: #37

I think this may be an issue with nested grid fields.

Upon clicking the queued export button, the page reloads, and simply displays the string SilverStripe\View\ViewableData_Customised

This is occurring on ElementForm submission exports, and any other gridfield I've tried it on within elemental Elements.

Got this module with a CWP update, once job runs the file exports but it only contains the "ID" and "Created" fields.

Bookmarking or refreshing the download page works at the Model Admin level but if I have the queued export button on a GridField for a has_many relationship on a data object then refreshing the page doesn't work. I get a blank page except for the string ViewableData_Customised.

As of #6, clicking a download link causes the file to be deleted. We should probably prevent double-clicks, as the second click would fail to download the file.

silverstripe/silverstripe-framework#8698 makes form field labels come out as My field instead of My Field. The tests are failing due to a case change.

Seems the admin_template isn't found after clicking the button. Might need a configuration update

Silverstripe Gridfield Queued Export stores CSV files in the assets/.exports dir. These files get deleted once downloaded, but if for some reason, you never download the file, the file will be stuck in limbo forever and never get deleted.

It also completely bypasses all the assets logic and directly writes and unlinks the files. There's some attempt to write a .htaccess file to block direct download from the file, but that method is fallible because your webserver could be configured to ignore .htaccess files or you might be running your site on NGINX or IIS. The file names are also random, which minimise the risk that someone will stumble on them.

It's arguable whatever this is an actual security vulnerability. I guess you need a lot of things to go wrong for the files to be disclosed publicly. It sure is not good security architecture.

At the very least, it's a GDPR problem because the CSV data could be stuck there without a way to delete it.

This is the bit that creates the file.

This is the bit that serves the file and delete it.

Notes

This was initially reported as a security issue. We decided to threat it as a regular issue since there isn't anything directly exploitable.

Apologies in advance, I don't have the specific scenario that caused the error, but I do have the simple fix! While doing SS5 upgrades and reviews, I'm PR'ing back what I can as I go.

Basically updateCMSFields in UserFormUseQueuedExportExtension.php is expecting $gridField = $fields->fieldByName('Root.Submissions.Submissions'); to always return a valid gridfield.

When I use this module along with elemental-userforms, in some scenario (which I can't recall - but I think is as commonplace as on creation of the userforms element) the Root.Submissions.Submissions gridfield doesn't exist, so when $gridField->getConfig() is called on null, an error is thrown.

I'm going to submit a PR, ideally could we just add if (empty($gridField)) return; between the two lines?

I can do more homework here, I'm just hoping that it's enough of a commonsense edit that you don't need me to 😅

PRs

• #81

How do we feel about adding a queued import feature to this module? appropriate or not?

Does the export respect filters set from the ModelAdmin?

Situation:

Clean Silverstripe 4.12 install. ModelAdmin with GridField Queued Export button.

Problem:
Job queues fine but always breaks immediately when the task starts to process the queue.
When the project is downgraded tot Silverstripe 4.11 all is fine.

Cause:
LeftAndMain now has afterHandleRequest that checks for $this->response->isError()
If an error is found, the current response (a GridFieldQueuedExportButtonResponse) is replaced by a new HTTPResponse representing an error state.
This is problematic because in the constructor of GridFieldQueuedExportButtonResponse a status of 500 is assigned to the response. As far as I can see this status is never altered after that and triggers the new afterHandleRequest functionality.

Fix:
This could be a fix. Perhaps someone can suggest someting more elaborate to detect the correct setting of the body on the response:

class GridFieldQueuedExportButtonResponse extends HTTPResponse
{
    /**
     * @var GridField
     */
    protected $gridField;

    public function __construct(GridField $gridField)
    {
        $this->gridField = $gridField;

        if($this->gridField) {
            parent::__construct('', 200);
        }
        else {
            parent::__construct('', 500);
        }
    }
}

Edit:

The specific error I'm seeing when I try to run the job via the CLI is:

Call to a member function getManipulatedList() on null at /var/www/vendor/silverstripe/gridfieldqueuedexport/src/Jobs/GenerateCSVJob.php:395)

PR

• #82

The extension yml file points to non namespaced classes