silverstripe / silverstripe-gridfieldqueuedexport Goto Github PK
View Code? Open in Web Editor NEWExport large data sets from your GridField in the Silverstripe CMS interface through async jobs
License: MIT License
Export large data sets from your GridField in the Silverstripe CMS interface through async jobs
License: MIT License
User Story
Acceptance Criteria
Sub Tasks
[ ]
[ ]
[ ]
Apologies in advance, I don't have the specific scenario that caused the error, but I do have the simple fix! While doing SS5 upgrades and reviews, I'm PR'ing back what I can as I go.
Basically updateCMSFields
in UserFormUseQueuedExportExtension.php
is expecting $gridField = $fields->fieldByName('Root.Submissions.Submissions');
to always return a valid gridfield.
When I use this module along with elemental-userforms
, in some scenario (which I can't recall - but I think is as commonplace as on creation of the userforms element) the Root.Submissions.Submissions
gridfield doesn't exist, so when $gridField->getConfig()
is called on null, an error is thrown.
I'm going to submit a PR, ideally could we just add if (empty($gridField)) return;
between the two lines?
I can do more homework here, I'm just hoping that it's enough of a commonsense edit that you don't need me to ๐
The extension yml file points to non namespaced classes
Can you please note in the documentation that out of the box this only auto-applies to UserDefinedForm (pages), and that to apply this to the submissions gridfield of ElementForm requires:
DNADesign\ElementalUserForms\Model\ElementForm:
extensions:
- SilverStripe\GridfieldQueuedExport\Extensions\UserFormUseQueuedExportExtension
(Alternatively, would this be an acceptable addition to the module's config.yml?)
How do we feel about adding a queued import feature to this module? appropriate or not?
Got this module with a CWP update, once job runs the file exports but it only contains the "ID" and "Created" fields.
https://app.travis-ci.com/github/silverstripe/recipe-kitchen-sink/jobs/537028781
SilverStripe\GridFieldQueuedExport\Tests\GenerateCSVJobTest::testGenerateExport()
Couldn't restore GridField
I think this may be an issue with nested grid fields.
Upon clicking the queued export button, the page reloads, and simply displays the string SilverStripe\View\ViewableData_Customised
This is occurring on ElementForm submission exports, and any other gridfield I've tried it on within elemental Elements.
As of #6, clicking a download link causes the file to be deleted. We should probably prevent double-clicks, as the second click would fail to download the file.
This will require a new client/lang folder, upstream master file on transifex, new source file, and update the PHP logic to include the necessary javascript files conditionally.
mkdir
is affected by the current user's mask, which can result in writing the .exports
dir (and sub dir/files) with permissions other than the declared 0770
(I'm seeing 750
).
This precludes the users with group perms from being able to cleanup the csv dir/file before downloading the csv file.
This wont affect those who execute the export-job and download action as the same user. But as one is likely handled by cron, and the other apache, chances are that different users will be involved.
An easy fix here is to use umask(0)
when making the dirs.
EDIT: turns out umask
is unsafe in a multi-theaded environment and chmod
should be used instead.
PR here: #37
Does the export respect filters set from the ModelAdmin?
On multi server environments sometime the error message "This export has already been downloaded. For security reasons each export can only be downloaded once." appears
This is because the queued job has completed, and the file is on one server, but not the server you are currently on, so the GridFieldQueuedExportButton.php file_exists() check will fail
The file will sync across, and refreshing the CMS will fix this, however this isn't obvious to the user
This message displays continuously after running the task
error-log.INFO: Next pending job could NOT be found or lock could NOT be obtained. [] []
exporting UDF submissions
GFQE version 2.8.1
The title ($gridField->Title()
) is more likely to be human-readable than the name - e.g. ModelAdmin generates a sanitised version of the FQCN of the DataObject.
Silverstripe Gridfield Queued Export stores CSV files in the assets/.exports
dir. These files get deleted once downloaded, but if for some reason, you never download the file, the file will be stuck in limbo forever and never get deleted.
It also completely bypasses all the assets logic and directly writes and unlinks the files. There's some attempt to write a .htaccess
file to block direct download from the file, but that method is fallible because your webserver could be configured to ignore .htaccess
files or you might be running your site on NGINX or IIS. The file names are also random, which minimise the risk that someone will stumble on them.
It's arguable whatever this is an actual security vulnerability. I guess you need a lot of things to go wrong for the files to be disclosed publicly. It sure is not good security architecture.
At the very least, it's a GDPR problem because the CSV data could be stuck there without a way to delete it.
This is the bit that creates the file.
silverstripe-gridfieldqueuedexport/src/Jobs/GenerateCSVJob.php
Lines 176 to 247 in 06a09ca
This is the bit that serves the file and delete it.
silverstripe-gridfieldqueuedexport/src/Forms/GridFieldQueuedExportButton.php
Lines 234 to 261 in 06a09ca
This was initially reported as a security issue. We decided to threat it as a regular issue since there isn't anything directly exploitable.
On a particular environment we encountered a permission-issue after a server-update. The permission is checked in the GridField_URLHandler to make sure the right user is accessing the export.
/src/Forms/GridFieldQueuedExporButton.php
public function checkExport($gridField, $request = null)
{
$id = $request->param('ID');
$job = QueuedJobDescriptor::get()->filter('Signature', $id)->first();
if ((int)$job->RunAsID !== Security::getCurrentUser()->ID) {
return Security::permissionFailure();
}
The problem seems to be that the Security::getCurrentUser()->ID is returned as a sting, and the $job->RunAsID is casted into an INT.
Can I make a PR to change this into
if ((int)$job->RunAsID !== (int)Security::getCurrentUser()->ID) {
Or is there some other voodoo going on, why my CurrentUser is returning a string?
silverstripe/framework: 4.8.0
silverstripe/gridfieldqueuedexport: 2.3.0
Seems the admin_template isn't found after clicking the button. Might need a configuration update
Thanks for a very useful module!
Situation:
7000+ rows export on a high load machine
Problem:
QueuedJobService has a default memory limit of 256Mb. Their method runJob has a check isMemoryTooHigh. In our case, that limit was met. The idea behind the way QueuedJobService handles this situation seems to be to release memory and let the next iteration of ProcessJobQueueTask take up where we left it.
It looks like silverstripe-gridfieldqueuedexport can not handle this situation. In the end I have an export with only the records from the last restart. This is easy to spot because the column headers are missing in the first line.
Then the part I am not sure about:
It looks like the problem might be that silverstripe-gridfieldqueuedexport uses write mode and not append mode for file handling in GenerateCSVJob->getCSVWriter():
$csvWriter = Writer::createFromPath($this->getOutputPath(), 'w');
Maybe the module is not build to account for this situation. In that case I think it is worth mentioning it in the README.
Hi there,
I am running silverstripe-gridfieldqueuedexport 2.1, SilverStripe 4.2 & PHP 7.1.22 and I came across a namespacing issue where the template for GridFieldQueuedExportButton cannot be found after clicking the export button.
None of the following templates could be found: SilverStripe\GridfieldQueuedExport\Forms\GridFieldQueuedExportButton in themes "Array ( [0] => silverstripe/admin:cms-forms [1] => $default ) " SSViewer.php:215
Although the template file is actually located under templates/SilverStripe/GridFieldQueuedExport/Forms/GridFieldQueuedExportButton.ss
(note the difference of GridField
vs Gridfield
.
Looks like the namespacing needs to be updated to match GridField
or I just changed the folder for the template to templates/SilverStripe/GridfieldQueuedExport/Forms/GridFieldQueuedExportButton.ss
for the meantime.
Cheers,
Alex
Situation:
Clean Silverstripe 4.12 install. ModelAdmin with GridField Queued Export button.
Problem:
Job queues fine but always breaks immediately when the task starts to process the queue.
When the project is downgraded tot Silverstripe 4.11 all is fine.
Cause:
LeftAndMain now has afterHandleRequest that checks for $this->response->isError()
If an error is found, the current response (a GridFieldQueuedExportButtonResponse) is replaced by a new HTTPResponse representing an error state.
This is problematic because in the constructor of GridFieldQueuedExportButtonResponse a status of 500 is assigned to the response. As far as I can see this status is never altered after that and triggers the new afterHandleRequest functionality.
Fix:
This could be a fix. Perhaps someone can suggest someting more elaborate to detect the correct setting of the body on the response:
class GridFieldQueuedExportButtonResponse extends HTTPResponse
{
/**
* @var GridField
*/
protected $gridField;
public function __construct(GridField $gridField)
{
$this->gridField = $gridField;
if($this->gridField) {
parent::__construct('', 200);
}
else {
parent::__construct('', 500);
}
}
}
Edit:
The specific error I'm seeing when I try to run the job via the CLI is:
Call to a member function getManipulatedList() on null at /var/www/vendor/silverstripe/gridfieldqueuedexport/src/Jobs/GenerateCSVJob.php:395)
silverstripe/silverstripe-framework#8698 makes form field labels come out as My field
instead of My Field
. The tests are failing due to a case change.
Bookmarking or refreshing the download page works at the Model Admin level but if I have the queued export button on a GridField for a has_many
relationship on a data object then refreshing the page doesn't work. I get a blank page except for the string ViewableData_Customised
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.