careset / zermelo Goto Github PK

Where it says "By Default, any numeric field will have statistical information will be passed on"

I think this means, "passed from the backend to the front end in JSON", but I am not sure.

There should be an option to turn off caching for the entire system at the configuration level.

Nothing in the current documentation details where the report will be accessible.

If this is configurable.. tell me where to do this...

What this probably means is that we should have an example config file in our README.md to help people know how to set things by default...

The tooltips for the menu icons in the default tabular report show above the icons, as they should.

However, when you click the "Maximize View" icon (which is a square shaped little icon that then turns orange...

the tooltips for the icons still appear above the icons, but they are unreadable now that they are at up at the top of the browser window... When the view is maximized the tooltips should appear below the icons.

If it is much easier to code, it is acceptable for the tooltips to always appear beneath the icons.

Apparently getCode() will get the first URI component.

Apparently getParameters() will get the rest of the URI component.

How do I get JSON variables? How do I get _GET variables? Do I use the Laravel request object? Does that live on the report object somewhere?

All of this should be documented in both the how-to and the sourcecode of the generated files.

The CACHE_TIMEOUT configuration variable being set to 0 should mean that although data is stored in the _cache table, it is always overwritten.

This does not seem to be respected.

Note: I do have multiple instances of the reporting engine on the same server using the same _cache database, but that should be a supported configuration...

The "real" ticket here is this:

#46

If we could just upgrade to this solution I would be very very happy...

I have an id that I want to use as part of a url I am building for a different column.

Previously, I was able to accomplish this using the unset function inside MapRow. Now using unset causes an error, and the instructions say "columns cannot be added or removed". The whole point of mapRow is that it supported adding and removing columns from the display.

A huge number of our existing reports support this functionality. If we are going to deprecated it, we need to have a very very good reason for doing so, and we also need to have a very clear mechanism for easily porting this code. Something like a "find replace" for unset that correctly removes the column or something...

-FT

The only thing in _GET is the token.

I am writing raw SQL for my reports, and I am accepting user input.

I need to have a consistent way to prevent SQL injections. Whatever that way is needs to be clearly documented in the template code and how-to.

Because ours is custom single sign on...

This should be config parameter.. specify an authentication middleware...

-FT

All of the very serious frustrating debugging that I have dealt with in not only this version of the reporting engine, but all previous versions has been related to caching. The caching was not set the way I thought it was, I thought I had turned off caching but I had not.

Caching is a killer feature for performance of very complex reports, but for developing reports and for all users that want to use the reporting engine as a "small data" browsing tool.. it is overkill and causes painful debugging headaches..

As a result, lets turn it entirely off by default. Lets remove the option to turn it on from the configuration files. I have added the following two functions to the report template

        /**
    * Should we enable the cache on this table?
    * This will improve the performance of very large and complex queries by only running the SQL once and then storing
    * the results in a dynamically creqted table in the _cache database.
    * But it also creates hard to debug update errors that are very confusing when changing GetSQL() contents.
    */
   public function isCacheEnabled(){
        return(false);
   }

        /**
    * How much time should pass (in seconds) before you update your _cache table for this report?
    * this only has an effect when isCacheEnabled is turned on.
    */
   public function howLongToCacheInSeconds(){
        return(600); //ten minutes by default
   }

Lets have the entire caching system pivot on those two functions. This way, when a person really wants the reports to be cached, they can do that for that report and that report only.

Of course, the system should still be configured to write out its results into the _cache database dynamically named table, but it should always be overwriting that in the default mode.

Does that make sense?

-FT

Should be handled on a per report basis with an obvious function call that determines whether a report should be cached.

It should be a function, because there will be cases where we really need to use caching, and updating the cache is very very expensive, but there is some logical way to determine that infrequent cache updates are required.

It would be good if we could use the each report as part of a massive, per-row form generation system.

So that you could have a couple of text boxes, or drop downs at the very top of the report...

Then you could make some decisions on a per-row basis, like "include this row because I checked the checkbox" or "for this row I chose option C rather than option default of A for this row using a select2 box, etc" and then post the entire form (using a botton at the top of the page) to some form processing URL that the report designer specified.

When the report was generated directly as HTML, this was easy to assert, because you just added a <form... > element to the starting html and a to the ending html section and because the report itself was always a very long all inclusive HTML page, this was no problem.

But now the report is something that loads through AJAX, sometimes into front-end viewer which is itself loaded through AJAX, makes thinking about how to make this functionality work a little harder.

-FT

as part of the summary area...

-FT

There needs to be lots of options on "show entries" For now lets do

• 10
• 20
• 50
• 100
• 500
• 1000
• 5000
• ALL (43,234)

Note 43,234 is just an example, but the all option should show the number that "All" would show...

Since the caching engine should allow us to easily see how many results there are for a report... let have the option of having the ability to show everything.

But lets remove the ALL option if results in under 50k results on a single page.. Or a lower number if 50k is not reasonable...

The results json should tell the front end whether the cache was leveraged

I keep having to choose 1000 over and over and over again...

When I choose a number it should remember it on all reports...

At least if that choice is anything less than "All"... I think it is reasonable to have to manually request each new report to give you all the results...

Should now be /app/Zermelo/ rather than /app/CareSetReports/

Right now they are converted by the blade template and write things out using htmlentities or some such.

see bolt software spec too

Then display the data vertically.

Display both the column name, and the column mouse over information (should be extra long descriptions of what each column means) and turn the data sideways.

There are several modes of this that might work.

1. Only display the vertical version, when there is one and only one row of data in the source data.
2. Allow for there to be a "flip" icon in the top left of the table, whenever there are less than 4 rows of data to display, and handle the flipping entirely on client side.

If the flipping between vert and horizontal is a button and under the user control.. create a momentary gif animation that explains visually what is happenig... i.e. that the rows are becoming columns and the columns are becoming rows.

Smart versions of this feature would understand the difference between data with lots of rows and few columns, few rows and lots of columns... few-few and many-many... There might be some combinations that are unworkable, and the interface should abandon those efforts when that is true.

Paging results should likely just disable this feature entirely.

On step 3 of configuration, "Edit the file /app/Zermelo/YourNewReportName You must fill in a reasonable GetSQL() function that returns either a single SQL text string, or an array of SQL text strings."

Should the default report namespace be Reports or Zermelo?

needs to be documented...

The tabular interface should be using bootstrap 4.1

Lets use the latest bootstrap by default from now on..

this might also require updating datatables css etc etc...

Lets suppose I get invalid input, because of a malformed url, or because someone entered the wrong information to a form that feeds a report.

How do I surface an "error" from within GetSQL that is appropriate for end user consumption?

How do I surface the technical details of that error? Do I log it?

Allow currency symbols to be customized. Not just '$'

across reports. So if I say "show me 1000" keep that as the setting as I change reports..

Perhaps each report should be able to enforce a starting default entries? So that a report that would be crippled by showing 1000 can avoid that by having a setting that forces only 10 (or whatever, to be shown?)

But for now, lets just remember what the setting was and respect it. KISS

To create a blank report template...

I would be nice if this was smart enough to ask

"What is the name of the report"
and
"What is the description of the report"

And have those parts of the report template automatically populated by those decisions...

It would also be nice to be able to use variations of the command to be able to create different report types... So eventually it might be

./artisan Zermelo:createTabular
./artisan Zermelo:createCard
./artisan Zermelo:createVega
./artisan Zermelo:createGraph

and have the corresponding thing happen...

The report is trying to load

/js/jquery.dataTables.yadcf.js

Which is apparently associated with https://github.com/vedmack/yadcf

Running artisan vendor:publish does not address the issue.

-FT

It seems really easy to get into a place where I cannot install anything because there is a missing function. I think it was api_prefix()

Please add and test some uninstall directions.. and see if this "interdependancy" bug can be removed somehow...

If this does not quickly replicate for you, then lets leave it and the next time I have the problem I will more carefully detail exactly the steps I took to get there..

-ft

No need to select whether to see 50, 100 or 1000 on one page, when there are only 7 results.

Similarly, there is no reason to have the "previous, 1, next' box when there is no second page.

The following command generates the following error...

$ php artisan vendor:publish --provider="CareSet\Zermelo\ServiceProvider"

In ProviderRepository.php line 208:

Class 'CareSet\ZermeloViewBlade\ServiceProvider' not found

I think I might have fixed this.. but it is hard to tell. Feel free to close this if you are not able to quickly replicate this behavior. Personally I suspect that it is a result of me "half installing" things... but I do not want to get hung up on this too much until we have made more decisions about the installation process.

Currently if a report has no results, it causes an error like so:

"message": "array_keys() expects parameter 1 to be array, null given",
"exception": "ErrorException",
"file": "/var/www/parenthood.set.care/vendor/careset/caresetreportengine/src/CareSet/CareSetReportEngine/Generators/ReportGenerator.php",
"line": 56,

Rather than having a "results = 0" style result.

This should not crash, but instead have some kind of display that says "Hey the SQL ran fine, but there were no results to show"....

-FT

Seems to want to always sort by report order.

/CareSetReport/RequestSummary

on parenthood as example of problem...

Write a series of test reports.

The test reports should rely on the same test data as DURC test data and/or the mysql meta information tables which should always be there.

The tests should include their purpose and what the expected result should be in the test report. They should basically test any reasonable parameter tweak that a given report can achieve.

For instance, OverrideHeader can dramatically change how a given column header is displayed and how it works. There should be one report that just exercises these permutations. Demonstrating how they work and also allowing us as developers to quickly see that they still work as expected.

MapRow should enable the creation of rows with bootstrap buttons or other html form components, as well as functioning links to other sites. There should be one report that just exercises this very heavily.

Basically this task is to start to use "reports as unit tests" for the project.

Frequently, we need to run "setup queries" before we actually run the report in GetSQL...

How do we do that in a way that does not break the query caching system completely?

Can we set it up so that some reports have to be run every time, without using the query?

Can we set it up so that if the GetSQL is doing preparatory work each time, that the engine knows that it cannot cache the #result?

Should we have a PrepSQL() function that separates this process out? So that the engine can know "the results of the preparation are different, therefore the GetSQL results will be different, therefore I cannot use the cache?"

-FT

The file name for the csv download (for now) should include the contents of the getCode() function, assuming they are not null.

We will do lots of later improvements to the download functionality.. but for now that should be enough...

Installing Zermelo, no command found. Error: "There are no commands defined in the "Zermelo" namespace."

Looked in ServiceProvider and doesn't define any commands. Perhaps this functionality got removed accidentally in a revision?

backend should support parameter for force cache refresh

Parenthood example:

/CareSetReport/DataProductGrid/5

It is too easy to forget that you have something in a particular search field and then wonder why your search is returning nothing...

If a search restriction is so limiting that it returns no results, then we should use the results area to let the user know what filter/search criteria are being applied... perhaps with little X icons to delete the filter and search and get some more results.. .

This problem is compounded by the fact that there are two different "levels" of search..

Awesome feature... but it does lead to some confusing results...

If a report is going to take time to load, that is fine... but it should tell the user with a spinner or some other indicator that things are not broken.

I think it would be cool to allow people to choose which fontawesome spinner to use... and to use a nice big one by default:

http://jsfiddle.net/chadkuehn/1twpzLz8/

And make it all work with composer.

Rather than continue to have the caching on by default vs off by default debate. (which has pros and cons either way) we should work to address the underlying usability problem with caching.

When you are writing a report, and the data is changing underneath, and you are expecting the report contents to change, but they do not... then this creates a very hard to find bug for a report builder role who does not, should not and potentially can not understand how the caching works.

So lets expose the following information and functionality to the default view:

• Whether the report was loaded from the cache, or by running the query
• How many times this report has been loaded from the cache
• When the cache that is powering the current data view was built
• A button to refresh the cache. (this should be a different feature, later)

Obviously depends on #35

As part of our "sprint acceptance" work, we need to be certain that, at a minimum, csv export downloads continue to work.

Not sure if this should be part of the acceptance testing process, or automated some other way.

But 95% of the usecase for the reporting engine is for for "report writer A" to create a report for "other user B" and then A sends a link to the report to B and then B immediately downloads the data into excel to work with it further.

If that breaks, or is unreliable, no one will ever trust the reporting engine. So we need to have some kind of reliable mechanism to ensure that it always works, at least in every case that the original report worked correctly....

-FT

Look at this report:

/CareSetReport/DataProductProgress/10

on the parenthood server for an example

I want to be able to have

cnt_thing_id and have it always map to "Count of Things" across every report.