rello / analytics Goto Github PK

When trying to import CSV from clipboard I have error message. When I check the log I see this:

{
	"reqId":"xxxxx",
	"level":3,
	"time":"2019-12-03T15:33:05+00:00",
	"remoteAddr":"xxxxx",
	"user":"xxxxx",
	"app":"PHP",
	"method":"POST",
	"url":"\/index.php\/apps\/analytics\/data\/importCSV",
	"message":"sizeof(): Parameter must be an array or an object that implements Countable at \/path\/to\/nextcloud\/apps\/analytics\/lib\/Controller\/DataLoadController.php#150",
	"userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0",
	"version":"17.0.1.1"
}

I tried with these CSVs:

A;B;B
1;2;3
3;4;5
6;7;8

A,B,B
1,2,3
3,4,5
6,7,8

May be this is not what is expected?

PHP Version Info:

php -v
PHP 7.3.12 (cli) (built: Nov 20 2019 19:02:54) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.12, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.12, Copyright (c) 1999-2018, by Zend Technologies

If you need more info let me know.

The file action menu provides an option to create a new Analytics report from a csv/txt file directly

Activity settings for Reports and Report Data

Activities for various events

Activity filter

Subjects:

const SUBJECT_DATASET_ADD = 'dataset_add';
const SUBJECT_DATASET_DELETE = 'dataset_delete';
const SUBJECT_DATASET_SHARE = 'dataset_share';
const SUBJECT_DATA_ADD = 'data_add';
const SUBJECT_DATA_ADD_API = 'data_add_api';

Are there any tutorials or documentation on Analytics app?
Specific question regarding datasources: Is there any way for Analytics to pull data from an SQL (MYSQL or MS SQL or SQlite) db?

https://jshare.com.cn/temp/ZOGGQw 当旭日图中有负值的时候显示的问题

The only available flow I see is "Write to Conversation". The video shows several others and a button to add more (which I don't have). How do I find and add additional flows?

uBlock origin is blocking asset loading because of the 'analytics/js' inside the path.

Firefox Console:

uBlock Origin Logger:

I will try to find which list contains this filter.

Since you moved to Chart.js i am missing some documentation guides.
With Highchart it was possible to have a csv like this to have multiple graphes

Name,Date,Amount
testname,2020-02-14,108
testname,2020-02-15,68
testname,2020-02-16,60
testname,2020-02-17,96
testname,2020-02-18,103
differentchart,2020-02-14,53
differentchart,2020-02-15,30
differentchart,2020-02-16,22
differentchart,2020-02-17,40
differentchart,2020-02-18,42

Now such file looks like this since the migration to chart.js.

Whats the correct way of using chart.js features trough a csv file? Are there even all features available, like horizontal bars or different point styles?

Installing analytics 2.4.0 into nextcloud 18.0.6 fails with the following error:

InvalidArgumentException: Index name "oc_analytics_threshold"."analytics_threshold_dataset_idx" is too long.

/var/www/nc.qcft.de/nextcloud/lib/private/DB/MigrationService.php - line 484:
OC\DB\MigrationService->ensureOracleIdentifierLengthLimit(Doctrine\DBAL\Schema\Schema {}, Doctrine\DBAL\Schema\Schema {}, 3)

/var/www/nc.qcft.de/nextcloud/lib/private/DB/MigrationService.php - line 414:
OC\DB\MigrationService->executeStep("020400Date20200515185352", false)

/var/www/nc.qcft.de/nextcloud/lib/private/Installer.php - line 157:
OC\DB\MigrationService->migrate()

/var/www/nc.qcft.de/nextcloud/apps/settings/lib/Controller/AppSettingsController.php - line 443:
OC\Installer->installApp("analytics")

/var/www/nc.qcft.de/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 170:
OCA\Settings\Controller\AppSettingsController->enableApps([ "analytics"], [])

/var/www/nc.qcft.de/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 99:
OC\AppFramework\Http\Dispatcher->executeController(OCA\Settings ... {}, "enableApps")

/var/www/nc.qcft.de/nextcloud/lib/private/AppFramework/App.php - line 125:
OC\AppFramework\Http\Dispatcher->dispatch(OCA\Settings ... {}, "enableApps")

/var/www/nc.qcft.de/nextcloud/lib/private/AppFramework/Routing/RouteActionHandler.php - line 47:
OC\AppFramework\App::main("OCA\\Settin ... r", "enableApps", OC\AppFramew ... {}, { _route: "s ... "})

<<closure>>
OC\AppFramework\Routing\RouteActionHandler->__invoke({ _route: "s ... "})

/var/www/nc.qcft.de/nextcloud/lib/private/Route/Router.php - line 299:
call_user_func(OC\AppFramew ... {}, { _route: "s ... "})

/var/www/nc.qcft.de/nextcloud/lib/base.php - line 1008:
OC\Route\Router->match("/settings/apps/enable")

/var/www/nc.qcft.de/nextcloud/index.php - line 38:
OC::handleRequest()

Apparently nextcloud/lib/private/DB/MigrationService.php enforces a maximum string length of 30 chars for index names, which analytics_threshold_dataset_idx violates with a count of 31.

The saveing of data source HTML grapper is not possible with the following combination. I tested the regex with no issues, but I cant save it. No error or warning given.

I suspect that the regex contains something creating the issue.

Datasource:
https://www.pollenwarndienst.at/prognose/3-tages-prognose.html?tx_scload_load%5Bzip%5D=8020

Regex:
/(<h3 class="polltitle">)(?<dimension>Erle)(\s.*\n){9,15}(.*<span class="invisible">\s*)(?<value>(Keine Belastung|niedrig|mittel|hoch|sehr hoch))(<\/span>)/

Result from https://pt.functions-online.com/preg_match_all.html:

  0 => 
  array (
    0 => '<h3 class="polltitle">Erle (Alnus)</h3>

                                        <div class="date">Dienstag, 17. März</div>
                                        

<div class="contaminationbar">
    
        <div class="number number_2">2</div>
        <div class="bar bar_2"></div>
    

    
        
        
        <span class="invisible"> mittel</span>',
  ),
  1 => 
  array (
    0 => '<h3 class="polltitle">',
  ),
  'dimension' => 
  array (
    0 => 'Erle',
  ),
  2 => 
  array (
    0 => 'Erle',
  ),
  3 => 
  array (
    0 => '        
',
  ),
  4 => 
  array (
    0 => '        <span class="invisible"> ',
  ),
  'value' => 
  array (
    0 => 'mittel',
  ),
  5 => 
  array (
    0 => 'mittel',
  ),
  6 => 
  array (
    0 => 'mittel',
  ),
  7 => 
  array (
    0 => '</span>',
  ),
)

Report customisation is available on top of the chart.
Fur the current version, it is only enabled for internal reports

Add Filter

Change Options

Change Drilldown

Report displays time in UTC, it not use local/server timezone.

Hello @blizzz, @juliushaertl

I would like to follow up our conversation regarding Workflow.
My last status was, that you were still working several basic classes.

• Do you have any update for me if it is worth starting already to provide an Operation?
• Can you direct me towards a reference app or how-to? (you showed my some prototypes back then)

thank you,
Rello

Hey there,

is it somehow possible to specify the color of a graph inside a csv? Maybe add a forth column or so.
I am not able to find any documentation to what is possible with the nextcloud analytics app, but its so useful!

Great product! :)

Could I please request the ability to purge the existing record-set and re-load, for scheduled data loads.

The thresholds, even though set, are not working. I tried on both graphs and tables. My source file is a csv from a local NC folder. Additionally, the notification option when a threshold is met is not available.

May I suggest a feature that might be handy? I have a graph report from local NC .csv file with hourly time-series, which gets quite dense day by day. It would be useful to have an option to select the range of the data showing on the graph (e.g. the last 1000 data points). I hope that it can be easily implemented.

It would be nice if the graphs would show the users timezone instead of the plain UTC values.

@juliushaertl @blizzz

I need your support again. I am seeing several logs related to the workflowengine and my app:

[workflowengine] Debug: Cannot handle event \OCP\Files::postWrite of Symfony\Component\EventDispatcher\GenericEvent against entity OCA\WorkflowEngine\Entity\File and operation OCA\Analytics\Flow\Operation

[no app in context] Debug: Deprecated event type for \OCP\Files::postWrite: Symfony\Component\EventDispatcher\GenericEvent

regarding the first one: am I still using the right generic Event?
https://github.com/Rello/analytics/blob/master/lib/Flow/Operation.php#L28

[workflowengine] Debug: Flow activation: rules were requested for operation Data Analytics

[workflowengine] Debug: No flow configurations is going to run Data Analytics

[PHP] Error: Undefined index: operation at /mnt/analytics/lib/Flow/Operation.php#97

the 3 last seem strange. if I understand correctly, workflow determines that DA is not relevant (I setup a rule for post-write and file name, but the file name was not matched).
But why is my onEvent Operation still being accessed afterwards?
https://github.com/Rello/analytics/blob/master/lib/Flow/Operation.php#L97

thank you!

Introduce a filter dialog for report analysis

• filter backend for internal data
• filter dialog
• drilldown dialog
• dynamic dialog for any dimension combination
• print layout
• share report

Here's the datasource as a csv:
https://tld-list.com/df/tld-list-details.csv

At import time it throws an error about a number at the 3rd column which is the column of the punycode, something like this: xn--80akhbyknj4f
Which triggers wrongly the parsing I guess because the import believes it's a number when it's just an alfanumeric string with dashes(-)

For now I will delete that column and hope in a successful import, nice job overall 😉

Hello Rello,

Thanks for your application.
Can you add the possibility to use your application with nextcloud black theme, please ?
See my screenshot capture

When installing analytics app from nextcloud web app store, it gave this error. Searched and guess it might be an error from release group.

https://jshare.com.cn/temp/ZOGGQw 当旭日图中有负值的时候显示的问题

Having a configuration option for a github API token would be quite nice globally or per report, so users don't run into the APIs rate limiting that fast.

Hello @nickvergessen

an I ask for your recommendation here:

I am trying to use the controller/service setup as recommended. But for keeping the app-structure similar to its modular setup, I am not sure what the recommendation is:

I want to implement an ApiDataController to enable REST/CORS. Generally, it would be a controller in the /Controller folder
But in Analytics context its a datasource - so I would like to group all of them in a /lib/Datasource folder

• does this make sense? is it best practise?
• would this still work with the route-setup?

'name' => 'apiservice#addData',
'url' => '/api/1.0/adddata/{datasetId}',

what is your idea on this?

It would be great if scheduled data loading could be configured to run more often than hourly, e.g. with a cronjob scheduled every 5 minutes.

I've upgraded from 1.1.0 to 1.2.0, all the graphs works ok but the tables are empty, no matters if I choose table+chart or table, if only shows the headers.

Hi there,
I have started working with your app as it is a real handy application - perfect for visualization.
Unfortunately I am not able to add all csv columns to Analytics:
I am monitoring my bandwith with speedtest-cli which puts the output into a csv file (see attachment, neede to change .csv to .txt as otherwise Github won't accept it)
Speedtest_WAN.txt

When I open it with Analytics I just see "Date", "Time" & "Ping (ms)" - but no Down- & Upload.
Is it possible to add more values into Analytics?
Or maybe the column name is "wrong" so Analytics cannot read it?

Thanks, Mathias

Hello,

I'm discovering your app, thanks
It seems that the button donate redirect to another of your app, it might be confusing for some person :

Hi, I am trying to use the Rest API to send a JSON array using the Python 3 code snippet below

payload = []
count = 0
while count < 9:
    now = datetime.now().strftime('%M:%S')
    payload.append({'dimension1': 'X axis', 'dimension2': now, 'dimension3': 1})
    payload.append({'dimension1': 'Y axis', 'dimension2': now, 'dimension3': 10})
    count = count + 2
    time.sleep(1)
print("Posting", json.dumps(payload)) #  using the dumps method to print and check if JSON is formed correctly
response = requests.post(url, headers=headers, json=payload, auth=(UserID, Password), verify=False)
print("Posted as ",response)

The output is as below

Posting [{"dimension1": "X axis", "dimension2": "21:37", "dimension3": 1}, {"dimension1": "Y axis", "dimension2": "21:37", "dimension3": 10}, {"dimension1": "X axis", "dimension2": "21:38", "dimension3": 1}, {"dimension1": "Y axis", "dimension2": "21:38", "dimension3": 10}, {"dimension1": "X axis", "dimension2": "21:39", "dimension3": 1}, {"dimension1": "Y axis", "dimension2": "21:39", "dimension3": 10}, {"dimension1": "X axis", "dimension2": "21:40", "dimension3": 1}, {"dimension1": "Y axis", "dimension2": "21:40", "dimension3": 10}, {"dimension1": "X axis", "dimension2": "21:41", "dimension3": 1}, {"dimension1": "Y axis", "dimension2": "21:41", "dimension3": 10}]
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:860: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Posted as <Response [200]>

Though I get <Response [200]>, not even a single record is shown on the basic line graph or its table that is at the 'url' on my next cloud server.

Whereas, when I have only one record in the json object (not as an array) and use the requests.post method retaining everything else to be the same, it works fine. Since it is time consuming to insert 1 record at a time for my application, I want to send an array.

I am using a self-signed certificate and would like to ignore the SSL warning for now, as it applies to either of the cases above and seems immaterial to my problem

Can you please let me know if the rest API method accepts a JSON Array? If yes, where am I going wrong?

First of all many thanks for this module! Works great for csv files and I am able to collect data via the REST API.

Is it possible to filter the data in the graph to display only the latest day(s) or hour(s) or group them per hour/day/week/...?

How is it possible to remove older data stored in internal tables? They can grow realy fast and I don't want to show them all at the same time.

Several different views of the same data would be great!

Regards,
berho

Introduce an advanced configuration page which can handle more options than sidebar

Provide a datasource to read json data by a given array-string

Input Parameter:

• URL of json
• Authentification
• json path
• timestamp t/f

Use Case / Example:
Monitor the Nextcloud instance via its internal API

• URL: http://server/nextcloud/ocs/v2.php/apps/serverinfo/api/v1/info?format=json
• Auth: user:token
• Path: ocs/data/nextcloud/storage
• timestamp t/f

Example Output:

Hello,

Thanks for the application, seems very neat. I appreciate the notification threshold.
In case of different magnitude order between objects, would it be possible to implement the possibility to have a secondary axis ?

For instance : Temperature, humidity, AIQ level are difficult to follow on the same axis.

Thanks

Hey there again .. :),

Apparently when I have less than 22 rows to be shown in a timeline chart the date changes to this weird 12AM, 4AM, 2AM format instead of the well formated date like Feb 20. Btw the csv file obviously doesn't change and the second column is 2020-02-20 for example. Bug or Feature? 💃

The expected one ( or when i have above 21 rows in a csv ):

The weird time formatted one ( below 22 rows in csv):

• more detailed definition of dataloads
• scheduling via cron (hourly / daily)

A whats-new popup can be pushed to the user

• Datasource to extract data via regex from external html page
• Available only in advanced config due to required parameters

Highcharts needs to be removed due to license issues.
Migration to Chart.js or C3 is being evaluated

Hello!

Is it possible to add such datasource? OnlyOffice app: https://apps.nextcloud.com/apps/onlyoffice

Datasources and connectors

• REST API
• file actions: crate report from Files app #4
• occ command for executing dataloads #16
• Datasource: JSON #21
• Event to register external datasources #71
• Datasource: OnlyOffice .xlsx table #20

Visualisation

• report groups
• more chart types (doughnut)
• advanced chart options #30
• secondary axis #36
• customization per data series #44
• #145
• thresholds in charts
• Multiple reports per dataset #193

Reports

• print layout #34
• Report filter and drilldown dialog #41
• persist selected filters #33 #36
• sharing on filters #33
• Dynamic filter (dates/ranges) #186

Integration

• NC Activity integration #2
• NC Notifications for alerts/thresholds #5
• NC Workflow integration #10
• NC BackgroundJob for snapshots #12
• Whats new #43
• Sharing: user + groups #60 (v2.5)
• NC20 Dashboard Widget #61
• NC20 Search #66
• NC Comments integration
• NC Projects integration

Others

• Translation
• Advanced configuration page for more options than sidebar #12

First of: Thank you! I love the idea and, if you succeed to realize your vision, it will be so much more useful then SensorLogger.

The issue:
I installed from the NC app store, but the app is completely non-responsive. The only two interactive elements on the fresh install are "New Report" and "- Create demo report on external CSV data", but both don't do anything when clicked.

Did I miss any settings? I could not find anything.

Hi @Rello, thank you for this app! I would like to use and contribute, but I have very little experience with nextcloud apps (yet).

I need (and could help to create) user documentation for nextcloud beginners.

I want to use the "website grabber".

Chapter Usage in the Wiki says

1. Report -> advanced config

Where do I find "advanced config"?

Maybe I need to satisfy the condition "the datasource is only available in advanced config with scheduling due to the required parameters"?

How shall I enable scheduling?

Originally posted by @markuskramerIgitt in #1 (comment)

https://jshare.com.cn/temp/ZOGGQw 当旭日图中有负值的时候显示的问题

Can't create report chart from HTML grabber. In Report->Datasource select "HTML grabber" not present. Any idea?

Analytics 2.4.1

https://www.odata.org/

I am mainly interesting in it for linking to OpenDataKit Central data, but it seems to be a format widely used with Office365 etc.

Thanks for considering it.

I want to POST data to the internal database with the following command but get an error message! Can you tell me what I am doing wrong?

curl -v -H Content-Type:application/json -u user:supersecretepassword -X POST -d {"dimension1":"Temperature","dimension2":2020-03-24T15:57:32.000Z,"dimension3":48} https://myserver.dom/index.php/apps/analytics/api/1.0/adddata/4

answer from server: {"success":false,"error":{"code":9002,"message":"Dimension 1 required"}}

Server: 18.0.2
DataAnalytics: 2.1.1

Regards