mangoautomation / ma-modules-public Goto Github PK

in mango3.6.6 with modbus module 3.6.6, at application start time,don't put log i/o Communication message，neet restart the datasource,then log i/o is ok.
in mang3.7.7,any time, don't put log i/o Communication message,just put the datasource start message.
such as
2020/08/28-11:30:59,549 1598585459549 data source heartbeet test started
2020/08/28-11:43:08,536 1598586188536 data source heartbeet test started
2020/08/28-11:45:28,331 1598586328331 data source heartbeet test started
2020/08/28-11:50:34,340 1598586634340 data source heartbeet test started
2020/08/28-11:57:38,180 1598587058180 data source heartbeet test started
2020/08/28-14:30:05,532 1598596205532 data source heartbeet test started

but no Communication message

Add a REST endpoint and UI functionality to be able to copy a data source (and optionally its points).

/v2/data-sources/copy/{xid}

Note this issue about performance which has been taken into consideration in the design of the endpoint:

MangoAutomation/ma-core-public#681

This is because the point link returns an AnnotatedPointValueTime with a sourceMessage that is null. Probably the solution is to not return such a thing, but perhaps also AnnotatedPointValueTime should do a null check in getAnnotation() ?

The Reports module needs a UI or to be dropped by Mango 4

This drop down is getting quite large and should be filterable by the attribute description.

Describe the bug
When doing SQL restore. The response shows that the restore failed.

If I look at the ma.log file it shows that the restore was successful.
This was done on an instance running MySQL

To Reproduce
Steps to reproduce the behaviour:
In v3 mangu UI go to /ui/administration/system/sql-backup
choose a backup file and click restore database

Expected behaviour
API response to match ma.log file entry

Screenshots
If applicable, add screenshots to help explain your problem.

Mango Version (please complete the following information):

• Core Version: [e.g. 3.5.5]
• API Version [e.g. 3.6.4]

Additional context
API response: {resourceId: "SYSACTION_18eb08e5-ddc8-4e9f-8aa6-f0307ca0291d",…} expires: 1573035081617 finished: true queueSize: 0 resourceId: "SYSACTION_18eb08e5-ddc8-4e9f-8aa6-f0307ca0291d" results: {completeBeforeCancel: false,…} cancelled: false completeBeforeCancel: false failed: true finished: true messages: [{level: "error", genericMessage: "Database restored", contextKey: null, contextualMessage: null}] taskId: "TR_SYSACTION_18eb08e5-ddc8-4e9f-8aa6-f0307ca0291d" threadName: "Temporary Resource Timeout for : SYSACTION_18eb08e5-ddc8-4e9f-8aa6-f0307ca0291d"

ma.log Snippet
INFO 2019-11-06T10:01:21,617 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.restore:319) - Attempting to restore database backup from: /opt/mango/backup/core-database-MYSQL-Nov-05-2019_112209.zip INFO 2019-11-06T10:01:29,549 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.restoreMysqlToDatabase:513) - Backup restored successfully formango in localhost:3306

Vulnerabilities

DepShield reports that this application's usage of commons-collections:commons-collections:3.2.1 results in the following vulnerability(s):

• (CVSS 9.8) [CVE-2017-15708] In Apache Synapse, by default no authentication is required for Java Remote Meth...
• (CVSS 7.5) [CVE-2015-6420] Serialized-object interfaces in certain Cisco Collaboration and Social Media; En...

Occurrences

commons-collections:commons-collections:3.2.1 is a transitive dependency introduced by the following direct dependency(s):

• net.sf.json-lib:json-lib:2.4
        └─ commons-collections:commons-collections:3.2.1

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

When adding or removing points from an eGauge BACnet device, the mapping of subsequent BACnet values further down in the memory map seemed to change, causing values to be assigned to the wrong point. According to BACnet, variables must be mapped according to the variable name, not its offset address. This way, when enabling or disabling a data variable that appears earlier in the memory map, the mapping of subsequent values will not be shifted due to the new map. If this is unrelated to the above issue, then I recommend we create a new issue.

This was brought up in this thread: https://forum.infiniteautomation.com/topic/4461/mango-api-post-rest-v2-point-values-import-point-values

As the model for the POST /rest/v2/point-values endpoint was not annotated with an @ ApiParam so swagger defaulted to using some model from somewhere else. Generally people can use one of the GET endpoints to get a decent example of what they would need to submit, but in this case none of the GET endpoints return something adhering to this model!

For Analog input and Analog output type points there should be inputs for the Multiplier and Additive settings.

when we write pointValueTime to one point, the value is a long string, the datatype is ALPHANUMERIC, not written successfully. but we reduce the length of the input value, the written is successful。when we set a long string value on the page to the point, the error log is "com.infiniteautomation.tsdb.IasTsdbRecordSizeException: Data length can not exceed 65536".

Are these two situations related? And whether the maximum data length is limited

The new data point selector is snazzy in providing tag access and many columns to sort / filter by, as well as make multiple selections at once. This could enable people to efficiently create the static watch list they are seeking in the watch list builder.

This was found on POST v2/data-sources but would apply to all create vo endpoints. If the user supplies an id in the JSON then the DAO layer will think it is updating a vo and try to extract it from the database and fail with a NPE at:

java.lang.NullPointerException: null
	at com.serotonin.m2m2.db.dao.AbstractDao.update(AbstractDao.java:263) ~[classes/:?]
	at com.serotonin.m2m2.db.dao.AbstractDao.update(AbstractDao.java:1) ~[classes/:?]
	at com.serotonin.m2m2.db.dao.AbstractBasicDao.update(AbstractBasicDao.java:628) ~[classes/:?]
	at com.serotonin.m2m2.db.dao.AbstractBasicDao.save(AbstractBasicDao.java:538) ~[classes/:?]
	at com.serotonin.m2m2.db.dao.AbstractBasicDao.save(AbstractBasicDao.java:526) ~[classes/:?]
	at com.serotonin.m2m2.db.dao.DataSourceDao.updateDataSource(DataSourceDao.java:187) ~[classes/:?]
	at com.serotonin.m2m2.db.dao.DataSourceDao.saveDataSource(DataSourceDao.java:166) ~[classes/:?]
	at com.serotonin.m2m2.rt.RuntimeManagerImpl.saveDataSource(RuntimeManagerImpl.java:323) ~[classes/:?]
	at com.infiniteautomation.mango.spring.service.DataSourceService.insert(DataSourceService.java:61) ~[classes/:?]
	at com.infiniteautomation.mango.spring.service.DataSourceService.insert(DataSourceService.java:1) ~[classes/:?]
	at com.infiniteautomation.mango.spring.service.AbstractVOService.insertFull(AbstractVOService.java:195) ~[classes/:?]
	at com.infiniteautomation.mango.rest.v2.DataSourcesRestController.save(DataSourcesRestController.java:123) ~[?:?]

As is noted in this thread, it is not intuitive one must use the enter key to create states for a new multistate point: https://forum.infiniteautomation.com/topic/4332/multistate-data-point-in-virtual-source

Further, if one adds a state, selects it as a start value, and then deletes the state from the state list, this will allow one to save. But, subsequent edits of the point will not save because the start value isn't found in the populated start value drop down list.

It has been seen that making a very large Point Value Rollup request via the API can cause Mango to crash, I investigated to see if there is a memory leak but this does not appear to be the case. It seems that a large amount of garbage is generated during these calls and is likely the culprit.

One suggestion was to create a system setting or use the existing UI setting (stored in the json data store) to do a quick calculation that would prevent returning too many values. However we need to consider that non web UI requests will be made to this endpoint so a limit for what the browser can handle will not be the same as what the limit for the max of the Mango server will be.

{
  "xid" : "mangoUI-settings",
  "name" : "UI Settings",
  "modelType" : "JSON_DATA",
  "validationMessages" : [ ],
  "readPermission" : "",
  "editPermission" : "edit-ui-settings",
  "publicData" : true,
  "jsonData" : {
    "pointValuesLimit" : 10000
  },
  "id" : 3
}

Add help to the new UI.

The REST controller doesn't provide feedback as to what was compiled successfully only what failed. On success the compiled-templates call is made to the REST api but this isn't displayed anywhere. It would be good to display this info in a table much like the legacy UI did.

We need the ability to export various types of VOs via RQL queries in a format that is easily downloadable and formatted exactly how a Mango import expects. The endpoints are all at GET ../export

REST Endpoints Needed

• Advanced Schedules
• Data Points
• Data Sources
• Event Detectors
• Excel Report Templates
• Global Scripts
• Mailing Lists
• Maintenance Events
• Publishers
• Users
• Watchlists

The UI should have a button on any list/table that will then use the RQL filter on the table to get the export file for download. Also any edit view should have a button that makes an RQL query xid=for_export to get the single item for download.

As a general rule use the export icon and place in top right edit view for single download or top right of table header in download all view.

UI Needed

• Data Sources

• Add download all button (Hover text to say Export all)
• Change existing export buttons to use new endpoints /export and /export-with-points

• Advanced Schedules

• Download all from list
• Download one by XID in edit window

• Data Points

• Add icon to table that uses RQL to filter

• Event Detectors

• Button to download all from list in edit view
• Button to download single in edit event detector view

• Excel Report Templates

• Download all from list
• Download one by XID in edit window

• Global Scripts

• Download all from list
• Download one by XID in edit window

• Mailing Lists

• Download all from list
• Download one by XID in edit window

• Maintenance Events

• Download all from list
• Download one by XID in edit window

• Publishers

• Add download using RQL from table
• Download one by XID in edit window

• Users

• Add download using RQL from table

• Watchlists

• Add download from table using filter

When testing please export and import the files to ensure they are formatted correctly before checking off a box.

If I set a value to the phase shift field in the new UI for a Numeric data point with a sinusoidal change type on a virtual data source, I see that save as 0 in the code and then upon reopening the page nothing is displayed in the phase shift field.

When I switched back to Mango 2.0.6, I once again lost all my background images on the graphical pages. Additionally, some of the graphical binary points did not render the correct .png image. These same issues occurred when I upgraded from 2.0.6 to 2.2.0. Actually, I've been having these problems every time I have upgraded, even before InfiniteAutomation purchased m2m2, back in the 1.12.x days.

We removed the units from the serialization of the text renderer in Mango 3.6.0. This means that the legacy report lost the units and fails on execution.

The solution is to upgrade the reports table and add units to it and ensure they are set on the renderer.

ERROR 2019-04-24T10:34:52,021 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$WorkItemRunnable.run:679) - Error in work item 
java.lang.NullPointerException: null
	at com.serotonin.m2m2.view.text.AnalogRenderer.getText(AnalogRenderer.java:106) ~[mango-3.6.0-beta.3.jar:?]
	at com.serotonin.m2m2.view.text.AnalogRenderer.getTextImpl(AnalogRenderer.java:100) ~[mango-3.6.0-beta.3.jar:?]
	at com.serotonin.m2m2.view.text.BaseTextRenderer.getText(BaseTextRenderer.java:81) ~[mango-3.6.0-beta.3.jar:?]
	at com.serotonin.m2m2.view.text.TextRenderer.getText(TextRenderer.java:58) ~[mango-3.6.0-beta.3.jar:?]
	at com.serotonin.m2m2.reports.web.ReportChartCreator$StreamHandler.startPoint(ReportChartCreator.java:619) ~[?:?]
	at com.serotonin.m2m2.reports.ReportDao.reportInstanceDataNoSQL(ReportDao.java:710) ~[?:?]
	at com.serotonin.m2m2.reports.web.ReportChartCreator.createContent(ReportChartCreator.java:138) ~[?:?]
	at com.serotonin.m2m2.reports.web.ReportWorkItem.execute(ReportWorkItem.java:182) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$WorkItemRunnable.run(BackgroundProcessingImpl.java:676) ~[mango-3.6.0-beta.3.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

Upon setting a value to my virtual point, the set point window stayed open, and as I mashed enter I set that point more and more times. I think it should close with each set, but could be interested in arguments on other use cases.

Currently all users can see all Maintenance events that are raised. The restrictions need to be done in the query by implementing the joinTables and joinPermisssions methods in the event type definition.

when a meta datasource initializing,if occer Point disabled or does not exist
com.serotonin.m2m2.rt.script.DataPointStateException
at com.serotonin.m2m2.rt.script.ScriptExecutor.convertScriptContext(ScriptExecutor.java:64)

then,the datasource status actually is not initializing sucess,it need back to stoped status,but it's status is running.

The full output of the 'source' is included which means every part of the data point model is a column.

There are many models and this would take time to re-code to have simpler models for the CSV output. My first thought here is to use a JsonView

Create multistate point with no change change type, define states and a start value and save it. Edit it again and no states appear in the state list!

I was only able to observe this with the no change change type.

Evinced by creating a schedule event that is currently active and going to events page and looking for it.

PreparedStatementCallback; SQL [INSERT INTO selectedWatchList (userId, watchListId) VALUES (?,?)]; Unique index or primary key violation: "PRIMARY KEY ON PUBLIC.SELECTEDWATCHLIST(USERID)"; SQL statement:
INSERT INTO selectedWatchList (userId, watchListId) VALUES (?,?) [23505-181]; nested exception is org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "PRIMARY KEY ON PUBLIC.SELECTEDWATCHLIST(USERID)"; SQL statement:
INSERT INTO selectedWatchList (userId, watchListId) VALUES (?,?) [23505-181]

I was cleaning up some watchlist's that had no points in them any longer and got this error.

One would expect the point locator of a multistate virtual point to export its values array into its point locator during a json export. It appears json export isn't implemented for either multistate random or multistate increment.

Oddly, virtual multistate points only let you define custom value sets if you've defined a change rule. Otherwise, you just get a good ole' multistate.

A result of this is that it's impossible to import a multistate virtual point that has a change type.

The project could not be analyzed because of build errors. Please review the error messages here. Another build will be scheduled within 24 hours. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

This data source needs help in the new UI

Vulnerabilities

DepShield reports that this application's usage of commons-beanutils:commons-beanutils:1.8.0 results in the following vulnerability(s):

• (CVSS 7.5) [CVE-2014-0114] Improper Input Validation

Occurrences

commons-beanutils:commons-beanutils:1.8.0 is a transitive dependency introduced by the following direct dependency(s):

• net.sf.json-lib:json-lib:2.4
        └─ commons-beanutils:commons-beanutils:1.8.0

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

As you can see on the images below, the reports are generating a "infinite" line when the data have a gap:

Correct data on point details:
http://i49.tinypic.com/347k9hy.jpg

Wrong data on report instance:
http://i45.tinypic.com/14v4b3l.jpg

Report configuration:
http://i46.tinypic.com/wvv7yr.png

Derby database:
https://www.dropbox.com/s/5ygwyoq5wzqc7w3/m2m2db.zip

Thanks,

When running a report on a point, if it is set to useUnitsAsSuffix with °F, Fahrenheit, or psi as the unit will cause issues when viewing a run report. For Fahrenheit, the units will be presented as [K?], and psi will throw a system exception. After some initial trying to solve the issue with Fahrenheit, it seems like the TextRenderer --> renderedUnit is written into the ReportInstancePoints table correctly, but when read back out any label specified for Fahrenheit in m2m2/util/UnitUtil (I wrote one to test, and tried psi because it's in that file... but btu works...) has disappeared and [K?] is all that remains.

The project could not be analyzed because of build errors. Please review the error messages here. Another build will be scheduled within 24 hours. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

It was asked if we could provide an option for specifying the content for the header of the value column for each point. Such as it's name, name + device Name etc.

This is feasible but we will run into problems when the headers are not unique as the data is processed by creating a CSV schema before pushing the values through it. One rule of the schema is that headers must be unique.

The project could not be analyzed because of build errors. Please review the error messages here. Another build will be scheduled within 24 hours. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

modbus point multiplier and additive sometimes cause double accuracy problem,return like 7.00000006,suggest use BigDecimal instead

In the new UI for the help it says:

Selecting a Send alarm emails alarm level above Ignore will cause these notifications to be sent when the event's alarm level is greater than or equal to the user's minimum alarm level,

I think we should add something like: "Set to 'Ignore' for no emails to be sent"

modbus module Log I/O enabled,when Files to keep change from big number to small,for example ,Files to keep change from 5 to 3,will cause ERROR 2015-07-14 11:47:15,399 (com.serotonin.log.RollingIOLog.sizeCheck:85) - java.nio.file.NoSuchFileException: D:\mango2\mango_automation_2.5.2_enterprise_demo\logs\modbusIO-3.log.3,

suggest,when datasource initializing,if the exist iolog count is more then configuration,delete the more.

another bug,the iolog file is not thorough closed when datasource stoped, you can enabled/disabled many times,then when the iofiles's sizes reached (File Size (in MB)),will cause iofiles being used error,can't rename it.

I added a custom image pack to: Mango ▸ overrides ▸ web ▸ modules ▸ sstGraphics ▸ web ▸ graphics ▸ custom ▸ LEDsmall. It was only accessible to Graphic Views when I place the files directly under the module.

The project could not be analyzed because of maven build errors. Please review the error messages here. Another build will be scheduled within 24 hours. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

• Add ability to export via CSV
• Add bulk endpoints that work like the data point bulk actions

It produces index out of bounds exceptions like this one and exceptions out of the data source's poll:

ERROR 2019-08-12T11:50:46,552 (com.serotonin.m2m2.util.timeout.TimeoutTask.run:61) - Uncaught Task Exception
java.lang.ArrayIndexOutOfBoundsException: -1
at com.serotonin.m2m2.virtual.rt.IncrementMultistateChangeRT.change(IncrementMultistateChangeRT.java:56) ~[?:?]
at com.serotonin.m2m2.virtual.rt.VirtualPointLocatorRT.change(VirtualPointLocatorRT.java:36) ~[?:?]
at com.serotonin.m2m2.virtual.rt.VirtualDataSourceRT.doPoll(VirtualDataSourceRT.java:52) ~[?:?]
at com.serotonin.m2m2.virtual.rt.VirtualDataSourceRT.doPollNoSync(VirtualDataSourceRT.java:33) ~[?:?]
at com.serotonin.m2m2.rt.dataSource.PollingDataSource.scheduleTimeoutImpl(PollingDataSource.java:183) ~[mango-3.6.4.jar:?]
at com.serotonin.m2m2.rt.dataSource.PollingDataSource$1.scheduleTimeout(PollingDataSource.java:85) ~[mango-3.6.4.jar:?]
at com.serotonin.m2m2.util.timeout.TimeoutTask.run(TimeoutTask.java:59) ~[mango-3.6.4.jar:?]
at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.6.4.jar:?]
at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.6.4.jar:?]
at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:336) ~[mango-3.6.4.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

the DataPointRT savePointValueDirectToCache method was modified to enable creating event tasks for values in certain situations, but the old helper method defaults to never. This is not exposed on something like the /rest/v2/point-values endpoints, so the POST endpoint for instance never fires event notify items.

This is a related issue in the core: MangoAutomation/ma-core-public#1322.

1. Some modules doesn't have the "lib" directory (ex: watchlist, reports, etc.) making the build.xml script fail.

2. Its not a better option to replace the coreHome property value from "/Users/Matthew/git/ma-core-public/Core" to "../../ma-core-public/Core"?

   With this change this structure will work fine:

   • Root folder
     • ma-core-public
     • ma-modules-public
     • ...

Thanks,