wso2 / shared-analytics Goto Github PK

Description:
When the client IP is a local, an empty string will be return as the city and when the DB is Oracle, empty is again considered as null.

Affected Product Version:
APIM-Analytics 2.1.0

Loganalyer log4j appender self logging handle using LogLog instead of using System.out and System.err .
We can use the LogLog class for handle the self logging in log appender.
org.apache.log4j.helpers.LogLog

Moved from https://wso2.org/jira/projects/SHAREDANLY/issues/SHAREDANLY-2

Description:
Duplicate entry for 'PRIMARY' key is coming when resolving the country in GEO_LOCATION_DATA.IP_LOCATION table [1].

ERROR {org.wso2.analytics.is.siddhi.extension.IpToCountryExtension} - Exception when resolving the country for given IP : 105.0.1.28 {org.wso2.analytics.is.siddhi.extension.IpToCountryExtension}
org.wso2.carbon.analytics.shared.geolocation.exception.GeoLocationResolverException: Error while saving the location to database
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.saveLocation(LocationResolverRdbms.java:161)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocationFromIp(LocationResolverRdbms.java:138)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocation(LocationResolverRdbms.java:61)
at org.wso2.carbon.analytics.shared.geolocation.impl.GeoLocationResolverUDFWithImprovedCache.getCountry(GeoLocationResolverUDFWithImprovedCache.java:47)
...
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '105.0.1.28' for key 'PRIMARY'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

It seems like this Duplicate entry for the primary key is coming when inserting data to IP_LOCATION table using the following SQL [2].
SQL_INSERT_LOCATION_INTO_TABLE = "INSERT INTO IP_LOCATION (ip,country_name," +"city_name) VALUES (?,?,?)";

And IP is the primary key for the IP_LOCATION table.

CREATE TABLE IP_LOCATION(ipvarchar(100) NOT NULL,country_namevarchar(200) DEFAULT NULL,city_name varchar(200) DEFAULT NULL, PRIMARY KEY (ip));

We can check whether the duplicate entry exception is there in a try-catch [3] and ignore that as the IP location is already inserted from the other node.

[1].

[2].

[3].

Suggested Labels:
IS-Analytics, HA

Affected Product Version:
IS-Analytics-5.3.0

OS, DB, other environment details and versions:
DB- MySQL

Steps to reproduce:

1. Configure an IS Analytics HA deployment.
2. Invoke from different IPs in a short period.
   (This might not reproduce always)

Below page has "localhost" hard coded which can cause hostname verification issues in certificates.

portal/modulesportal/modules/api.js

So this should be configurable. For instance it can use the hostname that can be configured in portal/configs/designer.json.

Description:
In APIM analytics 2.1.0. when geo stats are enabled the following error is throwing when the clientIp consists with local ips.

java.sql.BatchUpdateException: ORA-01400: cannot insert NULL into ("WSO2"."API_REQ_GEO_LOC_SUMMARY"."COUNTRY")
	at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:10032)

This happens because when the clientIP is a local, an empty string will be return as the country and when the DB is Oracle, empty is again considered as null.

Affected Product Version:
APIM-Analytics 2.1.0

Steps to reproduce:
Setup APIM 2.1.0 and APIM analytics 2.1.0 node locally and enable analytics in APIM side.
Enable geolocation statistic from APIM side.
Create an API and access it locally.
Run the "APIMAnalytics-APIM_GEO_LOCATION_STATS-APIM_GEO_LOCATION_STATS-batch1" from analytics side.
Use Oracle as the DB

persistInDataBase = true

Exception when resolving the country for given IP : 1.2.3.4 {org.wso2.analytics.is.siddhi.extension.IpToCountryExtension }
org.wso2.carbon.analytics.shared.geolocation.exception.GeoLocationResolverException: Error while saving the location to database
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.saveLocation(LocationResolverRdbms.java:161)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocationFromIp(LocationResolverRdbms.java:138)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocation(LocationResolverRdbms.java:61)
at org.wso2.carbon.analytics.shared.geolocation.impl.GeoLocationResolverUDFWithImprovedCache.getCountry(GeoLocationResolverUDFWithImprovedCache.java:47)
at org.wso2.analytics.is.siddhi.extension.IpToCountryExtension.execute(IpToCountryExtension.java:48)
at org.wso2.siddhi.core.executor.function.FunctionExecutor.execute(FunctionExecutor.java:92)
at org.wso2.siddhi.core.query.selector.attribute.processor.AttributeProcessor.process(AttributeProcessor.java:38)
at org.wso2.siddhi.core.query.selector.QuerySelector.processNoGroupBy(QuerySelector.java:104)
at org.wso2.siddhi.core.query.selector.QuerySelector.process(QuerySelector.java:86)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.processAndClear(ProcessStreamReceiver.java:154)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.process(ProcessStreamReceiver.java:80)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.receive(ProcessStreamReceiver.java:150)
at org.wso2.siddhi.core.stream.StreamJunction.sendData(StreamJunction.java:214)
at org.wso2.siddhi.core.stream.StreamJunction.access$200(StreamJunction.java:46)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:343)
at org.wso2.siddhi.core.stream.input.InputDistributor.send(InputDistributor.java:49)
at org.wso2.siddhi.core.stream.input.InputEntryValve.send(InputEntryValve.java:59)
at org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:51)
at org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39)
at org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:104)
at org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:146)
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher$QueueInputEventDispatcherWorker.run(QueueInputEventDispatcher.java:142)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (PRDGEOLOC.SYS_C0017586) violated
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:208)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1046)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1336)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3714)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1378)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.saveLocation(LocationResolverRdbms.java:158)
... 26 more

The following exception thrown for mapped IPv6 address with format ::ffff:192.168.2.1

java.lang.NumberFormatException: For input string: "::ffff:192"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getIpV4ToLong(LocationResolverRdbms.java:181)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocationFromLongValueOfIp(LocationResolverRdbms.java:104)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocationFromIp(LocationResolverRdbms.java:135)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocation(LocationResolverRdbms.java:61)
at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbmsTest.testGetLocation(LocationResolverRdbmsTest.java:116)

API Manager analytics are deployed in API Cloud. We have recently found an issue where log publishers are trying to publish logs to DAS but it throws an error saying connection failed etc, these logs are logged back in publisher node again and it causes a riffling effect trying to publish those errors back to DAS.
This causes API manager nodes to perform slow and causing a severe effect on API manager functionality too.

DAS node can go offline, can fail to handle incoming load etc. So we can not guarantee that DAS will always accept events from API manager publishers.
To fix this issue, API Manager log publisher should be able to configure what not to be published such as DAS connectivity issues, or logs from the particular tenant(super tenant) etc. API Manager analytics needs to be improved to configure a log filtering mechanism.

Moved from https://wso2.org/jira/projects/SHAREDANLY/issues/SHAREDANLY-4

Description:
API Manager Geo Location Stats break when client IP is in CIDR format (eg. 10.100.0.2/24).

TID: [-1] [] [2017-11-06 22:44:30,159]  WARN {org.apache.spark.scheduler.TaskSetManager} -  Lost task 2.0 in stage 356089.0 (TID 311137, platform-default-dasanalyzer-01): java.lang.Exception: Error while invoking method: getCountry, null
	at org.wso2.carbon.analytics.spark.core.udf.adaptor.UDF1Adaptor.call(UDF1Adaptor.java:64)
	at org.apache.spark.sql.UDFRegistration$$anonfun$register$25$$anonfun$apply$1.apply(UDFRegistration.scala:422)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:514)
	at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:686)
	at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
	at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: For input string: "70/32"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:492)
	at java.lang.Integer.parseInt(Integer.java:527)
	at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getIpV4ToLong(LocationResolverRdbms.java:181)
	at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocationFromLongValueOfIp(LocationResolverRdbms.java:104)
	at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocationFromIp(LocationResolverRdbms.java:135)
	at org.wso2.carbon.analytics.shared.geolocation.impl.LocationResolverRdbms.getLocation(LocationResolverRdbms.java:61)
	at org.wso2.carbon.analytics.shared.geolocation.impl.GeoLocationResolverUDF.getCountry(GeoLocationResolverUDF.java:43)
	at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.wso2.carbon.analytics.spark.core.udf.adaptor.UDF1Adaptor.call(UDF1Adaptor.java:60)
	... 21 more

Affected Product Version:

APIM Analytics 2.1.0

Steps to reproduce:

1. Enable Geo Location stats in APIM
2. Create an API and call it with X-Forwarded-For header in CIDR format.

This breaks Geo Stats and print above error log.