davetimmins / anywhere.arcgis Goto Github PK

Hi Dave,

This isn't a bug in your code, but I'm wondering if you've seen it or have a workaround.

I'm querying an entire hosted feature layer containing about 800K features, I'm requesting the GlobalId field, and another unique identifying field.

Most of the time this is working fine, but sometimes, and at random intervals (e.g. different number of loops each time), I'm getting the following error (some bits obscured by xxxx):

2019/03/19 11:56:11.907|INFO|Anywhere.ArcGIS.PortalGatewayBase|Exceeded query transfer limit (found 2000), batching query for rest/services/xxxx/FeatureServer/0/query - loop 219 |
2019/03/19 11:56:13.081|INFO|Anywhere.ArcGIS.PortalGatewayBase|Exceeded query transfer limit (found 2000), batching query for rest/services/xxxx/FeatureServer/0/query - loop 220 |
2019/03/19 11:56:14.239|INFO|Anywhere.ArcGIS.PortalGatewayBase|Exceeded query transfer limit (found 2000), batching query for rest/services/xxxx/FeatureServer/0/query - loop 221 |
2019/03/19 11:56:16.567|INFO|Anywhere.ArcGIS.PortalGatewayBase|Exceeded query transfer limit (found 2000), batching query for rest/services/xxxx/FeatureServer/0/query - loop 222 |
2019/03/19 11:56:19.389|INFO|Anywhere.ArcGIS.PortalGatewayBase|Exceeded query transfer limit (found 2000), batching query for rest/services/xxxx/FeatureServer/0/query - loop 223 |
2019/03/19 11:56:20.775|INFO|Anywhere.ArcGIS.PortalGatewayBase|Exceeded query transfer limit (found 2000), batching query for rest/services/xxxx/FeatureServer/0/query - loop 224 |
2019/03/19 11:56:27.789|INFO|Anywhere.ArcGIS.PortalGatewayBase|Exceeded query transfer limit (found 2000), batching query for rest/services/xxxx/FeatureServer/0/query - loop 225 |
2019/03/19 11:57:23.219|ERROR|xxx.DetectChanges|System.InvalidOperationException: Code 400: Cannot perform query. Invalid query parameters..
Unable to perform query. Please check your parameters.
   at Anywhere.ArcGIS.PortalGatewayBase.Get[T,TRequest](TRequest requestObject, CancellationToken ct)
   at Anywhere.ArcGIS.PortalGatewayBase.BatchQuery[T](Query queryOptions, CancellationToken ct)
   at xxx.DetectChanges.PopulateGlobalIdsIfNecessary(InputFileInstance inputFileInstance) in /Users/leigh/dev/xxxx/xxxx/DetectChanges.cs:line 835 Code 400: Cannot perform query. Invalid query parameters..
Unable to perform query. Please check your parameters.|

The feature service supports pagination.

I suspect I may be overloading the server. I'm wondering about inserting a configurable delay here after say 50 iterations or something like that.

Have you seen anything like this before?

I create this method, that connects to some Esri MapServer and gets all geometries and creates a GEOJSON:

public async Task<FeatureCollection<IGeoJsonGeometry>> QueryAllFromMapserviceLayer(string server, string service, string layer)
        {
            //http://app.anp.gov.br/arcgis/rest/services/ANP_Public_Confidential/MapServer/0/query

            var gateway = new PortalGateway(server); //"http://app.anp.gov.br/arcgis/"

            var query = new Query((service + "/MapServer/" + layer).AsEndpoint());
            query.ReturnGeometry = true;

            FeatureCollection<IGeoJsonGeometry> featureCollectionGeoJson = null;
            int? Wkid = 0;
            if (await this.GetLayerType(server, service, layer) == "esriGeometryPolyline")
            {
                var result = await gateway.BatchQuery<Anywhere.ArcGIS.Common.Polyline>(query);
                Wkid = result.SpatialReference.Wkid;
                //var count = result.Features.Count();
                featureCollectionGeoJson = FeatureCollectionExtensions.ToFeatureCollection<Anywhere.ArcGIS.Common.Polyline>(result.Features.ToList());
                
            }

            if (await this.GetLayerType(server, service, layer) == "esriGeometryPoint")
            {
                var result = await gateway.BatchQuery<Anywhere.ArcGIS.Common.Point>(query);
                Wkid = result.SpatialReference.Wkid;
                //var count = result.Features.Count();
                featureCollectionGeoJson = FeatureCollectionExtensions.ToFeatureCollection<Anywhere.ArcGIS.Common.Point>(result.Features.ToList());
            }

            if (await this.GetLayerType(server, service, layer) == "esriGeometryPolygon")
            {
                var result = await gateway.BatchQuery<Anywhere.ArcGIS.Common.Polygon>(query);
                Wkid = result.SpatialReference.Wkid;
                //var count = result.Features.Count();
                featureCollectionGeoJson = FeatureCollectionExtensions.ToFeatureCollection<Anywhere.ArcGIS.Common.Polygon>(result.Features.ToList());

                //por algum motivo ao gerar o geojson algumas features estão vindo zuadas... essa trecho remove as features zuadas do poligono
                foreach(var temp in featureCollectionGeoJson.Features)
                {
                    PointCollectionList fixedCoords = new PointCollectionList();
                    foreach (var coord in ((GeoJsonPolygon)temp.Geometry).Coordinates)
                    {
                        if (coord.Count > 3)
                            fixedCoords.Add(coord);
                    }

                    ((GeoJsonPolygon)temp.Geometry).Coordinates = fixedCoords;
                }
            }

            foreach (var geom in featureCollectionGeoJson.Features)
            {
                geom.Id = Guid.NewGuid().ToString();
            }
            featureCollectionGeoJson.Type = "FeatureCollection";
            featureCollectionGeoJson.CoordinateReferenceSystem = new Crs { Type = "name", Properties = new CrsProperties { Name = Wkid.HasValue ? "EPSG:" + Wkid : null} };

            return featureCollectionGeoJson;
        }

I used the following parameters:

• server=http://app.anp.gov.br/arcgis/
• service=ANP_Public_Confidential
• layer=2

The problem is that layer has some multilines then the geojson generated is invalid... Not represent the same features that original.

I think the problem is in this line:
featureCollectionGeoJson = FeatureCollectionExtensions.ToFeatureCollection<Anywhere.ArcGIS.Common.Polyline>(result.Features.ToList());

I think the lib need to create a new model for these kinds of geometries.

Need a way to Query with output statistics.

Is there any sample regarding ArcGIS Online? I always receive HTTP 404 using PortalGateway and ArcGISOnlineGateway. The unit tests are not functional?

This query can return any shape, but it is currently coded to only handle deserializing a Point, this causes any result that is any other shape, such as a polygon for Parcels, to not get deserialized.

Hi Dave,
Can you please suggest how I might be able to prevent it from swallowing TaskCanceledException, and FormatException?

I tried inheriting from PortalGateway but the Post method is not virtual.
I thought about injecting a custom implementation of ILog that would just throw the Exception in the WarnException() method but the ILog interface is internal.

I see there is a compiler directive at https://github.com/damianh/LibLog/blob/master/src/LibLog/ILog.cs#L8 that might allow me to do what I'm trying to accomplish but I thought I would reach out for some advice before I go any further.

Lastly, do you have any plans to implement Microsoft.Extensions.Logging.Abstractions in Anywhere.ArcGIS to replace ILog?

    [DataMember(Name = "domain")]
    public string Domain { get; set; }

Domain is a JSON object, not a string, so when queries are performed on feature servers with domain information, it causes a JSON parsing error.

Perhaps just remove the Domain property?

In the ArcGIS REST API, there is an ability to query records related to a particular objects. This appears to be the one thing I'm missing in this library.

https://developers.arcgis.com/rest/services-reference/query-related-records-feature-service-.htm

Hi,

I have ran into an interesting problem when querying an ArcGIS server with a complex geometry...

                Query query = new Query(service.AsEndpoint())
                {
                    SpatialRelationship = SpatialRelationshipTypes.Intersects,
                    Geometry = parcel.Geometry,
                    ReturnGeometry = true,
                    OutputSpatialReference = new Anywhere.ArcGIS.Common.SpatialReference { Wkid = wkid },
                };

In my case the text representation of this query is 85,249 characters but fails not due to a server issue but Uri.TryCreate reporting its too long to be valid at line 706 of PortalGatewayBase.cs. After some quick testing the limit seems to be 65,519 characters.

To workaround this problem i have simply switched some code around, from this

            **bool validUrl = Uri.TryCreate(url, UriKind.Absolute, out Uri uri);**
            if (!validUrl)
            {
                throw new HttpRequestException(string.Format("Not a valid url: {0}", url));
            }

            if (url.Length > MaximumGetRequestLength)
            {                
                _logger.DebugFormat("Url length {0} is greater than maximum configured {1}, switching to POST.", url.Length, MaximumGetRequestLength);
                return await Post<T, TRequest>(requestObject, ct).ConfigureAwait(false);
            }

To

            if (url.Length > MaximumGetRequestLength)
            {                
                _logger.DebugFormat("Url length {0} is greater than maximum configured {1}, switching to POST.", url.Length, MaximumGetRequestLength);
                return await Post<T, TRequest>(requestObject, ct).ConfigureAwait(false);
            }

            **bool validUrl = Uri.TryCreate(url, UriKind.Absolute, out Uri uri);**
            if (!validUrl)
            {
                throw new HttpRequestException(string.Format("Not a valid url: {0}", url));
            }

I haven't found a way to increase the maximum length the .NET framework will support. I'm not if you'd even want to drop the URI validation. I will submit a pull request if you approve of this change.

Cheers.

I found two issues when testing BatchQuery using Azure Functions 1.x against an AGOL Hosted Feature Service:

1. I get an error like this:

"message": "Exception while executing function: -> Nullable object must have a value.",
"errorDetails": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: ---> System.InvalidOperationException : Nullable object must have a value.\r\n at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)\r\n at async Anywhere.ArcGIS.PortalGatewayBase.BatchQuery[T](Query queryOptions,CancellationToken ct)\r\n
..."

I was able to fix this by changing PortalGatewayBase line 242 from:

exceeded = result.ExceededTransferLimit.HasValue && innerResult.ExceededTransferLimit.Value;

to

exceeded = result.ExceededTransferLimit.HasValue && innerResult.ExceededTransferLimit.HasValue && innerResult.ExceededTransferLimit.Value;

2. I wasn't actually getting back all the results from the feature service. I was expecting 5000+ features, but I was only getting the results from the first query. I was able to fix this by changing PortalGatewayBase line 241 from:

result.Features.ToList().AddRange(innerResult.Features);

to

result.Features = result.Features.Concat<Feature<T>>(innerResult.Features);

I didn't have time to test against various types of AGS services so it is possible that these changes were unique to my setup, but wanted to send them along just in case.

Thanks for sharing this great repo!

I came across a problem then trying to generate a token from a 10.2.2 server:

Exception in generating tokenFailed to generate a token for the user.Client ID is required for long expiration tokens

I have determined that specifying the referrer will resolve this problem and will submit a PR. There's several ways of fixing this but chose to specify the referrer when also specifying the username and password. This property, by default, is null; and so the only thing this code is doing is setting this property. There would be no code change to the TokenProvider class taking this approach.

Hi,

Just a question - how is it possible to add HTTP Referrer Header in order to access an ArcGIS MapServer service?

I cannot locate something like, in the examples...

Thanks,
George J.

I published a hosted feature layer from Pro to AGOL using AGOL defined scales for when the layer will draw. The resulting layer has a Min. Scale property of 577790.554289. When I try to batch query the layer, I get an error:
System.Private.CoreLib: Exception while executing function: XXXX. Newtonsoft.Json: Input string '577790.554289' is not a valid integer. Path 'minScale', line 1, position 1442.

Searching the code for MinimumScale, there are 3 places in the code. Two of them are defined as double? and one is defined as int (ServiceLayerDescriptionResponse class). Seems like MinimumScale and MaximumScale on this class should be changed to double?.

The arcserver usually return maximum 1000 records. How can I do a query to get all the features?

I wish I came across this project earlier in my development phase. I created something similar but not nearly as good as what I'm seeing here. One of my requirements is to hide WFS and WMS services behind and AGS API. If I'm understanding this project correctly, I could fill that requirement is there was such a thing as an OgcGateway. I don't have a question per-se, but would love to hear your comments and thoughts on this.

I have a service that is time enabled, and has an interval of roughly 12 days. It returns this:

"timeInfo": {
	"timeExtent": [
		1546300800000,
		253402214400000
	],
	"timeReference": null,
	"timeRelation": "esriTimeRelationOverlaps",
	"defaultTimeInterval": 0.39425893413496776,
	"defaultTimeIntervalUnits": "esriTimeUnitsMonths",
	"defaultTimeWindow": 0,
	"defaultTimeWindowUnits": "esriTimeUnitsMonths",
	"hasLiveData": false
},

On which DescribeServices() throws an error: One or more errors occurred. (Input string '0.39425893413496776' is not a valid integer. Path 'timeInfo.defaultTimeInterval', line 1, position 3188

Guys how I can convert this result to GeoJson string?

  var gateway = new PortalGateway(server); //"http://app.anp.gov.br/arcgis/"

            var query = new Query((service + "/MapServer/" + layer).AsEndpoint())
            {
                Where = "1=1"
            };
            query.ReturnGeometry = true;
            var result = await gateway.BatchQuery<Anywhere.ArcGIS.Common.Point>(query);
            var count = result.Features.Count();

You have a default MaximumGetRequestLength that switches to POST.

It checked this length then it adds the token and a few other things.

My token was 193 letters.

With the &f=json&token= this came to 207 extra letters.

It would be good to check the url after this is added or just reduce the max default max to 1800 or something reasonable (I have no idea the range of token lengths).

Workaround:

var secureGateway = await PortalGateway.Create("https://geoportal.example.com/server/", "username","password1");
secureGateway.MaximumGetRequestLength = 1000;

Every Ersi Geocoder service I ran "findAddressCanidates" query against is expecting a "SingleLine" parameter, the "Text" parameter returns no results when searching for a known address or parcel number, where as the "SingleLine" returns the expected results.

Even the ESRI world geocode service expecting this parameter for this query
URL: "http://geocode.arcgis.com/arcgis"
EndPoint: "/World/GeocodeServer"

When a token expires and it needs to be renewed, a cryptographic exception is thrown with text of Bad Length. The code below reproduces the issue. The actual issue is that GenerateToken gets double encrypted on renewal because the TokenRequest property is replaced inside Token Provider

Thank you!

GenerateToken t = new GenerateToken("user", "password");
var publicKey = new PublicKeyResponse
{
PublicKey = "10001",
Mod = "a92d33f398ef1d71c616730807b5722312ed42b94f0891299281abcc4fb7350b5eab970a6e52953fc695295735b9fd347b8b3fa7aa2f12c4a3ed423875aa276d"
};
GenerateToken e = new RsaEncrypter().Encrypt(t, publicKey.Exponent, publicKey.Modulus);

        GenerateToken b = new RsaEncrypter().Encrypt(e, publicKey.Exponent, publicKey.Modulus); 

According to Esri the REST API has a property for LocationType:

locationType
Specifies if the output geometry of PointAddress matches should be the rooftop point or street entrance location. Valid values are rooftop and street. The default value is street.

This property is not defined on the GeocodeOperation.

I intend to add it and submit a PR

There is a method to geocode, which is using the "find" query, however there is no method being exposed to handle the "findAddressCanidates" query, which uses the CustomGeocode class

The method:
public List<Point> Points { get { return this.Select(point => point != null ? new Point { X = point.First(), Y = point.Last() } : null) .Where(p => p != null) .ToList(); } }

in IGeometry returns points with coordinates {X=x,Y=z} for requests with Z values.

We ran into an issue if MapServer and GeoCode server URLs are on 2 different physical servers. In the current implementation, base the path of the Map-Server URL (RootUrl) and the relative path of GeoCode Server URL's are combined to get the final URL to get GeoCode. It would be great if we have the ability to override RelativeUrl property and BuildAbsoluteUrl method as virtual in ArcGISServerEndpoint. This will give us the flexibility to have an absolute URL for Geocode Server. Also, it would be great if you can update SingleInputCustomGeocode constructors to pass IEndPoint instead of ArcGISServerEndpoint. Also, it would be great if we can add another constructor to the same class to pass the endpoint as is to the base class instead of letting the base class create a new instance of ArcGISServerEndpoint.

Thanks a lot for all your efforts.

This line of code: https://github.com/davetimmins/Anywhere.ArcGIS/blob/master/src/Anywhere.ArcGIS/Common/IGeometry.cs#L634 causes an exception if I union two extents when both of them have spatial reference of null. Any chance this could be addressed?
Thank you very much for your efforts.

Hi @davetimmins, first of all thanks a lot for this project, I think it's a really useful package for everyone's working with ArcGIS Rest Services.
I installed Anywhere.ArcGIS version 1.2,1 on my project which is developed with .NET Framework 4.6.2.
On compilation everything's fine, but while running it throws an error like this:

Could not load file or assembly 'System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file.
System.IO.FileNotFoundException"

Have you ever experienced this issue? I saw you're targeting the package for netstandard 2.0, so .NET 4.6.2 should be fine, right?

I think it should be an issue of Microsoft and Visual Studio:
https://github.com/dotnet/corefx/issues/23306

But I wanted to know if someone is having this issue or it's just me.
Anyway thanks for your work!

The exception is thrown from here in the code:
https://github.com/davetimmins/Anywhere.ArcGIS/blob/dev/src/Anywhere.ArcGIS/PortalGatewayBase.cs#L392

A simple fix is to replace result.SetExpected(edits); with result?.SetExpected(edits);

The exception can be thrown in the case that a Cancellation is requested.

Hello, Dave.
We found an issue with AsRequestQueryString. If we do a spatial query on a complex shape, it will bust 65k limit of EscapeString in .NET Framework. Unfortunately, it switches to POST based on the serialized value, but it can never get that length based on this exception.

Do you have any suggestions or a fix?
Thank you.

Most cases, I will need to have either a Point or a Polygon when querying.
In both cases, I end up with a IGeometry.

JsonConvert.DeserializeObject does support passing the type at runtime.
Forcing to have the type at compile time makes it quite complex for a simple fetch.

It would be interesting to have something like:

var layer = await secureGateway.DescribeLayer(endpoint);
secureGateway.Query(queryOptions, layer.GeometryType);

Please add these two to the Field class in Query.cs:

    [DataMember(Name = "nullable")]
    public bool Nullable { get; set; }

    [DataMember(Name = "editable")]
    public bool Editable { get; set; }

Need to abstract out the Feature to not include a IGeometry for basic table ops.

Thanks for the great work, but I found drawingInfo is missing from type ServiceLayerDescriptionResponse， would you add it?

My test code is:

var gateway = new ArcGISOnlineGateway(
    "https://sampleserver6.arcgisonline.com/arcgis"
);
var response = await gateway.DescribeLayer(endpoint);
var endpoint = new ArcGISServerEndpoint(
    "Military/MapServer/3"
);

and the service with json response is: https://sampleserver6.arcgisonline.com/arcgis/rest/services/Military/MapServer/3?f=pjson