azure / azure-cosmos-table-dotnet Goto Github PK

CloudTableClient.GetServiceStats is broken in the Microsoft.Azure.Cosmos.Table. It was implemented in v 0.11.0-preview and was broken since then.

Repro:

        CloudStorageAccount storageAccount =
            CloudStorageAccount.Parse(connectionString);

        CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

        TableRequestOptions tableRequestOptions = new TableRequestOptions()
        {
            LocationMode = LocationMode.SecondaryOnly
        };

        ServiceStats stats = tableClient.GetServiceStats(tableRequestOptions); 

Here is a response:
<?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code>InvalidQueryParameterValue</m:code><m:message xml:lang="en-US">Value for one of the query parameters specified in the request URI is invalid. RequestId:84509fe0-7002-006b-7b5c-d11c71000000

The problem is in the TableRequestOptions copy constructor.
It does not copy LocationMode property and as a result request is sent to the primary instead of secondary.

Here is a workaround:

         CloudStorageAccount storageAccount =
            CloudStorageAccount.Parse(connectionString);

        CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

        TableRequestOptions tableRequestOptions = new TableRequestOptions()
        {
            LocationMode = LocationMode.SecondaryOnly
        };

        OperationContext operationContext = new OperationContext();
        operationContext.SendingRequest += (object sender, RequestEventArgs e) =>
        {
            string url = e.Request.RequestUri.AbsoluteUri;
            int dotIndex = url.IndexOf('.');
            string accountName = url.Substring(0, dotIndex);
            const string c_secondary = "-secondary";

            if (!accountName.EndsWith(c_secondary))
            {
                e.Request.RequestUri = new System.Uri(accountName + c_secondary + url.Substring(dotIndex));
            }
        };

        ServiceStats stats = tableClient.GetServiceStats(tableRequestOptions, operationContext);

Can not have nuget package Microsoft.Azure.Cosmos.Table and Microsoft.Azure.Search.Service in same project due the Microsoft.Azure.Cosmos.Table had a hard dependency to the Odata = 7.5. Where Microsoft.Azure.Search.Service just want the Microsoft.Spatial greater then.

NU1107: Version conflict detected for Microsoft.Spatial. Install/reference Microsoft.Spatial 7.5.3 directly to project MyProjects.AzureSearch to resolve this issue. 
 MyProjects.AzureSearch -> Microsoft.Azure.Search 10.1.0 -> Microsoft.Azure.Search.Service 10.1.0 -> Microsoft.Spatial (>= 7.5.3) 
 MyProjects.AzureSearch -> MyProjects.Core -> Microsoft.Azure.Cosmos.Table 1.0.6 -> Microsoft.OData.Core 7.5.0 -> Microsoft.Spatial (= 7.5.0).
Package restore failed. Rolling back package changes for 'MyProjects.AzureSearch'.

Please make a more lose dependency to Odata / Spatial, so you can use multiple of azures features in same project.

EntityProperty distinguishes between null values of different types. I was about to write code to support creating a property with a null GUID vs a null string vs a null Int32, etc. However, from a REST API standpoint, there is no such distinction - any null is a null, and a data type for the null is irrelevant when passed to the storage REST API. Having EntityProperty have a BooleanValue of type bool?, vs a GuidValue of type Guid? with a different PropertyType on these two instances doesn't fit the domain.

Note also that null values are never stored by Azure Table Storage, so there's no need to be able to read an EntityProperty with a null value and then worry about distinguishing a null bool? vs a null Guid? etc when reading - the service would never return a null value to begin with.

I understand that having TableEntity support properties with types like int? and Guid? is useful - I'm not suggesting any changes there. It's at the EntityProperty/DynamicTableEntity level that I think there's a problem (distinguishing null int? vs null bool? there doesn't fit the service's data model).

(Copied from Azure/azure-storage-net#548.)

Test issue

You can use the Azure portal to create tables in an Azure Cosmos DB account that contain dashes, e.g. "my-table-name". However, when I try to use that name in my Azure function project with the Microsoft.Azure.Cosmos.Table library I can no longer run the function because I get the following error:

Microsoft.Azure.WebJobs.Host: Error indexing method 'MyFunction'. Microsoft.Azure.WebJobs.Host: Validation failed for property 'TableName', value 'my-table-name'. The field TableName must match the regular expression '^[A-Za-z][A-Za-z0-9]{2,62}$'.

Changing the name to "myTableName" makes it work.

** SDK version
I am running a .net core 2.1 Azure function project.

** Issue reproduce steps
Create an Azure function. Add the Microsoft.Azure.Cosmos.Table nuget package and add an output trigger like this:

public static async Task MyFunction(
            ILogger log,
            [Table("%TableName%", Connection = "ConnectionString")] IAsyncCollector<CacheItem<string>> table)
{
    ...
}

Add the following entry to local.settings.json:

{
    ...
    "TableName": "my-table-name"
    ...
}

Add a table in your Table Storage Account named "my-table-name".

Then run the function. It will fail with the error:

Microsoft.Azure.WebJobs.Host: Error indexing method 'MyFunction'. Microsoft.Azure.WebJobs.Host: Validation failed for property 'TableName', value 'my-table-name'. The field TableName must match the regular expression '^[A-Za-z][A-Za-z0-9]{2,62}$'.

** Whether the issue is consistent or sporadic
The issue happens each time I start my function.

** Environment Summary
Ubuntu 20.04 with the following dotnet packages installed:

** Any other context, such as stack trace or log.

Hello,

I am having issues porting our existing WindowsAzure-based project into using Microsoft.Azure.Storage.* and Microsoft.Azure.Cosmos.Table (1.0.5, .NET Core 2.2). For queues, blob and files the CloudStorageAccount class can be shared. The Microsoft.Azure.Cosmos.Table package is extending the CloudStorageAccount class with CreateCloudTableClient(). It seems impossible now to re-use a CloudStorageAccount that is created using Microsoft.Azure.Storage.*. How should I work with a project referencing both queues and tables sharing the same CloudStorageAccount object? The Microsoft.Azure.Storage.* version also has properties like TableEndpoint, so it does not make any sense to me what I am doing wrong. Also the Microsoft documentation seems to lack information about using these new splitted NuGet packages, and still uses the WindowsAzure.Storage packages. Is this still preview stuff and not suited for production code?

Thank you.

Errors visible: cannot convert from 'Microsoft.Azure.Storage.CloudStorageAccount' to 'Microsoft.Azure.Cosmos.Table.CloudStorageAccount'

Hello!

We've tried to create a new instance of CloudTableClient per each request to storage (AddTransient in IoC container, for example) and it caused a memory leak.

So, after only a 500 requests to storage, we have a disproportional memory consuption:

CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

CloudTable table = tableClient.GetTableReference(TableName);
await table.CreateIfNotExistsAsync();

var entity = new TableEntity(
    partitionKey: "default",
    rowKey: Guid.NewGuid().ToString()
);

var insertResult = await table.ExecuteAsync(
    TableOperation.Insert(entity)
);

Parallel.For(0, 500, (i) =>
{
    var _table = storageAccount.CreateCloudTableClient().GetTableReference(TableName);
    //var _table = tableClient.GetTableReference(TableName);

    var result = _table.ExecuteAsync(
        TableOperation.Retrieve(entity.PartitionKey, entity.RowKey)
    ).GetAwaiter().GetResult();
});

But if we will use a single instance of CloudTableClient, everything will be ok:

The reason of this problem is that CloudTableClient doesn't call Dispose() for internal instance of Microsoft.Azure.Documents.Client.DocumentClient and there is no way to do it outside:

So it looks like the only one way to work with CloudTableClient is to use it as a singleton.
But, in this case, it would be great to know exactly, that CloudTableClient is thread-safe :) Could you please confirm this?

The property is named just "DateTime" rather than "DateTimeValue", unlike all the others (including DateTimeOffset).

I understanding fixing would be a breaking change (or require duplicates with both names). If you could change in the next major version, that would be appreciated. Thanks!

(Copied from Azure/azure-storage-net#551.)

It is well documented that the Azure Table Service & the Cosmos DB by extension place limit on entity sizes. To recap (from the doc):

• PartitionKey & RowKey must each be a max of 1 KB
• The entire row including the system columns (PK, RK & Timestamp) must be a max of 1 MB.
• There is also a limit of 255 columns (including PK, RK & Timestamp) that can be stored in a single row.

However, the SDK's Insert/Update/Delete methods allow you to send entities that violate these conditions.

There needs to be a way for the SDK to warn early (before making the REST API call) about these violations.

Which service(blob, file, queue, table) does this issue concern?

table storage

Which version of the SDK was used?

WindowsAzure.Storage 9.3.3 and
Microsoft.Azure.Cosmos.Table 1.0.1
Microsoft.Azure.Cosmos.Table 1.0.4
Microsoft.Azure.Cosmos.Table 1.0.5

Which platform are you using? (ex: .NET Core 2.1)

.NET Core 2.2

What problem was encountered?

The following exception is encountered when retrieving entities from Table Storage that have properties of type Double that are not set (or are NaN).

Entities were inserted with WindowsAzure.Storage 9.3.3 and retrieved with Microsoft.Azure.Cosmos.Table 1.0.1.

System.AggregateException: One or more errors occurred. (Unexpected EDM type from the Table Service: Edm.Double.) ---> Microsoft.Azure.Cosmos.Table.StorageException: Unexpected EDM type from the Table Service: Edm.Double. ---> System.InvalidOperationException: Unexpected EDM type from the Table Service: Edm.Double.
   at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.TableOperationHttpResponseParsers.ReadSingleItem(JToken token, String& etag)
   at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.TableOperationHttpResponseParsers.ReadQueryResponseUsingJsonParserMetadataAsync(Stream responseStream, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.TableOperationHttpResponseParsers.TableQueryPostProcessGenericAsync[TElement,TQueryType](Stream responseStream, Func`6 resolver, HttpResponseMessage resp, TableRequestOptions options, OperationContext ctx)
   at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.TableQueryRESTCommandGenerator.<>c__DisplayClass0_0.<<GenerateCMDForTableQuery>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor.ProcessEndOfRequestAsync[T](ExecutionState`1 executionState, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   --- End of inner exception stack trace ---

How can we reproduce the problem in the simplest way?

see: https://github.com/JeremyCade/bork.azure.tablestorage for an example of this behaviour.

Basic steps to reproduce:

1. Insert an entity into table storage with WindowsAzure.Storage 9.3.3 library.
2. Retrieve entity from table storage with Microsoft.Azure.Cosmos.Table 1.0.1
3. BORK!

Have you found a mitigation/solution?

No.

Aside

Where is the source for Microsoft.Azure.Cosmos.Table?

SDK version: 1.0.7
Issue reproduce steps: N/A
Whether the issue is consistent or sporadic: Consistent
Environment Summary: N/A
Any other context, such as stack trace or log:
it's impossible to control concurrency and TCP connection handling in a proper way with the SDK forcing the use of the now ancient and obsolete ClientHttpHandler as its inner-most handler. I'm stuck hacking a DelegatingHandler with use of Reflection and handwriting my own authorization header-logic decompiled from the SDK so I can use my own SocketsHttpHandler. In environments such as App Functions in consumption plan you only have 600 TCP connections and those run out fast if you're not able to control idle timeout, concurrency etc.

This is a huge architectural issue with the SDK that needs to be solved. I'd volunteer to fix this myself, but the SDK isn't even open-source (which it really should be!)

Describe the bug
CloudTable fails to insert instance of TableEntity when there's proeprty named "ResourceId"

To Reproduce
Our library is netstandard2.1 and it referencing the following Nuget package: Microsoft.Azure.Cosmos.Table version 1.0.6.
We have class that inherits from Microsoft.Azure.Cosmos.Table.TableEntity. And we have defined Guid property named ResourceId.Example:

`public class CustomRecord : TableEntity
{
public CustomRecord(string partitionKey, string rowKey) :
base(partitionKey, rowKey)
{
}

    public Guid ResourceId { get; set; }

}`

Trying to insert the record by using CloudTable.ExecuteAsync():

TableOperation tableOperation = TableOperation.InsertOrReplace(tableEntity); TableResult tableResult = await cloudTable.ExecuteAsync(tableOperation, cancellationToken).ConfigureAwait(false);

Expected behavior
Properly insert new record to CosmosDB table.

Actual behavior
Throws the following exception:
"Microsoft.Azure.Cosmos.Table.StorageException: Cannot implicitly convert type 'Newtonsoft.Json.Linq.JObject' to 'string'. An explicit conversion exists (are you missing a cast?)\r\n ---> Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot implicitly convert type 'Newtonsoft.Json.Linq.JObject' to 'string'. An explicit conversion exists (are you missing a cast?)\r\n at CallSite.Target(Closure , CallSite , Object , Object )\r\n at Newtonsoft.Json.Serialization.JsonDynamicContract.TrySetMember(IDynamicMetaObjectProvider dynamicProvider, String name, Object value)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateDynamic(JsonReader reader, JsonDynamicContract contract, JsonProperty member, String id)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)\r\n at Newtonsoft.Json.Se
rialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\r\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\r\n at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)\r\n at Microsoft.Azure.Cosmos.Tables.SharedFiles.EntityTranslator.DeserializeObject[TObject](String value)\r\n at Microsoft.Azure.Cosmos.Tables.SharedFiles.EntityTranslator.GetDocumentFromEntityProperties(IDictionary2 properties, String partitionKey, String rowKey, Boolean removeSystemGeneratedProperties)\r\n at Microsoft.Azure.Cosmos.Table.Extensions.EntityHelpers.GetDocumentFromEntity(ITableEntity entity, OperationContext context, TableRequestOpt ions options)\r\n at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionOperationHelper.HandleUpsertAsync(TableOperation operation, CloudTableClient client, CloudTable table, TableRequestOptions options, OperationContext context, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionOperationHelper.ExecuteOperationAsync[TResult](TableOperation operation, CloudTableClient client, CloudTable table, TableRequestOptions requestOptions, OperationContext operationContext, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionOperationHelper.ExecuteOperationAsync[TResult](TableOperation operation, CloudTableClient client, CloudTable table, TableRequestOptions requestOptions, OperationContext operationContext, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionRetryPolicy.ExecuteUnderRetryPolicy[TResult](Func1 executionMethod
, CancellationToken cancellationToken, OperationContext operationContext, TableRequestOptions requestOptions)\r\n at Microsoft.AzureData.DataProcessing.Storage.StorageAccount.TableCosmosDbStorageClient.WriteAsync[TEntity](IStorageContext storageContext, IEnumerable`1 entities, CancellationToken cancellationToken) in D:\Repos\AdsHybrid\AdsHybrid\src\DataProcessing\Microsoft.AzureData.DataProcessing.Storage\CosmosDb\TableCosmosDbStorageClient.cs:line 132\r\nRequest Information\r\nRequestID:\r\nRequestCharge:0\r\nRequestDate:\r\nStatusMessage:Server encountered an internal error.Please try again after some time.\r\nErrorCode:\r\nErrorMessage:Cannot implicitly convert type 'Newtonsoft.Json.Linq.JObject' to 'string'. An explicit conversion exists (are you missing a cast?)\r\n"

Environment summary
SDK Version: 3.1.200
OS Version: Windows 10

SDK version = 2.0.0

I see that this lib has a dependency on Newtonsoft.Json.
And I'm probably making a big assumption here. But I was hoping that by adding the relevant Newtonsoft.Json attributes that the deserialization process would occur correctly.

Is this an implemented feature ?
Or am I maybe doing it wrong ?

Issue reproduce steps

var csa = CloudStorageAccount.Parse("mystring");
var tc = csa.CreateCloudTableClient();
var tbl = tc.GetTableReference("foo"); //Has string column 'AgeString' and int column 'AgeInt'
var qry = tbl.CreateQuery<Foo>();
qry.ToList(); //Foo members are not deserialized correctly

public class Foo : TableEntity
{
	[JsonConverter(typeof(StringEnumConverter))]
	public ages AgeString { get; set; } //defaults to 0 irrespective of data in Azure
	public ages AgeInt {get;set;} //same
}

public enum ages
{
	One,
	Two
	Three
}

Hi. I'm using the Microsoft.Azure.Cosmos.Table package (version 1.0.6).

I intended to change the way CloudTableClient is initialized to take advantage of IHttpMessageHandlerFactory because we're facing scenarios of sockets exaustion derived from accesses to the table storage (the service that uses CloudTableClient is transient and there is no viable path to make it singleton).

The problem is that I don't see any way to do it.

RestExecutorConfiguration accepts a DelegatingHandler but it requires it to have a null inner handler. I don't see any way to pass in the HttpMessageHandler provided by IHttpMessageHandlerFactory.

Is this something that will change in the future? Any plans to support reusing HttpClient or HttpMessageHandler?

Thanks.

The following code generates "Microsoft.Azure.Cosmos.Table: Property delimiter: _ exists in property name" exception when using TableEntityAdapter.

public class TestEntity
{
    public Int32 Count_Current_Panels { get; set; }
    public Int32 Set_Current_Panels { get; set; }
    public Int32 Count_Calibrations { get; set; }
}

TableEntityAdapter entity = new TableEntityAdapter(testEntityInstance, partitionKey, rowKey);

        var table = await EnsureTable(tableName).ConfigureAwait(false);

        TableOperation insertOrReplaceOperation = TableOperation.InsertOrReplace(entity);

        TableResult result = await table.ExecuteAsync(insertOrReplaceOperation).ConfigureAwait(false);

If the TestEntity inherits TableEntity

public class TestEntity : TableEntity  
{
  ...
 }

ExecuteAsync() completes with no exception and there are no issues with Count_Current_Panels property name.

The issue exists only when TestEntity uses TableEntityAdapter and does not inherits TableEntity.

I have tried both Microsoft.Azure.Cosmos.Table.Client, Version=2.0.0.0 and Version 1.0.5

We are continuously addressing and improving the SDK. Please describe your issue along with the following information:

** SDK version

Confirmed in Nuget Package

• v1.0.4
• v1.0.6

** Issue reproduce steps

1. Create a POCO model class and a ModelAdapter class that inherits TableEntityAdatper.
2. Add a field to the underlying Azure table that does not exist on the model
3. Run a TableQuery<ModelAdapter> query and note the null reference

** Whether the issue is consistent or sporadic

Consistent.

** Environment Summary

ASP.NET Core, Azure App Service

** Any other context, such as stack trace or log.

NullReferenceException: Object reference not set to an instance of an object.

Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.SetProperty(object root, string propertyPath, object propertyValue, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext)
StorageException: Object reference not set to an instance of an object.

Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor.ExecuteAsync(RESTCommand cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)

NullReferenceException: Object reference not set to an instance of an object.

Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.SetProperty(object root, string propertyPath, object propertyValue, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext)
Microsoft.Azure.Cosmos.Table.EntityPropertyConverter+<>c__DisplayClass4_0.b__0(T current, KeyValuePair<string, EntityProperty> kvp)
System.Linq.Enumerable.Aggregate<TSource, TAccumulate>(IEnumerable source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> func)
Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.ConvertBack(IDictionary<string, EntityProperty> flattenedEntityProperties, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext)
Microsoft.Azure.Cosmos.Table.TableEntityAdapter.ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext)
Microsoft.Azure.Cosmos.Table.EntityUtilities.ResolveEntityByType(string partitionKey, string rowKey, DateTimeOffset timestamp, IDictionary<string, EntityProperty> properties, string etag)
Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.TableOperationHttpResponseParsers.ReadAndResolve(string etag, Dictionary<string, object> props, Func<string, string, DateTimeOffset, IDictionary<string, EntityProperty>, string, T> resolver, TableRequestOptions options)
Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.TableOperationHttpResponseParsers.TableQueryPostProcessGenericAsync<TElement, TQueryType>(Stream responseStream, Func<string, string, DateTimeOffset, IDictionary<string, EntityProperty>, string, TElement> resolver, HttpResponseMessage resp, TableRequestOptions options, OperationContext ctx, CancellationToken cancellationToken)
Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.TableQueryRESTCommandGenerator+<>c__DisplayClass1_0<TInput, TResult>+<b__2>d.MoveNext()
Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor.ProcessEndOfRequestAsync(ExecutionState executionState, CancellationToken cancellationToken)
Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor.ExecuteAsync(RESTCommand cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)

Unsure if related, but seems very similar to this issue: Azure/azure-storage-net#659

Describe the bug
When we connect o DocumentDB, unable to open connection to cosmoDB.
The same work when we have .Net Core 2.0
Below are the version we are using:
.Net Core 2.2
Package used:
Visual studion: 2017 (15.9.12)

Exception or Stack Trace
InnerException = {System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Http.Connec...

["One or more errors occurred. (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)"," at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.Task.Wait()\r\n at webapi.Controllers.ValuesController.Get() in C:\XXXi\ConsoleApp1\webapi\Controllers\ValuesController.cs:line 42 System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\r\n at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask)\r\n at System.Threading.Tasks.ValueTask1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Documents.Client.DocumentClient.HttpRequestMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n at Microsoft.Azure.Documents.Client.GatewayServiceConfigurationReader.GetDatabaseAccountAsync(Uri serviceEndpoint)\r\n at Microsoft.Azure.Documents.Routing.GlobalEndpointManager.GetDatabaseAccountFromAnyLocationsAsync(Uri defaultEndpoint, IList1 locations, Func`2 getDatabaseAccountFn)\r\n at Microsoft.Azure.Documents.Client.GatewayServiceConfigurationReader.InitializeReaderAsync()\r\n at Microsoft.Azure.Documents.Client.DocumentClient.InitializeGatewayConfigurationReader()\r\n at Microsoft.Azure.Documents.Client.DocumentClient.GetInitializationTask(IStoreClientFactory storeClientFactory)\r\n at Microsoft.Azure.Documents.Client.DocumentClient.EnsureValidClientAsync()\r\n at Microsoft.Azure.Documents.Client.DocumentClient.OpenPrivateInlineAsync(CancellationToken cancellationToken)"]

To Reproduce
Open a console application of .Net Core 2.2 with package:

Code Snippet
try
{

            WebProxy proxyObject = new WebProxy("XXXX");
            WebRequest.DefaultWebProxy = proxyObject;

            HttpClientHandler handler = new HttpClientHandler()
            {
                Proxy = proxyObject,
                UseProxy = true,
            };

            var accountKey = "XXX";

            var databaseAccountName = "XXX";
            var databaseEndPoint = new Uri(string.Format("https://{0}.documents.azure.com", databaseAccountName));
            var databaseAccountKey = accountKey;
            var connectionPolicy = new ConnectionPolicy
            {
                ConnectionMode = ConnectionMode.Gateway, //ConnectionMode.Gateway
                ConnectionProtocol = Protocol.Https,
                RetryOptions = new RetryOptions()
                {
                    MaxRetryAttemptsOnThrottledRequests = 6,
                    MaxRetryWaitTimeInSeconds = 5
                }
            };

            // var client = new DocumentClient(databaseEndPoint, databaseAccountKey, handler, connectionPolicy);
            var client = new DocumentClient(databaseEndPoint, databaseAccountKey, connectionPolicy);
            client.OpenAsync().Wait();

        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }

Expected behavior
Connection to DocumentDB (CosmoDB) should open with out error.

Setup (please complete the following information):

OS: Windows
IDE : Visual studion 2017 (15.9.12)
Version of the Library Microsoft.Azure.DocumentDB.Core=> Version="2.4.0"
Additional context
Its working with .Net Core 2.0

The README.md (and other release notes) says

The 2.0.0 Table SDK is in preview now... with... namespace consolidation to Cosmos DB endpoint

What does that mean exactly? Are Azure Tables being deprecated? Or absorbed into Cosmos? Will there still be a non-premium table storage option?

I have tried to use filtering on PartitionKey and it not working

void Main()
{
	var acc = Microsoft.Azure.Cosmos.Table.CloudStorageAccount.Parse("UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://ipv4.fiddler");
	var tables = acc.CreateCloudTableClient();
	var table = tables.GetTableReference("Tasks"); 
	table.DeleteIfExists();
    table.CreateIfNotExists();
	var entry = new AuditEntity("1", "some",new Dictionary<string,object>(),new Dictionary<string,Type>());
    TableOperation insert = TableOperation.Insert(entry);
    table.Execute(insert);
	entry = new AuditEntity("2", "some",new Dictionary<string,object>(),new Dictionary<string,Type>());
    insert = TableOperation.Insert(entry);
    table.Execute(insert);
	var subitemQuery = table.CreateQuery<AuditEntity>();
    subitemQuery.FilterString = TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal,Convert.ToString(1));
    subitemQuery.TakeCount = 1000;
    subitemQuery.FilterString.Dump();
	table.ExecuteQuery(subitemQuery).Count().Dump();
}
    public class AuditEntity : ITableEntity
    {
        public AuditEntity()
        {

        }
        public AuditEntity(string pk, string rk,Dictionary<string,object> props,Dictionary<string,Type> types)
        {
            PartitionKey = pk;
            RowKey = rk;
            Props = props;
            Types = types;
        }

        public string PartitionKey { get; set; }
        public string RowKey { get; set; }
        public Dictionary<string, object> Props { get; private set; }
        public Dictionary<string, Type> Types { get; private set; }
        public DateTimeOffset Timestamp { get; set; }
        public string ETag { get; set; }

        public void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext)
        {
            Props = properties.ToDictionary(p => p.Key, p => p.Value.PropertyAsObject);
            Types = properties.ToDictionary(p => p.Key, p => ToClr(p.Value.PropertyType));

        }

        private Type ToClr(EdmType propertyType)
        {
            switch (propertyType)
            {
                case EdmType.Binary:
                    return typeof(byte[]);
                case EdmType.Boolean:
                    return typeof(bool?);
                case EdmType.DateTime:
                    return typeof(DateTime?);
                case EdmType.Double:
                    return typeof(double?);
                case EdmType.Int32:
                    return typeof(int?);
                case EdmType.Int64:
                    return typeof(long?);
                case EdmType.String:
                    return typeof(string);
                case EdmType.Guid:
                    return typeof(Guid?);
                default:
                    throw new NotImplementedException();
            }
        }

        public IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext)
        {
            return Props
                .ToDictionary(kv => kv.Key,kv=> ToAzureProp(kv.Value, kv.Key));
        }

        private EntityProperty ToAzureProp(object arg,string key)
        {
            switch (arg)
            {
                case bool b:
                    return new EntityProperty(b);
                case double d:
                    return new EntityProperty(d);
                case string s:
                    return new EntityProperty(s);
                case byte bt:
                    return new EntityProperty((int)bt);
                case int i:
                    return new EntityProperty(i);
                case long l:
                    return new EntityProperty(l);
                case byte[] bArr:
                    return new EntityProperty(bArr);
                case DateTime dt:
                    return new EntityProperty(dt);
                case decimal m:
                    return new EntityProperty(Convert.ToString(m));
                case null:
                    var type = Types[key];
                    if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof(Nullable<>))
                    {
                        var subtype = type.GetGenericArguments().Single();
                        switch (subtype.Name)
                        {
                            case nameof(Decimal):
                                return EntityProperty.GeneratePropertyForString(null);
                            case nameof(Double):
                                return EntityProperty.GeneratePropertyForDouble(null);
                            case nameof(Int32):
                                return EntityProperty.GeneratePropertyForInt(null);
                            case nameof(Int64):
                                return EntityProperty.GeneratePropertyForLong(null);
                            case nameof(Boolean):
                                return EntityProperty.GeneratePropertyForBool(null);
                            case nameof(DateTime):
                                return new EntityProperty((DateTime?)null);
                            case nameof(DateTimeOffset):
                                return EntityProperty.GeneratePropertyForLong(null);
                            default:
                                if(subtype.IsEnum)
                                    return EntityProperty.GeneratePropertyForInt(null);

                                throw new NotImplementedException(subtype.Name);
                        }
                    }
                    else
                    {
                        switch (type.Name)
                        {
                            case nameof(String):
                                return EntityProperty.GeneratePropertyForString(null);
                            default:
                                throw new NotImplementedException();
                        }
                    }
                default:
                    if (arg.GetType().IsEnum)
                        return EntityProperty.GeneratePropertyForInt((int)(byte)arg);
                    throw new NotImplementedException();
            }
        }
    }

subitemQuery loads 2 items whenever it should load single item
also i used fiddler to check what kind of request it made and i expected something like ?$filter=PartitionKey eq '1' in a request url but it simply isnt there

Library version: 1.0.1

Hi,

I'm trying to fetch an entity from Azure Table Storage with the following code.

public async Task<T> GetAsync(string partitionKey, string rowKey)
{
    var operation = TableOperation.Retrieve<TableEntityAdapter<T>>(partitionKey, rowKey);
    var result = await _table.ExecuteAsync(operation);
    var complexEntity = result?.Result as TableEntityAdapter<T>;

    return complexEntity.OriginalEntity;
}

Inspecting the TableResult object namely result and looking into the Result property, I see the correct PartitionKey and RowKey properties. But the OriginalEntity property contain nulls for the partition and row key.

I can't seem to figure out why this happens but if I set the table operation to be Retrieve<T> instead of Retrieve<TableEntityAdapter<T>> then I get the correct keys except that now my I do not get the "complex entity".

I haven't been able to find anything about this behavior and currently blinded by this issue.

• SDK version
  Microsoft.Azure.Cosmos.Table 1.0.6
• Whether the issue is consistent or sporadic
  The issue is consistent.
• Environment Summary
  Windows 10 Pro x64
  Microsoft Visual Studio Professional 2019
  Version 16.4.6
  .NET Core 2.0 app but consumed by .NET Core 3.1 apps.

There is no sample or documentation on using queries like "$top" using this SDK. Please update the docs / samples to include this information.

For example, how do you use this SDK to implement this solution from the docs?
https://docs.microsoft.com/en-us/azure/cosmos-db/table-storage-design-guide#context-and-problem-6

Using a property named "ID" throws the below serialization exception

Cannot deserialize the current JSON object (e.g. {"name":"value"}) into type 'System.String' because the type requires a JSON primitive value (e.g. string, number, boolean, null) to deserialize correctly.
To fix this error either change the JSON to a JSON primitive value (e.g. string, number, boolean, null) or change the deserialized type so that it is a normal .NET type (e.g. not a primitive type like integer, not a collection type like an array or List) that can be deserialized from a JSON object. JsonObjectAttribute can also be added to the type to force it to deserialize from a JSON object.
Path 'Id.$t', line 1, position 13.

"Id" seems to be a reserved system property. We should probably throw a more meaningful exception.

The EntityProperty class has a bunch of validation (calls to this.EnforceType) that prevent turning a property of one type into another type.

The REST API for tables allows changing the property of a type when updating/merging. To be more precise, it doesn't even care what the old type was - the type of the property is determined by what is specified anytime the property is created/replaced/merged.

To match, the .NET API should likely not prevent reading a property as one type and then changing it to another type to write it back to the table.

(Copied from Azure/azure-storage-net#554.)

** SDK version
1.0.7 and 2.0.0
** Issue reproduce steps
Create a new UWP project targeting version 1903 and minimum version 18362
Install Microsoft.Azure.Cosmos.Table any of the above versions
Use the library from the UWP project
Build in release mode and run with the debugger
** Whether the issue is consistent or sporadic
Consistent
** Environment Summary
SDK Version: Windows 10 SDK (10.0.18362.0)

MCG : warning MCG0007: Unresolved P/Invoke method 'libc!sysctl' for method 'System.Int32 Microsoft.Azure.Documents.NativeMethods.Darwin.sysctl(System.Int32*, System.UInt32, System.Byte*, System.UInt32*, System.IntPtr, System.UInt32)'. Calling this method would throw exception at runtime. Please make sure the P/Invoke either points to a Windows API allowed in UWP applications, or a native DLL that is part of the package. If for some reason your P/Invoke does not satisfy those requirements, please use [DllImport(ExactSpelling=true) to indicate that you understand the implications of using non-UWP APIs.
MCG : warning MCG0007: Unresolved P/Invoke method 'libc!uname' for method 'System.Int32 Microsoft.Azure.Documents.NativeMethods.Unix.uname(System.IntPtr)'. Calling this method would throw exception at runtime. Please make sure the P/Invoke either points to a Windows API allowed in UWP applications, or a native DLL that is part of the package. If for some reason your P/Invoke does not satisfy those requirements, please use [DllImport(ExactSpelling=true) to indicate that you understand the implications of using non-UWP APIs.
MCG : warning MCG0007: Unresolved P/Invoke method 'ntdll!RtlGetVersion' for method 'System.Int32 Microsoft.Azure.Documents.NativeMethods.Windows.RtlGetVersion(Microsoft.Azure.Documents.NativeMethods.Windows.RTL_OSVERSIONINFOEX)'. Calling this method would throw exception at runtime. Please make sure the P/Invoke either points to a Windows API allowed in UWP applications, or a native DLL that is part of the package. If for some reason your P/Invoke does not satisfy those requirements, please use [DllImport(ExactSpelling=true) to indicate that you understand the implications of using non-UWP APIs.

** SDK version v1.0.x

Will Microsoft.Azure.Cosmos.Table have support for client side encryption, or the equivalence of EncryptionPolicy and EncryptionResolver in the old WindowsAzure.Storage nuget package?

https://docs.microsoft.com/en-us/azure/storage/common/storage-client-side-encryption#table-service-encryption

The old Azure Storage SDK API documentation seems to have been removed from https://docs.microsoft.com/en-us/dotnet/api/ , but the documentation can still be found in the following sites:

// TableEncryptionPolicy on docs.azure.cn:
https://docs.azure.cn/en-us/dotnet/api/microsoft.windowsazure.storage.table.tablerequestoptions.encryptionpolicy?view=azure-dotnet#Microsoft_WindowsAzure_Storage_Table_TableRequestOptions_EncryptionPolicy

// TableRequestOptions.EncryptionResolveron docs.azure.cn:
https://docs.azure.cn/en-us/dotnet/api/microsoft.windowsazure.storage.table.tablerequestoptions.encryptionresolver?view=azure-dotnet#Microsoft_WindowsAzure_Storage_Table_TableRequestOptions_EncryptionResolver

// TableEncryptionPolicy in the Azure Storage Java SDK
https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.storage.table.tablerequestoptions.getencryptionpolicy?view=azure-java-stable#com_microsoft_azure_storage_table_TableRequestOptions_getEncryptionPolicy__

My team would like to know whether this feature is even on the road map for Microsoft.Azure.Cosmos.Table, so that we can make an informed choice between the "legacy" package WindowsAzure.Storage vs. the new Microsoft.Azure.Cosmos.Table package. Thank you!

Corresponding to this nuget package https://www.nuget.org/packages/Microsoft.Azure.Cosmos.Table?

If so, any idea on when the source would be available? Or am I looking in the wrong place?

I'm stuck with version 7 because another package (Microsoft.Azure.Cosmos.Table) creates dependency issues. Can you please fix this issue?

We are continuously addressing and improving the SDK. Please describe your issue along with the following information:

I am not able to run my webjob on azure because of dependencies in Microsoft.Azure.Cosmos.Table: 1.0.6 (currently that's latest stable release).

** SDK version
csproj:

    <TargetFramework>net48</TargetFramework>
    <LangVersion>8.0</LangVersion>

app.config:

        <supportedRuntime version="v4.0" />

Microsoft.Azure.Cosmos.Table: 1.0.6

** Issue reproduce steps

                    CloudStorageAccount tableAccount = Microsoft.Azure.Cosmos.Table.CloudStorageAccount.Parse(Instance.ConnectionString);
                    Instance.StorageAccount = tableAccount;
                    CloudTableClient tableClient = tableAccount.CreateCloudTableClient();
                    Instance.Client = tableClient;

** Whether the issue is consistent or sporadic
0% locally (at least on my mac dev environment, with mono 6.8.x up-to-date)
100% on Azure

** Environment Summary
on Azure Web App: Configuration > General Settings
Stack: .NET
.NET Framework Version: V4.7

** Any other context, such as stack trace or log.

[04/14/2020 06:36:16 > 71b93c: ERR ] Unhandled Exception: Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ConsumeImages ---> System.TypeInitializationException: The type initializer for 'ImageConsumer.Functions' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Threading.Tasks, Version=4.0.10.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
[04/14/2020 06:36:16 > 71b93c: ERR ]    at Microsoft.Azure.Cosmos.Table.CloudTableClient..ctor(StorageUri storageUri, StorageCredentials credentials, TableClientConfiguration configuration)
[04/14/2020 06:36:16 > 71b93c: ERR ]    at Microsoft.Azure.Cosmos.Table.CloudStorageAccountExtensions.CreateCloudTableClient(CloudStorageAccount account, TableClientConfiguration configuration)

What I find bizarre is that I cannot find a way around it without updating Microsoft.Azure.Cosmos.Table to 2.0.0-preview (which does work in the Azure webjob environment). I tried installing the nuget package for 4.0.11, 4.3.0 - and providing a binding redirect. I saw no change though (other than the error first failing to find the redirect version, then the 4.0.10.0 version).
Also tried 4.0.10, but the install aborts with "Package restore failed. Rolling back package changes" and no error message to indicate what is going on. I thought that might be a mono issue.. but it is also true when I copy the project onto my virtual box and run it with visual studio 2019 in windows 10.

There is no system.threading.tasks reference I could add.

I also noticed that when I set the csproj to v4.7 exactly I can reproduce the error as a warning like "ound conflicts between different versions" but then it still runs locally anyway. Trying a different version like 4.7.1 also does not resolve the issue on the remote environment.

While debugging locally there is an excessive amount of noise in the Debug Output logs. The following doesn't serve any purpose to the average developer on our team and just makes it harder to watch for the information they are interested in. It would be great if there was at least a way to set the debug level programatically in our applications\libraries so we wouldn't have to see this unless we were troubleshooting something:

Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Starting operation with location Primary per location mode PrimaryOnly. Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: This operation can only be executed against the primary storage location. Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Starting asynchronous request to http://127.0.0.1:10002/devstoreaccount1/CDSRequest(). Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Setting payload format for the request to 'Json'. Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Waiting for response. Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Response received. Status code = 204, Request ID = 2b665035-8889-4197-8e3d-5c162ec14d37, Content-MD5 = , ETag = W/"datetime'2020-01-02T16%3A49%3A19.05Z'". Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Response headers were processed successfully, proceeding with the rest of the operation. Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Processing response body. Microsoft.Azure.Cosmos.Table Information: 0 : ec30069c-8bce-45fe-a3f0-6339d3fbebff: Operation completed successfully.

We are continuously addressing and improving the SDK. Please describe your issue along with the following information:

** SDK version
** Issue reproduce steps
** Whether the issue is consistent or sporadic
** Environment Summary
** Any other context, such as stack trace or log.

Attempting to use the Azure.Cosmos.Table libraries on an Ubuntu Linux container (specifically https://github.com/cloudfoundry/dotnet-core-buildpack/releases/tag/v2.2.12) is resulting in

System.MethodAccessException: Attempt by method 'Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionExecutor.ExecuteQuerySegmented(Microsoft.Azure.Cosmos.Table.TableQuery1<!!1>, Microsoft.Azure.Cosmos.Table.TableContinuationToken, Microsoft.Azure.Cosmos.Table.CloudTableClient, Microsoft.Azure.Cosmos.Table.CloudTable, Microsoft.Azure.Cosmos.Table.EntityResolver1<!!0>, Microsoft.Azure.Cosmos.Table.TableRequestOptions, Microsoft.Azure.Cosmos.Table.OperationContext)' to access method 'Microsoft.Azure.Documents.Client.TaskHelper.InlineIfPossible<Microsoft.Azure.Cosmos.Table.TableQuerySegment1<TResult>>(System.Func1<System.Threading.Tasks.Task1<Microsoft.Azure.Cosmos.Table.TableQuerySegment1<!!0>>>, Microsoft.Azure.Documents.IRetryPolicy, System.Threading.CancellationToken)' failed.
I've tried both the 1.0.1 and the 1.0.2-preview versions of the library and get the same results.

The code base is proprietary so I can't share the whole project but I did attach the class file that makes the call to the API as well as a sanitized csproj file that lists the versions of .NET Core and the nuget packages I'm using.

code.zip

EntityProperty currently supports values containing nulls, and for any property "foo" DynamicTableEntity supports both:

1. Adding a property named "foo" with a null value.
2. Not adding a property named "foo" at all.

But both of those options are treated identically by the Table Service REST API, as documented here (search for "null" on each page):

• https://docs.microsoft.com/en-us/rest/api/storageservices/update-entity2
• https://docs.microsoft.com/en-us/rest/api/storageservices/merge-entity
• https://docs.microsoft.com/en-us/rest/api/storageservices/insert-or-replace-entity
• https://docs.microsoft.com/en-us/rest/api/storageservices/insert-or-merge-entity

Because null values are treated as missing values by the storage service, they're never returned when querying entities (a null value will never be read back).

Given the shape of EntityProperty/DynamicTableEntity, I was about to write a bunch of code to handle reading/writing null values as distinct from a missing property value. I happened to test the storage service REST API to understand the distinction and thankfully discovered there is no difference, but the EntityProperty/DynamicTableEntity API was misleading in this regard. Having just one way to say a property doesn't have a value (it isn't in the dictionary) would simplify and clarify the .NET API here.

(Note that I'm not suggesting any changes to TableEntity - I agree it's useful to support int? and bool? properties there - this comment is only about the EntityProperty/DynamicTableEntity layer.)

(Copied from Azure/azure-storage-net#550.)

We have tried moving from Microsoft.WindowsAzure.Storage to Microsoft.Azure.Cosmos.Table, but hit some major performance obstacles in the process. This is currently a blocker for us.

After disabling client tracing (see issue see reported issue) we are still seeing a ~25% degradation in performance. From the Azure profiler traces, time is mainly spent in array allocation inside Microsoft.Azure.Cosmos.Table.RestExecutor.Utils.AsyncStreamCopier.StartCopyStreamAsyncHelper().

I have compared the disassembled IL from the Table library and the legacy API, and found that the Table library always allocates new arrays, while the legacy libraries uses a buffer manager to reuse buffers. I also found that the BufferManager setter has been removed from the CloudTableClient class in the new Table library, as compared to the WindowsAzure.Storage library.

I have two requests for getting the Table SDK on par with the legacy storage API:

1. Re-introduce the OperationContext.DefaultLogLevel property, to make it possible to control the trace log level.
2. Re-introduce support for BufferManager in the CloudTableClient.

SDK: 1.06
To reproduce: Insert and read back large number of entities. Noticeable performance decrease even with 1000.
Consistent: yes
Environment: Originally seen in deployment to azure services, reproduced locally with test app and test storage account. Using Azure Storage, not cosmos.

We've been working to upgrade to the latest version of our libraries and have hit a major snag with azure tables. The performance difference is pretty large and puts us into an unacceptable range for our clients. I see version 2 should improve performance, but no details to what level. Will performance be similar to 9.3.3?

Is there an ETA for the release of version 2?

What is the recommended path to upgrade to the latest storage libraries for Queues and Blobs without this major performance decrease for Tables?

Below is the output of my test app. Two identical sets of 1000 entities are inserted into two separate tables within the same account. They are then read back with no filter. The performance appears to be more noticeable with larger entities. The tests show similar slowdowns to what we experienced in our test environment.

Microsoft.Azure.CosmosDB.Table.TableContinuationToken class has WriteXml and ReadXml methods. Unfortunately, these methods do not exist in Microsoft.Azure.Cosmos.Table.TableContinuationToken, which makes upgrading from Microsoft.Azure.CosmosDB.Table to Microsoft.Azure.Cosmos.Table much more difficult.

SetServiceProperties fails with 400 Bad Request when sent to Azure Storage Emulator.

Code to reproduce:

        [Fact]
        public void SetAccountProperties_GetAccountProperties_ShouldMatchWhatWasSet()
        {
            // Arrange
            var account = CloudStorageAccount.DevelopmentStorageAccount;
            var tableClient = account.CreateCloudTableClient();
            var serviceProperties = new ServiceProperties
            {
                Logging = new LoggingProperties { RetentionDays = 7, LoggingOperations = LoggingOperations.All },
                DeleteRetentionPolicy = new DeleteRetentionPolicy { Enabled = true, RetentionDays = 7 },
                HourMetrics = new MetricsProperties { MetricsLevel = MetricsLevel.ServiceAndApi },
                MinuteMetrics = new MetricsProperties { MetricsLevel = MetricsLevel.Service }
            };

            // Act
            tableClient.SetServiceProperties(serviceProperties);
            var result = tableClient.GetServiceProperties();

            // Assert
            Assert.Equal(serviceProperties, result);
        }

Sniffing the request produces the following:

PUT /devstoreaccount1?comp=properties&restype=service HTTP/1.1
x-ms-client-request-id: 6051bd4b-4708-45d7-a1da-46f6e14d4c08
User-Agent: Azure-Cosmos-Table/1.0.1 (.NET CLR 4.0.30319.42000; Win32NT 6.2.9200.0)
x-ms-version: 2018-03-28
x-ms-date: Mon, 20 May 2019 19:20:04 GMT
Authorization: SharedKey devstoreaccount1:f/spsM88xzSo5sAkcpLxNUQg5dz18SP1xKGKl8BkkpQ=
Content-Length: 677
Host: 127.0.0.1:10002

?<?xml version="1.0" encoding="utf-8"?><StorageServiceProperties><Logging><Version>1.0</Version><Delete>true</Delete><Read>true</Read><Write>true</Write><RetentionPolicy><Enabled>true</Enabled><Days>7</Days></RetentionPolicy></Logging><HourMetrics><Version>1.0</Version><Enabled>true</Enabled><RetentionPolicy><Enabled>false</Enabled></RetentionPolicy><IncludeAPIs>true</IncludeAPIs></HourMetrics><MinuteMetrics><Version>1.0</Version><Enabled>true</Enabled><RetentionPolicy><Enabled>false</Enabled></RetentionPolicy><IncludeAPIs>false</IncludeAPIs></MinuteMetrics><DeleteRetentionPolicy><Enabled>true</Enabled><Days>7</Days></DeleteRetentionPolicy></StorageServiceProperties>

Notice the redundant questionmark in the beginning of the request body. After some investigation I came to a conclusion, that this questionmark is BOM, and this is what breaks the request.

The test project I've written uses target framework netcoreapp2.2

I'm working on migrating the Microsoft Orleans providers from WindowsAzure.Storage to Microsoft.Azure.Cosmos.Table. I've hit a snag with two classes that have changed visiblity. Namely, NameValidator and StorageErrorCodeStrings. These used to be public, but now they're internal. Here are examples of where they're being used (in my WIP fork/branch):

NameValidator.ValidateTableName
StorageErrorCodeStrings.ResourceNotFound
StorageErrorCodeStrings.OperationTimedOut

What are the recommended alternatives?

SDK version

cosmos-table-sdk/1.0.6

Issue reproduce steps

1. Setup local DotNet Core development, including Azure Strorage Emulator
2. Setup stunnel (https://www.stunnel.org/) to offload TLS-traffic, point it to localhost TCP/10002 for emulated Table Storage
3. Using Microsoft.Azure.Cosmos.Table.CloudStorageAccount.CloudTableClient with connection string (notice usage of HTTPS) AccountName=devstoreaccount1;TableEndpoint=https://localhost:10012/devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;
4. Observe the connection to fail with exception. Problem is with HTTP/1.1 400 The value for one of the HTTP headers is not in the correct format.

Whether the issue is consistent or sporadic

Consistent. Fully reproducable with ease.

Environment Summary

Visual Studio 2019, .Net Core 2.2, Azure Storage Emulator 5.10, Microsoft.Azure.Cosmos.Table 1.0.6.

Any other context, such as stack trace or log.

Mitigation

Not using HTTPS works. ConnectionString with TableEndpoint=http://localhost:10002/devstoreaccount1 works without problems. All other types of emulated storage work perfectly via stunnel. Using obsoleted Microsoft.WindowsAzure.Storage.Tables works perfectly via stunnel.

Traffic capture

Wireshark-captured HTTP-traffic is:

GET /devstoreaccount1/ HTTP/1.1
x-ms-version: 2018-06-18
User-Agent: Windows/10.0.17763 documentdb-netcore-sdk/2.1.3 cosmos-table-sdk/1.0.6
x-ms-date: Wed, 11 Mar 2020 09:03:03 GMT
Authorization: type%3dmaster%26ver%3d1.0%26sig%3dn4QR%2fBuIkiSnHksUzpiVAozyGf8PlbTFJ6Rggi6WMbA%3d
Host: localhost:10012

HTTP/1.1 400 The value for one of the HTTP headers is not in the correct format.
Content-Length: 371
Content-Type: application/xml
Server: Microsoft-HTTPAPI/2.0
x-ms-request-id: 2fa5a27e-088a-4218-b94b-920c00b8a395
Date: Wed, 11 Mar 2020 09:03:04 GMT

...<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code>InvalidHeaderValue</code>
  <message xml:lang="en-US">The value for one of the HTTP headers is not in the correct format.
RequestId:2fa5a27e-088a-4218-b94b-920c00b8a395
Time:2020-03-11T09:03:04.1336879Z</message>
</error>

According to the documentation:
https://docs.microsoft.com/en-us/rest/api/storageservices/writing-linq-queries-against-the-table-service

I should be able to create an IQuerable context.CreateQuery to consume in a downstream OData endpoint using Automapper projection.

Using dotnet core 3.1 and CreateQuery is not availabe on CloudTable.

Why is the method missing and is there a workaround to get an IQuerable to consume in my own OData Controller?

TLDR: Is it planned and/or possible to refactor this library to depend on the common classes found in Microsoft.Azure.Storage.Common 10.x? I'm trying to migrate code from WindowsAzure.Storage 9.3.3, but it's proving very difficult due to the duplication of common classes (StorageException, CloudStorageAccount, etc.).

We have a system built with Microsoft Orleans on .NET Standard, and we're using the built-in storage and stream providers for Azure Storage-based services (Blob, Queue, and Table). An issue keeps cropping up with WindowsAzure.Storage 9.3.3:

Azure/azure-storage-net#620

A contributor submitted a patch to try and fix the issue, but it was closed since Table has transitioned to the Cosmos DB team:

Azure/azure-storage-net#774

Sorry for the quick rant... but this transition has been rough... 7 months later, and we're still waiting for serious progress on the .NET Standard version of this package:

Azure/azure-storage-net#779

I tried to take a stab at "upgrading" the Microsoft Orleans providers to the newer libraries (Microsoft.Azure.Storage.Blob, Microsoft.Azure.Storage.Queue, and Microsoft.Azure.Cosmos.Table):

dotnet/orleans#5479

As noted in that PR, this is a difficult transition due to utilities like AzureStorageUtils that try to take care of cross-cutting concerns, like parsing details from StorageException. Additionally, some classes/methods have had their visibility changed.

Based on my research, the new Azure Storage SDKs have moved common classes like StorageException into Microsoft.Azure.Storage.Common. So back to the question, is it planned and/or possible to refactor this library to depend on Microsoft.Azure.Storage.Common? Even if the answer is "no", I think that would help projects, like Microsoft Orleans, come up with a migration strategy.

We upgraded to the next version of SDK to access our Azure Table storage.

We observed performance degradation of our application after that. We even created test applications with identical usage pattern to isolate it, and still see this performance hit.

We are using .NET Framework code, reading data from Azure table.

Old client: Microsoft.WindowsAzure.Storage - 9.3.2

New client: Microsoft.Azure.Cosmos.Table - 1.0.6

Here is one of the sample tests we tried to run:

public async Task ComparisionTest1()
{
    var partitionKey = CompanyId.ToString();

    {
        // Microsoft.Azure.Cosmos.Table
        var storageAccount = Microsoft.Azure.Cosmos.Table.CloudStorageAccount.Parse(ConnectionString);
        var tableClient = Microsoft.Azure.Cosmos.Table.CloudStorageAccountExtensions.CreateCloudTableClient(storageAccount);
        var tableRef = tableClient.GetTableReference("UserStatuses");
        var query = new Microsoft.Azure.Cosmos.Table.TableQuery<Microsoft.Azure.Cosmos.Table.TableEntity>()
                            .Where(Microsoft.Azure.Cosmos.Table.TableQuery.GenerateFilterCondition("PartitionKey", "eq", partitionKey));
        var result = new List<Microsoft.Azure.Cosmos.Table.TableEntity>(20000);

        var stopwatch = Stopwatch.StartNew();
        var tableQuerySegment = await tableRef.ExecuteQuerySegmentedAsync(query, null);
        result.AddRange(tableQuerySegment.Results);
        while (tableQuerySegment.ContinuationToken != null)
        {
            tableQuerySegment = await tableRef.ExecuteQuerySegmentedAsync(query, tableQuerySegment.ContinuationToken);
            result.AddRange(tableQuerySegment.Results);
        }

        stopwatch.Stop();
        Trace.WriteLine($"Cosmos table client. Elapsed: {stopwatch.Elapsed}");
    }

    {
        // Microsoft.WindowsAzure.Storage
        var storageAccount = Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse(ConnectionString);
        var tableClient = storageAccount.CreateCloudTableClient();
        var tableRef = tableClient.GetTableReference("UserStatuses");
        var query = new Microsoft.WindowsAzure.Storage.Table.TableQuery<Microsoft.WindowsAzure.Storage.Table.TableEntity>()
                            .Where(Microsoft.WindowsAzure.Storage.Table.TableQuery.GenerateFilterCondition("PartitionKey", "eq", partitionKey));
        var result = new List<Microsoft.WindowsAzure.Storage.Table.TableEntity>(20000);

        var stopwatch = Stopwatch.StartNew();
        var tableQuerySegment = await tableRef.ExecuteQuerySegmentedAsync(query, null);
        result.AddRange(tableQuerySegment.Results);
        while (tableQuerySegment.ContinuationToken != null)
        {
            tableQuerySegment = await tableRef.ExecuteQuerySegmentedAsync(query, tableQuerySegment.ContinuationToken);
            result.AddRange(tableQuerySegment.Results);
        }

        stopwatch.Stop();
        Trace.WriteLine($"Old table client. Elapsed: {stopwatch.Elapsed}");
    }
}

When test is ran in Azure environment, here is result:

Any thoughts, advise about it?

SDK version: 1.0.6

var continuationToken = new TableContinuationToken();
var tableQuery = new TableQuery<EmployeeTimeSummary>()
                .Where(q.Filter)
                .Select(q.Select)
                .Take(q.Take);
var querySegment = await table.ExecuteQuerySegmentedAsync(tableQuery, continuationToken);

Consistent issue
Environment: Visual Studio 2019 - Windows 10 (x64)
Stack Trace

Microsoft.Azure.Cosmos.Table.StorageException: Method 'Visit' in type 'QueryTokenVisitor' from assembly 'Microsoft.Azure.Cosmos.Table, Version=1.0.5.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
---> System.TypeLoadException: Method 'Visit' in type 'QueryTokenVisitor' from assembly 'Microsoft.Azure.Cosmos.Table, Version=1.0.5.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
at Microsoft.Azure.Cosmos.Tables.SharedFiles.ODataFilterTranslator.ToSql(String odataV4FilterString, Boolean isTableQuery, Boolean enableTimestampQuery)
at Microsoft.Azure.Cosmos.Tables.SharedFiles.QueryTranslator.GetSqlQuery(String selectList, String odataV4FilterString, Boolean isLinqExpression, Boolean isTableQuery, IList1 orderByItems, String tombstoneKey, Boolean enableTimestampQuery) at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionQueryHelper.QueryDocumentsAsync[TResult](Nullable1 maxItemCount, String filterString, IList1 selectColumns, TableContinuationToken token, CloudTableClient client, CloudTable table, EntityResolver1 resolver, TableRequestOptions requestOptions, OperationContext operationContext, Boolean isLinqExpression, IList1 orderByItems, String tombstoneKey) at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionExecutor.<>c__DisplayClass24_02.<b__0>d.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionExecutor.<>c__DisplayClass24_02.<<ExecuteQuerySegmentedInternalAsync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionRetryPolicy.ExecuteUnderRetryPolicy[TResult](Func1 executionMethod, CancellationToken cancellationToken, OperationContext operationContext, TableRequestOptions requestOptions)
at DGS.EnterpriseAPI.WebApi.Controllers.TimeSummaryController.GetAsync(ODataQueryCollectionOptions q) in C:\Users\trevo\source\repos\dgs\DGS.EnterpriseAPI\src\DGS.EnterpriseAPI.WebApi\Controllers\TimeSummaryController.cs:line 72
at lambda_method(Closure , Object )
at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
Request Information
RequestID:
RequestCharge:0
RequestDate:
StatusMessage:Server encountered an internal error.Please try again after some time.
ErrorCode:
ErrorMessage:Method 'Visit' in type 'QueryTokenVisitor' from assembly 'Microsoft.Azure.Cosmos.Table, Version=1.0.5.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.

The Partition and Row Keys have restrictions in terms of the characters allowed and the total length of the value.

There needs to be validation at the TableEntity level that checks these constraints, before the data is sent off to Azure to be processed.

When using the API version 1.0.6 and trying to do a filter query I get empty results.

Checking with Fiddler the request URL has the single quotes escaped:

Original query (tested in MS Azure Storage Explorer):
PartitionKey eq '[email protected]'

Request made with API (does not work):
GET /table?$filter=PartitonKey%20eq%20%27user%40domain.com%27

Manually crafted request which work:
GET /table?$filter=PartitonKey%20eq%20%'user%40domain.com'

Notice the single quotes are escaped on the incorrect request.

I'm querying Azure tables, not Cosmos DB, maybe there's a breaking change which render this API incompatible with that?

This comment suggested that OrderBy("RowKey") is now possible to sort data in a Table by RowKey achieving a similar behavior to what Storage Table offers.

However, trying a query with this feature failed.

I am using the NuGet Package Microsoft.Azure.Cosmos.Table 1.0.0 in a .NET Core 2.2 based console application.
The Cosmos DB Table API instance i use for testing is located in West Europe. If required i could share the Endpoint via private message.

The source code in my sample is here:

`var account = CloudStorageAccount.Parse(ConnectionString);
var tableClient = account.CreateCloudTableClient();
var table = tableClient.GetTableReference("telemetry");

        string startRowKey = "20190325095347586";
        string endRowKey = "20190325095347586";

        var where =
               TableQuery.CombineFilters(
                   TableQuery.GenerateFilterCondition("RowKey", QueryComparisons.GreaterThanOrEqual,
                       startRowKey),
                   TableOperators.And,
                   TableQuery.GenerateFilterCondition("RowKey", QueryComparisons.LessThanOrEqual,
                       endRowKey));

        var query = table.CreateQuery<TelemetryTableEntity>().Where(where)
            .OrderBy("RowKey");

        var result = table.ExecuteQuery<TelemetryTableEntity>(query);

        TableContinuationToken token = null;
        // Read entities from each query segment.
        do
        {
            TableQuerySegment<TelemetryTableEntity> segment = await table.ExecuteQuerySegmentedAsync(query, token);

            if (segment.RequestCharge.HasValue)
            {
                Console.WriteLine("Request Charge for Query Operation: " + segment.RequestCharge);
            }

            token = segment.ContinuationToken;
            foreach (TelemetryTableEntity entity in segment)
            {
                Console.WriteLine(entity);
            }
        }
        while (token != null);`

Full exception details:

Microsoft.Azure.Cosmos.Table.StorageException: Message: {"errors":[{"severity":"Error","location":{"start":26,"end":28},"code":"SC1001","message":"Syntax error, incorrect syntax near 'by'."}]} ActivityId: 03088207-a91e-4ec2-b1c3-8294167e4b6a, Microsoft.Azure.Documents.Common/2.2.0.0, Windows/10.0.17134 documentdb-netcore-sdk/2.1.3 ---> Microsoft.Azure.Documents.DocumentClientException: Message: {"errors":[{"severity":"Error","location":{"start":26,"end":28},"code":"SC1001","message":"Syntax error, incorrect syntax near 'by'."}]} ActivityId: 03088207-a91e-4ec2-b1c3-8294167e4b6a, Microsoft.Azure.Documents.Common/2.2.0.0, Windows/10.0.17134 documentdb-netcore-sdk/2.1.3 at Microsoft.Azure.Documents.Client.ClientExtensions.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings) at Microsoft.Azure.Documents.GatewayStoreModel.<>c__DisplayClass20_0.<<InvokeAsync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.Azure.Documents.BackoffRetryUtility1.<>c__DisplayClass1_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync(Func1 callbackMethod, Func3 callShouldRetry, Func1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync(Func1 callbackMethod, Func3 callShouldRetry, Func1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback)
at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteAsync(Func1 callbackMethod, IRetryPolicy retryPolicy, CancellationToken cancellationToken, Action1 preRetryCallback) at Microsoft.Azure.Documents.GatewayStoreModel.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, CancellationToken cancellationToken) at Microsoft.Azure.Documents.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Documents.Client.DocumentClient.ExecuteQueryAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.DocumentQueryClient.ExecuteQueryAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.ExecuteQueryRequestInternalAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.ExecuteQueryRequestAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.ExecuteRequestAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.ExecuteOnceAsync(IDocumentClientRetryPolicy retryPolicyInstance, CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.<>c__DisplayClass9_0.<<ExecuteInternalAsync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.Azure.Documents.BackoffRetryUtility1.<>c__DisplayClass1_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync(Func1 callbackMethod, Func3 callShouldRetry, Func1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync(Func1 callbackMethod, Func3 callShouldRetry, Func1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback)
at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteAsync(Func1 callbackMethod, IRetryPolicy retryPolicy, CancellationToken cancellationToken, Action1 preRetryCallback) at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.ExecuteInternalAsync(CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.ExecuteNextAsync(CancellationToken cancellationToken) at Microsoft.Azure.Documents.Query.ProxyDocumentQueryExecutionContext.ExecuteNextAsync(CancellationToken token) at Microsoft.Azure.Documents.Linq.DocumentQuery1.ExecuteNextPrivateAsync[TResponse](CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionQueryHelper.QueryDocumentsAsync[TResult](Nullable1 maxItemCount, String filterString, IList1 selectColumns, TableContinuationToken token, CloudTableClient client, CloudTable table, EntityResolver1 resolver, TableRequestOptions requestOptions, OperationContext operationContext, Boolean isLinqExpression, IList1 orderByItems)
at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionExecutor.<>c__DisplayClass19_02.<<ExecuteQuerySegmentedInternalAsync>b__0>d.MoveNext() --- End of inner exception stack trace --- at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionExecutor.<>c__DisplayClass19_02.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionRetryPolicy.ExecuteUnderRetryPolicy[TResult](Func1 executionMethod, CancellationToken cancellationToken, OperationContext operationContext, TableRequestOptions requestOptions) at CosmosDbTableClientDemo.Program.ExecuteCrossPartitionQueryWithOrderBy() in C:\Dev\Test\CosmosDbTableClientDemo\CosmosDbTableClientDemo\Program.cs:line 96 at CosmosDbTableClientDemo.Program.Main(String[] args) in C:\Dev\Test\CosmosDbTableClientDemo\CosmosDbTableClientDemo\Program.cs:line 15 Request Information RequestID:03088207-a91e-4ec2-b1c3-8294167e4b6a RequestCharge:0 RequestDate: StatusMessage:BadRequest ErrorCode: ErrorMessage:Message: {"errors":[{"severity":"Error","location":{"start":26,"end":28},"code":"SC1001","message":"Syntax error, incorrect syntax near 'by'."}]} ActivityId: 03088207-a91e-4ec2-b1c3-8294167e4b6a, Microsoft.Azure.Documents.Common/2.2.0.0, Windows/10.0.17134 documentdb-netcore-sdk/2.1.3

The structure of TelemetryTableEntity looks like this:

public enum SensorDataType
    {
        Number = 10,
        String = 20,
        Bool = 30,
        BoolArray = 31
    }

    public enum SensorClassification
    {
        Raw = 0,
        Min = 10,
        Avg = 20,
        Max = 30
    }

    public class TelemetryTableEntity : TableEntity
    {
        public DateTime MeasurementTimestamp { get; set; }

        public string SensorId { get; set; }

        public SensorDataType DataType { get; set; }

        public SensorClassification Classification { get; set; }

        public double NumValue { get; set; }
        public string StringValue { get; set; }
        public bool BoolValue { get; set; }
        public string BoolArrayValue { get; set; }
   
    }

As soon as i remove .OrderBy("RowKey") the query executes without exception.

I'm unable to debug on iOS simulator when implementing the Cosmos.Table api with my xamarin.forms app. I'm running Visual Studio Community on Windows with a mac paired on the network. I'm able to debug when I remove all references. Otherwise, the app builds and attempts to deploy on the simulator but exits before entering the first page of the app.

Microsoft.Azure.Cosmos.Table version 1.0.7

Steps to reproduce:

1. Create new xamarin forms project in Visual Studio Community 2019 (16.5.4)
2. Select blank project and update xamarin.forms (4.6.0.726+446-sha.d0dc59903-azdo.3681414) and xamarin.essentials (1.5.3.1)
3. Install Microsoft.Azure.Cosmos.Table (1.0.7) from nuget
4. Add the line "CloudStorageAccount temp;" or "CloudTableClient temp;" to MainPage.MainPage()
5. Connect to mac and try to run the iOS project
6. Observe it attempt to run the app then close

This issue is 100% consistent for me, it works when I remove the references. Here's the last few lines of the output log

Loaded assembly: /Users/tranandrew/Library/Developer/CoreSimulator/Devices/8D020B40-7D3F-46B3-85D5-3540AA7BAF24/data/Containers/Bundle/Application/E77E6302-81C4-410A-A38A-8649B46E9D0D/cosmostest.iOS.app/Microsoft.OData.Edm.dll [External] Loaded assembly: /Users/tranandrew/Library/Developer/CoreSimulator/Devices/8D020B40-7D3F-46B3-85D5-3540AA7BAF24/data/Containers/Bundle/Application/E77E6302-81C4-410A-A38A-8649B46E9D0D/cosmostest.iOS.app/Microsoft.Spatial.dll [External] [HotReload] (2020-05-02 22:14:26.4): INFO: XAML Hot Reload Initializing... [HotReload] (2020-05-02 22:14:26.6): WARN: (cosmostest.iOS) Unknown Breakpoint Hit: UIKit.UIApplication.Main(string[] args, System.IntPtr principal, System.IntPtr delegate)

SDK Version: 1.0.1
Steps to reproduce:

I have a table entity class with a DateTime property. When I query the resulting data using the Azure Document Client, the JSON returned is in a format I don't recognize. The reason I'm using the document client was to try and reproduce how the data is imported to Power BI Desktop. The DateTime appears correct when viewed in the portal or through the Microsoft Azure Storage Explorer.

Here is the resulting JSON value:

{
  "id": "tvmps_050e7bf8eedbaf55c4514c1a8d171f4d8e4b6ef9adbc4eb880abe569f401391c_p",
  "RunId": {
    "$t": 2,
    "$v": "e1c2345a-f638-455c-8970-a83b35dc931b"
  },
  "InstanceId": {
    "$t": 2,
    "$v": "tvmps_050e7bf8eedbaf55c4514c1a8d171f4d8e4b6ef9adbc4eb880abe569f401391c_p"
  },
  "StartTime": {
    "$t": 9,
    "$v": "00636988010372592895"
  },
  "EndTime": {
    "$t": 9,
    "$v": "00636988013347507596"
  },
  "MessagesProcessed": {
    "$t": 16,
    "$v": 0
  },
  "IsCompareInstance": {
    "$t": 8,
    "$v": false
  },
  "$pk": "e1c2345a-f638-455c-8970-a83b35dc931b",
  "$id": "tvmps_050e7bf8eedbaf55c4514c1a8d171f4d8e4b6ef9adbc4eb880abe569f401391c_p",
  "_rid": "uFV6ALUqhzkKAAAAAAAAAA==",
  "_self": "dbs/uFV6AA==/colls/uFV6ALUqhzk=/docs/uFV6ALUqhzkKAAAAAAAAAA==/",
  "_etag": "\"00000000-0000-0000-3b22-125cb40401d5\"",
  "_attachments": "attachments/",
  "_ts": 1563204560
}

The "StartTime" and "EndTime" fields are the ones I don't know what format it is. The documentation claims DateTime are stored as ISO-8601 format dates, which is how they appear in the portal/storage explorer, but that is not how they appear to be stored in the underlying JSON.

When I look at it in the Azure Storage Explorer it shows up like "2019-07-15T15:23:38.804Z" which looks like the correct ISO-8601 format.

I'm trying to determine what that value is from the document query so I can convert the date to a readable date in Power BI.

Exception "Parameter count mismatch"

I am trying to flatten complex object like List and getting this exception.
This is consistent, Am I doing something wrong here?

SDK Version - Microsoft.Azure.Cosmos.Table 1.1.0.6 and NetCore 2.2
Steps to reproduce :

        public class Root
        {
            public int Id { get; protected set; }
            public string DirName { get; protected set; }

            public List<Child> Childrens { get; protected set; }
            
            public Root() { }
            public Root(int id, string dirName, List<Child> childrens)
            {
                Id = id;
                DirName = dirName;
                Childrens = childrens;
            }
        }

        public class Child
        {
            public Child() { }
            public Child(string name)
            {
                Name = name;
            }
            public string Name { get; protected set; }
        }

Root root = new Root(1, "c", new List<Child>() { new Child("sys")});
Dictionary<string, EntityProperty> result = EntityPropertyConverter.Flatten(root, new OperationContext());
IDictionary<string, EntityProperty> result2 = TableEntity.Flatten(root, new OperationContext());

Exception "Parameter count mismatch." at
Dictionary<string, EntityProperty> result = EntityPropertyConverter.Flatten(root, new OperationContext());
Or
IDictionary<string, EntityProperty> result2 = TableEntity.Flatten(root, new OperationContext());

This issue is consistent.

Stack Trace :

   at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.<>c__DisplayClass5_0.<Flatten>b__1(PropertyInfo propertyInfo)
   at System.Linq.Enumerable.All[TSource](IEnumerable`1 source, Func`2 predicate)
   at Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.Flatten(Dictionary`2 propertyDictionary, Object current, String objectPath, HashSet`1 antecedents, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext)
   at Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.<>c__DisplayClass5_0.<Flatten>b__1(PropertyInfo propertyInfo)
   at System.Linq.Enumerable.All[TSource](IEnumerable`1 source, Func`2 predicate)
   at Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.Flatten(Dictionary`2 propertyDictionary, Object current, String objectPath, HashSet`1 antecedents, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext)
   at Microsoft.Azure.Cosmos.Table.EntityPropertyConverter.Flatten(Object root, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext)

Firstly may I state the problem DOES NOT occur when using master connection string from the storage account

** SDK version
1.0.6

** Issue reproduce steps
StorageV2 (general purpose v2), Read-access geo-redundant storage (RA-GRS)
Generate a SAS key with permissions as follows
Allowed services: Table
Allowed resource types: Object
Allowed permissions: Read
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(SASconnectionString);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient(new TableClientConfiguration());

inspect tableClient .StorageUri.SecondaryUri
OR
use TableRequestOptions.LocationMode.SecondaryOnly to force Exception

** Whether the issue is consistent or sporadic
consistent

** Environment
.Net Framework 4.7.2
connection string format is:
BlobEndpoint=https://.blob.core.windows.net/;QueueEndpoint=https://.queue.core.windows.net/;FileEndpoint=https://.file.core.windows.net/;TableEndpoint=https://.table.core.windows.net/;SharedAccessSignature=

** Any other context, such as stack trace or log.

Possibly related to Azure/azure-storage-net#948

We are updating our azure table storage to use Microsoft.Azure.Cosmos.Table 1.0.6 on framework net462.
The service also uses Microsoft.Azure.DocuementDb 2.4.0.

The update is failing to build due to a duplicate type Microsoft.Azure.Documents.Client.DocumentClient which is defined in both Microsoft.Azure.DocumentDb.Core 2.1.3 (included from Microsoft.Azure.Cosmos.Table 1.0.6) and Microsoft.Azure.Documents.Client 2.4.0 (included from Microsoft.Azure.DocumentDb 2.4.0).

"Error CS0433 The type 'DocumentClient' exists in both 'Microsoft.Azure.DocumentDB.Core, Version=2.1.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' and 'Microsoft.Azure.Documents.Client, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'"

Can anyone advise which versions we should be on to avoid this?
Thanks.

When using the code below, I get an "Object reference not set to an instance of an object" on the final line. This happens with Microsoft.Azure.Cosmos.Table version 1.07 and 1.06, however works perfectly fine on 1.05. Is there a breaking change with this new version. Please see code to reproduce and full stack trace below:

        CloudStorageAccount storageAccount =
        CloudStorageAccount.Parse("ConnectionStringRemovedForSecurity");
        CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

        CloudTable table = tableClient.GetTableReference("people");
        table.CreateIfNotExists();

        CustomerEntity customer1 = new CustomerEntity("Smith", "John");
        customer1.Email = "[email protected]";
        customer1.PhoneNumber = "111-555-0000";

        // Create the TableOperation that inserts the customer entity.
        var insertOperation = TableOperation.Insert(customer1);

        //Execute the insert operation.
        table.Execute(insertOperation);

        // Read storage
        TableQuery<CustomerEntity> query =
           new TableQuery<CustomerEntity>()
              .Where(TableQuery.GenerateFilterCondition("PartitionKey",
                  QueryComparisons.Equal, "Smith"));
        var list = table.ExecuteQuery(query).ToList();

Stack Trace
[NullReferenceException: Object reference not set to an instance of an object.]
Microsoft.Azure.Documents.<>c.<.cctor>b__9_0() +169
System.Lazy1.CreateValue() +161 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +27 System.Lazy1.get_Value() +12605132
Microsoft.Azure.Documents.CustomTypeExtensions.ByPassQueryParsing() +129
Microsoft.Azure.Documents.Query.d__3.MoveNext() +1253
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +102
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +64
Microsoft.Azure.Documents.Linq.d__361.MoveNext() +233 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +102 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +64 System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) +26 Microsoft.Azure.Cosmos.Table.Extensions.<QueryDocumentsAsync>d__21.MoveNext() +673
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +102
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +64
Microsoft.Azure.Cosmos.Table.Extensions.<b__0>d.MoveNext() +907

[StorageException: Object reference not set to an instance of an object.]
Microsoft.Azure.Cosmos.Table.Extensions.<b__0>d.MoveNext() +951
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +102
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +64
Microsoft.Azure.Cosmos.Table.Extensions.d__21.MoveNext() +893 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +102 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +64 Microsoft.Azure.Cosmos.Table.Extensions.TableExtensionExecutor.ExecuteQuerySegmented(TableQuery1 query, TableContinuationToken token, CloudTableClient client, CloudTable table, EntityResolver1 resolver, TableRequestOptions requestOptions, OperationContext operationContext) +270 Microsoft.Azure.Cosmos.Table.TableQuery1.ExecuteQuerySegmentedInternal(TableContinuationToken token, CloudTableClient client, CloudTable table, TableRequestOptions requestOptions, OperationContext operationContext) +367
Microsoft.Azure.Cosmos.Table.<>c__DisplayClass42_0.b__0(TableContinuationToken continuationToken) +317
Microsoft.Azure.Cosmos.Table.d__21.MoveNext() +66 System.Collections.Generic.List1..ctor(IEnumerable1 collection) +186 System.Linq.Enumerable.ToList(IEnumerable1 source) +61
CosmosTest.Controllers.HomeController.Index() in C:\Users\kelly\Dropbox (KWC)\KWC Private\Dev\Cosmos\CosmosTest\CosmosTest\Controllers\HomeController.cs:36
lambda_method(Closure , ControllerBase , Object[] ) +61
System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters) +14
System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary2 parameters) +157 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary2 parameters) +27
System.Web.Mvc.Async.<>c.b__9_0(IAsyncResult asyncResult, ActionInvocation innerInvokeState) +22
System.Web.Mvc.Async.WrappedAsyncResult2.CallEndDelegate(IAsyncResult asyncResult) +29 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) +32
System.Web.Mvc.Async.<>c__DisplayClass11_0.b__0() +58
System.Web.Mvc.Async.<>c__DisplayClass11_2.b__2() +228
System.Web.Mvc.Async.<>c__DisplayClass7_0.b__1(IAsyncResult asyncResult) +10
System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) +34
System.Web.Mvc.Async.<>c__DisplayClass3_6.b__4() +35
System.Web.Mvc.Async.<>c__DisplayClass3_1.b__1(IAsyncResult asyncResult) +100
System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult) +27
System.Web.Mvc.<>c.b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState) +11
System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +29 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) +45
System.Web.Mvc.<>c.b__151_2(IAsyncResult asyncResult, Controller controller) +13
System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +22 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult) +26
System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult) +10
System.Web.Mvc.<>c.b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState) +28
System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +29 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) +28
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result) +9
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +9836613
System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +50
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +163