https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/countoffunction

They are useful in summarize :)

For example, currently strcat_array fails because it requires:
strcat_array(dynamic(["A", "B", "C"]), ",")
while pykusto produces:
strcat_array(("A", "B", "C"), ",")

Should probably use dynamic()

Currently, the "render" function of Query has two options:

1. If the Query contains a PyKusto table, it creates: " table | query" render
2. If there is no PyKusto table, it emits the "table" part and creates only the "query".

However, if we use PyKusto to render the query only, we need another option - to produce " table | query" based on the table name, without providing a real Table.

f.strcat(f.tostring(col.claims_parsed['uti']),"==")
creates
strcat(tostring(col.claims_parsed['uti']),"==",)

Informative error messages are important, so when we have a test with assertRaises, we should verify not only the type of the exception, but also the contained error message

Example of desired Kusto query:
| extend d = pack('Name', name, 'Roles', pack_array(role1, role2, role3))
Where "name", "role1", "role2", "role3" are columns.

Suggested pykusto syntax:
.extend(d={'Name': col.name, 'Roled': [col.role1, col.role2, col.role3]})

if a column name is, for example, 'title', it should be parsed as "['title']", otherwise, Kusto throws an exception

TypeError Traceback (most recent call last)
in
----> 1 col['asd']

c:\python\python37-32\lib\site-packages\pykusto\expressions.py in getitem(self, name)
576
577 def getitem(self, name: str) -> Column:
--> 578 return Column[name]
579
580

TypeError: 'type' object is not subscriptable

• Introduce DynamicExpression and AnyExpression
• ArrayExpression and MappingExpression should return AnyExpression
• Delete Column.getitem
• Fix inheritance tree

If the joined table is from a different cluster, render it using the following syntax: cluster("a").database("b").table("c")

.extend(timeDelta=f.iff(col.day - col.day1 == 0, timedelta(last_seen_period), col.day - col.day1))

Subtraction between two AnyTypeColumn yields NumberExpression instead of TimespanExpression in this case.

examples:
Query().where(f.to_double(100 * t.numberField) > 0.2).render()
returns: TypeError: unsupported operand type(s) for *: 'int' and 'AnyTypeColumn'

Query().where(100 * f.to_double(t.numberField) > 0.2).render()
returns: TypeError: unsupported operand type(s) for *: 'int' and 'NumberExpression'

Query().where(col.of(100) * f.to_double(t.numberField) > 0.2).render()
renders as : "| where (['100'] * (todouble(numberField))) > 0.2" which doesn't compile.

I didn't have any more ideas...

Create a Kusto DataFrame class which is backed by a Kusto table instead of in- memory (similar to the Spark DataFrame class)

and while at it, more array operations...

In some tables, there is no "timestamp" field, but instead, we use the ingestion_time() function to filter the relevant period.

Working without it is unbearably slow and almost always fails as we can't filter first by timestamp.
That's why I think it's a relatively urgent feature.

Complex query strings generated by "render()" are hard to read, a pretty function could help debugging

kql equivalent:
| extend prod="prod=1"

f.iff function checks if both return options return the same type.
consider this code:

Query.extend(some_data_count=
                                  f.iff(some_condition,
                                        f.array_length(col.some_date),
                                        0)))

throws:
pykusto/functions.py", line 271, in iff raise TypeError("The second and third arguments must be of the same type") TypeError: The second and third arguments must be of the same type

Now that pykusto is on PyPI, you can update the installation instruction in the main readme file to install there rather than from github.

Expression classes, Query classes, etc.

Currently the operator bug_unpack of dynamic field is not supported

By default the column generated by mv-expand is of dynamic type. Usually it has some underlying concrete type, and it is useful to convert it to that type.

Example query which is unsupported:
| mv-expand name to typeof(string)

Suggested pykusto syntax:
.mv_expand(col.name.to_type(TypeName.STRING))

https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/best-practices

Provide link in documentation, perhaps raise warnings for certain violations.

The following syntax is not working:
.extend(timeDelta=iff(col.day-col.day1 == 0, timedelta(DAYS_BACK_DIFF), col.day-col.day1))

https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sampledistinctoperator

mv_epxand has an assignment option and currently pykusto doesn't support it.

name attribute should be renamed to _name