Union columns can never be `NULL` about arrow-datafusion HOT 7 CLOSED

Have custom logic for unions that looks up the child array to determine if the value is null

+1 for second option. I think we should check the children's nullability.

from arrow-datafusion.

I've proposed a fix in #11321.

from arrow-datafusion.

@alamb any idea on where I would start looking to try and fix this?

from arrow-datafusion.

I would suggest writing a standlone test case / reproducer as the first step

Then I suspect we can either help you find the code needed to be fixed (or maybe even someone would be interested in fixing it themselves)

from arrow-datafusion.

See #11314 as a demonstration of the problem for both dense and sparse unions.

After a bit of investigation, the issues lies in the first instance with

Then with this code in arrow-rs:

/// Returns a non-null [BooleanArray] with whether each value of the array is null.
/// # Error
/// This function never errors.
/// # Example
/// ...
pub fn is_null(input: &dyn Array) -> Result<BooleanArray, ArrowError> {
    let values = match input.logical_nulls() {
        None => BooleanBuffer::new_unset(input.len()),
        Some(nulls) => !nulls.inner(),
    };

    Ok(BooleanArray::new(values, None))
}

And then with this code

    /// Union types always return non null as there is no validity buffer.
    /// To check validity correctly you must check the underlying vector.
    fn is_null(&self, _index: usize) -> bool {
        false
    }

Ultimately with the spec

Unlike other data types, unions do not have their own validity bitmap. Instead,
the nullness of each slot is determined exclusively by the child arrays which
are composed to create the union.

Basically arrow is saying "we're not going to tell you if a union is null, you need to look in the child arrays", but datafusion isn't listening and is just asking the union if it's null in the naive way.

Two options to move forward as far as I can tell:

1. Decide unions in DF can never be null — I'll need to abandon unions in datafusion-functions-json and just return strings everywhere
2. Have custom logic for unions that looks up the child array to determine if the value is null

If (as I hope) we go for the second option, there's also the issue (as demonstrated by #11314) that the representation of "null" union items doesn't match other types, it shows {A=} instead of an empty string.

from arrow-datafusion.

I suppose there's a third option of updating arrow-rs to correctly calculate if a UnionArray is null, but I presume that works take much longer

from arrow-datafusion.

I suppose there's a third option of updating arrow-rs to correctly calculate if a UnionArray is null, but I presume that works take much longer

It would likely take longer

Note there is a method that takes into account child nullability that perhaps we could use instead of is_null:
https://docs.rs/arrow/latest/arrow/array/trait.Array.html#method.logical_nulls

Update: it turns out logical_nulls is incorrect for UnionArray: apache/arrow-rs#6017

from arrow-datafusion.