accessing item from list within a list about glom HOT 7 CLOSED

Hey @wdieter! Great example. First off, I think you're absolutely right that unnesting and flattening are common operations that we're going to have to look at supporting natively. For this simple example, flattening lists can be achieved using the old sum() function:

glom(data, {"names": ("data", ["name"]), "swims": (("data", ["custom"], ["hobbies"], [['swimming']]), lambda t: sum(t, [])) })
# {'swims': [True, False], 'names': ['bob', 'joey']}

You'll notice all I added to your spec was that lambda at the end. That's one nice affordance of glom, you can always drop to a function. Still, if you run into more examples like these, please keep us posted, because they're invaluable when it comes to building the native functionality. Thanks again!

from glom.

T[0] can also help un-nest things if you know there is going to be exactly one item.

spec = {"names": ("data", ["name"]), "swims": ("data", ["custom"], ["hobbies"], [(['swimming'], T[0])])}
glom(data, spec)
# {'swims': [True, False], 'names': ['bob', 'joey']}

from glom.

@wdieter sorry for the (almost) duplicate, I've missed your issue when creating mine.
@mahmoud @kurtbrose those helped a lot, I've managed to acomplish what I wanted.

from glom.

Thanks @mahmoud and @kurtbrose for the help there. I looked into the the issue a bit more and I'm not sure if the sum method works when there is more than one level of nesting or different levels of nesting, eg. [[['a']], [[['b']]]]. I didn't look into the solution utilizing T but my guess is you would also need to change it to something like [[(['swimming'], T[0][0])]]? This may work but the many layers of lists and tuples isn't so nice in my opinion. Also you already defined the access level in the first part of the spec so it feels a bit repetitive.

I think a flatten function should be able to handle one or more levels of nesting eg. both [['a'], ['b']] and [[['a']], [['b']]]. I found this recursive solution at https://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists that seems to work with one or many levels.

def flatten(x):
    if isinstance(x, collections.Iterable) and not isinstance(x, (str, bytes)):
        return [a for i in x for a in flatten(i)]
    else:
        return [x]

So if my dictionary now has three lists:

data2 = {"data":[{'name':'bob',
             'custom':{
                 'species':'person',
                 'hobbies': [{
                     'exercise': [{
                         "swimming": True}]
                     }]
                 }
            },
            {'name':'joey',
             'custom':{
                 'species':'cat',
                 'hobbies': [{
                     'exercise': [{
                         "swimming": True}]
                     }]
                 }
            }]
    }

My spec would be:

s2 = {"names": ("data", ["name"]),
"swims": ("data", ["custom"], ["hobbies"], [["exercise"]], [[['swimming']]],  lambda t: flatten(t))}

giving me

{'names': ['bob', 'joey'], 'swims': [True, False]}

And this flatten function works for the orginal dict too

from glom.

One more issue with sum(t, []) and also with the flatten func above: empty lists will be discarded. This can be an issue if you're trying to use glom to get a dict that can be passed to a dataframe and need to preserve consistent row counts where row[x] should represent one entity

In [23]: l = [['a'], [], ['b']]

In [24]: sum(l, [])
Out[24]: ['a', 'b']

My workaround:

In [25]: [item[0] if item else 'blank' for item in l]
Out[25]: ['a', 'blank', 'b']

from glom.

what about Coalesce(T[0])?

>>> glom([ [1], [], [3] ], [Coalesce(T[0], default=None)])
[1, None, 3]

from glom.

I think we've got this one sorted, I'm going to close out the issue. Feel free to chat further, and a PR for a snippet to this effect would be great!

from glom.