Comments (1)
Ended up doing some preprocessing on those specifically. Not a perfect solution, but does what I needed so far...
import sys
!{sys.executable} -m pip install flatten_json pandas boto3
import pandas as pd
import json
import csv
from pandas.io.json import json_normalize
def get_attribute(data, attribute, default_value):
return data.get(attribute) or default_value
def flatten_with_keys(data, parent_object, tag_name, key_name, value_name, concat_name):
data_list = data[parent_object]
df_data = pd.DataFrame([])
for i in range(len(data_list)):
data_tags = get_attribute(data_list[i], tag_name, None)
# used if statement as to deal with if the tag_name doesn't exist.
if data_tags:
df = pd.DataFrame(data_list[i][tag_name]).T
df.columns = df.loc[key_name]
df = df.loc[value_name].to_frame(name=data_list[i][concat_name])
df_data = pd.concat([df_data, df.T], sort=True)
dic_flattened = [flatten(d,'.',root_keys_to_ignore={tag_name}) for d in data_list]
df_flattened = pd.DataFrame(dic_flattened)
df_flattened.set_index(concat_name, inplace=True)
df_flattened
result = pd.concat([df_flattened, df_data], axis=1, sort=False)
return result
Example if you're running with aws cli where you want to flatten the vpc data. It's a bit of extra things, and not sure if it's going to work all the time.
import sys
!{sys.executable} -m pip install boto3
import boto3
ec2 = boto3.client('ec2')
vpc = ec2.describe_vpcs()
result_flattened = flatten_with_keys(vpc, 'Vpcs', 'Tags', 'Key', 'Value', 'VpcId')
result_flattened
from flatten.
Related Issues (20)
- More complex flattening? HOT 2
- ModuleNotFoundError for v0.1.8 HOT 11
- The util namespace HOT 4
- Still maintained? HOT 1
- flatten_preserve_lists returns a list instead of dict HOT 1
- flatten_preserve_lists is failed to flatten json data which includes an array with more than 10 rows HOT 1
- Error if a key is substring of another key HOT 1
- flatten_preserve_lists is failed to flatten data which has array in array
- Exploding non-nested lists rather than unpacking into separate columns
- Two different install instructions on PyPI HOT 2
- flatten option for root keys to ignore keeps some data from the root keys
- add type hints or stub file
- Several issues with flatten_preserve_lists
- Issue with unflatten
- Unflat and flat with list format a[0] instead of a.0
- Order preserving
- Wheel on pypi? HOT 1
- True Flattening
- Handling of Escaped Double Quotes
- The issue with the performance of unflattening process
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flatten.