charyeezy / aws-5g-network-performance-analytics Goto Github PK

Build a cloud-native network performance analytics solution on AWS for wireless service providers

This GitHub repository contains the code to support the AWS Big Data Blog post Build a cloud-native network performance analytics solution on AWS for wireless service providers. It contains the Python code for the AWS Lambda described in the post and some example source (XML) and output (CSV, JSON) files.

Collecting data using Amazon Kinesis Data Firehose or AWS Transfer for SFTP

For information about collecting PM files via element managers, see chapter 5 of Technical Specification 3GPP TS 31.432 on the 3GPP website.

Element managers act as collectors of XML measurement files that the network elements provide. They can send the files to an Amazon S3 bucket via Amazon Kinesis Data Firehose or AWS Transfer for SFTP.

Transforming data using a Lambda function

The Lambda function in this repository is written in Python 3.7 and it is associated with the ObjectCreated event type of the destination S3 bucket rawxml prefix (i.e. s3://wireless-pm/rawxml). We have created a Lambda layer to resolve the dependency of the xmltodict library. The function runs every time a new XML file is saved in this location.

The function transposes the XML files, converts them into CSV or JSON (depending on the value set in the function’s output_format environment variable), and formats the files with one record per measurement (measData), measure type, and value (measInfo).

The technical specification 3GPP TS 32.435 document on the 3GPP website provides three example XML files in the Annex section. As an example, these are also included in this repository, together with the transformed output in CSV and JSON format.

You can change the output field names in the get_record_header static method of the GPPXml class defined in the Lambda function. The fields md_mi_meas_name and md_mi_meas_value contain the measure name and measure value, respectively.

The transformed CSV and JSON files are saved in the raw_transform_csv and raw_transform_json prefixes, respectively, in the S3 bucket (only one format is created for each execution, depending on the value of the output_format environment variable). The following screenshot shows the S3 bucket overview on the Amazon S3 console.

License

This library is licensed under the MIT-0 License. See the LICENSE file.