Below is an extract from the logs after running
airflow test cm_etlp.parse_and_import_huawei_2g import_huawei_2g_cm_data "2018-03-10 23:13:19"
`
[2018-03-11 21:19:40,061] {bash_operator.py:94} INFO -
[2018-03-11 21:19:40,061] {bash_operator.py:94} INFO -
[2018-03-11 21:19:40,062] {bash_operator.py:94} INFO - -- AITFOTHPARA
[2018-03-11 21:19:40,062] {bash_operator.py:94} INFO - -- ---------------------------
[2018-03-11 21:19:40,062] {bash_operator.py:94} INFO - TRUNCATE TABLE hua_cm_2g.AITFOTHPARA;
[2018-03-11 21:19:40,197] {bash_operator.py:94} INFO - TRUNCATE TABLE
[2018-03-11 21:19:40,197] {bash_operator.py:94} INFO - \COPY hua_cm_2g.AITFOTHPARA FROM '/mediation/data/cm/huawei/2g/parsed/in/AITFOTHPARA.csv' CSV HEADER;
[2018-03-11 21:19:40,214] {bash_operator.py:94} INFO - psql:/mediation/conf/cm/hua_cm_2g_nbi_loader.cfg:8: ERROR: extra data after last expected column
[2018-03-11 21:19:40,214] {bash_operator.py:94} INFO - CONTEXT: COPY aitfothpara, line 2: "BSCBARG_20171017_RF.txt,2017-10-17 06:03:04,0,V900R019ENGC10SPC528,10.26.73.203,GO,,,,,,,,2,YES,NO,0..."
Cause:
The huawei MML parser is using the same parser config file as the ift-N XML parser. The result is the csv files have extra columns. The columns below that are produced by the Ift-N XML parser are not present in the MML files.
- ne_xsitype
- netype
- neversion
- neid
- module_type
- module_remark
- module_productversion
Unfortunately, the hua_cm_2g schema tables expect these column to be present in the imported csv.
Solution:
- Provide a seperate configuration file for the MML parser
- To maintain the column order configure the MML parser config file (hua_cm_2g_mml_parser.cfg) so that it has the same number of column as expected.
FileName --> FileName
varDateTime --> varDateTime
BSCID --> ne_xsitype
BAM_VERSION --> netype
OMU_IP --> neversion
MBSC MODE --> neid
module_type --> leave as is
module_remark --> leave as is
module_productversion --> leave as is