networktocode / circuit-maintenance-parser Goto Github PK

Environment

• circuit_maintenance_parser version:

Proposed Functionality

Use new Pydantic 2.0

Use Case

Environment

• Python version:
• circuit_maintenance_parser version:

Expected Behavior

Observed Behavior

Check CI execution from auto PRs

Error

RecursionError: maximum recursion depth exceeded in comparison

Steps to Reproduce

Proposed Functionality

Now that we can combine multiple DataParts we could take Stamp from the Email Date, and make it mandatory.

Use Case

This will add extra info to the Maintenance, about when it was created

Environment

• circuit_maintenance_parser version: 2.0.8

Proposed Functionality

Automated Parser/End-to-end test discoverability, so the test cases could be automatically discovered following some pattern

Use Case

Simplify development time, by dynamically fetching test cases, so no need to manually explicit them,

Environment

• Python version: 3.9
• circuit_maintenance_parser version: 2.0.3

Expected Behavior

Only unique Circuit IDs should be in one notification

Observed Behavior

Some Parsers return repeated Circuit IDs in the same notification

Steps to Reproduce

Proposed Functionality

Remove support for Python 3.6 as it is EOL.

Use Case

Allow updates to various dependencies that have already dropped support for 3.6.

Environment

• Python version: 3.x
• circuit_maintenance_parser version: latest

Expected Behavior

I expected that the invoke tasks would allow for the testing

Observed Behavior

• Errors due to no Dockerfile

Steps to Reproduce

1. clone repo
2. run invoke build
3. Get error that Dockerfile was not present

Environment

• Python version: <3.9>
• circuit_maintenance_parser version: <2.0.3>

We expect for the Zayo parser to pull the following fields from maintenance emails:

Circuit Id
Expected Impact
A Location Address
Z Location Address
Legacy Circuit Id

The Zayo parser is looking for data from the following fields:

Circuit Id
Expected Impact
A Location CLLI
Z Location CLLI
Legacy Circuit Id

Proposed Fix

diff --git a/circuit_maintenance_parser/parsers/zayo.py b/circuit_maintenance_parser/parsers/zayo.py
index 49e9b68..bfc78b7 100644
--- a/circuit_maintenance_parser/parsers/zayo.py
+++ b/circuit_maintenance_parser/parsers/zayo.py
@@ -64,8 +64,8 @@ class HtmlParserZayo1(Html):
             if len(head_row) < 5 or [self.clean_line(line) for line in head_row[:5]] != [
                 "Circuit Id",
                 "Expected Impact",
-                "A Location CLLI",
-                "Z Location CLLI",
+                "A Location Address",
+                "Z Location Address",
                 "Legacy Circuit Id",
             ]:
                 logger.warning("Table headers are not as expected: %s", head_row)

Environment

• Python version: 3.11
• circuit_maintenance_parser version: 2.6.0

Expected Behavior

no errors while installing.

Observed Behavior

when installing napalm-panos we see the lxml version error.

Steps to Reproduce

1. install lxml > 5 will circuit-maintenance-parser breaks

Environment

• Python version:
• circuit_maintenance_parser version:

Expected Behavior

All test data should be tested

Observed Behavior

https://github.com/networktocode/circuit-maintenance-parser/blob/develop/tests/unit/test_e2e.py#L643-L653

Steps to Reproduce

Environment

• Python version: 3.9.6
• circuit_maintenance_parser version: 2.1.0

Expected Behavior

Properly parse the email attached.

Observed Behavior

$ circuit-maintenance-parser --data-file a.eml --data-type email --provider-type zayo
Traceback (most recent call last):
  File "/home/jmaslak/.pyenv/versions/3.9.6/bin/circuit-maintenance-parser", line 8, in <module>
    sys.exit(main())
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/circuit_maintenance_parser/cli.py", line 47, in main
    parsed_notifications = provider.get_maintenances(data)
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/circuit_maintenance_parser/provider.py", line 118, in get_maintenances
    if self.exclude_filter_check(data) or not self.include_filter_check(data):
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/circuit_maintenance_parser/provider.py", line 79, in include_filter_check
    return self.filter_check(self._include_filter, data, "include")
  File "/home/jmaslak/.pyenv/versions/3.9.6/lib/python3.9/site-packages/circuit_maintenance_parser/provider.py", line 97, in filter_check
    data_part_content = data_part.content.decode().replace("\r", "").replace("\n", "")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 2254: invalid continuation byte

Steps to Reproduce

See above. The problem in the email appears to be the =C9's in the raw .EML file, that is part of the email signature. Note I annonymized key aspects of the EML file but that does not restrict reproduction of the bug. I have gzipped and attached the email file (since Github doesn't allow an EML that isn't zipped).

a.eml.gz

Environment

• Python version:3.9
• circuit_maintenance_parser version: 2.1.0

As part of the initial setup, I ran invoke tests --local

Expected Behavior

I expect all tests to pass

Observed Behavior

One of the unit test failed

tests/unit/test_e2e.py:516:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Seaborn()
data = NotificationData(data_parts=[DataPart(type='email-header-date', content=b'Mon, 16 Aug 2021 17:29:56 +0100'), DataPart(...px"> <br></div></blockquote></div> <div><br></div></div><div id="m_-8814736200534887510ZDeskInteg"></div><br></div>')])

    def get_maintenances(self, data: NotificationData) -> Iterable[Maintenance]:
        """Main entry method that will use the defined `_processors` in order to extract the `Maintenances` from data."""
        provider_name = self.__class__.__name__
        error_message = ""
        related_exceptions = []

        if self.exclude_filter_check(data) or not self.include_filter_check(data):
            logger.debug("Skipping notification %s due filtering policy for %s.", data, self.__class__.__name__)
            return []

        for processor in self._processors:
            try:
                return processor.process(data, self.get_extended_data())
            except ProcessorError as exc:
                process_error_message = (
                    f"- Processor {processor.__class__.__name__} from {provider_name} failed due to: %s\n"
                )
                logger.debug(process_error_message, traceback.format_exc())

                related_exc = rgetattr(exc, "__cause__")
                error_message += process_error_message % related_exc
                related_exceptions.append(exc)
                continue

>       raise ProviderError(
            (f"Failed creating Maintenance notification for {provider_name}.\nDetails:\n{error_message}"),
            related_exceptions=related_exceptions,
        )
E       circuit_maintenance_parser.errors.ProviderError: Failed creating Maintenance notification for Seaborn.
E       Details:
E       - Processor CombinedProcessor from Seaborn failed due to: SubjectParserSeaborn1 parser was not able to extract the expected data for each maintenance.
E         - Raw content: b'Fwd: [rd-notices] Re:[## 99999 ##] Emergency Maintenance Notification\r\n CID: AAA-AAAAA-AAAAA-AAA1-0000-00 TT#11111'
E         - Result: [{}]
E       - Processor CombinedProcessor from Seaborn failed due to: 5 validation errors for Maintenance
E       maintenance_id
E         field required (type=value_error.missing)
E       circuits
E         At least one circuit has to be included in the maintenance (type=value_error)
E       start
E         field required (type=value_error.missing)
E       end
E         field required (type=value_error.missing)
E       status
E         field required (type=value_error.missing)

circuit_maintenance_parser/provider.py:136: ProviderError

Steps to Reproduce

1. invoke tests --local

Environment

• Python version: 3.10.9
• circuit_maintenance_parser version: 2.2.0

Expected Behavior

Equinix maintenance notification is parsed correctly

Observed Behavior

Email parsing fails with error

Provider processing failed: Failed creating Maintenance notification for Equinix.
Details:
- Processor CombinedProcessor from Equinix failed due to: too many values to unpack (expected 3)

Steps to Reproduce

1. circuit-maintenance-parser --data-file equinix.eml --data-type email --provider-type equinix

Environment

• circuit_maintenance_parser version:

Proposed Functionality

Besides the data representing the Maintenance, the response could incorporate metadata to understand things like:

• Which Provider has been used to parse it?
• Which Parsers have been used?
• If the parsing consumed some resources, a summary of the amount of them (for instance, Openai tokens)

Use Case

Environment

• Python version:
• circuit_maintenance_parser version: 2.0.4 and later

Expected Behavior

No significant increase in memory consumption when loading the circuit_maintenance_parser library after upgrading from 2.0.3 to 2.0.4

Observed Behavior

Memory usage jumps by over 400 MB after import circuit_maintenance_parser with 2.0.4.

This appears to be due to #104, specifically due to a Geolocator object being instantiated at library load time, which in turn calls tzwhere.tzwhere(), which appears to allocate a large amount of memory.

Steps to Reproduce

1. python
2. import circuit_maintenance_parser

Environment

• Python version: 3.10.9
• circuit_maintenance_parser version: 2.2.0

Expected Behavior

Verizon maintenance notification is parsed correctly

Observed Behavior

Email parsing fails with error

Provider processing failed: Failed creating Maintenance notification for Verizon.
Details:
- Processor CombinedProcessor from Verizon failed due to: 'NoneType' object has no attribute 'find_all'

Steps to Reproduce

1. circuit-maintenance-parser --data-file verizon.eml --data-type email --provider-type verizon

Proposed Functionality

Add CLI command to parse the data from a file

Use Case

Interact with other tooling than Python code

Environment

• circuit_maintenance_parser version: v2.2.2

Return multiple maintenance windows in the event where a vendor sends us multiple activity dates. This feature should allow multiple maintenance windows to be returned as separate fields rather than one big window. Currently, multiple windows from vendor are being clumped into one large window (earliest start time + latest end time found).

Proposed Functionality

The proposed functionality of this would be allowing maintenance windows to only be active for the window that the vendor specifies, not all the time in between the earliest start time and latest end time. This increases availability of network resources in the event where operators move traffic away from network links during maintenance windows.

Use Case

An example email (sanitized) that we would receive:

                # Some Zayo notifications may include multiple activity dates.
                # For example, given:
                #
                # 1st Activity Date
                # 01-Nov-2021 00:01 to 01-Nov-2021 05:00 ( Mountain )
                # 01-Nov-2021 06:01 to 01-Nov-2021 11:00 ( GMT )
                #
                # 2nd Activity Date
                # 02-Nov-2021 00:01 to 02-Nov-2021 05:00 ( Mountain )
                # 02-Nov-2021 06:01 to 02-Nov-2021 11:00 ( GMT )
                #
                # 3rd Activity Date
                # 03-Nov-2021 00:01 to 03-Nov-2021 05:00 ( Mountain )
                # 03-Nov-2021 06:01 to 03-Nov-2021 11:00 ( GMT )

By testing the email file against circuit-maintenance-parser, we should be getting multiple maintenance objects rather than just one with the earliest start time and latest end time:

#circuit-maintenance-parser --data-file ~/zayo_test_email.eml --data-type html --provider-type zayo

Circuit Maintenance Notification #0
{
  "account": "Test",
  "circuits": [
    {
      "circuit_id": "ZAYO/TEST/CIRCUIT",
      "impact": "OUTAGE"
    }
  ],
  "end": 1635764400,
  "maintenance_id": "TTN-0001234567",
  "organizer": "[email protected]",
  "provider": "zayo",
  "sequence": 1,
  "stamp": 1635573660,
  "start": 1635746460,
  "status": "CONFIRMED",
  "summary": "Zayo Third-Party Provider will implement maintenance to perform temporary fiber relocation in order to proactively avoid outages.",
  "uid": "0"
}

Circuit Maintenance Notification #1
{
  "account": "Test",
  "circuits": [
    {
      "circuit_id": "ZAYO/TEST/CIRCUIT",
      "impact": "OUTAGE"
    }
  ],
  "end": 1635850800,
  "maintenance_id": "TTN-0001234567",
  "organizer": "[email protected]",
  "provider": "zayo",
  "sequence": 1,
  "stamp": 1635573660,
  "start": 1635832860,
  "status": "CONFIRMED",
  "summary": "Zayo Third-Party Provider will implement maintenance to perform temporary fiber relocation in order to proactively avoid outages.",
  "uid": "1"
}

Circuit Maintenance Notification #2
{
  "account": "Test",
  "circuits": [
    {
      "circuit_id": "ZAYO/TEST/CIRCUIT",
      "impact": "OUTAGE"
    }
  ],
  "end": 1635937200,
  "maintenance_id": "TTN-0001234567",
  "organizer": "[email protected]",
  "provider": "zayo",
  "sequence": 1,
  "stamp": 1635573660,
  "start": 1635919260,
  "status": "CONFIRMED",
  "summary": "Zayo Third-Party Provider will implement maintenance to perform temporary fiber relocation in order to proactively avoid outages.",
  "uid": "2"
}

Proposed Functionality

Add parsing for AWS Direct Connect notifications

Use Case

EXAMPLE 1
Subject

Reminder: AWS Direct Connect Planned Maintenance Notification [AWS Account: 11111111]

Body

Hello,

Planned maintenance has been scheduled on an AWS Direct Connect endpoint in Equinix SG2, Singapore, SGP from Mon, 13 Sep 2021 19:02:00 GMT to Tue, 14 Sep 2021 02:02:00 GMT for 7 hours. During this maintenance window, your AWS Direct Connect services listed below may become unavailable.

dxvif-XXXXXXXX

This maintenance is scheduled to avoid disrupting redundant connections at the same time.

If you encounter any problems with your connection after the end of this maintenance window, please contact AWS Support[1].

[1] https://aws.amazon.com/support

Sincerely,
Amazon Web Services

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210.

EXAMPLE 2
Subject

AWS Direct Connect Maintenance Complete [AWS Account: 111111]

Body

Hello,

We would like to inform you that the maintenance that was scheduled for AWS Direct Connect endpoint in Equinix LD5, Slough, GBR from Thu, 9 Sep 2021 03:30:00 GMT to Thu, 9 Sep 2021 04:30:00 GMT has been completed. Please find below your AWS Direct Connect services that would have been affected by this maintenance.

dxvif-xxxx
dxcon-ccccc


If you have any questions regarding the completed work, or if you would like to report a fault or adjust your contact information, please contact AWS support at https://aws.amazon.com/support .

Sincerely,
Amazon Web Services

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210.

Environment

• circuit_maintenance_parser version: 2.5.0

Proposed Functionality

Decouple core and parser functionalities to allow custom import of parser outside of the upstream library.

Use Case

Environment

• circuit_maintenance_parser version: 2.2.0

A test parser for Cogent notification emails with Content-Type: text/plain

Proposed Functionality

A text parser for cogent would provide additional coverage over incoming notifications by including emails of type Content-Type: text/plain

Use Case

Not sure if this is the right way to bring this issue up. Some of my cogent notifications are not able to be parsed on their own and I believe it is due to cogent's email formatting.

Cogent email of Content-Type: text/plain are not able to be parsed by the current Cogent provider parsers.
Example provided below:

─➤  circuit-maintenance-parser --data-file "cogent_eml.eml" --data-type email --provider-type cogent                                                                  2 ↵
Provider processing failed: Failed creating Maintenance notification for Cogent.
Details:
- Processor CombinedProcessor from Cogent failed due to: 6 validation errors for Maintenance
account
  field required (type=value_error.missing)
maintenance_id
  field required (type=value_error.missing)
circuits
  field required (type=value_error.missing)
start
  field required (type=value_error.missing)
end
  field required (type=value_error.missing)
status
  field required (type=value_error.missing)

Example email:
cogent_eml.eml.txt

Environment

• circuit_maintenance_parser version:

Proposed Functionality

Allow user to define a local folder/repository containing additional parser classes that can be used to parse email notifications.

Use Case

Users can quickly implement parsing extensions to support providers in their environment.
These parsing updates can still be submitted back upstream to the parent project when the user can assign time to work on the upstream PR.

Environment

• Python version: 3.8.13
• circuit_maintenance_parser version: 2.0.8

Expected Behavior

All tests to pass

Observed Behavior

Equinix Parser test fails. The parsed time stamp appears to be different than the expected timestamp.

Steps to Reproduce

1. Run pytest
2. Observe failure and diff

Environment

• circuit_maintenance_parser version: 2.0.5

Proposed Functionality

Per the BCOP, X-MAINTNOTE-STATUS can be omitted from an iCal notification (the only example I see of this in the BCOP is for a rescheduled maintenance window), but our Maintenance class defines status as a mandatory field. In general how should we handle iCal notifications that omit this field?

Use Case

I have a real-world example received from a provider for a "completed" notification that omits the X-MAINTNOTE-STATUS, so simply assuming that "no X-MAINTNOTE-STATUS implies "status=re-scheduled"" would not be correct in this case. We could possibly parse the DESCRIPTION or SUMMARY fields -- in this particular case the word "completed" is present in both -- but that's probably not desirable as a general solution? Looking for suggestions here.

Environment

• Python version: <3.9>
• circuit_maintenance_parser version: <2.0.3>

Expected Behavior

Colt email subject Colt Service Affecting Maintenance Notification - CRQ1-xxxxx [20/8/2022 20:00:00 GMT - 21/8/2022 01:00:00 GMT] for YYYYY, XYZ is not properly handle by the parser.

Colt email doesn't start with [ EXTERNAL ], this field could have been add by the email platform used during the first tests.

The regex in SubjectParserColt1 and SubjectParserColt2 should be modify to not enforce the presence of brackets at the beginning of the email.

Observed Behavior

Colt Parser is expecting COLT email to start with [ .* ], the 2 brackets should be see as facultative as well.

Environment

• circuit_maintenance_parser version: 2.1.0

Proposed Functionality

Add PCCW maintenance parser

Use Case

Useful for clients PCCW.

Environment

• circuit_maintenance_parser version: n/a

Proposed Functionality

Migrate to GitHub Actions

Use Case

Migrate off of Travis to GitHub Actions.

Environment

• Python version:
• circuit_maintenance_parser version: 2.1.0

Expected Behavior

Test data should not contain any public IP addresses for anonymity purposes.

Observed Behavior

Some files contains public IPv4 and IPv6 addresses.

Steps to Reproduce

Environment

• Python version:
• circuit_maintenance_parser version: 1.2.0

Expected Behavior

Backwards compatibility in minor-version-incrementing releases.

Observed Behavior

The MaintenanceNotification model was renamed to Parser by #26, meaning that any code (such as version 0.1.3 of https://github.com/nautobot/nautobot-plugin-circuit-maintenance) that makes use of this model will fail when attempting to import it:

 Traceback (most recent call last):
   File "/usr/local/bin/nautobot-server", line 8, in <module>
     sys.exit(main())
   File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 62, in main
     initializer=_configure_settings,  # Called after defaults
   File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
     management.execute_from_command_line([runner_name, command] + command_args)
   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
     utility.execute()
   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 377, in execute
     django.setup()
   File "/usr/local/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
     apps.populate(settings.INSTALLED_APPS)
   File "/usr/local/lib/python3.7/site-packages/django/apps/registry.py", line 122, in populate
     app_config.ready()
   File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/__init__.py", line 68, in ready
     super().ready()
   File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/__init__.py", line 92, in ready
     jobs = import_object(f"{self.__module__}.{self.jobs}")
   File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 44, in import_object
     spec.loader.exec_module(module)
   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
   File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/jobs.py", line 2, in <module>
     from .handle_notifications.handler import HandleCircuitMaintenanceNotifications
   File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/handle_notifications/handler.py", line 6, in <module>
     from circuit_maintenance_parser import ParsingError, init_parser, MaintenanceNotification
 ImportError: cannot import name 'MaintenanceNotification' from 'circuit_maintenance_parser' (/usr/local/lib/python3.7/site-packages/circuit_maintenance_parser/__init__.py)

Steps to Reproduce

Proposed Functionality

New LLM (Large Language Model) allow a sophisticated language processing that could get data from structures/unstructured notifications and retrieve part or total of the necessary information.

Use Case

This could avoid having to write custom parser for all Providers

Environment

• Python version: 3.7.9
• circuit_maintenance_parser version: 1.2.1

I'm starting to run into dependency hell with some plugins. There are some dependencies in this project (mainly click) which are outdated. Also development dependencies should probably be set to * as to not cause issues down the line but they are development dependencies

Expected Behavior

Plays nice in the sandbox with plugins dependent on newer versions of dependencies

Observed Behavior

this plugin is dependent on click 7

Environment

• Python version:
• circuit_maintenance_parser version:

Expected Behavior

Parsing failure should result in a clearly formatted exception and/or logging message to make it easy to troubleshoot and fix.

Observed Behavior

Due to the logic in https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/providers.py#L60 that catches all ParsingError exceptions raised by individual parser classes and discards them, the only log messages seen are generic, i.e.:

Parser HtmlParserCogent1 for provider Cogent was not successful

and the ultimately raised ParsingError is similarly generic:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/handle_notifications/handler.py", line 193, in process_raw_notification
    parsed_notifications = parser.process()
  File "/usr/local/lib/python3.7/site-packages/circuit_maintenance_parser/providers.py", line 81, in process
    raise ParsingError("None of the parsers was able to parse the notification")
circuit_maintenance_parser.errors.ParsingError: None of the parsers was able to parse the notification

This makes it very difficult to diagnose why the parsing failed.

Steps to Reproduce

Environment

• Python version:
• circuit_maintenance_parser version: 2.0.1

Expected Behavior

A Provider that uses a CombinedProcessor should be able to cleanly process multiple NotificationData in series without data bleeding over from one processing to the next

Observed Behavior

After successfully processing one NotificationData, then processing a second, incomplete NotificationData, the second processing returns a Maintenance that retains fields from the first processed NotificationData instead of raising a ProviderError due to the incomplete data.

Steps to Reproduce

Environment

• circuit_maintenance_parser version: 2.1.0

Proposed Functionality

Add Crown Castle maintenance parser

Use Case

Useful for clients of Crown Castle.

Proposed Functionality

Parsers could support different parsing implementations for vendors.

Use Case

When a provider offers different templates or it is transitioning, the library should provide the best parsed output.

For instance, Telstra has started using iCalendar format: https://twitter.com/NetworkAndrew/status/1402964730285789190

So, a provider parser should be able to parse multiple formats, and for each format, multiple implementations.

Environment

• Python version:
• circuit_maintenance_parser version:

Expected Behavior

When a problem is detected initializing NotificationData, the user of the init_data methods should not get the exception

Observed Behavior

Steps to Reproduce

Proposed Functionality

To create a valid Maintenance object we enforce a minimum set of attributes that it's validated via Pydantic.
The parser that is parsing the raw data from the notification has to get all the necessary information before creating the Maintenance and there are some identified cases where not all the necessary information is available in the raw payload that a notification is providing (more in Use Case).

Providing an optional extra_data attribute(s) together with the raw data could complement it with data that the provider is not embedding in the raw notification data, or even allow customization of the objects by the library consumer.

Use Case

This feature is based in, at least, 2 use cases:

1. A Provider sends all the maintenance information in a HTML or Calendar format, and attaches an extra CSV with the related Circuit IDs affected. This feature would enable to get the Circuit IDs and add them to the extra_data that would extend the data processed from the main information payload.
2. A Provider sends an email with the Maintenance ID in the "subject", so we would extend the data processed with it.

Environment

• circuit_maintenance_parser version: 2.1.0

Proposed Functionality

Add Hurricane Electric maintenance parser

Use Case

Useful for clients Hurricane Electric.

Environment

• circuit_maintenance_parser version: 2.2.3

Proposed Functionality

Remove the Cogent parser dependency on the tzwhere library that is unmaintained (last release 7 years ago).

Use Case

Allow for adopting newer Python versions.

Environment

• Python version:
• circuit_maintenance_parser version: 2.0.3

Expected Behavior

Cogent parser works well and tests passes

Observed Behavior

The Cogent parser doesn't provide the Timezone, so we use the Location to deduce the Timezone using an external API that often hits rate limit and it fails

Steps to Reproduce

Environment

• Python version: 3.8.7
• circuit_maintenance_parser version: 2.0.7

Expected Behavior

Notification parsed successfully

Observed Behavior

Notification parsing fails. No maintenance_id.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nautobot_circuit_maintenance/handle_notifications/handler.py", line 234, in get_maintenances_from_notification
    result = parser_provider.get_maintenances(data_to_process)
  File "/usr/local/lib/python3.8/site-packages/circuit_maintenance_parser/provider.py", line 136, in get_maintenances
    raise ProviderError(
circuit_maintenance_parser.errors.ProviderError: Failed creating Maintenance notification for Verizon.
Details:
- Processor CombinedProcessor from Verizon failed due to: 1 validation error for Maintenance
maintenance_id
  field required (type=value_error.missing)

Environment

• circuit_maintenance_parser version: 2.0.1

Proposed Functionality

All parsers should err on the side of including more information in the summary field versus trying to narrowly extract a single sentence or paragraph from the overall notification.

Use Case

Notifications often include many additional details beyond what we're currently capturing in most summary fields. In integrating the output of this parser library with other systems, it's better to preserve information rather than discarding it.

Frankly, I'm not entirely sure if this classifies as bug or feature request.
Sparkle started sending us maintenance notifications with this datetime format: MMM/DD/YYYY HH:mm UTC. For example: Jan/27/2023 03:00 UTC.

Currently circuit-maintenance-parser doesn't support this format and it's not a common/standard format. However, it can very much be parsed:

Python 3.10.9 (main, Dec 15 2022, 17:11:09) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import arrow
>>> a = "Jan/27/2023 03:00 UTC"
>>> arrow.get(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tdo/Documents/dev/MNA/lib/python3.10/site-packages/arrow/api.py", line 91, in get
    return _factory.get(*args, **kwargs)
  File "/Users/tdo/Documents/dev/MNA/lib/python3.10/site-packages/arrow/factory.py", line 254, in get
    dt = parser.DateTimeParser(locale).parse_iso(arg, normalize_whitespace)
  File "/Users/tdo/Documents/dev/MNA/lib/python3.10/site-packages/arrow/parser.py", line 208, in parse_iso
    raise ParserError(
arrow.parser.ParserError: Expected an ISO 8601-like string, but was given 'Jan/27/2023 03:00 UTC'. Try passing in a format string to resolve this.
>>> arrow.get(a, 'MMM/DD/YYYY HH:mm UTC')
<Arrow [2023-01-27T03:00:00+00:00]>

Environment

• Python version: 3.8
• circuit_maintenance_parser version: 2.2.2

Expected Behavior

We expected a successfully parsed Maintenance object.

Observed Behavior

We get an error as shown here:

(CMP) tdo@PlayStation [16:02:13] [~]
-> % circuit-maintenance-parser --data-file ~/Downloads/NETENG-22869.eml --data-type email --provider-type sparkle
Provider processing failed: Failed creating Maintenance notification for Sparkle.
Details:
- Processor CombinedProcessor from Sparkle failed due to: Unknown string format: Jan/27/202303:00 UTC

Steps to Reproduce

1. unzip sparkle-email.zip
2. circuit-maintenance-parser --data-file /path/to/sparkle.eml --data-type email --provider-type sparkle

Environment

• circuit_maintenance_parser version: v2.2.2

  Similar to #210, Lumen notifications are not accurately pulling all maintenance windows. In Lumens maintenances, Circuit Maintenance Parser (CMP) is actually only currently able to export one start/end time and in all cases, this is the last table that CMP scans. See my example in the Use Case.

Proposed Functionality

Allows operators to schedule in accurate maintenance windows for Lumen Maintenances.

Use Case

See the attached Lumen email in the comments of this issue.
Currently with this example, we extract the following start and end time:

Start: 2023/03/23-04:00 UTC
End: 2023/03/23-12:00 UTC

As you can see in the email, this is only the second window for the "Alternate Night".

We need to be able to capture both windows and have them scheduled as two separate events.

When running the circuit-maintenance-parser test on it, this should return something similar to the below:

 ✗  circuit-maintenance-parser --data-file ~/Downloads/Lumen_redacted.eml --data-type email --provider-type lumen
Circuit Maintenance Notification #0
{
  "account": "Company",
  "circuits": [
    {
      "circuit_id": "111111111",
      "impact": "OUTAGE"
    },
    {
      "circuit_id": "222222222",
      "impact": "OUTAGE"
    }
  ],
  "end": 1679479200,
  "maintenance_id": "123456789",
  "organizer": "[email protected]",
  "provider": "lumen",
  "sequence": 1,
  "stamp": 1678379003,
  "start": 1679457600,
  "status": "IN-PROCESS",
  "summary": "Lumen intends to carry out internal maintenance within its network.  This has been designated as ESSENTIAL. The nature of this work is to repair fiber and is required in order to avoid unplanned outages from damages related to natural causes.",
  "uid": "0"
}
Circuit Maintenance Notification #1
{
  "account": "Company",
  "circuits": [
    {
      "circuit_id": "111111111",
      "impact": "OUTAGE"
    },
    {
      "circuit_id": "222222222",
      "impact": "OUTAGE"
    }
  ],
  "end": 1679565600,
  "maintenance_id": "123456789",
  "organizer": "[email protected]",
  "provider": "lumen",
  "sequence": 1,
  "stamp": 1678379003,
  "start": 1679544000,
  "status": "IN-PROCESS",
  "summary": "Lumen intends to carry out internal maintenance within its network.  This has been designated as ESSENTIAL. The nature of this work is to repair fiber and is required in order to avoid unplanned outages from damages related to natural causes.",
  "uid": "0"
}

Environment

• circuit_maintenance_parser version: 2.0.1

Proposed Functionality

Today, a Provider processing a notification can only return either the maintenance objects or raise a ProviderError when it was not possible to get, at least, a maintenance object from the notification.
Some times the notification parsed is not actually containing any relevant information and it's not actually needed to be parsed and raise an error, just signal that it was not containing any relevant info.

Use Case

Being able to differentiate between parsing errors and non relevant notification would make the library clients able to handle each one in a different way.

Environment

• Python version:
• circuit_maintenance_parser version: 2.2.2

Expected Behavior

E2E testing for multiple maintenances should pass

Observed Behavior

Timestamps for Lumen PR #216 are not enable for e2e testing.

Steps to Reproduce

1. Add Lumen test 8 and run tests

Desired Behavior

Do not depend fail hard on external dependencies. Maybe catch results or maybe just provide some default info.

Observed Behavior

Flaky Cogent parser depending on external geolocation API:

FAILED tests/unit/test_utils.py::test_city_timezones[North Bergen, NJ-America/New_York] - geopy.exc.GeocoderUnavailable: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=North+Bergen%...

Steps to Reproduce

1. Run tests

Proposed Functionality

Enable the client of the library to initalize the Provider with a default provider type and organizer to overwrite the default one that every Provider has.

Use Case

This enable better customization, for instance, if for a specific client a network service provider uses a custom email or the client wants to use a custom type name.