Change Type

Addition

Proposed Changes

Compile a list of all models for which changes can be recorded in a branch. This is most NetBox models, with the natural exceptions of things like users, groups, permissions, etc.

This may need to be provided within the application itself, rather than in the form of static documentation, for two reasons:

1. It may be difficult to keep the list updated as new models are added to NetBox over time.
2. Users will need to know which plugin-provided models are supported, which is not feasible to determine universally.

Change Type

Addition

Proposed Changes

Add a new page in the documentation with details and examples showing how to employ branching for REST API requests using specific HTTP headers.

Plugin Version

v0.2.0

Proposed functionality

When viewing a ChangeDiff record via the REST API (/api/plugins/branching/changes/), the following attributes are included:

• original_data
• modified_data
• current_data

These each provide a complete representation of the changed object within the applicable context. However, there is no summary of these changes as exists in the web UI.

Use case

Providing the computed summary would make it directly consumable by the API client and obviate the need for the client to compute this locally.

External dependencies

None

Plugin Version

v0.2.0

Proposed functionality

Currently, a branch is activated by referencing its schema ID. For example:

/dcim/sites/3/?_branch=kqm2ghg3

This issue explores whether it is feasible to instead use the branch's name (or an analog, like slug) to indicate the desired branch. For example:

/dcim/sites/3/?_branch=maintenance-42

Use case

This would permit the use of human-significant branch identifiers rather than the randomly-generated schema IDs.

External dependencies

None

Plugin Version

v0.2.0

Proposed functionality

Create a table column which links to the corresponding branch (if any) for an ObjectChange record, and register it with the global changelog table using NetBox's plugins API.

Use case

This will provide a convenient mechanism for users to view the branch responsible for each change.

External dependencies

None

Proposed Changes

As per #33 (comment)

During the semantic release process netbox_branching/version.py should be written to with 3 fields injected in the build, then the plugin config netbox_branching/__init__.py should load the version from there.

Without this the plugin config version needs to be bumped manually, which will get out of sync of the semver / pypi process.

Justification

Follow the pattern of existing NBL plugin releases.

Plugin Version

0.3.0

NetBox Version

v4.1.0-beta1

Python Version

v3.12

Steps to Reproduce

1. Install NetBox v4.1-beta1 and branching v0.3.0
2. Install netbox-topology-views (this is only to demonstrate truncated field names and unrelated to the plugin directly) - note the plugin works as expected in v4.1
3. Create a Branch

Expected Behavior

Branch should be created

Observed Behavior

Job never completes due to an exception

Error: relation "public.netbox_topology_views_individualoptions_preselected_device_id_s" does not existLog:

2024-08-14T19:24:28.814958+00:00 nbl-branching-privatepreview-plugins python3[2572]: 19:24:28 default: handle(job=<Job: 6b74949b-b7a7-489d-aaeb-aac9aec84631>) (6b74949b-b7a7-489d-aaeb-aac9aec84631)

2024-08-14T19:24:31.262555+00:00 nbl-branching-privatepreview-plugins python3[3083]: 19:24:31 [Job 6b74949b-b7a7-489d-aaeb-aac9aec84631]: exception raised while executing (handle)

2024-08-14T19:24:31.262665+00:00 nbl-branching-privatepreview-plugins python3[3083]: Traceback (most recent call last):

2024-08-14T19:24:31.262711+00:00 nbl-branching-privatepreview-plugins python3[3083]:   File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute

2024-08-14T19:24:31.262741+00:00 nbl-branching-privatepreview-plugins python3[3083]:     return self.cursor.execute(sql)

2024-08-14T19:24:31.262769+00:00 nbl-branching-privatepreview-plugins python3[3083]:            ^^^^^^^^^^^^^^^^^^^^^^^^

2024-08-14T19:24:31.262798+00:00 nbl-branching-privatepreview-plugins python3[3083]:   File "/opt/netbox/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute

2024-08-14T19:24:31.262831+00:00 nbl-branching-privatepreview-plugins python3[3083]:     raise ex.with_traceback(None)

Note - this works fine in NetBox with no errors until you try to create a branch

In terms of the truncation

• actual database sequence is netbox_topology_views_individualoptions_preselected_devi_id_seq
• branching looks for public.netbox_topology_views_individualoptions_preselected_device_id_s
• the full sequence (if all chars permitted) - public.netbox_topology_views_individualoptions_preselected_device_role_id_seq

This can be resolved by shortening the field / table / sequence length as per this migration in the plugin - however there may be an opportunity for NetBox Branching to check for this behaviour to avoid exceptions / stuck branches.

Plugin Version

v0.2.0

NetBox Version

v4.1-dev

Python Version

3.10

Steps to Reproduce

1. Create two branches
2. In branch 1, delete an object.
3. In branch 2, modify some attribute of that object.
4. Attempt to merge branch 1.

Expected Behavior

The branch should be merged successfully.

Observed Behavior

The merge job fails with the following error:

KeyError('description')

Complete job log:

{
    "log": [
        "Merging branch Branch 1 (branch_kqm2ghg3)",
        "Applying change: Tenancy | tenant Wayne Enterprises deleted by admin",
        "'description'"
    ]
}

Plugin Version

0.3.1

NetBox Version

v4.1-beta1

Python Version

Python 3.12.3

Steps to Reproduce

Try to create a site on a branch using the header method specified in the docs like this:

    headers = {
        "Authorization": f"Token {NETBOX_TOKEN}",
        "Content-Type": "application/json",
        "Accept": "application/json",
        "X-Branch-Name": branch_name
    }
    
    site_endpoint = f"{NETBOX_URL}/api/dcim/sites/"
    
    data = {
        "name": site_name,
        "slug": site_slug
    }
    
    try:
        response = requests.post(site_endpoint, headers=headers, json=data)
        response.raise_for_status()
        print(f"Site '{site_name}' created successfully in branch '{branch_name}'.")
    except requests.exceptions.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err} - Status Code: {response.status_code}")
    except Exception as err:
        print(f"An error occurred: {err}")

Expected Behavior

The site should be created in the branch, as it is when using the URL parameter method, like this:

    headers = {
        "Authorization": f"Token {NETBOX_TOKEN}",
        "Content-Type": "application/json",
        "Accept": "application/json",
    }
    
    # Use the query string to specify the branch context
    site_endpoint = f"{NETBOX_URL}/api/dcim/sites/?_branch={branch_name}"
    
    data = {
        "name": site_name,
        "slug": site_slug
    }
    
    try:
        response = requests.post(site_endpoint, headers=headers, json=data)
        response.raise_for_status()
        print(f"Site '{site_name}' created successfully in branch '{branch_name}'.")
    except requests.exceptions.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err} - Status Code: {response.status_code}")
    except Exception as err:
        print(f"An error occurred: {err}")

Observed Behavior

When using the header (-H "X-NetBox-Branch: branch_id") method, the request writes to main instead of the branch, but reports success.

Plugin Version

v0.2.0

Proposed functionality

The "merge branch" confirmation dialog (pictured below) currently commits by default.

This FR proposes rendering this checkbox as unchecked initially.

Use case

Leaving this checkbox unchecked initially provides a safer default action (i.e. performing a dry run). It would also better abide by the convention of requiring an additional click to perform a potentially dangerous operation.

External dependencies

No response

Proposed Changes

The schema_id field on the Branch model should enforce uniqueness (uniqute=True).

Justification

In practice, this is very unlikely to be an issue, as schema IDs are generated randomly, but there's no reason not to enforce this.

Proposed Changes

Ensure that logging calls are included anywhere a relevant operation may take place, and remove any lingering debugging statements.

Justification

Ensures robust logging to assist in future troubleshooting and debugging.

Plugin Version

v0.2.0

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

1. Create a branch
2. Within main, change the status of a site from active to staging
3. Activate the branch
4. Change the status of that same site from active to planned
5. Synchronize the branch
6. Acknowledge the conflict and commit changes

Expected Behavior

The value in the branch should be updated to reflect that of main, and its changelog should be updated accordingly.

Observed Behavior

Within the branch, the site's status now shows "staging." However, the branch's change history (under the "changes ahead" tab) shows that the site's status will be changed to "planned" when merging the branch.

Plugin Version

v0.2.0

Proposed functionality

Within Branch.provision() all SQL commands used to copy data from main should be wrapped within an atomic transaction.

Use case

This will ensure that changes to main do not interfere with or disrupt the provisioning process, and that the resulting branch data reflects main at the point in time when the provisioning process was initialized.

External dependencies

None

Plugin Version

v0.2.0

Proposed functionality

Implement the ability to apply database migrations (e.g. from a NetBox upgrade) with one or more open branches. Currently, this is not supported as migration operations are applied only to the main schema.

Use case

This would enable users to reliably upgrade to new NetBox versions without needing to first merge any open branches.

External dependencies

No response

Plugin Version

v0.3.0

NetBox Version

v4.1.0-beta1

Python Version

3.11.6

Steps to Reproduce

Go to Background Tasks and click on an ID or try to run a script

Expected Behavior

Background Task should be shown

Observed Behavior

'str' object has no attribute '_meta'

Error during template rendering
In template /opt/netbox/netbox/templates/generic/object.html, error at line 118
'str' object has no attribute '_meta'

108 	    <li class="nav-item">
109 	      <a class="nav-link{% if not tab %} active{% endif %}" href="{{ object.get_absolute_url }}">{{ object|meta:"verbose_name"|bettertitle }}</a>
110 	    </li>
111 	
112 	    {# Include tabs for registered model views #}
113 	    {% model_view_tabs object %}
114 	  </ul>
115 	{% endblock tabs %}
116 	
117 	{% block alerts %}
118 	  {% plugin_alerts object %}
119 	{% endblock alerts %}
120 	
121 	{% block content %}{% endblock %}
122 	
123 	{% block modals %}
124 	  {% include 'inc/htmx_modal.html' %}
125 	{% endblock modals %}
126 	

Plugin Version

v0.2.0

NetBox Version

v4.1-dev

Python Version

3.10

Steps to Reproduce

1. Create & activate a branch
2. Edit an existing interface and assign one or more VLANs
3. Attempt to merge the branch

Expected Behavior

The branch should be merged successfully.

Observed Behavior

The merge job fails with the following error:

TypeError('Direct assignment to the forward side of a many-to-many set is prohibited. Use tagged_vlans.set() instead.')

Private Preview Participant Links

Bloomberg: https://netdev-community.slack.com/archives/C07CP24P6AX/p1721395501733989

Plugin Version

v0.2.0

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

1. Create two branches, A and B
2. In both branches, create a new interface named "eth0" on a device
3. Merge branch A
4. Attempt to merge branch B. An IntegrityError is raised, complaining that the name of the interface being created is not unique to the device (which is expected).
5. Delete the interface from branch B.
6. Attempt to merge branch B again.

Expected Behavior

The merge should be successful, as the offending object has been deleted.

Observed Behavior

The same exception continues to be raised, because the branch's history is being replayed in chronological order:

1. The interface with the offending name is created.
2. The interface is deleted.

We should be able to work around this by deferring constraint checks until the end of the transaction.

Plugin Version

0.3.0

NetBox Version

4.1-beta1

Python Version

3.11.5

Steps to Reproduce

Disclaimer: I have only tested with NetBox DNS so far.

1. Create a branch
2. Perform some arbitrary changes in NetBox DNS
3. Try to merge the branch

Expected Behavior

The branch is merged

Observed Behavior

The branch status shows the following error message:

When I look at the diffs, I see the following entry for the changes to the View object:

The data shown under 'before' are all None, which is definitely not correct - the same is true for the other objects. The only exception is with deletes, where the before data correctly represents the status before the object was deleted.

Plugin Version

v0.2.0

Proposed functionality

Update the background jobs to use the new JobRunner class introduced in NetBox v4.1.

Use case

Reduces boilerplate code and encourages standardization.

External dependencies

No response

Plugin Version

v0.2.0

Proposed functionality

When displaying a list of change records, group each tranche of records by request ID. The list entry should convey a summary of the changes therein (e.g. "created 48 interfaces") and retain some mechanism for viewing the individual records.

Use case

This will make the change log easier to interpret and simplify the review process.

External dependencies

No response

Plugin Version

v0.3.0

Proposed functionality

Introduce a new plugin configuration parameter (e.g. max_branches) to set the maximum number of branches that can be created.

Use case

This will provide a mechanism to guard against the erroneous or malicious creation of an excessive number of branches.

External dependencies

None

Change Type

Addition

Proposed Changes

Document the components of the plugin's public Python API, which are provided for consumption by other plugins or by custom scripts.

Plugin Version

v0.2.0

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

1. Create a branch
2. While the branch is being provisioned, delete it

Expected Behavior

It should not be possible to delete a branch which is currently in a transitional state (e.g. provisioning or merging), which may lead to a broken state.

Observed Behavior

The deletion is currently permitted.

Plugin Version

v0.3.0

NetBox Version

v4.1-beta1 - 2024-08-05

Python Version

Python 3.12.3

Steps to Reproduce

1. Create a branch
2. Add a site to the branch
3. Merge the branch
4. Revert the branch in dry run mode
5. Merge the branch

Expected Behavior

As the site is not removed from the main branch in dry run mode, I expect the branch to stay in the merged state so that I can see what happened in the job, instead of going back to the ready state which creates this issue

Observed Behavior

The site is not removed from the main branch and the branch goes back to the ready state with the new site in it again, which then creates a merge conflict

Plugin Version

v0.3.1

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

The signal receiver which watches for object events and creates ChangeDiffs in response does not currently filter out object types that should be excluded.

Expected Behavior

ChangeDiffs should be created/updated only for supported models, e.g. those returned by get_branchable_object_types().

Observed Behavior

A ChangeDiff is created/updated even for models which don't support branching.

Plugin Version

0.3.1

NetBox Version

v4.1-beta1

Python Version

Python 3.12.3

Steps to Reproduce

1. Create a branch
2. Create a site in main so that it shows up in the "changes behind" in the branch
3. Sync the branch in dry-run mode
4. Sync the branch with "commit" selected (non dry-run)

Expected Behavior

Step 4 should work correctly as there are no conflicts between main and the branch

Observed Behavior

ValidationError({'id': ['Site with this ID already exists.'], 'name': ['Site with this Name already exists.'], 'slug': ['Site with this Slug already exists.']})

Proposed Changes

Rewrite raw SQL statements which substitute a variable to pass them as separate arguments, rather then relying on string formatting. For example:

cursor.execute(f"CREATE TABLE {schema}.{table} ...")

would become

cursor.execute(f"CREATE TABLE %s.%s ...", [schema, table])

Justification

This approach ensures that any parameters are properly escaped. (Although there are currently no instances where we pass externally-sourced data into these commands, it's a good convention to adopt.)

Plugin Version

v0.3.1

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

1. Create and activate a branch
2. Create an instance of a model which supports change logging (i.e. it inherits from NetBox's ChangeLoggingMixin) but does not support branching (i.e. because its app has been excluded)
3. Inspect the change log

Expected Behavior

The creation of the object should be recorded in the global change log.

Observed Behavior

The object's creation is recorded in the branch-specific change log. However, the object exists in main as expected, because the model does not support branching.

Plugin Version

v0.2.0

Proposed functionality

In the branch selection dropdown (pictured below), include a direct link to the detail view of each branch.

Use case

This ensures that a direct link to each branch is available for convenience.

External dependencies

None

Plugin Version

v0.2.0

Proposed functionality

Provide a mechanism for reverting a previously-merged branch. This should effectively replay the associated change records in reverse chronological order, restoring the original state of each object in sequence.

Note that this will not erase or otherwise compromise the global change log: All of the original changes as well as the corresponding reversion will be recorded.

As with merging a branch, a revert should be an atomic operation: Either every associated change is reverted successfully, or the entire transaction is aborted and the original (pre-revert) database state is restored.

Use case

This will allow a user to conveniently roll back a set of changes made within a branch.

External dependencies

No response

Plugin Version

v0.2.0

Proposed functionality

Introduce the following custom event types, triggered by branch events:

• branch_provisioned
• branch_synced
• branch_merged
• branch_deprovisioned

Use case

These will enable branch actions to trigger event rules on NetBox v4.1 and later.

External dependencies

None

Plugin Version

v0.2.0

Proposed functionality

Implement a configurable setting which automatically deletes merged branches after some pre-defined amount of time has passed (similar to how NetBox can clean up old changelog entries). This should be disabled by default, as deleted branches can no longer be automatically reverted.

Use case

This will automatically clean up branches which are no longer needed and help reduce database sprawl.

External dependencies

No response

Plugin Version

v0.2.0

Proposed functionality

An example:

1. Create & activate a new branch
2. Create a new site
3. View the site
4. Deactivate the branch via the widget at top right

This returns a 404, because the request targets the site, which does not exist outside the branch.

Whenever the middleware detects a branch being activated/deactivated in the UI, it should check the response code. If it's a 404, redirect the user to the dashboard.

Use case

This would help avoid confusing disruptions in the user's workflow.

External dependencies

No response

Plugin Version

v0.2.0

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

1. Merge a branch which modifies at least one object
2. View a modified object within the UI

Expected Behavior

The object includes a notice indicating that it has been modified within the merged branch.

Observed Behavior

This notice should appear only for unmerged branches.

Plugin Version

v0.2.0 (aa7f933)

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

1. Create a site in main
2. Create a branch
3. Create a conflict on the site
4. Dry run merge after acknowledging the conflict

Expected Behavior

Dry run should complete successfully.

Observed Behavior

The AbortTransaction exception (which is intentionally raised to effect the dry run) is not handled as it should be.

Plugin Version

0.3.1

NetBox Version

4.1-beta1

Python Version

3.11.5

Steps to Reproduce

1. Create fresh NetBox DB
2. Install the current version of the branching plugin and add it to the configuration
3. Install NetBox DNS 1.1b4 from PyPI and add it to the configuration
4. Run the migrations and restart NetBox and RQ
5. Create a new branch "Test"
6. Activate the new branch
7. Create a NameServer object in NetBox DNS (e.g. "ns1.example.com")
8. Create a bunch of zones in NetBox DNS (via import):

zone1.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com
zone2.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com
zone3.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com
zone4.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com
zone5.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com
zone6.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com
zone7.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com
zone8.example.com,ns1.example.com,hostmaster.example.com,ns1.example.com

9. Deactivate the test branch
10. Try to merge the test branch

Expected Behavior

The branch is merged and the zones and the nameserver are now available in the main branch

Observed Behavior

This is not the only way I could reproduce this - at one point I got the same error just with an IP Address, without any NetBox DNS objects involved (unfortunately I didn't get a screen shot, though).

Not sure how this relates to #43 - so far I haven't been able to reproduce that one with a fresh DB.

Plugin Version

v0.2.0 (aa7f933)

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

• Create a branch
• View the swagger schema and form the merge POST request based on what is shown there
• Attempt to merge via POST request

Expected Behavior

Following the Swagger schema I expect to be able to form a valid merge POST request

Observed Behavior

The merge POST request throws a 500 KeyError because it requires the commit field toggle

Plugin Version

v0.3.1

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

1. Activate a branch
2. Create a new instance of a model which does not support branching

Expected Behavior

Our custom database router (BranchAwareRouter) should return the connection name corresponding to the active branch only for models which support branching.

Observed Behavior

The database router currently does not take into consideration the model being queried, and always returns the schema-specific connection name.

I haven't tied this to any undesirable behavior, but it should be corrected nonetheless.

Plugin Version

v0.3.0

NetBox Version

v4.1-beta1

Python Version

3.10

Steps to Reproduce

This applies to any fatal error which occurs within the database transaction employed to provision a new branch. #44 is a good example of this.

Expected Behavior

Any exception raised during this process should be handled cleanly, and the provisioning job should be aborted and marked as errored.

Observed Behavior

The exception is not handled, and the background job is left in a perpetual "running" state.

Plugin Version

v0.2.0

NetBox Version

v4.1-dev

Python Version

3.10

Steps to Reproduce

1. Open the "diff" tab for a branch
2. Use the quick search to filter for a specific object name

Expected Behavior

Only entries matching the search query should be displayed.

Observed Behavior

The search query has no effect.

Plugin Version

v0.2.0

Proposed functionality

When preparing to synchronize or merge a branch, display a summary of the changes to be applied. This should be an abridged version of the corresponding changes ahead & behind tabs.

Use case

This will provide additional context, serving as a sanity check prior to confirming the action.

External dependencies

None

Plugin Version

v0.2.0

Proposed functionality

Do not permit a branch to be merged until all of its conflicts have been resolved or acknowledged. A conflict is defined here as an object which has been modified in both the main and branch schemas with conflicting attributes.

For example, suppose Site A's description is changed from an empty string to two different values in the primary and branch schemas. Merging the branch in this scenario would effect data loss, as the value defined in the primary schema will be overwritten.

The user should be alerted to each such conflict and prompted to take one of three actions:

1. Keep the current value (ignoring the change within the branch)
2. Disregard the conflict (which may be expected) and apply the change
3. Abort the application of the branch

Use case

This will mitigate unintentional data loss while still preserving an optional path forward in the event such loss is deemed acceptable by the user.

External dependencies

No response

Plugin Version

v0.2.0

Proposed functionality

Extend the JSON representation of a ChangeDiff to include the summary diff, indicating only which attributes have changed and their current and original values.

The format of this representation is yet to be determined, but could look something like this:

{
    "status": {
        "old": "staging",
        "new": "active"
    },
    "description": {
        "old": "",
        "new": "Foo bar"
    }
}

Use case

Although a ChangeDiff retrieved via the REST API currently includes the complete set of original, main, and branch data, a client would need to reconstruct the summary diff locally. Providing this data (which is already conveyed via the web UI) makes it directly consumable by the client.

External dependencies

No response

Proposed Changes

The following test methods under test_request.RequestTestCase are currently failing:

• test_activate_branch
• test_deactivate_branch

django.db.utils.IntegrityError: insert or update on table "extras_cachedvalue" violates foreign key constraint "extras_cachedvalue_object_type_id_6f47d444_fk_django_co"
DETAIL:  Key (object_type_id)=(144) is not present in table "django_content_type".

Justification

Restore test suite integrity