Add real NHSD examples to some of the principles, including:

1. Cross-region SQL server on Azure (CB)
2. Active/active Postgres read-replicas on AWS (DS)
3. Ephemeral EC2 on AWS (NL)
4. Multi-master cosmos DB on Azure (RS?)

Clarify in quality-checks.md that Sonarqube Developer is only needed for static security code analysis (community edition is fine for anything else)

For example policies to prevent unencrypted <>

This term is used in a couple of different ways in NHSD. Would be useful to explain what we mean by it in this context.

Can we have some guidance around where open source should be used, and at a minimum we should be practicing "inner source".

https://en.m.wikipedia.org/wiki/Inner_source

I often find myself having to ask for read access to code within my own programme at NHS digital and my requests for read access to all developers by default falls on deaf ears.

I thought we'd included this, but I can't find anything apart from in the assessment. We should add a note in the reliability or observability sections.

Not a checklist, but some guidance

Regarding https://github.com/NHSDigital/software-engineering-quality-framework/blob/master/practices/cloud-services.md

Ref "Keep up to date"

Very often when a new service is commissioned there is a failure to identify, plan, and charge for the entire systems development life cycle - including maintenance, periodic re-platforming, and the eventual decommissioning.

I'd suggest we put in something to ensure that these things are considered at the outset (or at least ASAP retrospectively) so that programme roadmaps can be adjusted and so that necessary resource (human and financial) can be identified well in advance.

I think it would be useful to be able to offer some enterprise governance around CI/CD approach If we can document our thinking around this area with some example use cases we could look to take that to appropriate boards. I think it further strengthens the case for GitHub here.

Originally posted by @walteck in #166 (comment)

Regarding: https://github.com/NHSDigital/software-engineering-quality-framework/blob/master/practices/cloud-services.md

Ref: "Prefer serverless platform as a service (PaaS) over infrastructure as a service (IaaS)"

I think it is important to recognise that serverless architecture is a good example of TANSTAAFL [1]. What you gain in simplifying the infrastructure you often lose (and more) due to the additional difficulty of testing and debugging a distributed software architecture.

The problem is often not with the code itself, which can be unit tested, but with how to test the config and integration with other services. It's easier to build an IaaS stack locally than it is to mock out cloud services on your development machine.

I'd suggest we say that serverless is useful for isolated (or isolatable) components, especially those where demand tends to be "bursty" e.g. processing user signups, but that IaaS has benefits where the system has many interdependencies (assuming that deployment monolithically via IaaS is a realistic alternative).

[1] https://en.wikipedia.org/wiki/There_ain%27t_no_such_thing_as_a_free_lunch

Add the endorsed tools details and regex list to "Don't trust yourself or others!" in master/practices/security.md

(as an expand/contract section rather than as a new page)

And add a link from master/quality-checks.md

For example: vote-first on some review sections to try to avoid unintentional confirmation bias etc

... in general, and specifically in the context of how this promotes portability in terms of cloud vendors.

With a cautionary note about the dangers of over or premature abstraction.

Possibly as part of everything-as-code?

For example configuration to alert if unencrypted <> is detected

Regarding https://github.com/NHSDigital/software-engineering-quality-framework/blob/master/patterns/outsource-bottom-up.md

Ref: "Cloud native vs cloud agnostic"

I think you are very slightly implying cloud neutrality in this section but I don't think it would hurt to be more forward - we should be able to move entire services from one cloud provider to another if we want to. This would mean avoiding very niche components of each cloud in favour of commodity services (e.g. a PostgresDB is likely to be more or less the same regardless of which cloud it is in, and so this would be a more portable choice).

... in observability.md

The principle says we need to monitor expiry dates: we have examples of how we do that: we should include them in this repo

Rather than looking just at the KPIs as figures perhaps we could suggest to the terms as well to track a trend of changes of particular metrics in a set context.

A good example of how this could be done and what could be important is shown by Mark Richards, "Lesson 11 - Analyzing Architecture: Code Metrics" https://www.youtube.com/watch?v=pELKNy8B5Nw

Just a thought.

In 'Build Quality In' is says 'Three amigos analysis and elaboration to ensure requirements are fully understood at the point the work will be done."

This is good, however I think that another quality attribute to the requirements being understood is that the requirements are also testable.

Having well-understood, testable requirements will ensure that pair sessions (whether it's between devs or dev/test) are productive and effective.

For example, considering unintended data exposure via log files

(Idea from Alex)

For example, Azure-based services can use this to give insight into sustainability -

https://azure.microsoft.com/en-gb/blog/microsoft-sustainability-calculator-helps-enterprises-analyze-the-carbon-emissions-of-their-it-infrastructure/

Regarding https://github.com/NHSDigital/software-engineering-quality-framework/blob/master/practices/service-reliability.md

Do we need a sub-bullet under "Understand reliability requirements" around what level of out-of-hours support should be made available? (Software devs are not routinely on call OOH in all parts of Product Development)

Example:

• "Gold-level" services must have at least two software engineers and one infrastructure engineer available 24/7/365, all of whom must have experience in operating the system. Additional support/escalation must be available if service cannot be restored after 1 hour.
  ...
• "Bronze-level" services are supported through normal office hours on a "reasonable efforts" basis. Outside these hours, no guarantees of availability are made.

... (which are currently closed links) can now be repointed at the public versions

https://digital.nhs.uk/about-nhs-digital/our-work/nhs-digital-architecture/principles

We should add a library of endorsed/recommended GitHub actions, to save people having to find something where we already know of a decent solution

For example https://github.com/NHSDigital/software-engineering-quality-framework/blob/master/.github/workflows/action.yml isn't specific to this repo

Add a note (including in the review tool) that other concerns need to be considered alongside engineering concerns, e.g. are we building the right thing (even if we're building it in a good way)

As per existing links to architecture principles, service manual, NCSC, etc

https://en.wikipedia.org/wiki/Apdex could provide a useful means of distilling a performance-type test down to a numeric value, to drive an automated pass/fail gate?

Either as a new section in sonarqube.md, or a new page

With a link from quality-check.md

Ideas:

• Summary/overview is long with a lot of detail, move some of that to specific sections
• Break down basic requirements (e.g. logging, with meaningful logs) and advanced requirements (e.g. synthetic journeys)

Update engineering cop doc to reflect goals and tactics agreed in Miro board.

Principle to use the approved auth solutions

Was

• Consider restricting to TLS 1.2 or later only

Need to check if this should be a "MUST", not just "consider"

One consideration you might like to document is that when using cloud native services to deploy your infrastructure you have an additional security benefit in that the role which has permissions to amend your production infrastructure is only assumable by a cloud service (code build etc) and not assumable by any 'human' role.

Equally applying roles with different permissions to different stages in the deployment pipeline helps to ensure that, for example a deployment meant for a development account cannot actually be performed against a production account.

This quality framework is concerned with "rapid and safe delivery of high-quality software, at scale".

All types of engineering are influenced by human factors. I would argue that these human factors should be considered as part of this framework in order to ensure that delivery is rapid, safe, and sustainable.

Potential topics:

• teamwork / team structure
• relationships (e.g. with customers)
• learning processes (individual and organisational)
• "professional" behaviours and attitudes
• psychology (e.g. motivation, cognitive load, mental health)
• work/life balance

Some references to example cloud platform services which currently use AWS examples could be replaced or augmented with other examples, primarily from Azure.

In 'Testing' the Test Pyramid it mentioned, however another 'thinking tool' to help support a whole team approach and conversations around test automation and test planning is 'The Agile Testing Quadrant'. It encourages the whole team to think about testing first as it is a model to help build a common language and understanding.

Lisa Crispen has written a short article on it: https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/

Add details for vaults, e.g. KMS

Some other thoughts:

1. UK residency option should be used for NHS repos
2. User accounts should be work accounts (not sure if we should mention SSO or not)
3. All projects should have a -members and an -admins team, no project should be open for edits by non-team members; all projects should have a code owners file in the right place (and which references the team rather than individuals)

Originally posted by @andyblundell in #166 (review)

Regarding https://github.com/NHSDigital/software-engineering-quality-framework/blob/master/practices/service-reliability.md

Suggested addition:
It is good practice to conduct a post mortem after any service outage or significant degradation, with the goal of identifying improvements to systems and processes (rather than apportioning blame).