Git Product home page Git Product logo

core's Introduction

Introduction

Welcome to data.altinn.no (DAN)! This repository has a solution file containing several components

DAN CORE

DAN Core is the main component of DAN, implementing the public API. It utilizes that Altinn REST API for Service Owners for authorization (including consent) and maps requests for data sets to the relevant plugins implementing the codes.

The repository contains a .NET6 console application utilizing the isolated-process Azure Functions v4 runtime.

DAN Common

DAN Common contains all shared models and utilites used across all DAN projects. A nuget is built in Azure Devops which must be referred to in all plugin implementations.

Getting Started

Requirements

DAN can be used with any IDE / editor of your choosing, but we recommend Visual Studio 2022 or later, Visual Studio Code or Jetbrains Rider.

1. .NET 6 SDK and runtime

2. Azure Functions Core Tools

In order to build an run functions locally, you will need to install the Azure Functions Core Tools which depends on NodeJS.

3. Microsoft Azure Storage Emulator

In order to run functions locally, you will need a Azure Storage Emulator, preferably Azurerite installed and running on 127.0.0.1:10000 (default)

4. CosmosDB emulator

Download and install the CosmosDB emulator. For Windows, click here to download. See this page for information about running on Linux / MacOS.

After installation you will ned to create the default database. Go to https://localhost:8081/_explorer/index.html, select Explorer, Select "New Container". Name the database "nadobe-dev", and the container id "Accreditations". Supply "/owner" as partition key.

5. Redis

DAN is utilizing Redis for caching. See https://github.com/ServiceStack/redis-windows for options for Windows. Redis is also available for Linux / MacOS, and should be installed and run with default settings.

Contribute

The master branch is locked and cannot be pushed to directly, instead you must create a feature branch to which you commit all your code.

Feature branches should be named with the prefix Feature/ and be named in a descriptive manner (details yet to be determined). When you have pushed your feature branch, create a pull request against master and request a review from your team. When the pull request is approved, you may merge the branch.

Commits to master will initiate a full test run (unit, functional, integration) and will upon successful completion tag and deploy the master branch to the dev function slot in Azure.

A note about settings

Settings for the application is stored in Azure, and can be viewed in Azure Portal if you have access (this is not required for development). When developing and running functions locally, the settings are loaded from local.settings.json. Note that while this is a part of the repository for your convenience, these settings are only used locally and not in Azure,

core's People

Contributors

elsand avatar sondrejdigdir avatar renovate[bot] avatar erlendoksvoll avatar markciliavincenti avatar nlundee avatar danrj avatar

Stargazers

tednespippi avatar  avatar

Watchers

 avatar

core's Issues

Increase timeout for evidenceharvest and allow for customization

Description

Timeouts in plugins are typically 30 seconds and core cancels after 30 seconds, which means we produce errors instead of failing more gracefully.

Add timeoutsetting per plugin dataset (EvidenceCode model) and override default 30 second cancellation from core in evidenceharvester

Additional Information

No response

Tasks

No response

Fix end to end post-deploy tests

A few of the post deploy tests are currently failing because of amongst other things not fulfilling the tokenrequirement options.

These tests are good candidates for negative tests anyway :)

Upgrade Polly strategies to V8 API

Polly's made larger changes to their API in their SDK from version 7 to 8. Upgrading to V8 doesn't immediately require any changes, however how some policies are configured should be updated to be done the V8-way.

Polly has a migration guide from 7 to 8: https://www.pollydocs.org/migration-v8.html

I don't think this is urgent, however it should be done sooner rather than later as I would presume that when they go to V9 they won't support v7 methods anymore.

Remove support for ver2 from code and config

Description

Maskinporten's VER2 environment has reached EOL and has been removed. We no longer need to support more than 1 test environment (namely, test), so the settings and code should be removed (maskinportenauxurl and maskinportenauxwellknownurl) leaving test.maskinporten.no as the valid option.

These were introduced in #34 and should be removed from code and terraform.

Additional Information

No response

Tasks

No response

Create a test service context and test product in the staging environment

Description

We need to create a self-service test setup for new potential consumers of data from DAN in order for them to test the solution from a developer/technical standpoint

Additional Information

No response

Tasks

  • New service context "DAN-test"
  • New apim product (only in staging and dev) - DAN-test, with automatic subscription approval
  • New scope in test-maskinporten (dan:test), public and available for everyone
  • New dataset "DAN-test" in the digdir plugin that returns something
  • Explain how this is used under testing

Error message when person/organisation does not exist in Altinn

Description

When requesting consent the person/organisation must exist in Altinns databases. When they don't it just fails with a generic error.

"person/org xxxx cannot receive consent requests in Altinn" or something along those lines. Filter if ssn.

Steps To Reproduce

Request:
{"requestor":"980178986","subject":"998997801","evidenceRequests":[{"evidenceCodeName":"RestanserV2","requestConsent":true},{"evidenceCodeName":"MvaMeldingsOpplysning","requestConsent":true},{"evidenceCodeName":"Arbeidsgiveravgift","requestConsent":true},{"evidenceCodeName":"OppdragUtenlandskeVirksomheter","requestConsent":true},{"evidenceCodeName":"Firmaattest"},{"evidenceCodeName":"Enhetsregisterdata"},{"evidenceCodeName":"Aarsregnskap","parameters":[{"evidenceParamName":"NumberOfYears","value":1}]}],"consentReference":"schibsted-smb","externalReference":"Demo","languageCode":"no-nb"}

Response:
{
code: 1013,
description: "There was an error during an attempt to access a upstream source. The remote server might be down or overloaded",
innerExceptionMessage: "System.Net.Http.HttpRequestException: Server returned error status code: 400, ",
innerExceptionStackTrace: " at Nadobe.Core.Helpers.DelegatingHandlers.ExceptionDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\a\1\s\Core\Helpers\DelegatingHandlers\ExceptionDelegatingHandler.cs:line 39 at Polly.CircuitBreaker.AsyncCircuitBreakerEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext, ExceptionPredicates shouldHandleExceptionPredicates, ResultPredicates1 shouldHandleResultPredicates, ICircuitController1 breakerController) at Polly.AsyncPolicy1.ExecuteAsync(Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext) at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Nadobe.Core.Services.ConsentService.CreateConsentRequest(Accreditation accreditation, List1 evidenceCodesRequiringConsent) in D:\a\1\s\Core\Services\ConsentService.cs:line 324 at Nadobe.Core.Services.ConsentService.Initiate(Accreditation accreditation) in D:\a\1\s\Core\Services\ConsentService.cs:line 92 at Nadobe.Core.FuncAuthorization.<>c__DisplayClass8_0.<<Run>b__0>d.MoveNext() in D:\a\1\s\Core\FuncAuthorization.cs:line 128 --- End of stack trace from previous location where exception was thrown --- at Nadobe.Core.Filters.AzureFilterHelper.Handle(List1 attributes, ExecutionContext context, ILogger logger, HttpRequestMessage request, Func`1 action) in D:\a\1\s\Core\Filters\AzureFilterHelper.cs:line 98",
}

AccreditationList needs to be filtered on servicecontext and not just owner

Description

In order to prevent cross-context lookups

Additional Information

            while (query.HasMoreResults)
            {
                foreach (Accreditation accreditation in await query.ExecuteNextAsync<Accreditation>())
                {
                    var listOfServiceContexts = new List<string>();

                    if (!string.IsNullOrEmpty(accreditation.ServiceContext))
                        listOfServiceContexts.Add(accreditation.ServiceContext);

                    listOfServiceContexts.AddRange(accreditation.EvidenceCodes.FirstOrDefault().GetBelongsToServiceContexts());
                   
                    if (listOfServiceContexts.Contains(_requestContextService.ServiceContext.Name))
                        accreditations.Add(accreditation);
                }
            }

Tasks

No response

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • chore(deps): update dependency moq to 4.20.72
  • chore(deps): update dependency stackexchange.redis to 2.8.16
  • chore(deps): update dependency microsoft.azure.cosmos to 3.43.0
  • chore(deps): update dependency microsoft.azure.functions.worker.applicationinsights to 1.4.0
  • chore(deps): update dependency microsoft.azure.functions.worker.extensions.cosmosdb to 4.11.0
  • 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/codeql.yml
  • actions/checkout v4
  • github/codeql-action v3
  • actions/setup-dotnet v4
  • github/codeql-action v3
.github/workflows/common-release-nuget.yml
  • actions/checkout v4
  • actions/setup-dotnet v4
.github/workflows/core-ci.yaml
.github/workflows/core-cicd.yaml
  • actions/checkout v3
  • actions/setup-dotnet v3
  • actions/upload-artifact v3
  • actions/download-artifact v3
  • azure/functions-action v1
  • actions/download-artifact v3
  • azure/functions-action v1
  • azure/login v1
  • actions/download-artifact v3
  • azure/functions-action v1
.github/workflows/core-pr.yaml
  • actions/checkout v4
  • actions/setup-dotnet v4
.github/workflows/post-deploy-test.yml
.github/workflows/sonarcloud.yml
  • SonarSource/sonarcloud-github-action f5003fc9688ade81ce47b57a3fa97a8d3f12de4c
nuget
Dan.Common.UnitTest/Dan.Common.UnitTest.csproj
  • coverlet.collector 6.0.2
  • MSTest.TestFramework 3.5.1
  • MSTest.TestAdapter 3.5.1
  • Moq 4.20.70
  • Microsoft.NET.Test.Sdk 17.10.0
Dan.Common/Dan.Common.csproj
  • Microsoft.SourceLink.GitHub 8.0.0
  • StackExchange.Redis 2.8.0
  • Polly.Caching.Serialization.Json 3.0.0
  • Polly.Caching.Distributed 3.0.1
  • Newtonsoft.Json 13.0.3
  • Microsoft.Extensions.Http.Polly 8.0.7
  • Microsoft.Extensions.Caching.StackExchangeRedis 8.0.7
  • Microsoft.Azure.Functions.Worker.ApplicationInsights 1.3.0
  • Microsoft.Azure.Functions.Worker 1.23.0
  • Microsoft.Azure.Core.NewtonsoftJson 2.0.0
  • Microsoft.ApplicationInsights.WorkerService 2.22.0
  • AsyncKeyedLock 7.0.0
Dan.Core.UnitTest/Dan.Core.UnitTest.csproj
  • Polly.Caching.Memory 3.0.2
  • coverlet.collector 6.0.2
  • MSTest.TestFramework 3.5.1
  • MSTest.TestAdapter 3.5.1
  • Moq 4.20.70
  • Microsoft.NET.Test.Sdk 17.10.0
  • Microsoft.Extensions.Caching.Memory 8.0.0
Dan.Core/Dan.Core.csproj
  • System.ServiceModel.Security 6.0.0
  • System.ServiceModel.Primitives 8.0.0
  • System.ServiceModel.NetTcp 8.0.0
  • System.ServiceModel.Http 8.0.0
  • System.ServiceModel.Duplex 6.0.0
  • System.Private.ServiceModel 4.10.3
  • System.IdentityModel.Tokens.Jwt 8.0.1
  • Polly 8.4.1
  • Microsoft.IdentityModel.Tokens 8.0.1
  • Microsoft.IdentityModel.Protocols.OpenIdConnect 8.0.1
  • Microsoft.Extensions.Configuration.UserSecrets 8.0.0
  • Microsoft.Extensions.Caching.StackExchangeRedis 8.0.7
  • Microsoft.Extensions.Caching.Abstractions 8.0.0
  • Microsoft.Azure.Functions.Worker.Sdk 1.17.4
  • Microsoft.Azure.Functions.Worker.Extensions.Timer 4.3.1
  • Microsoft.Azure.Functions.Worker.Extensions.Http 3.2.0
  • Microsoft.Azure.Functions.Worker.Extensions.CosmosDB 4.10.0
  • Microsoft.Azure.Functions.Worker.ApplicationInsights 1.3.0
  • Microsoft.Azure.Functions.Worker 1.23.0
  • Microsoft.Azure.Cosmos 3.42.0
  • Microsoft.ApplicationInsights.WorkerService 2.22.0
  • jose-jwt 5.0.0
  • JmesPath.Net 1.0.330
  • GitInfo 3.3.5
  • Azure.Security.KeyVault.Secrets 4.6.0
  • Azure.Identity 1.12.0

  • Check this box to trigger a request for Renovate to run again on this repository

Plugin-skatteetaten datasets missing after deploy

Description

The expectation is that all datasets should appear as normal after deploy and repopulation of the evidencecode cache

Steps To Reproduce

Deploy new core code
Skatteetaten datasets are unavailable until the core cache of evidencecodes is forcibly refreshed

Introduce metadata at field level (evidence values)

Description

We need to add a "description" field on the evidence value model in order to create better metadata for published data sets.

The field should be included when requesting meta data and omitted when returning the actual values.

Additional Information

No response

Tasks

No response

Upgrade to isolated worker model and .NET 8

Description

Support for .NET 6 ends on 12 November 2024—upgrade your Azure Functions apps to .NET 8
You're receiving this email because you have one or more Azure Functions apps that use .NET 6.
In alignment with the end of community support, on 12 November 2024, support for .NET 6 in Azure Functions will end. Your apps that are hosted on Functions will continue to run, but security updates will no longer be available and we'll no longer provide customer service for .NET 6. Learn more about Azure Functions stack version support.

Additional Information

In addition the in-process model will be retired in 2026, so we need to make sure all plugins and functions are running .NET 8 and isolated.

Tasks

No response

Verify use of APIM management apis

Description

Required action
Please migrate all your existing tools and scripts from using the Direct Management API instances to the Azure Resource Manager-based REST API by 15 March 2025. If you have existing instances using Direct Management API, please rely on the Azure Resource Manager-based REST API, which is the standard for all Azure services today. After 15 March 2025, the Direct Management API won’t be supported.

Additional Information

No response

Tasks

No response

Remove certificate authentication support

Description

All clients are using maskinporten so certificate support should be removed from api mgmt, code and docs

Additional Information

No response

Tasks

  • Apim
  • Code
  • Docs

test

Description

test2

Additional Information

test 3

Tasks

No response

Better support for multiple dataset names

Description

In order to easier be able to reuse datasets and integrations between service contexts, evidencecodes should support an alias list in the evidence code model.

Core needs to be able to look for both evidence code names and aliases from getavailableevidencecodes when validation requests and also correctly route to the correct name when communication with the plugin.

Additional Information

No response

Tasks

No response

Demo service for data.altinn.no data

Altinn 3 app - belonging to testdepartementet to begin with

Authorization requirement is DAGL for an organization
After login present a landing page with options for data.altinn.no servicecontexts retrived from metadata api

It needs a jwk for authentication with maskinporten, clientid with all DAN scopes and apikeys for every servicecontext.

After selecting a context, present all datasets within that context using organization number that the user is currently representing, similar to the poc - each dataset in a collapsible accordion or similar with the possibility to mark them for inclusion in a report. If possible, create everything as generic as possible - using metadata about the datasets to choose how they are presented (key/value, json schema etc) and allow for help texts for each dataset and data element.

Simple button at the bottom for creating a pdf/report of the selected datasets and/or sending to email/altinn correspondence to a given recipient

https://github.com/data-altinn-no/poc-duediligence

Further ahead: inclue dialogporten items for the selected organization to further enhance "my business portal"

Revision of open datasets

Remove belongsto and requirements on open datasets unitbasic, roller, bilpleie, renhold, bemanning, firmaattest, årsregnskap and corresponding datasets in aggregate plugins

Envelope = false breaks Tilda datasets

Description

The hashtable solution for removing the DAN envelope from dataset responses doesn't work for Tilda because of multiple occurences of dataset values with the same name.
Either we make the envelope subject to an evidencecode flag "AllowEnvelopeRemoval" or we maybe create a tilda model that can be serialised where all the values are contained within a single object. Or we think of something smarter :)

Additional Information

No response

Tasks

No response

test

Description

asdsad

Additional Information

No response

Tasks

No response

Redefine deploy action for CI/CD

Description

Make sure steps are run in correct order - deploy-> test, gate and so on - consider making separate workflow for core instead of using the same flow for plugins AND core

Steps To Reproduce

Deploy

Update Cosmos sdk from v2 to v3

Vi må oppdatere cosmos-bruken i Core (det er eneste sted vi har direkte kommunikasjon med Cosmos).

Samtidig burde vi også bumpe til .net 8 på dancore.

Description

Update to version 3 of the Azure Cosmos DB .NET SDK by 31 August 2024
You're receiving this email because you've used version 2.x of the Azure Cosmos DB .NET SDK.

On 31 August 2024 we'll retire version 2.x of the Azure Cosmos DB .NET SDK, and you'll need to update your SDK to version 3.x by that date. Version 3 of the Azure Cosmos DB .NET SDK includes many usability and performance improvements over version 2 of the SDK as well as access to new Azure Cosmos DB features.

Version 3 includes:

Improved performance through stream API support.
Built-in support for change feed processor library and bulk operations.
Mockable APIs for easier unit testing.
Improved diagnostics and observability experience.
Access to new APIs and features including transactional batch, patch, hierarchial partition keys and many more.
Required action
To avoid service disruptions, migrate to Azure Cosmos DB .NET SDK version 3 by 31 August 2024.

Help and support
If you have questions, ask the community experts in Microsoft Q&A or post your them on our GitHub. If you have a support plan and require technical support, create a support request:

For Summary, type Cosmos DB .NET SDK.
For Issue type, select Technical.
For Subscription, select your subscription.
For Service, select My services, then select Cosmos DB.
For Resource, select your resource.
For Problem type, select SQL API SDK – NET.
For Problem subtype, select Error or unexpected result.

Additional Information

No response

Tasks

No response

JsonSchema revision

Description

Create jsonschemas or models that can be used to create json schemas where missing (fregperson, summertskattegrunnlag ++)
so that external clients know what models to expect

Additional Information

No response

Tasks

No response

test project

Description

dfbgdfgfdg

Additional Information

No response

Tasks

No response

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.