Disable HTTP protocol usage on Azure functions instances under SSL settings
https://microsoft.sharepoint.com/teams/TLS12Everywhere/SitePages/Azure-Functions.aspx

Currently the batch job manager terminates even there are corresponding scan tasks pending/running.

The expected job manager termination workflow:

1. Stop queuing batch task
2. Wait for pending task to complete
3. Terminate job manager

The sdk doesnt seem to have retry on failure response(https://github.com/Azure/ms-rest-nodeauth/blob/adea8415de97a8076006fbd9948123d09dfdd78d/lib/credentials/msiAppServiceTokenCredentials.ts#L94). So we may need to add retry & caching, make class singleton for results returned by CredentialsProvider.

Refer https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token#error-handling

Currently, in Azure function sometimes, we have received the below error in production:
Result: Failure

Exception: Error: MSI: Failed to retrieve a token from "http://127.0.0.1:41709/MSI/token/?resource=https%3A%2F%2Fvault.azure.net&api-version=2017-09-01" with an error: {"ExceptionMessage":"","ErrorCode":"service_unavailable","ServiceErrorCodes":null,"StatusCode":503,"Message":null,"CorrelationId":"8b0bb15d-11f7-4a3b-8fbb-3a55ca98a884"}
Stack: Error: MSI: Failed to retrieve a token from "http://127.0.0.1:41709/MSI/token/?resource=https%3A%2F%2Fvault.azure.net&api-version=2017-09-01" with an error: {"ExceptionMessage":"","ErrorCode":"service_unavailable","ServiceErrorCodes":null,"StatusCode":503,"Message":null,"CorrelationId":"8b0bb15d-11f7-4a3b-8fbb-3a55ca98a884"}
    at MSIAppServiceTokenCredentials.<anonymous> (D:\home\site\wwwroot\get-report-func\index.js:56446:23)
    at Generator.next (<anonymous>)
    at fulfilled (D:\home\site\wwwroot\get-report-func\index.js:56384:58)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

The solution had single logging provider Logger() in a corresponding library /Logger. To support HTTP request singleton pattern the new logging provider was introduced ContextAwareLogger(). The sparse usage of a new logging provider introduced inconsistent logging pattern where some packages use Logger() (for example the azure-services package) and web-api package uses ContextAwareLogger().

As result the Logger() indirectly being used in HTTP context where ContextAwareLogger() is used as well which do not let log context related event properly.

Expected implementation pattern

The context aware pattern should be implemented on IoC container level instead. Any other providers that require context awareness should be part on that container (like logging library). The context IoC container should follow HTTP request singleton pattern.

Describe the bug

Throttle client requests on API management, in case of high traffic.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

The object property should be expanded correctly for logging.

[Trace][error][properties - { source: 'webApiScanJobManager' }] === An error occurred while adding new task {"status":"ClientError","taskId":"task_dc2a1e5a-dfbb-4274-bc02-44c88d8ec512_1636d7dac58c76b95633","error":{"code":"InvalidPropertyValue","message":{"lang":"en-US","value":"The value provided for one of the properties in the request body is invalid.\nRequestId:9278f373-0952-4a07-a879-5b40af6698bd\nTime:2020-01-24T21:14:57.7025004Z"},"values":[{"key":"PropertyName","value":"name"},{"key":"PropertyValue","value":"TASK_ARGUMENTS"},{"key":"Reason","value":"Duplicate Environment Setting name"}]},"[email protected]":"https://..."} to the job on-demand-url-scan-schedule:job-35.
[Trace][info][properties - { '0': '[',
  '1': 'o',
  '2': 'b',
  '3': 'j',
  '4': 'e',
  '5': 'c',
  '6': 't',
  '7': ' ',
  '8': 'O',
  '9': 'b',
  '10': 'j',
  '11': 'e',
  '12': 'c',
  '13': 't',
  '14': ']',
  source: 'webApiScanJobManager' }] === Pool load statistics

Describe the bug
A clear and concise description of what the bug is.

Area

• Install script
• Update script
• Service

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Log files
If applicable, any log files / console output that might help explain what the issue might be.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Restrict access to Azure Web API instance to Azure API Management service 's IP range only

https://docs.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions

Describe the bug

Currently if the user submits scan for page that is not accessible, we return internal server error.

For scan failure like 401, 404, etc, we should be returning proper error state to customer instead of just giving internal server error message.

To support scan priority processing the request storage queue size should be small enough to not hold priority scan request for too long. Optimal queue size should be 2x of a single batch processing ie priority message should be red no longer than 2x time of average single message processing.
To achieve this batch pool manager calculated size data need to be connected with scan request sender.

Dev environment currently tries to access service principal to add reply URL to app during installation.

Describe the bug

When the url requested for scan loads with failure error code, we don't send the error code returned by the page. We simply return state as httpErrorCode failure. It is better to always send the final response code of the scanned page.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

The Batch service will terminate long running scan tasks. In this case no corresponding state will be updated within Cosmom DB. The scan state will be marked as 'running'.

There should be a mechanism to handle batch scan tasks state after external termination. Either job manager should re-validate tasks states periodically or job scheduler post task should do re-validation.

In azure cosmos sdk 3.x they removed partition key in RequestOption and FeedOption, we need to specify the pk in the query.
See:
Azure/azure-sdk-for-js#6841

If the url is unreachable, the reponse should have proper error code instead of returning internal server error.

The scan request queue message should be abstracted from Page document.

[Exception][properties - { source: 'scanRequestSender' }] === { VError: Error occurred while executing job: RequestBodyTooLargeThe request body is too large and exceeds the maximum permissible limit.
RequestId:72500c2d-1003-001a-1e51-33cfc4000000
Time:2019-07-05T16:46:24.4987695Z65536
at Logger.trackExceptionAny (/mnt/batch/tasks/workitems/scan-req-schedule/job-19/scan-req-sender-task/wd/sender.js:150998:29)
at ScanRequestEntryPoint. (/mnt/batch/tasks/workitems/scan-req-schedule/job-19/scan-req-sender-task/wd/sender.js:151392:24)
at Generator.throw ()
at rejected (/mnt/batch/tasks/workitems/scan-req-schedule/job-19/scan-req-sender-task/wd/sender.js:151350:65)
at process._tickCallback (internal/process/next_tick.js:68:7)
jse_shortmsg: 'Error occurred while executing job',
jse_cause:
{ Error: RequestBodyTooLargeThe request body is too large and exceeds the maximum permissible limit.
RequestId:72500c2d-1003-001a-1e51-33cfc4000000
Time:2019-07-05T16:46:24.4987695Z65536
at new RestError (/mnt/batch/tasks/workitems/scan-req-schedule/job-19/scan-req-sender-task/wd/sender.js:20190:28)
at /mnt/batch/tasks/workitems/scan-req-schedule/job-19/scan-req-sender-task/wd/sender.js:19014:37
at process._tickCallback (internal/process/next_tick.js:68:7)
code: undefined,
statusCode: 413,

Describe the bug
delete the app registration created in clean install script
Area

• Install script
• Update script
• Service

Expected behavior
A clean install should remove the service principal created before.

Describe the bug

After deploying azure function, sometimes the function may not be available to publish code.
In this case running "func azure functionapp publish ..." will silently fail with log:

Publishing 'web-api' scripts to 'web-api-allyfuncappxjohj2cwcdj4c' Function App...
Can't find app with name "web-api-allyfuncappxjohj2cwcdj4c"
Successfully published 'web-api' scripts to 'web-api-allyfuncappxjohj2cwcdj4c' Function App.

We should check if the azure function exists & retry.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

When health monitoring orchestration reports unavailability E2E tests must provide results to easy diagnosis the service failure point. This bug renders out all benefit of running e2e in production.

The batch pool may experience intermittent Internet connectivity issue. This may require re-run failed scans.

{ TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded\n at Promise.then (/mnt/batch/tasks/shared/batch-web-api-scan-runner/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21)\n -- ASYNC --\n at Frame. (/mnt/batch/tasks/shared/batch-web-api-scan-runner/node_modules/puppeteer/lib/helper.js:111:15)\n at Page.goto (/mnt/batch/tasks/shared/batch-web-api-scan-runner/node_modules/puppeteer/lib/Page.js:629:49)\n at Page. (/mnt/batch/tasks/shared/batch-web-api-scan-runner/node_modules/puppeteer/lib/helper.js:112:23)\n at Page. (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180776:55)\n at Generator.next ()\n at /mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180753:71\n at new Promise ()\n at module.exports.../scanner/dist/page.js.__awaiter (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180749:12)\n at Page.scanForA11yIssues (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180775:16)\n at Scanner. (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180897:40)\n -- ASYNC --\n at Page. (/mnt/batch/tasks/shared/batch-web-api-scan-runner/node_modules/puppeteer/lib/helper.js:111:15)\n at Page. (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180776:55)\n at Generator.next ()\n at /mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180753:71\n at new Promise ()\n at module.exports.../scanner/dist/page.js.__awaiter (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180749:12)\n at Page.scanForA11yIssues (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180775:16)\n at Scanner. (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180897:40)\n at Generator.next ()\n at fulfilled (/mnt/batch/tasks/workitems/on-demand-url-scan-schedule/job-1534/task_acb3de9c-3155-4779-9944-f8a7b3e9e7e5_81d9e3eee4a6701c083e/wd/web-api-scan-runner.js:180873:58) name: 'TimeoutError' }

We need this to copy private secrets that are required (for eg: for security & monitoring in vms)

Describe the bug

{
    "url": "https://www.bing.com",
    "priority": 0
}

instead of

[
  {
    "url": "https://www.bing.com",
    "priority": 0
  }
]

will error on invalid url / 500 instead of malformed request and consequently be harder to debug for a consumer.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

Describe the bug
A clear and concise description of what the bug is.

Area

• Install script
• Update script
• Service

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Log files
If applicable, any log files / console output that might help explain what the issue might be.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Currently, the installation script does not check the existence of VMSS resource groups or retry when enabling system-assigned managed identity. This can cause errors if the resource group does not yet exist.

Describe the bug

In general the documentation is very sparse. The documentation that is present is wrong (asks for the function host to be started" before starting the debug from VS Code. This isn't true. As a part of debugging it does start up the host.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

The Cosmos DB related install scripts should be updated due to AZ API has been obsoleted

[setup-cosmos-db] Checking if database 'scanner' exists in Cosmos account ... in resource group ...
This command has been deprecated and will be removed in a future release. Use 'cosmosdb sql database, cosmosdb mongodb database, cosmosdb cassandra keyspace or cosmosdb gremlin database' instead.
This command is implicitly deprecated because command group 'cosmosdb database' is deprecated and will be removed in a future release. Use 'cosmosdb sql database, cosmosdb mongodb database, cosmosdb cassandra keyspace or cosmosdb gremlin database' instead.

[setup-cosmos-db] Checking if collection 'a11yIssues' exists in db 'scanner' of cosmosAccount ... in resource group ...
This command has been deprecated and will be removed in a future release. Use 'cosmosdb sql container, cosmosdb mongodb collection, cosmosdb cassandra table, cosmosdb gremlin graph or cosmosdb table' instead.
This command is implicitly deprecated because command group 'cosmosdb collection' is deprecated and will be removed in a future release. Use 'cosmosdb sql container, cosmosdb mongodb collection, cosmosdb cassandra table, cosmosdb gremlin graph or cosmosdb table' instead.

Adding secret for cosmosDbUrl in key vault ...
This command has been deprecated and will be removed in a future release. Use 'cosmosdb keys list' instead.

Describe the bug

This has to be done after a month. (after Jan 1) so that all the old documents that has scanError as string would have been deleted.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

Describe the bug

The rest api always returns 404 with no err message when any error occurs.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:
send a scan status get request with an invalid guid.

Expected behavior

the response should indicate that the input is not valid.

Log files

Screenshots

Additional context

Is your feature request related to a problem? Please describe.

Spoke with @manishsat and he mentioned batching is no longer required and we should remove batching from our API spec and implementations.

Describe the solution you'd like

This work will include changing the API spec as well as cutting down unwanted code in the API layer only. Both submit and get will no longer be batched.

• Post scans will take in a single URL with priority
• There will be a single GET scan endpoint that takes in a single url. The batch endpoint will be dropped as part of a different feature.
• The GET report endpoint remains unchanged.

Scope is limited to changing the API spec and API layer. Everything else will remain as is and everything underneath is still handled the same way. A separate feature will work to remove the flab and re-architect everything below the API layer.

Work with Uday and come up with a way to transition smoothly.

Key Vault's soft delete feature allows recovery of the deleted vaults and vault objects, known as soft-delete

https://docs.microsoft.com/en-us/azure/key-vault/key-vault-ovw-soft-delete

Describe the bug
A clear and concise description of what the bug is.

Area

• Install script
• Update script
• Service

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Log files
If applicable, any log files / console output that might help explain what the issue might be.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Scan only files of content type - text/html
We observed a link to yaml file that has content type as "application/octet-stream" timing out

Restrict access to Azure Worker function instance to Cosmos DB service 's IP range only

https://docs.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions

Describe the bug

When the scan request sender hits 429's it fails (without logging to application insights).
Examples - https://github.com/microsoft/accessibility-insights-service/blob/master/packages/web-api-scan-request-sender/src/sender/on-demand-dispatcher.ts#L35

https://github.com/microsoft/accessibility-insights-service/blob/master/packages/web-api-scan-request-sender/src/sender/on-demand-scan-request-sender.ts#L20

We should go through our code and find all instances where we can possibly hit 429.

A common way of handling these would be great.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

We log the 429 into application insights along with appropriate data.

Log files

Running Scan request sender for on demand requests
[Trace][info][properties - { source: 'webApiScanRequestSender' }] === [Sender] Maximum queue size configuration set to 2000
[Trace][info][properties - { source: 'webApiScanRequestSender' }] === [Sender] Current queue size is 446
[Exception][properties - { source: 'webApiScanRequestSender' }] === { VError: Error occurred while executing action.
at Logger.trackExceptionAny (/mnt/batch/tasks/workitems/on-demand-scan-req-schedule/job-19760/on-demand-scan-req-sender-task/wd/sender.js:221355:29)
at WebApiScanRequestSenderEntryPoint. (/mnt/batch/tasks/workitems/on-demand-scan-req-schedule/job-19760/on-demand-scan-req-sender-task/wd/sender.js:222048:24)
at Generator.throw ()
at rejected (/mnt/batch/tasks/workitems/on-demand-scan-req-schedule/job-19760/on-demand-scan-req-sender-task/wd/sender.js:222005:65)
at process._tickCallback (internal/process/next_tick.js:68:7)
jse_shortmsg: 'Error occurred while executing action.',
jse_info:
{ error:
{ code: 429,
body:
'{"code":"429","message":"Message: {\"Errors\":[\"Request rate is large\"]}\r\nActivityId: 262a7648-44a4-4e07-9b7f-13aaee548910, Request URI: /apps/4f5c042d-76fb-4ce6-bda3-517e6ef3984f/services/1ddcee52-6dbd-4f48-b37c-f554a4589315/partitions/025e9987-207d-48ed-99be-59cc04da3dc4/replicas/132136347263726188p/, RequestStats: \r\nRequestStartTime: 2019-10-21T11:50:41.6881406Z, RequestEndTime: 2019-10-21T11:50:41.6981437Z, Number of regions attempted:1\r\nResponseTime: 2019-10-21T11:50:41.6981437Z, StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-westus2-fd2.documents.azure.com:16824/apps/4f5c042d-76fb-4ce6-bda3-517e6ef3984f/services/1ddcee52-6dbd-4f48-b37c-f554a4589315/partitions/025e9987-207d-48ed-99be-59cc04da3dc4/replicas/132136347263726188p/, LSN: 4981, GlobalCommittedLsn: 4981, PartitionKeyRangeId: , IsValid: True, StatusCode: 429, SubStatusCode: 3200, RequestCharge: 0.38, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, ResourceType: Document, OperationType: Delete\r\n, SDK: Microsoft.Azure.Documents.Common/2.7.0"}',
headers:
{ 'transfer-encoding': 'chunked',
'content-type': 'application/json',
'content-location':
'https://allycosmosli56mzbng5jpe-westus2.documents.azure.com/dbs/onDemandScanner/colls/scanRequests/docs/1e9f3f3e-8c41-6d81-017b-8b1566d09429',
server: 'Microsoft-HTTPAPI/2.0',
'x-ms-retry-after-ms': '897',
lsn: '4981',
'x-ms-schemaversion': '1.8',
'x-ms-quorum-acked-lsn': '4981',
'x-ms-substatus': '3200',
'x-ms-current-write-quorum': '3',
'x-ms-current-replica-set-size': '4',
'x-ms-xp-role': '1',
'x-ms-global-committed-lsn': '4981',
'x-ms-number-of-read-regions': '0',
'x-ms-transport-request-id': '2077',
'x-ms-cosmos-llsn': '4981',
'x-ms-cosmos-quorum-acked-llsn': '4981',
'x-ms-request-charge': '0.38',
'x-ms-serviceversion': 'version=2.7.0.0',
'x-ms-activity-id': '262a7648-44a4-4e07-9b7f-13aaee548910',
'strict-transport-security': 'max-age=31536000',
'x-ms-gatewayversion': 'version=2.7.0',
date: 'Mon, 21 Oct 2019 11:50:41 GMT',
'x-ms-throttle-retry-count': 9,
'x-ms-throttle-retry-wait-time-ms': 6728 },
activityId: '262a7648-44a4-4e07-9b7f-13aaee548910',
substatus: 3200,
retryAfterInMilliseconds: 897,
requestHeaders:
{ 'Cache-Control': 'no-cache',
'x-ms-version': '2018-06-18',
'User-Agent':
'linux/4.15.0-1060-azure Nodejs/v10.16.3 azure-cosmos-js/2.1.7',
'x-ms-documentdb-partitionkey': '["pageScanRequestDocuments"]',
'x-ms-date': 'Mon, 21 Oct 2019 11:50:34 GMT',
Accept: 'application/json',
authorization:
'type%3Dmaster%26ver%3D1.0%26sig%3DMiFlbUrqqyPz2zhn9tMWBfy9JRzdVL%2FbxgZd9FLNL0Y%3D' } } },
message: 'Error occurred while executing action.' }

Screenshots

Additional context

The GET scan response HTTP payload malformed and contains misleading data (see below). The payload should be in the following format:

{
"scanId": "1e9dc9b0-2522-6411-f0d5-cc6817d96315",
"url": "https://support.microsoft.com/en-us",
"run": {
"state": "failed",
"timestamp": "2019-09-21T18:13:54.812Z",
"error": ""
}
}

Malformed payload:
{
"scanId": "1e9dc9b0-2522-6411-f0d5-cc6817d96315",
"url": "https://support.microsoft.com/en-us",
"scanResult": {
"state": "unknown",
"issueCount": 0
},
"reports": [
{
"reportId": "",
"href": "",
"format": "sarif"
}
],
"run": {
"state": "failed",
"timestamp": "2019-09-21T18:13:54.812Z",
"error": "{ TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded..."
}
}

Investigate using SAS tokens instead of access keys for accessing Azure Blob storage

Corresponding Azure DevOps task: User Story 1672989: Investigate using SAS tokens instead of access keys for accessing Azure Blob Storage

Describe the bug

Currently all the logs stdout/stderr) are sitting on the host machine and we will loose them, if we recycle/reimage the vms.

Area

• Install script
• Update script
• [X ] Service

We need to make change while creating task from job manager to specify the location where we want logs to be generated.

https://docs.microsoft.com/en-us/azure/batch/batch-task-output-files

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

Currently, web-worker is consuming cosmos container client directly to delete batch request document. We need to move BatchScanRequestDataService to service library

Describe the bug

Currently we are building the azure function app(dotnet build ont he csproj) in release definition before publishing. We should be building in the build definition instead & let the release definition do only the deployment.

Area

• Install script
• Update script
• Service

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Log files

Screenshots

Additional context

The href link should be fully qualified URL that match to the GET scan report REST API including the corresponding API version:

/scans/{scanid}//reports/{reportid}?api-version=1.0

Describe the bug
A clear and concise description of what the bug is.

Area

• Install script
• Update script
• Service

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Log files
If applicable, any log files / console output that might help explain what the issue might be.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

The cosmosContainerClient.mergeOrWriteDocument() API should have support for the option to set existing DB document property to undef on merge ie merged document should have existing property removed when for merging client document property is set to null for instance.

The REST API is exposed through Azure API Management service. The scan response report link should have APIM host name instead of Azure Function host name.

The Azure APIM should have policy added that set Forwarded HTTP header. The func app should use Forwarded host= header to set correct host name in a report link. Reference to Add a Forwarded header policy example.

{
"scanId": "1e9e4937-6a37-64e1-cfd4-85cdcfb8ec2e",
"url": "https://code.visualstudio.com/",
"scanResult": {
"state": "fail",
"issueCount": 6
},
"reports": [
{
"reportId": "1e9e493a-bd7c-61c0-bfcd-316aa1961397",
"format": "sarif",
"links": {
"rel": "self",
"href": "https://web-api-allyfuncappgghrfwzvmcdly.azurewebsites.net/api/scans/1e9e4937-6a37-64e1-cfd4-85cdcfb8ec2e/reports/1e9e493a-bd7c-61c0-bfcd-316aa1961397?api-version=1.0"
}
}
],
"run": {
"state": "completed",
"timestamp": "2019-10-01T21:37:30.587Z"
}
},

Describe the bug
In the get scan request the response header for content/type is currently set to text/plain. Should it perhaps bet set to "application/json" to better reflect what is being sent? Additionally all other requests should also set this header.

A possible solution would be to use a decorator class that would set the appropriate headers for all outgoing responses.

Area

• Install script
• Update script
• Service

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Log files
If applicable, any log files / console output that might help explain what the issue might be.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

The scan batch task should have priority assigned that is corresponding to the client scan request priority.

Refactor OnDemandPageScanRunResultProvider API as follows:

1. The updateScanRun() should return the updated document version that has been returned by underlying Azure Cosmos DB client on write operation. Missing this return value result read document after write that is redundant and expensive Cosmos operation

2. Add single version APIs: readScanRun(), writeScanRun()

3. The saveSarifReport() API should return the created blob path

The GET scan response should NOT contain the internal server error. The REST API should return pre-canned message to the client like "Internal server error occured when run URL scan" .

The accepted scan priority range is from -1000 to 1000

We have duplicate code Batch.ts, PoolLoadGenerator which can be made generic & consumed by both our job manager packages.