k-phoen / backstage-plugin-confluence Goto Github PK

Backstage 1.1.0 was released recently. Would be great to follow their release cadence and keep nested dependencies in sync.

Getting the following error when trying to setup the plugin. URL is correct, authorization is correct, have tested a basic CURL request and am able to return successfully. Doesn't give a lot more information than that in the logs. Any suggestions?

Thank you, this plugin looks amazing. Really excited to get it implemented.

I’ve noticed a discrepancy in the data fields returned by the getDocumentInfo function in the Backstage plugin for Confluence. Specifically, the lastModifiedBy field is expected to be data.version.by.publicName, but in the data returned from Confluence for my organization, the equivalent field is data.version.by.displayName.

Here’s the current return statement:

return [{ title: data.title, text: this.stripHtml(data.body.storage.value), location: ${this.wikiUrl}${data._links.webui}, spaceKey: data.space.key, spaceName: data.space.name, ancestors: ancestors, lastModifiedBy: data.version.by.publicName, lastModified: data.version.when, lastModifiedFriendly: data.version.friendlyWhen }];

And here’s what would work for my organization’s data:

return [ { title: data.title, text: this.stripHtml(data.body.storage.value), location:${this.wikiUrl}${data._links.webui}, spaceKey: data.space.key, spaceName: data.space.name, ancestors: ancestors, lastModifiedBy: data.version.by.displayName, lastModified: data.version.when, lastModifiedFriendly: new Date(data.version.when).toLocaleDateString(), }, ];

Additionally, I’ve found an issue with the stripHtml function. It doesn’t seem to be stripping HTML tags in a reliable manner, and some unwanted special characters are still left in the output.

I would appreciate it if these issues could be looked into

To keep the project up-to-date and functioning effectively, we need to establish a regular maintenance schedule that includes the following tasks:

Dependencies Update

1. Regularly update all project dependencies to their latest versions.
2. Ensure compatibility with the new backend system.
3. Identify and replace any deprecated packages with suitable alternatives.
4. Verify that all dependencies are compatible with the existing codebase.
5. Conduct regular reviews and merging of pull requests (PRs).

We have been exploring this plugin and have just upgraded Backstage to 1.11.x

I think the SearchResults setup has changed:

<SearchResult>
    <CatalogSearchResultListItem icon={<CatalogIcon />} />
    <TechDocsSearchResultListItem icon={<DocsIcon />} />
</SearchResult>

The

ConfluenceResultListItem

Does not match - and I cant get the changes described in Readme to work (have tried some simple changes).
We will continue to investigate - but sharing for visibility.

I am thinking about adding some new functionality to the plugin.
It's currently in the hack/explore phase.

However I am struggling to get my own Backstage App to link to a local checkout of this package and allow easy development.

I wondered how your flow works, and how you do your build/dev cycle.

You can see my proposed changes here:
https://github.com/dlaird-ovo/backstage-plugin-confluence/tree/feat/add_dynamic_spaces

The idea here is that we can avoid modifying app-config to add spaces - but instead users can update the resource and on the next iteration it will index those spaces.

Doesn't seem to be working.

{
  "error": {
    "name": "InputError",
    "message": "Invalid query string: [\n  {\n    \"code\": \"custom\",\n    \"message\": \"Invalid input\",\n    \"path\": [\n      \"types\",\n      0\n    ]\n  }\n]",
    "stack": "InputError: Invalid query string: [\n  {\n    \"code\": \"custom\",\n    \"message\": \"Invalid input\",\n    \"path\": [\n      \"types\",\n      0\n    ]\n  }\n]\n    at <anonymous> (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/@backstage/plugin-search-backend/src/service/router.ts:182:13)\n    at handleReturn (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express-promise-router/lib/express-promise-router.js:24:23)\n    at /mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express-promise-router/lib/express-promise-router.js:64:7\n    at handleReturn (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express-promise-router/lib/express-promise-router.js:24:23)\n    at /mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express-promise-router/lib/express-promise-router.js:64:7\n    at Layer.handle [as handle_request] (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/layer.js:95:5)\n    at next (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/route.js:149:13)\n    at Route.dispatch (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/route.js:119:3)\n    at Layer.handle [as handle_request] (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/layer.js:95:5)\n    at /mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/index.js:284:15\n    at Function.process_params (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/index.js:346:12)\n    at next (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/index.js:280:10)\n    at Layer.handle [as handle_request] (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/layer.js:91:12)\n    at trim_prefix (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/index.js:328:13)\n    at /mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/mnt/c/Users/u1175114/Gitlab-tstenv/backstage_poc/backstage/node_modules/express/lib/router/index.js:346:12)"
  },
  "request": {
    "method": "GET",
    "url": "/query?term=Backstage&types%5B0%5D=confluence"
  },
  "response": {
    "statusCode": 400
  }
}

I'm new to Backstage. I tried installing this plugin but I'm not sure if I did anything wrong. I replaced the existing default async function createPlugin. But now I'm getting error when running it in local

2023-02-01T11:52:43.015Z techdocs info Creating Local publisher for TechDocs type=plugin
Backend failed to start up ReferenceError: ElasticSearchSearchEngine is not defined
    at createPlugin (webpack-internal:///./src/plugins/search.ts:74:26)
    at main (webpack-internal:///./src/index.ts:101:95)

Hi, I get this issue while trying to run the plugin.
Somehow it cannot index it. Bu I see not specific error in the backend.

I'm not even sure which rest api it tries to run...

{
"error": {
"name": "MissingIndexError",
"message": "Missing index for confluence. This could be because the index hasn't been created yet or there was a problem during index creation.",
"stack": "MissingIndexError: Missing index for confluence. This could be because the index hasn't been created yet or there was a problem during index creation.\n at LunrSearchEngine.query (/Users/em/backstage/backstage/node_modules/@backstage/plugin-search-backend-node/dist/index.cjs.js:329:13)\n at /Users/em/backstage/backstage/node_modules/@backstage/plugin-search-backend/dist/cjs/router-8d1d26a0.cjs.js:250:67\n at handleReturn (/Users/em/backstage/backstage/node_modules/express-promise-router/lib/express-promise-router.js:24:23)\n at /Users/em/backstage/backstage/node_modules/express-promise-router/lib/express-promise-router.js:64:7\n at handleReturn (/Users/em/backstage/backstage/node_modules/express-promise-router/lib/express-promise-router.js:24:23)\n at /Users/em/backstage/backstage/node_modules/express-promise-router/lib/express-promise-router.js:64:7\n at Layer.handle [as handle_request] (/Users/em/backstage/backstage/node_modules/express/lib/router/layer.js:95:5)\n at next (/Users/em/backstage/backstage/node_modules/express/lib/router/route.js:144:13)\n at Route.dispatch (/Users/em/backstage/backstage/node_modules/express/lib/router/route.js:114:3)\n at Layer.handle [as handle_request] (/Users/em/backstage/backstage/node_modules/express/lib/router/layer.js:95:5)"
},
"request": {
"method": "GET",
"url": "/query?term=authz&types%5B0%5D=confluence"
},
"response": {
"statusCode": 500
}
}

I'm looking at least 4 confluence sites which I'd like to add to the search aggregator.
Can I get some pointers on how to modify the backend to accomplish this?

Is it possible to add confluence as a section under TechDocs rather than integrate into search?

Thanks

Hi,

Thanks for the plugin, noticed that when we have a custom domain for confluence, unable to use the plugin.

How would one get this working for on prem i.e on server Atlassian?

Hi, I am setting up the plugin and I came across an issue during the execution of the scheduled task that queries the confluence spaces.

As per code, if a space is returning a non-ok response, the execution is terminated, and the relevant error is printed.
I'd like to propose a change, where if a space returns a non-ok response, the execution continues, and the error generated from the bad space is logged.

My backstage instance produces this error:

2023-03-24T12:15:00.934Z search info exploring space DEVT type=plugin
2023-03-24T12:15:01.654Z search info exploring space MORP type=plugin
2023-03-24T12:15:02.080Z search warn non-ok response from confluence type=plugin
2023-03-24T12:15:02.081Z search error Collating documents for confluence failed: Error: Request failed with 404 Not Found type=plugin documentType=confluence
2023-03-24T12:15:02.081Z backstage error Request failed with 404 Not Found type=taskManager task=search_index_confluence stack=Error: Request failed with 404 Not Found
    at ConfluenceCollatorFactory.get (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:146:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ConfluenceCollatorFactory.getDocumentsFromSpace (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:93:20)
    at async ConfluenceCollatorFactory.getDocumentsFromSpaces (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:83:29)
    at async ConfluenceCollatorFactory.execute (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:40:27)
    at async next (node:internal/streams/from:85:11)
(node:922) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 11 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)

I have 3 spaces configured:

spaces: ['DEVT', 'MORP', 'REL']

DEVT and REL are valid, MORP doesn't exist. As you can see from the logs, MORP triggers the error, and as the execution terminates REL is never explored, and ultimately pages from DEVT and REL never get indexed.

The second error, more problematic for us, appeared after a few failed executions of the scheduled task:

2023-03-24T12:19:48.887Z backstage error Request failed with status 500 There was a problem performing the search query: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? type=errorHandler stack=Error: There was a problem performing the search query: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at /workspace/node_modules/@backstage/plugin-search-backend/dist/index.cjs.js:258:15
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)

It appears that the broken collation leaves the index in a bad state, causing the search query to fail for all results types.
and the search page shows infinite loading in the UI :

Looking at the code from ``,

private async getDocumentsFromSpace(space: string): Promise<string[]> {
        const documentsList = [];

        this.logger.info(`exploring space ${space}`);

        let next = true;
        let requestUrl = `${this.wikiUrl}/rest/api/content?limit=1000&status=current&spaceKey=${space}`;
        while (next) {
            const data = await this.get<ConfluenceDocumentList>(requestUrl);
            if (!data.results) {
                break;
            }

            documentsList.push(...data.results.map(result => result._links.self));

            if (data._links.next) {
                requestUrl = `${this.wikiUrl}${data._links.next}`;
            } else {
                next = false;
            }
        }

        return documentsList;
    }

it might be sufficient to change the logic to something like continue instead of break when a bad space is explored:

...
while (next) {
            const data = await this.get<ConfluenceDocumentList>(requestUrl);
            if (!data.results) {
                continue;
            }

            documentsList.push(...data.results.map(result => result._links.self));
...