theapsgroup / steampipe-plugin-gitlab Goto Github PK

not sure what diagnostic information to offer.

10:50:48 ❯ steampipe plugin install theapsgroup/gitlab                                                                       

panic: runtime error: index out of range [-1]

goroutine 14 [running]:
github.com/turbot/steampipe/cmd.doPluginInstall.func2(0xc000796ea0?)
	/home/runner/work/steampipe/steampipe/cmd/plugin.go:270 +0xb1
github.com/gosuri/uiprogress.(*Bar).Bytes(0xc00058c200)
	/home/runner/go/pkg/mod/github.com/gosuri/[email protected]/bar.go:200 +0x288
github.com/gosuri/uiprogress.(*Bar).String(...)
	/home/runner/go/pkg/mod/github.com/gosuri/[email protected]/bar.go:214
github.com/gosuri/uiprogress.(*Progress).print(0xc00070c8a0)
	/home/runner/go/pkg/mod/github.com/gosuri/[email protected]/progress.go:127 +0x96
github.com/gosuri/uiprogress.(*Progress).Listen(0xc00070c8a0)
	/home/runner/go/pkg/mod/github.com/gosuri/[email protected]/progress.go:114 +0x29
created by github.com/gosuri/uiprogress.(*Progress).Start
	/home/runner/go/pkg/mod/github.com/gosuri/[email protected]/progress.go:134 +0x56
~
10:50:49 ❯ steampipe -v                                                                                                     
steampipe version 0.14.4

❯ uname -a  
Darwin dredl0ck 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64 x86_64

❯ go version                                    
go version go1.18.2 darwin/amd64

No real experience with steampipe yet, so perhaps my setup isn't correct.

Currently the projects tables do not include the array of strings for associated topics, add these fields!

We require gitlab_notification_setting table for our compliance mod, please let us know if there are any plans to add this table.

Reference:
https://pkg.go.dev/github.com/xanzy/[email protected]#NotificationSettingsService.GetGlobalSettings

We require gitlab_page_domain table for our compliance mod, please let us know if there are any plans to add this table.

Reference:
https://pkg.go.dev/github.com/xanzy/[email protected]#PagesDomainsService.ListPagesDomains

There's a need to be able to obtain certain statistics about projects such as disk space used, etc - this is available in the API wrapper so should add it.

ListOptions has been updated to include:

// For keyset-based paginated result sets, name of the column by which to order
OrderBy string `url:"order_by,omitempty" json:"order_by,omitempty"`
// For keyset-based paginated result sets, the value must be `"keyset"`
Pagination string `url:"pagination,omitempty" json:"pagination,omitempty"`
// For keyset-based paginated result sets, sort order (`"asc"`` or `"desc"`)
Sort string `url:"sort,omitempty" json:"sort,omitempty"`

With a newer Steampipe SDK we can potentially push these down and reduce pagination calls?

Currently the plugin cannot be built as a standalone postgres FDW as it differs from the standard way how the plugin main.go and go.mod have to look like.

go.mod and main.go should be fixed accordingly so that one can use the steampipe-postgres-fdw Makefile process to build standalong postgres-FDW like so:

# get makefile and standlone FDW toolchain
git clone https://github.com/turbot/steampipe-postgres-fdw.git
cd steampipe-postgres-fdw

# build the plugin
# currently does not work. Will work after this fix
make standalone plugin=gitlab plugin_github_url=github.com/theapsgroup/steampipe-plugin-gitlab

See:

• https://turbot-community.slack.com/archives/C044P668806/p1719409368775559

Iterate available endpoints and create new tables as appropriate.

Add a table that links Repo Files to each repository, this will allow for searching files/file-content.

Needs to be limited, since files can be large and content is returned (base64 enc).

I'd like to be able to specify the branch to use for the gitlab_project_repository table, as right now it can only ever get the file tree from the default branch from a repository.

I see in the Gitlab API Wrapper library that there is a Ref option that can be used with the ListTree API call: https://github.com/xanzy/go-gitlab/blob/main/repositories.go#L54

I'm unsure if I'm running into a me problem or an issue with the plugin, sorry. Am I doing something wrong below?

Our group is on the GitLab Ultimate plan. That said, we're getting it as part of an OSS sponsorship. From my understanding, we should be able to access the epic object from the REST API.

I have some issues with known epics. Let's use this one as an example.

Note the issues high-line#18 and subway#1247.

When I go to query gitlab_issue.epic_issue_id, however, the epic_issue_id comes back as NULL.

Below you can see a few issues, including high-line#18 and subway#1247 coming back as NULL.

Typo on row 40 in docs/index.md
~./steampipe/config/gitlab.spc -> ~/.steampipe/config/gitlab.spc

Would it make sense to have this set by default? That would make those who have GITLAB_TOKEN already set in their environment for other reasons have an excellent OOB experience.

How would the SQL look?

This fails:

steampipe query "select * from gitlab_project where group_id=$MY_GRP_ID" --output csv
group_id ForeingKey is not there, what is the correct option?

1. Please add the goreleaser.yaml. You can copy the one from steampipe-plugin-vault, but edit id the line to say id: "steampipe-gitlab"
2. Please make sure you test against the cloud version of gitlab as well as your local version. The plugin should work with both, but the cloud version will reveal some multi-tenant considerations that you must address.
3. For instance, many APIs return ALL items by default. While this may be sufficient in a private instance, the scale of the shared cloud version will make paging through all the results unfeasible in any practical way. For example, select * from gitlab_project returns ALL the public projects, in addition to your own private projects. We have addressed this in the github plugin with the my pattern - you will see that we have split some apis into 2 tables, one which requires a qualifier but will return ANY items, and one that does not that only returns "your" resources. As an example, the github_repository table can return information about ANY public repo, but you must specify where full_name=, whereas the the github_my_repository does NOT require a qualifier, but only returns repos that you own or contribute to. Please try to use this pattern as well - let me know if you have questions.
4. Paging seems to be broken, at least against the cloud version. I did not test all tables, but select * from gitlab_project and select * from gitlab_user both return only the first 30 items.

We require gitlab_protected_branch table for our compliance mod, please let us know if there are any plans to add this table.

Reference:
https://pkg.go.dev/github.com/xanzy/[email protected]#ProtectedBranchesService.GetProtectedBranch

We require gitlab_settings table for our compliance mod, please let us know if there are any plans to add this table.

Reference:
https://pkg.go.dev/github.com/xanzy/[email protected]#SettingsService.GetSettings

Hi

I'm trying to query the gitlab_user table with users that are in a group: gitlab_group_member but I keep running into the same error. This is on gitlab.com:

Query 1 ERROR at Line 1: : ERROR:  rpc error: code = Unknown desc = when using the gitlab_user table with GitLab Cloud, `List` call requires an '=' qualifier for one or more of the following columns: 'id', 'username'

This are the queries I tried:

SELECT
	*
FROM
	gitlab_group_member AS m,
	gitlab_user AS u
WHERE
	u.id = m.id
	AND group_id = <group_id>;

SELECT
	*
FROM (
	SELECT
		id AS group_member_id
	FROM
		gitlab_group_member
	WHERE
		group_id = <group_id>) AS m, gitlab_user
WHERE
	id = group_member_id;

SELECT
	*
FROM
	gitlab_group_member AS m
	JOIN gitlab_user AS u ON u.id = m.id
WHERE
	group_id = <group_id>;

We require gitlab_group_hook table for our compliance mod, please let us know if there are any plans to add this table.

Reference:
https://pkg.go.dev/github.com/xanzy/[email protected]#GroupsService.ListGroupHooks

The query returns no data. No error message given. The project_id is correct, queries to other tables work fine for the same id.

Plugin version:

{
    "hub.steampipe.io/plugins/theapsgroup/gitlab@latest": {
      "name": "hub.steampipe.io/plugins/theapsgroup/gitlab@latest",
      "version": "0.4.0",
      "image_digest": "sha256:6aacfb9c115176d8b9b9919e8f70df790b646d24db7663cd156bbeb292463ee1",
      "binary_digest": "sha256:5a51dd4bea9469e018b0f94987b27bdca110f73d5697f9bd7e65a52aa0ad53a6",
      "binary_arch": "amd64",
      "installed_from": "us-docker.pkg.dev/steampipe/plugins/theapsgroup/gitlab:latest",
      "last_checked_date": "2023-05-17T10:28:52Z",
      "install_date": "2023-05-09T11:11:44Z",
      "imageDigest": "sha256:6aacfb9c115176d8b9b9919e8f70df790b646d24db7663cd156bbeb292463ee1",
      "installedFrom": "us-docker.pkg.dev/steampipe/plugins/theapsgroup/gitlab:latest",
      "lastCheckedDate": "2023-05-17T10:28:52Z",
      "installDate": "2023-05-09T11:11:44Z"
    }
}

Gitlab Version: 15.10.7-ee

Steampipe Version:

steampipe@3a6c42c71856:/workspace$ steampipe -v    
Steampipe v0.19.4

Query:

select
  *
from
  gitlab_project_job
where
  project_id = 912
limit 30;

To reproduce run the following query

> select
  path,runners_token
from
  gitlab_group

Using the same token I can access the runner registration token using the api so the problem appears to be with the gitlab plugin
curl --fail --header "Private-Token: ${GITLAB_TOKEN}" "https://$URL/api/v4/groups/$id"

Review documentation & improve consistency, review query examples and add some additional ones.

Also ensure the descriptions for tables/columns is more consistent.

It's been a while since we updated the version of https://github.com/xanzy/go-gitlab (current: 0.55.0, latest 0.77.0)

• Update to v0.77.0
• Remediate any issues caused to current tables/columns by updating

We require this column in gitlab_group table for our compliance mod, please find the reference below.

Reference:
https://github.com/xanzy/go-gitlab/blob/v0.55.0/groups.go#L719

When performing a query on project to get statistics all columns are null. However, when setting statistics=true on an API call to the same project id the expected results are returned. I looked and there are references to statistics where stats = true in the project. I don't see a connection option.

GitlabEE self hosted version 16.9.5-ee.

Example Steampipe query:
select id,name,repository_size,storage_size, lfs_objects_size from gitlab_project where id = 4766

+------+--------+-----------------+--------------+------------------+
| id   | name   | repository_size | storage_size | lfs_objects_size |
+------+--------+-----------------+--------------+------------------+
| 4766 | 24reen | <null>          | <null>       | <null>           |
+------+--------+-----------------+--------------+------------------+

curl ... "https://gitlab..../api/v4/projects/4766?statistics=true" | jq | grep size
"storage_size": 12271071469,
"repository_size": 668659,
"wiki_size": 0,
"lfs_objects_size": 12270402810,
"job_artifacts_size": 0,
"pipeline_artifacts_size": 0,
"packages_size": 0,
"snippets_size": 0,
"uploads_size": 0

If someone has the time, it would be awesome if you could take a look.

Attempting query

select * from gitlab_project

returns the following error.

[HV000] ERROR: rpc error: code = Internal desc = 'List' call for table 'gitlab_project' is missing 2 required quals: column:'owner_id' operator: = column:'owner_username' operator: =

Querying by known project_id works fine.

select * from gitlab_project where id=XXX

Am I missing a config somewhere? Updated to latest plugin (0.4.1)

Thanks !

• Verify Statistics in gitlab_project table.
• Verify ref column on gitlab_project_repository
• Upgrade Steampipe SDK Version
• Upgrade GitLab SDK Version
  • Group struct has EmailsDisabled=>EmailsEnabled & SharedRunnersEnabled replaced by a new struct SharedRunnersSetting
  • Verify Changes on underlying MergeRequest struct do not break tables.
  • Verify removal of Public (bool) field on Project struct does not break tables.
  • AuditEvent now have EventType property
  • Commit now has Author property
  • ContainerRegistry now has Status property
  • GroupMember now has MemberRole property

As part of periodic Steampipe maintenance all plugins should be upgraded to use SDK 1.8.2 as well as golang v1.17.

We require this column in gitlab_project table for our compliance mod, please find the reference below.

Reference:
https://github.com/xanzy/go-gitlab/blob/v0.55.0/projects.go#L39

Thanks so much for making this happen!

Any plans to add support for Epics and Iterations?

Hello, I'd like to be able to check that sensitive variables are set to Protected in our CICD pipelines in GitLab, e.g. to ensure that production AWS AK/SK aren't exposed to developers. However the current plugin doesn't expose CICD variables and their attributes.

Would it be possible to add GitLab CICD variables and their attributes to a Steampipe table?

Thanks,
Guillaume

I ran the following query against all projects that had issues with the foo label to see which ones did not have the bar/fluffy.json file. It errored out. It should not have.

Query for example:

SELECT
  p.full_path
FROM
  gitlab_issue AS i
  LEFT JOIN gitlab_project_repository_file AS f ON f.project_id = i.project_id AND f.file_path = 'bar/fluffy.json',
  gitlab_project AS p
WHERE
  i.labels ? 'foo' AND f.file_path IS NULL AND p.id = i.project_id
GROUP BY
  p.full_path;

What I expected was a list of p.full_path's where bar/fluffy.json did not exist. What I got was this error:

Error: unable to obtain repository file bar/fluffy.json for project_id 521 on ref main
GET https://lab.fluffy.cat/api/v4/projects/521/repository/files/bar/fluffy.json: 404 {message: 404 Commit Not Found} (SQLSTATE HV000)

I tried a similar query using the gitlab_project_repository table and got an error when a repository didn't exist which I also think should return an empty response. I'll have a PR of some sort for this.

Environment

Steampipe Cloud

To reproduce

1. Install the GitLab plugin from https://cloud.steampipe.io/user//create/connection
2. Query the gitlab_epic table. You do not need to include the due_date field that seems to cause the problem.
3. Observe error ERROR: rpc error: code = Unknown desc = interfaceToColumnValue failed for column 'due_date': cannot convert 2022-12-16 to a time.Time (SQLSTATE HV000)

Add fields for epic to gitlab_issue tables. see docs

"epic": {
      "id": 123456,
      "iid": 123,
      "title": "Some Title Here",
      "url": "/groups/my-group-name/-/epics/123",
      "group_id": 10000000,
      "human_readable_end_date": "Jan 16, 2099",
      "human_readable_timestamp": "some text"
    }

Upgrade to:
steampipe-plugin-sdk v3.1.0
Go version 1.18

The gitlab events api is useful for assessing the performance of an individual developer or a team or to compare projects.

I'd be interested in implementing this, but curious what people thing about how it should work. And if people would want it.

Option 1 would be to have a gitlab_events table and if user_id isn't specified in a where clause, just get the logged in user's events. If it is, get that/those user_id's events.

Option 2 would have two tables: gitlab_my_events for the logged in user and gitlab_user_events with a requirement that user_id must be specified in a where clause.

Option 2 seems idiomatically correct based on the existing tables.

Some tables don't currently have columns for all data returned, this seems wasteful as we may have a use for the data that is currently omitted.

• Example objects returned from API against table columns & add columns (or additional tables if necessary) to increase coverage of data.

There is a distinct lack of logging occurring in this plugin, at the very minimum errors should be output to logs, but it would be nicer to have more detailed and consistent logging.