phillipj / gitlab-search Goto Github PK

Command line tool to search for contents in GitLab repositories

License: MIT License

JavaScript 0.93% Reason 99.07%

gitlab-search's Introduction

GitLab Search

This is a command line tool that allows you to search for contents across all your GitLab repositories. That's something GitLab doesn't provide out of the box for non-enterprise users, but is extremely valuable when needed.

Prerequisites

Install Node.js
Create a personal GitLab access token with the read_api scope.

Installation

$ npm install -g gitlab-search

To finish the installation you need to configure the personal access token you've created previously:

$ gitlab-search setup <your personal access token>

That will create a .gitlabsearchrc file in the current directory. That configuration file can be placed in different places on your machine, valid locations are described in the rc package's README. You can decide where that file is saved when invoking the setup command, see more details in its help:

$ gitlab-search setup --help

Usage

Searching through all the repositories you've got access to:

$ gitlab-search [options] [command] <search-term>

Options:
  -V, --version                            output the version number
  -g, --groups <group-names>               group(s) to find repositories in (separated with comma)
  -f, --filename <filename>                only search for contents in given a file, glob matching with wildcards (*)
  -e, --extension <file-extension>         only search for contents in files with given extension
  -p, --path <path>                        only search in files in the given path
  -a, --archive [all,only,exclude]         search only in archived projects, exclude archived projects, search in all projects (default is all)
  -h, --help                               output usage information

Commands:
  setup [options] <personal-access-token>  create configuration file

Use with Self-Managed GitLab

To search a self-hosted installation of GitLab, setup has options for, among other things, setting a custom domain:

$ gitlab-search setup --help

Usage: setup [options] <personal-access-token>

create configuration file

Options:
  --ignore-ssl            ignore invalid SSL certificate from the GitLab API server
  --api-domain <name>     domain name or root URL of GitLab API server,
                          specify root URL (without trailing slash) to use HTTP instead of HTTPS (default: "gitlab.com")
  --dir <path>            path to directory to save configuration file in (default: ".")
  --concurrency <number>  limit the amount of concurrent HTTPS requests sent to GitLab when searching,
                          useful when *many* projects are hosted on a small GitLab instance
                          to avoid overwhelming the instance resulting in 502 errors (default: 25)
  -h, --help              display help for command

Debugging

If something seems fishy or you're just curious what gitlab-search does under the hood, enabling debug logging helps:

$ DEBUG=1 gitlab-search here-is-my-search-term
Requesting: GET https://gitlab.com/api/v4/groups?per_page=100
Using groups: name-of-group1, name-of-group2
Requesting: GET https://gitlab.com/api/v4/groups/42/projects?per_page=100
Requesting: GET https://gitlab.com/api/v4/groups/1337/projects?per_page=100
Using projects: hello-world, my-awesome-website.com
Requesting: GET https://gitlab.com/api/v4/projects/666/search?scope=blobs&search=here-is-my-search-term
Requesting: GET https://gitlab.com/api/v4/projects/999/search?scope=blobs&search=here-is-my-search-term

License

MIT

gitlab-search's People

Contributors

Stargazers

Watchers

gitlab-search's Issues

Is it possible to search all branches?

This might be a question about what's possible with Gitlab's API or it might be a feature request, I'm not sure...

Is it possible to search all branches? I think currently the results are only based on the master branch?

Personal repositories not included

Repositories that isn't related to a group, but rather just put directly on a GitLab account, is currently not included when searching through repositories for contents.

Some repositories missing from results

Hi,

Description of problem
I'm scanning my repos for part of a htaccess rule using:
gitlab-search "download.php?file" -f ".htaccess"

Expected result
I expected to get about 5 different results but only got 3.

I manually verified the missing repos did have the string in a htaccess file.

Notes
All of the repos are owned by the group the dev team are a part of (no personal repos).

Comparing a repo that is found vs one that isn't I can't see any standout differences in the settings.

Thanks for the great little tool by the way!
( grumbles about how Gitlab needs global search )

Should search in wiki page also.

It does not searching in wiki page. Please add this feature to searching in wiki page.

Present results in a more compact way

Hello (first of all thanks for this nice little tool!)
In large repositories search results list is very very long.
It would be good to be able to configure the way results are presented, so that they are more 'compact':

Only repository names (+maybe number of hits?)
Repository names + general file URLs (one per file, regardless of how many occurences)
Repository names + urls to specific lines
Repository names + line urls + single line of usage (no surrounding context)

Kind regards,
Bartosz

Iterate over multiple GitLab domains

Hey @phillipj, thanks for the nifty tool!

It would be great that GitLab Search handles more than one GitLab instance, either by a mechanism of contexts, or simply by allowing the user to add as many keys and domains as needed.

TypeError: domain.enter is not a function

I tried setting this up for a custom domain with the following command:

gitlab-search setup --domain "gitlab.imat.us" "MY_API_TOKEN"

and received the following error:

events.js:176
    domain.enter();
           ^

TypeError: domain.enter is not a function
    at Command.emit (events.js:176:12)
    at Command.module.exports.350.Command.parseOptions (/usr/lib/node_modules/gitlab-search/dist/index.js:5528:14)
    at Command.listener (/usr/lib/node_modules/gitlab-search/dist/index.js:5077:23)
    at emitTwo (events.js:106:13)
    at Command.emit (events.js:191:7)
    at Command.module.exports.350.Command.parseArgs (/usr/lib/node_modules/gitlab-search/dist/index.js:5449:12)
    at Command.module.exports.350.Command.parse (/usr/lib/node_modules/gitlab-search/dist/index.js:5272:21)
    at Object.935 (/usr/lib/node_modules/gitlab-search/dist/index.js:14851:9)
    at __webpack_require__ (/usr/lib/node_modules/gitlab-search/dist/index.js:22:30)
    at startup (/usr/lib/node_modules/gitlab-search/dist/index.js:36:19)

I was able to work around this by adding "domain": "gitlab.imat.us" to the .gitlabsearchrc file manually. Everything seemed to work as I would have expected after that.

Thank you

Your project saved me a bunch of time. Thank you.

Add setup option to allow invalid certificates

While having an on-premise hosted GitLab server, some organisations out in the wild might be running self signed not-verifiable TLS certificates for whatever reason. Currently this tools explodes badly when trying to communicate with such a server.

Being able to explicitly configure "ignore TLS certificate" upon setting up this tool, would avoid that mentioned explosion.

npm install fails

> [email protected] prepublish
> npm run build


> [email protected] build
> rimraf dist && bsb -make-world && ncc build lib/js/src/Main.bs.js --out dist

bsb: no work to do.
bsb: no work to do.
bsb: no work to do.
    at runCmd (evalmachine.<anonymous>:1:47392)
    at Object.819 (evalmachine.<anonymous>:1:44264)
    at __webpack_require__ (evalmachine.<anonymous>:1:216)
    at startup (evalmachine.<anonymous>:1:353)
    at module.exports.8 (evalmachine.<anonymous>:1:385)
    at evalmachine.<anonymous>:1:395 {
  opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
  library: 'digital envelope routines',
  reason: 'unsupported',
  code: 'ERR_OSSL_EVP_UNSUPPORTED'
}

Anyone up for a Typescript rewrite?

Seeing this tool still attracts attention, people actively starring and forking.

It's a shame the project is more or less stalled. I personally have a hard time making changes to the project, since it's written in a language & stack I havent touched for many years. This project was created as a fun learning experiment to get hands-on with ReasonML.

To me the ReasonML barrier seems like a logical reason for others to also have a hard time contributing with fixes & new features, which is being asked for in issues...

Sooooo: TypeScript is eating the world. Why not consider a rewrite? The project it certainly small enough, shouldnt take an -enormous- effort.

Would any be keen on giving it a shot?

node version

what is the minimum version of node for this project?

JSON output

Hi,
I recently discovered your project and it does exactly what I'm looking for :)
I'm wondering if a JSON output is something you are considering ?
It would be great when you use your work as (kind of) API.

A way to send config config via command line as well, but this is secondary (creating/removing .gitlabsearchrc on the fly)

Personal token starting with "-" doesn't work

Just tried running setup with a personal access token starting with a minus (-):

$ gitlab-search setup -superSecretToken
error: missing required argument 'personal-access-token'

Doesn't seem to recognise that token argument if it starts with a minus, which obviously some GitLab tokens do.

This might be affect by my use of zsh.

Output archived-state of projects

As we've got a lot of archived projects, and they are often not relevant in searches, it would be really helpful to get a hint in the output if a project is archived. Or even better would be a parameter to restrict the search to projects that haven't been archived.

Ability to query GitLab API which is accessible by http rather than https

Our privately hosted GitLab instance is accessible only with http://gitlab_domain/api/v4 but gitlab-search always uses https:// prefix when constructing URL. Would be nice if this could be configurable or overridden when --api-domain option includes protocol as well, i.e,
gitlab-search setup mytoken --api-domain http://my_git_domain.

DEBUG get 401 err

GitLab Rate Limit

The GitLab.com docs (https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits) provide a 10 request per minute rate limit for "Advanced, project, and group search API (for a given IP address)"

So, user-beware. If you have lots of repos/groups/projects whatever they're called, you're gonna have a bad time.

FR: Subgroup search

Today i found this nice utility to help search my gitlab instance.

However there seems to be a limitation that i can only search through projects directly below a given group a.
So i would get results from:
a / my-project
a / another project
However what i don't get is results from:
a / sub-group / awesome-project

Is this intended behaviour? I would love to see two things here:

recursive search when specifying a group
sub-group search (gitlab-search -g a/b my-search-term)

Prebuild .re to .js before publishing version

Currently all users of this tool has to install the entire Bucklescript platform of dependencies to build ReasonML source code to JavaScript for Node.js to execute.

Initially the project did prebuild the local ReasonML files in ./src to JavaScript files in ./lib/js. But that didn't work out when users installed this command line tool, because all the ReasonML/Bucklescript dependencies also had to be built. Therefore the easiest possible way was chosen just to get this tool out the door; build it on the end-user's computer.

Recently discovered that @zeit/ncc could be used to build everything into one JavaScript file with all dependencies included. At a distance that sounds like the perfect solution to avoid end-users having to download a ton of stuff, and rather just get a single file of prebuilt JavaScript from npmjs.com.

Group name should be part of search results

Currently search results only contains the respective repository name. That should also be prefixed with the group that repository belongs to.

Add LICENSE

Thanks for creating this project!

I'd suggest adding a license, MIT or Apache.

groups does not work

The Parameter 'groups' has no influence at the behaviour - it's still searching through all groups (api-level 4) :(

crash when no rights to crawl the whole tree

Context:

On a self hosted gitlab (15.9.3), consider this tree:

/public_group/
/private_group/sub_private_group1/sub_deep_private_group1/
/private_group/sub_private_group2/

I get granted access to sub_deep_private_group1.

I search something, with debug mode activated.

Bug:

Something exploded! Error: Request failed with status code 401

I don't know why, but the very first GET /api/v4/groups?per_page=100 we can see on debug mode return some groups I don't have access to (in my example it will return public_group private_group sub_private_group1 sub_deep_private_group1).

So when the next GET try to get the projects on all these groups, that ends with a 401 or 404 on inaccessible groups (private_group and sub_private_group1) and makes gitlab-search crash.

Expected:

No crash and search results! And either ignore these inaccessible groups or display a warning.

Because as it is right now I can't use this tool anymore :(

force port 443

Hello,

I cannot force on port 443, the first request uses the HTTPS protocol on the default port, but from the second request it switches to port 80

$ cat .gitlabsearchrc 
{"token":"XXXXXXXXXXXXX","domain":"https://gitlab.intranet.mycompany.fr","ignoreSSL":true}

$ DEBUG=1 gitlab-search here-is-my-search-term
Requesting: GET https://gitlab.intranet.mycompany.fr/api/v4/groups?per_page=100
Requesting: GET https://gitlab.intranet.mycompany.fr:80/api/v4/groups?order_by=name&owned=false&page=2&per_page=100&sort=asc&statistics=false&with_custom_attributes=false
Something exploded! Error: write EPROTO 139896227686272:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)

$ cat .gitlabsearchrc 
{"token":"XXXXXXXXXXXXX","domain":"https://gitlab.intranet.mycompany.fr:443"}

$ DEBUG=1 gitlab-search here-is-my-search-term
Requesting: GET https://gitlab.intranet.mycompany.fr:443/api/v4/groups?per_page=100
Requesting: GET https://gitlab.intranet.mycompany.fr:80/api/v4/groups?order_by=name&owned=false&page=2&per_page=100&sort=asc&statistics=false&with_custom_attributes=false
Something exploded! Error: write EPROTO 140273917437824:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)

Best Regards
Merouane

Not getting full results

We have an internal GitLab CE server (11.10.4). Our repos are set up in group\subgroup\projects for the most part.

When I search for words, sometimes I get no results back (even though I can go to the repo with that keyword, use the UI search, and get multiple responses), or I get some results back (from one directory tree, but not others).

The projects that return results appear to be consistent between each run, but I'm not seeing a pattern between which ones return and which do not. The account I used to generate the API token is a maintainer on all of the repos, but not a site admin.

proxy

not respect npm config set proxy http://127.0.0.1:8118