go get becomes a deprecated way to install binaries https://golang.org/doc/go-get-install-deprecation, but go install fails with error:

go install github.com/augmentable-dev/askgit@latest: github.com/augmentable-dev/[email protected]
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

For listing the reviews of a GitHub pull request (PullRequestReview in the GraphQL API)

As mentioned in: #207 (comment), it would be valuable to have a way to list the available tables (table-valued-functions) provided by askgit and have the method documented. This PRAGMA could be a simple option, but it mixes tables provided by this project and other modules (the JSON functions for example).

It may be worth doing this at the application level and offering a subcommand or "built-in" view/table that lists the askgit tables.

Hi

Interesting project! Currently trying to run https://docs.mergestat.com/miscellaneous/cloning-all-org-repos against github enterprise, but failing on auth. Is there any way to set the custom github enterprise url so I can auth to that instead of github.com?

Is it expected that the basic command from README is so heavy? I initially thought that there's something wrong with my invocation, I did this:

docker run --rm -v `pwd`:/repo:ro augmentable/askgit "SELECT * FROM commits"

Then my computer just seemed to be stuck. I was seeing resource usage like this for over a minute:

Then it eventually finished after about 2.5 minutes but I was seriously worried that I'm doing something wrong, e.g., not escaping the SQL query correctly.

What does it do on the first run? Is it building some sort of database behind the scenes? Would even "simpler" queries like SELECT count(*) FROM commits take similarly long?

We can list commits from a repository on disk, but not via the GitHub API. We should implement a github_repo_commits table that can do this.

Upgrading askgitdev/askgit/askgit with Homebrew on Linux (Ubuntu 20.04) is failing:

> uname -srm
Linux 5.14.11-051411-generic x86_64

> lsb_release -d
Description:    Ubuntu 20.04.3 LTS

> brew outdated
askgitdev/askgit/askgit (v0.4.7) < v0.4.8

> brew upgrade
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/cask).
==> Updated Casks
Updated 1 cask.

Updating Homebrew...
==> Upgrading 1 outdated package:
askgitdev/askgit/askgit v0.4.7 -> v0.4.8
==> Downloading https://github.com/askgitdev/askgit/archive/v0.4.8.tar.gz
Already downloaded: /home/giermulnik/.cache/Homebrew/downloads/bc83f30eb7ec1aa03e0e8e020c5cd9006e5ccb1da98eb05d36d61777e2d14864--askgit-0.4.8.tar.gz
==> Upgrading askgitdev/askgit/askgit
  v0.4.7 -> v0.4.8

==> make
Last 15 lines from /home/giermulnik/.cache/Homebrew/Logs/askgit/01.make:

-- nuking .build/
-- building .build/libaskgit.so
-- building .build/askgit
# github.com/libgit2/git2go/v32
/home/giermulnik/.cache/Homebrew/go_mod_cache/pkg/mod/github.com/libgit2/git2go/[email protected]/Build_system_dynamic.go:12:3: error: #error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
   12 | # error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
      |   ^~~~~
# github.com/libgit2/git2go/v32
/home/giermulnik/.cache/Homebrew/go_mod_cache/pkg/mod/github.com/libgit2/git2go/[email protected]/Build_system_static.go:12:3: error: #error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
   12 | # error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
      |   ^~~~~
make: *** [Makefile:17: .build/libaskgit.so] Error 2
make: *** Waiting for unfinished jobs....
make: *** [Makefile:23: .build/askgit] Error 2

If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
  https://github.com/askgitdev/homebrew-askgit/issues

> brew info libgit2
libgit2: stable 1.3.0 (bottled), HEAD
C library of Git core methods that is re-entrant and linkable
https://libgit2.github.com/
/home/linuxbrew/.linuxbrew/Cellar/libgit2/1.3.0 (102 files, 4.8MB) *
  Poured from bottle on 2021-10-01 at 00:16:35
From: https://github.com/Homebrew/linuxbrew-core/blob/HEAD/Formula/libgit2.rb
License: GPL-2.0-only
==> Dependencies
Build: cmake ✔, pkg-config ✔
Required: libssh2 ✔
==> Options
--HEAD
        Install HEAD version
==> Analytics
install: 1,052 (30 days), 1,994 (90 days), 6,492 (365 days)
install-on-request: 140 (30 days), 193 (90 days), 598 (365 days)
build-error: 0 (30 days)

Build log as follows:

simon@mason:~/tmp$ go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit
github.com/augmentable-dev/askgit (download)
github.com/go-git/go-git (download)
github.com/go-git/go-billy (download)
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
github.com/go-git/gcfg (download)
Fetching https://gopkg.in/warnings.v0?go-get=1
Parsing meta tags from https://gopkg.in/warnings.v0?go-get=1 (status code 200)
get "gopkg.in/warnings.v0": found meta tag get.metaImport{Prefix:"gopkg.in/warnings.v0", VCS:"git", RepoRoot:"https://gopkg.in/warnings.v0"} at https://gopkg.in/warnings.v0?go-get=1
gopkg.in/warnings.v0 (download)
github.com/mitchellh/go-homedir (download)
github.com/jbenet/go-context (download)
Fetching https://golang.org/x/net/context?go-get=1
Parsing meta tags from https://golang.org/x/net/context?go-get=1 (status code 200)
get "golang.org/x/net/context": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/context?go-get=1
get "golang.org/x/net/context": verifying non-authoritative meta tag
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
golang.org/x/net (download)
github.com/emirpasic/gods (download)
github.com/sergi/go-diff (download)
Fetching https://golang.org/x/crypto/openpgp?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp?go-get=1 (status code 200)
get "golang.org/x/crypto/openpgp": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp?go-get=1
get "golang.org/x/crypto/openpgp": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
github.com/kevinburke/ssh_config (download)
github.com/xanzy/ssh-agent (download)
Fetching https://golang.org/x/crypto/ssh?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh?go-get=1
get "golang.org/x/crypto/ssh": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/ssh/knownhosts?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh/knownhosts?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh/knownhosts": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh/knownhosts?go-get=1
get "golang.org/x/crypto/ssh/knownhosts": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/proxy?go-get=1
Parsing meta tags from https://golang.org/x/net/proxy?go-get=1 (status code 200)
get "golang.org/x/net/proxy": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/proxy?go-get=1
get "golang.org/x/net/proxy": verifying non-authoritative meta tag
github.com/imdario/mergo (download)
github.com/mattn/go-sqlite3 (download)
github.com/gitsight/go-vcsurl (download)
github.com/olekukonko/tablewriter (download)
github.com/mattn/go-runewidth (download)
github.com/spf13/cobra (download)
github.com/spf13/pflag (download)
../go/src/github.com/go-git/go-git/remote.go:9:2: code in directory /home/simon/go/src/github.com/go-git/go-billy/osfs expects import "github.com/go-git/go-billy/v5/osfs"

On investigation, https://github.com/go-git/go-billy/v5/osfs does not exist but https://github.com/go-git/go-billy/osfs does. Suggest this is bit-rot caused by the upstream package changing its directory structure?

It'd be nice if we could just run docker run -v pwd:/repo:ro gitqlite "SELECT * FROM commits" without needing to clone and build the docker image locally.

We've had a user request for a github_branches table-valued function for returning a list of branches on a GitHub repository and associated metadata, something like:

SELECT * FROM github_branches('mergestat', 'mergestat')

The associated query would probably look something like this (with more columns selected):

query { 
  repository(owner:"mergestat", name:"mergestat") {
    refs(refPrefix: "refs/heads/", first: 100) {
      nodes {
        id, name, target {
          ...on Commit {
            oid
          }
        }
      }
    }
  }
}

Note that refs can also be accessed via the git functions by interacting with a repo directly (either already on disk or executing a clone to a tmp directory)

SELECT * FROM refs('https://github.com/mergestat/mergestat')

would be somewhat equivalent, just querying the git objects directly vs going through the GitHub API

Could you upgrade to the latest libgit2? With the current one I have issues of missing symbols

Currently, the --format json flag outputs line delimited json (https://jsonlines.org/) to the CLI for better streaming support (results are printed as they arrive).

We should add support for a "regular" JSON output (better name?) that outputs a more standard JSON array:

[
 {"col1": "hello, world", "col2": 123},
 {"col1": "hello, world!", "col2": 124},
  ...
]

Maybe it's worth renaming current json out put to ndjson and use json for "regular" json output? It would break backwards compatibility - but I think that should be okay

would be neat!

Like

…but for GitLab

请教一下，SQL语法的解析如何实现的？又是如何和git查询接口对接上的呢

Currently, it's pretty easy to hit a GitHub rate limit when running queries that use the GitHub API tables. We could consider implementing a back-off-retry strategy to pause execution when we encounter one, wait an appropriate amount of time, and retry/continue

The locator package currently only specifies an HTTP locator, we should add support for an SSH one as well: https://github.com/askgitdev/askgit/blob/main/pkg/locator/locator.go#L58

When no 'Default repository' in the 'Query Settings' is set, the error message for

SELECT count(*) FROM commits('')

is problem executing query: failed to open "/var/task": must be an http(s) repo

The error message could indicate that this might be the case because no default repo was set.

—--—-—----

The greyed out placeholder text in the settings dialog makes it seem like the MergeStat repo is the default always (unless overwritten)

I'm running OSX Catalina with libgit2 v1.1.0:

$ brew info --installed --json | jq '.[] | select(.name == "libgit2") | .versions'
{
  "stable": "1.1.0",
  "head": "HEAD",
  "bottle": true
}

I cannot build askgit due to a git2go version incompatibility with my libgit2 version:

$ make build
go build -v -tags="sqlite_vtable,static,system_libgit2" askgit.go
github.com/augmentable-dev/askgit/pkg/gitlog
github.com/libgit2/git2go/v30
# github.com/libgit2/git2go/v30
../../../../go/pkg/mod/github.com/libgit2/git2go/[email protected]/git_system_static.go:11:3: error: "Invalid libgit2 version; this git2go supports libgit2 v1.0"
# error "Invalid libgit2 version; this git2go supports libgit2 v1.0"
  ^
1 error generated.
make: *** [build] Error 1

The git2go module has very strict libgit2 version dependencies: https://github.com/libgit2/git2go#which-go-version-to-use

I was able to successfully build askgit by replacing github.com/libgit2/git2go/v30 with github.com/libgit2/git2go/v31, but I imagine this would break things for those with older libgit2 versions.

Some functionality, like the GitHub functions, requires a configuration value (GITHUB_TOKEN) set via an env variable. It's a bit clumsy to specify this every time (GITHUB_TOKEN=123 askgit "SELECT ....") and would be great if there were a way to store the token (or any other config value) in a "cache" or managed config area on disk, so that a user doesn't need to specify it on every run of the command

I've been trying to work out how to filter file paths affected by a specific commit. Or in other words, show me the files modified by a particular commit, or maybe all the files affected when a particular merge happens.

As far as I can tell, each commit hash in the files table has a record for every file in the tree at that time. I'm struggling to see where in the data model the file paths changed in each commit might be available.

Thanks!

#136 re-adds support for GitHub tables again. We should add some tests to validate the behavior in these tables, ideally ones that don't make actual calls to the GitHub API as was done before (let's mock the responses as we should)

See here for context. It would be useful to be able use mappings in a .mailmap of a repo to de-duplicate authors in queries.

I'm not entirely sure how we add support for it - maybe as a helper function that takes the contents of a .mailmap and an email address, and returns the associated name.

Something like SELECT mailmap(<mailmap-contents>, '[email protected]')

Would be nice to have PAGER support so that the produced table results are more easily browseable.

Most SQL command line clients support this. I could pipe the results into less -SinFX, but would be nice if I didn't have to on each query.

Currently, GitHub API tables make GraphQL requests to the GitHub API (using this library). However, we fetch all columns for a table, even if they are unused in the query. This makes it easy to hit the GitHub "secondary rate limit," which is based on query complexity (fetching more items from the GraphQL API is more expensive). It also makes requests with unused data slower than they need to be, and skips over a big purpose of GraphQL (being able to selectively choose which fields to retrieve).

We should investigate how to only fetch the columns/fields we need for these API requests. We should be able to access the colUsed field from the SQLite virtual table interface as well.

I don't get that query:

SELECT count(*) AS commits, SUM(additions) AS additions, SUM(deletions) AS deletions, author_email FROM commits GROUP BY author_email ORDER BY commits

According to the docs there is no additions and deletions column for the commits table.
Are they missing in the docs?

See this page for more information. Basically, it would be useful if the commits table could allow for something like:

SELECT * FROM commits('some-repo', 'v4.0.0..v5.0.0')

to select only the commits that occurred after v4.0.0 up to v5.0.0

See this issue for some context as well

See #256

Total PRs Opened
Total PRs Merged
Total PRs Closed
Total Authors
Avg. Comment Count
Avg. Commit Count
Avg. Files Modified
Avg. Lines added/removed (maybe all these on one line?)
Avg. Time to merge

Breakdown by Author (GitHub login)
Login
Total PRs opened by author in period
Total PRs merged in period (belonging to author)
Total PRs remaining open in period (and avg age?)
Avg time to merge

it would be nice to mention libgit2 requirement in README

My environment: macOS Big Sur, go1.16.5 darwin/amd64

What do you think of the idea of the parsed .mailmap file from the repos (if there is one) being represented by a queryable table in MergeStat?

This would allow constructing a query to identify all email addresses of contributors that have not been assigned a canonical name through the .mailmap file.

See here: go-git/go-git#140 and here: go-git/go-git#228

We should be able to support the GIT_SSL_NO_VERIFY in the CLI so that users can clone HTTPs repos with self-signed certs

in mysql, we can use "show tables" ,"desc table_name".
really looking forward meet this feature

A table-valued GitHub API function for listing commits in a pull request, something like github_pr_commits that return all the commits in a PR, via the GitHub API

When trying to install by running this:

go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit

I get this:

➜ go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit

github.com/libgit2/git2go (download)
cannot find package "github.com/libgit2/git2go/v30" in any of:
	/usr/local/go/src/github.com/libgit2/git2go/v30 (from $GOROOT)
	/home/duncan/.go/src/github.com/libgit2/git2go/v30 (from $GOPATH)

My environment is as follows:

 ➜ go version
go version go1.15.5 linux/amd64

➜ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/duncan/.cache/go-build"
GOENV="/home/duncan/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/duncan/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/duncan/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build228718343=/tmp/go-build -gno-record-gcc-switches"

 ➜ neofetch --backend off

OS: Ubuntu 20.04.1 LTS x86_64 
Kernel: 5.4.0-52-generic 
Shell: bash 5.0.17 
DE: Xfce 
Memory: 38419MiB / 64206MiB 

The docker image we reference in the README is very out of date: https://hub.docker.com/r/augmentable/askgit (and belongs to an old, no-longer relevant org). We should:

• Update the image we publish on Docker Hub
• Update the README to use the correct/latest image
• Setup a pipeline to publish docker images automatically so that they remain up to date

This was "discovered" in #207, thank you @thcipriani

https://github.com/mergestat/mergestat/blob/main/cmd/summarize/commits/commits.go#L63-L70

It's possible for author_name / author_email to be null, in which case we have a Scan error - should use sql.NullString in these cases

I was wondering where in the repo is the SQL getting parsed? How is it being parse?

We should allow a user to supply OWNER, COLLABORATOR or ORGANIZATION_MEMBER association values to the github_org_repos and github_user_repos tables as an optional argument.

$ go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit
go/src/github.com/go-git/go-git/remote.go:9:2: code in directory /home/federico/go/src/github.com/go-git/go-billy/osfs expects import "github.com/go-git/go-billy/v5/osfs"

See this issue for additional context: mergestat/homebrew-mergestat#20

Hi to all,
imagine I have repo in which I update a txt file day by day.

Is there a way to have the version of this file on 2021-12-21?

A query like SELECT * FROM myFile.txt AS OF TIMESTAMP('2021-12-21'); that give me in output that file at that date?

Thank you

Hey team, thank you for providing this awesome tool which helps me a lot traversing the data in git repos. I'm wondering that do we have any plan to add binaries in the GitHub releases (for example, on GitHub Actions)? It seems not so friendly to the new users who have to build from source to install and try.

Info about the recently implemented branches and tags tables should be added to the documentation in the README, similar to what's done for commits and files

askgit: error while loading shared libraries: libgit2.so.1.1

I'm one of the package maintainers for Arch Linux and I also maintain a few packages on the AUR, which mergestat can be found on.

Just wondering if there's any interest in RELRO/PIE being applied to the binary & shared library?

I generally try and apply these to all the Go-related packages that I maintain due to our Go package guidelines. I've found that mergestat seems to be working fine with these applied, as per this commit.

It would be nice to release this as a downloadable set of binaries. I have had some experience with GoReleaser and I have to say it's a pretty nice little tool, especially if you're working in Go. I'd prefer to be able to just brew install gitqlite, which GoReleaser has good support for: https://goreleaser.com/customization/homebrew/

Similar to output for mergestat summarize commits and mergestat summarize blame (in format), a mergestat summarize repo-issues owner/repo command:

• Total issues opened
• Total issues closed
• Total authors
• Average time to close
• Average participant count
• Average comment count

Breakdown by Author (GitHub login)

• Login
• Total Issues opened by author in period
• Issues remaining open in period (and avg age?)
• Issues closed in period
• Average time to close

Fetching issues can take time, by default only show a limited time frame (maybe the last 6 months or year)

Is there someway to update for example to committer email address etc or a commit message using SQL.

I'm not very familiar with Go, so perhaps I'm doing something wrong?

$ go install -v -tags=sqlite_vtable github.com/augmentable-dev/gitqlite
can't load package: package github.com/augmentable-dev/gitqlite: cannot find package "github.com/augmentable-dev/gitqlite" in any of:
        /usr/lib/go-1.10/src/github.com/augmentable-dev/gitqlite (from $GOROOT)
        /home/erez/go/src/github.com/augmentable-dev/gitqlite (from $GOPATH)

askgit has trouble with repository directories that contain characters that are either special to Go's %q string encoding or special to sqlite. Some characters cause askgit to exit with "unrecognized token" trying to create the virtual table, while others make it further and fail (or produce no data) when executing SQL statements.

Some special characters do not render well on github, so I've included equivalent shell commands for making these directories.

The following directories behave the same. select count(*) from commits returns no results (not the number 0 - it returns an empty resultset), while select count(*) from files panics with "panic: invalid handle":

• back\slash (mkdir 'back\slash')
• thing� (mkdir 'thing'$'\357\267\220')
• new line (mkdir 'new'$'\n''line')
• doublequotes");--injection (mkdir 'doublequotes");--injection')

The following directories all fail without running the SQL query, with an error like unrecognized token: "");":

• quotation"marks (mkdir 'quotation"marks')
• comma",separated" (mkdir 'comma",separated"')

This seems to be caused by building a SQL string using fmt.Sprintf and %q, which quotes/escapes strings in a way that Go understands rather than in a way that sqlite understands.
Go will format " in the middle of a string as \", which Sqlite considers to be a literal backslash character followed by the end of a string, which is why most directories with the double quote character result in "unrecognized token".
For other characters it seems like Go will escape them (e.g. \ becomes \\, newline becomes \n), and Sqlite happily passes the escaped versions to the modules' Create functions, which presumably try and fail to open a directory named e.g. back\\slash instead of back\slash

This can be reproduced in the tests by changing the fixture repo from "repo" to e.g. "repo\\" or "repo\""