storj / dbx Goto Github PK

View Code? Open in Web Editor NEW

24.0 24.0 3.0 2.26 MB

A neat codegen-based database wrapper written in Go

License: Apache License 2.0

Go 100.00%

dbx's Introduction

Storj V3 Network

Storj is building a distributed cloud storage network. Check out our white paper for more info!

Storj is an S3-compatible platform and suite of distributed applications that allows you to store data in a secure and distributed manner. Your files are encrypted, broken into little pieces and stored in a global distributed network of computers. Luckily, we also support allowing you (and only you) to retrieve those files!

Contributing
Start using Storj
License
Support

Contributing to Storj

All of our code for Storj v3 is open source. If anything feels off, or if you feel that some functionality is missing, please check out the contributing page. There you will find instructions for sharing your feedback, building the tool locally, and submitting pull requests to the project.

A Note about Versioning

While we are practicing semantic versioning for our client libraries such as uplink, we are not practicing semantic versioning in this repo, as we do not intend for it to be used via Go modules. We may have backwards-incompatible changes between minor and patch releases in this repo.

Start using Storj

Our wiki has documentation and tutorials. Check out these three tutorials:

License

This repository is currently licensed with the AGPLv3 license.

For code released under the AGPLv3, we request that contributors sign our updated Contributor License Agreement (CLA) v2 so that we can relicense the code under Apache v2, or other licenses in the future.

Support

If you have any questions or suggestions please reach out to us on our community forum or file a ticket at https://support.storj.io/.

dbx's People

Contributors

Stargazers

Watchers

Forkers

refaqtor haifenghuang erikvv

dbx's Issues

use generics to reduce query boilerplate

The generated code has a significant amount of boilerplate which can be reduced by using generics.

For example:

	defer mon.Task()(&ctx)(&err)

	var __embed_stmt = __sqlbundle_Literal("SELECT bucket_storage_tallies.bucket_name, bucket_storage_tallies.project_id, bucket_storage_tallies.interval_start, bucket_storage_tallies.total_bytes, bucket_storage_tallies.inline, bucket_storage_tallies.remote, bucket_storage_tallies.total_segments_count, bucket_storage_tallies.remote_segments_count, bucket_storage_tallies.inline_segments_count, bucket_storage_tallies.object_count, bucket_storage_tallies.metadata_size FROM bucket_storage_tallies ORDER BY bucket_storage_tallies.interval_start DESC")

	var __values []interface{}

	var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
	obj.logStmt(__stmt, __values...)

	for {
		rows, err = func() (rows []*BucketStorageTally, err error) {
			__rows, err := obj.driver.QueryContext(ctx, __stmt, __values...)
			if err != nil {
				return nil, err
			}
			defer __rows.Close()

			for __rows.Next() {
				bucket_storage_tally := &BucketStorageTally{}
				err = __rows.Scan(&bucket_storage_tally.BucketName, &bucket_storage_tally.ProjectId, &bucket_storage_tally.IntervalStart, &bucket_storage_tally.TotalBytes, &bucket_storage_tally.Inline, &bucket_storage_tally.Remote, &bucket_storage_tally.TotalSegmentsCount, &bucket_storage_tally.RemoteSegmentsCount, &bucket_storage_tally.InlineSegmentsCount, &bucket_storage_tally.ObjectCount, &bucket_storage_tally.MetadataSize)
				if err != nil {
					return nil, err
				}
				rows = append(rows, bucket_storage_tally)
			}
			if err := __rows.Err(); err != nil {
				return nil, err
			}
			return rows, nil
		}()
		if err != nil {
			if obj.shouldRetry(err) {
				continue
			}
			return nil, obj.makeErr(err)
		}
		return rows, nil
	}

Could be replaced by generated code like:

	defer mon.Task()(&ctx)(&err)
	var __values []interface{}
	return queryAll(obj,
		"SELECT bucket_storage_tallies.bucket_name, bucket_storage_tallies.project_id, bucket_storage_tallies.interval_start, bucket_storage_tallies.total_bytes, bucket_storage_tallies.inline, bucket_storage_tallies.remote, bucket_storage_tallies.total_segments_count, bucket_storage_tallies.remote_segments_count, bucket_storage_tallies.inline_segments_count, bucket_storage_tallies.object_count, bucket_storage_tallies.metadata_size FROM bucket_storage_tallies ORDER BY bucket_storage_tallies.interval_start DESC",
		values,
		func(r *BucketStorageTally) []any {
			return []any{&bucket_storage_tally.BucketName, &bucket_storage_tally.ProjectId, &bucket_storage_tally.IntervalStart, &bucket_storage_tally.TotalBytes, &bucket_storage_tally.Inline, &bucket_storage_tally.Remote, &bucket_storage_tally.TotalSegmentsCount, &bucket_storage_tally.RemoteSegmentsCount, &bucket_storage_tally.InlineSegmentsCount, &bucket_storage_tally.ObjectCount, &bucket_storage_tally.MetadataSize}
		}
	)

Add support for JSONB to .dbx

Acceptance Criteria:

JSONB can be specified as a type in a .dbx file

use generics to reduce field boilerplate

There's a significant duplication of the field code:

type OauthToken_Scope_Field struct {
	_set   bool
	_null  bool
	_value string
}

func OauthToken_Scope(v string) OauthToken_Scope_Field {
	return OauthToken_Scope_Field{_set: true, _value: v}
}

func (f OauthToken_Scope_Field) value() interface{} {
	if !f._set || f._null {
		return nil
	}
	return f._value
}

func (OauthToken_Scope_Field) _Column() string { return "scope" }

Could be replaced by:

type OauthToken_Scope_Field struct { NullableField[string] }

func OauthToken_Scope(v string) OauthToken_Scope_Field {
	return OauthToken_Scope_Field{NullableField: FieldOf(v)}
}
func (OauthToken_Scope_Field) _Column() string { return "scope" }

It's not clear how big the benefit is here. Also, maybe there's a better approach for this.

add support for documentation

It's difficult for the data team to figure out what each field means. Instead of needing to manually annotate, let's add comments directly to the database. That way tools should be able to read out that information.

https://www.postgresql.org/docs/current/sql-comment.html

CRDB requires GROUP BY to be before ORDER BY

It looks that if GROUP BY is after ORDER BY crdb is returning error ERROR: at or near "group": syntax error (SQLSTATE 42601)

Sample query

read limitoffset (
	select bucket_metainfo.project_id bucket_metainfo.name
	where bucket_metainfo.project_id > ?
	where bucket_metainfo.name > ?
	groupby bucket_metainfo.project_id bucket_metainfo.name
	orderby (
		asc bucket_metainfo.project_id 
		asc bucket_metainfo.name
	)
)

SELECT bucket_metainfos.project_id, bucket_metainfos.name FROM bucket_metainfos WHERE bucket_metainfos.project_id > ? AND bucket_metainfos.name > ? ORDER BY bucket_metainfos.project_id, bucket_metainfos.name GROUP BY bucket_metainfos.project_id, bucket_metainfos.name LIMIT ? OFFSET ?

See example https://review.dev.storj.io/c/storj/storj/+/11211 and failing test TestBatchBuckets/Cockroach