guregu / null Goto Github PK

Unmarshaling JSON into an intermediate interface{} results in all numbers being decoded to float64s:

    var v interface{}
    json.Unmarshal(data, &v)

Later on it casts the float64 to an int64, but there's no guarantee that the passed value was actually an integer or that it didn't overflow the int64:

    i.Int64 = int64(x)

So if I have it decode a non-integer such as 123.45 it gets silently cast to 123 when I would have expected an error. If I have a number that doesn't fit in an int64 such as 9223372036854775808, it overflows silently to a negative number as well. I also tested the max int64 size of 9223372036854775807 and found it was overflowing for unknown reasons. Perhaps the intermediate float64 resulted in some loss of precision?

I think the intermediate unmarshal to interface{} should be avoided if possible, instead attempting to unmarshal the []byte directly into an int64.

hi
how your versioning works?
can you plz help me understand
v4 v8 and none really? :)

Hello Greg (hope I got the name right :) )

First of all, great library for marshalling and unmarshalling Golang's SQL null datatypes.
Really saved me a lot of time and effort.

However, I noticed that this library is missing one piece of functionality which is storing null values in database.
I have gone through the library and could not find it anywhere. I had to implement this part in my own codebase.
I would be happy to raise a Pull Request so that this functionality can be merged to master.

Please do let me know if I can go ahead with raising a pull request.

From the documentation I understand that types are either fully treated as zero or null values in Go and JSON.

I am trying to understand if I can use null values in Go but have them encoded to zero in JSON.

Hi guregu,
My teammates and I really like your library, as it has helped us a lot when going from JSON to go structs to databases (thank you!). One small issue we found is unmarshalling large numbers from JSON. Right now, because there is only one number type in Javascript, integers in JSON that are large enough are decoded in scientific notation. Because of this, these numbers cannot be unmarshalled into integer types, in particular your null.Int and zero.Int types. I was wondering if you had any thoughts on the issue.

If you're interested, I've already written an implementation that passes all of the tests, if you want to take a look.

Unmarshalling and marshalling types in the null package isn't symmetric. For example:

type T struct {
  Name null.String
}

in := []byte(`{"Name":""}`)
var t T
json.Unmarshal(in, &t)
fmt.Printf("%+v\n", t) // => {Name:{NullString:{String: Valid:false}}}
out, _ := json.Marshal(t)
fmt.Println(string(out)) // => {"Name":null}

Why not have in unmarshal to {Name:{NullString:{String: Valid:true}}}? In addition to clearing up the issue shown above it would make the null package incredibly useful for the common issue of how to determine if a RESTful request intends to set a value to the zero value or if it just wasn't included in the request (a PATCH-like request).

Right now the common recommendation is to use pointers for the fields in your structs as described here, but I'd much prefer using the null package for this if I could.

Let me know what you think. I'm happy to create a PR for this if you think this idea has legs.

What version of Go are you using (go version)?

$ go version
go version go1.15.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

https://play.golang.org/p/7kgBsW80Mzf

What is the problem?

If we would have not empty decoder.errorContext.FieldStack and we received error of fmt.wrapError type from UnmarshalJSON then later addErrorContext wouldn't be able to set proper value for err.Field and err.Struct because error type won't match with *UnmarshalTypeError.

https://github.com/golang/go/blob/a7e16abb22f1b249d2691b32a5d20206282898f2/src/encoding/json/decode.go#L181-L183
https://github.com/golang/go/blob/a7e16abb22f1b249d2691b32a5d20206282898f2/src/encoding/json/decode.go#L255-L258

Like this library, very clean & nice to use.

Would you be interested in adding support for / accepting a PR with an implementation of null.Duration ?

When using gorp or other ORMs it would be nice to be able to do the same with the Time type

Hey,

I have a question. The null.String type does not have a key specified. Thus, govet as well as sonarqube give me the warning:

vendor/gopkg.in/guregu/null.v3.String composite literal uses unkeyed fields

for a struct using this type. I was wondering if it's possible to change that or if there is a fix for this?

example

StringFrom("")
``
value

sql.NullString{Value:"", Valid:true}

which is wrong

cannot use value.Network (type string) as type "gopkg.in/volatiletech/null.v6".String in assignment

I tried this with Gin-Gonic

type AddUserRequest struct {
  // some other properties
  DateOfBirth               null.Time   `json:"user_birthday" time_format:"2006-01-02"`
}

When I send

{
  // some other stuff
  "user_birthday": "2000-04-05"
}

I still get the (infamous?) parsing time \"\"2000-04-05\"\" as \"\"2006-01-02T15:04:05Z07:00\"\": cannot parse \"\"\" as \"T\"" error.

Is there a workaround for this issue? I don't want to send a string and parse manually.

Thanks!

The docs state the following about null.String.IsZero:

IsZero returns true for null or empty strings, for future omitempty support. (Go 1.4?)
Will return false s if blank but non-null.

Which looks like a contradictory statement. In a test with an empty string, the latter held true: false is returned if the string is equal to "". Could either the behavior of IsZero() change to return true if string equals empty string or the docs be clarified?

Thanks.

Hey guys,

is it on purpose that this module can not support uint64 (bigint unsigned)?

casting uint64 to int64 will cause a loss of the last bit. And therefore I would have wrong results in DB

type Thing struct {
  Id string `db:"id"`
  number null.Int `db:"name"`

Database has the schema:

CREATE TABLE `thing` (
`id` varchar(25) NOT NULL,
`number` bigint unsigned DEFAULT NULL
)

Implemtation of null.IntFrom

Is it sqlx not supporting it? How can I work around that?! :)
related to: go-sql-driver/mysql#715 ?

can add custom time format support for Time?

add a global variable with name is TimeFormats and type is []string

It seems that all types in null except null.Time has IsZero() method, is it intended?

I'd like to work towards releasing a new version.

v3 stuff

• Merge #36 because it's backwards compatible, need to add zero support first.

v4 stuff

• Fix #47 (backwards incompatible)
• Take the opportunity to get rid of some cruft? If we drop support for marshaling from sql.NullXXX JSON objects (as in

  null/int.go

  Lines 70 to 71 in 80515d4

  	case map[string]interface{}:
  	err = json.Unmarshal(data, &i.NullInt64)

  ) we can speed up the library a bit and avoid having to unmarshal into interface{} first. I doubt many people use this anyway.

Can we make StringFrom verify string argument for "emptiness"?

func StringFrom(s string) String {
    return NewString(s, true)
}

to

func StringFrom(s string) String {
        if s == "" {
            return NewString(s, false)
        }
        return NewString(s, true)
}

It is useful in:

result := null.StringFrom(ctx.Request.Header.Get("some-header"))

And many other cases.

Or a new method with this behaviour?

Purpose: to allow cases where field it's value does not exist inside an wrapping object to be sent to the server as NULL JSON/JSONB Value (where structure type is json.RawMessage)

If someone could implement it it would be even better.

1. defind web api dto struct

type Dto struct {
   Start null.Time
}

2. send this json to web api

{ 
  "Start": null
}

4. service: use gorm update entry

gorm.Model(&MyTable).Where("id = 1").Update(&dto)

5. Hope to generate sql

update my_table set start = null  where id = 1

The start field is now ignored when used in this way

In JSON there are effectively two types of null.

1) When the key is provided, and the value is explicitly stated as null.
2) When the key is not provided, and the value is implicitly null.

This is important when constructing a PATCH endpoint because:

1. When the key is provided, with a null value, the field should be set to null
2. When the key is not provided, the value should not be changed

There is a more detailed discussion of this issue here, with example code of how to solve the problem:

https://www.calhoun.io/how-to-determine-if-a-json-key-has-been-set-to-null-or-not-provided/

It would be nice if the null library can support this.

Any ideas or opinions on how to handle generics in this library?
I played around with creating a generic Null[T] type, but I wonder if it's even useful at that point, because we are essentially recreating the Optional type you might find in other languages. At what point is it going too far?
Maybe ideally we could make generic Null/Zero types and use the materialized version of it for each pre-existing type.

I'm also curious if we'll see a generic sql.Null[T] type in the standard library, which might obsolete a good chunk of this library, especially if this change lands: golang/go#5901 (which lets you easily specify custom marshalers for JSON, moving the responsibility from the object itself to the encoding site).

See also: golang/go#48702

Less of an issue, and more of an FYI. I have been keeping tabs on the (un)marshaling/(de)encoding issue with omitempty, and it looks like they have a adjacent proposal that may provide a workable solution:

golang/go#5901

Didn't see anywhere else to leave a comment about this, and I figured it is relevant to this project in particular (I know its relevant to my usage of this project! Thanks by the way!)

How can we check null.String value where is null or where is empty ?

I see the function IsZero work not as expected.

I use beego with "nuu.v4", create struct

type SysUserBean struct {
UserId int64 orm:"auto" json:"userId"
Username null.String json:"username"
}

when run "bee run", then tips, "field: models.SysUserBean.Username, unsupport field type &{{ %!s(bool=false)}}, may be miss setting tag", can't go on

how to fix it

I found this error

/home/myname/go/src/github.com/guregu/null/time.go:15:2: undefined: sql.NullTime

Ubuntu 18.04.4 LTS
go version 1.12.10

Each struct of this library uses sql package (persistence or infrastructure layer), while it can be used for JSON marshaling (presentation layer).
It seems to be against some software architecture patterns based on layered architecture or etc.

type JsonUser struct { ID null.Int json:"id" db:"id"Email null.Stringjson:"email" db:"email"Password null.Stringjson:"-" db:"password"FirstName null.Stringjson:"first_name" db:"first_name"LastName null.Stringjson:"last_name" db:"last_name"CreatedAt null.Time json:"created_at" db:"created_at"UpdatedAt null.Time json:"updated_at" db:"updated_at"`
}

func UserByID(w http.ResponseWriter, req *http.Request) {
vars := mux.Vars(req)
if userID, err := strconv.Atoi(vars["id"]); err != nil {
fmt.Fprint(w, response.Success("No record found.", http.StatusNotFound, nil).JSON())
return
} else {
jsonUser := &JsonUser{}
err = DB.QueryRow("SELECT id, email, first_name, last_name, created_at, updated_at FROM users WHERE id = ? LIMIT 1;", userID).Scan(&jsonUser.ID, &jsonUser.Email, &jsonUser.FirstName, &jsonUser.LastName, &jsonUser.CreatedAt, &jsonUser.UpdatedAt)
if err != nil {
fmt.Fprint(w, response.Error(err.Error(), http.StatusInternalServerError).JSON())
return
}
fmt.Fprint(w, response.Success("Request successful!", http.StatusOK, jsonUser).JSON())
}

return

}`

CreatedAt is not null, but UpdatedAt is for the particular record I'm retrieving.

The error that is being thrown is: "sql: Scan error on column index 4: null: cannot scan type []uint8 into null.Time: [50 48 49 56 45 48 56 45 50 50 32 48 48 58 53 54 58 50 50]"

It doesn't seem that this package's Time is not handling []uint8 but I'm not sure how to get this working. Thanks!

/root/go/pkg/mod/gopkg.in/guregu/[email protected]/zero/time.go:15:2:
undefined: sql.NullTime

// NewInt creates a new Int from a string
func NewIntFromString(i string, valid bool) Int {
if valid {
inter, err := strconv.Atoi(i)
if err != nil {
return Int{
NullInt64: sql.NullInt64{
Int64: 0,
Valid: false,
},
}
}
in := int64(inter)
return Int{
NullInt64: sql.NullInt64{
Int64: in,
Valid: valid,
},
}
}
return Int{
NullInt64: sql.NullInt64{
Int64: 0,
Valid: false,
},
}
}
test:
func TestNewIntFromString(t *testing.T) {
type args struct {
i string
valid bool
}
tests := []struct {
name string
args args
want Int
}{
{
name: "equal zero",
args: args{
i: "0",
valid: false,
},
want: IntFrom(0),
// TODO: Add test cases.
},
{
name: "equal one",
args: args{
i: "1",
valid: true,
},
want: IntFrom(1),
// TODO: Add test cases.
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := NewIntFromString(tt.args.i, tt.args.valid); !reflect.DeepEqual(got, tt.want) {
t.Errorf("NewIntFromString() = %v, want %v", got, tt.want)
}
})
}
}

I use this like:
id:= zero.NewIntFromString(c.GetQuery("id"))
where c is a gin.Context (could do similar with http.GetForm)

Testing my models on GraphQL, I detected some issues regarding text-marshaling implementation in this library.

null.Time returns "null"

func (t Time) MarshalText() ([]byte, error) {
	if !t.Valid {
		return []byte("null"), nil
	}
	return t.Time.MarshalText()
}

Other null.X implementations

// MarshalText implements encoding.TextMarshaler.
// It will encode a blank string when this String is null.
func (s String) MarshalText() ([]byte, error) {
	if !s.Valid {
		return []byte{}, nil
	}
	return []byte(s.String), nil
}

// MarshalText implements encoding.TextMarshaler.
// It will encode a blank string if this Int is null.
func (i Int) MarshalText() ([]byte, error) {
	if !i.Valid {
		return []byte{}, nil
	}
	return []byte(strconv.FormatInt(i.Int64, 10)), nil
}

My question/curiosity is: why does null.Time text-marshaling return a "null" string instead of an empty string like the others?

I'm trying to compare nullable types but I couldn't find an existing easy way to do it.

Are you interested in adding support for / accepting a PR that adds functions like the following for all nulltypes?

func StringEquals(a, b null.String) bool {
	return a.Valid == b.Valid && (! a.Valid || a.String == b.String)
}

// StringEquals(null.NewString("foo", false), null.NewString("foo", false)) -> true
// StringEquals(null.NewString("foo", false), null.NewString("bar", false)) -> true
// StringEquals(null.NewString("foo", true), null.NewString("foo", false)) -> false
// StringEquals(null.NewString("foo", true), null.NewString("foo", true)) -> true
// StringEquals(null.NewString("foo", true), null.NewString("bar", true)) -> false

I'm not entirely sure about the workings of the zerotypes (I don't use them). If I got it right, they can just be compared using their string/bool/int/... value so they don't need the extra function, right?

could not import gopkg.in/guregu/null.V3 (no required module provides package "gopkg.in/guregu/null.V3")

I tried v4 also with the same error

Does this library support golang 1.13.

We just upgraded our application to 1.13 and now are getting:

2020/08/24 00:12:38 Failed to build app: building app with command [go build -o /tmp/staging/usr/local/bin/start bitbucket.org/x1x1x1x1/y1y1y1/z1z1z1], env [PATH=/go/bin:/usr/local/go/bin:/builder/google-cloud-sdk/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=cb5c804c1756 HOME=/builder/home BUILDER_OUTPUT=/builder/outputs DEBIAN_FRONTEND=noninteractive GOROOT=/usr/local/go/ GOPATH=/go GOPATH=/tmp/staging/srv/gopath]: err=exit status 2, out=# github.com/guregu/null
srv/gopath/src/github.com/guregu/null/time.go:15:2: undefined: sql.NullTime

It looks like there were some changes to the SQL libraries in the new versions of go lang. Is there something we need to support the changes in 1.13

I realize that the db driver doesn't support uint as a type and that javascript only have the number type. Basically meaning a rather small performance hit because of the conversion we would need to do, not to mention the complications with overflow.

But i still think in some of the structs we have it would make sense to have a null.Uint just for cosmetic purposes and to enforce correct types at the point of storage to avoid errors. Would a PR with uint type(s) be accepted?

There might be a super simple answer to this, but as a newbie at Go, I'm having an issue.

I'm fetching some data from MSSQL and have set up some struct properties as null.String since the DB can contain null values.

Before json encoding the structs, I need to manipulate them a bit and need to do some simple if statements, like: if row.x == "y" { //do stuff } but since row.x has the type null.String, that comparison fails with:

mismatched types null.String and string

If I try to convert it to string first: var x = string(row.x), I get:

cannot convert (type null.String) to type string

What is the best way to handle this?