@zudov I propose a renaming of the library: googlica -> gogolica.

There is a few reasons:

1. The new one will get less hits on google. Also, while googling googlica, it gets autocorrected to "google", which is not nice.
2. There is objectively strong ties to the haskell library
3. Allows to refer to Gogol, with pictures and great quotes like this one:

"You can't imagine how stupid the whole world has grown nowadays."

Nikolai Vasilievich Gogol

Currently we simply allow the user to specify alt=media in order to get redirected to the downloadable content (I suppose the server then returns a SEE OTHER).

"Media Download" section however recommends constructing the download url by prefixing the path with /download/, which avoids the redirect.

Also there is a useMediaDownloadService, I'm not sure, but probably it means than when it's true the alt=media won't work. Or maybe it would, nevertheless they recommend to prefix url with /download/ at all times 🤷‍♂️

I think it would be pretty nice if when the mediaDownload is true, we generate additional function (with -media) suffix that would perform that /download/ request.

So instead of (taken from README) doing:

(gcs/objects-get "my-new-bucket-name" "my-file-name" {:alt "media"})

One would do:

(gcs/objects-get-media "my-new-bucket-name" "my-file-name")

Alternative interface would be to allow some {:media-download true} option as an argument to normal objects-get, but it feels nice to have a separate function, since the functionality of "get the metadata of an object" and "get content of an object" feels like different functions (what you do with the response and the purpose of it is quite different).

The support for alt parameter would remain as part of #26, since it also allows to specify alternative response formats, we should be aware of that and decode the content appropriately.

We should have some helpers and preferred practices (documented in the README) for handling various errors that are produced by either gogolica itself, like:

• failed to load the credentials
• given body doesn't match the required schema
• the parameter that's conditionally required wasn't provided
• ...

as well as the errors returned by google's services:

• bucket doesn't exist doesn't exist
• queue is full
• machine hasn't started yet
• user rate limit exceeded
• service rate limit
• ....

As usually, it's worth looking at how gogol does it. There is a hierarchy of errors, the errors are thrown as exceptions, convenient lens machinery is provided to focus on particular types of errors, and there are handy catching, trying functions.

I guess clojure can do that quite well by putting the error data into ex-info and providing some predicates (service-error?, credentials-error?, bucket-not-found-error?, rate-limit-error?) on those exceptions and using something like slingshot to catch them conveniently.

That was initially a comment on #21, but I thought it deserves a separate issue since it's not about the 500 errors, but more about the errors that you would actually want to handle in your code. #21 is more about the retries on availability errors.

Right now we're only handling get methods, while we should support all the http verbs.

There are certain parameters that can be passed to ALL methods of google APIs. We should allow passing them.

More details at https://developers.google.com/discovery/v1/using#discovery-common-parameters

Not all the characters are allowed in bucket names: https://cloud.google.com/storage/docs/naming#objectnames

We shouldn't be converting all the model json to kebab case. While it is nicer and more idiomatic to do destructuring with it, Google APIs expect parameters to be passed in with their syntax in requests.

The following code is being generated now:

While we could of course convert back the parameters to CamelCase, it might be that the conversion is not lossless, and I don't feel safe doing CamelCase -> kebab-case -> CamelCase.

Right now generated code doesn't contain any newlines, so it's pretty unreadable.
We should use something like cljfmt to format it.

We have a half-baked implementation of uri-template substitution, that takes a path string like "/a/{foo}/b/{bar}" and generates a Clojure vector with the proper symbols like ["/a/" foo "/b/" bar].

Testing this on the storage model works, but if we run it on all the 3931 methods, we discover that it's broken on some of them because it doesn't cover all of the standard.

https://developers.google.com/identity/protocols/application-default-credentials

The namespaces of generated modules look like gogolica.storage.v1, and our utility namespaces are like gogolica.auth, gogolica.utils, gogolica.model and so on.

To distinguish between them, we should consider somehow distinguish our "operational" namespaces from the generated one. I like the names that we currently generate, so maybe we could move the operational modules a level deeper. Something like gogolica.gen.{auth,utils,model,...}.

However, gogolica.gen is boring, and maybe we should give a nice name to the thing that would be generating the actual user-facing gogolica. I propose either Viy (more evil), or Bulba (more epic). Then we can have gogolica.viy or gogolica.bulba.

For parameter like:

:projection
  {:type "string",
   :description "Set of properties to return. Defaults to noAcl.",
   :enum ["full" "noAcl"],
   :enumDescriptions
   ["Include all properties."
    "Omit owner, acl and defaultObjectAcl properties."],
   :location "query"}

We could spew out the code like:

:projection
  (case (name projection)
    "no-acl" "noAcl"
    (name projection))

to allow kebab case.

#10 (spec-checking) and #19 (docstring generation) would benefit from recognizing them too.

In API models every method has a parameterOrder field which specifies a canonical order in which the required parameters should be placed.
We need to take it in account when generating a vector of function's required parameters.

Some older APIs have a "feature" called dataWrapper. This means that all of the response/requests have to be wrapped in {data: ...} singleton.

See https://developers.google.com/discovery/v1/using#discovery-features

Since the json files define JSON schemas for the response of each method, we should use this information to spec the generated code, with either core.spec or plumatic/schema.

All the methods have an optional-parameters map in the end, which contains the optional parameters, but it's not optional in itself, and it has to be specified even when empty.
This makes for an awkward UI.

We should have an additional arity, where that map is actually optional.

We currently get the models by submoduling the go repo and then having a bash script that copies things from there, but instead one could just get them from API explorer via HTTP:

http https://www.googleapis.com/discovery/v1/apis/storage/v1/rest | jq '.resources | keys'                                                             
[                                                                                                                                                                                 
  "bucketAccessControls",
  "buckets",
  "channels",
  "defaultObjectAccessControls",
  "notifications",
  "objectAccessControls",
  "objects",
  "projects"
]

I think we should get rid of irrelevant submodule and either rewrite the script to fetch all things via curl and jq, or just have code that can fetch them inside gogolica.gen.model.

Checking in the kilolines of generated code can get quite messy, but this can be mitigated by establishing and documenting some sensible rules.

A couple of ideas to get started:

• the commits that include generated things should never include anything that was written by hand
• "regenerate code/documentation/models" things should be entirely done by self-contained scripts, that are checked into the repo. No arbitrary command line fiddling.
• those scripts should perform the full pipeline of:
  • clean the repo
  • regenerate the relevant thing and stage it
  • request for confirmation and commit it
  • push and create a pull request (should be easy via hub)
• the commit as well as pull request messages should be generated automatically and be useful:
  • specify the name of the script that produced it
  • include the hashes and versions of the upstream things
  • BONUS: include summary of changes (e.g. a list of affected namespaces/methods/functions)
• we should specify some fictional commit author, while the person responsible for the commit is specified as commiter
  • BONUS: Give them a name, identity, create them a github user and automate them via CI. Then they could be as bold as to specify themselves as a commiter.

The MVP interface could be just a lazy-seq + instructions on how to consume it. We might adopt some streaming API later if it's needed.

Some methods require authentication/authorization, while some are open.

I have no idea how to do this yet, but the general API reference says that we should use Oauth2, so we should probably implement that.

Currently when mapping over :resources we treat it as a flat list, but SUDDENLY, resources can be nested, so we end up only grabbing the top level routes at the moment.

> (-> storage-model :resources :projects :resources :serviceAccount :methods :get)
{:id "storage.projects.serviceAccount.get",
 :path "projects/{projectId}/serviceAccount",
 :httpMethod "GET",
 :description
 "Get the email address of this project's Google Cloud Storage service account.",
 :parameters
 {:projectId
  {:type "string",
   :description "Project ID",
   :required true,
   :location "path"},
  :userProject
  {:type "string",
   :description
   "The project to be billed for this request, for Requester Pays buckets.",
   :location "query"}},
 :parameterOrder ["projectId"],
 :response {:$ref "ServiceAccount"},
 :scopes
 ["https://www.googleapis.com/auth/cloud-platform"
  "https://www.googleapis.com/auth/cloud-platform.read-only"
  "https://www.googleapis.com/auth/devstorage.full_control"
  "https://www.googleapis.com/auth/devstorage.read_only"
  "https://www.googleapis.com/auth/devstorage.read_write"]}

Right now lots of stuff is "tested" including repl outputs as comments, let's move this into actual tests.

See here.

The parameters list for every method has some information about the type and the meaning of every parameter.
While the actual typechecking of the params should be covered in #10, here we should include both the type and documentation of the params in the method docstring.

Google specifies some guidelines about handling errors, which we should follow:

When uploading media, be sure to follow these best practices related to error handling:

• Resume or retry uploads that fail due to connection interruptions or any 5xx errors, including:

  • 500 Internal Server Error
  • 502 Bad Gateway
  • 503 Service Unavailable
  • 504 Gateway Timeout

• Use an exponential backoff strategy if any 5xx server error is returned when resuming or retrying upload requests. These errors can occur if a server is getting overloaded. Exponential backoff can help alleviate these kinds of problems when there is a high volume of requests or heavy network traffic.

• When you receive an error other than a 5xx error, you do not need to use an exponential backoff strategy. Instead, limit the number of times you retry the request. For example, your code could report an error to the user after retrying a request 10 times.

E.g. when reading the storage json, we take the name and version top level keys (which values are storage and v1 in this case), and generate the following ns definition:

(ns googlica.storage.v1
  (:gen-class)
  (:require [clj-http.client :as client])

Simple upload is within scope of issue #12, while the more complex resumable upload deserves its own issue.

HOWTO: https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload