brendanhay / gogol Goto Github PK

View Code? Open in Web Editor NEW

278.0 278.0 103.0 118.49 MB

A comprehensive Google Services SDK for Haskell.

License: Other

Haskell 99.99% Makefile 0.01% Shell 0.01% Nix 0.01%

google haskell

gogol's People

Contributors

Stargazers

Watchers

Forkers

wagonhq arrowplum justinmanley traytonwhite hitscotty feumilieu runeksvendsen dtulig gbordyugov cthulhuden rueshyna mbj wizek yuhangwang shokohara jamesthompson steshaw karshan miladz68 guoshimin silky nakaji-dayo amarpotghan saurabhnanda paparga adetokunbo termina1 jtdaugherty jkarni thumphries choyuri mutantmell ehamberg betterteamapp rumuki alexanderkjeldaas kchaikasnap31 ababkin donatello vacationlabs alphasheets-development soostone yuanwang-wf tmcgilchrist stealthmate polynomial capital-match smaccoun lemois folq nick96 sopvop buecking futurice srid korayal luigy obsidiansystems kikaitech agrue kim tarsbase jamesmcmahon alistairb tonicebrian lostbean arche-tool ilyakooo0 pigsinablanket iammrinal0 peterbecich gwils kevoriordan madjar paulvictor earnestresearch kfowler kayvank aravindgopall shinzui mospina krisis superrare hsenag tylermerz hendysg paidright maksar tommyengstrom purefn costarastrology yaitskov muqadma colonelpanic8 m4dc4p chordify mercurytechnologies tripshot

gogol's Issues

Haxl support

It would be great if gogol would support Haxl out of the box. I see that it relies in part on servant, and I see that servant-haxl-client is a thing, but I'm not sure how possible it is to use the two together. It seems to me that both libraries have code that overlaps: the part that issues the actual HTTP request.

How are you meant to "re-scope" an env?

I've got a module in my project that's the single interface to all of my usage of google cloud services. Currently, each individual function is rebuilding the env each time, which seems pretty wasteful. I would rather the user be able to pass in an env to each function.

The problem comes with the scopes needed for each function:

Require the user to initialize an env on their own that happens to have the full set of scopes that the application needs. This seems like a good bit of documentation I guess as an env would "accumulate" scopes sort of like how usage of multiple typeclass-constrained functions accumulates constraints.
Let them pass in any old env and append the scopes I need.

The problem with 1 is if one of my google functions uses multiple google functions internally, the required type signature does not seem to concatenate lists but forces me to specify them individually, duplicates and all. So if I use foo which requires '[a, b, c] and bar which requires '[b, c, d], then fooBar's constraints won't require (HasScope s '[a, b, c, d] ~ True) => Env s -> ... but rather (HasScope s '[a, b, c] ~ True, HasScope s '[b, c, d]) => Env s -> ..., which can get really repetitive the more you use. I am not willing to forgo writing type signatures for toplevel functions.

The problem with 2 is envScopes :: Lens' a (Proxy s) uses a Lens' which does not allow type-changing, so I cannot do something like let env' = env & envScopes .~ neededScopes because that changes s and that lens will not allow it.

I tried to look at the examples for guidance but because they set up the env and set scopes in one shot, they can't really model the scenario of the user passing in an env to something that demands a certain set of scopes. What should I do?

gogol-storage: Multipart Upload incomplete/corrupted

Hi,

I'm currently trying to upload a 2MB image to a bucket, but the upload seems to be corrupted. I'm running the example from examples/gogol-examples.

> stack ghci gogol-examples
ghci> example "my-bucket" "image.jpg"
...

ghci> :! gsutil ls gs://my-bucket
2035740  2016-03-14T20:59:51Z  gs://my-bucket/image.jpg
TOTAL: 1 objects, 2035740 bytes (1.94 MiB)
ghci> :! stat -f "%z" image.jpg
2035466

I discovered that this lib uses AltMedia for the uploadType query parameter. It resolves to "media" as a value for the parameter, but this is wrong for multipart uploads and should be "multipart" (https://cloud.google.com/storage/docs/json_api/v1/objects/insert).

For a quick fix, I changed the Line 65 to:

instance ToText AltMedia where toText = const "multipart"

and the image is now correctly uploaded.

> stack ghci gogol-examples
ghci> example "my-bucket" "image.jpg"
ghci> :! gsutil ls -rl gs://my-bucket/
   2035466  2016-03-14T21:10:40Z  gs://my-bucket/image.jpg
TOTAL: 1 objects, 2035466 bytes (1.94 MiB)

Please update gogol-datastore to v1

Hi there

There's a final version of the Datastore API out, v1: https://cloud.google.com/datastore/docs/reference/rest/#collection-v1projects

Will you update gogol-datastore to support this version?

Thank you.

request to publish changes

@brendanhay Could you publish new changes?
I really need this change, #66.

Thanks!

Application default credentials read the wrong path on Google Compute Engine

I'm trying to use gogol with GCE. I'm getting this error when connecting:

RetrievalError (StatusCodeException (Status {statusCode = 404, statusMessage = "Not Found"})

... and later on ...

The requested URL <code>/instance/service-accounts/default/token</code> was not found on this server.

...

("X-Request-URL","GET http://metadata.google.internal:80instance/service-accounts/default/token")

It looks like the correct path is this:

$ curl -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
<< a valid token >>

But it looks like gogol is missing part of the path. Or I'm doing something dumb...

[Help/Advice needed] base64: input: invalid encoding at offset: 1

Hi there. Thanks for this awesome library!

I'm trying out gogol-compute and I'm getting a somewhat weird error when I try to run the example below. I'm trying to get a list of instances for a particular project.
From what I can tell from the debug output I successfully connect to the google compute api, get a bearer token and make a request but then when the debug output would normally print the body (with I assume, the instances json in there) I get the following error:

Main.hs: base64: input: invalid encoding at offset: 1

I tried setting the logger to Info and Error with no luck. I also tried ignoring the result of the api call
but the same error happens.

{-# LANGUAGE OverloadedStrings #-}

module Main where

import           Control.Lens                           ((&), (.~), (<&>), (?~))
import qualified Data.Aeson                             as JSON
import qualified Data.ByteString.Lazy                   as BS
import           Data.Maybe                             (fromJust)
import           Data.Text                              (Text)
import           Network.Google
import           Network.Google.Auth
import           Network.Google.Auth.ApplicationDefault
import           Network.Google.Auth.ServiceAccount
import           Network.Google.Compute
import           System.IO                              (stdout)

import qualified Data.Text                              as Text

main = do
    lgr  <- newLogger Debug stdout
    bs <- BS.readFile "service_account.json"
    manager <- newManager tlsManagerSettings
    let creds = fromJSONCredentials bs
    case creds of
        Left s -> error s
        Right c -> do
            env  <- (newEnvWith c lgr manager) <&> (envScopes .~ computeReadOnlyScope)
            r <- runResourceT . runGoogle env $ do
                send $ (instancesList "testing" "us-east1-c")
            print r

Do not know how to use some endpoints - cannot find corresponding *Response definitions for those

I successfully used endpoints like Network.Google.Resource.Container.Projects.Zones.Clusters.List for instance.

But i cannot seem to figure out where the corresponding *Response definition is for this call:
https://hackage.haskell.org/package/gogol-compute-0.2.0/docs/Network-Google-Resource-Compute-Networks-List.html

Or perhaps it does not exist in the json definition and someone is supposed to add it? I wouldn't mind extending the json spec if needed. Is there some guide that explains how to do this?

Thanks

gogol-pubsub: PubSub projects.topics.publish method generates incorrect URL

According to the official docs from Google, the URL format should be:

POST https://pubsub.googleapis.com/v1/{topic=projects/*/topics/*}:publish

However, the publish URL being generated is missing the colon between the topic and "publish": /v1/projects/my-project-here/topics/topic-testerpublish

It works fine though if I add a colon myself, but I doubt that's the intended way to use the library:

{-# LANGUAGE OverloadedStrings #-}
import Control.Lens           ((&), (.~), (<&>), (?~))
import Data.Text              (Text)
import Network.Google
import Network.Google.PubSub
import System.IO              (stdout)

import qualified Data.Text as Text

example :: IO PublishResponse
example = do
    lgr  <- newLogger Debug stdout
    env  <- newEnv <&> (envLogger .~ lgr) . (envScopes .~ pubSubScope)

    runResourceT . runGoogle env $ do
      send $ projectsTopicsPublish 
        (publishRequest & prMessages .~ [pubsubMessage & pmData ?~ "Hi!"])
        "projects/my-project-here/topics/topic-tester:"

gogol-core-0.0.1 does not compile

Citing from http://hydra.cryp.to/build/1732331/nixlog/1/raw:

Building gogol-core-0.0.1...
Preprocessing library gogol-core-0.0.1...

src/Network/Google/Data/Time.hs:37:18:
    Could not find module ‘Servant.Common.Text’
    Use -v to see a list of the files searched for.

I suppose that module has been removed from servant-0.6.x?

A way to disable Auth (for use with emulators)?

I'm trying to use gogol-datastore with the Datastore Emulator, for local testing a debugging. I can't find any definitive information on it, but I assume this emulator just ignores OAuth headers, but I'm unable to test, since I can't figure out a way to run requests while ignoring authentication.

What's the right way to run requests against an emulator, auth-wise?

Design question: newtypes

Hey there!

I'm taking gogol for a spin. I'm a long time amazonka user and noticed that theres some departers from the amazonka design. In particular, in amazonka, things like BucketName are a newtype over text which I really like because it helps prevent you from mixing up something that's semantically a BucketName with one of the most ubiquitous types. I noticed in gogol-storage its just a Text. I'm curious if this was a conscious decision and if so, why?

Request: Support for Analytics Measurement Protocol

Hey!

Love the library. I was wondering if the Analytics Measurement Protocol is something you could add here. It's unauthenticated so I'm not sure if it falls into the same category as the other APIs you generate the bindings for, but I at least wanted to ask. :)

Documentation on how the generation works

I'm curious about how the generation works, and I suspect so might be other people. Does it use this?: https://developers.google.com/discovery/

Datastore: Not exporting internal To/FromJSON instances

Many Network.Google.Datastore.Value types -- and types contained within these -- have sensible Data.Aeson.ToJSON/FromJSON instances, because their purpose is to map to JSON types.

This goes for types like ValueNullValue (toJSON = const Data.Aeson.Null), ArrayValue (toJSON = ^. avValues) and Entity:

instance ToJSON Entity where
    toJSON e = Object $
        fmap toJSON . view epAddtional .
        fromMaybe (entityProperties Map.empty) $
        (e ^. eProperties)

However, these types also need to be serialized, and sent over the API, hence this library's ToJSON and FromJSON instances, which are used for this internal purpose.

I would like to ask if it's possible to contain these instances internally in the library, such that I can create an elegant wrapper around gogol-datastore which has the right Aeson instances for the various types, thus allowing users to convert seamlessly between eg. a value & vStringValue and a Data.Aeson.String.

I have created JSON conversion functions for all the useful Network.Google.Datastore.Value types, as well as eg. ArrayValue and ValueNullValue. However, these implementations really belong as ToJSON/FromJSON instances of Network.Google.Datastore.Value, ArrayValue and ValueNullValue, such that toJSON and fromJSON knows which lenses to use for conversion, for example vIntegerValue or vDoubleValue to convert to/from a JSON Number.

Support for error handling with response body?

gogol doesn't offer much context for API errors right now, since it uses checkStatus to throw ServiceError on non-success response statuses. (And checkStatus doesn't have access to the response body.)

Is this intentional, or a planned change in the future? It's hard to work with these APIs without any error context beyond the raw status code.

These libraries (amazonka etc) are amazing btw 👍😀🔧

Network.Google.PubSub.Types:PullRequest API Type Error

I've just made my first subscription pull request to PubSub for an existing subscription.

I noticed that my Network.Google.PubSub.Types:PullRequest field prMaxMessages which is typed as a Maybe Int32 apparently is required to be a non optional value.

https://github.com/brendanhay/gogol/blob/master/gogol-pubsub/gen/Network/Google/PubSub/Types/Product.hs#L559

Here's the ServiceError that got returned:

ServiceError (ServiceError' {_serviceId = ServiceId "pubsub:v1", _serviceStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, _serviceHeaders = [("Vary","Origin"),("Vary","X-Origin"),("Vary","Referer"),("Content-Type","application/json; charset=UTF-8"),("Content-Encoding","gzip"),("Date","Sat, 11 Mar 2017 19:06:21 GMT"),("Server","ESF"),("Cache-Control","private"),("X-XSS-Protection","1; mode=block"),("X-Frame-Options","SAMEORIGIN"),("X-Content-Type-Options","nosniff"),("Alt-Svc","quic=\":443\"; ma=2592000; v=\"36,35,34\""),("Transfer-Encoding","chunked")], _serviceBody = Just "{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"A required argument is missing in the request: (argument=\\\"max_messages\\\").\",\n    \"status\": \"INVALID_ARGUMENT\"\n  }\n}\n"})

If there's anything I can do to help fix this I'd be happy to submit a PR. Thank you so much for this incredible work @brendanhay. I've said it before but this and amazonka are a tour de force in Haskell.

Please provide a proper license tag in the Cabal file

My understanding is that MPL-2.0 would be the correct choice.

Feature Request: Google Places API

Would be nice to have an ability to use Google Places API with this library.
https://developers.google.com/places/

Missing docs for gogol-bigquery package

Could you provide some documentation about the usage of this client lib?
One example that fetches data from a public dataset would be sufficient.

"Couldn't match type [...] ("https://www.googleapis.com/auth/datastore" Data.Type.Equality.== "https://www.googleapis.com/auth/datastore") [...] with ‘'True’"

I have this function, which I use to make Datastore requests:

sendReq' :: ( DatastoreM m
            , HasScope '[AuthDatastore] a
            , GoogleRequest a)
         => (ProjectId -> a)
         -> m (Rs a)
sendReq' mkReq = do
    pid <- getPid
    liftGoogle $ Google.send (mkReq pid)

The DatastoreM monad just has a few things in a config, like the ProjectId. I use it like this:

runQueryReq :: DatastoreM m
         => Maybe TxId
         -> RunQueryRequest
         -> m RunQueryResponse
runQueryReq txM req =
    sendReq' (projectsRunQuery txReq)
  where
    txReq = maybe req (`atomically` req) txM

but I get the error below for the above function, and I can't figure out why. It happens for all projects, not just projectsRunQuery.

It's solved by adding HasScope '[AuthDatastore] <Project> to the constraint list for the function that needs to consume from sendReq' (eg. runQueryReq), but then I also need to add it to the next function, and so on, until I'm out in a class instance definition, and I need to use UndecidableInstances in order to enable it here.

Can you tell me why this happens, and if there's something I'm doing wrong, or if it's GHC acting up?

   /Users/rune/IdeaProjects/promissory-note-app/paychan-datastore/src/DB/Request/Query.hs:43:5: error:
        • Couldn't match type ‘(("https://www.googleapis.com/auth/datastore"
                                 Data.Type.Equality.== "https://www.googleapis.com/auth/cloud-platform")
                                Data.Type.Bool.|| ("https://www.googleapis.com/auth/datastore"
                                                   Data.Type.Equality.== "https://www.googleapis.com/auth/datastore"))
                               Data.Type.Bool.|| Network.Google.Auth.Scope.HasScope'
                                                   '[]
                                                   '["https://www.googleapis.com/auth/cloud-platform",
                                                     "https://www.googleapis.com/auth/datastore"]’
                         with ‘'True’
            arising from a use of ‘sendReq'’
        • In the expression: sendReq' (projectsRunQuery txReq)
          In an equation for ‘runQueryReq’:
              runQueryReq txM reqT
                = sendReq' (projectsRunQuery txReq)
                where
                    txReq = maybe req (`atomically` req) txM
                    req = unTagged reqT

Unable to get past environment creation with newEnv - "Failure refreshing token from accounts.google.com/o/oauth2/v2/auth"

This happens when I run the provided example: https://github.com/brendanhay/gogol/tree/develop/examples/src/Example

I have credentials setup by gcloud in my ~/.config/gcloud/application_default_credentials.json as per default.

getting this runtime error (when executing the newEnv line):
TokenRefreshError (Status {statusCode = 400, statusMessage = "Bad Request"}) "Failure refreshing token from accounts.google.com/o/oauth2/v2/auth" Nothing

Observation: logger does not log anything, even though I used Debug/Trace mods and looking at the code (https://github.com/brendanhay/gogol/blob/develop/gogol/src/Network/Google/Internal/Auth.hs#L288-L289) it should be logging things.

I've tried printing the offending request and response and here is what i found:

Request {
  host                 = "accounts.google.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("Content-Type","application/x-www-form-urlencoded")]
  path                 = "/o/oauth2/v2/auth"
  queryString          = ""
  method               = "POST"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}

Response:

<!DOCTYPE html><html lang=en><meta charset=utf-8><meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\"><title>Error 400 (OAuth2 Error)!!1</title><style>*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{color:#222;text-align:unset;margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px;}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}pre{white-space:pre-wrap;}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}</style><div id=\"af-error-container\"><a href=//www.google.com/><span id=logo aria-label=Google></span></a><p><b>400.</b> <ins>That\226\128\153s an error.</ins><p><script src='https://ssl.gstatic.com/accounts/o/3918119028-common_lib.js'></script><style>\n    #request_info_header {\n      cursor: default;\n      outline: none;\n      padding-left: 14px;\n      padding-top: 10px;\n    }\n\n    #request_info_items {\n      line-height: 18px;\n      list-style-type: none;\n      margin-top: 8px;\n      padding-left: 14px;\n    }\n\n    .param_entry {\n      margin-top: 2px;\n    }\n\n    .goog-zippy-expanded,\n    .goog-zippy-collapsed {\n      list-style: none;\n      padding: 2px 0 1px 15px;\n      position: relative;\n    }\n\n    .goog-zippy-expanded:before {\n      content: url(https://ssl.gstatic.com/ui/v1/zippy/arrow_down.png);\n      left: 1px;\n      position: absolute;\n      top: 7px;\n    }\n\n    .goog-zippy-collapsed:before {\n      content: url(https://ssl.gstatic.com/ui/v1/zippy/arrow_right.png);\n      left: 3px;\n      position: absolute;\n      top: 6px;\n    }\n  </style><p id=\"errorCode\"><b>Error: invalid_request</b></p><p id=\"errorDescription\">Required parameter is missing: response_type</p><p id=\"errorUri\"><a target=\"_blank\" href=\"http://code.google.com/apis/accounts/docs/OAuth2.html\">Learn more</a></p><div id=\"request_info_header\">Request Details<ul id=\"request_info_items\"><li class=\"param_entry\" id=\"param_entry_0\">grant_type=refresh_token</li><li class=\"param_entry\" id=\"param_entry_1\">client_id=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com</li><li class=\"param_entry\" id=\"param_entry_2\">client_secret=d-FL95Q19q7MQmFpd7hHD0Ty</li><li class=\"param_entry\" id=\"param_entry_3\">refresh_token=1/1Yrj_4sULbMQca3rqX3BO919jyj2JR1LDGXy9M3Y9Bw</li></ul></div><script type=\"text/javascript\">lso.doZippy(\"request_info_header\", \"request_info_items\");</script><p> <ins>That\226\128\153s all we know.</ins></div>

viewing this response in a browser tells the following story:

Error: invalid_request
Required parameter is missing: response_type

Request Details
grant_type=refresh_token
client_id=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com
client_secret=<CLEANED>
refresh_token=<CLEANED>

usersDraftsList in gogol-gmail discards first argument

Looks like you may not want the argument at all here:

usersDraftsList
    :: Text
    -> UsersDraftsList
usersDraftsList pUdlUserId_ =
    UsersDraftsList
    { _udlUserId = "me"
    , _udlPageToken = Nothing
    , _udlMaxResults = 100
    }

FromFile Credentials not working with service_account type

gogol is fantastically exciting! (But trying to stumble through OAuth is painful... can the core gogol lib walk us through this a bit more for newbies?)

Here's my current stumbling block. Currently the docs on the Credentials data type say that its ok to use "service_account" credentials:

Load the Application Default Credentials from a specific file path. The file can be formatted as either a service_account or an authorized_user.

Ok, so I followed Google's instructions for "Application Default Credentials", and got a json file with a private key in it, which says it is in fact of type "service_account". Specifically, it's of the form:

{
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "[email protected]",
  "client_id": "....apps.googleusercontent.com",
  "type": "service_account"
}

Then for my Credentials data type I point it at this file with FromFile "blah.json". But when trying to execute newEnv I get this error:

mytest-exe: InvalidFileError "blah.json" "key \"client_secret\" not present"

Hmm, that looks like it's trying to use the other form of credentials, not the "service_account" type...

CC @craigcitro

Where can I "exchange code for token"

Google's OAuth docs show pictures of the back-and-forth protocol to get user consent, get a response code, and exchange it for a token that can actually be used for API calls.

(I've been doing this for a while with handa-gdata but without knowing how it works.)

Now I'm staring at the haddock Indices for gogol, gogol-core, and gogol-oauth2 and I see very little mention of "token"s. Where are the modules/types to:

form the request URL for user consent?
exchange the code for a token (type OAuthToken)?

authorized_user token is not refresh correctly

First of all, thank all contributors for this great lib!

I am learning Haskell, so bear with me. I encountered TokenRefreshError (Status {statusCode = 400, statusMessage = "Bad Request"}) "Failure refreshing token from accounts.google.com/o/oauth2/v2/auth" Nothing, when I try bigquery project list.

code looks like

example :: IO BigQuery.ProjectList
example = do
    lgr  <- Google.newLogger Google.Debug stdout
    m <- liftIO (newManager tlsManagerSettings) :: IO Manager
    c <- Google.getApplicationDefault m

    env  <- Google.newEnvWith c lgr m <&>
          (Google.envLogger .~ lgr)
        . (Google.envScopes .~ BigQuery.bigQueryScope)

    runResourceT . Google.runGoogle env $ Google.send BigQuery.projectsList

main :: IO ()
main = do
    projects <- example
    print projects

i am using authorized_user type credential generated by gcloud auth application-default login, i verified the credential is loaded and it is FromUser, but the refresh endpoint got invoked is

[Client Request] {
  host      = accounts.google.com:443
  secure    = True
  method    = POST
  timeout   = ResponseTimeoutDefault
  redirects = 10
  path      = /o/oauth2/v2/auth
  query     = 
  headers   = content-type: application/x-www-form-urlencoded

but the correct endpoint should be https://www.googleapis.com/oauth2/v4/token accord to this.

i found there is tokenRequest and accountsRequest functions, it seems tokenRequest should be the right one for authorized user token, and exchange is invoking tokenRequest. So I cannot find where is the problem. (I tried on two MacOs machines)

Missing `refresh_token` when parsing Service Account JSON

Given the following code:

{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE OverloadedStrings #-}
module Lib
    ( fetchSpreadsheetValues
    ) where

import System.Environment
import System.IO (stdout)
import Network.Google(newLogger, newEnvWith, envLogger, LogLevel(Debug), envScopes)
import Network.Google.Auth.ApplicationDefault(fromJSONCredentials)
import Network.Google.Resource.Sheets.Spreadsheets.Get(spreadsheetsGet, sgAccessToken)
import Network.Google.Sheets(spreadsheetsScope)
import Network.HTTP.Client(newManager, defaultManagerSettings)
import Control.Lens           ((&), (.~), (<&>), (?~))
import qualified Data.ByteString.Lazy.Char8 as B
import qualified Data.Text as T

fetchSpreadsheetValues :: IO ()
fetchSpreadsheetValues = do
    serviceAccountJson <- getEnv "SERVICE_ACCOUNT_JSON"
    let credentials = fromJSONCredentials $ B.pack serviceAccountJson
    case credentials of
        Left errMessage -> putStrLn errMessage
        Right creds -> do
            apiKey <- getEnv "API_KEY"
            spreadSheetId <- getEnv "SPREADSHEET_ID"
            let getRequest = spreadsheetsGet $ T.pack spreadSheetId
            let authorizedRequest = sgAccessToken .~ (Just $ T.pack apiKey) $ getRequest

            manager <- newManager defaultManagerSettings
            lgr <- newLogger Debug stdout
            env <- (newEnvWith creds lgr manager) <&> (envScopes .~ spreadsheetsScope)

            putStrLn apiKey

I'm getting this output

Failed parsing service_account: Error in $: Unable to parse key contents from "private_key", Failed parsing authorized_user: Error in $: key "refresh_token" not present

Any idea what's going on? The JSON string in the SERVICE_ACCOUNT_JSON environment variable parses fine in a python shell into the equivalent of the file downloaded from the Google IAM console.

affiliates and classroom 0.2.0: Ambiguous occurrence ‘Link’

In Stackage Nightly:

Preprocessing library gogol-affiliates-0.2.0...
[ 1 of 14] Compiling Network.Google.Affiliates.Types.Sum ( gen/Network/Google/Affiliates/Types/Sum.hs, dist/build/Network/Google/Affiliates/Types/Sum.o )
[ 2 of 14] Compiling Network.Google.Affiliates.Types.Product ( gen/Network/Google/Affiliates/Types/Product.hs, dist/build/Network/Google/Affiliates/Types/Product.o )

gen/Network/Google/Affiliates/Types/Product.hs:380:8: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Affiliates/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Affiliates.Types.Product.Link’,
                             defined at gen/Network/Google/Affiliates/Types/Product.hs:310:1

gen/Network/Google/Affiliates/Types/Product.hs:407:22: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Affiliates/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Affiliates.Types.Product.Link’,
                             defined at gen/Network/Google/Affiliates/Types/Product.hs:310:1

gen/Network/Google/Affiliates/Types/Product.hs:412:26: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Affiliates/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Affiliates.Types.Product.Link’,
                             defined at gen/Network/Google/Affiliates/Types/Product.hs:310:1

gen/Network/Google/Affiliates/Types/Product.hs:418:28: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Affiliates/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Affiliates.Types.Product.Link’,
                             defined at gen/Network/Google/Affiliates/Types/Product.hs:310:1
:

Preprocessing library gogol-classroom-0.2.0...
[ 1 of 46] Compiling Network.Google.Classroom.Types.Sum ( gen/Network/Google/Classroom/Types/Sum.hs, dist/build/Network/Google/Classroom/Types/Sum.o )
[ 2 of 46] Compiling Network.Google.Classroom.Types.Product ( gen/Network/Google/Classroom/Types/Product.hs, dist/build/Network/Google/Classroom/Types/Product.o )

gen/Network/Google/Classroom/Types/Product.hs:641:8: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Classroom/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Classroom.Types.Product.Link’,
                             defined at gen/Network/Google/Classroom/Types/Product.hs:625:1

gen/Network/Google/Classroom/Types/Product.hs:650:24: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Classroom/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Classroom.Types.Product.Link’,
                             defined at gen/Network/Google/Classroom/Types/Product.hs:625:1

gen/Network/Google/Classroom/Types/Product.hs:657:15: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Classroom/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Classroom.Types.Product.Link’,
                             defined at gen/Network/Google/Classroom/Types/Product.hs:625:1

gen/Network/Google/Classroom/Types/Product.hs:661:17: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Classroom/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Classroom.Types.Product.Link’,
                             defined at gen/Network/Google/Classroom/Types/Product.hs:625:1

gen/Network/Google/Classroom/Types/Product.hs:664:19: error:
    Ambiguous occurrence ‘Link’
    It could refer to either ‘Network.Google.Prelude.Link’,
                             imported from ‘Network.Google.Prelude’ at gen/Network/Google/Classroom/Types/Product.hs:21:1-39
                             (and originally defined in ‘servant-0.10:Servant.Utils.Links’)
                          or ‘Network.Google.Classroom.Types.Product.Link’,
                             defined at gen/Network/Google/Classroom/Types/Product.hs:625:1
:
:

Help with Env/HasScope stuff

So I'm trying to run all the MonadGoogle functions I've created, but I get the following error when I try to run everything using runResourceT . runGoogle (storeEnv :: Env '[AuthDatastore])

/Users/rune/IdeaProjects/paychan-datastore/src/Test.hs:40:62: error:
    • Couldn't match type ‘Network.Google.Auth.Scope.HasScope'
                             '[AuthDatastore] (Scopes BeginTransactionResponse)’
                     with ‘'True’
        arising from a use of ‘testDB’
    • In the second argument of ‘($)’, namely ‘testDB pid tstData’
      In a stmt of a 'do' block:
        runResourceT . runGoogle (env :: Env '[AuthDatastore])
        $ testDB pid tstData
      In the expression:
        do { putStrLn $ "Using project: " ++ cs pid;
             env <- defaultAppDatastoreEnv;
             tstData <- genTestData numPayments;
             runResourceT . runGoogle (env :: Env '[AuthDatastore])
             $ testDB pid tstData }

Relevant snippet (error appears at testDB pid tstData in runPaymentTest - line 40 in this file):

type AuthDatastore = "https://www.googleapis.com/auth/datastore"

runPaymentTest :: ProjectId -> Word -> IO Int
runPaymentTest pid numPayments = do
    putStrLn $ "Using project: " ++ cs pid
    storeEnv <- defaultAppDatastoreEnv
    tstData  <- genTestData numPayments
    -- Run
    runResourceT . runGoogle (storeEnv :: Env '[AuthDatastore]) $
        -- ERROR: "Couldn't match type
        --    HasScope' '[AuthDatastore] (Scopes BeginTransactionResponse)’ with ‘'True’"
        testDB pid tstData

testDB :: ( MonadCatch m
          , MonadGoogle '[AuthDatastore] m
          ,    HasScope '[AuthDatastore] BeginTransactionResponse
          ,    HasScope '[AuthDatastore] LookupResponse
          ,    HasScope '[AuthDatastore] RollbackResponse
          ,    HasScope '[AuthDatastore] CommitResponse )
       => ProjectId -> Pay.ChannelPairResult -> m Int
testDB pid Pay.ChannelPairResult{..} = do
    let sampleRecvChan = Pay.recvChan resInitPair
        sampleKey = Pay.getSenderPubKey sampleRecvChan
        paymentList = reverse $ init resPayList
    DB.insertChan pid sampleRecvChan
    -- Safe lookup + update/rollback
    res <- M.forM paymentList (doPayment pid sampleKey)
    return $ length res

defaultAppDatastoreEnv :: IO (Env '[AuthDatastore])
defaultAppDatastoreEnv = do
    manager <- HTTP.newManager HTTP.tlsManagerSettings
    logger <- Google.newLogger Google.Error stderr
    Google.newEnv <&>
        (envLogger .~ logger) .
        (envScopes .~ datastoreScope) .
        (envManager .~ manager)

Relevant snippet from gogol-datastore:

gogol/gogol-datastore/gen/Network/Google/Resource/Datastore/Projects/BeginTransaction.hs

Line 167 in da4fc87

instance GoogleRequest ProjectsBeginTransaction where

AdWords API

Is there an easy way to get the AdWords API included in this?

I think it is a little different to the other Google APIs so maybe it is not possible to use the same approach.

Doens't compile with base-4.7 (GHC-7.8)

src/Network/Google/Types.hs:318:34:
    Not in scope: ‘foldMap’
    Perhaps you meant one of these:
      ‘CL.foldMap’ (imported from Data.Conduit.List),
      ‘ifoldMap’ (imported from Control.Lens),
      ‘CL.foldMapM’ (imported from Data.Conduit.List)

made bound revisions on Hackage

Request: Support contacts (People API)

Hey, thank you for creating this library! I was wondering if you could wrap the People API, i.e. Google Contacts? The Google Discovery Service link is here:

https://developers.google.com/apis-explorer/#p/people/v1/

`Byte` should be base64Url encoded `ByteString` not `Word8`

First of all thanks for your excellent work. I noticed this problem while I was trying to use gmail-api. mraw should be base64Url encoded ByteString instead of a Word8. Google description here: https://developers.google.com/discovery/v1/type-format . I think other gogol-* libraries affected as well by this.

Make a release

Hi!

I would really appreciate a new hackage release of all libs, would help using gogol with latest GHC and servant.

Thank you!

Support `validMasterVersions` in ServerConfig

Currently the ServerConfig looks like this:

data ServerConfig = ServerConfig'
    { _scValidNodeVersions     :: !(Maybe [Text])
    , _scDefaultImageFamily    :: !(Maybe Text)
    , _scValidImageFamilies    :: !(Maybe [Text])
    , _scDefaultClusterVersion :: !(Maybe Text)
    } deriving (Eq,Show,Data,Typeable,Generic)

Hitting gcloud container get-server-config --log-http returns a result like this:

{
  "defaultClusterVersion": …,
  "validNodeVersions": […],
  "defaultImageType": …,
  "validImageTypes": [...],
  "validMasterVersions": [...]
}

Can we get _scValidMasterVersions in the datatype? Thanks!

Storage: How to fetch the contents of an object?

Thanks for creating gogol! 😄

My goal is to download an object from Storage and write its contents to a file. But I'm stumped by the use of ResumableSources.

Calling download works just fine. It finds the object, yielding a Stream (aka: ResumableSource (ResourceT IO) ByteString).

But I don't know how to work with this Stream. I've tried to use Data.Conduit.($$+-) to put the contents in a file, to no avail. Nothing at all happens.

Here's my code:

fetchFile key = do
  lgr ← newLogger Debug stdout
  env ← newEnv <&> (envLogger .~ lgr) . (envScopes .~ storageReadWriteScope)
  runResourceT . runGoogle env $ do
    -- Fetch file into Stream
    stream ← download (objectsGet bucketName key)
    -- Attempt to write stream into file. Nothing happens.
    return $ stream $$+- sinkFile "./my_file_name"

Could you guide me in the right direction, please?

Thanks!

assistance needed: trying to hit beta gcloud endpoints

I'm trying to hit this API endpoint:
https://us.gcr.io/v2/<gcp_project_name>/<my_image_name>/tags/list (the endpoint that the following CLI command hits: gcloud beta container images list-tags us.gcr.io/<gcp_project_name>/<my_image_name>)

I realize that gogol likely does not support this API, but I suspect that I could reuse gogol's auth mechanism to authenticate with this endpoint.
I don't mind hand-crafting a gogol request for this endpoint, but would really appreciate an advice or assistance as to how to go about it

Thanks

Firebase Cloud Messaging support

Thanks for this wonderful package. Any plans to support FCM?

Could you provide a simple example of how gogol-gmail should be used?

I understand this is very early stage work and there is a lot to do, but I am really like a hen before an egg while looking at http://hackage.haskell.org/package/gogol-gmail-0.1.0. I have no idea how to start using it. In particular, I don't see how this works with OAuth2 authentication flow...

Thanks for the great work in making this available in Haskell!

Feature request: support viewing and setting labels

Currently only tags are supported. It used to be that tags and labels are coupled somehow. It's no longer the case: https://cloud.google.com/compute/docs/labeling-resources#labels_tags

Installed application credentials are inadequate.

Unless I'm missing something, the InstalledApplication credentials can only handle the initial authorization, and does not allow to store refresh token or use refresh token to authenticate.

The "Installed application" flow should be something like

if have refresh token - use it to initialize store, otherwise do the authorization token dance and get refresh/access token.
Run your app, which refreshes access token if needed
Store the refresh token since it could have been changed when refreshing.

So currently missing building blocks would be

data Credentials s = FromRefreshToken RefereshToken
                | ...
retrieveStoredAuth :: MonadIO m => Store s -> m (Auth s)

And installed app usage would be like

initialiseCreds :: OAuthClient -> Logger -> Manager -> IO (Credentials s)
initialiseCreds client logger manager = do
    refreshToken <- readSavedToken
    case refreshToken of
      Just t -> pure (FromRefreshToken t)
      Nothing -> do
          T.putStrLn (formUrl ...)
          code <- OAuthCode <$> T.getLine
          pure (FromClient client code)

main = do
    logger <- newLogger
    manager <- newManager
    creds <- initializeCreds client logger manager
    env <- newEnvWith ....
    runApp
    auth <- retreiveStoreAuth (env^.envStore)
    saveRefreshToken auth

On top of that we can build something more like a google python api does - let users store this auth information in a json file before program exists, and try to read it same way as ServiceAccount or AuthorizedUser does.

{ "client_id" : "..."
, "client_secret" : "..."
, "access_token" : ".."
, "token_expiration" : "2016-12-01T..."
, "refresh_token" : "..."
}

For all use cases google recommends storing refresh tokens and reusing them, so this may be useful not only for "Installed application" case.

What do you think?

Replace prototype generator with valid JSON schema compatible types/parser etc.

The current implementation of the parser/AST reification is just a hack port from the amazonka generator.

Next step is to rewrite and conform to the JSON Schema specification.

Wrong type for Gogol.BigQuery.jclQuote

The jclQuote lens should actually be optional. Currently it is a required text, but it should have been Maybe Text.

The consequence of the current implementation is that it is impossible to process a JSON payload without getting an error about this field. The error is:

Only CSV imports may specify a quote character

Setting jclQuote to an empty Text has no effect on the error.

OOMs from fresh build

When doing a clean stack build:

gogol-datastore-0.0.1: install
gogol-dataflow-0.0.1: install
Progress: 102/158
--  While building package gogol-compute-0.0.1 using:
      /home/gambogi/.stack/setup-exe-cache/setup-Simple-Cabal-1.22.4.0-x86_64-linux-ghc-7.10.2 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.22.4.0/ build lib:gogol-compute --ghc-options  -ddump-hi -ddump-to-file
    Process exited with code: ExitFailure 1
    Logs have been written to: /home/gambogi/projects/gogol/.stack-work/logs/gogol-compute-0.0.1.log

    Configuring gogol-compute-0.0.1...
    Preprocessing library gogol-compute-0.0.1...
    [  1 of 194] Compiling Network.Google.Compute.Types.Sum ( gen/Network/Google/Compute/Types/Sum.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Network/Google/Compute/Types/Sum.o )
    [  2 of 194] Compiling Network.Google.Compute.Types.Product ( gen/Network/Google/Compute/Types/Product.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Network/Google/Compute/Types/Product.o )
    ghc: out of memory (requested 1048576 bytes)

Not sure if this is primarily a gogol, stack, cabal, or ghc issue, but I figured I'd start here.

Respect parameterOrder

Google Cloud Dataproc API

Can you generate library for google cloud dataproc api?

Api key based auth

Would it be possible to add apikey auth? For example, Translate API doesn't need token exchange, but works by just including a secret key in the url (https://cloud.google.com/translate/v2/getting_started?csw=1).

Note: I tried from devel branch head with a service account, and got Forbidden, so I assume Translate API only accepts the api key or OAuth (but the latter is not needed, since there is no user data here).

Slightly related: is it possible to pass quotaUser and/or userIp parameter for that API? See https://support.google.com/cloud/answer/6158858?hl=en&ref_topic=6262490 .

Build failure with http-client 0.5

> /tmp/stackage-build8$ stack unpack gogol-0.1.0
Unpacked gogol-0.1.0 to /tmp/stackage-build8/gogol-0.1.0/
> /tmp/stackage-build8/gogol-0.1.0$ runghc -clear-package-db -global-package-db -package-db=/var/stackage/work/builds/nightly/pkgdb Setup configure --package-db=clear --package-db=global --package-db=/var/stackage/work/builds/nightly/pkgdb --libdir=/var/stackage/work/builds/nightly/lib --bindir=/var/stackage/work/builds/nightly/bin --datadir=/var/stackage/work/builds/nightly/share --libexecdir=/var/stackage/work/builds/nightly/libexec --sysconfdir=/var/stackage/work/builds/nightly/etc --docdir=/var/stackage/work/builds/nightly/doc/gogol-0.1.0 --htmldir=/var/stackage/work/builds/nightly/doc/gogol-0.1.0 --haddockdir=/var/stackage/work/builds/nightly/doc/gogol-0.1.0 --flags=
Configuring gogol-0.1.0...
> /tmp/stackage-build8/gogol-0.1.0$ runghc -clear-package-db -global-package-db -package-db=/var/stackage/work/builds/nightly/pkgdb Setup build
Building gogol-0.1.0...
Preprocessing library gogol-0.1.0...
[ 1 of 13] Compiling Network.Google.Internal.Multipart ( src/Network/Google/Internal/Multipart.hs, dist/build/Network/Google/Internal/Multipart.o )

src/Network/Google/Internal/Multipart.hs:17:1: warning: [-Wunused-imports]
    The qualified import of ‘Data.CaseInsensitive’ is redundant
      except perhaps to import instances from ‘Data.CaseInsensitive’
    To import instances alone, use: import Data.CaseInsensitive()
[ 2 of 13] Compiling Network.Google.Compute.Metadata ( src/Network/Google/Compute/Metadata.hs, dist/build/Network/Google/Compute/Metadata.o )

src/Network/Google/Compute/Metadata.hs:229:11: error:
    Not in scope: ‘Client.checkStatus’
    Module ‘Network.HTTP.Conduit’ does not export ‘checkStatus’.

gogol-datastore missing colon in request

I'm currently experimenting with the gogol-datastore library, and experienced an bug. Here my example code:

{-# LANGUAGE OverloadedStrings #-}
module Main where

import Network.Google as Google
import Network.Google.Datastore
import Control.Lens
import System.IO              (stdout)
import Control.Monad.Trans    (liftIO)


main = do
    let papegoPartition = partitionId & piProjectId ?~ "my-project-id" & piNamespaceId ?~ "dev"
    logger <- Google.newLogger Google.Debug stdout
    env <- Google.newEnv <&> (Google.envLogger .~ logger) . Google.allow datastoreScope

    let proofId = key
                    & kPartitionId ?~ papegoPartition
                    & kPath .~ [pathElement & peKind ?~ "Proof" & peName ?~ "proof_abc"]
        proofEntity = entity
                        & eKey ?~ proofId
    putStrLn "Entity:"
    print proofEntity


    let upsertProof = commitRequest
                        & crMutations .~ [mutation & mUpsert ?~ proofEntity]
                        & crMode ?~ NonTransactional

    print upsertProof

    runResourceT . runGoogle env $ do
        resp <- Google.send (projectsCommit upsertProof "my-project-id")
        liftIO $ print resp

    putStrLn "Done!"

Results into a 404 with this debug output:

[Client Request] {
  host      = datastore.googleapis.com:443
  secure    = True
  method    = POST
  timeout   = Just 70000000
  redirects = 10
  path      = /v1beta3/projects/my-project-idcommit
  query     = ?pp=true&alt=json
  headers   = authorization: Bearer XXX; accept: application/json; content-type: application/json
  body      = {"mutations":[{"upsert":{"key":{"partitionId":{"namespaceId":"dev","projectId":"my-project-id"},"path":[{"kind":"Proof","name":"proof_abc"}]}}}],"mode":"NON_TRANSACTIONAL"}
}
[Client Response] {
  status  = 404 Not Found
  headers = date: Tue, 26 Jul 2016 13:29:12 GMT; content-type: text/html; charset=UTF-8; server: ESF; content-length: 1616; x-xss-protection: 1; mode=block; x-frame-options: SAMEORIGIN; x-content-type-options: nosniff; alternate-protocol: 443:quic; alt-svc: quic=":443"; ma=2592000; v="36,35,34,33,32,31,30,29,28,27,26,25"
}

The Problem is the path: /v1beta3/projects/my-project-idcommit which should be /v1beta3/projects/my-project-id:commit (see REST-API Reference)

Changing

 resp <- Google.send (projectsCommit upsertProof "my-project-id")

resp <- Google.send (projectsCommit upsertProof "my-project-id:")

solves it as quick workaround.