Reference docs : https://gapic-generator-python.readthedocs.io/en/stable/getting-started/local.html

Running the following command to setup the project locally produces error
protoc google/cloud/vision/v1/image_annotator.proto --proto_path=../api-common-protos/ --proto_path=. --python_gapic_out=../dest/

Output

google/longrunning/operations.proto:30:8: Option "php_namespace" unknown.
google/cloud/vision/v1/image_annotator.proto: Import "google/longrunning/operations.proto" was not found or had errors.
google/cloud/vision/v1/image_annotator.proto:63:16: "google.longrunning.Operation" is not defined.

Flake raises warnings about unused imports.

See https://github.com/googleapis/google-cloud-python-private/pull/504

google/cloud/workflows_v1alpha1/services/workflows/pagers.py:2:1: F401 'copy' imported but unused
google/cloud/workflows_v1alpha1/services/workflows/client.py:3:1: F401 'typing.Mapping' imported but unused
google/cloud/workflows_v1alpha1/services/workflows/client.py:3:1: F401 'typing.Optional' imported but unused
google/cloud/workflows_v1alpha1/services/workflows/transports/grpc.py:2:1: F401 'typing.Sequence' imported but unused
google/cloud/workflows_v1alpha1/services/workflows/transports/grpc.py:2:1: F401 'typing.Tuple' imported but unused
tests/unit/workflows_v1alpha1/test_workflows.py:10:1: F401 'google.api_core.operation' imported but unused
tests/unit/workflows_v1alpha1/test_workflows.py:18:1: F401 'google.protobuf.empty_pb2 as empty' imported but unused

Address comments in googleapis/google-cloud-python#5472

The version regex probably fails some corner cases:

• alpha10, beta10, etc.
• alpha or beta with no number after it.

The introduction of schema.wrappers.OperationType has unexpected side effects in grpc.py.j2 (where you actually do need access to the google.longrunning.Operation pb2 type, and also has side effects in attempting to handle module writing. At the moment, this makes LRO flatly not work.

Fix this.

The title tag contains a helpful docstring.

---
type: com.google.api.codegen.SampleConfigProto
schema_version: 1.2.0
samples:
- service: google.cloud.language.v1.LanguageService
  rpc: AnalyzeSentiment
  title: Analyze sentiment of text
  region_tag: language_sentiment_text
  description: Analyze sentiment of text

The sampletester uses the generated manifest to map generated sample files to test cases.

When running...

protoc \
    --proto_path={wherever}/api-common-protos/ --proto_path=. \
    kiosk.proto \
    --python_out=. --pyclient_out=.

...the python_out and pyclient_out components do not agree on where kiosk_pb2.py ends up. The internal plugin places it in the main directory (kiosk_pb2.py), but the GAPIC tries to import it based on its package (kiosk/kiosk_pb2.py).

This means that trying to run the code would raise ImportError because of an invalid import.

The generator currently exposes an idiom to templates on Proto objects, a property spelled top, which returns a shim of the proto that is only aware of messages and enums at the top level (for the purposes of iterating only over top-level messages and enums).

There are two problems with this:

• It is easy to forget to invoke proto.top.messages instead of proto.messages (and the resulting bugs from making this mistake can be difficult to debug if they are centered around imports).
• There is virtually no reason to ever iterate over messages and enums and not want them restricted to the top level.

Therefore, refactor this to remove top, still keeping messages and enums from all levels included, but override their __iter__ to only yield top-level messages and enums.

pagers.py is missing an import. I generated the Cloud Billing Budgets API with the latest release.

https://github.com/googleapis/google-cloud-python-private/blob/14acc8f7dce0da8fd30fec5a9d8d25febcf7d477/billing_budgets/google/cloud/billing/budgets_v1alpha1/services/budget_service/pagers.py#L5
pagers.py

from google.cloud.billing.budgets_v1alpha1.types import budget_service

Other files in the library have the budget_model import. https://github.com/googleapis/google-cloud-python-private/blob/14acc8f7dce0da8fd30fec5a9d8d25febcf7d477/billing_budgets/google/cloud/billing/budgets_v1alpha1/services/budget_service/client.py#L12-L13
client.py

from google.cloud.billing.budgets_v1alpha1.types import budget_model
from google.cloud.billing.budgets_v1alpha1.types import budget_service

See https://github.com/googleapis/google-cloud-python-private/pull/503/files for the full generation.

EDIT: Added code snippets for those who don't have access to the repo.

See: googleapis/google-cloud-python#7220 for how this manifests in the BigQuery Storage API.

The manifest should not include a sample's region tag if the tag is the empty string.

From: googleapis/google-cloud-python#7043

The members of a collections.namedtuple have no semantic information in their docstrings:

>>> import collections
>>> Foo = collections.namedtuple("Foo", ["bar", "baz", "qux"], verbose=True)
class Foo(tuple):
    'Foo(bar, baz, qux)'

    __slots__ = ()

    _fields = ('bar', 'baz', 'qux')

    def __new__(_cls, bar, baz, qux):
        'Create new instance of Foo(bar, baz, qux)'
        return _tuple.__new__(_cls, (bar, baz, qux))

    @classmethod
    def _make(cls, iterable, new=tuple.__new__, len=len):
        'Make a new Foo object from a sequence or iterable'
        result = new(cls, iterable)
        if len(result) != 3:
            raise TypeError('Expected 3 arguments, got %d' % len(result))
        return result

    def __repr__(self):
        'Return a nicely formatted representation string'
        return 'Foo(bar=%r, baz=%r, qux=%r)' % self

    def _asdict(self):
        'Return a new OrderedDict which maps field names to their values'
        return OrderedDict(zip(self._fields, self))

    def _replace(_self, **kwds):
        'Return a new Foo object replacing specified fields with new values'
        result = _self._make(map(kwds.pop, ('bar', 'baz', 'qux'), _self))
        if kwds:
            raise ValueError('Got unexpected field names: %r' % kwds.keys())
        return result

    def __getnewargs__(self):
        'Return self as a plain tuple.  Used by copy and pickle.'
        return tuple(self)

    __dict__ = _property(_asdict)

    def __getstate__(self):
        'Exclude the OrderedDict from pickling'
        pass

    bar = _property(_itemgetter(0), doc='Alias for field number 0')

    baz = _property(_itemgetter(1), doc='Alias for field number 1')

    qux = _property(_itemgetter(2), doc='Alias for field number 2')

Common protos are still pb2 modules.

Ideally we should use wrapped modules. Augment googleapis-common-protos (the 1.6.0 beta) by adding wrapped modules. (We should leave the old ones for back-compat.)

Robust request setup and response handling validation require analysis of the corresponding protobuf message types, e.g. to make sure that referenced attributes exist, that enums are assigned valid variants, and so forth.

Cleanup tracker for converting macro and variable names to snake_case.

Include baseline sample files generated via a combination of a dummy api proto and some reasonably feature-comprehensive sample configs.

In an ideal world, these are autogenerated, either via git-hook magic or ci magic. If they cannot be autogenerated, then they are used in comparison tests so that editors are forced to keep them up to date.

What

Currently client library documentation contains autogenerated links to cloud.google.com. The URLs are derived heuristically, and are sometimes incorrect. For example, autogenerated Natural Language docs point to https://cloud.google.com/language/ instead of the expected https://cloud.google.com/natural-language/ (context).

How

These URLs are already specified by internal configuration metadata. We need to open source this configuration and consume it in the code generation process.

Right now, our Python auto-gen creates a package based on the longrunning service YAML, which does a few things that are strange/exceptional:

• It writes a package that is not really a package, and instead a subset of it is put within google.gax. This might be a reasonable strategy in principle (if we did it everywhere), but it is inconsistent with how every other package works.
• Our packages print code which expect them to be a full packge, and so this raises an exception in longrunning's case. See googleapis/gax-python#171.
• This also makes tracking metrics difficult, because there is no obvious way to send in the gapic/ metric right now.

We either need to fix the longrunning special case to not break, or separate longrunning out into a separate package.

Let's investigate why the generator depends on the grpc runtime.

What

The current behavior is to declare all gRPC and GAPIC packages as namespace packages except the version package; that is, for an API living in the package google.cloud.foo.bar.v1, we will declare all packages as namespace packages except v1. This behavior originated in Packman and has been ported to Toolkit.

Currently, it's always necessary to declare google, google.cloud, google.cloud.grpc, and google.cloud.gapic as namespace packages, as these are shared across many packages on PyPI that may be installed together. Declaring deeper packages as namespace packages is not necessary, but not harmful either, AFAIK, and opens up the possibility of splitting up types across multiple packages, which may be an important future use case (as we've seen with the logging AuditData types).

Do we need special logic for the version package?

Why

• Not declaring the version package as a namespace package allows us to declare package-level types in the version-level __init__.py. This may have been the original rationale for this strategy
• We don't currently use the __init__.py to define anything, so we can simplify the packaging logic by simply declaring all packages as namespace packages.

cc: @bjwatson @lukesneeringer

If an API has no namespace (e.g. package kiosk; as opposed to package google.kiosk;), the resulting package name is silly: -kiosk.

Tie together the samplegen functionality into the generator, read in the sample_config, and generate samples.

What:

• Move content of __init__ docstring to class docstring for a service (see http://pythonhosted.org/gax-google-pubsub-v1/_modules/google/pubsub/v1/publisher_api.html#PublisherApi).
• Remove the superfluous "Constructor" title
• Remove the unnnecessary "Returns" section

Why:

Per @jonparrott, "most libraries actually document the constructor in the class docstring, so I would advise putting the args section here into the class docstring and omitting the docstring for the __init__ method. Typically you don't want to separately document __init__ unless there is more than one way to construct the class (e.g. @classmethod helper constructors)."

PEP 257 indicates that the __init__ method should have a docstring, but the class docstring is effectively the __init__ docstring.

How:

Update appropriate generator .snip code.

Ensure that Sphinx is configured correctly: http://www.sphinx-doc.org/en/stable/ext/autodoc.html#confval-autoclass_content

Create a corresponding issue in https://github.com/googleapis/gax-python to do whatever we do here, for consistency.

• Get autogen unit test coverage to 100%
• Add cover session to nox.py

If the 'response' key is not present in a sample spec, the generator must infer that the correct response handling mechanism is to print a dump of the response object in an idiomatic format.

Running in kiosk/protos (after having run COMPILE-PROTOS.sh in the parent directory), I get this message:

$ protoc kiosk.proto --proto_path=./api-common-protos --proto_path=. --pyclient_out=.
kiosk/__init__.py: Tried to write the same file twice.

This is with no configuration options, so I'm not surprised that it failed, but was surprised by the type of problem. Are there error messages that I can watch for? (protoc plugins can write errors to stdout)

What:

Improve documentation for service constructor arguments, and make it clear that they are all optional.

Also fix links to supporting objects, such as grpc.beta.implementations.Channel, grpc.beta.implementations.ClientCredentials, and google.gax.construct_settings() in the Sphinx rendered documentation.

This is not the same as googleapis/gapic-generator#197, which refers to method call options.

Why:

They are standard across our auto-generated Python surfaces, and the purpose of the arguments is not obvious. For example, why would I change service_path or port?

How:

This is mostly a documentation effort. @omaray do you have any thoughts on this?

This 'except' block is the only line missing unit test coverage.

try:
    _client_info = gapic_v1.client_info.ClientInfo(
        gapic_version=pkg_resources.get_distribution("google-cloud-workflows").version
    )
except pkg_resources.DistributionNotFound:
    _client_info = gapic_v1.client_info.ClientInfo()

https://github.com/googleapis/google-cloud-python-private/blob/530c9c710b768c444eba4bd7c8639096affae89a/workflows/google/cloud/workflows_v1alpha1/services/workflows/client.py#L431

Using latest image (sha256:d3389258b42e24eea781dadefdd6a2c65d4075518ad73a034c8c0c4d19e84af1)

The comma after the last arg in the function argument list raises a syntax error only in 3.5.

class Workflows(metaclass=WorkflowsMeta):
    """Manages workflow programs."""

    def __init__(
        self,
        *,
        host: str = "workflows.googleapis.com",
        credentials: credentials.Credentials = None,
        transport: Union[str, WorkflowsTransport] = None,
    ) -> None:

__________________________________ ERROR collecting tests/unit/workflows_v1alpha1/test_workflows.py __________________________________
.nox/unit-3-5/lib/python3.5/site-packages/_pytest/python.py:498: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
.nox/unit-3-5/lib/python3.5/site-packages/py/_path/local.py:701: in pyimport
    __import__(modname)
<frozen importlib._bootstrap>:968: in _find_and_load
    ???
<frozen importlib._bootstrap>:957: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:673: in _load_unlocked
    ???
.nox/unit-3-5/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:149: in exec_module
    exec(co, module.__dict__)
tests/unit/workflows_v1alpha1/test_workflows.py:13: in <module>
    from google.cloud.workflows_v1alpha1.services.workflows import Workflows
google/cloud/workflows_v1alpha1/__init__.py:2: in <module>
    from .services.workflows import Workflows
google/cloud/workflows_v1alpha1/services/workflows/__init__.py:2: in <module>
    from .client import Workflows
E     File "/usr/local/google/home/busunkim/github/google-cloud-python-private/workflows/google/cloud/workflows_v1alpha1/services/workflows/client.py", line 59
E       ) -> None:
E       ^
E   SyntaxError: invalid syntax

Currently, if the LRO annotation is absent but google.longrunning.Operation is the response type, it will generate a non-working library. This should become an error.

Add a format-checking task in that runs autopep8 (or black) on the checked in code. If there's a non-trivial diff, error out.

There is no documentation for the fields of the

google.cloud.monitoring_v3.types.MetricDescriptor
google.cloud.monitoring_v3.types.MessageOptions
(and many others)

in the generated docmentation here:

https://google-cloud-python.readthedocs.io/en/latest/monitoring/gapic/v3/types.html

Or consider

google.cloud.redis_v1beta1.types.Operation

in

https://google-cloud-python.readthedocs.io/en/latest/redis/gapic/v1beta1/types.html

These classes have fields that are generated from protobuf messages yet there is no documentation.

My attempts to use auto completion in common IDEs like pycharm, visual studio code, emacs's elpy all fail. Is there an IDE that does auto completion successfully on these types I can use?

Thanks

For example, see http://pythonhosted.org/gax-google-pubsub-v1/google.protobuf.timestamp_pb2.html#module-google.protobuf.timestamp_pb2. This manually written sample code is poorly formatted in the Python doc.

We will probably need some kind of hint in the proto file that it's sample code, such as the markdown conventional ```.

Runs pylint on repo. If a clean bill of health is not presented, error out.

Make sure that attribute expressions, .e.g. the 'body' of the statement

{ "print", ["%s", "$resp.mollusc.cephalopod.squids"] }

refer to correctly defined attribute hierarchies based on the types of the leftmost token.

Using SSH for cloning the repos may cause permission denied error.

git clone [email protected]:googleapis/gapic-generator-python.git
git clone [email protected]:googleapis/googleapis.git
git clone [email protected]:googleapis/api-common-protos.git

They can be replaced with HTTPS version.

If you wrap a reference so that the '<' that starts a ref is on a different line than the ref it will inject spaces to the link and fail. See the following commit/line for an example:
googleapis/google-cloud-python@568779a#diff-ed5bfbb1dceb1a0c0c401ca805fa9b16L443

The following should hold true:

from google.cloud.gapic import language_v1beta2
from google.cloud.proto.language.v1beta2 import language_service_pb2

assert language_service_pb2.Document is language_v1beta2.Document

This can be easily accomplished with an __init__.py in language_v1beta2:

from google.cloud.proto.language.v1beta2.language_service_pb2 import Document, ...

__all__ = [
    'Document',
    ...
]

What:

Show how to implement client-side streaming for methods that declare either bi-di or client streaming.

Why:

We show how to stream data from the server, which is fairly obvious. However, it takes effort to figure out a good pattern for client-side streaming.

How:

Use the model to determine whether the stream keyword was declared on the request, response, or both. For example, streaming_recognize is declared to be bi-di.

For bi-di streaming, include something like the following code snippet:

>>> class Requests(object):
>>>   def __iter__(self):
>>>     while not_done:
>>>       # generate client-side streaming element
>>>       yield cloud_speech_pb2.StreamingRecognizeRequest()
>>> requests = Requests()
>>> for element in api.streaming_recognize(requests):
>>>   # process server-side streaming element
>>>   pass

The sample for just client or server streaming would be simpler.

Use the sample config and protobuf service descriptor to determine the necessary python imports for generated samples to instantiate a client and reference enum variants.

If a class docstring does not exist, then Sphinx will not generate a documentation page for that class. Arguably this is a bug in Sphinx, but it's one we need to work around.

This happens for us when the proto file does not have a comment describing the service. One example is logging_config.proto and logging_metrics.proto for the Logging API. I added service level comments in the https://github.com/googleapis/googleapis repo, but the upstream source does not have them.

We should enhance our surface generator to be robust about this, and generate a default docstring when the service does not have a comment. We should also emit a warning in the generator output.

The 1.6.0b9 version is lagged (missing e.g. google.types.expr_pb2, see googleapis/google-cloud-python#8034).

If protos are in a subpackage (a package beneath the version number), the subpackages are effectively wiped out in the directory structure.

      Returns:
           ~.operation.Operation:
               An object representing a long-running operation.
               The result type for the operation will be
               :class:`~.empty.Empty`: A generic empty message that you
               can re-use to avoid defining duplicated empty messages
               in your APIs. A typical example is to use it as the
               request or the response type of an API method. For
               instance:

                   service Foo {
                     rpc Bar(google.protobuf.Empty) returns
               (google.protobuf.Empty);     }

               The JSON representation for `Empty` is empty JSON object
               `{}`.

Sphinx doesn't seem to like the indentation.

sphinx.errors.SphinxWarning: /usr/local/google/home/busunkim/github/google-cloud-python-private/workflows/google/cloud/workflows_v1alpha1/services/workflows/client.py:docstring of google.cloud.workflows_v1alpha1.Workflows.delete_workflow:33:Block quote ends without a blank line; unexpected unindent.

• code
• pull request

We're getting coverage errors due to the google.cloud.bigquery_datatransfer module never being imported in the unit tests.

googleapis/google-cloud-python#8630 (comment)

We should have an auto-generated unit test similar to the one Tres suggests.

As mentioned in #35, there is a potential for naming collisions:

• No proto file can have the same snake-cased name as an RPC on a service.
• No proto file can import a proto file of the same name from two different directories.

I'm getting this error whenever I run the docker container:

/usr/src/gapic-generator-python/docker-entrypoint.sh: line 38: syntax error near unexpected token `;;'
/usr/src/gapic-generator-python/docker-entrypoint.sh: line 38: `      ;;'

I changed this line to convert the else if into an elif because otherwise the fi for the second if was missing, and that did the trick.

Now I don't understand if I'm doing something wrong (I think I'm not) or if this was not tested, but the currently published version in the Docker Hub does not work for me because of this.

The changes in #6 make it such that the gRPC transport class returns a callable from a @property (the underlying gRPC stub), while the HTTP transport class simply has a method (not a property) which does the thing.

We should make these consistent. There is significant value in the approach the gRPC transport now takes because it exposes the underlying gRPC stub callable, which can be called in unique ways (e.g. using .with_call or .future. The HTTP callable will not have these features, but the basic interface should remain consistent.

To speed merge and reduce churn it is proposed to defer formatting issues raised in PR #137 to a subsequent PR.