Git Product home page Git Product logo

java-datastore's Introduction

Google Cloud Datastore Client for Java

Java idiomatic client for Cloud Datastore.

Maven Stability

Quickstart

If you are using Maven with BOM, add this to your pom.xml file:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>26.40.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-datastore</artifactId>
  </dependency>

If you are using Maven without the BOM, add this to your dependencies:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-datastore</artifactId>
  <version>2.20.0</version>
</dependency>

If you are using Gradle 5.x or later, add this to your dependencies:

implementation platform('com.google.cloud:libraries-bom:26.40.0')

implementation 'com.google.cloud:google-cloud-datastore'

If you are using Gradle without BOM, add this to your dependencies:

implementation 'com.google.cloud:google-cloud-datastore:2.20.0'

If you are using SBT, add this to your dependencies:

libraryDependencies += "com.google.cloud" % "google-cloud-datastore" % "2.20.0"

Authentication

See the Authentication section in the base directory's README.

Authorization

The client application making API calls must be granted authorization scopes required for the desired Cloud Datastore APIs, and the authenticated principal must have the IAM role(s) required to access GCP resources using the Cloud Datastore API calls.

Getting Started

Prerequisites

You will need a Google Cloud Platform Console project with the Cloud Datastore API enabled.

Follow these instructions to get your project set up. You will also need to set up the local development environment by installing the Google Cloud Command Line Interface and running the following commands in command line: gcloud auth login and gcloud config set project [YOUR PROJECT ID].

Installation and setup

You'll need to obtain the google-cloud-datastore library. See the Quickstart section to add google-cloud-datastore as a dependency in your code.

About Cloud Datastore

Cloud Datastore is a fully managed, schemaless database for storing non-relational data. Cloud Datastore automatically scales with your users and supports ACID transactions, high availability of reads and writes, strong consistency for reads and ancestor queries, and eventual consistency for all other queries.

See the Cloud Datastore client library docs to learn how to use this Cloud Datastore Client Library.

See the [Google Cloud Datastore docs][cloud-datastore-activation] for more details on how to activate Cloud Datastore for your project.

See the [Datastore client library docs][datastore-client-lib-docs] to learn how to interact with the Cloud Datastore using this Client Library.

Creating an authorized service object

To make authenticated requests to Google Cloud Datastore, you must create a service object with credentials. You can then make API calls by calling methods on the Datastore service object. The simplest way to authenticate is to use Application Default Credentials. These credentials are automatically inferred from your environment, so you only need the following code to create your service object:

import com.google.cloud.datastore.Datastore;
import com.google.cloud.datastore.DatastoreOptions;

Datastore datastore = DatastoreOptions.getDefaultInstance().getService();

For other authentication options, see the Authentication page.

Storing data

Objects in Datastore are known as entities. Entities are grouped by "kind" and have keys for easy access. In this code snippet, we will create a new entity representing a person and store that data by the person's email. First, add the following imports at the top of your file:

import com.google.cloud.datastore.Entity;
import com.google.cloud.datastore.Key;
import com.google.cloud.datastore.KeyFactory;

Then add the following code to put an entity in Datastore.

KeyFactory keyFactory = datastore.newKeyFactory().setKind("Person");
Key key = keyFactory.newKey("[email protected]");
Entity entity = Entity.newBuilder(key)
    .set("name", "John Doe")
    .set("age", 51)
    .set("favorite_food", "pizza")
    .build();
datastore.put(entity);

Later, if you want to get this entity back, add the following to your code:

Entity johnEntity = datastore.get(key);

Running a query

In addition to retrieving entities by their keys, you can perform queries to retrieve entities by the values of their properties. A typical query includes an entity kind, filters to select entities with matching values, and sort orders to sequence the results. google-cloud-datastore supports two types of queries: StructuredQuery (that allows you to construct query elements) and GqlQuery (which operates using GQL syntax) in string format. In this tutorial, we will use a simple StructuredQuery.

Suppose that you've added more people to Datastore, and now you want to find all people whose favorite food is pizza. Import the following:

import com.google.cloud.datastore.Query;
import com.google.cloud.datastore.QueryResults;
import com.google.cloud.datastore.StructuredQuery;
import com.google.cloud.datastore.StructuredQuery.PropertyFilter;

Then add the following code to your program:

Query<Entity> query = Query.newEntityQueryBuilder()
    .setKind("Person")
    .setFilter(PropertyFilter.eq("favorite_food", "pizza"))
    .build();
QueryResults<Entity> results = datastore.run(query);
while (results.hasNext()) {
  Entity currentEntity = results.next();
  System.out.println(currentEntity.getString("name") + ", you're invited to a pizza party!");
}

Cloud Datastore relies on indexing to run queries. Indexing is turned on by default for most types of properties. To read more about indexing, see the Cloud Datastore Index Configuration documentation.

Updating data

Another thing you'll probably want to do is update your data. The following snippet shows how to update a Datastore entity if it exists.

KeyFactory keyFactory = datastore.newKeyFactory().setKind("keyKind");
Key key = keyFactory.newKey("keyName");
Entity entity = datastore.get(key);
if (entity != null) {
  System.out.println("Updating access_time for " + entity.getString("name"));
  entity = Entity.newBuilder(entity)
      .set("access_time", DateTime.now())
      .build();
  datastore.update(entity);
}

The complete source code can be found at UpdateEntity.java.

Complete source code

In AddEntitiesAndRunQuery.java we put together all the code to store data and run queries into one program. The program assumes that you are running on Compute Engine or from your own desktop. To run the example on App Engine, simply move the code from the main method to your application's servlet class and change the print statements to display on your webpage.

Testing

This library has tools to help write tests for code that uses the Datastore.

On your machine

You can test against a temporary local Datastore by following these steps:

  1. Install Cloud SDK and start the emulator

To determine which host/port the emulator is running on:

$ gcloud beta emulators datastore env-init

# Sample output:
#   export DATASTORE_EMULATOR_HOST=localhost:8759
  1. Point your client to the emulator
DatastoreOptions options = DatastoreOptions.newBuilder()
.setProjectId(DatastoreOptions.getDefaultProjectId())
.setHost(System.getenv("DATASTORE_EMULATOR_HOST"))
.setCredentials(NoCredentials.getInstance())
.setRetrySettings(ServiceOptions.getNoRetrySettings())
.build();
Datastore datastore = options.getService();
  1. Run your tests

Example Applications

  • Bookshelf - An App Engine app that manages a virtual bookshelf.
    • This app uses google-cloud to interface with Cloud Datastore and Cloud Storage. It also uses Cloud SQL, another Google Cloud Platform service.
  • Flexible Environment/Datastore example - A simple app that uses Cloud Datastore to list the last 10 IP addresses that visited your site.
  • SparkDemo - An example of using google-cloud-datastore from within the SparkJava and App Engine Flexible Environment frameworks.
    • Read about how it works on the example's README page.

Samples

Samples are in the samples/ directory.

Sample Source Code Try it
Native Image Datastore Sample source code Open in Cloud Shell
Quickstart Sample source code Open in Cloud Shell
Avg Aggregation On Kind source code Open in Cloud Shell
Avg Aggregation With Limit source code Open in Cloud Shell
Avg Aggregation With Order By source code Open in Cloud Shell
Avg Aggregation With Property Filter source code Open in Cloud Shell
Count Aggregation In Transaction source code Open in Cloud Shell
Count Aggregation On Kind source code Open in Cloud Shell
Count Aggregation With Gql Query source code Open in Cloud Shell
Count Aggregation With Limit source code Open in Cloud Shell
Count Aggregation With Order By source code Open in Cloud Shell
Count Aggregation With Property Filter source code Open in Cloud Shell
Count Aggregation With Stale Read source code Open in Cloud Shell
Multiple Aggregations In Gql Query source code Open in Cloud Shell
Multiple Aggregations In Structured Query source code Open in Cloud Shell
Sum Aggregation On Kind source code Open in Cloud Shell
Sum Aggregation With Limit source code Open in Cloud Shell
Sum Aggregation With Order By source code Open in Cloud Shell
Sum Aggregation With Property Filter source code Open in Cloud Shell
Create a union between two filters source code Open in Cloud Shell
Query Profile Explain source code Open in Cloud Shell
Query Profile Explain Aggregation source code Open in Cloud Shell
Query Profile Explain Analyze source code Open in Cloud Shell
Query Profile Explain Analyze Aggregation source code Open in Cloud Shell
Task List source code Open in Cloud Shell

Troubleshooting

To get help, follow the instructions in the shared Troubleshooting document.

Supported Java Versions

Java 8 or above is required for using this client.

Google's Java client libraries, Google Cloud Client Libraries and Google Cloud API Libraries, follow the Oracle Java SE support roadmap (see the Oracle Java SE Product Releases section).

For new development

In general, new feature development occurs with support for the lowest Java LTS version covered by Oracle's Premier Support (which typically lasts 5 years from initial General Availability). If the minimum required JVM for a given library is changed, it is accompanied by a semver major release.

Java 11 and (in September 2021) Java 17 are the best choices for new development.

Keeping production systems current

Google tests its client libraries with all current LTS versions covered by Oracle's Extended Support (which typically lasts 8 years from initial General Availability).

Legacy support

Google's client libraries support legacy versions of Java runtimes with long term stable libraries that don't receive feature updates on a best efforts basis as it may not be possible to backport all patches.

Google provides updates on a best efforts basis to apps that continue to use Java 7, though apps might need to upgrade to current versions of the library that supports their JVM.

Where to find specific information

The latest versions and the supported Java versions are identified on the individual GitHub repository github.com/GoogleAPIs/java-SERVICENAME and on google-cloud-java.

Versioning

This library follows Semantic Versioning.

Contributing

Contributions to this library are always welcome and highly encouraged.

See CONTRIBUTING for more information how to get started.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Code of Conduct for more information.

License

Apache 2.0 - See LICENSE for more information.

CI Status

Java Version Status
Java 8 Kokoro CI
Java 8 OSX Kokoro CI
Java 8 Windows Kokoro CI
Java 11 Kokoro CI

Java is a registered trademark of Oracle and/or its affiliates.

java-datastore's People

Contributors

andreamlin avatar athakor avatar benwhitehead avatar capstan avatar chingor13 avatar dependabot[bot] avatar eaball35 avatar eddavisson avatar elharo avatar garrettjonesgoogle avatar gcf-owl-bot[bot] avatar jainsahab avatar jesselovelace avatar jmdobry avatar kolea2 avatar mpeddada1 avatar neenu1995 avatar neozwu avatar pcostell avatar pongad avatar release-please[bot] avatar renovate-bot avatar stephaniewang526 avatar suraj-qlogic avatar suztomo avatar tswast avatar vam-google avatar wsh avatar yihanzhen avatar yoshi-automation avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

java-datastore's Issues

Add OpenCensus tracing/stats support

We're in the process of moving for AppEngine Standard to Flex, thus we're now using the Google Cloud Datastore APIs. Unfortunately, these APIs are not using OpenCensus Tracing and Stats to transmit relevant data.

Please consider this as a feature request for parity.

Manual copy from googleapis/google-cloud-datastore#248

@alysyk please follow along here for progress/work related to your request.

com.example.datastore.QuickstartSampleIT: testQuickstart failed

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.


commit: 947befe
buildURL: Build Status, Sponge
status: failed

Test output
com.google.cloud.datastore.DatastoreException: I/O error
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:164)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:564)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:561)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:560)
	at com.google.cloud.datastore.DatastoreImpl.commitMutation(DatastoreImpl.java:553)
	at com.google.cloud.datastore.DatastoreImpl.delete(DatastoreImpl.java:538)
	at com.example.datastore.QuickstartSampleIT.deleteTestEntity(QuickstartSampleIT.java:46)
	at com.example.datastore.QuickstartSampleIT.tearDown(QuickstartSampleIT.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
	at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:136)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:105)
	at com.google.datastore.v1.client.Datastore.commit(Datastore.java:87)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:162)
	... 39 more
Caused by: java.io.IOException: Error getting access token for service account: 400 Bad Request
POST https://oauth2.googleapis.com/token
{"error":"invalid_grant","error_description":"Invalid JWT Signature."}
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:612)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:164)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:149)
	at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:946)
	at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
	at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
	at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:91)
	... 41 more
Caused by: com.google.api.client.http.HttpResponseException: 400 Bad Request
POST https://oauth2.googleapis.com/token
{"error":"invalid_grant","error_description":"Invalid JWT Signature."}
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1116)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:609)
	... 49 more

datastore.testing.ITLocalDatastoreHelperTest: testStartStopReset failed

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.


commit: b842b7d
buildURL: Build Status, Sponge
status: failed

Test output
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:840)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at com.google.cloud.testing.BaseEmulatorHelper.sendPostRequest(BaseEmulatorHelper.java:198)
	at com.google.cloud.datastore.testing.LocalDatastoreHelper.stop(LocalDatastoreHelper.java:300)
	at com.google.cloud.datastore.testing.ITLocalDatastoreHelperTest.testStartStopReset(ITLocalDatastoreHelperTest.java:165)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)

Starting LocalDatastoreHelper stalls if `beta` gcloud component is not installed

Environment details

  • OS type and version: Debian GNU/Linux 10 (buster) (on Docker)
  • Java version: openjdk version "1.8.0_275"
  • datastore version(s): com.google.cloud:google-cloud-datastore:jar:1.105.0

Steps to reproduce

  1. Remove beta component: gcloud components remove beta
  2. Run the code which uses LocalDatastoreHelper

Code example

import com.google.cloud.datastore.testing.LocalDatastoreHelper;

public class Test {
  public static void main(String[] args) throws Exception {
    LocalDatastoreHelper emulator = LocalDatastoreHelper.create();
    emulator.start();

    System.out.println("Started!");
  }
}

How I found this

  • Removed beta component
root@17224e8a6010:/app# gcloud components remove beta --quiet
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚    These components will be removed.     โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚         Name         โ”‚  Version   โ”‚ Size โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ gcloud Beta Commands โ”‚ 2019.05.17 โ”‚      โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”˜
...
Uninstall done!
  • Run code example, then it stalls.
root@17224e8a6010:/app# mvn exec:java -Dexec.mainClass=Test
  • Ctrl+Z and run ps to find out what command is executed via LocalDatastoreHelper
^Z
[1]+  Stopped                 mvn exec:java -Dexec.mainClass=Test
root@17224e8a6010:/app# ps -ef 
...
root      1840  1795  0 17:40 pts/1    00:00:00 python2 -S /root/google-cloud-sdk/lib/gcloud.py beta emulators datastore start --host-port=localhost:40079 --consistency=0.9
  • Execute the same command at the terminal, then it prompts whether continue to install beta component
root@17224e8a6010:/app# python2 -S /root/google-cloud-sdk/lib/gcloud.py beta emulators datastore start --host-port=localhost:40079 --consistency=0.9
You do not currently have this command group installed.  Using it 
requires the installation of components: [beta]


Your current Cloud SDK version is: 322.0.0
Installing components from version: 322.0.0

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚     These components will be installed.     โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚         Name         โ”‚  Version   โ”‚   Size  โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ gcloud Beta Commands โ”‚ 2019.05.17 โ”‚ < 1 MiB โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

For the latest full release notes, please visit:
  https://cloud.google.com/sdk/release_notes

Do you want to continue (Y/n)?  

Possible solution

Add --quiet option to run emulator, so as to silently install beta component.
pomu0325@e2cb8f2

Datastore emulator fails to start without project id property

Environment details

  1. GCloud datastore emulator 2.1.0
  2. Google Cloud SDK 327.0.0
  3. OS type and version: NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" / MacOS Catalina 10.15.7 (19H15)
  4. Java version: OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10) / OpenJDK Runtime Environment (build 11.0.7+10)
  5. Libs:

Steps to reproduce

This version emulator fails to start without --project property

  1. gcloud beta emulators datastore start
ERROR: (gcloud.beta.emulators.datastore.start) The required property [project] is not currently set.
You may set it for your current workspace by running:

  $ gcloud config set project VALUE

or it can be set temporarily by the environment variable [CLOUDSDK_CORE_PROJECT]

Starting emulator with Java code like this:

LocalDatastoreHelper.create(settings.getConsistency(), settings.getPort());

Causes: no messages in logs, the application failed to call the Datastore then with I/O exceptions, i.e. curl doesn't work.

Workaround

Set env variable: CLOUDSDK_CORE_PROJECT with the projectId value

Questions

  1. Why LocalDatastoreHelper doesn't pass --project on startup and doesn't expect this from the builder?
  2. Shouldn't LocalDatastoreHelper #start fail to start on this kind of errors?

Besides the outcomes of these questions, I would expect this class to fail to start or log something.

datastore.testing.ITLocalDatastoreHelperTest: testStartStopReset failed

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.


commit: a695f61
buildURL: Build Status, Sponge
status: failed

Test output
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:840)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at com.google.cloud.testing.BaseEmulatorHelper.sendPostRequest(BaseEmulatorHelper.java:199)
	at com.google.cloud.datastore.testing.LocalDatastoreHelper.stop(LocalDatastoreHelper.java:300)
	at com.google.cloud.datastore.testing.ITLocalDatastoreHelperTest.testStartStopReset(ITLocalDatastoreHelperTest.java:165)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)

datastore.it.ITDatastoreTest: testSetLimit failed

Note: #101 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.


commit: 74f138f
buildURL: Build Status, Sponge
status: failed

Test output
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at com.google.cloud.datastore.it.ITDatastoreTest.testSetLimit(ITDatastoreTest.java:881)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

Google DataStore iterating over QueryResults is too slow

I have around 20,000 entities stored in google datastore. The datastore project is in us-central and I am querying from my laptop in California. When I query for all entities and iterate over the results it is taking about 30 seconds. This seems too slow. Am I doing something wrong?

Here is the sample scala code. I am using google-cloud-datastore version 1.105.3 and scala 2.12

    import com.google.cloud.datastore._

    println(LocalDateTime.now()) //-----> 2020-12-20T10:36:24.242
    val q = Query.newEntityQueryBuilder().setKind("Test").build()
    val datastore = DatastoreOptions.newBuilder().setProjectId("my-project-id").build().getService
    val es = datastore.run(q,  Seq.empty[ReadOption]: _*)
    
    println(LocalDateTime.now()) //---> 2020-12-20T10:36:24.881
    
    while (es.hasNext) {
          val e = es.next()      // Just iterating over the results.
        }
    
    println(LocalDateTime.now()) //----> 2020-12-20T10:36:53.870 (~29 seconds later)

The query run returns almost immediately but it seems like the getting the results is taking time.

Also, when I try to get the count of the entities using the suggested method of setting the offset to Int.MaxValue I always get getSkippedResults as 1000. I verified that es.hasNext is false as well.

Customize gcloudCommand of LocalDatastoreHelper

I am using Spring Cloud GCP. It has the auto-configuration to start datastore emulator when starting the app engine devserver. However, when I read through the set up of LocalDatastoreHelper, I could see that the gcloudCommand is hard-coded. Are there any ways that I can customize the options of gcloudCommand ?

    List<String> gcloudCommand = new ArrayList<>(Arrays.asList(GCLOUD_CMD_TEXT.split(" ")));
    gcloudCommand.add(GCLOUD_CMD_PORT_FLAG + "localhost:" + getPort());
    gcloudCommand.add(CONSISTENCY_FLAG + consistency);
    gcloudCommand.add("--no-store-on-disk");
    GcloudEmulatorRunner gcloudRunner =
        new GcloudEmulatorRunner(gcloudCommand, VERSION_PREFIX, MIN_VERSION);
    if (gcdPath != null) {
      gcloudCommand.add("--data-dir=" + gcdPath.toString());
    }

I want to remove the option --not-store-on-disk and would like to change the --data-dir to my local folder.
Moreover, could you help me to find the location of the current data-dir. I do not know where it is.
Thanks

Simple query taking 12 seconds

public List<Campaign3> getLiveCampaignsForAccount(Long accountId) {
        EntityQueryRequest request = datastoreAdapter.getEm().createEntityQueryRequest("SELECT * FROM " + ENTITY_NAME + " WHERE accountId=@1 and live = @2");
        request.addPositionalBinding(accountId);
        request.addPositionalBinding(true);
        QueryResponse<Campaign3> response = datastoreAdapter.getEm().executeEntityQueryRequest(Campaign3.class, request);
        return response.getResults();

Transaction doesn't update entity

I use your cloud functions to react some event in topic.
When even has occurred function has to select an entity to change a state.
But it doesn't work. The library doesn't update en entity in Datastore.

Environment details

  1. GCP Cloud Function

Steps to reproduce

  1. Create a cloud function
  2. Add a dependency to work with Datastore (v2.1.2)
  3. Prepare some entities
  4. The function has to select some entities by query
  5. Then update it

Code example

Datastore datastore = DatastoreOptions.getDefaultInstance().getService()

Query<Entity> query = Query.newEntityQueryBuilder()
                    .setKind("MyEntity")
                    .setFilter(eq("phoneNumber", "+15551014397"))
                    .build();

QueryResults<Entity> results = datastore.run(query);

Entity entity = results.next();

Transaction transaction = datastore.newTransaction();

Entity updatedEntity = Entity.newBuilder(entity)
        .set("state", 3)
        .build();

transaction.update(updatedEntity);

transaction.commit();

Stack trace

The result is success. No error.
But the entity in Datastore has not been updated with field entity.state = 3 - there is previous value/

datastore.testing.ITLocalDatastoreHelperTest: testStartStopResetWithBuilder failed

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.


commit: 03b6d5e
buildURL: Build Status, Sponge
status: failed

Test output
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:840)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at com.google.cloud.testing.BaseEmulatorHelper.sendPostRequest(BaseEmulatorHelper.java:198)
	at com.google.cloud.datastore.testing.LocalDatastoreHelper.stop(LocalDatastoreHelper.java:300)
	at com.google.cloud.datastore.testing.ITLocalDatastoreHelperTest.testStartStopResetWithBuilder(ITLocalDatastoreHelperTest.java:185)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)

Temporary animal-sniffer-annotations exclusion rules in poms

As of grpc-api:1.26.0 and guava:28.1-android there is a failure for animal-sniffer-annotations

There is no actual difference between the single annotation present in the versions, so it is being excluded for the time being.

The exclusion should be removed from the poms when there is no longer a disparity.

Synthesis failed for java-datastore

Hello! Autosynth couldn't regenerate java-datastore. ๐Ÿ’”

Here's the output from running synth.py:

ernal/gapic_generator_python/requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl
  Saved ./Jinja2-2.11.2-py2.py3-none-any.whl
Collecting MarkupSafe==1.1.1 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
  Saved ./MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting protobuf==3.13.0 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/30/79/510974552cebff2ba04038544799450defe75e96ea5f1675dbf72cc8744f/protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl
  Saved ./protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pypandoc==1.5 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/d6/b7/5050dc1769c8a93d3ec7c4bd55be161991c94b8b235f88bf7c764449e708/pypandoc-1.5.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmpfs/tmp/tmp0xmgeb7r/setuptools-tmp/setuptools/__init__.py", line 6, in <module>
        import distutils.core
      File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/_distutils_hack/__init__.py", line 82, in create_module
        return importlib.import_module('._distutils', 'setuptools')
      File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    ModuleNotFoundError: No module named 'setuptools._distutils'
    
    ----------------------------------------
 (Command "python setup.py egg_info" failed with error code 1 in /tmpfs/tmp/pip-build-pt2oeod4/pypandoc/
)
ERROR: no such package '@gapic_generator_python_pip_deps//': pip_import failed: Collecting click==7.1.2 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
  Saved ./click-7.1.2-py2.py3-none-any.whl
Collecting google-api-core==1.22.1 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/e0/2d/7c6c75013105e1d2b6eaa1bf18a56995be1dbc673c38885aea31136e9918/google_api_core-1.22.1-py2.py3-none-any.whl
  Saved ./google_api_core-1.22.1-py2.py3-none-any.whl
Collecting googleapis-common-protos==1.52.0 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/03/74/3956721ea1eb4bcf7502a311fdaa60b85bd751de4e57d1943afe9b334141/googleapis_common_protos-1.52.0-py2.py3-none-any.whl
  Saved ./googleapis_common_protos-1.52.0-py2.py3-none-any.whl
Collecting jinja2==2.11.2 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl
  Saved ./Jinja2-2.11.2-py2.py3-none-any.whl
Collecting MarkupSafe==1.1.1 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
  Saved ./MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting protobuf==3.13.0 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/30/79/510974552cebff2ba04038544799450defe75e96ea5f1675dbf72cc8744f/protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl
  Saved ./protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pypandoc==1.5 (from -r /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_python/requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/d6/b7/5050dc1769c8a93d3ec7c4bd55be161991c94b8b235f88bf7c764449e708/pypandoc-1.5.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmpfs/tmp/tmp0xmgeb7r/setuptools-tmp/setuptools/__init__.py", line 6, in <module>
        import distutils.core
      File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/_distutils_hack/__init__.py", line 82, in create_module
        return importlib.import_module('._distutils', 'setuptools')
      File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    ModuleNotFoundError: No module named 'setuptools._distutils'
    
    ----------------------------------------
 (Command "python setup.py egg_info" failed with error code 1 in /tmpfs/tmp/pip-build-pt2oeod4/pypandoc/
)
INFO: Elapsed time: 2.107s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kbuilder/.cache/synthtool/java-datastore/synth.py", line 52, in <module>
    proto_path=f'google/{service}/{version}',
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 62, in java_library
    service, version, "java", tar_strip_components=0, **kwargs
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 183, in _generate_code
    shell.run(bazel_run_args)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bazel', '--max_idle_secs=240', 'build', '//google/datastore/v1:google-cloud-datastore-v1-java']' returned non-zero exit status 1.
2020-08-30 14:43:44,641 autosynth [ERROR] > Synthesis failed
2020-08-30 14:43:44,642 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 9407c4a docs: update libraries-bom (#211)
2020-08-30 14:43:44,647 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-08-30 14:43:44,652 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 690, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 539, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 670, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 375, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 273, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/tmpfs/src/github/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

datastore.it.ITDatastoreTest: testRunStructuredQuery failed

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.


commit: 6822385
buildURL: Build Status, Sponge
status: failed

Test output
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at com.google.cloud.datastore.it.ITDatastoreTest.testRunStructuredQuery(ITDatastoreTest.java:577)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

Synthesis failed for java-datastore

Hello! Autosynth couldn't regenerate java-datastore. ๐Ÿ’”

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to a new branch 'autosynth'
Cloning into '/tmpfs/tmp/tmpxgwz8b6o/googleapis'...
Cloning into '/tmpfs/tmp/tmpxgwz8b6o/synthtool'...
Switched to branch 'autosynth-self'
Note: checking out '97864cbbffdb29420745491eaabb212da9b48147'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 97864cb chore(deps): update dependency com.google.cloud:libraries-bom to v5.3.0 (#115)
Note: checking out 'd741cd976975c745d0199987aff0e908b8352992'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d741cd97 chore: enable gapicv2 for firestore/v1 API
Note: checking out '52638600f387deb98efb5f9c85fec39e82aa9052'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 5263860 build(java): set GOOGLE_CLOUD_PROJECT env for samples/integration tests (#484)
Switched to a new branch 'autosynth-self-2'
2020-04-30 13:59:09 [INFO] Running synthtool
2020-04-30 13:59:09 [INFO] ['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/tmpfs/src/github/synthtool/synthtool/__init__.py", line 21, in <module>
    from synthtool import update_check
  File "/tmpfs/src/github/synthtool/synthtool/update_check.py", line 19, in <module>
    import packaging.version
ModuleNotFoundError: No module named 'packaging'
2020-04-30 13:59:09 [ERROR] Synthesis failed
HEAD is now at 97864cb chore(deps): update dependency com.google.cloud:libraries-bom to v5.3.0 (#115)
Switched to branch 'autosynth-self'
Note: checking out '97864cbbffdb29420745491eaabb212da9b48147'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 97864cb chore(deps): update dependency com.google.cloud:libraries-bom to v5.3.0 (#115)
Previous HEAD position was d741cd97 chore: enable gapicv2 for firestore/v1 API
HEAD is now at 7e1c7603 Add an OAuth scope annotation in build_service proto file
Previous HEAD position was 5263860 build(java): set GOOGLE_CLOUD_PROJECT env for samples/integration tests (#484)
HEAD is now at 6b685a2 fix: synthtool path (#515)
Switched to a new branch 'autosynth-90'
2020-04-30 13:59:09 [INFO] Running synthtool
2020-04-30 13:59:09 [INFO] ['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/tmpfs/src/github/synthtool/synthtool/__init__.py", line 21, in <module>
    from synthtool import update_check
  File "/tmpfs/src/github/synthtool/synthtool/update_check.py", line 19, in <module>
    import packaging.version
ModuleNotFoundError: No module named 'packaging'
2020-04-30 13:59:09 [ERROR] Synthesis failed
HEAD is now at 97864cb chore(deps): update dependency com.google.cloud:libraries-bom to v5.3.0 (#115)
Switched to branch 'autosynth'
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 576, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 457, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 566, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 352, in synthesize_loop
    synthesize_inner_loop(toolbox, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 362, in synthesize_inner_loop
    synthesizer, len(toolbox.versions) - 1
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(self.environ)
  File "/tmpfs/src/github/synthtool/autosynth/synthesizer.py", line 115, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

com.example.datastore.QuickstartSampleIT: testQuickstart failed

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.


commit: d188c4c
buildURL: Build Status, Sponge
status: failed

Test output
com.google.cloud.datastore.DatastoreException: The project java-docs-samples-testing does not exist or it does not contain an active Cloud Datastore or Cloud Firestore database. Please visit http://console.cloud.google.com to create a project or https://console.cloud.google.com/datastore/setup?project=java-docs-samples-testing to add a Cloud Datastore or Cloud Firestore database. Note that Cloud Datastore or Cloud Firestore always have an associated App Engine app and this app must not be disabled.
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:164)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:564)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:561)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:560)
	at com.google.cloud.datastore.DatastoreImpl.commitMutation(DatastoreImpl.java:553)
	at com.google.cloud.datastore.DatastoreImpl.delete(DatastoreImpl.java:538)
	at com.example.datastore.QuickstartSampleIT.deleteTestEntity(QuickstartSampleIT.java:46)
	at com.example.datastore.QuickstartSampleIT.tearDown(QuickstartSampleIT.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
Caused by: com.google.datastore.v1.client.DatastoreException: The project java-docs-samples-testing does not exist or it does not contain an active Cloud Datastore or Cloud Firestore database. Please visit http://console.cloud.google.com to create a project or https://console.cloud.google.com/datastore/setup?project=java-docs-samples-testing to add a Cloud Datastore or Cloud Firestore database. Note that Cloud Datastore or Cloud Firestore always have an associated App Engine app and this app must not be disabled., code=NOT_FOUND
	at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:136)
	at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:185)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:96)
	at com.google.datastore.v1.client.Datastore.commit(Datastore.java:87)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:162)
	... 39 more

Migrate google-cloud-java/google-cloud-datastore to use GAPIC low-level libraries

Right now, https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-datastore depends on https://github.com/GoogleCloudPlatform/google-cloud-datastore, which is suboptimal for several reasons:

  1. No support for gRPC.
  2. Lack of uniformity with other google-cloud-java clients.
  3. Continued dependence on the separate google-cloud-datastore artifact, which frequently leads to confusion because of the unfortunate naming similarity. We'd like to deprecate this artifact altogether.

LocalDatastoreHelper fails to start if used in an environment that cannot access the "gcd" GCS bucket

Is your feature request related to a problem? Please describe.
When attempting to use the Downloadable emulator, LocalDatastoreHelper downloads from a hardcoded URL to the file in the "gcd" GCS bucket:

EMULATOR_URL = new URL("http://storage.googleapis.com/gcd/tools/" + FILENAME);

When used in an environment that cannot access the "gcd" GCS bucket, LocalDatastoreHelper fails to start (e.g. when running in a GCE VM protected by a VPC-SC perimeter).

Describe the solution you'd like
If I could provide the emulator file URL in an environment variable, I could copy the file from the "gcd" bucket into another bucket that is accessible from the restricted environment and use that URL.

Describe alternatives you've considered
N/A

High Datastore Query Latency

I am reporting an issue I am seeing when using the java-datastore library within an Apache Beam transform (running on GCP Dataflow). The beam job was running slow so i decided to profile it using the GCP Profiler. I was able to identify that most of the wall time was spent waiting on calls to Datastore (within a beam transform) using the Java Datastore library. I wrote a test job that just makes simple datastore reads within a beam transform and saw the same results. Here is a screenshot of the profiler:

image

You can see in the screenshot that on average it is taking around 19s to run these simple Datastore queries. I understand it is best practice to use the beam datastore library in order to parallelize/batch these calls to Datastore but for our use case we need these sequential synchronous reads (that the Java Datastore library provides) within the beam transform. Anyways I was wondering if anyone had any insight into why these calls might be taking as long as they do? You can see in the screenshot that most of the wall time is spent waiting on the SocketInputStream.socketRead0() method which indicates possible network or remote server latency.

Here is a screenshot of the test job DoFn code i wrote that i ran the above profile on:

image

datastore.it.ITDatastoreTest: testTransactionWithRead and testTransactionWithQuery fail if the test DB type is Firestore in Datastore mode

These tests always timeout when testing with a Firestore in Datastore Mode DB. I think the reason they pass in continuous tests is because the type of DB they use (in project gcloud-devel) is "Cloud Datastore".
I believe the root cause of the timeouts is the transaction isolation and consistency behavior differences between the 2 Datastore DB types, documented here and here.
TL;DR
With Firestore in datastore mode when 2 concurrent transactions read or write the same data, the lock held by one transaction can delay the other transaction (hence the timeouts), but with Cloud Datastore the contention is reported as an error at commit.

Environment details

All environments

Steps to reproduce

  1. Create a Firestore in Datastore Mode DB (or use an existing DB, if its type is known to be "Firestore in Datastore Mode")
  2. Set the GOOGLE_CLOUD_PROJECT env var to the project ID that owns the DB
  3. Set the GOOGLE_APPLICATION_CREDENTIALS env var to the absolute path of the key file for a service account that has permission to write to the DB
  4. run the tests, e.g.
GOOGLE_CLOUD_PROJECT=some-project GOOGLE_APPLICATION_CREDENTIALS=/tmp/sa-key.json mvn -P enable-integration-tests verify

Stack trace

it.ITDatastoreTest.testTransactionWithQuery

org.junit.runners.model.TestTimedOutException: test timed out after 100 seconds
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
	at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1332)
	at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:948)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
	at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:97)
	at com.google.datastore.v1.client.Datastore.commit(Datastore.java:85)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:162)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:564)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:561)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:560)
	at com.google.cloud.datastore.DatastoreImpl.commitMutation(DatastoreImpl.java:553)
	at com.google.cloud.datastore.DatastoreImpl.put(DatastoreImpl.java:513)
	at com.google.cloud.datastore.DatastoreHelper.put(DatastoreHelper.java:53)
	at com.google.cloud.datastore.DatastoreImpl.put(DatastoreImpl.java:488)
	at com.google.cloud.datastore.it.ITDatastoreTest.testTransactionWithQuery(ITDatastoreTest.java:284)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

it.ITDatastoreTest.testTransactionWithRead
this test would also timeout if this code was updating ENTITY3 as stated in the comment.

java.lang.AssertionError: Expecting a failure
	at org.junit.Assert.fail(Assert.java:89)
	at com.google.cloud.datastore.it.ITDatastoreTest.testTransactionWithRead(ITDatastoreTest.java:254)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

External references such as API reference guides

Synthesis failed for java-datastore

Hello! Autosynth couldn't regenerate java-datastore. ๐Ÿ’”

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to branch 'autosynth'
Running synthtool
['/tmpfs/src/git/autosynth/env/bin/python3', '-m', 'synthtool', 'synth.py', '--']
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/synth.py.
On branch autosynth
nothing to commit, working tree clean
HEAD detached at FETCH_HEAD
nothing to commit, working tree clean
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:6aec9c34db0e4be221cdaf6faba27bdc07cfea846808b3d3b964dfce3a9a0f9b
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/datastore/artman_datastore.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/LookupRequest.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/Query.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/GqlQueryParameter.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/QueryOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/LookupResponse.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PropertyOrder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/CompositeFilterOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/FilterOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/QueryResultBatchOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ArrayValueOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/EntityOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/BeginTransactionRequestOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/AllocateIdsResponse.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/KindExpression.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ReserveIdsResponse.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PropertyOrderOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/MutationOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/Key.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/Projection.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RunQueryRequestOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/GqlQueryParameterOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ProjectionOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/TransactionOptions.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ReadOptionsOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/CommitResponseOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/AllocateIdsRequest.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/Value.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RollbackResponseOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/Filter.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/LookupResponseOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/CommitRequestOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/TransactionOptionsOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/QueryResultBatch.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RollbackRequest.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ArrayValue.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/AllocateIdsRequestOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/DatastoreProto.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/AllocateIdsResponseOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/BeginTransactionResponse.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ReserveIdsRequestOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/KeyOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/Entity.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/Mutation.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/EntityResultOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/LookupRequestOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PartitionId.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/CommitRequest.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PropertyReference.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PropertyReferenceOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/MutationResult.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/GqlQueryOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/CommitResponse.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RunQueryResponseOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PartitionIdOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ReadOptions.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/MutationResultOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/GqlQuery.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RunQueryResponse.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ValueOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ReserveIdsResponseOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/CompositeFilter.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/EntityProto.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/KindExpressionOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RollbackRequestOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RollbackResponse.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/RunQueryRequest.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/EntityResult.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/ReserveIdsRequest.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/BeginTransactionRequest.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PropertyFilter.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/BeginTransactionResponseOrBuilder.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/QueryProto.java.
synthtool > Replaced '// Generated by the protocol buffer compiler.  DO NOT EDIT!' in /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java/proto-google-cloud-datastore-v1/src/main/java/com/google/datastore/v1/PropertyFilterOrBuilder.java.
synthtool > Running java formatter on 96 files
synthtool > Running java formatter on 73 files
synthtool > Wrote metadata to synth.metadata.
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmpfs/src/git/autosynth/working_repo/synth.py", line 66, in <module>
    templates = common_templates.java_library()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/common.py", line 75, in java_library
    return self._generic_library("java_library", **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/common.py", line 43, in _generic_library
    if not kwargs["metadata"]["samples"]:
KeyError: 'samples'

Synthesis failed

Google internal developers can see the full log here.

datastore.it.ITDatastoreTest: testSkippedResults failed

Note: #97 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.


commit: 964f59e
buildURL: Build Status, Sponge
status: failed

Test output
java.lang.AssertionError: expected:<2> but was:<0>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at com.google.cloud.datastore.it.ITDatastoreTest.testSkippedResults(ITDatastoreTest.java:874)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

LocalHost Check For Emulator URL Throws Exception, Causes Invalidation and Connection Failure

Environment details

  1. Client Library: google-cloud-datastore 2.1.3
  2. OS type and version: Mac OS Big Sur 11.6
  3. Java version: 1.8.0_301
  4. datastore version(s): 2.1.3
  5. Emulator Version: latest at time of posting

Steps to reproduce

  1. Default emulator start --host=::1 --port=8658
  2. $(gcloud beta emulators datastore env-init)
  3. in Java: DatastoreOptions.getDefaultInstance().getService();

Code example

HttpDatastoreRpc.class

private static boolean isLocalHost(String host) { // "::1:8658"
    if(!host.isEmpty() {
        try {
            String normalizedHost = "http://" + removeScheme(host);
            InetAddress hostAddr = InetAddress.getByName(new URL(normalizedHost )).getHost()); //throws Exception
            return hostAddr.isAnyLocalAddress() || hostAddr.isLoopbackAddress();
        } catch (Exception var3) {
        }
    }

    return false;
}

Stack trace

java.lang.IllegalArgumentException: Project endpoint: "::1:8658/v1/projects/my-project" must include scheme.

datastore.it.ITDatastoreTest: testSkippedResults failed

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.


commit: 6822385
buildURL: Build Status, Sponge
status: failed

Test output
java.lang.AssertionError: expected:<2> but was:<3>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at com.google.cloud.datastore.it.ITDatastoreTest.testSkippedResults(ITDatastoreTest.java:874)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

datastore emulator is shutdown unexpectedly

The issue appears every time during test run. Probably it's related to big number of write/read/delete operations but i'm not sure. Please let me know if any other logs are necessary.

[datastore] Nov 26, 2020 4:47:12 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub <init>
[datastore] INFO: Local Datastore initialized:
[datastore] 	Type: High Replication
[datastore] 	Storage: In-memory
[datastore] API endpoint: http://localhost:8081
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore]
[datastore]   export DATASTORE_EMULATOR_HOST=localhost:8081
[datastore]
[datastore] Dev App Server is now running.
[datastore]
[datastore] The previous line was printed for backwards compatibility only.
[datastore] If your tests rely on it to confirm emulator startup,
[datastore] please migrate to the emulator health check endpoint (/). Thank you!
[datastore] The health check endpoint for this emulator instance is http://localhost:8081/Nov 26, 2020 4:47:49 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[datastore] INFO: Adding handler(s) to newly registered Channel.
[datastore] Nov 26, 2020 4:47:49 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[datastore] INFO: Detected non-HTTP/2 connection.
[datastore] Nov 26, 2020 4:47:50 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[datastore] INFO: Adding handler(s) to newly registered Channel.
[datastore] Nov 26, 2020 4:47:50 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[datastore] INFO: Detected HTTP/2 connection.
[datastore] Nov 26, 2020 4:47:53 PM io.netty.util.ResourceLeakDetector reportTracedLeak
[datastore] SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
[datastore] Recent access records:
[datastore] Created at:
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:221)
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:199)
[datastore] 	io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:255)
[datastore] 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
[datastore] 	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
[datastore] 	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
[datastore] 	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
[datastore] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
[datastore] 	java.lang.Thread.run(Thread.java:748)
[datastore] Nov 26, 2020 4:47:53 PM io.netty.util.ResourceLeakDetector reportTracedLeak
[datastore] SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
[datastore] Recent access records:
[datastore] Created at:
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:221)
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:199)
[datastore] 	io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:255)
[datastore] 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
[datastore] 	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
[datastore] 	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
[datastore] 	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
[datastore] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
[datastore] 	java.lang.Thread.run(Thread.java:748)
[datastore] Nov 26, 2020 4:47:53 PM io.netty.util.ResourceLeakDetector reportTracedLeak
[datastore] SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
[datastore] Recent access records:
[datastore] Created at:
[datastore] 	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137)
[datastore] 	io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
[datastore] 	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
[datastore] 	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
[datastore] 	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
[datastore] 	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508)
[datastore] 	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
[datastore] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
[datastore] 	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[datastore] 	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[datastore] 	java.lang.Thread.run(Thread.java:748)
[datastore] Nov 26, 2020 4:47:53 PM io.netty.util.ResourceLeakDetector reportTracedLeak
[datastore] SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
[datastore] Recent access records:
[datastore] Created at:
[datastore] 	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
[datastore] 	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
[datastore] 	io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1762)
[datastore] 	io.netty.buffer.CompositeByteBuf.copy(CompositeByteBuf.java:1417)
[datastore] 	io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1194)
[datastore] 	io.gapi.emulators.grpc.HttpAdapter$UnaryMethodHandler.handle(HttpAdapter.java:517)
[datastore] 	io.gapi.emulators.grpc.HttpAdapter.handleRequest(HttpAdapter.java:191)
[datastore] 	io.gapi.emulators.netty.HttpHandler.channelRead(HttpHandler.java:52)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.HttpHandler$1.onValue(HttpHandler.java:64)
[datastore] 	io.gapi.emulators.netty.HttpHandler$1.onValue(HttpHandler.java:54)
[datastore] 	io.gapi.emulators.grpc.HttpAdapter.handleRequest(HttpAdapter.java:162)
[datastore] 	io.gapi.emulators.netty.HttpHandler.channelRead(HttpHandler.java:52)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.UnsupportedApiVersionHandler.channelRead(UnsupportedApiVersionHandler.java:96)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.SimpleHttpHandler.channelRead(SimpleHttpHandler.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.SimpleHttpHandler.channelRead(SimpleHttpHandler.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.SimpleHttpHandler.channelRead(SimpleHttpHandler.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.SimpleHttpHandler.channelRead(SimpleHttpHandler.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.SimpleHttpHandler.channelRead(SimpleHttpHandler.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.SimpleHttpHandler.channelRead(SimpleHttpHandler.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.gapi.emulators.netty.SimpleHttpHandler.channelRead(SimpleHttpHandler.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
[datastore] 	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
[datastore] 	io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
[datastore] 	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
[datastore] 	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
[datastore] 	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
[datastore] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
[datastore] 	java.lang.Thread.run(Thread.java:748)
[datastore] *** shutting down gRPC server since JVM is shutting down
[datastore] *** server shut down
[datastore] Nov 26, 2020 4:47:57 PM com.google.cloud.datastore.emulator.impl.BaseLocalDatastore cleanupActiveServices
[datastore] INFO: scheduler shutting down.

Cannot use Cloud Datastore Java APIs with Graal VM.

Playing with the native example of Quarkus IO https://quarkus.io/ located at:

https://github.com/quarkusio/quarkus-quickstarts/tree/master/getting-started

Adding a dep with

       <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-datastore</artifactId>
      <version>1.62.0</version>
    </dependency>

And using the Cloud Datastore API (simple imports/usage)

Notice I do not define GAE App Engine APIs which I do not want to use (it would not work anyway in a Java11 titanium pure runtime.

I then compile with:

mvn install -Dnative-image.docker-build -Pnative
...
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
[quarkus-quickstart-runner:6] analysis: 22,469.29 ms
Printing call tree to /project/reports/call_tree_quarkus-quickstart-runner_20190312_181219.txt
Printing list of used classes to /project/reports/used_classes_quarkus-quickstart-runner_20190312_181222.txt
Printing list of used packages to /project/reports/used_packages_quarkus-quickstart-runner_20190312_181222.txt
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: com.google.appengine.api.urlfetch.HTTPMethod. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: com.google.appengine.api.urlfetch.HTTPMethod. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Trace:
at parsing com.google.api.client.extensions.appengine.http.UrlFetchTransport.buildRequest(UrlFetchTransport.java:116)
Call path from entry point to com.google.api.client.extensions.appengine.http.UrlFetchTransport.buildRequest(String, String):
at com.google.api.client.extensions.appengine.http.UrlFetchTransport.buildRequest(UrlFetchTransport.java:113)
at com.google.api.client.extensions.appengine.http.UrlFetchTransport.buildRequest(UrlFetchTransport.java:50)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:901)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:183)
at com.google.datastore.v1.client.Datastore.commit(Datastore.java:87)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:152)
at com.google.cloud.datastore.DatastoreImpl$4.call(DatastoreImpl.java:496)
at com.google.cloud.datastore.DatastoreImpl$4.call(DatastoreImpl.java:493)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:176)
at com.google.cloud.datastore.QueryResultsImpl.sendRequest(QueryResultsImpl.java:72)
at com.google.cloud.datastore.QueryResultsImpl.computeNext(QueryResultsImpl.java:93)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
at java.util.AbstractCollection.containsAll(AbstractCollection.java:317)
at java.util.RegularEnumSet.containsAll(RegularEnumSet.java:199)
at com.oracle.svm.core.amd64.AMD64CPUFeatureAccess.verifyHostSupportsArchitecture(AMD64CPUFeatureAccess.java:158)
at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:131)
at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

com.example.datastore.QuickstartSampleIT: testQuickstart failed

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.


commit: 947befe
buildURL: Build Status, Sponge
status: failed

Test output
com.google.cloud.datastore.DatastoreException: I/O error
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:164)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:564)
	at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:561)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:560)
	at com.google.cloud.datastore.DatastoreImpl.commitMutation(DatastoreImpl.java:553)
	at com.google.cloud.datastore.DatastoreImpl.delete(DatastoreImpl.java:538)
	at com.example.datastore.QuickstartSampleIT.deleteTestEntity(QuickstartSampleIT.java:46)
	at com.example.datastore.QuickstartSampleIT.tearDown(QuickstartSampleIT.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
	at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:136)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:105)
	at com.google.datastore.v1.client.Datastore.commit(Datastore.java:87)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:162)
	... 39 more
Caused by: java.io.IOException: Error getting access token for service account: 400 Bad Request
POST https://oauth2.googleapis.com/token
{"error":"invalid_grant","error_description":"Invalid JWT Signature."}
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:612)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:164)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:149)
	at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:946)
	at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
	at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
	at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:91)
	... 41 more
Caused by: com.google.api.client.http.HttpResponseException: 400 Bad Request
POST https://oauth2.googleapis.com/token
{"error":"invalid_grant","error_description":"Invalid JWT Signature."}
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1116)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:609)
	... 49 more

Transaction "add"s are not multi-thread safe

The add(entity) method in a Transaction object is not multi-thread safe. Other methods (e.g. put, delete) may also not be safe but we have traced a failure in our server specifically to add. Our server uses Scala Futures and so uses multiple threads of execution.

The particular scenario we tracked down was in our new user creation processing where we add 4 entities in a transaction. However there was a probability that one of the entities didn't get saved. It wasn't always the same entity kind that "disappeared". We traced this by adding debug logging after the 'add' calls that shows the adds successfully completing - but subsequent processing would find one of the entities missing. We did note that it always seemed to be just one entity missing and when an entity did go missing it seemed to be the first entity kind to be added (thread execution order was not consistent across invocations).

We added synchronize on each of our DataStore calls and haven't encountered this issue since. So at a minimum this issue can serve as a warning to others. Maybe a note could be added to the documentation. But we are also patiently waiting for issue #22 (async support) to be implemented and would hope that explicit synchronization to not be needed.

Queries that start at a cursor report that they start at the very beginning until you go to the first element

A query result from a query that starts at a cursor will say getCursorAfter is actually the default cursor (Cursor{bytes=}) rather than the cursor it started with.

This creates complications for testing / resuming a result set that was never iterated.

The old Java 8 datastore SDK didn't have this issue.

Environment details

  1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
    General, Core, and Other are also allowed as types
  2. OS type and version: any
  3. Java version: 8/11
  4. datastore version(s): current master (0.87.1)

Steps to reproduce

Start cloud_datastore_emulator on localhost:8081
Run code example

Code example

    import com.google.cloud.datastore.*;
    Datastore ds = DatastoreOptions.newBuilder().setProjectId("project").setHost("localhost:8081").build().getService();
    ds.put(Entity.newBuilder(Key.newBuilder("project", "Kind", "name-01").build()).build());
    ds.put(Entity.newBuilder(Key.newBuilder("project", "Kind", "name-02").build()).build());
    ds.put(Entity.newBuilder(Key.newBuilder("project", "Kind", "name-03").build()).build());
    QueryResults<Entity> run = ds.run(Query.newEntityQueryBuilder().setKind("Kind").build());
    run.next();
    Cursor cursor = run.getCursorAfter();
    QueryResults<Entity> run2 = ds.run(Query.newEntityQueryBuilder().setKind("Kind").setStartCursor(cursor).build());
    // expected:
    assert run2.getCursorAfter().equals(cursor);
    // actual: run2.getCursorAfter() is the default cursor until run2.next() is called

Synthesis failed for java-datastore

Hello! Autosynth couldn't regenerate java-datastore. ๐Ÿ’”

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to a new branch 'autosynth'
Cloning into '/tmpfs/tmp/tmprt4_t8z2/googleapis'...
Cloning into '/tmpfs/tmp/tmprt4_t8z2/synthtool'...
Switched to branch 'autosynth-self'
Note: checking out 'b484256c30ac9117eb82688c801966b6ace856df'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at b484256 deps: update dependency com.google.api.grpc:proto-google-common-protos to v1.18.0 (#117)
Note: checking out 'd741cd976975c745d0199987aff0e908b8352992'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d741cd97 chore: enable gapicv2 for firestore/v1 API
Note: checking out '52638600f387deb98efb5f9c85fec39e82aa9052'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 5263860 build(java): set GOOGLE_CLOUD_PROJECT env for samples/integration tests (#484)
Switched to a new branch 'autosynth-self-2'
2020-05-05 21:31:00 [INFO] Running synthtool
2020-05-05 21:31:00 [INFO] ['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
2020-05-05 21:31:01,015 synthtool > Executing /tmpfs/src/github/synthtool/working_repo/synth.py.
On branch autosynth-self-2
nothing to commit, working tree clean
2020-05-05 21:31:01,148 synthtool > Ensuring dependencies.
2020-05-05 21:31:01,160 synthtool > Cloning googleapis.
2020-05-05 21:31:01,161 synthtool > Using precloned repo /tmpfs/tmp/tmprt4_t8z2/googleapis
2020-05-05 21:31:01,165 synthtool > Generating code for: //google/datastore/v1:google-cloud-datastore-v1-java.
2020-05-05 21:31:01,275 synthtool > Failed executing bazel build //google/datastore/v1:google-cloud-datastore-v1-java:

Starting local Bazel server and connecting to it...
Server crashed during startup. Now printing /home/kbuilder/.cache/bazel/_bazel_kbuilder/5528ad577e097974c79b31ce121db9ff/server/jvm.out
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000715200000, 247463936, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 247463936 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmpfs/tmp/tmprt4_t8z2/googleapis/hs_err_pid20881.log

2020-05-05 21:31:01,276 synthtool > Wrote metadata to synth.metadata.
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmpfs/src/github/synthtool/working_repo/synth.py", line 52, in <module>
    proto_path=f'google/{service}/{version}',
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 63, in java_library
    return self._generate_code(service, version, "java", **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 177, in _generate_code
    shell.run(bazel_run_args)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bazel', 'build', '//google/datastore/v1:google-cloud-datastore-v1-java']' returned non-zero exit status 37.
2020-05-05 21:31:01 [ERROR] Synthesis failed
HEAD is now at b484256 deps: update dependency com.google.api.grpc:proto-google-common-protos to v1.18.0 (#117)
Switched to branch 'autosynth-self'
Note: checking out 'b484256c30ac9117eb82688c801966b6ace856df'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at b484256 deps: update dependency com.google.api.grpc:proto-google-common-protos to v1.18.0 (#117)
Previous HEAD position was d741cd97 chore: enable gapicv2 for firestore/v1 API
HEAD is now at a3a0bf0f BREAKING_CHANGE: Removing TimeSeriesQueryLanguageCondition as an alert condition type. The condition type is unsupported and unused. It was originally added for the Monitoring Query Language Alpha feature. refactor!: Drop support for TimeSeriesQueryLanguageCondition as an alert condition type.
Previous HEAD position was 5263860 build(java): set GOOGLE_CLOUD_PROJECT env for samples/integration tests (#484)
HEAD is now at ab88356 fix: make .kokoro-autosynth executable (#522)
Switched to a new branch 'autosynth-108'
2020-05-05 21:31:01 [INFO] Running synthtool
2020-05-05 21:31:01 [INFO] ['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
2020-05-05 21:31:01,605 synthtool > Executing /tmpfs/src/github/synthtool/working_repo/synth.py.
On branch autosynth-108
nothing to commit, working tree clean
2020-05-05 21:31:01,736 synthtool > Ensuring dependencies.
2020-05-05 21:31:01,747 synthtool > Cloning googleapis.
2020-05-05 21:31:01,748 synthtool > Using precloned repo /tmpfs/tmp/tmprt4_t8z2/googleapis
2020-05-05 21:31:01,752 synthtool > Generating code for: //google/datastore/v1:google-cloud-datastore-v1-java.
2020-05-05 21:31:01,862 synthtool > Failed executing bazel build //google/datastore/v1:google-cloud-datastore-v1-java:

Starting local Bazel server and connecting to it...
Server crashed during startup. Now printing /home/kbuilder/.cache/bazel/_bazel_kbuilder/5528ad577e097974c79b31ce121db9ff/server/jvm.out
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000715200000, 247463936, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 247463936 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmpfs/tmp/tmprt4_t8z2/googleapis/hs_err_pid20916.log

2020-05-05 21:31:01,863 synthtool > Wrote metadata to synth.metadata.
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmpfs/src/github/synthtool/working_repo/synth.py", line 52, in <module>
    proto_path=f'google/{service}/{version}',
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 63, in java_library
    return self._generate_code(service, version, "java", **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 177, in _generate_code
    shell.run(bazel_run_args)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bazel', 'build', '//google/datastore/v1:google-cloud-datastore-v1-java']' returned non-zero exit status 37.
2020-05-05 21:31:01 [ERROR] Synthesis failed
HEAD is now at b484256 deps: update dependency com.google.api.grpc:proto-google-common-protos to v1.18.0 (#117)
Switched to branch 'autosynth'
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 584, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 465, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 574, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 360, in synthesize_loop
    synthesize_inner_loop(toolbox, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 370, in synthesize_inner_loop
    synthesizer, len(toolbox.versions) - 1
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(self.environ)
  File "/tmpfs/src/github/synthtool/autosynth/synthesizer.py", line 115, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/unmanaged_dependency_check.yaml
  • actions/checkout v4
  • actions/setup-java v4
maven
datastore-v1-proto-client/pom.xml
  • com.google.cloud:google-cloud-datastore-parent 2.20.1
  • com.google.truth:truth 1.4.2
google-cloud-datastore-bom/pom.xml
  • com.google.cloud:sdk-platform-java-config 3.31.0
  • com.google.cloud:google-cloud-datastore 2.20.1
  • com.google.api.grpc:grpc-google-cloud-datastore-admin-v1 2.20.1
  • com.google.api.grpc:proto-google-cloud-datastore-v1 0.111.1
  • com.google.api.grpc:proto-google-cloud-datastore-admin-v1 2.20.1
google-cloud-datastore/pom.xml
  • com.google.cloud:google-cloud-datastore-parent 2.20.1
  • com.google.guava:guava-testlib 33.2.1-jre
  • com.google.truth:truth 1.4.2
grpc-google-cloud-datastore-admin-v1/pom.xml
  • com.google.cloud:google-cloud-datastore-parent 2.20.1
pom.xml
  • com.google.cloud:sdk-platform-java-config 3.31.0
  • com.google.api.grpc:proto-google-cloud-datastore-admin-v1 2.20.1
  • com.google.api.grpc:grpc-google-cloud-datastore-admin-v1 2.20.1
  • com.google.cloud:google-cloud-datastore 2.20.1
  • com.google.api.grpc:proto-google-cloud-datastore-v1 0.111.1
  • com.google.cloud.datastore:datastore-v1-proto-client 2.20.1
  • junit:junit 4.13.2
  • org.easymock:easymock 5.2.0
  • org.apache.maven.plugins:maven-compiler-plugin 3.13.0
  • com.google.errorprone:error_prone_core 2.28.0
proto-google-cloud-datastore-admin-v1/pom.xml
  • com.google.cloud:google-cloud-datastore-parent 2.20.1
proto-google-cloud-datastore-v1/pom.xml
  • com.google.cloud:google-cloud-datastore-parent 2.20.1
samples/install-without-bom/pom.xml
  • com.google.cloud.samples:shared-configuration 1.2.0
  • com.google.cloud:google-cloud-datastore 2.20.0
  • junit:junit 4.13.2
  • com.google.truth:truth 1.4.2
  • org.codehaus.mojo:build-helper-maven-plugin 3.6.0
samples/native-image-sample/pom.xml
  • com.google.cloud.samples:shared-configuration 1.2.0
  • com.google.cloud:libraries-bom 26.40.0
  • junit:junit 4.13.2
  • com.google.truth:truth 1.4.2
  • org.junit.vintage:junit-vintage-engine 5.10.2
  • org.graalvm.buildtools:junit-platform-native 0.10.2
  • org.apache.maven.plugins:maven-surefire-plugin 3.2.5
  • org.graalvm.buildtools:native-maven-plugin 0.10.2
samples/pom.xml
  • com.google.cloud.samples:shared-configuration 1.2.0
  • org.apache.maven.plugins:maven-deploy-plugin 3.1.2
  • org.sonatype.plugins:nexus-staging-maven-plugin 1.7.0
samples/snapshot/pom.xml
  • com.google.cloud.samples:shared-configuration 1.2.0
  • com.google.cloud:google-cloud-datastore 2.20.1
  • junit:junit 4.13.2
  • com.google.truth:truth 1.4.2
  • org.codehaus.mojo:build-helper-maven-plugin 3.6.0
samples/snippets/pom.xml
  • com.google.cloud.samples:shared-configuration 1.2.0
  • com.google.cloud:libraries-bom 26.40.0
  • junit:junit 4.13.2
  • com.google.truth:truth 1.4.2
regex
.kokoro/presubmit/graalvm-native-17.cfg
  • com.google.cloud:sdk-platform-java-config 3.31.0
.kokoro/presubmit/graalvm-native.cfg
  • com.google.cloud:sdk-platform-java-config 3.31.0
.github/workflows/unmanaged_dependency_check.yaml
  • com.google.cloud:sdk-platform-java-config 3.31.0

  • Check this box to trigger a request for Renovate to run again on this repository

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.