[RFE] Parallel Filestore deploy and destroy about hpc-toolkit HOT 7 CLOSED

Hi Federico,
As far as we know, that is a filestore issue. I will investigate that and come back to you shortly.

from hpc-toolkit.

Hi Federico,

I created a simple blueprint to try to reproduce the error and, here, the instances seem to be created in parallel.

I also had the impression of seeing this sequential creation before, could it have been circumstantial (based on the number of things being already created)? Could you try again with -parallelism=20 during the terraform apply?

FYI, the yaml I used for test now was:

# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---

blueprint_name: filestore-test

vars:
  project_id:  ## Set GCP Project ID Here ##
  deployment_name: filestore-test
  region: us-central1
  zone: us-central1-c

deployment_groups:
- group: primary
  modules:
  - id: network1
    source: modules/network/vpc

  - id: homefs
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home

  - id: homefs2
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home2

  - id: homefs3
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home3

  - id: homefs4
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home4

  - id: compute
    source: modules/compute/vm-instance
    kind: terraform
    use: [network1]
    settings:
      name_prefix: vm
      instance_count: 1

And the output seems to be parallel.

What do you think?

from hpc-toolkit.

Hey Carlos,

The output is definitely in parallel but on GCP the Filestores go up one at the time.

The following test has just been done on 1.4.1

from hpc-toolkit.

We are still working to track down if the observed behavior is a result of the terraform provider implementation or google cloud implementation.

from hpc-toolkit.

I found the root cause of this serialization to be described here: hashicorp/terraform-provider-google#9007
I am pursuing both the terraform and the internal Filestore team to see if we can change back the implementation in the provider and fix the underlying issue. This will take time.

from hpc-toolkit.

GoogleCloudPlatform/magic-modules#6672 should address the issue.

from hpc-toolkit.

v4.41.0
of the Terraform provider for Google Cloud removes the constraint for serial creation of Filestore instances. Any new blueprint deployment should automatically get this release of the provider. You can upgrade an existing environment by running terraform init -upgrade in an existing Terraform folder.

from hpc-toolkit.