What is Magic Modules?

Magic Modules is used to generate support for Google Cloud Platform services in declarative infrastructure management tools With Magic Modules, instead of hand-writing similar code for dozens of GCP services across several declarative tools, developers transcribe the services' REST-like APIs into Magic Modules' declarative resource definitions.

Once service APIs are encoded into resource definitions, they're used to generate first-class support for GCP services in every tool Magic Modules supports. Ideally, GCP's service APIs will always behave declaratively. While that's not always the case. Magic Modules is there to help by mapping from imperative APIs to declarative resources.

Magic Modules performs numerous transformations to represent imperative APIs as declarative resources, including:

For these reasons, mapping APIs into a declarative resource model requires thoughtful work from developers as they transcribe APIs into Magic Modules' API definitions.

What does Magic Modules support?

At the time of writing (May 2019), Magic Modules generates integrations with GCP services ("product"s) for the following tools ("providers"):

In addition, Magic Modules generates support for several companion features/tools:

What you'll accomplish

In this codelab, you'll add support for a new GCP service to Magic Modules, generating and testing it in a single provider, Terraform.

What you'll learn

What you'll need

It's also recommended that you have a working knowledge of Terraform and HTTP response codes. This codelab explains Magic Modules concepts, and not necessarily Terraform ones.

Cloning Magic Modules

You'll be working out of the Magic Modules GitHub repository. To start, clone the Magic Modules repository locally.

git clone https://github.com/GoogleCloudPlatform/magic-modules.git && cd magic-modules

Next, check out the mm-product-codelab branch.

git checkout mm-product-codelab

Setting up Magic Modules

Recommended: Use the bootstrap script

To get started right away, use the bootstrap script:

tools/bootstrap

./tools/bootstrap

Alternative: Manually perform setup

Otherwise, follow these manual steps:

First, check out the Terraform Google provider repositories. Magic Modules generates most but not all of the provider, and relies on an existing checkout of the provider to build correctly.

git clone https://github.com/terraform-providers/terraform-provider-google.git $GOPATH/src/github.com/terraform-providers/terraform-provider-google

cd $GOPATH/src/github.com/terraform-providers/terraform-provider-google

git checkout mm-product-codelab

git clone https://github.com/terraform-providers/terraform-provider-google-beta.git $GOPATH/src/github.com/terraform-providers/terraform-provider-google-beta

cd $GOPATH/src/github.com/terraform-providers/terraform-provider-google-beta

git checkout mm-product-codelab

Second, finish getting Magic Modules set up by installing the Ruby gems it needs to run:

bundle install

Verify your setup

Now, you can verify you're ready with:

tools/doctor

./tools/doctor

Using Magic Modules

Now that you've set up Magic Modules and both Google provider repositories, you can test out Magic Modules to validate your setup, and get familiar with using it.

Generating a Terraform provider

Run the following to generate the google Terraform provider using Magic Modules:

 bundle exec compiler -a -v "ga" -e terraform -o "$GOPATH/src/github.com/terraform-providers/terraform-provider-google"

Alternatively, google-beta can be generated with:

bundle exec compiler -a -v "beta" -e terraform -o "$GOPATH/src/github.com/terraform-providers/terraform-provider-google-beta"

You're using Bundler to invoke compiler.rb, where a series of options are defined. Using the first command, you've specified

-a

Generate all products

-v "ga"

Generate ga (generally available) version fields and resources

-e terraform

Use the terraform "engine" in Magic Modules

-o $PATH

output to the defined path

There are two other useful parameters:

-p "products/redis"

Generate the "redis" product (conflicts with -a)

-t "Instance"

Target (generate exclusively) the "Instance" resource

Building the provider

After running the command to generate the google provider, you can run make build from the provider directory to ensure it builds correctly.

cd $GOPATH/src/github.com/terraform-providers/terraform-provider-google
make build

After that, you can run a test to verify that it's working correctly. See the CONTRIBUTING guide and Provider Reference if you're unfamiliar with running tests on the provider.

Run a test to make sure you've correctly set up your environment:

make testacc TEST=./google TESTARGS='-run=TestAccComputeAddress_addressBasicExample'

What's your starting point?

For this codelab, we'll be adding support for a "new" GCP product, Cloud Memorystore. Cloud Memorystore is a fully-managed in-memory data store service for Redis. It's been released as a generally available product, but for the purposes of this codelab we'll be treating some features as if they're in beta.

"Beta" Features

During this codelab, pretend that authorizedNetwork and reservedIpRange are only configurable using the Beta API and not the GA (v1) one.That will let you explore the differences between versions.

Concepts

You were introduced to a few Magic Modules concepts through the prior commands. Here's a brief glossary so you're familiar with them before we break in to them with more detail.

You can see how these concepts map to the GCP REST API by breaking up the URL for a VM Instance resource:

https://redis.googleapis.com/v1/projects/{{projectId}}/locations/{{locationId}}/instances/{{instanceId}}

Researching Cloud Memorystore

As discussed previously, a product in Magic Modules is a rough mapping to a service in GCP. Cloud Memorystore is an example where the Magic Modules product maps 1:1 to a GCP service.

First, you need to determine how the product is named. While the service's brand name is Cloud Memorystore, it isn't always true that the API is named the same. We're more concerned with the naming of the API in Magic Modules, and you can pull that name from the REST reference for the product.

Based on the service URL of redis.googleapis.com, you can infer that the product's API name is redis. In addition, this documentation states that all URIs are relative to a base url of https://redis.googleapis.com.

From here, this codelab will primarily refer to Cloud Memorystore as Redis, based on the API name. With this information- the name and base url -you can start constructing the Magic Modules product.

API definitions (api.yaml)

In Magic Modules, each product has a folder under the products/ directory, named after the product's API name. Inside that folder, you can expect to find a few files:

api.yaml

The API definitions for the product, recorded based on the REST API

terraform.yaml

Terraform's "overrides", corrections to the API definitions and extra tool-specific information. This codelab will touch on this more in a future section. ansible.yaml and inspec.yaml may exist for those tools.

Other files

Other tools may have placed other unrelated files in a product folder as well.

Since you haven't defined a product folder for Redis yet, you can look at one for another file such as Cloud Build (cloudbuild) for an example.

api.yaml files, called API definitions, are made up of Ruby objects marshaled to YAML. API definitions are made up of Ruby types defined in the api/ directory of the repository. Their entries contain type markup that allows Magic Modules to unmarshal the YAML definitions directly into data objects for use in templates.

You can find a reference for the fields used in each marshaled object by browsing the api/ directory, such as product.rb. Alternatively, you can browse other products to learn by convention:

products/pubsub/api.yaml

--- !ruby/object:Api::Product
name: Pubsub
display_name: Cloud Pub/Sub
versions:
  - !ruby/object:Api::Product::Version
    name: ga
    base_url: https://pubsub.googleapis.com/v1/
scopes:
  - https://www.googleapis.com/auth/pubsub
apis_required:
  - !ruby/object:Api::Product::ApiReference
    name: Cloud Pub/Sub API
    url: https://console.cloud.google.com/apis/library/pubsub.googleapis.com/
objects:
  - !ruby/object:Api::Resource
    name: 'Topic'
...

From this, you can see a few important details:

scopes is a required field. While it's value would differ on older products, for newer GCP products it that shares a common value of https://www.googleapis.com/auth/cloud-platform.

Other than the optional apis_required field, you're missing one piece of product-level information for Redis, the version. Looking back at the REST reference for the product can you determine what it is? (A solution is provided below)

Creating api.yaml

From the root directory of the Magic Modules repo, create a redis/ directory under products/ and an api.yaml file inside. Then, open the file in your text editor of choice.

mkdir products/redis
touch products/redis/api.yaml

Next, fill out a base api.yaml file:

--- !ruby/object:Api::Product
name: Redis
display_name: Cloud Memorystore
scopes:
  - https://www.googleapis.com/auth/cloud-platform
versions:
  - !ruby/object:Api::Product::Version
    name: ga
    base_url: https://redis.googleapis.com/v1/
objects:

Adding asynchronous Operation handling

Across GCP, APIs use an "Operation" object to represent asynchronous operations. They can be polled, and return the status of a long-running operation. Pubsub doesn't return operations, but Redis does. The operation object has a common structure across most APIs.

You can pull the async definition from another product such as AccessContextManager and include this in your Redis product definition.

products/accesscontextmanager/api.yaml

--- !ruby/object:Api::Product
name: AccessContextManager

...

async: !ruby/object:Api::Async
  operation: !ruby/object:Api::Async::Operation
    path: 'name'
    base_url: '{{op_id}}'
    wait_ms: 1000
  result: !ruby/object:Api::Async::Result
    path: 'response'
    resource_inside_response: true
  status: !ruby/object:Api::Async::Status
    path: 'done'
    complete: true
    allowed:
      - true
      - false
  error: !ruby/object:Api::Async::Error
    path: 'error'
    message: 'message'

...

Adding Redis Instance to api.yaml

With support for the Redis product added, it's time to break in to the single resource, Redis Instance. A reference of the supported fields in a resource is available in api/resource.rb. In addition, you can use the Pubsub Topic resource as a reference.

products/pubsub/api.yaml

 - !ruby/object:Api::Resource
    name: 'Topic'
    base_url: projects/{{project}}/topics
    create_verb: :PUT
    input: true
    collection_url_response: !ruby/object:Api::Resource::ResponseList
      items: 'topics'
    references: !ruby/object:Api::Resource::ReferenceLinks
      guides:
        'Managing Topics':
          'https://cloud.google.com/pubsub/docs/admin#managing_topics'
      api: 'https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics'
    description: |
      A named resource to which messages are sent by publishers.
    properties:
      - !ruby/object:Api::Type::String
        name: 'name'
        required: true
        description: 'Name of the topic.'
      - !ruby/object:Api::Type::KeyValuePairs
        name: 'labels'
        description: |
          A set of key/value label pairs to assign to this Topic.
...

Reviewing the definitions for Pubsub Topic, you can see that:

For Redis Instance, based on the resource REST reference:

In addition, a description and reference_links value will need to be defined.

Based on the information you've gathered, you can create an entry for Redis Instance.

  - !ruby/object:Api::Resource
    name: 'Instance'
    base_url: projects/{{project}}/locations/{{region}}/instances
    create_url: projects/{{project}}/locations/{{region}}/instances?instanceId={{name}}
    update_verb: :PATCH
    update_mask: true
    description: |
      A Google Cloud Redis instance.
    references: !ruby/object:Api::Resource::ReferenceLinks
      guides:
        'Official Documentation':
          'https://cloud.google.com/memorystore/docs/redis/'
      api: 'https://cloud.google.com/memorystore/docs/redis/reference/rest/'
    properties:
...

Adding property definitions to api.yaml

With a skeleton for the Redis resource defined, you're now able to add properties to the resource. Magic Modules' API definitions represent REST API fields with typed property entries. For example, the description field in GCP is typically an optional string.

- !ruby/object:Api::Type::String
  name: 'description'
  description: |
    An optional description of this resource.

On the other hand, some properties might be another type, or required by the API.

- !ruby/object:Api::Type::Integer
  name: 'maxNumReplicas'
  required: true
  description: |
    The maximum number of instances that the autoscaler can scale up to. This is required when creating or updating an autoscaler. The maximum number of replicas should not be lower than minimal number of replicas.

A full reference of the types and properties available on those types is available in api/type.rb. The most notable properties are:

Given the REST reference for Redis Instance and the contents of api/type.rb, you're able to transcribe most of the fields into Redis Instance's api.yaml entry.

You should work through some of the fields yourself, although a full definition of the supported fields at the time of writing will be provided prior to a later step. If you'd like to save some time while still covering most of the process, try these fields:

What are tool-specific overrides?

Each downstream tool defines a "override" type to allow Magic Modules authors to:

These can be applied to properties and resources. In addition, a "provider config" can be used to add tool-specific values to a product.

In Terraform:

For example, if a property has required: true defined, Terraform's overrides can define required: false to make the property optional in Terraform.

Product-level Terraform Config

Terraform uses a limited amount of tool-specific config. You can use this template for the terraform.yaml file, placing it at products/redis/terraform.yaml.

# Copyright 2019 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

--- !ruby/object:Provider::Terraform::Config
overrides: !ruby/object:Overrides::ResourceOverrides

files: !ruby/object:Provider::Config::Files
  compile:
<%= lines(indent(compile('provider/terraform/product~compile.yaml'), 4)) -%>

This adds some files generated based on product-level information. Currently, that's exclusively the generated provider entries.

Adding the product to the provider map

When adding a new product to Magic Modules, you'll need to add the product's generated provider entries to the provider's map between resource names (like google_redis_instance) and their resource definitions. You can modify third_party/terraform/utils/provider.go.erb to add it.

The relevant part of that file contains the following;

third_party/terraform/utils/provider.go.erb

func ResourceMapWithErrors() (map[string]*schema.Resource, error) {
        return mergeResourceMaps(
                        <% unless version == 'ga' -%>
                        // start beta-only products
                        GeneratedBinaryAuthorizationResourcesMap,
                        GeneratedContainerAnalysisResourcesMap,
                        GeneratedSecurityScannerResourcesMap,
                        // end beta-only products
                        <% end -%>
                        GeneratedAccessContextManagerResourcesMap,
                        GeneratedAppEngineResourcesMap,
                        GeneratedComputeResourcesMap,

Add GeneratedRedisResourcesMap to the list of generated entries.

Fine-tuning the Redis Instance Resource with overrides

In Terraform, resource-level overrides are used to configure specific behaviours of the resource at import time, generate documentation and some limited tests, and attach custom code when necessary.

You can use the Pubsub Topic overrides as an example:

products/pubsub/terraform.yaml

 Topic: !ruby/object:Overrides::Terraform::ResourceOverride
    id_format: "projects/{{project}}/topics/{{name}}"
    examples:
      - !ruby/object:Provider::Terraform::Examples
        name: "pubsub_topic_basic"
        primary_resource_id: "example"
        vars:
          topic_name: "example-topic"
    properties:

In that resource, an explicit id_format is added and an example is added.

That file contains the following, mapping the primary_resource_id and vars values into the template at compile time.

templates/terraform/examples/pubsub_topic_basic.tf.erb

resource "google_pubsub_topic" "<%= ctx[:primary_resource_id] %>" {
  name = "<%= ctx[:vars]['topic_name'] %>"

  labels = {
    foo = "bar"
  }
}

You can create a similar example for Redis Instance, using a few of the properties you defined previously. For example, you could create the following:

templates/terraform/examples/redis_instance_basic.tf.erb

resource "google_redis_instance" "<%= ctx[:primary_resource_id] %>" {
  name           = "<%= ctx[:vars]["instance_name"] %>"
  memory_size_gb = 1
}

Using an example like that you'll end up with a Redis Instance config similar to the following:

Instance: !ruby/object:Overrides::Terraform::ResourceOverride
    id_format: "projects/{{project}}/locations/{{region}}/instances/{{name}}"
    examples:
      - !ruby/object:Provider::Terraform::Examples
        name: "redis_instance_basic"
        primary_resource_id: "cache"
        vars:
          instance_name: "memory-cache"

You can use property overrides to adjust the behaviour of fields in Terraform. This can include overriding api.yaml properties directly, such as setting required: false or the transformations available in overrides/terraform/property_override.rb.

One commonly used Terraform-specific override is default_from_api, used to mark fields as having a default when an unset value is sent to the API. Other overrides allow you to add Terraform-specific fields like diff suppress functions and "statefuncs" that transform values as they're recorded in Terraform state.

Turning back to Pubsub Topic, only a single property is overridden. The name field—normally the base_url of the resource in that API—is collapsed to a short name similar to older APIs like Compute. That's accomplished using a custom expander and flattener. While these two are shared between multiple resources, it's also common for custom code snippets to be namespaced for a product/resource, indicating it makes assumptions relevant for a single product or resource API.

products/pubsub/terraform.yaml

   properties:
      name: !ruby/object:Overrides::Terraform::PropertyOverride
        diff_suppress_func: 'compareSelfLinkOrResourceName'
        custom_expand: templates/terraform/custom_expand/resource_from_self_link.go.erb
        custom_flatten: templates/terraform/custom_flatten/name_from_self_link.erb

Common Terraform Property Overrides

There are several common overrides Terraform uses when preserving behaviour of resources that we previously handwritten, or to enable a better user experience.

     region: !ruby/object:Overrides::Terraform::PropertyOverride
        ignore_read: true
        required: false
        default_from_api: true

Redis Instance Property Overrides

Applying the common overrides for region and name and using the provided custom expand for name, you'll end up with the following:

   properties:
      name: !ruby/object:Overrides::Terraform::PropertyOverride
        custom_expand: 'templates/terraform/custom_expand/redis_instance_name.erb'
        custom_flatten: 'templates/terraform/custom_flatten/name_from_self_link.erb'
      region: !ruby/object:Overrides::Terraform::PropertyOverride
        ignore_read: true
        required: false
        default_from_api: true

After toying with the API and seeing what defaults are returned, you would find you need to add:

      alternativeLocationId: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true
      locationId: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true
      redisVersion: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true

Generated Operation Object

Previously, you added the definition for an Operation to the product with the async block.

Because Terraform was previously handwritten, only a subset of Operation handlers are generated at the time of writing. We mark resources using generated Operation handlers with autogen_async: true to indicate that an operation handler should be generated and used. Since Redis is a new resource, you should add that to the Terraform override definitions.

You'll end up with an api.yaml and terraform.yaml file similar to the following. If you encounter issues in the following steps, look for differences between your files and these partial solutions.

products/redis/api.yaml

# Copyright 2019 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

--- !ruby/object:Api::Product
name: Redis
display_name: Cloud Memorystore
scopes:
  - https://www.googleapis.com/auth/cloud-platform
versions:
  - !ruby/object:Api::Product::Version
    name: ga
    base_url: https://redis.googleapis.com/v1/
async: !ruby/object:Api::Async
  operation: !ruby/object:Api::Async::Operation
    path: 'name'
    base_url: '{{op_id}}'
    wait_ms: 1000
  result: !ruby/object:Api::Async::Result
    path: 'response'
    resource_inside_response: true
  status: !ruby/object:Api::Async::Status
    path: 'done'
    complete: true
    allowed:
      - true
      - false
  error: !ruby/object:Api::Async::Error
    path: 'error'
    message: 'message'
objects:
  - !ruby/object:Api::Resource
    name: 'Instance'
    base_url: projects/{{project}}/locations/{{region}}/instances
    create_url: projects/{{project}}/locations/{{region}}/instances?instanceId={{name}}
    update_verb: :PATCH
    update_mask: true
    description: |
      A Google Cloud Redis instance.
    references: !ruby/object:Api::Resource::ReferenceLinks
      guides:
        'Official Documentation':
          'https://cloud.google.com/memorystore/docs/redis/'
      api: 'https://cloud.google.com/memorystore/docs/redis/reference/rest/'
    properties:
      - !ruby/object:Api::Type::String
        name: name
        description: |
          The ID of the instance or a fully qualified identifier for the instance. 
        required: true
        input: true
      - !ruby/object:Api::Type::String
        name: 'region'
        description: |
          The name of the Redis region of the instance.
        required: true
        input: true
        url_param_only: true
      - !ruby/object:Api::Type::Integer
        name: memorySizeGb
        description: Redis memory size in GiB.
        required: true
      - !ruby/object:Api::Type::Integer
        name: port
        description: The port number of the exposed Redis endpoint.
        output: true
      - !ruby/object:Api::Type::String
        name: redisVersion
        description: |
          The version of Redis software. If not provided, latest supported
          version will be used. Updating the version will perform an
          upgrade/downgrade to the new version. Currently, the supported values
          are REDIS_3_2 for Redis 3.2.
        input: true
      - !ruby/object:Api::Type::String
        name: alternativeLocationId
        description: |
          Only applicable to STANDARD_HA tier which protects the instance
          against zonal failures by provisioning it across two zones.
          If provided, it must be a different zone from the one provided in
          [locationId].
        input: true
      - !ruby/object:Api::Type::Time
        name: createTime
        description: |
          The time the instance was created in RFC3339 UTC "Zulu" format,
          accurate to nanoseconds.
        output: true
      - !ruby/object:Api::Type::String
        name: currentLocationId
        description: |
          The current zone where the Redis endpoint is placed.
          For Basic Tier instances, this will always be the same as the
          [locationId] provided by the user at creation time. For Standard Tier
          instances, this can be either [locationId] or [alternativeLocationId]
          and can change after a failover event.
        output: true
        input: true
      - !ruby/object:Api::Type::String
        name: displayName
        description: |
          An arbitrary and optional user-provided name for the instance.
      - !ruby/object:Api::Type::String
        name: host
        description: |
          Hostname or IP address of the exposed Redis endpoint used by clients
          to connect to the service.
        output: true
      - !ruby/object:Api::Type::KeyValuePairs
        name: 'labels'
        description: Resource labels to represent user provided metadata.
      - !ruby/object:Api::Type::KeyValuePairs
        name: 'redisConfigs'
        description: |
          Redis configuration parameters, according to http://redis.io/topics/config.
          Please check Memorystore documentation for the list of supported parameters:
          https://cloud.google.com/memorystore/docs/redis/reference/rest/v1/projects.locations.instances#Instance.FIELDS.redis_configs
      - !ruby/object:Api::Type::String
        name: locationId
        description: |
          The zone where the instance will be provisioned. If not provided,
          the service will choose a zone for the instance. For STANDARD_HA tier,
          instances will be created across two zones for protection against
          zonal failures. If [alternativeLocationId] is also provided, it must
          be different from [locationId].
        input: true
      - !ruby/object:Api::Type::Enum
        name: tier
        description: |
          The service tier of the instance. Must be one of these values:

          - BASIC: standalone instance
          - STANDARD_HA: highly available primary/replica instances
        values:
          - :BASIC
          - :STANDARD_HA
        default_value: :BASIC
        input: true

products/redis/terraform.yaml

# Copyright 2019 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

--- !ruby/object:Provider::Terraform::Config
overrides: !ruby/object:Overrides::ResourceOverrides
  Instance: !ruby/object:Overrides::Terraform::ResourceOverride
    id_format: "projects/{{project}}/locations/{{region}}/instances/{{name}}"
    autogen_async: true    
    examples:
      - !ruby/object:Provider::Terraform::Examples
        name: "redis_instance_basic"
        primary_resource_id: "cache"
        vars:
          instance_name: "memory-cache"
    properties:
      name: !ruby/object:Overrides::Terraform::PropertyOverride
        custom_expand: 'templates/terraform/custom_expand/redis_instance_name.erb'
        custom_flatten: 'templates/terraform/custom_flatten/name_from_self_link.erb'
      region: !ruby/object:Overrides::Terraform::PropertyOverride
        ignore_read: true
        required: false
        default_from_api: true
      alternativeLocationId: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true
      locationId: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true
      redisVersion: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true
files: !ruby/object:Provider::Config::Files
  compile:
<%= lines(indent(compile('provider/terraform/product~compile.yaml'), 4)) -%>

Generating your resource

By now, you've recorded full API and override definitions for the Redis Instance resource, and added support for Redis to Magic Modules. Next, you can test the Terraform integration to make sure it works as expected. You can generate it with the full command, or by generating it with only the Redis product.

 bundle exec compiler -a -v "ga" -e terraform -o "$GOPATH/src/github.com/terraform-providers/terraform-provider-google"

or

 bundle exec compiler -p products/redis -v "ga" -e terraform -o "$GOPATH/src/github.com/terraform-providers/terraform-provider-google"

This will add a few files to the Terraform Google Provider repo's directory:

Testing your resource

Now that the resource is generated and has a test, you can exercise the test to see if the resource works as expected. From $GOPATH/src/github.com/terraform-providers/terraform-provider-google, run the following replacing the test name with your generated test if it's different:

make testacc TEST=./google TESTARGS='-run=TestAccRedisInstance_redisInstanceBasicExample'

Now that your Redis Instance resource is working, you can revisit the fields we're considering Beta during the codelab, authorizedNetwork and reservedIpRange. If these fields were truly beta-only, you wouldn't see the fields in the v1 REST reference but would see them in the v1beta1 reference.

Basic Property Definitions

Configuration of beta fields is the same as GA fields, adding standard api.yaml and terraform.yaml definitions:

     - !ruby/object:Api::Type::String
        name: authorizedNetwork
        input: true
        description: |
          The full name of the Google Compute Engine network to which the
          instance is connected. If left unspecified, the default network
          will be used.
     - !ruby/object:Api::Type::String
        name: reservedIpRange
        input: true
        description: |
          The CIDR range of internal addresses that are reserved for this
          instance. If not provided, the service will choose an unused /29
          block, for example, 10.0.0.0/29 or 192.168.0.0/29. Ranges must be
          unique and non-overlapping with existing subnets in an authorized
          network.
authorizedNetwork: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true
        custom_expand: 'templates/terraform/custom_expand/redis_instance_authorized_network.erb'
     reservedIpRange: !ruby/object:Overrides::Terraform::PropertyOverride
        default_from_api: true

The custom expander for authorizedNetwork is already included in the repo. If you're interested in what it does, the contents are posted below:

templates/terraform/custom_expand/redis_instance_authorized_network.erb

func expand<%= prefix -%><%= titlelize_property(property) -%>(v interface{}, d TerraformResourceData, config *Config) (interface{}, error) {
        fv, err := ParseNetworkFieldValue(v.(string), d, config)
        if err != nil {
                return nil, err
        }
        return fv.RelativeLink(), nil
}

Right now, Magic Modules thinks these fields are at GA, and a command with -v "ga" would include them. You'll need to add a minimum version so that they're generated only when -v "beta" is provided.

Adding a minimum version

You can apply a version to a field by marking it with a min_version value in the API definitions. Magic Modules supports two versions based on the GCP Product Launch Stages:

Minimum versions can be applied to any of a product, resource, or property. By adding the min_version: beta to the api.yaml definitions for the properties, they'll be ignored while generating the GA provider and added to the Beta provider.

Including min_version attributes, the api.yaml definition for the fields will look like this, with terraform.yaml unchanged:

     - !ruby/object:Api::Type::String
        name: authorizedNetwork
        min_version: beta
        input: true
        description: |
          The full name of the Google Compute Engine network to which the
          instance is connected. If left unspecified, the default network
          will be used.
     - !ruby/object:Api::Type::String
        name: reservedIpRange
        min_version: beta
        input: true
        description: |
          The CIDR range of internal addresses that are reserved for this
          instance. If not provided, the service will choose an unused /29
          block, for example, 10.0.0.0/29 or 192.168.0.0/29. Ranges must be
          unique and non-overlapping with existing subnets in an authorized
          network.

Adding a versioned example

When writing examples in terraform.yaml, they'll be generated at both GA and Beta by default. Examples support min_version to configure the version the corresponding test is generated at. If they're written for a resource in beta (or whose product is in beta) they'll inherit that version automatically.

When an example is versioned at beta, you'll need to specify the Terraform provider used in the tests for that example similar to real-world use of the google-beta provider per the provider versions guide for the Google providers.

Consider if you versioned the example used previously in this codelab:

      - !ruby/object:Provider::Terraform::Examples
        name: "redis_instance_basic"
        min_version: beta
        primary_resource_id: "cache"
        vars:
          instance_name: "memory-cache"

You would need to change the config to the following:

templates/terraform/examples/redis_instance_basic.tf.erb

resource "google_redis_instance" "<%= ctx[:primary_resource_id] %>" {
  provider = "google-beta"

  name           = "<%= ctx[:vars]["instance_name"] %>"
  memory_size_gb = 1
}

provider "google-beta" {
  region = "us-central1"
  zone   = "us-central1-a"
}

Versioned examples will show up in the documentation at every version but will only generate tests at versions where they're supported.

Now that you've added the Redis Instance resource including versioned fields to Magic Modules, you're almost done!

Handwritten Files

The Google provider existed for several years before Magic Modules was used to manage it, and many parts of it remain handwritten today. Handwritten files are used for legacy resources not yet ported to MM, advanced tests, and to add shared provider-wide code. These handwritten files are included under the third_party/terraform directory, divided by type of file.

Handwritten files sometimes end in .erb indicating they're versioned. Inside these files, you'll see Ruby ERB conditionals that control which code is generated at which version of the provider. If adding beta-only code, it may be necessary to modify inside these conditionals or to add new ones.

Testing advanced functionality

In addition to any generated example tests you've written, you'll want to exercise most of the fields of the resource in handwritten tests, as well as test if updating the resource works.

Before writing a test file, you can inspect a generated test file. Inside, you'll see a few different parts:

You can add your own simple tests as a handwritten file by copying a generated test + config pair into a new file and modifying the name and values as appropriate. In the Google provider, we try to exercise each field at least once in handwritten tests.

In order to perform an update, you'll need to add more test steps. For a simple single-stage test with a single config, you'll see a single config step followed by an import step:

func testAccRedisInstance_redisInstanceFooConfig(context map[string]interface{}) string {
        return Nprintf(`
resource "google_redis_instance" "cache" {
  name           = "memory-cache-%{random_suffix}"
  memory_size_gb = 1
}
`, context)
}
                Steps: []resource.TestStep{
                        {
                                Config: testAccRedisInstance_redisInstanceFooConfig(context),
                        },
                        {
                                ResourceName:            "google_redis_instance.cache",
                                ImportState:             true,
                                ImportStateVerify:       true,
                                ImportStateVerifyIgnore: []string{"region"},
                        },
                },

To test update, you'll need to add a second config function alongside a second pair of config+import steps to the test function:

func testAccRedisInstance_redisInstanceFooConfig(context map[string]interface{}) string {
        return Nprintf(`
resource "google_redis_instance" "cache" {
  name           = "memory-cache-%{random_suffix}"
  memory_size_gb = 1
}
`, context)
}

func testAccRedisInstance_redisInstanceBarConfig(context map[string]interface{}) string {
        return Nprintf(`
resource "google_redis_instance" "cache" {
  name           = "memory-cache-%{random_suffix}"
  memory_size_gb = 2
}
`, context)
}
                Steps: []resource.TestStep{
                        {
                                Config: testAccRedisInstance_redisInstanceFooConfig(context),
                        },
                        {
                                ResourceName:            "google_redis_instance.cache",
                                ImportState:             true,
                                ImportStateVerify:       true,
                                ImportStateVerifyIgnore: []string{"region"},
                        },
                },
{
                                Config: testAccRedisInstance_redisInstanceBarConfig(context),
                        },
                        {
                                ResourceName:            "google_redis_instance.cache",
                                ImportState:             true,
                                ImportStateVerify:       true,
                                ImportStateVerifyIgnore: []string{"region"},
                        },
                },

Adding your documentation page to the sidebar

In order to have your resource appear on the Google provider docsite sidebar, you'll need to create the sidebar entry. You can add the file to third_party/terraform/website-compiled/google.erb as part of your PR. For example, an entry for Redis will look like the following:

third_party/terraform/website-compiled/google.erb

   <li<%%= sidebar_current("docs-google-redis") %>>
    <a href="#">Google Redis (Cloud Memorystore) Resources</a>
    <ul class="nav nav-visible">
      <li<%%= sidebar_current("docs-google-redis-instance") %>>
      <a href="/docs/providers/google/r/redis_instance.html">google_redis_instance</a>
      </li>
    </ul>
    </li>

Congratulations, you've successfully added a new product and resource to Magic Modules, and tested it in Terraform!

What's next?

Now that you're familiar with Magic Modules, you can add new products and resources, or modify existing ones to add new versions or features. See the Magic Modules README for next steps on submitting changes.

Further reading

Reference docs