Supercharge Your Platform with Custom Terraform Providers
Integrating your custom platform API with your IaC can be a game changer.
What will you learn
In this article you will learn about the benefits of integrating your custom platform API with your Infrastructure as Code (IaC) tooling and how this integration can be a game changer. Additionally, you will discover how to create a custom Terraform provider using the newly available generator capabilities.
Develop your own Terraform Provider
Modern platforms often revolve around a central API that not only provides access to data but also acts as a powerful interface for automation. By integrating your API's resources into Terraform, you not only unlock new possibilities but also enhance your existing Terraform capabilities, creating a seamless experience for platform users by combining cloud provider interactions with streamlined platform management.
With a custom Terraform provider, you gain the flexibility to extend Terraform's capabilities to manage unique or specialized resources that are not covered by existing providers. This allows for seamless integration and automation of infrastructure management, tailored to your specific needs and API functionalities.
Imagine a scenario where your CI/CD pipeline serves as the orchestrator for provisioning infrastructure. With a custom Terraform provider, your pipeline could seamlessly interact with your platform's API to deploy resources, enforce configurations, and manage updates. This integration simplifies automation, reduces manual intervention, and ensures consistency across environments.
For a clean, complete and executable example of creating a custom Terraform provider, refer to my GitHub repository, which walks you through the process step by step.
Focusing on practical use cases like these demonstrates how a custom Terraform provider can drive efficiency, extend Terraform's capabilities, and align your infrastructure management with your development workflows.
How to generate your own terraform provider?
Prerequisites
API defined by the OpenAPI standard (3.1.x or 3.0.x).
Some familarity with Golang
Overview
Terraform offers you some tools to implement the provider based on the OpenAPI specification that your API implements.
OpenAPI Provider Spec Generator: is a command-line tool that converts an OpenAPI Specification (OAS) into a Provider Code Specification.
Framework Code Generator: is a CLI utility that automatically produces Terraform Provider code from a specified provider configuration.
OpenAPI Provider Spec Generator
The OpenAPI Provider Spec Generator simplifies the creation of custom Terraform providers by translating an OpenAPI Specification (OAS) into a Provider Code Specification. This output serves as a blueprint for generating Terraform provider code, streamlining the process and reducing manual effort.
What It Will Provide
Structured Resource Mapping: Translates the API's endpoints and schemas into a format that aligns with Terraform's resource structure.
Generates a Provider Specification: Produces an output that serves as a detailed blueprint for generating Terraform provider code.
Support for OpenAPI Standards: Works with OpenAPI 3.1.x and 3.0.x specifications, making it versatile for modern API definitions.
What It Will Not Provide
Despite its utility, the generator does not deliver a fully functional Terraform provider out of the box. There are several aspects it does not handle:
Business Logic Implementation: You’ll need to define the specific behaviours, validations, or transformations that map your API's logic to Terraform resources.
Provider-Specific Customizations: Custom code for advanced features, such as handling complex authentication flows or specialized API interactions, must be written separately.
Operational Readiness: The generator doesn’t handle deployment, testing, or maintenance of the provider; these responsibilities remain part of your development process.
This tool requires two inputs to make it work:
OpenAPI (3.1.x or 3.0.x) version
Generator Config: developers must specify which API operations (e.g., GET, POST) and components correspond to Terraform resources, data sources, or providers. These mappings are defined in a YAML configuration file, which is then used to generate the provider code specification.
# Generator config
version:
0.1.0
provider:
name: api
# Terraform datasources
data_sources:
order:
read:
path: /orders/{id}
method: GET
schema:
ignores:
- service.metadata
# Terraform resources
resources:
order:
read:
path: /orders/{id}
method: GET
create:
path: /orders
method: POST
update:
path: /orders/{id}
method: PATCH
How to use the OpenAPI Provider Spec Generator
Once you have the inputs ready, you can proceed to run the generator.
go install github.com/hashicorp/terraform-plugin-codegen-openapi/cmd/tfplugingen-openapi@latestGenerate the provider code specification.
tfplugingen-openapi generate \ --config <path/to/generator_config.yml> \ --output <output/for/provider_code_spec.json> \ <path/to/openapi_spec.json>
Framework Code Generator
The purpose of the Framework Code Generator is to take the Provider Code Spec and generate the necessary Terraform provider scaffolding that adheres to the Terraform Plugin Framework. While the OpenAPI Provider Spec Generator focuses on translating an OpenAPI specification into a Provider Code Spec, the Framework Code Generator transforms that spec into functional provider code.
This division ensures a clean separation of responsibilities: the first generator defines what the provider should do, and the second generator builds the provider code that implements it.
What It Will Provide
Code Scaffolding: Generates boilerplate code for Terraform providers, resources, and data sources, adhering to best practices.
Schema and Interface Support: Prepares schemas and ensures compatibility with Terraform Plugin Framework interfaces.
Error Handling and Logging: Offers patterns for robust error diagnostics and integrated logging.
Resource Stubs: Provides placeholders for resources and data sources, ready for customization.
What It Will Not Provide
Complete Custom Logic: While it generates the structure, the framework does not provide the implementation for specific API interactions or business logic.
Authentication Mechanisms.
API Client Libraries: Assumes you either have an existing API client library or will generate one separately. The framework won't create an SDK for your API.
Production-Ready Security Practices: While it provides templates for secure interaction, the framework doesn't enforce advanced encryption or runtime security best practices.
Testing: testing of the code.
How to use the Framework Code Generator
This tool also requires two inputs:
Provider Code Spec: the one we obtained using the OpenAPI Provider Spec Generator
Plugin Framework: which can be generated with this tool.
Installation
go installgithub.com/hashicorp/terraform-plugin-codegen-framework/cmd/tfplugingen-framework@latest
Generate the Plugin Framework mentioned above.
tfplugingen-framework generate all \ --input provider_code_spec.json \ --output generated
<provider_code_spec.json> is the file that we got in step 2 in the OpenAPI Provider Spec Generator
The generated Go files will include a _gen.go suffix and will feature a DO NOT EDIT notice at the top of each file.
These files include, among other things, the JSON specification of the API for the resource/data source in Terraform typing.
Scaffold command: generates starter code for a data source, provider, or resource.
tfplugingen-framework scaffold <data-source | provider | resource > \ --name example \ --force \ --output-dir provider
Executing this command generates the following scaffold code in ./output/scaffold/data_source/example_data_source.go
package provider
import (
"context"
"github.com/hashicorp/terraform-plugin-framework/datasource"
"github.com/hashicorp/terraform-plugin-framework/datasource/schema"
"github.com/hashicorp/terraform-plugin-framework/types"
)
var _ datasource.DataSource = (*exampleDataSource)(nil)
func NewExampleDataSource() datasource.DataSource {
return &exampleDataSource{}
}
type exampleDataSource struct{}
type exampleDataSourceModel struct {
Id types.String `tfsdk:"id"`
}
func (d *exampleDataSource) Metadata(ctx context.Context, req datasource.MetadataRequest, resp *datasource.MetadataResponse) {
resp.TypeName = req.ProviderTypeName + "_example"
}
func (d *exampleDataSource) Schema(ctx context.Context, req datasource.SchemaRequest, resp *datasource.SchemaResponse) {
resp.Schema = schema.Schema{
Attributes: map[string]schema.Attribute{
"id": schema.StringAttribute{
Computed: true,
},
},
}
}
func (d *exampleDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
var data exampleDataSourceModel
// Read Terraform configuration data into the model
resp.Diagnostics.Append(req.Config.Get(ctx, &data)...)
if resp.Diagnostics.HasError() {
return
}
// Read API call logic
// Example data value setting
data.Id = types.StringValue("example-id")
// Save data into Terraform state
resp.Diagnostics.Append(resp.State.Set(ctx, &data)...)
}
In this code // Read API call logic comment, you implement the logic for performing CRUD operations on your API using Terraform. Here, you will also define the resource or data source blocks in Terraform syntax.
Order update example code implementation:
// Schema defines the schema for the resource.
func (r *orderResource) Schema(_ context.Context, _ resource.SchemaRequest, resp *resource.SchemaResponse) {
resp.Schema = schema.Schema{
Description: "Manages an order.",
Attributes: map[string]schema.Attribute{
"id": schema.StringAttribute{
Computed: true,
},
"last_updated": schema.StringAttribute{
Computed: true,
},
"order_id": schema.StringAttribute{
Description: "Order state ID",
Required: true,
},
"state": schema.StringAttribute{
Description: "Order state",
Required: true,
},
},
}
}
// date updates the resource and sets the updated Terraform state on success.
func (r *orderResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
// Get current state
var u orderResourceModel
diags := req.Plan.Get(ctx, &u)
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}
// Generate API request body from plan
oId := order.OrderId(u.OrderId.ValueString())
_, err := r.client.PatchOrder(oId, order.OrderUpdateRequest{State: u.State.ValueString()})
if err != nil {
resp.Diagnostics.AddError(
"Error updating the order resource",
fmt.Sprintf("%v", err),
)
return
}
u.ID = types.StringValue(strconv.FormatInt(time.Now().Unix(), 10))
u.LastUpdated = types.StringValue(time.Now().Format(time.RFC850))
// Set refreshed state
diags = resp.State.Set(ctx, &u)
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}
}
Testing
Since this is Go code, you can write tests for it, and I highly recommend doing so. Writing tests provides a fast feedback cycle, allowing you to validate your logic early without the need to integrate your custom provider binary into your Terraform workflow.
func TestAccOrderResource_Update(t *testing.T) {
// Define initial and expected values
initialOrder := order.Order{
OrderId: "o-abcd123",
State: "pending",
ServiceId: "s-asdf123",
CustomerId: "c-obedwasg",
CloudAccount: order.CloudAccount{
Provider: "AWS",
Id: "1234567",
},
}
updatedOrder := order.Order{
OrderId: "o-abcd123",
State: "completed",
ServiceId: "s-asdf123",
CustomerId: "c-obedwasg",
CloudAccount: order.CloudAccount{
Provider: "AWS",
Id: "1234567",
},
}
initialApiResponse := order.OrdersResponse{
Orders: []order.Order{initialOrder},
}
updatedApiResponse := order.OrdersResponse{
Orders: []order.Order{updatedOrder},
}
// Setup the mock client
stubClient := testdata.StubClient{
GetOrderForServiceResponse: initialApiResponse,
PatchOrderResponse: updatedApiResponse, // Mock the update response
}
// Define provider factories
testAccProtoV6ProviderFactories := map[string]func() (tfprotov6.ProviderServer, error){
"api": providerserver.NewProtocol6WithError(NewProviderWithStubClient(stubClient)),
}
// Run the resource test
resource.Test(t, resource.TestCase{
ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
IsUnitTest: true,
Steps: []resource.TestStep{
{
// Create and update testing
Config: providerConfig + `
resource "api_order" "test" {
order_id = "o-abcd123"
state = "pending"
}
`,
Check: resource.ComposeAggregateTestCheckFunc(
resource.TestCheckResourceAttr("api_order.test", "order_id", "o-abcd123"),
resource.TestCheckResourceAttr("api_order.test", "state", "pending"),
),
},
{
// Update state
Config: providerConfig + `
resource "api_order" "test" {
order_id = "o-abcd123"
state = "completed"
}
`,
Check: resource.ComposeAggregateTestCheckFunc(
resource.TestCheckResourceAttr("api_order.test", "order_id", "o-abcd123"),
resource.TestCheckResourceAttr("api_order.test", "state", "completed"),
),
},
},
})
}
Designing and Deploying the Provider
We have explored how to generate resources and data sources, the essential components that define how Terraform interacts with specific elements of an API.
But what about the provider itself? The provider is the central piece that ties these components together, defining the configuration, authentication, and overall structure needed for seamless interaction with the API.
How do you design a provider.go file to encapsulate all resources and data sources effectively, ensuring they operate cohesively? Furthermore, how do you deploy this provider to make it ready for use in Terraform workflows?
How do you write a provider.go to encapsulate all the resources and data sources, and then deploy it?
The following code serves as an example of how to encapsulate the provider logic, integrating configuration, authentication, and resource management into a cohesive implementation. It demonstrates provider-level functionality, including basic authentication via an API password. While this approach lays a strong foundation for managing resources and data sources, it's worth noting that basic authentication is not the most secure method and should be revisited before deploying to production environments to ensure robust security practices.
package api_provider
// same package for resource_example.go and datasource_example.go
import (
"context"
"os"
"import-your-custom-api-client>"
"github.com/hashicorp/terraform-plugin-framework/datasource"
"github.com/hashicorp/terraform-plugin-framework/path"
"github.com/hashicorp/terraform-plugin-framework/provider"
"github.com/hashicorp/terraform-plugin-framework/provider/schema"
"github.com/hashicorp/terraform-plugin-framework/resource"
"github.com/hashicorp/terraform-plugin-framework/types"
"github.com/hashicorp/terraform-plugin-log/tflog"
)
// Ensure the implementation satisfies the expected interfaces
var (
_ provider.Provider = New()
)
// New is a helper function to simplify provider server and testing implementation.
// offers an interface for the API
func New() provider.Provider {
var c client.ApiClient = &client.SdkClient{}
return &apiProvider{client: c}
}
// apiProvider is the provider implementation.
type apiProvider struct {
client client.ApiClient
}
// apiProviderModel maps provider schema data to a Go type.
type apiProviderModel struct {
Host types.String `tfsdk:"host"`
Username types.String `tfsdk:"username"`
Password types.String `tfsdk:"password"`
}
// Metadata returns the provider type name.
func (p *apiProvider) Metadata(_ context.Context, _ provider.MetadataRequest, resp *provider.MetadataResponse) {
resp.TypeName = "api"
}
// Schema defines the provider-level schema for configuration data.
func (p *apiProvider) Schema(_ context.Context, _ provider.SchemaRequest, resp *provider.SchemaResponse) {
resp.Schema = schema.Schema{
Description: "Interact with API.",
Attributes: map[string]schema.Attribute{
"host": schema.StringAttribute{
Description: "URI for API. May also be provided via API_HOST environment variable.",
Optional: true,
},
"username": schema.StringAttribute{
Description: "Username for API. May also be provided via API_USERNAME environment variable.",
Optional: true,
},
"password": schema.StringAttribute{
Description: "Password for API. May also be provided via API_PASSWORD environment variable.",
Optional: true,
Sensitive: true,
},
},
}
}
// Configure prepares a HashiCups API client for data sources and resources.
func (p *apiProvider) Configure(ctx context.Context, req provider.ConfigureRequest, resp *provider.ConfigureResponse) {
tflog.Info(ctx, "Configuring API client")
// Retrieve provider data from configuration
var config apiProviderModel
diags := req.Config.Get(ctx, &config)
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}
host := os.Getenv("API_HOST")
username := os.Getenv("API_USERNAME")
password := os.Getenv("API_PASSWORD")
if !config.Host.IsNull() {
host = config.Host.ValueString()
}
if !config.Username.IsNull() {
username = config.Username.ValueString()
}
if !config.Password.IsNull() {
password = config.Password.ValueString()
}
// If any of the expected configurations are missing, return
// errors with provider-specific guidance.
if host == "" {
resp.Diagnostics.AddAttributeError(
path.Root("host"),
"Missing API Host",
"The provider cannot create the API client as there is a missing or empty value for the API host. "+
"Set the host value in the configuration or use the API_HOST environment variable. "+
"If either is already set, ensure the value is not empty.",
)
}
if username == "" {
resp.Diagnostics.AddAttributeError(
path.Root("username"),
"Missing API Username",
"The provider cannot create the API client as there is a missing or empty value for the API username. "+
"Set the username value in the configuration or use the API_USERNAME environment variable. "+
"If either is already set, ensure the value is not empty.",
)
}
if password == "" {
resp.Diagnostics.AddAttributeError(
path.Root("password"),
"Missing API Password",
"The provider cannot create the API client as there is a missing or empty value for the API password. "+
"Set the password value in the configuration or use the API_PASSWORD environment variable. "+
"If either is already set, ensure the value is not empty.",
)
}
if resp.Diagnostics.HasError() {
return
}
ctx = tflog.SetField(ctx, "api_host", host)
ctx = tflog.SetField(ctx, "api_username", username)
ctx = tflog.SetField(ctx, "api_password", password)
ctx = tflog.MaskFieldValuesWithFieldKeys(ctx, "api_password")
tflog.Debug(ctx, "Creating client")
p.client.ConfigureClient(username, password, host)
// type Configure methods.
resp.DataSourceData = p.client
resp.ResourceData = p.client
tflog.Info(ctx, "Configured client", map[string]any{"success": true})
}
// Register DataSources defines the data sources implemented in the provider.
func (p *apiProvider) DataSources(_ context.Context) []func() datasource.DataSource {
return []func() datasource.DataSource{
NewOrderDataSource,
}
}
// Register Resources defines the resources implemented in the provider.
func (p *apiProvider) Resources(_ context.Context) []func() resource.Resource {
return []func() resource.Resource{
NewOrderResource,
}
}
Build the provider
To use your custom Terraform provider, you need to build the Go binary. This step compiles your provider code into an executable file that Terraform can utilize. Once the binary is built, your provider is ready for integration with Terraform.
#!/bin/bash
# Define the binary name
bin_name="provider"
go_build() {
go mod tidy
local os=$1
local arch=$2
env GOOS="$os" GOARCH="$arch" CGO_ENABLED=0 go build -ldflags="-s -w" -o "${bin_name}-${os}-${arch}"
}
# Build for different OS and architectures
go_build linux amd64
go_build darwin amd64
go_build darwin arm64
Terraform Provider Installation
Prepare Your Custom Provider
Before proceeding with the Terraform installation, ensure your custom provider is uploaded to a registry. The registry acts as a distribution point, enabling Terraform to locate, download, and use the provider seamlessly during configuration. For experimentation or private use, you might consider the public Terraform Registry, which supports versioning and easy integration. However, keep in mind that the public registry requires your provider to be open-source. Alternatively, private registries, such as those hosted via Artifactory or GitHub Releases, can be set up for controlled, internal usage and testing environments.
Why Must the Provider Be Stored in a Registry First?
Terraform relies on a registry to fetch provider binaries when executing configurations. Storing your provider in the registry ensures:
Accessibility: Terraform can locate and download the provider automatically.
Version Control: The registry maintains versions of your provider, allowing you to specify the exact version needed for your configuration.
Team Collaboration: A registry provides a central location for shared providers, ensuring all team members work with the same version.
Security: For private providers, using a registry ensures they are securely hosted and only accessible to authorized users.
Uploading the provider to the registry is a prerequisite for Terraform to discover and use it in your projects.
Host configuration for using the provider
Create $HOME/.terraformrc
provider_installation {
network_mirror {
url = "https://example.com/path/to/providers/"
include = ["*/"]
}
direct {
exclude = ["terraform.local/*/*"]
}
filesystem_mirror {
path = "<$HOME>/.terraform.d/plugins"
}
}
Log in to the registry to be able to download the provider.
terraform login registry.example.com
Declare your provider in your Terraform configuration
terraform {
required_version = ">= 1.0"
required_providers {
api = {
source = "example-provider"
version = "2.5.0"
}
}
}
# set the order state to completed
resource "api_order" "example" {
order_id = "o-123456"
state = "completed"
}
Summary
Integrate your custom platform API with Infrastructure as Code (IaC) using Terraform to enhance automation, streamline infrastructure management, and unify cloud interactions with platform-specific operations. Learn to create a custom Terraform provider leveraging OpenAPI specifications and harness code generation tools to produce provider scaffolding. This guide covers the full process from generating the provider code and encapsulating resources to deploying the provider in a registry for version control and accessibility. Unlock new automation capabilities and ensure consistent infrastructure management across environments.
Key points to include:
Custom Terraform Provider: Extend Terraform's capabilities to manage unique resources not covered by existing providers, enabling seamless integration and automation tailored to your API functionalities.
OpenAPI Provider Spec Generator: Simplifies the creation of custom Terraform providers by translating an OpenAPI Specification into a Provider Code Specification.
Framework Code Generator: Transforms the Provider Code Spec into functional provider code, ensuring a clean separation of responsibilities.
Provider Design and Deployment: Learn how to encapsulate resources and data sources in a
provider.gofile, build the provider binary, and host it in a registry for accessibility and version control.Terraform Configuration: Instructions on configuring Terraform to use the custom provider, including an example of managing an order resource.
By following these steps, you can enhance your infrastructure management, align it with your development workflows, and unlock new automation possibilities.
Additionally, for a clean, complete, and executable example of creating a custom Terraform provider, you can refer to my GitHub repository, which walks you through the process step by step.