Nomad OpenID Connect with AWS Identity Provider

This page describes how to integrate Nomad as an OpenID Connect (OIDC) provider with AWS IAM Identity.

Prerequisites

You must have the following before continuing.

A running Nomad cluster, version 1.7.x or later, with TLS enabled.
An AWS account with the necessary permissions to create IAM roles, policies, hosted zones, and certificates.
Terraform installed locally and configured to communicate with AWS.

Set up and configure the Nomad cluster domain

In order to use Nomad as an identity provider, you need to have a trusted SSL certificate for the domain used by the cluster. You will use AWS Certificate Manager (ACM) in this example.

Create a Hosted Zone

Use the aws_route53_zone resource.

variable domain_name {
    type = "string"
    default = "<DOMAIN_NAME>"
}

resource "aws_route53_zone" "example" {
    name = var.domain_name
}

This configuration requires the following information:

<DOMAIN_NAME>: The domain name of your Nomad cluster without the protocol or port. var.domain_name is used throughout the examples on this page to reference <DOMAIN_NAME>.

Generate an SSL Certificate

Use the aws_acm_certificate and aws_acm_certificate_validation resources to create a request and provide validation for an SSL certficiate.

resource "aws_acm_certificate" "example" {
  domain_name       = var.domain_name
  validation_method = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_acm_certificate_validation" "example" {
  certificate_arn         = aws_acm_certificate.example.arn
  validation_record_fqdns = [var.domain_name]
}

Configure Application Load Balancer and DNS Alias

Create an ALB

Use the aws_lb resource.

resource "aws_lb" "test" {
  name               = "test-lb-tf"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [<SECURITY_GROUP_ID>]
  subnets            = [<SUBNET_IDS>]

  enable_deletion_protection = true

  access_logs {
    bucket  = <S3_BUCKET_ID>
    prefix  = "test-lb"
    enabled = true
  }
}

This configuration requires the following information:

<SECURITY_GROUP_ID>: A list of security groups IDs that you want to apply to the load balancer.
<SUBNET_IDS>: A list of subnet IDs that you want to attach to the load balancer.
<S3_BUCKET_ID>: The ID of an S3 bucket to store the load balancer access logs in.

Create a load balancer listener

Use the aws_lb_listener resource.

resource "aws_lb_listener" "example" {
  load_balancer_arn = <LB_ARN>
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-2016-08"
  certificate_arn   = <CERT_ARN>

  default_action {
    type             = "forward"
    target_group_arn = <LB_TARGET_GROUP_ARN>
  }
}

This configuration requires the following information:

<LB_ARN>: The AWS resource name of the load balancer. This will be aws_lb.test.arn if you are using the examples from this page.
<CERT_ARN>: The AWS resource name of the load balancer's certificate. This will be aws_acm_certificate_validation.example.certificate_arn if you are using the examples from this page.
<LB_TARGET_GROUP_ARN>: The AWS resource name of the target group for the load balancer to route traffic to.

Create a DNS Alias for the ALB

Use the aws_route53_record resource.

resource "aws_route53_record" "www" {
  zone_id = <ROUTE53_ZONE_ID>
  name    = var.domain_name
  type    = "A"

  alias {
    name                   = <LB_ALIAS_DNS_NAME>
    zone_id                = <LB_ALIAS_ZONE_ID>
    evaluate_target_health = true
  }
}

This configuration requires the following information:

<ROUTE53_ZONE_ID>: The ID of the hosted zone. This will be aws_route53_zone.example.zone_id if you are using the examples from this page.
<LB_ALIAS_DNS_NAME>: The DNS name of the load balancer. This will be aws_lb.test.dns_name if you are using the examples from this page.
<LB_ALIAS_ZONE_ID>: The zone ID of the load balancer. This will be aws_lb.test.zone_id if you are using the examples from this page.

Configure the AWS IAM Identity Provider

Create an OIDC Identity Provider in AWS

Use the aws_iam_openid_connect_provider resource.

data "tls_certificate" "example" {
  url = <CERT_DOMAIN_NAME>
}

resource "aws_iam_openid_connect_provider" "nomad" {
  # Nomad HTTPS URL
  url = <CERT_DOMAIN_NAME>

  client_id_list = [
    "aws",
  ]

  thumbprint_list = [data.tls_certificate.example.certificates.[0].sha1_fingerprint]
}

This configuration requires the following information:

<CERT_DOMAIN_NAME>: The domain name of the load balancer certificate. This will be aws_acm_certificate.example.domain_name if you are using the examples from this page.

Create an IAM policy for OIDC Federated Users

Use the aws_iam_role resource to create an appropriate IAM role for federated users. This will be specific to your use-case. This example is setting S3 bucket access.

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/$OIDC_PROVIDER"]
    }

    actions = ["sts:AssumeRoleWithWebIdentity"]

    condition {
        test = "StringEquals"
        variable = "$OIDC_PROVIDER:aud"
        values = "aws"
    }
  }
}

resource "aws_iam_role" "s3_all_access_role" {
  name = "s3_all_access_role"

  # Terraform's "jsonencode" function converts a
  # Terraform expression result to valid JSON syntax.
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
            "s3:*",
            "s3-object-lambda:*"
        ]
        Resource = "*"
      },
    ]
  })

  tags = {
    tag-key = "tag-value"
  }
}

resource "aws_iam_policy" "policy" {
  name        = "nomad-oidc-policy"
  description = "A policy for federated Nomad OIDC"
  policy      = data.aws_iam_policy_document.assume_role.json
}

resource "aws_iam_role_policy_attachment" "test-attach" {
  role       = aws_iam_role.s3_all_access_role.name
  policy_arn = aws_iam_policy.policy.arn
}

This configuration requires the following information:

<AWS_ACCOUNT_ID>:The ID of the AWS account where the IAM role from the previous step was created.

Configure Nomad for OIDC

Update the Nomad server configuration

Modify the Nomad server configuration file. Add the oidc_issuer attribute and set the value to the domain name for the Nomad cluster. This enables the HTTP endpoint in Nomad that allows third parties to discover Nomad's OIDC configuration.

server {
  enabled = true
  [...]
  oidc_issuer = "https://<DOMAIN_NAME>"
  [...]
}

This configuration requires the following information:

<DOMAIN_NAME>: The domain name of your Nomad cluster without the protocol or port.

Restart the Nomad server agent to apply the configuration changes.

Run a sample Nomad job

Create and run the jobspec file

Create a jobspec file named s3-upload.nomad.hcl, add the following configuration to it, and save the file.

job "s3" {
  type = "batch"
  group "bucket" {
    task "copy" {
      driver = "docker"
      config {
        image = "public.ecr.aws/aws-cli/aws-cli"
        command = "s3"
        args = ["cp", "/local/test.txt", "s3://<S3_BUCKET_NAME>/test-nomad.txt"]
      }

      identity {
        name = "aws"
        aud = ["aws"] 
        file = true
        env = true
        ttl = "1h" 
        change_mode = "restart"
      }

      template {
        destination = "local/test.txt"
        change_mode = "restart"
        data        = <<EOF
Job:          {{ env "NOMAD_JOB_NAME" }}
Alloc:        {{ env "NOMAD_ALLOC_ID" }}
EOF
      }

      env {
        AWS_ROLE_ARN = "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>"
        # The format of the token file is nomad_$NAME_OF_IDENTITY.jwt
        AWS_WEB_IDENTITY_TOKEN_FILE = "${NOMAD_SECRETS_DIR}/nomad_aws.jwt"
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

This configuration requires the following information:

<S3_BUCKET_NAME>: The name of the S3 bucket where the test file will be saved.
<AWS_ACCOUNT_ID>: The ID of the AWS account where the IAM role from the previous step was created.
<IAM_ROLE_NAME>: The name of the IAM role from previous steps. This will be s3_all_access_role if you are using the examples from this page.

Submit the job to Nomad.

$ nomad job run s3-upload.nomad.hcl

Verify that the job completed successfully.

$ nomad job status s3

Verify that the file was also uploaded to the S3 bucket.