Bootstrap Guide

Complete guide to bootstrapping AWS infrastructure for the NorthBuilt RAG System.

Overview

The bootstrap process sets up foundational AWS resources required before deploying the main Terraform infrastructure:

1. S3 Bucket - Stores Terraform state file
2. DynamoDB Table - Provides state locking
3. IAM OIDC Provider - Enables GitHub Actions authentication
4. IAM Role - Allows GitHub Actions to deploy infrastructure

Why Bootstrap is Needed

Chicken-and-Egg Problem:

• Terraform needs AWS credentials to run
• GitHub Actions needs IAM role to get credentials
• IAM role creation requires AWS credentials

Solution:

• Use personal AWS credentials (one time) to create OIDC provider + role
• GitHub Actions uses OIDC to assume role for all future deployments
• No long-lived AWS access keys needed

Prerequisites

• AWS CLI configured with admin credentials
• GitHub CLI (gh) installed and authenticated
• Terraform installed (>= 1.13.0)
• GitHub repository created

Quick Start

Option 1: Automated Script (Recommended)

# Run OIDC setup script
./.github/setup-oidc.sh

# Add GitHub secrets
gh secret set AWS_ROLE_ARN --body "arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):role/GitHubActionsOIDCRole"
gh secret set GOOGLE_CLIENT_ID --body "..."
gh secret set GOOGLE_CLIENT_SECRET --body "..."

# Deploy via GitHub Actions
git push origin main

Option 2: Manual Bootstrap

Follow the step-by-step guide below.

Step-by-Step Bootstrap

Step 1: Create S3 Bucket for Terraform State

# Set variables
export AWS_REGION="us-east-1"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export STATE_BUCKET="nb-rag-sys-terraform-state"

# Create S3 bucket
aws s3api create-bucket \
  --bucket $STATE_BUCKET \
  --region $AWS_REGION

# Enable versioning (for state recovery)
aws s3api put-bucket-versioning \
  --bucket $STATE_BUCKET \
  --versioning-configuration Status=Enabled

# Enable encryption
aws s3api put-bucket-encryption \
  --bucket $STATE_BUCKET \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }]
  }'

# Block public access
aws s3api put-public-access-block \
  --bucket $STATE_BUCKET \
  --public-access-block-configuration \
    BlockPublicAcls=true,\
IgnorePublicAcls=true,\
BlockPublicPolicy=true,\
RestrictPublicBuckets=true

# Add lifecycle policy (optional - keep last 10 versions)
aws s3api put-bucket-lifecycle-configuration \
  --bucket $STATE_BUCKET \
  --lifecycle-configuration '{
    "Rules": [{
      "Id": "DeleteOldVersions",
      "Status": "Enabled",
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 90
      }
    }]
  }'

echo "[OK] S3 bucket created: s3://$STATE_BUCKET"

Step 2: Create DynamoDB Table for State Locking

# Set variables
export LOCK_TABLE="nb-rag-sys-terraform-locks"

# Create DynamoDB table
aws dynamodb create-table \
  --table-name $LOCK_TABLE \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region $AWS_REGION

# Enable Point-in-Time Recovery
aws dynamodb update-continuous-backups \
  --table-name $LOCK_TABLE \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

# Wait for table to be active
aws dynamodb wait table-exists --table-name $LOCK_TABLE

echo "[OK] DynamoDB table created: $LOCK_TABLE"

Step 3: Create GitHub OIDC Provider

# Get GitHub OIDC provider thumbprint
THUMBPRINT=$(curl -s https://token.actions.githubusercontent.com/.well-known/openid-configuration \
  | jq -r '.jwks_uri' \
  | xargs curl -s \
  | openssl x509 -fingerprint -noout \
  | sed 's/://g' \
  | awk -F= '{print tolower($2)}')

# Create OIDC provider
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list $THUMBPRINT

echo "[OK] OIDC provider created"

Step 4: Create IAM Role for GitHub Actions

# Get GitHub repository details
export GITHUB_ORG=$(gh repo view --json owner --jq '.owner.login')
export GITHUB_REPO=$(gh repo view --json name --jq '.name')
export ROLE_NAME="GitHubActionsOIDCRole"

# Create trust policy
cat > /tmp/trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:${GITHUB_ORG}/${GITHUB_REPO}:*"
        }
      }
    }
  ]
}
EOF

# Create IAM role
aws iam create-role \
  --role-name $ROLE_NAME \
  --assume-role-policy-document file:///tmp/trust-policy.json \
  --description "Role for GitHub Actions to deploy Terraform"

# Create permissions policy
cat > /tmp/permissions-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::nb-rag-sys-*",
        "arn:aws:s3:::nb-rag-sys-*/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:*"
      ],
      "Resource": [
        "arn:aws:dynamodb:*:*:table/nb-rag-sys-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "lambda:*"
      ],
      "Resource": [
        "arn:aws:lambda:*:*:function:nb-rag-sys-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:*"
      ],
      "Resource": [
        "arn:aws:iam::*:role/nb-rag-sys-*",
        "arn:aws:iam::*:policy/nb-rag-sys-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "apigateway:*"
      ],
      "Resource": [
        "arn:aws:apigateway:*::/*"
      ]
    },
    {
      "Effect": "Allow",
        "Action": [
        "cognito-idp:*"
      ],
      "Resource": [
        "arn:aws:cognito-idp:*:*:userpool/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:*"
      ],
      "Resource": [
        "arn:aws:secretsmanager:*:*:secret:nb-rag-sys-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudfront:*"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:*",
        "logs:*"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:*"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "sts:GetCallerIdentity"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

# Attach policy to role
aws iam put-role-policy \
  --role-name $ROLE_NAME \
  --policy-name TerraformDeploymentPolicy \
  --policy-document file:///tmp/permissions-policy.json

# Get role ARN
ROLE_ARN=$(aws iam get-role --role-name $ROLE_NAME --query 'Role.Arn' --output text)

echo "[OK] IAM role created: $ROLE_ARN"

Step 5: Add GitHub Secrets

# Add AWS role ARN
gh secret set AWS_ROLE_ARN --body "$ROLE_ARN"

# Add Google OAuth credentials
echo "Enter Google Client ID:"
read GOOGLE_CLIENT_ID
gh secret set GOOGLE_CLIENT_ID --body "$GOOGLE_CLIENT_ID"

echo "Enter Google Client Secret:"
read -s GOOGLE_CLIENT_SECRET
gh secret set GOOGLE_CLIENT_SECRET --body "$GOOGLE_CLIENT_SECRET"

# Optional: Add integration API keys
echo "Enter Fathom API key (or press Enter to skip):"
read -s FATHOM_KEY
if [ -n "$FATHOM_KEY" ]; then
  gh secret set FATHOM_API_KEY --body "$FATHOM_KEY"
fi

echo "[OK] GitHub secrets configured"

Step 6: Verify Configuration

# List GitHub secrets
gh secret list

# Test AWS credentials via GitHub Actions
gh workflow run terraform-deploy.yml --ref main

# Watch workflow
gh run watch

Bootstrap Terraform Module (Alternative Approach)

Instead of bash scripts, you can create a separate Terraform module for bootstrap resources.

bootstrap/main.tf

terraform {
  required_version = ">= 1.13.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

  # No backend - stores state locally
}

provider "aws" {
  region = var.aws_region
}

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "project_prefix" {
  description = "Project prefix for resource naming"
  type        = string
  default     = "nb-rag-sys"
}

variable "github_org" {
  description = "GitHub organization name"
  type        = string
}

variable "github_repo" {
  description = "GitHub repository name"
  type        = string
}

# S3 bucket for Terraform state
resource "aws_s3_bucket" "terraform_state" {
  bucket = "${var.project_prefix}-terraform-state"

  tags = {
    Name      = "${var.project_prefix}-terraform-state"
    ManagedBy = "terraform-bootstrap"
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_lifecycle_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    id     = "delete-old-versions"
    status = "Enabled"

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }
}

# DynamoDB table for state locking
resource "aws_dynamodb_table" "terraform_locks" {
  name         = "${var.project_prefix}-terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }

  point_in_time_recovery {
    enabled = true
  }

  tags = {
    Name      = "${var.project_prefix}-terraform-locks"
    ManagedBy = "terraform-bootstrap"
  }
}

# GitHub OIDC provider
resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = [
    "sts.amazonaws.com"
  ]

  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1"  # GitHub OIDC thumbprint
  ]

  tags = {
    Name      = "github-actions-oidc"
    ManagedBy = "terraform-bootstrap"
  }
}

# IAM role for GitHub Actions
data "aws_caller_identity" "current" {}

resource "aws_iam_role" "github_actions" {
  name = "GitHubActionsOIDCRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = aws_iam_openid_connect_provider.github.arn
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
          }
          StringLike = {
            "token.actions.githubusercontent.com:sub" = "repo:${var.github_org}/${var.github_repo}:*"
          }
        }
      }
    ]
  })

  tags = {
    Name      = "github-actions-oidc-role"
    ManagedBy = "terraform-bootstrap"
  }
}

# IAM policy for GitHub Actions
resource "aws_iam_role_policy" "github_actions" {
  name = "TerraformDeploymentPolicy"
  role = aws_iam_role.github_actions.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:*"
        ]
        Resource = [
          "arn:aws:s3:::${var.project_prefix}-*",
          "arn:aws:s3:::${var.project_prefix}-*/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "dynamodb:*"
        ]
        Resource = [
          "arn:aws:dynamodb:*:*:table/${var.project_prefix}-*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "lambda:*"
        ]
        Resource = [
          "arn:aws:lambda:*:*:function:${var.project_prefix}-*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "iam:*"
        ]
        Resource = [
          "arn:aws:iam::*:role/${var.project_prefix}-*",
          "arn:aws:iam::*:policy/${var.project_prefix}-*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "apigateway:*"
        ]
        Resource = [
          "arn:aws:apigateway:*::/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "cognito-idp:*"
        ]
        Resource = [
          "arn:aws:cognito-idp:*:*:userpool/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "secretsmanager:*"
        ]
        Resource = [
          "arn:aws:secretsmanager:*:*:secret:${var.project_prefix}-*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "cloudfront:*",
          "cloudwatch:*",
          "logs:*",
          "bedrock:*",
          "sts:GetCallerIdentity"
        ]
        Resource = ["*"]
      }
    ]
  })
}

# Outputs
output "state_bucket_name" {
  description = "S3 bucket name for Terraform state"
  value       = aws_s3_bucket.terraform_state.id
}

output "lock_table_name" {
  description = "DynamoDB table name for state locking"
  value       = aws_dynamodb_table.terraform_locks.name
}

output "github_actions_role_arn" {
  description = "IAM role ARN for GitHub Actions"
  value       = aws_iam_role.github_actions.arn
}

output "next_steps" {
  description = "Next steps after bootstrap"
  value = <<EOT
Bootstrap complete! Next steps:

1. Add GitHub secrets:
   gh secret set AWS_ROLE_ARN --body "${aws_iam_role.github_actions.arn}"
   gh secret set GOOGLE_CLIENT_ID --body "..."
   gh secret set GOOGLE_CLIENT_SECRET --body "..."

2. Update terraform/backend.tf:
   bucket         = "${aws_s3_bucket.terraform_state.id}"
   dynamodb_table = "${aws_dynamodb_table.terraform_locks.name}"

3. Initialize main Terraform:
   cd terraform
   terraform init

4. Deploy via GitHub Actions:
   git push origin main
EOT
}

bootstrap/terraform.tfvars

aws_region     = "us-east-1"
project_prefix = "nb-rag-sys"
github_org     = "craftcodery"
github_repo    = "compass"

Using Bootstrap Module

# Navigate to bootstrap directory
cd bootstrap

# Initialize Terraform
terraform init

# Plan bootstrap resources
terraform plan

# Apply bootstrap resources
terraform apply

# View outputs
terraform output

# Add GitHub secrets (from output instructions)
gh secret set AWS_ROLE_ARN --body "$(terraform output -raw github_actions_role_arn)"

# Return to main directory
cd ..

# Initialize main Terraform with remote backend
cd terraform
terraform init

Troubleshooting

OIDC Provider Already Exists

# Delete existing provider
aws iam delete-open-id-connect-provider \
  --open-id-connect-provider-arn arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com

# Recreate
./.github/setup-oidc.sh

State Bucket Access Denied

# Check bucket policy
aws s3api get-bucket-policy --bucket nb-rag-sys-terraform-state

# Verify IAM role has S3 permissions
aws iam get-role-policy --role-name GitHubActionsOIDCRole --policy-name TerraformDeploymentPolicy

DynamoDB Lock Timeout

# List locks
aws dynamodb scan --table-name nb-rag-sys-terraform-locks

# Force unlock (get LockID from scan output)
terraform force-unlock LOCK_ID

GitHub Actions Authentication Failed

# Verify OIDC provider exists
aws iam list-open-id-connect-providers

# Verify role trust policy
aws iam get-role --role-name GitHubActionsOIDCRole --query 'Role.AssumeRolePolicyDocument'

# Check GitHub secret
gh secret list

Cleanup

To destroy bootstrap resources:

# Using bootstrap module
cd bootstrap
terraform destroy

# Or manually
aws s3 rb s3://nb-rag-sys-terraform-state --force
aws dynamodb delete-table --table-name nb-rag-sys-terraform-locks
aws iam delete-role-policy --role-name GitHubActionsOIDCRole --policy-name TerraformDeploymentPolicy
aws iam delete-role --role-name GitHubActionsOIDCRole
aws iam delete-open-id-connect-provider --open-id-connect-provider-arn arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com

---

Last updated: 2025-12-30