Adding API Integrations to Helix Services

This guide covers how to add external API integrations to Helix services, including library installation, configuration, and secret management across environments.

Overview

When adding a new API integration (e.g., Stripe, SendGrid, Twilio, etc.), you need to:

Install the npm library in the specific service
Configure API keys for local development
Store secrets in AWS Secrets Manager
Update Terraform to inject secrets
Update GitHub Actions workflows
Test in all environments

Step-by-Step Guide

Step 1: Install the npm Library

Install the library in the specific service that needs it, not in the root package.json.

# Navigate to the service directory
cd apps/your-service

# Install the library
npm install <library-name>

# Example: Adding Stripe to kira-service
cd apps/kira-service
npm install stripe

# Return to root and update lock file
cd ../..
npm install

Important: Each service has its own package.json. Install dependencies in the service that uses them.

Step 2: Add Configuration Interface

Create a config interface for the new integration.

Location: apps/your-service/src/config/

// apps/kira-service/src/config/stripe.config.ts
export interface StripeConfig {
  apiKey: string;
  webhookSecret?: string;
  apiVersion?: string;
}

export const getStripeConfig = (): StripeConfig => {
  const apiKey = process.env.STRIPE_API_KEY;
  
  if (!apiKey) {
    throw new Error('STRIPE_API_KEY is required');
  }

  return {
    apiKey,
    webhookSecret: process.env.STRIPE_WEBHOOK_SECRET,
    apiVersion: '2023-10-16',
  };
};

Step 3: Create Service Wrapper

Create a service to encapsulate the API client.

// apps/kira-service/src/integrations/stripe.service.ts
import { Injectable, OnModuleInit } from '@nestjs/common';
import Stripe from 'stripe';
import { getStripeConfig } from '../config/stripe.config';

@Injectable()
export class StripeService implements OnModuleInit {
  private stripe: Stripe;

  onModuleInit() {
    const config = getStripeConfig();
    this.stripe = new Stripe(config.apiKey, {
      apiVersion: config.apiVersion as any,
    });
  }

  async createCustomer(email: string, name: string): Promise<Stripe.Customer> {
    return this.stripe.customers.create({
      email,
      name,
    });
  }

  // Add more methods as needed
}

Step 4: Configure for Local Development

4.1 Update `.env.example`

Add the new environment variable to the service's .env.example:

# apps/kira-service/.env.example

# Existing vars...
OPENAI_API_KEY=sk-...

# New integration
STRIPE_API_KEY=sk_test_...
STRIPE_WEBHOOK_SECRET=whsec_...

4.2 Update `.env` (Local Development)

Create or update your local .env file:

# .env (root directory)

# Service-specific
STRIPE_API_KEY=sk_test_your_test_key_here
STRIPE_WEBHOOK_SECRET=whsec_your_webhook_secret

4.3 Update `docker-compose.yml`

Add the environment variable to the service in docker-compose.yml:

kira-service:
  build:
    context: .
    dockerfile: Dockerfile.template
    args:
      SERVICE_NAME: kira-service
  environment:
    NODE_ENV: development
    PORT: 3004
    # Existing vars
    OPENAI_API_KEY: ${OPENAI_API_KEY}
    # New integration
    STRIPE_API_KEY: ${STRIPE_API_KEY}
    STRIPE_WEBHOOK_SECRET: ${STRIPE_WEBHOOK_SECRET:-}
  ports:
    - '3004:3004'

Note: Use ${VAR_NAME:-} for optional variables (provides empty string as default).

Step 5: Add to AWS Secrets Manager

5.1 Update Terraform Secrets Module

Add the new secret to terraform/modules/secrets/main.tf:

# Stripe API Key
resource "aws_secretsmanager_secret" "stripe_api_key" {
  name        = "${var.environment}/helix/stripe-api-key"
  description = "Stripe API key for payment processing"

  tags = merge(var.tags, {
    Name    = "${var.environment}-helix-stripe-api-key"
    Service = "kira-service"
  })
}

resource "aws_secretsmanager_secret_version" "stripe_api_key" {
  secret_id     = aws_secretsmanager_secret.stripe_api_key.id
  secret_string = var.stripe_api_key
}

# Stripe Webhook Secret (optional)
resource "aws_secretsmanager_secret" "stripe_webhook_secret" {
  name        = "${var.environment}/helix/stripe-webhook-secret"
  description = "Stripe webhook secret for event verification"

  tags = merge(var.tags, {
    Name    = "${var.environment}-helix-stripe-webhook-secret"
    Service = "kira-service"
  })
}

resource "aws_secretsmanager_secret_version" "stripe_webhook_secret" {
  secret_id     = aws_secretsmanager_secret.stripe_webhook_secret.id
  secret_string = var.stripe_webhook_secret
}

5.2 Update Terraform Variables

Add to terraform/modules/secrets/variables.tf:

variable "stripe_api_key" {
  description = "Stripe API key for payment processing"
  type        = string
  sensitive   = true
}

variable "stripe_webhook_secret" {
  description = "Stripe webhook secret for event verification"
  type        = string
  sensitive   = true
  default     = ""
}

5.3 Update Terraform Outputs

Add to terraform/modules/secrets/outputs.tf:

output "stripe_api_key_secret_arn" {
  description = "ARN of Stripe API key secret"
  value       = aws_secretsmanager_secret.stripe_api_key.arn
}

output "stripe_webhook_secret_arn" {
  description = "ARN of Stripe webhook secret"
  value       = aws_secretsmanager_secret.stripe_webhook_secret.arn
}

# Update all_secret_arns map
output "all_secret_arns" {
  description = "Map of all secret ARNs"
  value = {
    # ... existing secrets
    stripe_api_key      = aws_secretsmanager_secret.stripe_api_key.arn
    stripe_webhook_secret = aws_secretsmanager_secret.stripe_webhook_secret.arn
  }
}

5.4 Wire Secrets in Main Terraform

Update terraform/main.tf to pass variables to secrets module:

module "secrets" {
  source = "./modules/secrets"
  
  # ... existing variables
  
  # New integration
  stripe_api_key        = var.stripe_api_key
  stripe_webhook_secret = var.stripe_webhook_secret
}

Add variables to root terraform/variables.tf:

variable "stripe_api_key" {
  description = "Stripe API key for payment processing"
  type        = string
  sensitive   = true
}

variable "stripe_webhook_secret" {
  description = "Stripe webhook secret"
  type        = string
  sensitive   = true
  default     = ""
}

5.5 Update ECS Task Definition

Update terraform/modules/ecs-cluster/main.tf to inject the secret:

resource "aws_ecs_task_definition" "services" {
  for_each = local.services

  # ... existing configuration

  container_definitions = jsonencode([
    {
      name  = each.key
      image = "${var.ecr_repository_urls[each.key]}:${var.image_tag}"
      
      secrets = concat(
        # Common secrets...
        
        # Service-specific secrets
        each.key == "kira-service" ? [
          {
            name      = "OPENAI_API_KEY"
            valueFrom = var.openai_api_key_secret_arn
          },
          {
            name      = "STRIPE_API_KEY"  # NEW
            valueFrom = var.stripe_api_key_secret_arn
          },
          {
            name      = "STRIPE_WEBHOOK_SECRET"  # NEW
            valueFrom = var.stripe_webhook_secret_arn
          }
        ] : [],
        # ... other services
      )
    }
  ])
}

Add variables to terraform/modules/ecs-cluster/variables.tf:

variable "stripe_api_key_secret_arn" {
  description = "ARN of Stripe API key secret"
  type        = string
  default     = ""
}

variable "stripe_webhook_secret_arn" {
  description = "ARN of Stripe webhook secret"
  type        = string
  default     = ""
}

Update terraform/main.tf to pass ARNs to ECS module:

module "ecs_cluster" {
  source = "./modules/ecs-cluster"
  
  # ... existing configuration
  
  # New secret ARNs
  stripe_api_key_secret_arn    = module.secrets.stripe_api_key_secret_arn
  stripe_webhook_secret_arn    = module.secrets.stripe_webhook_secret_arn
}

Step 6: Configure GitHub Secrets

6.1 Add Staging Secrets

Go to: Repository → Settings → Secrets and variables → Actions
Add new secrets:
- STAGING_STRIPE_API_KEY = sk_test_...
- STAGING_STRIPE_WEBHOOK_SECRET = whsec_test_...

6.2 Add Production Secrets

Same as staging but with production values:

PROD_STRIPE_API_KEY = sk_live_...
PROD_STRIPE_WEBHOOK_SECRET = whsec_live_...

Step 7: Update GitHub Actions Workflows

Update .github/workflows/deploy-staging.yml:

- name: Terraform Plan
  run: |
    terraform plan \
      # ... existing vars
      -var="stripe_api_key=${{ secrets.STAGING_STRIPE_API_KEY }}" \
      -var="stripe_webhook_secret=${{ secrets.STAGING_STRIPE_WEBHOOK_SECRET }}" \
      -out=tfplan

Update .github/workflows/deploy-production.yml:

- name: Terraform Plan
  run: |
    terraform plan \
      # ... existing vars
      -var="stripe_api_key=${{ secrets.PROD_STRIPE_API_KEY }}" \
      -var="stripe_webhook_secret=${{ secrets.PROD_STRIPE_WEBHOOK_SECRET }}" \
      -out=tfplan

Step 8: Test the Integration

Test Locally

# Set environment variables
export STRIPE_API_KEY=sk_test_your_key

# Start the service
make dev

# Or specific service
make run-service SERVICE=kira-service

Test in Docker

# Build and run with docker-compose
docker-compose up kira-service

# Check logs
docker-compose logs -f kira-service

Test in Staging

# Deploy to staging
git checkout develop
git push origin develop

# Monitor deployment in GitHub Actions
# Check CloudWatch logs after deployment

Common Patterns

Pattern 1: API Client with Retry Logic

import { retry } from '@helix/shared/utils';

@Injectable()
export class ExternalApiService {
  async callWithRetry<T>(fn: () => Promise<T>): Promise<T> {
    return retry(fn, 3, 1000); // 3 retries, 1s delay
  }

  async getData(): Promise<any> {
    return this.callWithRetry(() => 
      this.client.get('/endpoint')
    );
  }
}

Pattern 2: Circuit Breaker for External APIs

import { CircuitBreaker } from '@helix/shared/utils';

@Injectable()
export class ExternalApiService {
  private circuitBreaker: CircuitBreaker;

  constructor() {
    this.circuitBreaker = new CircuitBreaker({
      failureThreshold: 5,
      timeout: 30000,
      resetTimeout: 60000,
    });
  }

  async callApi(): Promise<any> {
    return this.circuitBreaker.execute(() =>
      this.client.get('/endpoint')
    );
  }
}

Pattern 3: Caching API Responses

@Injectable()
export class ExternalApiService {
  constructor(
    private readonly redis: RedisService,
  ) {}

  async getCachedData(key: string): Promise<any> {
    // Check cache first
    const cached = await this.redis.get(`api:${key}`);
    if (cached) {
      return JSON.parse(cached);
    }

    // Fetch from API
    const data = await this.client.get(`/endpoint/${key}`);

    // Cache for 5 minutes
    await this.redis.setex(`api:${key}`, 300, JSON.stringify(data));

    return data;
  }
}

Checklist

Before merging your API integration, verify:

Troubleshooting

Issue: Secret not available in container

Check:

# Verify secret exists in AWS
aws secretsmanager get-secret-value --secret-id staging/helix/stripe-api-key

# Check ECS task definition
aws ecs describe-task-definition --task-definition staging-helix-kira-service

# Check container logs
aws logs tail /ecs/staging/helix/kira-service --follow

Issue: Terraform can't find variable

Solution: Ensure variable is defined at all levels:

Root variables.tf
Module variables.tf
Passed to module in main.tf
Used in GitHub Actions workflow

Issue: API calls failing

Debug:

// Add detailed logging
@Injectable()
export class ExternalApiService {
  async callApi(): Promise<any> {
    try {
      const result = await this.client.get('/endpoint');
      console.log('API call succeeded:', result);
      return result;
    } catch (error) {
      console.error('API call failed:', {
        message: error.message,
        status: error.response?.status,
        data: error.response?.data,
      });
      throw error;
    }
  }
}

Example: Complete Integration (Stripe)

See the complete example in apps/kira-service/src/integrations/stripe/ which demonstrates:

Configuration management
Service wrapper with error handling
Webhook verification
Retry logic
Circuit breaker
Comprehensive tests

Need Help?

Terraform issues: See docs/deployment/terraform-overview.md
Secret management: See docs/deployment/deployment-guide.md#secrets-management
Service architecture: See docs/services/
Docker issues: See docs/architecture/docker-setup.md

Overview​

Step-by-Step Guide​

Step 1: Install the npm Library​

Step 2: Add Configuration Interface​

Step 3: Create Service Wrapper​

Step 4: Configure for Local Development​

4.1 Update .env.example​

4.2 Update .env (Local Development)​

4.3 Update docker-compose.yml​

Step 5: Add to AWS Secrets Manager​

5.1 Update Terraform Secrets Module​

5.2 Update Terraform Variables​

5.3 Update Terraform Outputs​

5.4 Wire Secrets in Main Terraform​

5.5 Update ECS Task Definition​

Step 6: Configure GitHub Secrets​

6.1 Add Staging Secrets​

6.2 Add Production Secrets​

Step 7: Update GitHub Actions Workflows​

Step 8: Test the Integration​

Test Locally​

Test in Docker​

Test in Staging​

Common Patterns​

Pattern 1: API Client with Retry Logic​

Pattern 2: Circuit Breaker for External APIs​

Pattern 3: Caching API Responses​

Checklist​

Troubleshooting​

Issue: Secret not available in container​

Issue: Terraform can't find variable​

Issue: API calls failing​

Example: Complete Integration (Stripe)​

Need Help?​