Deployment Notes

This page summarizes the operational pieces used by the API features that the Python client calls.

Clustering Compute

The /search/cluster endpoint and agent clustering currently run through the Lambda compute backend in production deployments.

Configure the API with:

EIGENLAKE_CLUSTERING_BACKEND=lambda
EIGENLAKE_CLUSTERING_LAMBDA_NAME=eigenlake-clustering

The SDK call does not need to know where compute runs:

clusters = idx.search.cluster(
    filter={"status": {"$in": ["failed", "failure", "error"]}},
    algorithm="dbscan",
    dbscan_min_samples=4,
)

The response includes backend so callers can see that the work ran on Lambda.

Future compute backends may include GPUs, Spark, and automatic compute selection. In that model EigenLake will choose the compute target based on request size, latency needs, and resource requirements.

Lambda Isolation

The clustering Lambda runs the same k-means, DBSCAN, and hyperparameter tuning implementation as the API, but outside the FastAPI process. The API invokes Lambda synchronously; AWS starts or reuses an execution environment automatically and freezes or retires it after the invoke.

Deploy the Lambda from the API repository:

bash lambda/eigenlake_cluster/scripts/deploy_lambda.sh \
  --function-name eigenlake-clustering \
  --role-arn arn:aws:iam::<account-id>:role/<lambda-execution-role> \
  --region us-east-1

Minimum Lambda environment:

EIGENLAKE_DATABASE_URL=postgresql+psycopg://...
EIGENLAKE_AWS_REGION=us-east-1

Prefer IAM roles for AWS credentials. If your database is private, attach the Lambda to subnets and security groups that can reach it.

The API role also needs permission to invoke the function:

{
  "Effect": "Allow",
  "Action": "lambda:InvokeFunction",
  "Resource": "arn:aws:lambda:<region>:<account-id>:function:eigenlake-clustering"
}

EC2 Deployment

The API repository includes a GitHub Actions deployment workflow that can deploy to an EC2 instance over SSH. A typical instance needs:

sudo apt update
sudo apt install -y git python3 python3-venv curl
sudo mkdir -p /opt/eigenlake/eigenlake-api
sudo chown -R ubuntu:ubuntu /opt/eigenlake

Store deployment values as GitHub Actions secrets, including the EC2 host, SSH key, target directory, service name, and runtime environment values. The workflow pulls the latest code, installs dependencies, and restarts the API service.

Docs Deployment

The Python client docs are built from packages/eigenlake/docs with MkDocs. The docs publish workflow builds the site and pushes the generated output to:

EigenLake-Org/eigenlake-docs

GitHub Pages serves the public docs at:

https://docs.eigenlake.dev/

The DNS record should be:

docs.eigenlake.dev CNAME eigenlake-org.github.io

Keep the Cloudflare proxy status as DNS only for GitHub Pages custom-domain certificate provisioning.