Skip to content

Vector Topic Modeling

EigenLake discovers topics from existing document embeddings and grounds each topic in the original text stored in record properties.

Run topic modeling

result = idx.search.topics(
    filter={"status": {"$eq": "open"}},
    limit=10_000,
    min_topics=2,
    max_topics=20,
    text_fields=["subject", "description"],
    metadata_fields=["priority", "product"],
    top_terms=10,
    representatives_per_topic=3,
)

for topic in result["topics"]:
    print(topic["topic_id"], topic["count"], topic["label"])
    print([term["term"] for term in topic["terms"]])
    print(topic["text_coverage"])

Topic discovery uses deterministic spherical k-means with cosine similarity. When num_topics is omitted, EigenLake evaluates the requested range on a deterministic sample and selects the best sampled cosine silhouette score.

Each assignment contains uuid, topic_id, and cosine similarity. Similarity is not a probability.

Topic terms and text coverage

After grouping embeddings, EigenLake calculates class-based TF-IDF over the selected text_fields. This identifies words and phrases that distinguish each topic from the other discovered topics.

Records without text still participate in semantic grouping. The response reports text coverage globally and per topic. A topic with no usable text gets a stable Topic <id> fallback label and empty terms.

If text_fields is empty, the API infers descriptive string fields from the index schema. Specify the fields explicitly when stable labeling is important.

Optional LLM labels

result = idx.search.topics(
    text_fields=["subject", "description"],
    label_mode="llm",
)

LLM labeling is a presentation layer. Topic membership, centroids, assignments, and c-TF-IDF terms are calculated first and never changed by the LLM.

The API sends only top terms and up to three truncated representative texts per topic to the configured Codex app-server. keyword_label remains in the response for auditability. If labeling is unavailable or incomplete, label falls back to keyword_label and labeling.fallback_reason explains why.

The default is label_mode="keywords", which is deterministic and does not call an external model.

Metadata facets

metadata_fields adds per-topic categorical value counts:

result = idx.search.topics(
    text_fields=["message"],
    metadata_fields=["priority", "region"],
)

Metadata is explanatory only. It does not affect semantic topic assignments. Use filter when metadata should restrict the analyzed snapshot.

Workload limits

  • Maximum snapshot: 10,000 embeddings
  • Minimum: three valid, finite, non-zero embeddings
  • Distance: cosine only
  • One topic assignment per embedding
  • Default timeout: 180 seconds

Long documents that need multiple topic assignments should be chunked before indexing.