Contrastive Failure Embeddings for Lakehouse Reliability Engineering

Abstract

This study proposes a contrastive embedding approach for reliability engineering in lakehouse environments. The objective is to learn a vector representation of failure episodes that captures operational similarity, enabling recurrence detection, nearest-neighbor retrieval, and similarity-based analysis of historical incidents. Failure records are constructed from normalized telemetry features reflecting execution behavior, resource usage, runtime outcomes, and contextual metadata. A neural encoder is trained using a supervised contrastive loss with temperature-controlled scaling, pulling same-class failure pairs together while pushing apart pairs from different classes to produce geometrically well-separated clusters suited to cosine-similarity retrieval. Because the encoder maps failures into a continuous space rather than assigning discrete class labels, new failure types can be accommodated without retraining: a novel failure occupies a previously unoccupied region of the space, preserving the open-set flexibility that production environments require. The method extends deterministic failure fingerprinting with a learned representation layer for cases where exact matching is insufficient, organizing failures by operational similarity rather than surface-level error attributes. Contrastive failure embeddings offer a practical mechanism for improving incident retrieval in lakehouse environments and a foundation for intelligent operational assistants.