Building a Data Filter using Composable Observability: Cutting Costs Without Cutting Corners

TL;DR?

Let's tackle a real challenge: observability costs are out of control. Rather than arbitrary data reduction, composability can optimize cost while preserving incident response capabilities. This represents a shift toward smarter, more cost-effective Observability.

You don't need to wait for a vendor to reinvent observability. You can build it yourself—today. Composable observability for data filtration means:

•  Filtering with budgets, not blunt-force rules
• Buffering data locally for fast replay during incidents
• Automating triggers with golden signals or AI
• Optimizing tradeoffs continuously

The Core Problem

Observability platforms typically charge based on ingestion volume. While reducing data lowers costs, indiscriminately cutting data compromises the primary function of observability: effective incident response.

I've been in situations where key telemetry was missing during a critical incident because of sampling. When the signal isn't there, you can't trace root-cause. That''s not savings—that's risk.

So the question becomes: How can we reduce costs without losing fidelity when it matters most?

Enter Composable Observability

The idea is simple: build your own observability pipeline. Think of it like modular architecture—composing the parts you need, where you need them.

Here's a high-level flow:

• Deploy a data filter to control vendor-bound data
• Store a local buffer of all data for a short time window.
• Deploy that buffer to replay data so you have the complete picture during an incident.
• Integrate AI optimization logic over time for better cost-performance balance.

Why Budget-Based Filtering Beats Static Drops

Dropping a fixed percentage of data (e.g., 80%) sounds simple, but it's blunt-force and often lacks context. Budget-based filtering allocates ingestion quotas to services, teams, or hosts based on operational priorities. Some services may need more headroom, others less. It's flexible, dynamic, and better aligned to how your systems actually behave.

Critical services receive higher fidelity while less-critical components operate with reduced telemetry.

The Buffer: Your Incident-Time Lifesaver

To avoid the tradeoff between filtering and visibility, we introduce a local buffer—a temporary storage layer (e.g., an S3 bucket or similar) that holds 100% of telemetry for 1-4 hours.

When something breaks, you can replay that data to your vendor. Most platforms handle delayed ingestion just fine. The key differentiator is the triggering mechanism.

Triggering the Flush, Automatically

You might choose to base your replay decision on application golden signals. Error rate increases, latency spikes, or other anomalies can trigger automated data flushing. This creates a self-monitoring pipeline responsive to actual operational conditions, in real time.

Smarter Pipelines, Not Just Cheaper Ones

You could call this whole approach a "smart telemetry hub." It's local, composable, and fully under your control. Vendors don't need to change anything. You build the pipeline before data leaves your environment.

And because you built it, you can evolve it—plug in ML to detect patterns, or let an AI agent adjust thresholds dynamically. It's observability with brains and a budget.

Why current Observability Falls Short

Let's be honest—Observability today is mostly about cloud-hosted dashboards, alerts, and logs. It's reactive, vendor-controlled, and expensive. Even the newer tools with better UIs and smarter alerts are still stuck in the same paradigm. Composable observability flips the script. It brings intelligence and control to the data pipeline itself—it is no longer just about how data is visualized. That's a massive leap. It's smarter, cheaper, and—most importantly—yours.

Thoughts? Join us on slack: MyDecisive community slack

Ari