Enterprise-Grade Security, Powered by 100% OpenTelemetry

With the emergence of dynamic variables and the smart telemetry hub, it's now possible to build security solutions that rival expensive, bespoke products from elite security vendors—without the price tag or lock-in. OpenTelemetry (OTel) isn't just for monitoring anymore. It's becoming a foundation for real-time, programmable security across your organization.

In this post, we'll explore how dynamic variables in a smart telemetry hub can turn your observability pipeline into a flexible, enterprise-scale security platform. We'll show how OTel processors can redact, filter, and transform sensitive business data on the fly—helping you enforce privacy policies without constantly changing source code. We'll also look at how a well-structured OTel deployment empowers SecOps teams to manage compliance and data governance independently, via clean interfaces that support self-service.

From Passive Telemetry to Active Data Governance

Picture this: Your enterprise has adopted a smart telemetry hub. Telemetry flows in from across the org and routes cleanly to multiple destinations. All seems well—until your SecOps team reaches out. They've just completed an audit and discovered that sensitive data—credit card numbers, social security info, emails—is making its way into a third-party observability vendor.

But you've deployed a programmable hub, not a dumb router. So, you act. You start authoring OTel processors to scrub personally identifiable information (PII) from your telemetry streams. It works. The sensitive data never leaves your environment. You've just turned observability into a privacy enforcement layer.

Now the rest of the organization takes notice. Every engineering team wants to tweak filters for their specific application logic, use cases, and local regulations. Suddenly, one central question emerges:

How do you give individual teams control over local PII scrubbing—while giving SecOps confidence that global standards are enforced? And do you need a separate OTel collector for every team to achieve this.

That's the challenge we'll explore next.

The Smart Telemetry Hub: A Shared Platform for Security at Scale in Four Steps

Assume it is reasonable to start addressing your security challenges by redacting sensitive customer information before it hits a SaaS observability backend. Then, use the smart telemetry hub acts as a centralized control plane—with scoped delegation—enabling both flexibility and enforcement. Teams get fine-grained control over their processing logic, while SecOps retains the ability to audit and enforce compliance across the enterprise.

And then widen the lens showing how this architecture brings security, DevOps, and data teams into true collaboration. By shifting from static dashboards and raw exhaust to active, intelligent telemetry streams, OTel becomes more than data pipelines—it becomes a platform.

Step One: Using dynamic OTel to deliver a PII solution that scales to multiple apps all with complex rules

Here is the redactionProcessor from OpenTelemetry contrib: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/redactionprocessor/README.md

Many people use it to maintain compliance with respect to their observability solution / vendor of choice. Out of the box, the component offers you the ability to scrub for patterns of known PII data lurking in your telemetry. This happens more than you would think, the lurking PII data that is. I have yet to find an enterprise that is not leaking PII data to their observability vendor.

The question is, what can dynamic OTel do that the static component cannot already do? There are multiple value propositions for dynamic OTel, but we will cover only two now.

The use cases:

• PII rules that differ across teams. Rarely can we set up security pipelines and forget about them. Most of us need to keep up with changing business rules as well as differences in code bases. Dynamic OTel can help us here as well but we need to make the solution–and pipelines - easy to use for SecOps to help us keep our PII rules up to date.
• Filtering + enrichment for SIEM.

There is a third use case which is automatic detection and scrubbing of PII data. This is generally done with language-specific detectors. But how would you plug this into your telemetry pipeline directly and avoid paying a vendor like Datadog for a proprietary implementation that also runs after egress of the data out of your network: https://www.datadoghq.com/product/sensitive-data-scanner/

For now we won't go into the details but the detectors can be hosted inside the smart telemetry hub directly. This allows your telemetry hub to scrub PII data before it leaves your network. Not only is this faster and safer, it is cheaper than paying or relying on a legacy observability vendor to solve this for you.

PII rules that differ per team

Let's take a look at the OpenTelemetry redactionProcessor. One of the most critical things we want to control is whether or not the data in our telemetry stream can be considered safe. Here is the configuration stanza that controls this behavior:

processors:
 redaction:
  # allow_all_keys is a flag which when set to true, which can disables the
  # allowed_keys list. The list of blocked_values is applied regardless. If
  # you just want to block values, set this to true.
  allow_all_keys: false
  # allowed_keys is a list of span/log/datapoint attribute keys that are kept on the span/log/datapoint and
  # processed. The list is designed to fail closed. If allowed_keys is empty,
  # no attributes are allowed and all span attributes are removed. To
  # allow all keys, set allow_all_keys to true.
  allowed_keys:
   - description
   - group
   - id
   - name

If we were using vanilla OTel, we would consider setting allow_all_keys to true globally. With such a configuration, we leave it up to SecOps and dev teams to manually secure every field of every bit of telemetry our applications emit. This way no team has to learn and configure their OTel pipelines–their data will always pass through to observability vendors. But now we have added a burden where every team must contribute to the redaction configuration, meaning replacing sensitive information with "X's" or deleting sensitive fields altogether.

We might even decide to centrally locate the redaction processor and perhaps there will be collisions where team 1 and team 2 emit different data using the same field names and now, OTel-based redaction management starts to feel like source control and conflict resolution–we have to merge rules across teams every day. This is not good.

Surely we do not want the variable named allow_all_keys to be set globally to true or false. If this variable is set to true that means we either trust all our data to not contain PII or that we have handled each key manually with a rule that scrubs that key for us. (To avoid confusion, I like to think of the word "keys" in this context to mean fields or attributes in the data stream.)

The easiest solution would be to do this in the agent, not the gateway collector. But, imagine a medium-sized company with 200 services and maybe 2 or all 3 public cloud providers. Could we possibly write a manual rule for each key in each telemetry data type in each region in each cloud? We still need to give SecOps the ability to check every single architectural component for compliance on every release cycle and that's a lot of work.

It would seem that centrally managed PII data redaction is too cumbersome to coordinate and has side effects like cross-application configuration collisions whereas decentralized management solves that problem but leaves a lot of work for humans to audit and manage the PII posture across our enterprise.

Most SecOps teams would demand that we buy a PII scrubber from a dedicated vendor at this point. The combination of fields and regexes seems both daunting and dangerous to try to get right. The constant scanning of hundreds of local configs seems just as daunting.

So let's make these settings variables and let's push them centrally to the smart telemetry hub.

allow_all_keys: $myAppRedactorBoolean

  Allowed_keys:
    $myAppAllowedKeysList

Now you can set the redactor to allow certain keys that pertain to just your specific application or service.

We need one more trick which is to use data processors to label the data as to which set of rules apply.

Using the OpenTelemetry attributesProcessor: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/README.md

We add an attribute called mdai_resourceName and now we can use dynamic variables to control our redaction behavior.

In the smart telemetry hub, we can store a table of PII redaction configurations as follows:

Application / Resource Name	AllowedKeysList
Application_1	key1; key2; key3; key4; …
Application_2	key1; key2
Application_3	key1; key2; ...

Instead of having a static block of attribute names that need scrubbing, this dynamic version of the redactionProcessor will inspect the mdai_resourceName and conditionally load in the variable list of keys that need to be dropped before transmission.

Step Two: Going beyond dropping keys to scrubbing data

What about all these regexes from the example documentation for OpenTelemetry?

# blocked_key_patterns is a list of blocked span attribute key patterns. Span attributes
# matching the regexes on the list are masked.
blocked_key_patterns:
 - ".*token.*"
 - ".*api_key.*"
# blocked_values is a list of regular expressions for blocking values of
# allowed span attributes. Values that match are masked
blocked_values:
 - "4[0-9]{12}(?:[0-9]{3})?" ## Visa credit card number
 - "(5[1-5][0-9]{14})"    ## MasterCard number

Take the simple example at the end of the quoted text above. Visa and Mastercard regexes. What if our business decides one day to accept Discover card? Now we need to add another regex to the drop rules for blocked_values.

Blocked_values:
$myBlockedCreditCardRegexes

The solution is the same as before.

Step Three: Properly Obscuring Data

The OpenTelemetry redactionProcessor gives us access to more than the ability to drop keys that are dangerous or identify data via regex that might be lurking in unknown key-value pairs. The processor can take a field and obscure it. Since the field is destined for an observability platform eventually, it might be a good idea to make query tools and dashboards still work in that product. For example, let's say a site reliability engineer would like to be able to query the number of unique email addresses provided to your company per hour. Using a hash function allows for consistent obscurity for data so that query aggregation functions like count() and unique() still work.

# hash_function defines the function for hashing the values instead of
# masking them with a fixed string. By default, no hash function is used
# and masking with a fixed string is performed.
hash_function: md5

I am no security expert but I have done enough enterprise data projects to know that advertising to your entire dev team that you simply md5 hash all your data to obscure it is never going to pass compliance. You would need to salt (Wikipedia entry for reference) that data and rotate the SALT itself.

This requires the hash_function to take multiple arguments and those arguments need to change over time. But OpenTelemetry did not account for this. Do we now fork the redactionProcessor? Dynamic variable-capable OpenTelemetry solves this problem for you.

# hash_function defines the function for hashing the values instead of
# masking them with a fixed string. By default, no hash function is used
# and masking with a fixed string is performed.
hash_function: $myHashingFunction
Hash_salt: $myDailySALT

The SALT can be rotated daily so brute force attacks are no longer trivial. The function itself can be obscured from prying eyes which also makes it harder to attack the data. Now you have redaction that is open source and enterprise class.

CRITICALLY: your observability vendor is not involved.

While Datadog charges for PII scrubbing, you don't need them anymore. While New Relic offers column-level obfuscation based on loginID (some engineers see a column while some do not), you do not need that error-prone configuration. You can now just obscure the data you need to and be more confident that the data isn't in observability streams anymore.

Step Four: Storing and Working with Variables Outside OTel and K8S

Looking back to the Principled Architecture blog entry there was a reference to a "scalars and variables db." In the smart telemetry hub, this capability can be implemented with ValKey configured for persistence (along with a few custom modules that make ValKey more than storage for strings).

Let's look back at three of the challenges we have not yet addressed:

1. We needed teams' separate configuration requirements to not force us to deploy many different OTel configurations. This is addressed by making everything a dynamic variable and coupling the values we plug in at runtime to the mdai_resourceName field which acts like a tag for our data passing through the hub.
2. We needed a central place for configuring application PII scrubbing policies. ValKey acts as that central storage. Don't worry about the operational complexity or security of ValKey. The storage could be Postgres or DynamoDB or Vault or other options.
3. We needed a way for SecOps to be able to audit or edit our rules as rules, not as YAML configuration files. Because we have centrally located the data in a database / data store SecOps just checks that the variables are present in the OTel configuration we deployed and then they scan each team's configuration variables for current settings. SecOps no longer needs to read all the code to all the applications and services we are running in production.

In conclusion, the business logic is in source control. The OpenTelemetry pipelines are in a separate source control managed by pipeline and site reliability engineers. And the values that drive data redaction in our PII scrubbers are in a database and get plugged into the system at runtime. Those variables can change value regularly and can be looked up as part of pipeline execution at full production scale.

A New Era for OpenTelemetry: Dynamic, Secure, and Collaborative

At the core of this revolution is a mindset shift: telemetry isn't just something you collect—it's something you manage. Actively. Securely. Intelligently.

Smart telemetry hubs built on OpenTelemetry mark a new phase in observability—one where dynamic, real-time data processing becomes foundational. This is how we move from passive monitoring to true enterprise intelligence. Not by adding more dashboards, but by treating telemetry itself as a programmable substrate for security, governance, and innovation.

Tell us what you think. Join us on slack: MyDecisive community slack

Ari