Add Message Attestation for ServerToAgent messages

[liustanley]

Resolves #265, see proposal document for context.

Summary

This PR introduces an optional, end-to-end integrity mechanism for ServerToAgent messages based on X.509 certificate chains. When both the Server and the Agent opt in, every ServerToAgent message sent after the initial handshake carries a signature that the Agent verifies against a pre-configured trust anchor (a CA certificate) that is distinct from the TLS certificate authority used to establish the transport.

The mechanism allows OpAMP deployments to separate the distribution server from the authoritative source of OpAMP messages. A compromised distribution server cannot, in this model, push arbitrary configuration or commands to an Agent, because every message must be signed by a key whose trust chain validates against the Agent's pre-configured root.

Strict opt-in: implementations that do not implement Message Attestation require no changes. Existing OpAMP deployments are unaffected until both sides opt in.

Changes

Wire-level changes (all Status: [Development]):

• AgentCapabilities.RequiresPayloadTrustVerification = 0x00010000
• ServerCapabilities.OffersPayloadTrustVerification = 0x00000080
• ServerToAgent.trust_chain_response = 12 (new TrustChainResponse message)
• ServerToAgent.signature = 13

The new "Message Attestation" section in specification.md covers the threat model, trust model, opt-in semantics, capability negotiation, connection-time handshake, in-session per-message verification, algorithm selection (any X.509-supported), certificate requirements, failure modes (all result in agent disconnect), and out-of-scope items.

Client-facing changes

To enable verification of messages, the OpAMP client is configured with a root certificate to trust at startup via a config value:

extensions:
  opamp:
    server:
      payload_ca: /etc/opamp/securecorp-ca.pem
      ws:
        endpoint: wss://127.0.0.1:4320/v1/opamp

When provided, the client will indicate to the server that it requires authenticated payloads, and use the provided root certificate to verify messages the server sends.

Motivation

The OpAMP Server is currently treated as both a point of distribution and the authoritative source of OpAMP messages. If an OpAMP Server is compromised, clients can be subject to significant security vulnerabilities such as forced restarts, breaches of sensitive information, and unintended binary updates. By requiring messages to be signed by an authoritative source, an attacker gaining access to the OpAMP Server would not be able to sign messages that do not meet strict validation/type requirements. OpAMP Agents will also have the capability to reject messages with invalid certificates, and can immediately terminate the connection, preventing further attacks.

[linux-foundation-easycla]

• ✅ login: liustanley / name: Stanley Liu (65b74b8)
• ✅ login: truthbk / name: Jaime Fullaondo (1f75ca1, 65b74b8)
• ❌ The email address for the commit (1f75ca1, 65b74b8) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

[michel-laterman]

Suggested change

	pre-configured trust anchor (a CA certificate) that is **distinct** from
	the TLS certificate authority used to establish the transport.
	pre-configured trust anchor (a CA certificate) that MUST differ from
	the TLS certificate authority used to establish the transport.

you use the MUST wording later on

[domodwyer]

I'm interested in drives this requirement for separate CAs? It would require a customer that deploys message attestation to maintain / utilise two distinct CAs which is a bit of a burden.

If it is to split the security domains between the TLS + attestation, then I think we get that already using the Key Usage constraints on the certificates themselves:

• A server using TLS requires the serverAuth key usage bit to be set.
• This spec requires that the codeSigning bit be set.

So this lets us have two distinct chain "roles" that can chain to the same root, but are isolated from each other. You could go as far as to have the codeSigning intermediate CA stored separately from the serverAuth intermediate, physically isolating them too.

[michel-laterman]

What's the guidance if the trust root is set to expire?
Is some external process needed in order to update?

[truthbk]

If a root is expired the assumption is there will be some manual or out-of-band redistribution of the new trust root certs to the clients. Baking in rotation of the root cert into the protocol would allow for attackers to hijack the entire attestation process.

[michel-laterman]

When an agent terminates, should it retry?

Do we want to use a specific error_message attribute instead of the ServerToAgent.error_response?

[truthbk]

IHMO retries (with an exponential backoff to prevent hurting the OpAMP servers) are OK; it would allow the clients to self-recover on some class of attacks, for example DNS-based, without having to do anything specific on the client-side (in our example, once the DNS issue is resolved).

I personally don't see a need to have a dedicated message attribute, but I don't have a strong opinion on my side.

[tigrannajaryan]

I am blocking this for now to prevent accidental merging.

This needs a thorough review.

I am also not sure we want this now, it is not aligned with anything we have on the roadmap: #321

If we make the call we want this in the spec, this is going to require a prototype demonstrating it (not needed for now) and we will need to decide when we want this to be added (possibly after 1.0).

[domodwyer]

Disclaimer: I provided technical help for the proposal.

I agree - I think getting a working prototype nailed down would really help shake out any issues 👍

I understand the roadmap to V1 is already being worked on so the timing isn't great, but I do think there's value in at least investigating what a more secure protocol might look like.

The primary concern with OpAMP as it proposed for V1, is that it's a security single-point-of-failure. If you compromise the distribution server (which necessarily sits in a risky network position, reachable by many machines) then it's game over: the attacker has full control of the fleet.

This is especially bad if you're a 3rd party provider like Datadog (my employer) as a single nginx exploit or similar could yield control of many thousands of machines across many distinct organisations. This kind of exposure becomes a significant business risk and isn't something that's easy to accept - it unfortunately fundamentally restricts our adoption of the open protocol we're otherwise big fans of!

We're definitely motivated to help develop any solutions 🙏

[domodwyer]

I'm interested in drives this requirement for separate CAs? It would require a customer that deploys message attestation to maintain / utilise two distinct CAs which is a bit of a burden.

If it is to split the security domains between the TLS + attestation, then I think we get that already using the Key Usage constraints on the certificates themselves:

• A server using TLS requires the serverAuth key usage bit to be set.
• This spec requires that the codeSigning bit be set.

So this lets us have two distinct chain "roles" that can chain to the same root, but are isolated from each other. You could go as far as to have the codeSigning intermediate CA stored separately from the serverAuth intermediate, physically isolating them too.

[domodwyer]

So I think there's two other properties that this proposal doesn't currently address that we might want to consider - they're both part of a class of attacks we expect for our internal protocol to mitigate:

• Message replay: if an attacker obtains a valid and signed message, they can reuse that message and send it to the same agent again, or a different agent that was not the intended recipient.
• Rollback attack: related to the above, a previously captured message can be sent to revert any subsequent changes (like "roll back a config change").

A classic replay attack in this context would be capturing a validly signed ServerToAgentCommand.type = Restart message for an Agent, and then resending it every time that agent connects, causing a DoS of the Agent using the attested message.

A more subtle attack would be to use the above replay trick to obtain a legitimate remote config message that compels an Agent to upload sensitive files to the server¹ or perform some similarly risky action, and then broadcast that command to all connected agents instead of just the intended recipient.

Notably an attacker can't change the content of these messages (that would invalidate the signature). They're constrained to only replaying messages previously sent by the server.

Fixes are relatively easy:

• To prevent cross-session replay attacks, you can make use of a session nonce ("number used once") that is randomly generated by the Agent, unique to the connection and is part of the signed ServerToAgent payloads sent to that Agent. Upon receiving a payload, the ID is checked to match. That prevents a message destined for Agent A from being replayed to Agent B.

• To prevent intra-session replay attacks: maintain a sequence number such that an old message cannot be resent later in the session to roll back the effects of a message (e.g. cannot "undo" a "change config" request) - much like your car probably does. There's a similar mechanism already in place for AgentToServer messages already, but for a different purpose.

Note: there is the instance_id field which goes a long way to preventing cross-Agent replay attacks, but the protocol allows the server to push an "instance ID change" request (ref) which could be used to switch the Agent to an instance ID that is part of a previously captured request. I'm not familiar enough with the implementation to know if this is a viable path, and if not, the existing instance ID may suffice as a session nonce, but it should be documented as a load-bearing part of the protocol security if used as such.

Footnotes

1. https://opentelemetry.io/docs/specs/opamp/#security-considerations ↩

[domodwyer]

Is this paragraph implying that the server MUST validate payloads are valid before they are sent to the client?

If so, I would consider skipping this - it's not going to help you in the case of an attack (the attacker can simply send their message directly, bypassing the check) but it imposes a cost in the happy path as every message must be validated (latency + CPU).

[domodwyer]

Another constraint: the certificate must have a dNSName or iPAddress SAN entry that matches the hostname (or IP address if IPs are used) of the opamp distribution server.

This will be required by any X509 verification library - it is what is used to ensure you're using the right certificate for the server you're connected to.

[domodwyer]

I think this presents an interesting future development if justified by usage - being able to safely delegate parts of operating the fleet to 3rd parties or higher-risk infra, while retaining control over risk factors is a nice feature.

[domodwyer]

I think this is probably going to be problematic - the protobuf encoding does not have a canonical representation. The docs specifically state that, even in "deterministic output" mode:

The serializer can generate different output for many reasons, including but not limited to the following variations:

• The protobuf schema changes in any way.
• The application being built changes in any way.
• The binary is built with different flags (eg. opt vs. debug).
• The protobuf library is updated.

Which isn't going to provide sufficient guarantees when deployed across different versions / implementations / platforms / etc.

In general signing a payload that also contains the signature is definitely tricky and kind of forces this "serialise -> sign -> reserialise" dance that then requires the serialisation be deterministic on all platforms.

In an ideal world, the signature would be part of the metadata sent alongside the signed payload, rather than within the payload itself. For the HTTP transport that is easy - the signature can be a header value. But the WebSocket transport don't provide a metadata side channel like like HTTP headers, and the protocol doesn't currently have any provision for metadata alongside the message payload.

Here's some possible solutions - they're all shifting the signature out of the payload (to make a "detached signature") but there might be other ways to do this that I've not thought about.

Negotiated Type Switch

After the feature flag exchange has resulted in negotiating the use of signed messages, switch to using a different type instead of ServerToAgent if and only if message attestation is enabled:

message SignedServerToAgent {
    bytes payload = 1; // Serialised ServerToAgent
    bytes signature = 2;
}

// ServerToAgent unchanged

Clients will need to have a new check early in the message processing path, but then processing continues as usual:

payload := conn.Read();

var message ServerToAgent
if usingSigning {
    message = verify_and_parse(payload);
} else {
    message = parse_server_to_agent(payload);
}

// Continues like normal

Where verify_and_parse() parses the SignedServerToAgent protobuf message, and validates the signature, and then returns the inner ServerToAgent. The rest of the message processing continues like normal, using the ServerToAgent as it does today.

Pros:

• Protocol is fully backwards compatible with existing clients.

Cons:

• Protocol is now not fully described by a single type for the server -> agent direction.

Explicit Protocol Metadata

Obviously the protocol is not yet v1, so technically a breaking change could be made to support attestation directly - either by modifying the protobuf:

// Redefine the top-level type:
message ServerToAgent {
    ServerToAgentPayload payload = 1;
//or  bytes payload = 1;

    // metadata here
    bytes signature = 2;
}

message ServerToAgentPayload {
    // Existing fields here
}

This adds user friction though and probably wouldn't be well received, but it is a "one time" cost prior to stability.

Alternatively, by changing the WebSocket header to explicitly indicate that a signature is present alongside the encoded protobuf payload:

┌────────────┬────────────────────────────────────────┬───────────────────┐
│header      │varint encoded unsigned 64 bit integer  │1-10 bytes         │
├────────────┼────────────────────────────────────────┼───────────────────┤
│data        │Encoded Protobuf message,               │0 or more bytes    │
│            │either AgentToServer or ServerToAgent   │                   │
├────────────┼────────────────────────────────────────┼───────────────────┤
│metadata    │serialised metadata                     │0 or more bytes    │
└────────────┴────────────────────────────────────────┴───────────────────┘

They're both equivalent except the former couples the metadata to protobuf.

Pro:

• Explicit "bag of metadata" separates concerns from "payload content" in the same way HTTP / gRPC does and can be reused.

Cons:

• Breaking change - probably a difficult option at this point.