structured logging: Add a list of fields to Metadata

smalis-msft commented

2025-06-04 13:07:26 -05:00

(Migrated from github.com)

It would be handy to have a list of fields, and some basic metadata about them (name, formatting options, etc) available in Metadata when structured logging is being used.

My desired use case here is something that is currently possible with tracing: In OpenVMM we have implemented a 'tagging' system for events, where by including a field with a known name and a value of tracing::Empty we can filter out the logging of events based on the presence or absence of these tags. See https://github.com/microsoft/openvmm/blob/main/vm/cvm_tracing/src/lib.rs if you're curious as to how this all works. And since tracing::Empty doesn't generate any output these tags are completely transparent to the final produced logs. I'm hoping Value::null() could be used for the same in log?

It would be handy to have a list of fields, and some basic metadata about them (name, formatting options, etc) available in Metadata when structured logging is being used. My desired use case here is something that is currently possible with tracing: In [OpenVMM](https://github.com/microsoft/openvmm) we have implemented a 'tagging' system for events, where by including a field with a known name and a value of tracing::Empty we can filter out the logging of events based on the presence or absence of these tags. See https://github.com/microsoft/openvmm/blob/main/vm/cvm_tracing/src/lib.rs if you're curious as to how this all works. And since tracing::Empty doesn't generate any output these tags are completely transparent to the final produced logs. I'm hoping Value::null() could be used for the same in log?

smalis-msft commented

2025-06-04 13:48:58 -05:00

(Migrated from github.com)

I guess could be done by visiting the whole record instead of relying on Metadata and enabled? It'd be nice if this could all be handled in enabled though.

KodrAus commented

2025-06-04 16:07:09 -05:00

(Migrated from github.com)

Hi @smalis-msft 👋

log's metadata is simpler than tracing's. It doesn't include a list of key-values that will be present on the resulting log record. The closest feature it has to what you're currently doing is the target, which can be set to an arbitrary string, but is usually the name of the module that generated the event.

log also doesn't have an equivalent of tracing::Empty. If a key is present then it's defined and can be serialized, even if its value is null. If a key is not present then it's undefined.

You could probably implement this as a wrapper over log, by defining macros that accepted your tags, then filtered based on those, then forwarded matching records through to the underlying logger.

If we were to implement something like this in log itself, it might look like this:

Add an internal &'a [&'a str] list to Metadata to describe the keys that appear in the macro, and a Metadata::has_key(&self, &str) -> bool method to query whether a key is present.
Add a :meta annotation to the log macros for key-values that should be present in the metadata, but not the resulting log record.

Hi @smalis-msft :wave: `log`'s metadata is simpler than `tracing`'s. It doesn't include a list of key-values that will be present on the resulting log record. The closest feature it has to what you're currently doing is the target, which can be set to an arbitrary string, but is usually the name of the module that generated the event. `log` also doesn't have an equivalent of `tracing::Empty`. If a key is present then it's defined and can be serialized, even if its value is `null`. If a key is not present then it's undefined. You could probably implement this as a wrapper over `log`, by defining macros that accepted your tags, then filtered based on those, then forwarded matching records through to the underlying logger. If we were to implement something like this in `log` itself, it might look like this: 1. Add an internal `&'a [&'a str]` list to `Metadata` to describe the keys that appear in the macro, and a `Metadata::has_key(&self, &str) -> bool` method to query whether a key is present. 2. Add a `:meta` annotation to the `log` macros for key-values that should be present in the metadata, but not the resulting log record.

👍 1 ❤️ 1

smalis-msft commented

2025-06-04 16:25:10 -05:00

(Migrated from github.com)

1 is exactly what I had in mind, and might be a good addition in general. 2 is an interesting thought, but I think we could also just skip certain fields in our particular implementation of Log to accomplish the same thing without any new crate functionality.

👍 1

smalis-msft commented

2025-06-05 09:48:09 -05:00

(Migrated from github.com)

Or perhaps 1 should include a new KeyMetadata struct that can include more than just the name? Either way works.

KodrAus commented

2025-06-14 13:09:53 -05:00

(Migrated from github.com)

We try to keep the API surface area here in log small, so as a start I think we could commit to Metadata::has_key using data populated by the macros, and consider expanding that over time. I don't think we should expose a public way to set the keys on MetadataBuilder yet, until we decide on what the metadata API for key values might look like.

We try to keep the API surface area here in `log` small, so as a start I think we could commit to `Metadata::has_key` using data populated by the macros, and consider expanding that over time. I don't think we should expose a public way to set the keys on `MetadataBuilder` yet, until we decide on what the metadata API for key values might look like.

👍 1

Rows
Columns

structured logging: Add a list of fields to Metadata #685