Duplication of mbedtls-framework

irwir commented

2025-10-19 07:08:10 -05:00

(Migrated from github.com)

The current directory structure requires two references and thus two copies of the framework when building mbedTLS library; the same for local git repositories.
Maybe "flat" repository structure could be considered, where each repository is directory at the same level?
Directories for mbedTLS:
[mbedtls]
[tf-psa-crypto]
[mbedtls-framework]

For TF-PSA-crypto:
[tf-psa-crypto]
[mbedtls-framework]

In this case framework might be reused for both mbedTLS and TF-PSA-crypto.

The current directory structure requires two references and thus two copies of the framework when building mbedTLS library; the same for local git repositories. Maybe "flat" repository structure could be considered, where each repository is directory at the same level? Directories for `mbedTLS`: [mbedtls] [tf-psa-crypto] [mbedtls-framework] For `TF-PSA-crypto`: [tf-psa-crypto] [mbedtls-framework] In this case framework might be reused for both `mbedTLS` and `TF-PSA-crypto`.

davidhorstmann-arm commented

2025-10-23 02:59:28 -05:00

(Migrated from github.com)

Hi, thanks for the suggestion!

I think there are some good reasons why we have 2 submodules in that way at the moment (although I appreciate that it is a bit annoying to use).

One problem is that the framework submodule is also shared with Mbed TLS 3.6, so that limits what we can do in terms of changing the directory structure (it's an LTS, so users are relying on it to be the same). We made a submodule /framework when we released 3.6 and then we were committed to having that same structure for the lifetime of 3.6.

However, when 3.6 reaches end-of-life (in March 2027) we will have more freedom. It's possible that we'll be able to do away with the framework entirely, since we can separate it into (things that are used by crypto) and (things that are used only by TLS/X.509).

@ronald-cron-arm made most of the design decisions around the way the framework works. Ronald, do you have anything to add to this?

Hi, thanks for the suggestion! I think there are some good reasons why we have 2 submodules in that way at the moment (although I appreciate that it is a bit annoying to use). One problem is that the framework submodule is also shared with Mbed TLS 3.6, so that limits what we can do in terms of changing the directory structure (it's an LTS, so users are relying on it to be the same). We made a submodule `/framework` when we released 3.6 and then we were committed to having that same structure for the lifetime of 3.6. However, when 3.6 reaches end-of-life (in March 2027) we will have more freedom. It's possible that we'll be able to do away with the framework entirely, since we can separate it into (things that are used by crypto) and (things that are used only by TLS/X.509). @ronald-cron-arm made most of the design decisions around the way the framework works. Ronald, do you have anything to add to this?

gilles-peskine-arm commented

2025-10-23 08:28:41 -05:00

(Migrated from github.com)

We have a hard constraint that TF-PSA-Crypto should work without involving Mbed TLS, so it needs direct access to the framework. We also don't want to require users to track down dependencies, so the framework goes under the project, not separately, and likewise tf-psa-crypto goes under mbedtls. As a consequence, the nesting mbedtls/tf-psa-crypto/framework must exist.

We could avoid the duplication in several ways, but it doesn't come for free.

For background, the goal of the framework repository is to contain files that are shared between the two projects, or shared between maintained branches of one projects, such as parts of build scripts, test scripts and test data. Both projects consume files from the framework. The framework has a single branch, so at any time, Mbed TLS and TF-PSA-Crypto might use the same framework commit, or Mbed TLS might need a more recent commit, or TF-PSA-Crypto might need a more recent commit — at least we won't have the two projects wanting separate branches of the framework.

We could decide that Mbed TLS only ever consumes the framework through TF-PSA-Crypto. But that would require that every time Mbed TLS needs a framework change, we have to update TF-PSA-Crypto's framework pointer, then Mbed TLS's tf-psa-crypto pointer. So far we haven't judged this acceptable.

We could decide that when TF-PSA-Crypto is consumed by Mbed TLS, crypto's framework is not checked out and not used. This should be doable because the framework should never become incompatible with either project. But it requires a lot of work in build scripts, to make the framework location configurable. It's also beneficial only if you don't tell git to do a recursive submodule update, which we can do, but might not be applicable when Mbed TLS itself is a submodule of some other project.

There's no solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins.

We have a hard constraint that TF-PSA-Crypto should work without involving Mbed TLS, so it needs direct access to the framework. We also don't want to require users to track down dependencies, so the framework goes under the project, not separately, and likewise `tf-psa-crypto` goes under `mbedtls`. As a consequence, the nesting `mbedtls/tf-psa-crypto/framework` must exist. We could avoid the duplication in several ways, but it doesn't come for free. For background, the goal of the framework repository is to contain files that are shared between the two projects, or shared between maintained branches of one projects, such as parts of build scripts, test scripts and test data. Both projects consume files from the framework. The framework has a single branch, so at any time, Mbed TLS and TF-PSA-Crypto might use the same framework commit, or Mbed TLS might need a more recent commit, or TF-PSA-Crypto might need a more recent commit — at least we won't have the two projects wanting separate branches of the framework. We could decide that Mbed TLS only ever consumes the framework through TF-PSA-Crypto. But that would require that every time Mbed TLS needs a framework change, we have to update TF-PSA-Crypto's `framework` pointer, then Mbed TLS's `tf-psa-crypto` pointer. So far we haven't judged this acceptable. We could decide that when TF-PSA-Crypto is consumed by Mbed TLS, crypto's framework is not checked out and not used. This should be doable because the framework should never become incompatible with either project. But it requires a lot of work in build scripts, to make the framework location configurable. It's also beneficial only if you don't tell git to do a recursive submodule update, which we can do, but might not be applicable when Mbed TLS itself is a submodule of some other project. There's no solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins.

👍 1

ronald-cron-arm commented

2025-10-23 09:15:48 -05:00

(Migrated from github.com)

There's solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins.

You mean "There's NO solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins" ?

> There's solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins. You mean "There's NO solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins" ?

irwir commented

2025-10-23 09:49:22 -05:00

(Migrated from github.com)

We have a hard constraint that TF-PSA-Crypto should work without involving Mbed TLS, so it needs direct access to the framework. We also don't want to require users to track down dependencies, so the framework goes under the project, not separately, and likewise tf-psa-crypto goes under mbedtls. As a consequence, the nesting mbedtls/tf-psa-crypto/framework must exist.

Right now in mbedtls repository: include, library and framework are at the same level, but framework is a link.
The idea is to move contents of the current root of two repositories into a subdirectory.

The root of mbedtls repository contains only mbedtls real directory, while tf-psa-crypto and framework are links.
The root of tf-psa-crypto repository contains only tf-psa-crypto real directory, and framework is a link.
Paths in the scripts have to be rearranged, but is this really impossible without nesting?

> We have a hard constraint that TF-PSA-Crypto should work without involving Mbed TLS, so it needs direct access to the framework. We also don't want to require users to track down dependencies, so the framework goes under the project, not separately, and likewise `tf-psa-crypto` goes under `mbedtls`. As a consequence, the nesting `mbedtls/tf-psa-crypto/framework` must exist. Right now in `mbedtls` repository: `include`, `library` and `framework` are at the same level, but `framework` is a link. The idea is to move contents of the current root of two repositories into a subdirectory. The root of `mbedtls` repository contains only `mbedtls` real directory, while `tf-psa-crypto` and `framework` are links. The root of `tf-psa-crypto` repository contains only `tf-psa-crypto` real directory, and `framework` is a link. Paths in the scripts have to be rearranged, but is this really impossible without nesting?

gilles-peskine-arm commented

2025-10-23 09:57:30 -05:00

(Migrated from github.com)

You mean "There's NO solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins" ?

Oops, yes.

The idea is to move contents of the current root of two repositories into a subdirectory.

That wouldn't help. You'd still have the Mbed TLS project with a framework subdirectory and a tf-psa-crypto subdirectory, and the tf-psa-crypto subdirectory would also have a framework subdirectory.

but framework is a link.

Having a link has its own problems (does Windows support symlinks well these days?), and doesn't solve anything. framework can't be a link in TF-PSA-Crypto because it has to work without Mbed TLS. If we decide that framework can be a link to tf-psa-crypto/framework in Mbed TLS, then we could just rewrite our scripts to use the path tf-psa-crypto/framework instead, and we wouldn't need a link, but we'd be in the situation that we don't want, where an Mbed TLS framework update requires a TF-PSA-Crypto framework update.

> You mean "There's NO solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins" ? Oops, yes. > The idea is to move contents of the current root of two repositories into a subdirectory. That wouldn't help. You'd still have the Mbed TLS project with a `framework` subdirectory and a `tf-psa-crypto` subdirectory, and the `tf-psa-crypto` subdirectory would also have a `framework` subdirectory. > but `framework` is a link. Having a link has its own problems (does Windows support symlinks well these days?), and doesn't solve anything. `framework` can't be a link in TF-PSA-Crypto because it has to work without Mbed TLS. If we decide that `framework` can be a link to `tf-psa-crypto/framework` in Mbed TLS, then we could just rewrite our scripts to use the path `tf-psa-crypto/framework` instead, and we wouldn't need a link, but we'd be in the situation that we don't want, where an Mbed TLS framework update requires a TF-PSA-Crypto framework update.

irwir commented

2025-10-23 10:15:10 -05:00

(Migrated from github.com)

Having a link has its own problems

That exactly what we see now; and to make things even more troublesome - nested links.
But these links are in github. Locally there would be no links, it is three ordinary directories (local repositories) at the same level - this is the goal of the suggestion.

> Having a link has its own problems That exactly what we see now; and to make things even more troublesome - nested links. But these links are in github. Locally there would be no links, it is three ordinary directories (local repositories) at the same level - this is the goal of the suggestion.

gilles-peskine-arm commented

2025-12-04 08:29:11 -06:00

(Migrated from github.com)

I should add that I personally think that having two copies of the framework is bad enough that we should pursue another solution (always use the outer framework, or always use the inner framework), despite the downsides of the other solutions.

During routine maintenance, it's very common that the two copies of the framework are out of synch. Every once in a while, this has a subtle effect that causes mysterious problems.

The latest such problem was https://github.com/Mbed-TLS/mbedtls/issues/10529 — generated files being inconsistent depending on exactly how they were generated, because Mbed TLS had a framework submodule that was a couple of weeks more recent than its tf-psa-crypto submodule and consequently more recent than tf-psa-crypto/framework.

Unfortunately, changing this is hard because:

We have a lot of scripts that use a specific path, that we would need to rewrite. And the longer we let the problem fester, the harder it becomes to fix it.
In order to change the strategy, we have to accept some downside, and so far we haven't managed to accept the downsides of either solution.

I should add that I personally think that having two copies of the framework is bad enough that we should pursue another solution (always use the outer framework, or always use the inner framework), despite the downsides of the other solutions. During routine maintenance, it's very common that the two copies of the framework are out of synch. Every once in a while, this has a subtle effect that causes mysterious problems. The latest such problem was https://github.com/Mbed-TLS/mbedtls/issues/10529 — generated files being inconsistent depending on exactly how they were generated, because Mbed TLS had a `framework` submodule that was a couple of weeks more recent than its `tf-psa-crypto` submodule and consequently more recent than `tf-psa-crypto/framework`. Unfortunately, changing this is hard because: 1. We have a lot of scripts that use a specific path, that we would need to rewrite. And the longer we let the problem fester, the harder it becomes to fix it. 2. In order to change the strategy, we have to accept some downside, and so far we haven't managed to accept the downsides of either solution.

irwir commented

2025-12-04 09:22:57 -06:00

(Migrated from github.com)

We have a lot of scripts that use a specific path, that we would need to rewrite. And the longer we let the problem fester, the harder it becomes to fix it.

Replacement of framework/... with ../framework/... would do in most cases if flat repository structure was accepted. In any case, this job should be much simpler than recent repository split.

> 1. We have a lot of scripts that use a specific path, that we would need to rewrite. And the longer we let the problem fester, the harder it becomes to fix it. Replacement of `framework/...` with `../framework/...` would do in most cases if flat repository structure was accepted. In any case, this job should be much simpler than recent repository split.

bjwtaylor commented

2025-12-05 02:24:07 -06:00

(Migrated from github.com)

Had an interesting problem related to this yesterday. The issue was that the version of the framework in mbedtls was correct, however tf-psa-crypto was wrong. From what I can tell if you build a tf-psa-crypto PR the CI updates the mbedtls framework version to match that of tf-psa-crypto, so this reverted the correct change to the mbedtls version and caused a failure that was initially not initially reproducible locally. I suspect there we other factors that contributed to my fork being in this state, however though it would be useful to add this in for further discussion.

Rows
Columns

Duplication of mbedtls-framework #227