lwip submodule should point to a tag or branch tip in the remote repository

ali1234 commented

2022-03-10 11:40:12 -06:00

(Migrated from github.com)

Currently the lwip submodule points to an arbitrary commit c385f31076b27efb8ee37f00cb5568783a58f299. Most git servers, including GNU Savannah, only allow shallow clones of branch tips and tags. This means if you do a shallow recursive clone of pico-extras (for example, in a CI pipeline), you will get the following error:

Error: error: Server does not allow request for unadvertised object c385f31076b27efb8ee37f00cb5568783a58f299
Error: fatal: Fetched in submodule path 'lib/lwip', but it did not contain c385f31076b27efb8ee37f00cb5568783a58f299. Direct fetching of that commit failed.

User must instead do a full clone of the submodule, which might mean doing a full clone of pico-extras too. This is slow, wastes bandwidth and electricity, uses up extra Actions minutes, and hammers the GNU servers.

This can be fixed by either changing the submodule to point to an existing tag or branch (if branch, you have to keep on updating the submodule whenever the branch tip changes), or by forking the upstream repository and adding a tag to the commit you want to use, and then adjusting the submodule to point to your fork.

Currently the lwip submodule points to an arbitrary commit c385f31076b27efb8ee37f00cb5568783a58f299. Most git servers, including GNU Savannah, only allow shallow clones of branch tips and tags. This means if you do a shallow recursive clone of pico-extras (for example, in a CI pipeline), you will get the following error: Error: error: Server does not allow request for unadvertised object c385f31076b27efb8ee37f00cb5568783a58f299 Error: fatal: Fetched in submodule path 'lib/lwip', but it did not contain c385f31076b27efb8ee37f00cb5568783a58f299. Direct fetching of that commit failed. User must instead do a full clone of the submodule, which might mean doing a full clone of pico-extras too. This is slow, wastes bandwidth and electricity, uses up extra Actions minutes, and hammers the GNU servers. This can be fixed by either changing the submodule to point to an existing tag or branch (if branch, you have to keep on updating the submodule whenever the branch tip changes), or by forking the upstream repository and adding a tag to the commit you want to use, and then adjusting the submodule to point to your fork.

lurch commented

2022-03-10 11:59:46 -06:00

(Migrated from github.com)

I believe the contents of this repo are basically "unsupported, use at your own risk", and I'm not aware of any plans to actively update the contents of this repo.

Looking at https://savannah.nongnu.org/projects/lwip/ I see that lwIP 2.1.2 was released in November 2018 and lwIP 2.1.3 was released in November 2021. I've never looked at any of the lwIP stuff myself, so would you be willing to do some testing to see if STABLE-2_1_3_RELEASE would be a suitable tag for us to pin the submodule to?

I believe the contents of this repo are basically "unsupported, use at your own risk", and I'm not aware of any plans to actively update the contents of this repo. Looking at https://savannah.nongnu.org/projects/lwip/ I see that lwIP 2.1.2 was released in November 2018 and lwIP 2.1.3 was released in November 2021. I've never looked at any of the lwIP stuff myself, so would you be willing to do some testing to see if `STABLE-2_1_3_RELEASE` would be a suitable tag for us to pin the submodule to?

kilograham commented

2022-03-10 12:14:27 -06:00

(Migrated from github.com)

yes we can update the tag if you find one that works

ali1234 commented

2022-03-20 12:44:42 -05:00

(Migrated from github.com)

I have never used lwip either. I don't know what it is supposed to do. The only reason I even know it exists is because of the build failure it causes. There are no mentions of it anywhere in pico-sdk or pico-playground repositories and the only use in pico-extras is where the platform code initializes it. As such it isn't clear how I would determine whether it is working correctly or not, since it doesn't actually do anything.

I can tell you that 2.1.x is a stable release with backported fixes and therefore does not contain every commit present in c385f3107 because it branched when 2.1.0 was released in 2018.

I have never used lwip either. I don't know what it is supposed to do. The only reason I even know it exists is because of the build failure it causes. There are no mentions of it anywhere in pico-sdk or pico-playground repositories and the only use in pico-extras is where the platform code initializes it. As such it isn't clear how I would determine whether it is working correctly or not, since it doesn't actually do anything. I can tell you that 2.1.x is a stable release with backported fixes and therefore does not contain every commit present in c385f3107 because it branched when 2.1.0 was released in 2018.

lurch commented

2022-03-21 05:29:57 -05:00

(Migrated from github.com)

pico-extras is effectively a "use at your own risk" collection of not-yet-officially-supported code, so it might be better to copy out the parts that you need into your own project, rather than trying to clone the whole repo during your CI process? 🤷‍♂️

`pico-extras` is effectively a "use at your own risk" collection of not-yet-officially-supported code, so it might be better to copy out the parts that you need into your own project, rather than trying to clone the whole repo during your CI process? :man_shrugging:

kilograham commented

2022-03-21 12:09:35 -05:00

(Migrated from github.com)

then, you should be able just add:

set(SKIP_LWIP 1)

to the top of your CMakeLists.txt (before you init the SDK)

then, you should be able just add: ``` set(SKIP_LWIP 1) ``` to the top of your CMakeLists.txt (before you init the SDK)

ali1234 commented

2022-03-21 12:27:11 -05:00

(Migrated from github.com)

That won't help because the problem occurs during the cloning of the repository:

$ git clone --recursive --depth 1 --shallow-submodules https://github.com/raspberrypi/pico-extras
Cloning into 'pico-extras'...
remote: Enumerating objects: 153, done.
remote: Counting objects: 100% (153/153), done.
remote: Compressing objects: 100% (118/118), done.
remote: Total 153 (delta 14), reused 121 (delta 6), pack-reused 0
Receiving objects: 100% (153/153), 131.68 KiB | 807.00 KiB/s, done.
Resolving deltas: 100% (14/14), done.
Submodule 'lwip' (https://git.savannah.nongnu.org/git/lwip.git) registered for path 'lib/lwip'
Cloning into '/home/al/pico-extras/lib/lwip'...
remote: Counting objects: 851, done.        
remote: Compressing objects: 100% (742/742), done.        
remote: Total 851 (delta 176), reused 497 (delta 71)        
Receiving objects: 100% (851/851), 1.84 MiB | 2.14 MiB/s, done.
Resolving deltas: 100% (176/176), done.
remote: Total 0 (delta 0), reused 0 (delta 0)
error: Server does not allow request for unadvertised object c385f31076b27efb8ee37f00cb5568783a58f299
Fetched in submodule path 'lib/lwip', but it did not contain c385f31076b27efb8ee37f00cb5568783a58f299. Direct fetching of that commit failed.

That won't help because the problem occurs during the cloning of the repository: ``` $ git clone --recursive --depth 1 --shallow-submodules https://github.com/raspberrypi/pico-extras Cloning into 'pico-extras'... remote: Enumerating objects: 153, done. remote: Counting objects: 100% (153/153), done. remote: Compressing objects: 100% (118/118), done. remote: Total 153 (delta 14), reused 121 (delta 6), pack-reused 0 Receiving objects: 100% (153/153), 131.68 KiB | 807.00 KiB/s, done. Resolving deltas: 100% (14/14), done. Submodule 'lwip' (https://git.savannah.nongnu.org/git/lwip.git) registered for path 'lib/lwip' Cloning into '/home/al/pico-extras/lib/lwip'... remote: Counting objects: 851, done. remote: Compressing objects: 100% (742/742), done. remote: Total 851 (delta 176), reused 497 (delta 71) Receiving objects: 100% (851/851), 1.84 MiB | 2.14 MiB/s, done. Resolving deltas: 100% (176/176), done. remote: Total 0 (delta 0), reused 0 (delta 0) error: Server does not allow request for unadvertised object c385f31076b27efb8ee37f00cb5568783a58f299 Fetched in submodule path 'lib/lwip', but it did not contain c385f31076b27efb8ee37f00cb5568783a58f299. Direct fetching of that commit failed. ```

kilograham commented

2022-03-21 12:32:41 -05:00

(Migrated from github.com)

well it is the only submodule, so you don't need to do a recursive clone...

... should be a workaround until we update it.

well it is the only submodule, so you don't need to do a recursive clone... ... should be a workaround until we update it.

peterharperuk commented

2023-02-13 11:24:10 -06:00

(Migrated from github.com)

This seems to work for me on the sdk
git clone --recursive --depth 1 --shallow-submodules https://github.com/raspberrypi/pico-sdk

For lwip in the sdk, we are a few commits from the tip of master (not entirely sure why we made that decision) presumably it's moved on a few commits since mid 2022.

Not sure what there is to do here, I guess it would do no harm to move to the tip.

This seems to work for me on the sdk git clone --recursive --depth 1 --shallow-submodules https://github.com/raspberrypi/pico-sdk For lwip in the sdk, we are a few commits from the tip of master (not entirely sure why we made that decision) presumably it's moved on a few commits since mid 2022. Not sure what there is to do here, I guess it would do no harm to move to the tip.

ali1234 commented

2023-02-13 13:43:10 -06:00

(Migrated from github.com)

I am not sure why it works now. It could be because the remote now points to the github mirror instead of savannah, and github might configure their servers differently. They never used to allow this either though.

kilograham commented

2023-05-30 17:04:38 -05:00

(Migrated from github.com)

um; i'm a bit confused, as this issue seems to refer repeatedly to pico-extras not pico-sdk... in any case it doesn't seem like #1369 is a suitable fix anyway, so won't fixing this for now - please re-open if there is still an issue

um; i'm a bit confused, as this issue seems to refer repeatedly to `pico-extras` not `pico-sdk`... in any case it doesn't seem like #1369 is a suitable fix anyway, so won't fixing this for now - please re-open if there is still an issue

ali1234 commented

2023-05-30 17:23:44 -05:00

(Migrated from github.com)

This issue was opened when lwip was a submodule of pico-extras. It is now a submodule of pico-sdk, and you moved the issue here when lwip was moved.

kilograham commented

2023-05-30 17:25:01 -05:00

(Migrated from github.com)

ha

kilograham commented

2023-05-30 17:25:42 -05:00

(Migrated from github.com)

anyways, i'm not sure there is much to be done, unless lwip makes a newer tag, right?

ali1234 commented

2023-05-30 17:27:44 -05:00

(Migrated from github.com)

It does seem that this is no longer a problem. I am not sure why because github definitely is not advertising github.com/lwip-tcpip/lwip@239918ccc1 (you can confirm this with git ls-remote.) It seems to be a change in the behavior of the git command itself, to download the full remote even when you ask for a shallow clone - and then discard anything it doesn't need.

(Could still also be related to github server configuration, since when this issue was opened the submodule was not hosted on Github.)

It does seem that this is no longer a problem. I am not sure why because github definitely is not advertising https://github.com/lwip-tcpip/lwip/tree/239918ccc173cb2c2a62f41a40fd893f57faf1d6 (you can confirm this with `git ls-remote`.) It seems to be a change in the behavior of the git command itself, to download the full remote even when you ask for a shallow clone - and then discard anything it doesn't need. (Could still also be related to github server configuration, since when this issue was opened the submodule was not hosted on Github.)

ali1234 commented

2023-05-30 17:44:16 -05:00

(Migrated from github.com)

This definitely seems like a difference in github behavior. Attempting to shallow clone an old tag of pico-extras:

git clone --branch sdk-1.2.0 --depth 1 --recursive --shallow-submodules https://github.com/raspberrypi/pico-extras

still results in the failure (because it is fetching from savannah).

This definitely seems like a difference in github behavior. Attempting to shallow clone an old tag of pico-extras: git clone --branch sdk-1.2.0 --depth 1 --recursive --shallow-submodules https://github.com/raspberrypi/pico-extras still results in the failure (because it is fetching from savannah).

Rows
Columns

lwip submodule should point to a tag or branch tip in the remote repository #1244