gh-libusb/hidapi

Watch

Fork 0

macOS dylib minor version numbers differ between cmake and autotools builds #622

New issue

Closed

opened 2023-09-03 12:18:06 -05:00 by ryandesign · 6 comments

ryandesign commented

2023-09-03 12:18:06 -05:00

(Migrated from github.com)

When building hidapi 0.14.0 on macOS, the dynamic library's minor version information is different depending on whether cmake or autotools was used for the build.

With autotools:

% otool -L /opt/local/lib/libhidapi.dylib | head -n2
/opt/local/lib/libhidapi.dylib:
	/opt/local/lib/libhidapi.0.dylib (compatibility version 1.0.0, current version 1.0.0)

With cmake:

% otool -L /opt/local/lib/libhidapi.dylib | head -n2
/opt/local/lib/libhidapi.dylib:
	/opt/local/lib/libhidapi.0.dylib (compatibility version 0.0.0, current version 0.14.0)

With autotools the library's major version is 0 and its minor version is 1.0.0 (compatible back to minor version 1.0.0) whereas with cmake the library's major version is still 0 but its minor version has regressed to 0.14.0 (compatible back to minor version 0.0.0).

This is bad because the major and minor library version that was available at compile time is baked into each binary that uses the library. For example, here's a program that was built with hidapi-compiled-with-autotools:

% otool -L /opt/local/bin/mspdebug | head -n2
/opt/local/bin/mspdebug:
	/opt/local/lib/libhidapi.0.dylib (compatibility version 1.0.0, current version 1.0.0)

If I then replace libhidapi with one compiled with cmake, the program will not run*:

% mspdebug
dyld: Library not loaded: /opt/local/lib/libhidapi.0.dylib
  Referenced from: /opt/local/bin/mspdebug
  Reason: Incompatible library version: mspdebug requires version 1.0.0 or later, but libhidapi.0.dylib provides version 0.0.0
zsh: abort      mspdebug

This means that if a user wants to switch from hidapi compiled with autotools to hidapi compiled with cmake, they have to recompile all their software that links with libhidapi. This is a barrier to getting people to switch from autotools to cmake, as I believe you want us to do.

You have to be very careful to match your previous library version numbering scheme when switching to a different build system. libtool has a very special way of defining library versions that you have to follow. What's not explained there, because I guess they considered it an implementation detail that you didn't need to know about if you never stopped using libtool, is that library minor versions on macOS are 1.0.0 higher than on other operating systems because when Mac OS X was first introduced it could not handle library minor versions less than 1.0.0 but there was existing Unix software using libtool that specified library versions less than 1.0.0. macOS has been able to handle library minor versions less than 1.0.0 for a long time now but of course libtool can never remove that workaround or else projects' library minor versions would decrease when they updated libtool.

*What I did not realize until I researched it for this bug report is that Apple removed these library version checks completely in macOS 12, and in macOS 10.14(?) through macOS 11, it only runs the checks if the library was built for a macOS version earlier than 10.14. But since I'm sure you want your software to work on the widest range of systems possible, you should continue to follow the original spirit of the library minor version numbers by never letting them decrease (for any given major library version number; when the major library version number increases, of course the library minor version number can reset).

When building hidapi 0.14.0 on macOS, the dynamic library's minor version information is different depending on whether cmake or autotools was used for the build. With autotools: ``` % otool -L /opt/local/lib/libhidapi.dylib | head -n2 /opt/local/lib/libhidapi.dylib: /opt/local/lib/libhidapi.0.dylib (compatibility version 1.0.0, current version 1.0.0) ``` With cmake: ``` % otool -L /opt/local/lib/libhidapi.dylib | head -n2 /opt/local/lib/libhidapi.dylib: /opt/local/lib/libhidapi.0.dylib (compatibility version 0.0.0, current version 0.14.0) ``` With autotools the library's major version is 0 and its minor version is 1.0.0 (compatible back to minor version 1.0.0) whereas with cmake the library's major version is still 0 but its minor version has regressed to 0.14.0 (compatible back to minor version 0.0.0). This is bad because the major and minor library version that was available at compile time is baked into each binary that uses the library. For example, here's a program that was built with hidapi-compiled-with-autotools: ``` % otool -L /opt/local/bin/mspdebug | head -n2 /opt/local/bin/mspdebug: /opt/local/lib/libhidapi.0.dylib (compatibility version 1.0.0, current version 1.0.0) ``` If I then replace libhidapi with one compiled with cmake, the program will not run*: ``` % mspdebug dyld: Library not loaded: /opt/local/lib/libhidapi.0.dylib Referenced from: /opt/local/bin/mspdebug Reason: Incompatible library version: mspdebug requires version 1.0.0 or later, but libhidapi.0.dylib provides version 0.0.0 zsh: abort mspdebug ``` This means that if a user wants to switch from hidapi compiled with autotools to hidapi compiled with cmake, they have to recompile all their software that links with libhidapi. This is a barrier to getting people to switch from autotools to cmake, as I believe you want us to do. You have to be very careful to match your previous library version numbering scheme when switching to a different build system. libtool has a [very special way](https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html) of defining library versions that you have to follow. What's not explained there, because I guess they considered it an implementation detail that you didn't need to know about if you never stopped using libtool, is that [library minor versions on macOS are 1.0.0 higher than on other operating systems](https://github.com/autotools-mirror/libtool/blob/1ec8fa28dcb29500d485c136db28315671ec4c3b/build-aux/ltmain.in#L7015-L7016) because when Mac OS X was first introduced it could not handle library minor versions less than 1.0.0 but there was existing Unix software using libtool that specified library versions less than 1.0.0. macOS has been able to handle library minor versions less than 1.0.0 for a long time now but of course libtool can never remove that workaround or else projects' library minor versions would decrease when they updated libtool. *What I did not realize until I [researched it](https://github.com/giordano/macos-compatibility-version) for this bug report is that Apple removed these library version checks completely in macOS 12, and in macOS 10.14(?) through macOS 11, it only runs the checks if the library was built for a macOS version earlier than 10.14. But since I'm sure you want your software to work on the widest range of systems possible, you should continue to follow the original spirit of the library minor version numbers by never letting them decrease (for any given major library version number; when the major library version number increases, of course the library minor version number can reset).

Youw commented

2023-09-04 04:17:53 -05:00

(Migrated from github.com)

This means that if a user wants to switch from hidapi compiled with autotools to hidapi compiled with cmake, they have to recompile all their software that links with libhidapi.

Can you elaborate what kind of software are you talking about?
I can't come up with a scenario where such happens and that is a blocker.
Based on #620 I only assume that "the software" is being developed/evolved (as it needs hidapi_darwin.h) so it will be re-compiled anyway.

> This means that if a user wants to switch from hidapi compiled with autotools to hidapi compiled with cmake, they have to recompile all their software that links with libhidapi. Can you elaborate what kind of software are you talking about? I can't come up with a scenario where such happens and that is a blocker. Based on #620 I only assume that "the software" is being developed/evolved (as it needs `hidapi_darwin.h`) so it will be re-compiled anyway.

ryandesign commented

2023-09-04 06:35:53 -05:00

(Migrated from github.com)

Can you elaborate what kind of software are you talking about?

I gave the example of mspdebug because it was one of the software programs in MacPorts that depends on hidapi and that has few other dependencies, but any software linking with the library would be affected.

Based on #620

That's a different matter. I'm a developer with MacPorts. The inciting incident that caused me to look into hidapi was the report to us five months ago that hidapi_darwin.h was not installed and my reencountering that issue yesterday. My investigation led me to discover that the cause was the difference in build system, so I filed #620. I discovered the recommendation to switch to cmake, which prompted me to investigate the viability of doing so, which led me to discover the dylib versioning difference and the inability (without workarounds, which we can use, with a little difficulty) to build both shared and static libraries at the same time.

> Can you elaborate what kind of software are you talking about? I gave the example of [mspdebug](https://ports.macports.org/port/mspdebug) because it was one of the software programs in MacPorts that depends on hidapi and that has few other dependencies, but any software linking with the library would be affected. > Based on #620 That's a different matter. I'm a developer with MacPorts. The inciting incident that caused me to look into hidapi was the [report to us five months ago that hidapi_darwin.h was not installed](https://trac.macports.org/ticket/67189) and my reencountering that issue yesterday. My investigation led me to discover that the cause was the difference in build system, so I filed #620. I discovered the recommendation to switch to cmake, which prompted me to investigate the viability of doing so, which led me to discover the dylib versioning difference and the inability (without workarounds, which we can use, with a little difficulty) to build both shared and static libraries at the same time.

ryandesign commented

2023-09-04 06:46:10 -05:00

(Migrated from github.com)

One difficulty in fixing the library versioning problem is that in the cmake build system you are using your project's version number as the library version number:

github.com/libusb/hidapi@c19ae126d8/mac/CMakeLists.txt (L18-L27)

These should not be coupled. There should be separate variables for the components of your library version number in the cmake build system as there already are in the autotools build system:

github.com/libusb/hidapi@c19ae126d8/configure.ac (L8-L15)

One difficulty in fixing the library versioning problem is that in the cmake build system you are using your project's version number as the library version number: https://github.com/libusb/hidapi/blob/c19ae126d8856b55af34d64ee11641920475961c/mac/CMakeLists.txt#L18-L27 These should not be coupled. There should be separate variables for the components of your library version number in the cmake build system as there already are in the autotools build system: https://github.com/libusb/hidapi/blob/c19ae126d8856b55af34d64ee11641920475961c/configure.ac#L8-L15

Youw commented

2023-09-04 07:18:15 -05:00

(Migrated from github.com)

TL;DR; version of the library built with CMake will remain unchanged by default.

These should not be coupled.

That is true if the project contains multiple components possibly of different versions or if the library has its own (different) versioning stategy.
As for HIDAPI - the library is the project. Everything else is just the helper infrastructure around it. At least this is the way I see it.

There should be separate variables

Most probably not a variable. The version of the library is not something that user should deside at build time.

as there already are in the autotools build system

As you may see from the blame - no one ever bothered to manage the library version with Autotools. The version always was 0.0.0 (or 1.0.0 because of the libtool behavior on macOS) and it makes no sense at all. I don't believe it was that way because someone designed it or that it has a special meaning - it is likely that way because no one really took care if it.
When I look at the library (e.g. with otool) I want to be able to know it version w/o checking with the package manager (in case of macOS - it could be a MacPorts, homebrew or no package manager at all, in which case - how do I check the version?).

In other words - I find current behavior of the Autotools to be a bug (e.g. #392) which is "fixed" in CMake.

But I totally understand the pain this may cause in scenarios like MacPorts when you upgrade only a single library w/o recompilation of all others.
I haven't consideret it earlier mostly because MacPorts is not the first package manager that has switched from Autotools to CMake and no one else had any problems with it. And if it wheren't for legacy behavior of libtool on macOS - MacPorts probably wouldn't hit this as an issue as well.

Specifically for cases like this one, I think it should suffise adding a macOS-specific CMake option HIDAPI_USE_LEGACY_LIBRARY_VERSION which when set to true would use version 1.0.0 for target/library version.

@ryandesign is there anything else stopping MacPorts from switching to CMake build system for HIDAPI?

TL;DR; version of the library built with CMake will remain unchanged by default. --- > These should not be coupled. That is true if the project contains multiple components possibly of different versions or if the library has its own (different) versioning stategy. As for HIDAPI - _the library is the project_. Everything else is just the helper infrastructure around it. At least this is the way I see it. > There should be separate variables Most probably not a _variable_. The version of the library is not something that user should deside at build time. > as there already are in the autotools build system As you may [see](https://github.com/libusb/hidapi/commit/6d50f8fa0f889566cdcc4affe9244ce0ae28d8fc#diff-49473dca262eeab3b4a43002adb08b4db31020d190caaad1594b47f1d5daa810R16-R19) from the blame - no one ever bothered to manage the library version with Autotools. The version _always_ was `0.0.0` (or `1.0.0` because of the `libtool` behavior on macOS) and it makes no sense at all. I don't believe it was that way because someone designed it or that it has a special meaning - it is likely that way because no one really took care if it. When I look at the library (e.g. with `otool`) I want to be able to know it version w/o checking with the package manager (in case of macOS - it could be a MacPorts, homebrew or no package manager at all, in which case - how do I check the version?). In other words - I find current behavior of the Autotools to be a bug (e.g. #392) which is "fixed" in CMake. --- But I totally understand the pain this may cause in scenarios like MacPorts when you upgrade only a single library w/o recompilation of all others. I haven't consideret it earlier mostly because MacPorts is not the first package manager that has switched from Autotools to CMake and no one else had any problems with it. And if it wheren't for legacy behavior of `libtool` on macOS - MacPorts probably wouldn't hit this as an issue as well. Specifically for cases like this one, I think it should suffise adding a macOS-specific CMake option `HIDAPI_USE_LEGACY_LIBRARY_VERSION` which when set to true would use version `1.0.0` for target/library version. @ryandesign is there anything else stopping MacPorts from switching to CMake build system for HIDAPI?

Youw commented

2023-10-23 04:08:01 -05:00

(Migrated from github.com)