Building both static and shared libraries at the same time

MaesterZ commented

2022-06-10 03:22:39 -05:00

(Migrated from github.com)

Nomadic Labs is using libusb and hidapi for the Octez project at https://gitlab.com/tezos/tezos written in Ocaml.

Is there a clean way to compile and install both static and shared libraries at the same time?

@pietro submitted an MR to Alpine aports to have both static and shared libraries for package libusb-dev and is now available in Alpine 3.15+:

https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/23613

We did the same for package hidapi-dev but the switch to cmake removed the static libraries from Alpine 3.15 starting with 0.11.0:

https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/23615
https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/hidapi/APKBUILD

At the moment we have no other solutions than building from source or use a custom Alpine package. We copied the current GitHub workflow for Ubuntu and run the configure/build/install twice:

https://gitlab.com/nomadic-labs/hidapi-apk/-/blob/main/APKBUILD#L21

# Shared libraries
cmake -B build/shared -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=lib -DBUILD_SHARED_LIBS=TRUE
cmake --build build/shared
cmake --install build/shared
# Static libraries
cmake -B build/static -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=lib -DBUILD_SHARED_LIBS=FALSE
cmake --build build/static
cmake --install build/static

I have no experience with cmake, my concern is that we overwrite the following files instlalling twice:

/usr/lib/cmake/hidapi/hidapi-config-version.cmake
/usr/lib/cmake/hidapi/hidapi-config.cmake
/usr/lib/cmake/hidapi/hidapi-release.cmake
/usr/lib/cmake/hidapi/hidapi.cmake

Note: Alpine aports MR https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/35251

Nomadic Labs is using `libusb` and `hidapi` for the Octez project at https://gitlab.com/tezos/tezos written in Ocaml. Is there a clean way to compile and install both static and shared libraries at the same time? [@pietro ](https://github.com/abate) submitted an MR to Alpine `aports` to have both static and shared libraries for package `libusb-dev` and is now available in Alpine `3.15+`: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/23613 We did the same for package `hidapi-dev` but the switch to `cmake` removed the static libraries from Alpine `3.15` starting with `0.11.0`: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/23615 https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/hidapi/APKBUILD At the moment we have no other solutions than building from source or use a custom Alpine package. We copied the [current GitHub workflow for Ubuntu](https://github.com/libusb/hidapi/blob/master/.github/workflows/builds.yml#L74) and run the configure/build/install twice: https://gitlab.com/nomadic-labs/hidapi-apk/-/blob/main/APKBUILD#L21 ``` # Shared libraries cmake -B build/shared -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=lib -DBUILD_SHARED_LIBS=TRUE cmake --build build/shared cmake --install build/shared # Static libraries cmake -B build/static -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=lib -DBUILD_SHARED_LIBS=FALSE cmake --build build/static cmake --install build/static ``` I have no experience with `cmake`, my concern is that we overwrite the following files instlalling twice: ``` /usr/lib/cmake/hidapi/hidapi-config-version.cmake /usr/lib/cmake/hidapi/hidapi-config.cmake /usr/lib/cmake/hidapi/hidapi-release.cmake /usr/lib/cmake/hidapi/hidapi.cmake ``` Note: Alpine `aports` MR https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/35251

Youw commented

2022-06-10 04:52:09 -05:00

(Migrated from github.com)

The way you're building it right now is the cleanest possible with CMake at the moment.

The only implication of the overwriting CMake files is that when a CMake user would try to use HIDAPI with
find_package(hidapi) it will only find target(s) (static or shared) that where installed the last.

In your example, CMake's find_package will find only targets for static HIDAPI.

This may be somethign that could be improved on HIDAPI side, but I need to understad one thing:

When a library is built and installed in the system, and there is both static and shared variants of the library + .pc pkg-config file(s).

Normally, if the user needs to use the library, the common approach is to use pkg-config, e.g.:
add to link flags: pkg-config --libs zlib
and usually that would be something like: -L/opt/local/lib -lz
which will link to a shared version of the library.
Having: pkg-config --libs zlib --static would give the same -L/opt/local/lib -lz and will link against shared library anyway, even though static version is installed and available too.

How exactly user gets to choose the static version of the library w/o manually hard-coding the path to it?

The way you're building it right now is the cleanest possible with CMake at the moment. The only implication of the overwriting CMake files is that when a CMake user would try to use HIDAPI with [`find_package(hidapi)`](https://cmake.org/cmake/help/latest/command/find_package.html) it will only find target(s) (static or shared) that where installed the last. In your example, CMake's `find_package` will find only targets for static HIDAPI. --- This may be somethign that could be improved on HIDAPI side, but I need to understad one thing: When a library is built and installed in the system, and there is both static and shared variants of the library + `.pc` pkg-config file(s). Normally, if the user needs to use the library, the common approach is to use pkg-config, e.g.: add to link flags: `pkg-config --libs zlib` and usually that would be something like: `-L/opt/local/lib -lz` which will link to a _shared_ version of the library. Having: `pkg-config --libs zlib --static` would give the same `-L/opt/local/lib -lz` and will link against shared library anyway, even though static version is installed and available too. How exactly user gets to choose the static version of the library w/o manually hard-coding the path to it?

Youw commented

2022-06-10 04:55:16 -05:00

(Migrated from github.com)

https://gitlab.com/nomadic-labs/hidapi-apk/-/blob/main/APKBUILD#L30

BTW: I recommend installing static version first, and then shared, so by default your CMake users would use a shared version of the library - to be consistent with pkgconfig users in general.

> https://gitlab.com/nomadic-labs/hidapi-apk/-/blob/main/APKBUILD#L30 BTW: I recommend installing static version first, and then shared, so by default your CMake users would use a shared version of the library - to be consistent with pkgconfig users in general.

netfab commented

2022-06-14 10:36:54 -05:00

(Migrated from github.com)

Hi,

How exactly user gets to choose the static version of the library w/o manually hard-coding the path to it?

In ¹ we can read :

Unfortunately, at least as of February 2013, there is no easy way to tell at once to an Autotools-based build system that you
intend to link statically; this means that the most common way to signal this to pkg-config becomes the command ./configure
LDFLAGS="-static" PKG_CONFIG="pkg-config --static".

However, if you want to link to a specific library statically (and others dynamically), you can use an option for the linker for that specific library :

PKG_CHECK_MODULES([LIBHIDAPI], [hidapi-libusb >= 0.10.0])
LIBHIDAPI_LIBS="-l:libhidapi-libusb.a"

See the following option in ² (but the documentation seems not very clear to me) :

-l namespec
--library=namespec

Tested here and it seems to work fine.

Hi, > How exactly user gets to choose the static version of the library w/o manually hard-coding the path to it? In ¹ we can read \: > Unfortunately, at least as of February 2013, there is no easy way to tell at once to an Autotools-based build system that you > intend to link statically; this means that the most common way to signal this to pkg-config becomes the command ./configure > LDFLAGS="-static" PKG_CONFIG="pkg-config --static". However, if you want to link to a specific library statically (and others dynamically), you can use an option for the linker for that specific library \: > PKG_CHECK_MODULES([LIBHIDAPI], [hidapi-libusb >= 0.10.0]) > LIBHIDAPI_LIBS="-l:libhidapi-libusb.a" See the following option in ² (but the documentation seems not very clear to me) \: > -l namespec > --library=namespec Tested here and it seems to work fine. 1. https://autotools.info/pkgconfig/dependencies.html#pkgconfig.static-link 2. https://sourceware.org/binutils/docs/ld/Options.html

Youw commented

2022-06-14 16:35:11 -05:00

(Migrated from github.com)

This may be somethign that could be improved on HIDAPI side

My analysis:

LIBHIDAPI_LIBS="-l:libhidapi-libusb.a"

I'd say this is practically a "manually hard-coding the path" (library name).

LDFLAGS="-static" PKG_CONFIG="pkg-config --static"

CMake doesn't have any similar machanism(s), so I don't think it worth making anything HIDAPI-specific in this matter on CMake level.

Basically, for now I'd say what is done at https://gitlab.com/nomadic-labs/hidapi-apk/-/blob/main/APKBUILD#L37 is fine (and can be used as a reference for others who need a similar thing).
https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/hidapi/APKBUILD - uses the alternative approach, which technically is fine as well.
Both ways would give the same result.

The only true alternative, is to do something that, for instance, libjpeg-turbo does (here and here), but that would complicate CMake scripts for HIDAPI a lot, has complicated corner-cases (e.g. the Framework libraries on macOS), and I'm not sure it is really required.

> This may be somethign that could be improved on HIDAPI side My analysis: --- > LIBHIDAPI_LIBS="-l:libhidapi-libusb.a" I'd say this is practically a "manually hard-coding the path" (library name). > LDFLAGS="-static" PKG_CONFIG="pkg-config --static" CMake doesn't have any similar machanism(s), so I don't think it worth making anything HIDAPI-specific in this matter on CMake level. --- Basically, for now I'd say what is done at https://gitlab.com/nomadic-labs/hidapi-apk/-/blob/main/APKBUILD#L37 is fine (and can be used as a reference for others who need a similar thing). https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/hidapi/APKBUILD - uses the alternative approach, which technically is fine as well. Both ways would give the same result. --- The only true alternative, is to do something that, for instance, libjpeg-turbo [does](https://github.com/libjpeg-turbo/libjpeg-turbo/blob/a405519e961aeee3096f7bff45be6330dfa054b8/CMakeLists.txt#L184-L187) ([here](https://github.com/libjpeg-turbo/libjpeg-turbo/blob/a405519e961aeee3096f7bff45be6330dfa054b8/CMakeLists.txt#L650-L651) and [here](https://github.com/libjpeg-turbo/libjpeg-turbo/blob/main/CMakeLists.txt#L598,L614)), but that would complicate CMake scripts for HIDAPI a lot, has complicated corner-cases (e.g. the Framework libraries on macOS), and I'm not sure it is really required.

netfab commented

2022-06-15 04:52:47 -05:00

(Migrated from github.com)

LIBHIDAPI_LIBS="-l:libhidapi-libusb.a"

I'd say this is practically a "manually hard-coding the path" (library name).

Yes, but it is always possible to generate with cmake an additional *.pc file to link to the static archive.
In pc/hidapi-libusb-static.pc.in :

Libs: -L${libdir}                                                                                      
Libs.private: -L${libdir} -l:lib@OUTPUT_NAME@.a

And in cmake files :

index e5d7d51..d073545 100644
--- a/libusb/CMakeLists.txt
+++ b/libusb/CMakeLists.txt
@@ -65,3 +65,4 @@ if(HIDAPI_INSTALL_TARGETS)
 endif()
 
 hidapi_configure_pc("${PROJECT_ROOT}/pc/hidapi-libusb.pc.in")
+hidapi_configure_pc("${PROJECT_ROOT}/pc/hidapi-libusb-static.pc.in" "hidapi-libusb")
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 504b205..c8b0d4b 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -51,6 +51,9 @@ function(hidapi_configure_pc PC_IN_FILE)
     set(VERSION "${VERSION}${VERSION_SUFFIX}")
     set(prefix "${CMAKE_INSTALL_PREFIX}")
     set(exec_prefix "\${prefix}")
+    foreach(arg IN LISTS ARGN)
+        set(OUTPUT_NAME "${arg}")
+    endforeach()
     if(IS_ABSOLUTE "${CMAKE_INSTALL_LIBDIR}")
         set(libdir "${CMAKE_INSTALL_LIBDIR}")
     else()

Once all this stuff built and installed, the user may chose between static or dynamic linking by choosing a different *pc file :

$ pkg-config --libs hidapi-libusb
-lhidapi-libusb
$ pkg-config --libs hidapi-libusb-static --static
-l:libhidapi-libusb.a

I was just answering the question how a user can choose to link to the static library when static and dynamic are both installed. I do not know cmake more than that.

> > LIBHIDAPI_LIBS="-l:libhidapi-libusb.a" > > I'd say this is practically a "manually hard-coding the path" (library name). Yes, but it is always possible to generate with cmake an additional *.pc file to link to the static archive. In pc/hidapi-libusb-static.pc.in \: ``` Libs: -L${libdir} Libs.private: -L${libdir} -l:lib@OUTPUT_NAME@.a ``` And in cmake files \: ```diff --git a/libusb/CMakeLists.txt b/libusb/CMakeLists.txt index e5d7d51..d073545 100644 --- a/libusb/CMakeLists.txt +++ b/libusb/CMakeLists.txt @@ -65,3 +65,4 @@ if(HIDAPI_INSTALL_TARGETS) endif() hidapi_configure_pc("${PROJECT_ROOT}/pc/hidapi-libusb.pc.in") +hidapi_configure_pc("${PROJECT_ROOT}/pc/hidapi-libusb-static.pc.in" "hidapi-libusb") diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 504b205..c8b0d4b 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -51,6 +51,9 @@ function(hidapi_configure_pc PC_IN_FILE) set(VERSION "${VERSION}${VERSION_SUFFIX}") set(prefix "${CMAKE_INSTALL_PREFIX}") set(exec_prefix "\${prefix}") + foreach(arg IN LISTS ARGN) + set(OUTPUT_NAME "${arg}") + endforeach() if(IS_ABSOLUTE "${CMAKE_INSTALL_LIBDIR}") set(libdir "${CMAKE_INSTALL_LIBDIR}") else() ``` Once all this stuff built and installed, the user may chose between static or dynamic linking by choosing a different *pc file \: > $ pkg-config --libs hidapi-libusb > -lhidapi-libusb > $ pkg-config --libs hidapi-libusb-static --static > -l:libhidapi-libusb.a I was just answering the question how a user can choose to link to the static library when static and dynamic are both installed. I do not know cmake more than that.

Youw commented

2022-06-15 05:27:31 -05:00

(Migrated from github.com)

pkg-config --libs hidapi-libusb-static --static

Technically that is an entirely different library, which happens to have (almost) the same name. But thanks for suggestion.

I was just answering the question how a user can choose to link to the static library when static and dynamic are both installed. I do not know cmake more than that.

I guess the complete answer I was looking for:

to link all autotools dependencies statically: LDFLAGS="-static" PKG_CONFIG="pkg-config --static"
to link a specific library statically - almost certainly need to hard-code a static library name

> pkg-config --libs hidapi-libusb-static --static Technically that is an entirely different library, which happens to have (almost) the same name. But thanks for suggestion. > I was just answering the question how a user can choose to link to the static library when static and dynamic are both installed. I do not know cmake more than that. I guess the complete answer I was looking for: - to link all autotools dependencies statically: `LDFLAGS="-static" PKG_CONFIG="pkg-config --static"` - to link a specific library statically - _almost_ certainly need to hard-code a static library name

netfab commented

2022-06-17 03:21:53 -05:00

(Migrated from github.com)

pkg-config --libs hidapi-libusb-static --static

Technically that is an entirely different library, which happens to have (almost) the same name.

To be entirely complete on this subject, the additional pc file can be installed in a non-standard directory :

$ pwd
/usr/local/share/pkgconfig

$ ls
hidapi-libusb.pc

$ pkg-config --libs hidapi-libusb
-lhidapi-libusb

$ PKG_CONFIG_LIBDIR=/usr/local/share/pkgconfig pkg-config --libs hidapi-libusb --static
-l:libhidapi-libusb.a

$ pkg-config --list-all | grep hidapi-libusb
hidapi-libusb                  hidapi-libusb - C Library for USB HID device access from
                                      Linux, Mac OS X, FreeBSD, and Windows.
                                      This is the libusb implementation.

$ PKG_CONFIG_LIBDIR=/usr/local/share/pkgconfig pkg-config --list-all
hidapi-libusb                  hidapi-libusb - C Library for USB HID device access from
                                      Linux, Mac OS X, FreeBSD, and Windows.
                                      This is the libusb implementation, static version
                                      (you must use the pkg-config --static option).

But, yes, it seems that there is no other way.

> > pkg-config --libs hidapi-libusb-static --static > > Technically that is an entirely different library, which happens to have (almost) the same name. To be entirely complete on this subject, the additional pc file can be installed in a non-standard directory \: ``` $ pwd /usr/local/share/pkgconfig $ ls hidapi-libusb.pc $ pkg-config --libs hidapi-libusb -lhidapi-libusb $ PKG_CONFIG_LIBDIR=/usr/local/share/pkgconfig pkg-config --libs hidapi-libusb --static -l:libhidapi-libusb.a $ pkg-config --list-all | grep hidapi-libusb hidapi-libusb hidapi-libusb - C Library for USB HID device access from Linux, Mac OS X, FreeBSD, and Windows. This is the libusb implementation. $ PKG_CONFIG_LIBDIR=/usr/local/share/pkgconfig pkg-config --list-all hidapi-libusb hidapi-libusb - C Library for USB HID device access from Linux, Mac OS X, FreeBSD, and Windows. This is the libusb implementation, static version (you must use the pkg-config --static option). ``` But, yes, it seems that there is no other way.

mcuee commented

2022-08-16 08:52:32 -05:00

(Migrated from github.com)

Interesting there are similar discussions here -- looks like a common CMake question.

https://github.com/avrdudes/avrdude/issues/952

My comment here:

https://github.com/libusb/libusb/pull/1134#issuecomment-1119262956

For the library developers (like libusb project), this is required if the developers need to release the binaries.
But I do not think this is an issue for CMake. libftdi project is using CMake and it provides both shared and static libraries in single build.
http://developer.intra2net.com/git/?p=libftdi

Interesting there are similar discussions here -- looks like a common CMake question. * https://github.com/avrdudes/avrdude/issues/952 My comment here: * https://github.com/libusb/libusb/pull/1134#issuecomment-1119262956 > For the library developers (like libusb project), this is required if the developers need to release the binaries. > But I do not think this is an issue for CMake. libftdi project is using CMake and it provides both shared and static libraries in single build. > http://developer.intra2net.com/git/?p=libftdi

mcuee commented

2022-08-16 08:56:22 -05:00

(Migrated from github.com)

For example, if I only build the default dynamic link version, then I will get wrong static link option using pkg-config.

mcuee@UbuntuSwift3:~/build$ pkg-config --libs hidapi-libusb --static
-L/usr/local/lib -lhidapi-libusb

For example, if I only build the default dynamic link version, then I will get wrong static link option using pkg-config. ``` mcuee@UbuntuSwift3:~/build$ pkg-config --libs hidapi-libusb --static -L/usr/local/lib -lhidapi-libusb ```

Youw commented

2022-08-16 09:02:54 -05:00

(Migrated from github.com)

So far I haven't seen a single pkg-config package, that would provide link options to link against static version of the library, when pass --static paramater to pkg-config.

So far I haven't seen a single pkg-config package, that would provide link options to link against static version of the library, when pass `--static` paramater to `pkg-config`.

Youw commented

2022-08-16 09:06:16 -05:00

(Migrated from github.com)

a common CMake question

A common question of those who started using CMake after Autotools. Context/bias is important.

> a common CMake question A common question of those who started using CMake _after Autotools_. _Context/bias_ is important.

👍 2

mcuee commented

2022-08-16 09:09:40 -05:00

(Migrated from github.com)

So far I haven't seen a single pkg-config package, that would provide link options to link against static version of the library, when pass --static paramater to pkg-config.

I can see libusb-1.0 (autools) and libftdi-1.0 (CMake) try to differentiate between dynamic linking and static linking.

mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --libs libusb-1.0
-L/usr/local/lib -lusb-1.0
mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --static --libs libusb-1.0
-L/usr/local/lib -lusb-1.0 -ludev -latomic -lpthread
mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --libs libftdi1
-L/usr/local/lib -lftdi1 -lusb-1.0
mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --libs libftdi1 --static
-L/usr/local/lib -lftdi1 -lusb-1.0 -ludev -latomic -lpthread

> So far I haven't seen a single pkg-config package, that would provide link options to link against static version of the library, when pass `--static` paramater to `pkg-config`. I can see libusb-1.0 (autools) and libftdi-1.0 (CMake) try to differentiate between dynamic linking and static linking. ``` mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --libs libusb-1.0 -L/usr/local/lib -lusb-1.0 mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --static --libs libusb-1.0 -L/usr/local/lib -lusb-1.0 -ludev -latomic -lpthread mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --libs libftdi1 -L/usr/local/lib -lftdi1 -lusb-1.0 mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --libs libftdi1 --static -L/usr/local/lib -lftdi1 -lusb-1.0 -ludev -latomic -lpthread ```

Youw commented

2022-08-16 09:12:28 -05:00

(Migrated from github.com)

mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --static --libs libusb-1.0
-L/usr/local/lib -lusb-1.0 -ludev -latomic -lpthread

True, but pass those to the linker of you own library/application - you're still going to be linked against shared/.so version of libusb, right?

> mcuee@UbuntuSwift3:~/build/hidapi/build$ pkg-config --static --libs libusb-1.0 -L/usr/local/lib -lusb-1.0 -ludev -latomic -lpthread True, but pass those to the linker of you own library/application - you're still going to be linked against shared/.so version of libusb, right?

mcuee commented

2022-08-16 09:13:56 -05:00

(Migrated from github.com)

@Youw
Yes I agree they are not correct either -- use use libusb-1.0.a instead. I remember macOS is correct. I will check again.

@Youw Yes I agree they are not correct either -- use use libusb-1.0.a instead. I remember macOS is correct. I will check again.

mcuee commented

2022-08-16 09:18:04 -05:00

(Migrated from github.com)

@Youw

avrdude-libhidapi is basically repackged hidapi (old version) adapted to MSVC static build. For MSVC, it makes sense to use static build. Is it possible to modify hidapi so that there is no need for avrdude to use a fork version of hidapi?

I tend to see that avrdude has legitimate reasons to use avrdude-libwinusb (at least when libusb-1.0.23/1.0.24 are broken under Windows for libusb0.sys) and avrdude-libftdi (to use FTDI vendor driver) for MSVC build (Static linking) but no good reasons to use a fork version of hidapi for MSVC build.

Ref: github.com/avrdudes/libhidapi@e3700e951f

@Youw avrdude-libhidapi is basically repackged hidapi (old version) adapted to MSVC static build. For MSVC, it makes sense to use static build. Is it possible to modify hidapi so that there is no need for avrdude to use a fork version of hidapi? I tend to see that avrdude has legitimate reasons to use avrdude-libwinusb (at least when libusb-1.0.23/1.0.24 are broken under Windows for libusb0.sys) and avrdude-libftdi (to use FTDI vendor driver) for MSVC build (Static linking) but no good reasons to use a fork version of hidapi for MSVC build. Ref: https://github.com/avrdudes/libhidapi/commit/e3700e951f762ef92871ff4fc94586e4d1c042a6

Youw commented

2022-08-16 09:26:49 -05:00

(Migrated from github.com)