Need API to get the HID Report Descriptor or a similar data structure #249
Labels
No labels
3rdparty
Android
API
Author does not respond
bsd
bug
build system/CI
ci-virtual-device
CMake
Contributions Welcome
Core
dependencies
device-specific
documentation
don't_merge
duplicate
enhancement
Example
Haiku
hidraw
hotplug
info required
invalid
libusb
macOS
Misc
Not an issue
question
Release
Test
Test GUI
Windows
wontfix
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
gh-libusb/hidapi#249
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
We need a function to read the HID Report Descriptor or an equal data structure with information about the fields in the different HID reports.
Background:
For the cross platform Mixxx DJ software we need a solution, to allow the end user to map HID controller buttons, faders, knobs, RGB LEDs, etc. These are different on any DJ controller. Therefore this mapping it must be customizeable by end users.
As input we need information about the field, e.g.: a 12 bit field at position n of Input report with ID0x02 with a logical minimum of -2048 and a maximum of +2047.
Potentially helpful links:
https://github.com/DIGImend/usbhid-dump
https://github.com/DIGImend/hidrd
Those are both licensed under the GPL, so they couldn't be used directly in hidapi unless the maintainers are willing to change the hidapi license. Nevertheless they could be useful examples to reference.
https://eleccelerator.com/tutorial-about-usb-hid-report-descriptors/
Go check:
https://github.com/todbot/win-hid-dump
https://github.com/todbot/mac-hid-dump
Actually, it somewhat comes for free as the basic usage page and usage are extracted from the report descriptor which is read here:
github.com/libusb/hidapi@0ea5e9502f/libusb/hid.c (L644)In my eyes it'd make sense to directly store this descriptor either
struct hid_device_info(simpler) orstruct hid_device_info(feels more elegant)That said, it might be nice or meaningful to somehow incorporate this quite nice descriptor to string parser based on which I'm momentarily trying to make a generic parser to help extract meaningful values from given reports without having to know the report layout beforehand (primarily trying to make it work with my use cases, but for simple devices should be fine).
Thanks for that info @tschiemer. I have not looked into what it would take to implement this. If you have a branch of hidapi you are working on, please open a pull request for this upstream repository and I'd be happy to test it with my Native Instruments Traktor Kontrol S4 Mk3.
Ok, can see to it, but might take a moment
While this indeed already available and easy to get for libusb and hidraw backends, one of the first things about HIDAPI is being cross-platform.
I agree that this would be very useful, but Windows backend doesn't have an API to get a descriptor at all.
The descriptor could be partially reconstructed, but still, it is not an easy task.
I know one successfull attempt to reconstruct a HID descriptor using WinAPI calls, but that one is written on C# (https://github.com/todbot/win-hid-dump is using it). If someone could port it to C - I'd gladly introduce a new API (to get HID descriptor) for all platforms/backends.
There's also the approach to use the generic WinUSB driver (comes with Win7 or newer) instead of Windows High-Level-HID-API: https://github.com/libusb/hidapi/issues/61#issuecomment-638397298
As far as I understood it, the WinUSB driver gives a userspace Windows program full access to an USB device, but locks it exclusive.
Yeah, and someone actually needs to make a driver for each specific device. And the problem not even with making a driver - a template inf file can be generated. But signing a driver properly - that requires some effort.
And you'd need to compile a libusb backend on Windows, which isn't compilable of-the-box yet.
Ok, that doesn't sounds like what we're looking for.
I think it could be a good idea to start with libusb or hidraw to figure out what the API should be then implement it for other backends.
@Be-ing so not quite a hidapi branch, but as a proof of concept I implemented this puredata external to parse input reports based on the report descriptor, works for me on macos and a rpi4/linux with a joystick and mouse (see releases for binaries).
It's a bit of a mess as it uses libusb to find devices according to search criteria (more than hidapi provides out of the box, but meaningful in my opinion and based on hidapi-libusb internals) and reads/parses the report descriptor but relies on hidapi to actually receive input reports (and thus on linux the device listing is set to detach devices as in the hidapi driver..) - should have had a deeper look at hidapi's (or hidraw) interface before..
At the moment only the raw values as given in the input report are extracted without making use of any of the logical/physical min/max, unit, exponential etc values but adding that shouldn't be too big of a deal..
Could not test this with any feature/output reports or more complicated report descriptors, as I have neither the devices nor much more capacity..
The hid report parser just creates a datastructure that's more easy to handle but I would guess it won't work for all descriptors (collection items are somewhat ignored; note that the handling of output and feature items likely will need some fixing in the parser..).
As external it "publishes" any changed values (usage page, usage, value), similar to the apple HIDIOManager value callbacks.
I guess construction of output reports can be achieved by inverting the logic of the value extractor.
Maybe this is helpful in some form
I implemented such a reconstructor in C and will start a Pull Request for HIDAPI soon. Stay tuned!
I started PR #262 with a C implementation of the Windows Report-Descriptor code.
I think libusb HID backend has already the codes to partially reconstruct the descriptors. Some from the USB HUB driver, some from the USB HID driver. But that may not be suitable for HIDAPI as it is also for bluetooth devices.
https://github.com/libusb/libusb/blob/master/libusb/os/windows_winusb.c#L3728
static int hid_open(int sub_api, struct libusb_device_handle *dev_handle)
FYI I have just tested win-hid-dump. It seems I got some problems with it.
https://github.com/todbot/win-hid-dump/issues/2
But HIDSharp seems to work (not checking the contents). I have a long list of HID device in the system (Windows 10, Dell Laptop with a docking station). Edit to add: it is not stable for my system and the output does not seem to be correct, at least for my Microchip PICKit 2).
click to expand
mac-hid-dump output seems to be a bit too simple.
Edit to add: but external tools can parse the output.
Example mentioned by #262
https://eleccelerator.com/usbdescreqparser/
Raw HID reports (part of Logitech USB Receiver):
Parsed output from the above website.
I tested your new code, on my Windows 7 system. unfortunateley it always fails with:
I would be glad if your approach would work, because gather the real Report Descriptor would be better than any reconstruction.
@DJm00n I tried RawInputDemo under Windows 10 using VS2019, no problem running it but the RawInputDemo.exe does not seem to do anything. How do I use it?
@JoergAtGithub #262 seems to running fine at my Dell Windows 10 laptop with a docking edition and quite some HID devices. The only problem is that I do not know the original HID report descriptor of the devices other than the device I have access to the FW (say Microchip based Simple HID Demo, PICkit 2, Lakeview Research HID demo) and they are on the simpler side.
Further test shows that HIDSharp is not so stable for my system, sometime it hangs.
Pull request #262 seems to be pretty stable and product consistent results (but I need to check whether it is correct or not).
@mcuee sorry. I forgot to mention - it outputs info to debug console window in msvs.
This is what I got in the VS2019 output window. Where can I see the debug console? Sorry but I seldom use VS.
I found that pull request #262 is not correct for Microchip PICKit 2 and then I run HIDSharp again and it is also not correct.
Original HID report descriptor
Dumped by pull request #262
https://eleccelerator.com/usbdescreqparser/ output.
As a comparison, this is what I get from HIDSharp which is also not correct.
The HID Report Descriptors that HIDSharp (and
win-hid-dump) emit are reconstructions based on what is available via Windows HID APIs, it is not the actual descriptor sent by the device, but rather a "consolation-of-a-parsed-data-structure".It should be semantically (to the OS) the same, however. If it's not, it usually means the descriptor sent by the device is wrong in some subtle way. For vendor usages, this doesn't much matter, as then all that really matters is REPORT_ID, REPORT_COUNT, and REPORT_SIZE
I understand, but in this case, it is quite a bit OFF. It is totally different from what the device does. REPORT_COUNT and REPORT_SIZE are totally wrong.
#262 actually looks better in this case.
@todbot Maybe this device is indeed strange, it has two USB configurations but that should not matter as Windows will ignore the second configuration.
Full USB Descriptor C source code for reference.
click to expand
Also, @mcuee since this is a PICkit2 and not your own device, have you uninstalled all the Microchip Windows drivers? Vendor drivers usually really mess up USB HID libraries. (Some even create fake HID devices that mask the real device. Not what's going on here I think)
@todbot This is a standard HID device and no vendor driver is needed (ignore the second configuration, which was a legacy from PICkit 1, and meant for macOS last time). I know this Microchip PICKit 2 very well as I was helping Microchip to test the cross platform pk2cmd applications many years ago.
@todbot This is a good point. In that case, it seems #262 is actually okay for Microchip PICkit 2. Report size of 8 bytes is correct. Report count of 1 is actually correct as well. Report ID is in fact also correct (0, or no report ID).
The following raw report is reported by mac-hid-dump (https://github.com/todbot/mac-hid-dump). It matches exactly as the Firmware codes. Good.
Translation from https://eleccelerator.com/usbdescreqparser/
The following is from USB Prober under macOS Big Sur (Mac Mini M1), results seem to be similar to mac-hid-dump.
mac-hid-dump is very accurate accurate. It matches exactly my FW codes.
FW:
So we seem to have a good solution for Linux/FreeBSD already. For macOS, it seems to me mac-hid-dump is a good base.
Only Windows may be challenging. But it seems to me #262 works reasonably well based on my limited testing.
usbhid-dump under Linux is indeed very accurate too. It matched exactly the FW codes.
For 0925 7001: Lakeview Research - Generic HID
For 04D8 0033: Microchip Technology Inc. - PICkit 2 Microcontroller Programmer
Did you change the Report Descriptor of your test device between the tests? Otherwise I can't understand why Report count of 1 is Ok here.
MacOS and Linux provide APIs for reading the Report descriptor of an HID device. Therefore you get the unmodified Report Descriptor on these operating systems.
Unfortunately the Windows HID subsystem doesn't provide such an API. What I did in #262 is to reconstruct a Report Descriptor from the available high-level HID API (same technical approach as win-hid-dump). This reconstructed Report Descriptor has the same logical meaning in most aspects, as the original Report Descriptor, but is bytewise different.
There is another promissing approach by @DJm00n , which unfortunately doesn't work for me yet, to download the real HID report Descriptor using Windows USB API: https://github.com/DJm00n/RawInputDemo/tree/hid_report_descriptor_dump
I have asked in the libusb-devel to see if Tim Roberts has some suggestions.
No I did not change. So looks like there are still issues.
BTW, I am not that confident of the report parser website or Linux hidrd for the devices. I will probably check the output from Apple USB Prober under macOS.
Answer from Tim in libusb-devel mailing list
On Sat, Jun 19, 2021 at 12:14 PM Tim Roberts wrote:
So it seems to me the only possibility is probably the usbview trick. It is only for USB HID devices though, not Bluetooth.
https://github.com/microsoft/Windows-driver-samples/tree/master/usb/usbview
https://github.com/DJm00n/Windows-driver-samples/commits/usb-hid-descriptor/usb/usbview
I need to check again the following.
https://github.com/DJm00n/RawInputDemo/tree/hid_report_descriptor_dump
There is also the reconstructor from Google Chromiums WebHID implementation, which accesses the not official documented struct _HIDP_PREPARSED_DATA directly instead of using the official functions for this.
https://chromium.googlesource.com/chromium/src/+/73fdaaf605bb60caf34d5f30bb84a417688aa528/services/device/hid/hid_preparsed_data.cc
This approach could solve the remaining issues with #262 and win_hid_dump, where HidP_SetData returns HIDP_STATUS_IS_VALUE_ARRAY ( button array with usage 0). I think this case is an error in the HidP_SetData function, which Google circumvented by accessing the _HIDP_PREPARSED_DATA direct, which is the input for HidP_SetData.
This should be solveable, it is the second case in rd_determine_value_bitpositions, where I commented "// EXPERIMENTAL - No device available for test":
github.com/libusb/hidapi@b54a041cad/windows/hid.c (L871)win_hid_dump does this better here, but therefore fails with the report_size, which #262 determines correct . But it proves that the needed data are available. The win_hid_dump implementation of this case is here:
github.com/IntergatedCircuits/HidSharp@e8359220e0/HidSharp/Platform/Windows/WinHidDevice.ReportDescriptorReconstructor.cs (L161)Personally I think using unofficiallly documented internal struct _HIDP_PREPARSED_DATA is not the way to go for HIDAPI. But anyway, I have asked in the libusb-devel mailing list again.
https://marc.info/?t=162372687300002&r=1&w=2
I will check the general question in OSR forum as well.
https://community.osr.com/discussion/292947/how-to-get-hid-report-descriptor-using-windows-api
I think the best possible approach would be, to try the USB hub trick to read the real Report Descriptor, and if it fails return the reconstructed Report Descriptor instead. The later is needed anyway, because the USB hub trick will not work with Bluetooth HID devices.
Just tried this one and it does not seem to work at all for USB HID Report Descriptor, tested on my home Acer Windows 10 laptop.
https://github.com/DJm00n/Windows-driver-samples/tree/usb-hid-descriptor/usb/usbview
For example, this is the output from my mode of Lakeview Research Demo.
For this particular device, HIDSharp is about right.
Then I tried RawInputDemo here with my home Acer Windows 10 laptop and it crashed.
https://github.com/DJm00n/RawInputDemo/tree/hid_report_descriptor_dump
Call stack
Edit to add: if not plugging in the Lakeview Research HID demo, it will not crash but I do not see any output of HID report descriptors.
Thanks for working on this @mcuee and @JoergAtGithub!
There is one general issue with "usbview" or "IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTION" way:
Those would give the descriptor of the entire HID device (usb interface) which is basically what we can get on Linux/macOS.
But that wouldn't be accurate in case of Windows, because Windows creates a unique virtual HID device for each usage_page/usage combination of the HID device.
Each such virtual device accepts only a subset of reports that are corresponding to a specific usage_page/usage combination.
When we are (attempting) to reconstruct the describtor based on capabilities reported by WinAPI - we at least prividing more-or-less accurate information about the virtual device (even if it is not identical to the real descriptor).
If we provide the entire descriptor of a real HID interface/device - it will contain reports that cannot be accepter by the device.
Indeed this can be a potential problem. Windows treats them just like USB Composite Devices.
Windows HID Top-Level Collections:
https://docs.microsoft.com/en-us/windows-hardware/drivers/hid/hid-collections
In Windows, the HID device setup class (HIDClass) generates a unique physical device object (PDO) for each Top Level Collection described by the Report Descriptor. Microsoft defines a top-level collection as a HID collection that is not nested within another collection. An unnested collection is always a top-level collection, regardless of its HID type. In particular, a top-level collection does not have to be an Application collection, as defined by the USB HID Standard.
There are some arguments about this.
Ref: https://community.osr.com/discussion/287808/ioctl-usb-get-descriptor-from-node-connection
For a HID report descriptor, the Recipient bits of bmRequestType must be set to Interface (1).
IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTION sets bmRequestType to 0x80.
From
github.com/DJm00n/RawInputDemo@026261520bExample Logitech USB Receiver. It is a USB Composite Device, under Windows Device Manager, it will show as 9 devices, one USB Composite Device parent, then six devices in Human Interface Devices section, one in mouse and one in keyboard.
HIDSharp will show it as six HID devices, one for Interface 0, one for each top level collection of Interface 1 (col 1/2/3/4/5)..
Full run-log of HIDSharp under my Acer laptop with the Logitech USB receiver installed.
click to expand
USB Prober output from Mac Mini M1.
click to expand
The HidSharp report descriptor reconstructor doesn't implement reconstruction of the collection structure, but #262 does.
Can you please upload the output for this device with #262 too.
I guess I wasn't technically correct, as usage_page/usage combination isn't always a Top Level Collection (I was referencing to my knowledge/experience at the time, instead of check with the documentation), but that doesn't change the root of the problem.
This is the run log from #262 under my Acer Windows 10 laptop (with Acer built-in HID device and the Logitech M238 USB wireless mouse). I just use the following branch.
https://github.com/JoergAtGithub/hidapi/tree/windows_report_descriptor_reconstructor
click to expand
You can compare with the macOS USB Prober output of the particular USB receiver (part of the Logitech M238 wireless USB mouse)
USB Prober output from Mac Mini M1.
click to expand
BTW there are no answers here in OSR forum. I guess this is a real difficult question.
https://community.osr.com/discussion/292947/how-to-get-hid-report-descriptor-using-windows-api
This is a serious issue for cross platform software based on HIDAPI in general, because such an device behaves different on Linux/Mac vs. Windows. The report descriptor is only one aspect were this unsolved issue appears again.
A proper fix would require, to group these virtual devices during device enumeration and represent them to the API as the same single HID device as on other platforms.
One potential way to sort out the issue is to create a libusb backend for Windows. It will have limited usage for people who are willing to use WinUSB driver to replace the Windows HID driver (and even the USB Composite Device Parent driver if that is the case)..
We're doing qite the opposite for other platforms: we report same device multiple times, so each such unique device has its own usage_page/usage filled up.
And it kind of bits off a general desing of HIDAPI - to be as simple as possible and just be a thin proxy over an OS-specific API/implementation, just enough to keep the uniform API across different platforms. (libusb backend is kind-of the most complicated in this matter, as it practically implements its own USB HID driver, but let it be the only exception.)
What I want to suggest instead is to start with a question:
If I understand the initial problem correctly, a user application just wants to know the information about the reports, e.g.:
This is a bit out of scope of the initial HIDAPI design as well. As Alan always mentioned, that if you want to use HIDAPI, you probably already know the device you want to work with, and its API/reports etc. Clearly there are cases when a user still needs more than that (I guess
Mixxx DJ SWis an example of such), and maybe we think about this problem from a different angle:E.g. as "HIDAPI specific data structure", with a number of input/output reports and its characteristics.
That would require parsing a HID descriptor on all platforms except Windows (parsing is already partially implemented, maybe not for everything, but still), and on Windows we would fill such a structure directly from WinAPI. I haven't looked at the details, but unless there is something Windows doesn't provide, that a user application might need (the "HIDAPI specific data structure" would need to contain), that should do it in a much simpler manner.
It would also make things simpler for all user applications, as there is no need to perform HID descriptor parsing.
And yes, this kind of sound like a big feature to me, but it should be quite usefull.
To me the goal may need to be a bit modest: to get REPORT_ID, REPORT_COUNT, and REPORT_SIZE correct.
As for hid report parser, there are a few past efforts.
https://github.com/mcuee/libhid-win/tree/master/hidparser
https://github.com/supercollider/hidapi/tree/main/hidapi_parser
My motivation is to gather the report internal structure for the represenation in Mixxx. To offer our users the correct bit range for each control (Rotary knobs, Faders, Buttons, LEDs) to map it without programming skills. Therefore it's primary LogicalMinimum/Maximum, Report ID and of cause the position of the bits in a report.
I don't see a need for an report descriptor parser inside HIDAPI. (The debug output in #262 is not intended to be released). I think many applications on top of HIDAPI are often written in higher level languages, and the developers of it want these data in the object oriented data structures of these languages anyway. Furthermore, parsing the report descriptor is not a difficult task. And it can be encapsulated in an own library.
The difficulty with the WIN32 API is not to create the report descriptor structure, it's to gather the bit positions of a capability from WIN32 API. After digging deep into this topic, I think the bytewise report descriptor is the best achievable interface between HIDAPI and high-level applications. It's a simple data structure (just an array of bytes) and it's well defined in the HID spec.
You said it yourself.
It is not easy-enough to ask each user to have it on its own, but it is definitely easier than reconstructing the descriptor back using WinAPI, or trying to merge multiple Windows devices into a one HIDAPI device, or trying to get a descriptor from Bluetooth HID device on Windows.
Making HIDAPI uniform across all platforms doesn't mean making it "as on Linux".
I started the work on #262 after @Youw stated:
It took me 3 weeks of intensive work to achieve this. If there's no quality issue with my implementation, I expect that my contribution is accepted.
So the idea is to have a new API like the following and not to include HID reports parser.
Take note I am just using #263 as an example but I have not tested #263.
https://github.com/libusb/hidapi/pull/263/files
It seems that there are no issues with macOS (eg:https://github.com/todbot/mac-hid-dump/blob/main/mac-hid-dump.c), Linux hidraw, and libusb backend. But someone needs to do it.
As for Windows, #262 seems to be reasonably well and I will carry out more tests. We can always label this Windows implementation as EXPERIMENTAL.
I am for this idea to include this new API.
For further context, the end goal for Mixxx is to create a declarative system for mapping HID report fields to callbacks which manipulate application state when the state of the field changes between received reports. This would help unify the APIs for mapping MIDI and HID devices.
@Be-ing
Is an API like "static int get_hid_report_descriptor(IOHIDDeviceRef device, wchar_t* buf, size_t len)" good enough for you? You have to carry out the HID report parsing in the Mixxx application.
BTW, have you guys give #262 a test for Mixxx (USB and Bluetooth)? Most of my devices are USB HID device with the exception of Xbox One S controller (USB/Bluetooth).
Based on my testing of #262 and HIDSharp, it seems to me that HID report can be written in many different variations and it is not easy to get right (maybe impossible) if you do not know the HID report format of a particular device.
But if you know the device, then you do not need to guess and you can send the right request. That is probably the initial thought of Alan Ott -- you need to know the device in order to use HIDAPI (and libusb for that matter).
A wild idea: we provide the API for macOS and Linux hidraw and libusb backend, and also Windows. But for Windows, user may need to provide a template HID report descriptor (based on the dumping from Linux and macOS) for the API to get right. If not, then it is based on the #262 or similar best efforts (not guaranteed to work).
Example: compare the hidpp Linux and Windows code, Windows code has to provide the hard coded HID Report Descriptors.
https://github.com/cvuchener/hidpp/blob/master/src/libhidpp/hid/RawDevice_windows.cpp (native Windows HID API).
https://github.com/cvuchener/hidpp/blob/master/src/libhidpp/hid/RawDevice_linux.cpp (hidraw)
Issues mentioned in hidpp
https://github.com/cvuchener/hidpp/issues/12
https://github.com/cvuchener/hidpp/pull/14
From hidpp author cvuchener
I would like to add a function
int HID_API_EXPORT HID_API_CALL hid_get_report_descriptor(hid_device *dev, unsigned char *data, size_t length);which returns the real report descriptor for libusb/hidraw/MacOS and the reconstructed Report Descriptor on Windows.
For the Mixxx case only the machine readable Report Descriptor is needed, no text representation. But if somebody else needs such a text representation, he could implement it on top of this new API.
The function name sounds good to me.
@mcuee regarding Chromium HID descriptor reconstruction code. You can look at real
HIDP_PREPARSED_DATAin leaked Windows XP source code. Seehidparse.h.But keep in mind that exact struct could be changed since WinXp and that struct was hidden for a reason:
PS: Also there is code for kernel debugger command !hidkd.hidppd.
@DJm00n Thanks for the info.
@DJm00n I looked at the code from https://github.com/DJm00n/RawInputDemo/tree/hid_report_descriptor_dump and I kind of know why it does not work for my device.
Ref: https://community.osr.com/discussion/287808/ioctl-usb-get-descriptor-from-node-connection
For a HID report descriptor, the Recipient bits of bmRequestType must be set to Interface (1).
IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTION sets bmRequestType to 0x80.
https://github.com/DJm00n/RawInputDemo/blob/hid_report_descriptor_dump/RawInputLib/RawInputDevice.cpp#L308
Apparently 0x80 does not work for my device. The FW checks the request is sent to the HID interface number.
BTW, I was puzzled why hidapi was not able to find my Microchip PICkit 2 programmer. Then I remember that many years ago (in May 2008) I submitted a trivial kernel patch (released in 2.6.26) to blacklist the device from kernel HID driver.
usbhid-dump does not care as it is based on libusb.
mcuee@rpi400Ubuntu:~/build/libusb/hidapi$ usbhid-dump -d 04d8:0033
001:017:000:DESCRIPTOR 1624363237.908977
06 00 FF 09 01 A1 01 19 01 29 40 15 00 26 FF 00
75 08 95 40 81 02 19 01 29 40 91 02 C0
@mcuee
I added that comment because I aware of that peculiarity of IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTION.
It is documented here: https://docs.microsoft.com/windows-hardware/drivers/ddi/usbioctl/ns-usbioctl-_usb_descriptor_request
But on my Windows 10 1909 PC it really works and I can dump HID Report Descriptors.
It may be that this Windows version just caching that descriptors (they are get requested by HIDClass driver after device connection). Because didn't saw these request via Wireshark (I captured USB traffic with USBPcap) while running my program. Just a speculation.
Not so sure if we can get some ideas from here too. But it is using Windows API so probably it has the same limitation.
https://github.com/rene-aguirre/pywinusb/tree/master/pywinusb/hid
pywinusb uses
HidD_GetPhysicalDescriptor(the only routine I've found inside, that is closely related to HID descriptor)I believe someone tried to use this function with no luck so far
The HID Physical Descriptor is something different. It's a seldom used optional data structure of the HID protocol, which provides information about the specific part or parts of the human body that are activating a control. In case of a race car control, this would be "Both hands" for the steering wheel, and "Right hand" for the gear shift stick.
pyusb uses standard Win32 API like HidD_GetPreparsedData(), HidP_GetCaps(), HidP_GetButtonCaps(), HidP_GetValueCaps(). So I guess it has the same limitation as the others.
https://github.com/rene-aguirre/pywinusb/blob/master/pywinusb/hid/core.py
https://github.com/rene-aguirre/pywinusb/blob/master/pywinusb/hid/usage_pages.py
The only thing I am not so sure is that if it can use the self.__input_report_templates to have the user give the known HID Report Descriptors if the user knows (for example from macOS or Linux).
Basically my idea is that a new API can do this.
The hidapi can use the API to decide the report ID, report size and report count, and the data can be used to sort out issues like #210.
Interestingly, just notice that libusb xusb example has the following codes. I do not have the HID test device in xusb.c so I had never tested this. I doubt it would work for Windows HID driver.
https://github.com/libusb/libusb/blob/master/examples/xusb.c#L616
Control transfer codes for the libusb Windows HID backend.
https://github.com/libusb/libusb/blob/master/libusb/os/windows_winusb.c#L3957
Okay here is the history of libusb HID Windows backend and xusb example.
https://github.com/libusb/libusb/blob/master/libusb/os/windows_winusb.c#L3452
https://github.com/libusb/libusb/blob/master/examples/xusb.c
The xusb example hid report reconstruction code was actually based on the libusb-win32 1.0 branch.
https://github.com/mcuee/libusb-win32/blob/master/libusb1/src/dll/usbi_backend_hid.c#L287
From Pete Batard's email dated 6-April-2010.
https://marc.info/?l=libusb-devel&m=127050528618692&w=2
Quick hack to have libusb xusb example to dump the HID report descriptor for HID device.
https://github.com/libusb/libusb/blob/master/examples/xusb.c
Run log under Windows with the generic Lakeview Research HID demo mod
HID report parser output: the output is actually pretty good and correct.
Mod xusb output for Microchip PICKit 2.
HID report parser output. Again, it is actually correct.
So it seems Pete has done some work on libusb Windows backend and xusb example to dump the HID report descriptor, and the codes seem to do reasonably well for my tested device (because he may have the same PICKit 2 as I). But I think it may not cover other devices and may not be generic.
libusb Windows HID backend does not support multiple HID top collections. So there is a severe limitation. For example, for the Logitech Unifying receiver, it can only dump of the top level collections.
HID report parser output
Comparing raw descriptors is not a good idea. I did it because I was too lazy to write a descriptor parser at the time and it stayed much too long. It created issues with equivalent descriptors that have some items swapped (not changing the meaning of the whole descriptor). So I had to compare with two different strings and it became an even bigger issue when more than two alternative became possible.
I finally implemented what I should have done long ago: parsing the raw descriptor and checking the high-level data. Which lead to the Windows issue. I opted to use the reverse-engineered preparsed data I found in a link around here. Unlike the public API, "values" and "buttons" are not separated (so "buttons" have their report size/count), you have the exact position of the field in the report, which makes it easier to reconstruct the report descriptor. It enabled me to build a very minimal report descriptor that was enough for my specific use case, but may not be for hidapi (I only store flags, usages, report size and count for each field).
I did not look much at the other data members in the preparsed data except for logical minimum/maximum and there may be issues with them. Logical minimum/maximum are some times swapped or 1 may be replaced by 0 (e.g. (1, 3) becomes (3, 0)). I have no idea what is causing those changes.
@cvuchener Just wondering where are your codes using the reverse-engineered praparsed data. Thanks.
It's not merged yet:
github.com/cvuchener/hidpp@e6e9b301b7/src/libhidpp/hid/RawDevice_windows.cpp@Youw Could you please update the branch https://github.com/libusb/hidapi/tree/get-descriptor to the latest state of the master branch.
done
From #262 test result, it can be seen that getting Windows right is not easy with the native HID driver. But using libusb (with WinUSB driver) it is easy to dump the HID report descriptor, that is anyway expected (same as the libusb backend for HIDAPI).
Ref: I have tested the method (using WinUSB driver to replace the native HID driver) with Logitech USB Receiver and it matches the dump from macOS USB prober.
https://github.com/libusb/hidapi/pull/262#issuecomment-878127839
Of course this is not the solution for HIDAPI, just a easy method for me to test without using macOS/Linux.
@mcuee easier method is to use wireshark tool with usbpcap installed - then you can see and dump usb and bluetooth packets that are 100% flying by the wire.
This is way I used to dump all those HID report descriptors: https://github.com/DJm00n/ControllersInfo
@DJm00n Thanks for the pointer. Unfortunately usb-pcap does not for work for my Windows 10 PC (hangs during installation and all my the USB port will not work) so I can not use wireshark.
Interesting I found one old Cypress Control Center application which also has some HID report descriptor parse function. I have not looked into the codes in details yet. I was actually playing with the EZ-USB FX2LP example FW which has nothing to do with HID.
AN4053: it has the C# source code for the Control Center application.
https://www.cypress.com/documentation/application-notes/an4053-streaming-data-through-isochronous-or-bulk-endpoints-ez-usb
It is also part of the SuiteUSB package from Cypress.
https://www.cypress.com/documentation/software-and-drivers/suiteusb-34-usb-development-tools-visual-studio
Output from Cypress Control Center for FX2LP (not the new one for FX3).
For the Logitech Receiver, it will display as many devices in the tree view.
I will compare with the macOS USB Prober output mentioned here.
https://github.com/libusb/hidapi/issues/249#issuecomment-864510041
For the first device.
click to expand for the output
Second device.
click to expand for the output
Third device.
click to expand for the output
4th device.
click to expand for the output
5th device.
click to expand for the output
6th device.
click to expand for the output
This looks like a dump of what WIN32 API returns, but it does not contain information about the bit positions. This does not help.
For Microchip PICkit 2 it reports the following.
click to expand for the output
Looks like you are right.
Test result for the macOS implementation.
https://github.com/libusb/hidapi/pull/303 (now merged here: https://github.com/libusb/hidapi/tree/get-descriptor)
Mostly it is okay but there is a bit strange output for Logitech USB Receiver.
@Youw Please take a look at the results in #303 to see if it is a problem or not. Thanks.
Just tested #303 with a bluetooth mouse (Logitech M557) and it works fine as well under macOS (Mac Mini M1).
I created #306 (a variant of #262) but using the reverse-engineered praparsed data struct approach. This seems to cover the remaining edge cases.
For Windows #306 seems to be much better than #262 with the first quick check. It seems to be working pretty well.
For Linux hidraw, it seems to me Alan Ott (orginal developer of hidapi) has the codes.
https://github.com/torvalds/linux/blob/master/samples/hidraw/hid-example.c
@Youw It seems that the macOS codes can not print out the HID report descriptor from my Microsoft Xbox One S Wireless Controller (bluetooth connection). On the other hand, mac-hid-dump has no issues.
click to expand
@mcuee, could you try to modify this line?
github.com/libusb/hidapi@26d45bf427/mac/hid.c (L794)Make it:
and let see if it helps
@Youw No, this does not seem to be good enough.
click to expand
interesting
need to debug
hid_open_path- on which step does it fail, and with what error@youw Any pointer of debug messages to print out? So you think the issue is not the added codes
github.com/libusb/hidapi@0db25f9132but with existing hid_open_path codes for macOS?Okay, here is the git diff I used to troubleshoot the issue.
Here is the output.
click to expand
@Youw So basically the Logitech device does not show proper path. mac_hid_dump (https://github.com/todbot/mac-hid-dump/blob/main/mac-hid-dump.c) does not use the path so it does not have such issue.
@Youw Okay, I figured out the issue, the issue is not with your codes, but rather #236. I have two external hubs (USB 3.0 --> USB 2.0) so the path is too long. My External Hubs are more complicated hubs (nested hub).
Once I move the Logitech receiver to the root hub, it will mostly succeed. If I move it to the Port 7 of the first external USB 3.0 hub , it is still not working though.
If I moved it to the Port 1 of the external USB hub, it will mostly succeed.
If I move it to the root hub, almost everything will succeed, except one.
Full output once I move the receiver to the root hub.
click to expand
GJ finding that out. So it works from the ROOT hub with the original implementation (with
kIOHIDOptionsTypeSeizeDevice)?Yes that is correct, except one device as mentioned above. I tend to think it may still be due to #236.
https://github.com/libusb/hidapi/pull/322 has been merged. So I apply the two patches from the macOS get-descriptor branch and here is the result. The way I apply the patch may not be so correct but the patching is successful.
Only one output is a bit strange.
@Youw The issue goes away once I run with sudo. Not so sure why.
Not sure either.
Does
IOHIDDeviceOpen(dev->device_handle, kIOHIDOptionsTypeNone);makes any difference?Indeed that makes the difference. With the change, no need to use sudo.
That means that there is another application/daemon running in the background that has that device opened.
Just the latest test result from the get-descriptor branch under Windows.
Click for the run log
Can I somehow help to get the
get_descriptorbranch merged?Get me fired from my current/main job :D
Actually you can:
One of my biggest conserns is
rd_item_byte. A classical linked list of a 1-byte items is one of the least efficient data structures possible.Right now the
byte_listis defined as:struct rd_item_byte *byte_list = NULL;.A much better aproach would be to have something like:
in this case we still going to have convenient
rd_append/write...functions, but no dynamic allocations per-byte.My second consern is the implementation of
hid_winapi_descriptor_reconstruct_pp_data.As a human being I refuse to read/understand/check it line-by-line, and because it was written by enother human being it may contain accidental issues.
So I think we will have it mostly as is (except my consern above), but run it thru some static analyzers (we have setup project for coverity, and run unit-tests we have with ASAN enabled (maybe even have it enabled for our CI).
If no issues found (or after all issues fixed) - that will be the last item in my checklist for that branch.
I will look, how I can adapt the proposed rd_buffer approach!
To have static code checkers in CI, makes sense to me. I wonder, how it will handle the preparsed_data itself, where the memory allocation happens in WIN32 API code and only a pointer to the struct of variable size is returned. Everything else is under our control, and therefore, should be fixable.
Just another test results.
run log for hidtest
One more test results with latest commit.
run log for hidtest
Comparison with libusbK kList utility for the Mi Wireless Mouse USB Receiver, by attaching libusb0.sys filter to the device.
Great that this PR has finally been merged.