WIP: Implemented Windows Report-Descriptor Reconstructor #262
No reviewers
Labels
No labels
3rdparty
Android
API
Author does not respond
bsd
bug
build system/CI
ci-virtual-device
CMake
Contributions Welcome
Core
dependencies
device-specific
documentation
don't_merge
duplicate
enhancement
Example
Haiku
hidraw
hotplug
info required
invalid
libusb
macOS
Misc
Not an issue
question
Release
Test
Test GUI
Windows
wontfix
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
gh-libusb/hidapi!262
Loading…
Reference in a new issue
No description provided.
Delete branch "windows_report_descriptor_reconstructor"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is implements code to reconstruct a HID report descriptor using from the information provide by the Windows API.
The code contains debug printout and it writes a text file of the reconstructed report descriptor bytes into one text file per HID device, which can be read by visualization tools like: https://eleccelerator.com/usbdescreqparser/
In most cases the generated report descriptor is logical identical to the original, which I read using the free version of USBlyszer. But it's not byte wise identical. The order of the bytes differ, handling of reundant information differs and padding bits can be defined in a different way.
There is one real limitation, where i couldn't find a way to reconstruct the correct logical meaning:
If multiple button bit field have the Usage 0, it seems impossible to determine the bit positions, because in case of zero, API functions like HidP_SetUsageValueArray do not set any bit.
There is one solveable bug the determition Report Count for ValueArrays with UsageMin...UsageMax specified. This special case is handled by win-hid-dump (
github.com/IntergatedCircuits/HidSharp@e8359220e0/HidSharp/Platform/Windows/WinHidDevice.ReportDescriptorReconstructor.cs (L161)) correct but not in this PR (github.com/libusb/hidapi@b54a041cad/windows/hid.c (L871))But the win-hid-dump implementation proves, that this is solveable based on the data provided by WIN32 API
Lets have this against feature branch: https://github.com/libusb/hidapi/tree/get-descriptor
When we have implementation for all platforms - we will merge to master
Link for reference: https://github.com/IntergatedCircuits/HidSharp/blob/master/HidSharp/Platform/Windows/WinHidDevice.ReportDescriptorReconstructor.cs
@JoergAtGithub After several attempts I finally managed how to get USB HID Report Descriptors from device!
You can do it via specially crafted
IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTIONcall on parent USB HUB device.See my fork of usbview tool: https://github.com/DJm00n/Windows-driver-samples/tree/usb-hid-descriptor
Until now, I believed this is impossible for a user space application. Fantastic!
To get the original report descriptor from the device is clearly better than any reconstruction approach!
I just tried your code on Win7 VS2019, but got just this error message:
===>HID Descriptor<===
@JoergAtGithub can you paste whole output for this device?
[Port2] : USB-Eingabegerät
I get this message for all HID devices I've, except the keyboard. There I get a bytes, but they are not a report descriptor.
I cannot get descriptor my mouse too. Seems that is because of active boot protocol on it:
I can get descriptors on my keyboard and DualShock4 .
Also there seems some memleak in code. I’ll try to debug further.
Its sad but I don't know why it is working with my DualShock4.
According to docs of
IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTIONUSB stack ignores the value ofbmRequestand inserts a value of0x80(Endpoint_In). But to do proper HID Report Descriptor request over USB I need to do such request with0x81(Interface_In) value to do request on corresponding Interface.Seems there is no any way to do HID Report Descriptor request from userspace properly...
Also I found another descriptor reconstruction code in Chromium:
https://bugs.chromium.org/p/chromium/issues/detail?id=1009717
github.com/chromium/chromium@73fdaaf605They reverse-engineered
HIDP_PREPARSED_DATAstruct definingPreparsedDataHeaderandPreparsedDataItemstructs and trying to output Report Descriptor from it. SeeCreateHidCollectionInfo(). Maybe could be useful.Sad, that IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTION does not read the real ReportDescriptor for all devices.
The reverse-engineered HIDP_PREPARSED_DATA struct from Chromium might be helpfull to:
Did you try the reconstructor from this PR? Did it work for your devices?
That assumes the HID device is a USB device though. Couldn't it be Bluetooth?
Not for libusb backend, of course.Right, but this PR is for the Windows backend.
This PR doesn't use IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTION and should also work for Bluetooth HID
Ah, pardon me, mentally I was still thinking about the other issue we discussed today.
@JoergAtGithub hi!
I have rewritten my code that is trying to get HID Report Descriptor from USB devices. Can you give it a try?
https://github.com/DJm00n/RawInputDemo/tree/hid_report_descriptor_dump
It successfully dumps descriptor from my keyboard, dualshock4, dualsense, stadia controller etc into debug output in MSVS. Now without memleaks and crashes :)
Cannot dump descriptor from my mouse because it supports only Boot Interface Subclass (see 4.2 Subclass in HID spec).
It uses
IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTIONunder the hood.PS: Bluetooth and Bluetooth LE another story that I'll try to investigate.
@DJm00n I should be able to test this over the weekend.
I have tried HIDSharp and win-hid-dump (based on HIDSharp) as mentioned in #249.
Why you don't test the code of this PR? This PR uses the the same principle as HIDSharp, but implements much more report constructor elements, incl. the complete reconstruction of the collection structure and correct size of multi-byte items.
I will answer you in the related issue: https://github.com/libusb/hidapi/issues/249
Let's use the comment section here only for the review of the code of this PR.
@JoergAtGithub I agree that we should use #249 for testing or ideas of different implementation methods, and we only talk about this PR here.
I should be able to test this pull request soon. I would prefer a demo or test program for testing and not debug print though.
@JoergAtGithub This pull request seems to work fine.
I build the following branch using VS2019
https://github.com/JoergAtGithub/hidapi/tree/windows_report_descriptor_reconstructor
Here is the run-log.
click to expand
It generates many txt files as seen in the run-log file mentioned above. There is a difficulty to correlate the txt file with the device.
I have the Microchip PICkit 2 with simple HID report. It is not that correct.
Original HID report descriptor
Dumped by this pull request [Edit: fixed]
https://eleccelerator.com/usbdescreqparser/ output.
As a comparison, this is what I get from HIDSharp which is also not correct.
Thanks for testing!
Are you sure, that the data in the last post are from the same device? It looks completly different.
Oops, you are right. That is the difficulty I am facing. I will change that one. After fixing that, it is not that correct to me.
I search the run-log file and this is the closest and it does not match the FW source codes.
Full USB Descriptor C source code for reference.
click to expand
@JoergAtGithub Based on the comments in #249 , the output from this pull request is actually not bad. REPORT_ID (0), REPORT_COUNT (1), and REPORT_SIZE (8) are actually correct.
@JoergAtGithub Based on this new understanding, I will say this pull request actually works pretty well.
Another test device (my mod HID FW)
Debug log from this pull request.
Original FW source code.
Raw data captured from this pull request.
Parser output
An approach to test this PR with third party HID devices is, to read the real Report Descriptor on Linux or MacOS, and compare it with the output of this reconstructor code.
Is it possible to print the device VID:PID in the debug log so that it is easy to correlate the hid report descriptor dump text file with the HID device?
Done!
Now it is much easier for me to correlate the device. Thanks.
Debug output from #262, and I can confirm it is correct for my modded Lakeview Research Generic HID device.
Original FW codes.
Equivalent FW codes:
Reformat of the raw debug text file
Test result for Microchip PICkit 2. There seems to be a problem with report count reporting from #262.
It seems to me the FW source codes may be a bit confusing. Let me check if the source code matches the device.
Original FW source code
FW hex file
From FW hex file
HID report parser output: https://eleccelerator.com/usbdescreqparser/
FYI HIDSharp is totally wrong with Microchip PICkit 2.
There is a problem with the Microchip Simple HID demo as well. Report size is not correct.
FW source codes:
Again HIDSharp is worse than #262.
It seems to me that HID report can be of many different variations and it is not easy to get right if you do not know the HID report format.
But if you know the device, then you can get it right. That is probably the initial thought of Alan Ott.
1.)
The "Test result for Microchip PICkit 2. There seems to be a problem with report count reporting" is the second issue described in the description of this PR. I'm pretty sure, that I can fix this case, because in these case (value array(report_size > 1) with Usage-Range(Usagemin/Max)):
If I adapt the code from HidSharp correct into #262, I can get both.
2.)
The second case is more difficult, because WIN32 API seems to detect it wrong (button instead of value). But I hope I can solve this too, because:
If I adapt the code from HidSharp correct into #262, I can get ReportCount and Report Size too.
This is all related to the code branch I marked as Experimental, because I don't have a device with Usage-Range:
github.com/libusb/hidapi@b54a041cad/windows/hid.c (L871)I will provide an update tonight, which hopefully fixes these two cases.
That will be great. I can test soon after you have the fixes.
The problem is slightly different than I though, I will need another day to provide a proper fix
No issues. Take your time. This is a difficult task and so far I will say this pull request produces the best results for me under Windows.
The last commit doesn't contain changes in the reconstructor code itself, it just moves the call of it into the hid_get_report_descriptor function introduced in #303 .
Latest test shows no more txt files.
click to expand for the debug log using hidtest
The debug log is a bit strange for the Dell keyboard.
But the https://eleccelerator.com/usbdescreqparser output looks better.
Could you provide the original Report Descriptor of the Delll device as well. Read on another platform - BTW this PR should now work with MacOS as well, because I merged the code from #303 in.
Yes I will get the Report Descriptor from macOS or Linux.
HIDSharp Output for the Dell USB Keyboard.
I use a brutal force method to get the Windows HID descriptor by changing the Windows HID driver to WinUSB. Then I use libusb xusb test to extract the HID report descriptor. I believe this value will be correct.
https://github.com/libusb/libusb/blob/master/examples/xusb.c#L616
Minor hack to use the "-s"
Now this is what I get.
https://eleccelerator.com/usbdescreqparser/ output
I can confirm the brutal method will work.
Here is the macOS dump of the Logitech USB receiver (Interface 0)
Here is the brutal method (using WinUSB driver and "xusb -s".
Brutal force method for the Dell Optical Mouse. Looks like #262 is not so correct.
https://eleccelerator.com/usbdescreqparser/ output
#262 output:
No, this is the intended behavior. The three extra items:
are const padding bit's to fill up the number of input report bits to full bytes. If these bits are defined explicitly in the report descriptor or not can only guessed by a reconstructor. But this doesn't change the logical meaning.
Thanks for the explanation for the padding. Still all in all I think this approach is difficult. Maybe you can take a look at the reverse engineered hid_preparsed_data idea mentioned here https://github.com/libusb/hidapi/issues/249#issuecomment-873029494.
Yes, I also think, that the reengineering approach has the potential for a more accurate reconstruction. It can hopefully fix all the functional relevant bit position issues, but for non-functional items, like the constant padding-bits at the report end, the reconstruction will still remain on assumptions.
I will start as second PR, based on the #262 reconstructor, bút using the reengineering approach to gather the data. Than we can compare both and decide which should be used.
That will be great. Now that I figured out a way to dump the correct USB HID device descriptors under Windows, it is pretty easy for me to carry out the tests under Windows without relying on Linux/macOS.
I hope I could fix this issue in the latest commit, could you please test this device again. Does it now report the correct ReportCount of 64?
Could you also provide the output for the Dell keyboard of this version. I added some debug output, which hopefully helps to understand the root cause for this issue.
This is the result of Microchip PICkit 2 with latest git commit
github.com/JoergAtGithub/hidapi@9bd708ae6e.#262 does not get the report count 64 correct.
Original source code
Output from xusb (with Windows HID driver)
This is actually basically correct.
Using xusb and WinUSB driver (which will get the exact report descriptor)
Another device of interests, Dell USB Mouse, #262 got the exact report correctly.
xusb using WinUSB driver
Parser output
#262 output
One more device: Logitech Dual Action Gamepad
xusb output with Windows HID driver, which is obviously wrong.
xusb output with WinUSB driver
Parser output
#262 output
Could you please provide the output with the Dell Keyboard. I need the debug output for this device.
I do not have the keyboard with me now so you have to wait for a few days when I will get access to that one.
Thanks for testing again. The Dell keyboard is of interest, because it has an identical report descriptor to mine keyboard, except of the LogicalMax/UsageMax value, which is 0xFF instead of 0xDD. And mine with 0xDD works. I added some debug output to see what fails in the 0xFF case with Dell.
@mcuee I commited another version to address the Microchip PICkit 2 case. could you please test this device again.
Yes the latest code is good.
@JoergAtGithub
Full log from hidtest.exe. Please take not that "HIDP_STATUS_BUFFER_TOO_SMALL" was triggered for the Logitech USB receiver. It is probably a difficult device to get right due to the complexity.
click to expand for the full run log
Just FYI as well that I also tested with a bluetooth mouse and here is the output from #262. I will need to use my Mac Mini M1 to dump the original HID report descriptor. I do not know how to do it under Windows.
Oops I click the wrong button and accidently closed the pull request so I just reopened it.
macOS output
Parser output:
The following device may be a difficult one as well -- Microsoft Nano Receiver v1.0.
Click to expand for pull request 262 output
This is not a real error, this is the debug output, I added to understand the issue with the Dell keybord.
Output for a Logitech USB Keyboard. Same output as #306 and at least the report ID, report size and count are all correct.
Output for the ESP32 BLE HID device, same problem as #306 that second report is not working.
May I ask if you have access to the Dell keyboard again? It would help to see what the debug output added to the latest #262 shows for this device.
Yes I should have access to the Dell keyboard soon and I will test it out.
This is the output:
#306 looks okay for the Dell keyboard.
Full debug log with the latest git with quite some devices attached, including the Logitech USB Receiver, Dell USB Mouse, Dell USB Keyboard and the Plantronics USB headset.
Click to expand for the full debug log
Latest run log for your reference.
Click to expand for the run log
Am I correct that this one can be closed in favor of #306 ?
I close this PR in favor of #306, which generates a better reconstruction
Pull request closed