HID reports of more than 64 bytes and libusb backend #274
Labels
No labels
3rdparty
Android
API
Author does not respond
bsd
bug
build system/CI
ci-virtual-device
CMake
Contributions Welcome
Core
dependencies
device-specific
documentation
don't_merge
duplicate
enhancement
Example
Haiku
hidraw
hotplug
info required
invalid
libusb
macOS
Misc
Not an issue
question
Release
Test
Test GUI
Windows
wontfix
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
gh-libusb/hidapi#274
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
At Mixxx, users reported problems with USB 2.0 High-Speed devices, which are limited to Full-Speed mode for some reasons. Under MacOS and Windows they operate as High-Speed device.
This is not only a performance limitation, but it has functional implications for the HID communication, because USB Full-Speed is limited to a package size of 64 Bytes instead of 1024 in High-Speed mode.
The result is, that HIDAP returns a 79Byte input report correct on Windows and MacOS, but two reports on Linux (libusb backend), the first with correct report ID, the second with report ID 0x00.
How could a cross platform application using HIDAPI handle this:
Do I understand it correctly that the problem on Linux with libusb backend?
Yes, it's reported by two independend Linux libusb users. I'm myself just relaying this, since I've neither the device nor Linux.
It is hard to define in a first place what is a "misconfiguration". From USB stack POV, if a device is successfully initialized in Full-Speed it is a successfull configuration. Otherwise it wouldn't be configured at all.
How would a client application know what it should expect (unless it is a known device, expected to run in a specific mode)?
And one more time: hidapi is agnostic of the device bus most of the time. On Windows/macOS/hidraw communicationis done using an OS API and there is nothing USB-specific.
It is only negotiated by the OS/USB driver/stack and device itself. The application has no control of this. Each device always tries to initialize itself in a highest possible mode, unless explicitly required otherwise by the device FW/HW limitations.
Again no cross-platform solution is possible. And right now there is no API to get the descriptor. And the USB speed mode is ony easily can be checked by libusb backend.
The device's FW is way mo easily can check/know current USB configuration.
If device's FW can be modified, I'd rather suggest to make an API and ask the device in which USB mode is it running.
Modification of the the device firmware is not possible. The device is a commerical DJ controller, which is sold for use with the commercial DJ software Traktor Pro for Windows and MacOS. The users try to use it on the cross platform DJ software Mixxx under Linux, which is based on HIDAPI.
Mixxx works with this device on MacOS and even on a Windows 10 VM under Linux, but not if under Linux with libusb backend.
What about hidraw backend, as anyone tried it with that device?
If there is no known issues with Windows/macOS, I think it would be feasible to perform an explicit check on Linux using libusb directly, even before trying to open it with hidapi.
hidraw is not yet tried. Further investigation is needed to understand this specific case.
But I think this is a general issue which could also occur with other HID devices, maybe also on other operating systems. Therfore it would be better to detect this somehow as error in hid_open instead of adding a check on application level.
That's what I'm trying to say: a USB/HID device running at Full-Speed - is not an error.
As for the inability to send packets larger than 64 bytes: this looks like an undesirable behavior of a specific device, but that's definitely not a HID issue.
HID protocol is not design to send large packets in a first place. It is specifically designed to send many small packets (2/4/8/16 bytes of data), and that's what is being done by all devices I've seen. I've had one device that has been using 64 bytes data packets, and that was considered an edge case.
For ceses when large data packets needs to be send between host and device, usually different protocols are used, like USB BULK.
After giving it a second though, we can call a case like this a "broken device" or "hardware missconfiguration" which can be detected by libusb backend of hidapi.
If someone is going to develop a patch to detect such a thing for libusb backend, I don't see
strongreasons not to accept it.I have a Native Instruments Traktor Kontrol S4 Mk3, one of the affected devices. Here is the output of
lsusb -tfor the device:The HID component is at 480M speed, so I don't think the bug is in the kernel. The audio interface of the device works fine with Linux.
I don't know if the bug here is in libusb or hidapi's use of libusb. My hunch is the latter but I have no evidence to demonstrate that.
I don't believe libusb (as a result - hidapi too) has any control of how to initialize the device, select speed, etc. USB stack driver is responsible for that.
Do you have the same issue on Application level on your machine with this configuration?
@JoergAtGithub can you ask users who reported this issue to share the output of
lsusb -ttoo?It seems that libusb can only read a device's speed with
libusb_get_device_speed()but has no API to set the speed.Here is the
lsusb -voutput with formatting... thewMaxPacketSize 0x0040 1x 64 bytesfor the HID endpoint is interesting...How do I tell hidapi to use the hidraw backend instead of libusb? I don't see any documentation for this.
./configure --helphas no information. It looks likemakebuilds both hidraw and libusb. If I uninstall thelibusbx-develpackage on Fedora, hidapi fails to build.You can still have both hidraw and libusb backends of hidapi installed and use either one of them.
Technically those are two different libraries (even though with identical public API/ABI).
If you want to switch the backend for you application, you need to link against libhidapi-hidraw instead of libhidapi-libusb.
If you have a prebuilt application, it is already hard linked to one of the backends (presumably libhidapi-libusb in your case), and there is no "official" way to change it.
I hacked the application build system to use hidraw instead of libusb. Now it fails to open the device, both with
hid_open_pathandhid_openwithout any error message from hidapi indicating the problem...Here is the application code.
I'm confused why the manufacturer set the wMaxPacketSize for the HID endpoint to 64 when they made the HID protocol have packets larger than 64 byes. I'm also confused how the packets are not split on macOS and Windows... do those operating systems concatenate the packet fragments in their HID drivers??
@JoergAtGithub what makes you think that the devices are opened as Full-Speed USB 1 devices? Is it only that the packets are split at 64 bytes or is there some other hint?
One of the Linux users reported this:
AFAIK the slowest element in the USB bus chain defines the speed. I assumed, that the HUB limits the speed of the HID endpoint to Full-Speed.
The hub operates at 480 Mb too. Here is
lsusb -vwith a USB 3.0 drive plugged into the USB-A port on the device:I don't think the USB speed is the problem here. I think it is the wMaxPacketSize of 64.
I wouldn't be suprised.
Does the HID specification say to concatenate the packet fragments in this case with wMaxPacketSize smaller than the report size? I wouldn't be surprised if this is an edge case where the specification does not say what to do... I'm still at a loss why the manufacturer would create this situation.
The HID specification has this to say about wMaxPacketSize:
I'm unclear exactly what that means but it seems that it permits wMaxPacketSize to be shorter than the report size.
This would make some sense. libusb by itself has no awareness that the split USB packets should be concatenated for the HID protocol. However hidapi can determine this.
hidapi would need to parse the report descriptor to determine this, so I guess #249 is a prerequisite for this?
Isn't the report_descriptor already available internal in the libusb backend:
github.com/libusb/hidapi@b72a367555/libusb/hid.c (L643)Yes, getting the report descriptor is there. Parsing it is another task. hidapi would need to compare the size of the reports described in the report descriptor to wMaxPacketSize. If any reports are bigger than wMaxPacketSize, keep incoming packets that are equal in length to wMaxPacketSize in a buffer, then concatenate the buffer with the short part when the incoming packet length is equal to the long report length minus wMaxPacketSize.
The quick and ugly hack around this is for the application to concatenate the packets for devices know to be affected by this, but that means the application requires platform-specific code to use a cross-platform library. :/
Advanced parsing and packet (re-)construction/verification was never the intention of hidapi. It always tried to be the thinest proxy library possible, with remarks to keeping the API uniform across platforms.
In defence of the application side workaround for a specific device - I believe the reconstruction code can be written uniformly for all platforms, it just that some of the branches of the code will be executed when run on linux only.
This issue makes the API not uniform across platforms.
This could not be done in a generic way for any device because that would require hidapi to expose the wMaxPacketSize -- or possibly directly invoke libusb to get that, which kinda defeats the point of using hidapi. For the few devices known to be affected by this for Mixxx, it isn't hard to hack around this in the device-specific JavaScript code that maps the HID I/O to application logic.
I don't think, that it's neccessary to have wMaxPacketSize to implement this. The only information you need is, the size of each report by ReportID. And this information can be gathered from the report descriptor.
With this information you could check if the report is complete, or if another data packet needs do be received.
I got the hidraw backend working by adding a new udev rule. The hidraw backend does not split the big report. The application receives a single 79 byte report like on Windows and macOS. I did not realize the libusb and hidraw backends require different udev rules. I think linking to the hidraw backend would be an acceptable workaround.
That would require hidapi (or the application) to do some hacky guesswork. If the incoming packet is not equal to the size of a known report from the report descriptor, store it in a buffer. Then, when another incoming packet is another size that isn't in the size of a known report descriptor, concatenate it with the buffer.
Not any packet of unknown size - you have to read the next packet until the report size is reached, and than you return the concatenated report to the user.
Okay, that seems considerably simpler to implement.
Since you proved, that the hidraw Linux kernel module handles this case correct, there is at least one working OpenSource implementation available, which can be taken as reference to fix HIDAPIs libusb backend implementation.
Here is Linux's hidraw driver's implementation of hidraw_read.
Don't see anything regarding concatination there, even remotely.
I guess the concatenation happens at a lower layer of the kernel, presumably somewhere in drivers/hid/hid-core.c. The common entry point of the different wire protocols is hid_input_report. Or maybe it happens somewhere in drivers/hid/usbhid/hid-core.c.
It is the kernel driver's job. libusb is a user mode library.
Ref: https://community.arm.com/developer/tools-software/tools/f/keil-forum/26534/small-question-about-hid-usb-device
wMaxPacketSize of the interrupt IN (or OUT) endpoint determines the division unit of the report.
For full-speed device, wMaxPacketSize is limited to less than or equal to 64 bytes.
For low-speed, it's 8 bytes.
When the report size is greater than wMaxPacketSize, the report is split into packets of wMaxPacketSize. For example, 128 bytes report is divided into two 64 bytes packets. And then, these packets are exchanged one packet by one over the interrupt endpoint.
I'm not sure which HID example you are starting on.
In most examples, you have to code it so that above division works.
bMaxPacketSize of the default endpoint (EP0) also determines the division unit of the transfer.
For HID reports, this value affects to the report exchange using Get_Report and Set_Report() requests.
Usually in most examples, the USB firmware stack cares it for you.
Tsuneo
Technically libusb backend should not need to worry about individual packet but rather the transfer. The transfer size should be of the report size. The data on the wire is based on packet, but then the transfer can split into multiple packets.
This is probably what happens under the hidraw backend, as well as the Windows and macOS backend using OS native HID driver. I am not so sure why the hidapi libusb backend is different in this case.
Most of the HID device will have the wMaxPacketSize as 8 bytes (low speed) or 64 bytes (full speed). I remember seeing old reports that Windows default HID driver (at least for older version of Windows) does not support high speed interrupt endpoints. That does limit the speed of HID device. However, it does not limit the report size.
https://github.com/libusb/libusb/blob/master/libusb/os/windows_winusb.c#L3728
static int hid_open(int sub_api, struct libusb_device_handle *dev_handle)
The libusb Windows HID backend has some codes to figure out the report size.
Same way it is done by hidapi on Windows backend. The problem here is on Linux with libusb backend for hidapi
For Linux libusb backend of HIDAPI, can we do the same thing? Figure out the report size, and then use Control Transfer or Interrupt transfer to request for the report size. Then the Linux usbfs driver will handle the split of packet.
Am I missing something obvious?
I believe we can parse hid descriptor (which we already doing for some other purposes) and find out the largest report size, and use it as a length parameter passed to
libusb_fill_interrupt_transfer.Now some questions:
Because in all cases other than libusb - the backend is an OS/Driver implementation. hidapi just uses it thru an API.
Libusb backend is basically an alternative implementation of such "driver" written by Alan, Ludovic and a few other folks from the community. libusb backend is the only backend that implements a low-level USB/HID protocol.
Yes I agree that we are implementing the USB/HID protocol with libusb, however, I am not so convinced about the packet splitting thingy.
I believe the answers are.
But I actually do not know the internal of libusb very well. I will ask in the libusb mailing list.
That would be the most desirable behavior from this issue/feature POV.
I have asked the question in libusb-devel mailing list for confirmation. Hopefully experts there can check whether my answers are correct or not.
Here is the reply from Tim Roberts, the expert in USB, and a top Windows Driver expert in the world.
For INPUT report (and also Feature IN report) the report is sent by the device.
On Tue, Jun 15, 2021 at 2:29 AM Tim Roberts wrote:
Further clarification for OUTPUT report.
On Wed, Jun 16, 2021 at 12:30 AM Tim Roberts wrote:
BTW, it is not that clear whether Windows has good support for high speed HID device. This is not related to this issue, just FYI only.
This is my question in OSR NTDEV forum and so far no answers.
https://community.osr.com/discussion/292942/speed-of-high-speed-usb-hid-device
Just for reference:
USB HID spec 1.11 specifies this behaviour in chapter 8.4:
That's some piece of work.
The recent merge of #451 in git master should help, right?
I don't think #451 is actually needed for this particual one. Inside of the LIBUSB backend a Report Descriptor was always available (even though, the function for it was partially broken until #444).
But again: parse a descriptor, figure an each report size, have a "smart" read function, which will analyze the USB packet size, and concatenate the data when the report is larger than USB packet size - that is huge amount of work, and I don't see any volunteers.
I think we can promote the hidraw backend for Linux and tell the user that libusb backend has various issues including this one (even go to the extreme to say libusb backend is only for legacy Linux versions which has issues with hidraw). In that case, this becomes a non-issue for Linux, macOS and Windows users.
Then the only users who are affected are FreeBSD users (hidapi is not really useful under NetBSD/OpenBSD because the user basically needs to rebuild the kernel to detach the kernel hid driver).
@Youw
The other thing is to provide an advice for application developers using the libusb backend to work-around the issues. That may help as well.
@cederom
Just wondering if you encounter such HID devices under FreeBSD. Thanks.
@hselasky
Sorry to ping you as well. Do you think there is a way you can provide an insight whether FreeBSD libusb can provide some help here for hidapi users under FreeBSD. Thanks.
The reason I ask, is that I remember FreeBSD creates a ugen devce along with uhid so that HIDAPI can work under FreeBSD using the libusb backend. And FreeBSD libusb implementation is different from main libusb project.
Hey there :-) What / How can I test?
Are you sure this is not a problem of a cable / hub / bios settings / faulty device firmware?
For my USB HID mouse/keyboard all uses FullSpeed (12M):
Just a quick glance at different devel kits some of them choose HighSpeed (480M) some FullSpeed (12M) it depends on the attached device.. and from what I remember this is up to device to select speed it wants based on the electrical pull-ups-downs on the bus and on the protocol.
For my KT-LINK it looks like this:
Some Infineon devkit:
ST-LINK V2:
nRF52840PDK:
@cederom
Ignore the USB speed part. Rather the issue is with USB HID device with HID report size greater than wMaxPacketSize of 64Bytes. Do you have such USB HID device?
No clue sorry :-) For testing it may be good to create some sort of firmware on a cheap device like nRF52840-Dongle with USB Device block.. maybe BadUsb.. in order to be able to generate various situations on the controllable device firmware side?
@mcuee: USB devices usually indicate end of data by sending a packet having size less than wMaxPacketSize.
There is a flag to allow short terminated packets. USB is packet oriented. When you setup a libusb transfer to receive 2x wMaxPacketSize, if any of those two packets are short, then that libusb transfer will complete.
The clear stall command may be used to sync/reset the stream, but many vendors don't care about implementing it.
As a general mechanism I would read wMaxPacketSize at a time. Then at each completion, you check the length of the content (this is given by the HID payload). If the payload is short terminated or you have enough data, then you proceed. This will have to be repeated every wMaxPaketSize bytes.
--HPS
Of course, a device only sending wMaxPacketSize chunks of data may end up out of sync!
From what I see, there is a work-around from the application side -- the application has to know the HID device and know the max HID report size.
There is no guessing work needed from hidapi or the application. The application developer has to know the device's each report size in the end.
@Youw
I think we should maintain this way and not to make HIDAPI too complicated.
In this case, I think we can treat this issue as a documentation issue -- hidapi libusb backend will have such limitations and application will have to develop their own workaround if they encounter devices like this.
@hselasky
Do you think that FreeBSD can develop its own hidapi compatible library and not use the hidapi libusb backend? License is not the issue here but because the libusb backend limitation.
@mcuee : FreeBSD's libusb already has a backend abstraction:
https://github.com/freebsd/freebsd-src/blob/main/lib/libusb/libusb20_ugen20.c
What you could do is to enumerate all the FreeBSD HIDRAW devices and use some generic HID device configuration and endpoint profile, and emulate a USB device over other services in FreeBSD. Is that what you are asking?
--HPS
Haha, what I am asking is a FreeBSD specific backend (eg: using hidraw) for HIDAPI so that FreeBSD HIDAPI users no longer need to use the libusb backend. The libusb backend for FreeBSD was a hack to get hidapi to work under FreeBSD as uhid can not be used.
I am not so sure if there are such hidraw driver for OpenBSD/NetBSD now.
Reference:
https://reviews.freebsd.org/D27992
In the long term, I think the libusb backend will become not important at all and can be deprecated.
Try asking: wulf@FreeBSD.org
@wulf7
Just wondering if you are interested in this enhancement. It should benefit FreeBSD HIDAPI users.
I already added hidraw enumeration support to https://github.com/wulf7/libudev-devd/
So, most probably, addition of linux hidapi backend to build is only thing left.
And exactly this is not given here: It's not uniform accross platforms!
Using the libusb backend requires different code in the cross-platform applications build on top of hidapi. In case of Mixxx, we have mappings for over 100 devices, and it's very unlikely, that a mapping developer uses more than one operating system.
Different to the other backends, the libusb backend is not only a wrapper to a HID implementation - it's a HID implementation on it's own. Therefore it must contain more functionality than the other backends - and it does!
For example it's allready the only backend, which contains an HID report descriptor parser - used to determine Usage and Usage-Table:
github.com/libusb/hidapi@b516e97048/libusb/hid.c (L270-L273)It would be only a few lines to extend this existing parser to determine the size of each report.
Just wondering why do you need to use libusb backend (other than for FreeBSD users)? I thought using hidraw was good enough under Linux?
Hmm, are you saying it is trival to fix this paticualr issue? I get a different impression from the comment by Youw,
If that is the case, then PR will be welcome.
Yes, we will switch our official Linux builds to hidraw backend in the next major release, due to this issue. (This will be a lot of hazzle for the end users due to changed udev rules.)
But unofficial ports for BSD exist - and the maintainers of these ports do not have all these devices to debug the mappings on their platform.
I said it's easy to determine the size of all reports from the report descriptor. But just knowing the size, does not fix the data transport.
Let's count on core FreeBSD developers like @wulf7 and @hselasky to fix the issue for FreeBSD users, by using the FreeBSD hidraw backend to replace libusb backend for hidapi.
@JoergAtGithub are these device mappings something that could be easily copied and recreated as static set of descriptors uploaded to a (physical/virtual) usb device that could emulate selected hid device or this require whole bunch of functioning devices of various kinds?
Unfortunately not, each consist of several hundred to thousands lines of Javascript code. DJ controllers are fairly complex HID devices - this is why some have such large reports.
@cederom
If you just want to test this issue, I think you can get an HID FW sample code for your preferred USB device, modify the FW to have wMaxPacketSize=64 bytes and the HID report size to be greater than 64Bytes, say 79 bytes to match the first report.
@JoergAtGithub I mean at the USB low level.. at the device descriptors.. hid descriptors and reports.. USB transfers.. I think of sniff-and-replay USB traffic for a given device that could be then replied as part of test automation.. would that make sense?
Indeed, I missunderstood that. Test automation on USB low level wit a virtual device should be possible.
But note that wMaxPacketSize must not always be defined by the HID device itself. The 64 bytes is the limit for USB full speed devices. But if you use a USB low speed hub to connect this USB high speed device, wMaxPacketSize is reduced to 8 bytes, and you will receive chunks of 8 bytes.
64 Bytes is not the limit for High Speed USB Interrupt Endpoint, but rather Full Speed USB Interrupt Endpoint.
However, it is not clear whether Windows even support high speed HID device or not. Please refer to my comment above.
USB 2.0 Spec
https://www.usb.org/document-library/usb-20-specification
5.7.3 Interrupt Transfer Packet Size Constraints
An endpoint for an interrupt pipe specifies the maximum size data payload that it will transmit or receive.
The maximum allowable interrupt data payload size is 64 bytes or less for full-speed. High-speed endpoints
are allowed maximum data payload sizes up to 1024 bytes. A high speed, high bandwidth endpoint
specifies whether it requires two or three transactions per microframe. Low-speed devices are limited to
eight bytes or less maximum data payload size. This maximum applies to the data payloads of the data
packets; i.e., the size specified is for the data field of the packet as defined in Chapter 8, not including other
protocol-required information. The USB does not require that data packets be exactly the maximum size;
i.e., if a data packet is less than the maximum, it does not need to be padded to the maximum size.
From https://community.osr.com/discussion/292942/speed-of-high-speed-usb-hid-device
No, not at all. If (Windows support high speed HID device and) you have a maximum bandwidth endpoint, you'll get
3072 bytes per microframe. That means the 64kB transfer is going to be split up to span 21 microframes, or 3 complete frames. And that would be 24MB/s. If they don't support the multiple transactions, you'd get 1024 bytes per microframe, and the 64kB transfer would span 8 complete frames, at 8MB/s.
From https://learn.microsoft.com/en-us/windows-hardware/drivers/hid/hid-transports
Input/Output/Feature Report Length
The Hidclass/Hidparse driver pair defines lengths of HID Input, Output, and Feature Reports. The limit is 8 KB (minus 1 bit). Even if a HID minidriver can request a transfer of more than 8 KB for a report, only reports smaller than 8 KB are successfully transferred.
As mentioned above, you may want to change "high speed" to "full speed" in your comments. And it would be a good testing method for this issue if you can find a "USB low speed hub". At least I have not encountered one "USB low speed hub" myself since I was playing with USB device (from 2005 to now). I do have one USB full speed hub somewhere.
Yes, wMaxPacketSize may vary, but the principle about short terminated data transfers is the same. That's why it is important to send one byte extra, if the payload is exactly divisible by wMaxpacketSize. Else you risk getting out of sync at some point.
HID spec says that there is an exception for the longest report.
https://www.usb.org/sites/default/files/hid1_11.pdf
8.4 Report Constraints
The following constraints apply to reports and to the report handler:
An item field cannot span more than 4 bytes in a report. For example, a 32-bit item must start on a byte boundary to satisfy this condition.
Only one report is allowed in a single USB transfer.
A report might span one or more USB transactions. For example, an application that has 10-byte reports will span at least two USB transactions in a low-speed device.
All reports except the longest which exceed wMaxPacketSize for the endpoint must terminate with a short packet. The longest report does not require a short packet terminator.
Each top level collection must be an application collection and reports may not span more than one top level collection.
If there are multiple reports in a top level collection then all reports, except the longest, must terminate with a short packet.
A report is always byte-aligned. If required, reports are padded with bits (0) until the next byte boundary is reached.
Some general comments:
Some embedded USB host controllers may have problems carrying such packets. These multi-DATA USB packet payload are sometimes incorrectly done on the wire. I have one USB device only working with EHCI from INTEL, and stops working on XHCI from INTEL. I sniffed the wire and there was a bug there how the PID's (packet type field) was encoded. I don't remember the exact details, but it was not according to spec, and some old USB HC's allowed that and so the product was shipped.
HID frequently use USB interrupt endpoints. One packet is exchanged per microframe (different between FULL/HIGH speed), and the data toggle must be the same, but doesn't have to toggle like BULK. Even if the hardware does it like this, there is something called interrupt moderation. USB jobs may be finished, but the USB host operating system doesn't see them until a CPU interrupt is generated. The number of IRQ's per second is 8000 by default for HIGH speed USB, if I'm not mistaken. Single jobs may not be submitted faster than this. Sometimes you can submit multiple jobs, but not all USB HC's can queue them in hardware, and still process them one by one. The next job is only started after the previous job has finished, and an IRQ has been generated.
@hselasky
Do you know FreeBSD supports High Speed USB HID device with wMaxPacketSize of the Interrupt Endpoint = 1024 Bytes?
It is no clear whether Windows support it or not, so that usual HID devices only have wMaxPacketSize of the Interrupt Endpoint <= 64 Bytes.
@mcuee : Yes, FreeBSD supports all features of the interrupt transport endpoint for all USB speeds.
FreeBSD doesn't support "USB stream mode endpoints".
Windows at least, did not support USB control transfers having payload above 4095 / 4096 bytes, last time I checked. FreeBSD can do up to 65535 bytes, by splitting the jobs, not needing to allocate a 64K buffer, which I guess the reason why Windows did not want to implement this.
If someone steps up to write an hidraw backend for FreeBSD (or modify the existing Linux one so it works with FreeBSD), I think deprecating the libusb backend, or at least documenting it as less favored, would be a decent way to handle this bug.
Just want to mention that initial report is a bit confusing. Later it is clear that the speed is not the real issue (full speed or high speed or even Low Speed). The issue is the report size (one report will normally done in single USB transfer by the OS kernel HID driver, if not using libusb) can be greater than packet size.
[To be confimred -- confirmed to be wrong]
It seems to me because Windows inbox HID driver most likely does not really support Interrupt IN/OUT Endpoint with wMaxPacketSize greater than 64 bytes, so for the High Speed USB HID device out in the market, the Interrupt IN/OUT Endpoint will still have wMaxPacketSize at 64Bytes.Later version of Windows may or may not have such limitations but the device out in the market have to support older version of Windows.[/To be confirmed -- confirmed to be wrong]
On the other hand, the HID report size can be bigger. Again Windows is the limitation here at 8KB.
Same for low speed USB HID device where the Interrupt Endpoint will have wMaxPacketSize at most 8 Bytes. Again the report size can be bigger
In any case, the HID report size (IN, OUT, Feature report) can be greater than wMaxPacketSize of the respective Interrupt Endpoint or Control Endpoint. Therefore hidapi libusb backend has to know the report size to decide whether it has finished processing of the report or not.
If we ask the application developer to carry out workaround in the application, it may be a big issue for the more complex applications like Mixx in this issue report (with mappings for over 100 devices).
Just an example of High Speed USB HID device -- Microchip PICKit 4 (USB Composite Device, with first interface to the HID interface). You can see that the wMaxPacketSize of the Interrupt IN endpoint for the HID interface is still 64 bytes.
But in this case, the report size is also 64 Bytes so it will not encounter issues using hidapi-libusb.
lsusb -tinfo under Ubuntu Linux 20.04.USBTreeviewer connection information under Windows 11.
hidtest output under Linux.
HID report patser output
Past discussions and comments by the original author of HIDAPI Alan Ott.
On 30-Jan-2013
Past discussions and comments by the original author of HIDAPI Alan Ott.
On 25-Feb-2014
Application level work-around from probe-rs
Also here from Microchip (hard-coded table)
https://github.com/microchip-pic-avr-tools/pyedbglib/blob/main/pyedbglib/hidtransport/toolinfo.py
Alan Ott mentioned about VRPN application side work-around. I believe it is here (512 Bytest report)
https://github.com/vrpn/vrpn/blob/master/vrpn_HumanInterface.C
So yes there are quite some devices with >64 Bytes HID report size. I need to find one of them and check if they still use wMaxPacketSize=64 bytest or if any of them use higher one.
@MCUdude
Hi Hans, would you please do me a favor to dump the USB Prober output of your Atmel-ICE or Power Debuger if you still have them. Thanks.
Something like this:
Or you can post the output from
lsusb -vvvunder macOS or Linux. You can install usbutils under macOS Homebrew.And also the outout of hidapi hidtest or mac-hid-dump.
https://github.com/todbot/mac-hid-dump
Can not confirm. I own the laptop with USB touchscreen which input report is greater than 100 bytes. It had Windows 8.0 preinstalled.
There are certainly devices with input report size greater than 64 Bytes as mentioned in the above (like 512 Bytes or even 1024 Bytes). However, please check the wMaxPacketSize of the Interupt IN endpoint of your USB touch deviice. I bet it is still 64 Bytes.
wMaxPacketSize is 64 as it is USB 2.0 LowSpeed device
I think you mean to say USB 2.0 Full Speed Device.
It is good that you have such device, probably you can use it to test the current HIDAPI FreeBSD implementation which use libusb. And if you would like to work on FreeBSD hidraw backend for HIDAPI, it can be useful as a comparison.
[To be confimred --> confirmed to be wrong]
It seems to me because Windows inbox HID driver most likely does not really support Interrupt IN/OUT Endpoint with wMaxPacketSize greater than 64 bytes, so for the High Speed USB HID device out in the market, the Interrupt IN/OUT Endpoint will still have wMaxPacketSize at 64Bytes.Later version of Windows may or may not have such limitations but the device out in the market have to support older version of Windows.[/To be confirmed --> confirmed to be wrong]
Hopefully some people will have some High Speed USB HID device to check if their HID interrupt Endpoint have greater than 64 Bytest wMaxPacketSize or not. So far I have only seen High Speed USB HID Devices with their Interrupt Endpoint having only 64 Bytes wMaxPacketSize.
I really need a picture, but this is what I've got:
I'm confused about the discussion regarding Windows. Why does it matter? Is there some use case for using the libusb backend on Windows instead of Windows' HID API?
Hi HPS,
Sorry the request is to @MCUdude (another Hans) who has the access to the two High Speed USB Composite devices with an HID interface (Atmel-ICE and Atmel Power Debugger). What you have (Jtag ICE mkII and AVR ISP mkII) are not HID devices.
Ahh, it is a side topic -- I am trying to understand why High Speed USB HID device developers would not choose to have the Interrupt IN endpoint's wMaxPacketSize to be higher value (up to 1024 Bytes). From what I read the HID report size is usually less than 1024 Bytes. In that case they can simplify their Firmware development efforts (send the report in one packet and no need to deal with multiple packets on the device side). Of course the issue is not as bad as this one since the device firmware developer only needs to deal with one paticular device or a group of similar devices.
As for hidapi Windows libusb backend, in fact there are some use cases.
Interestingly I see a similar issue from the libfido2 project. But they do not use libusb backend. I see they have platfrom specific backend and even one hidapi backend.
Issue:
Fixes:
@wulf7 and @hselasky
FYI as well. libfido2 uses FreeBSD hidraw on FreeBSD13+.
I think it is also good for hidapi to move to FreeBSD hidraw for newer version of FreeBSD.
https://github.com/Yubico/libfido2/blob/main/src/hid_freebsd.c
@mcuee I've only borrowed an Atmel ICE, so I cant test with this at the moment, but here are the Power Debugger dump, which is very similar hardware vice:
lsusb -vvv with only the Power Debugger plugged in
mac-hid-dump with only the Power Debugger plugged in
@MCUdude
Great. So we can see that the HID report size is 512 Bytes and the Interrupt IN/OUT Endpoint's wMaxPacketSize is 512 Bytes.
Since the Power Debugger works well under Windows, so this confirms that Windows does support HID interface with the Interrupt IN/OUT Endpoint's wMaxPacketSize greater than 64 Bytes.
The hid report parser output.
https://eleccelerator.com/usbdescreqparser/
I hope my modification to Jan Axelson's FX2HID FW is correct. In that case, I can probably carry out some tests for this issue in different OS.
INPUT/OUTPUT report size 128 bytes, loop back, no report ID.
(wMaxPacketSize = 64, high speed or full speed).
Indeed I can reproduce the issue using hidapitester under Linux.
Using libusb backend, I can reproduce the issue under Linux. You can see that it only reads back 64 Bytes.
The documentation of
libusb_interrupt_transfer()states: "You should also check the transferred parameter for interrupt writes. Not all of the data may have been written."https://libusb.sourceforge.io/api-1.0/group__libusb__syncio.html#ga0f171a72904a552fc43e6e6564d108a3
@JoergAtGithub and @Youw
Just wondering if you are still looking at this issue or not. Thanks.
This is clearly a bug in hidapi libusb backend, confirmed by several affected users.
But I'm not working on a fix curently.
Agree.
I don't have capacity for this either.
I have also encountered this issue in a HID device with reports smaller than 64 bytes which has a
wMaxPacketSizeof 8.I have created PR #728 to address this issue. I do not have a device with input reports over 64 bytes, but I believe it should work just as well. I'll have to see what other unusual HID devices I have around to test with.
And as an aside about the usefulness of the libusb backend: I encountered this issue when programming for an embedded Linux device that is stuck on the 4.x kernel. I need to use
HIDIOCSINPUTfor a particular HID device, but that is not supported until the 5.11 kernel. So I cannot use hidraw.Unfortunately my test shows #728 is not working yet.
The other approach is to use hidraw for FreeBSD, unfortunately it does not seem to fully work either for high speed USB HID device with >64 bytes of HID report.
@cederom and @dl8dtl
Please take a look at this issue when you are looking at PR #728 and #730. Thanks.
Guys can you please provide test steps on how to verify patches on FreeBSD? Do I need some specific device? :-)
Yes, you need certain specific test device. I have add the comments in #728 and #730.