xxh64sum: Argument list too long. #87

New issue

Closed

opened 2016-08-25 05:42:44 -05:00 by chellasundar · 2 comments

chellasundar commented

2016-08-25 05:42:44 -05:00

(Migrated from github.com)

Hi Yann,

Good Evening.

Hope all is well.

I try to run the script using xxh32sum and xx64sum file.Its show some error message.

-bash: /root/xxHash-master/xxh64sum: Argument list too long.

Our total file is 29000 Nos.

How many numbers of files can support xxh32sum and xxh64sum script.

Any alternate opensource or paid software is available to check the checksum with sub folder and files.

Thanks,
Chellasundar SR

Hi Yann, Good Evening. Hope all is well. I try to run the script using xxh32sum and xx64sum file.Its show some error message. -bash: /root/xxHash-master/xxh64sum: Argument list too long. Our total file is 29000 Nos. How many numbers of files can support xxh32sum and xxh64sum script. Any alternate opensource or paid software is available to check the checksum with sub folder and files. Thanks, Chellasundar SR

Cyan4973 commented

2016-08-25 06:17:06 -05:00

(Migrated from github.com)

I suspect the problem is that the list of files is so large that it ends up being too large for the shell command buffer, which is typically limited to 2 MB in modern Linux distributions.

I'm afraid there is no solution yet for this problem within xxhsum.
You will have to provide the list of files in smaller batches.

zstd solves this issue by providing a "recursive" mode, doing the expansion internally, instead of relying on shell. In which case, it's no longer limited by shell command maximum size.
I guess a future development could be to adapt this mode to xxhsum.

I suspect the problem is that the list of files is so large that it ends up being too large for the shell command buffer, which is typically limited to 2 MB in modern Linux distributions. I'm afraid there is no solution yet for this problem within `xxhsum`. You will have to provide the list of files in smaller batches. `zstd` solves this issue by providing a "recursive" mode, doing the expansion internally, instead of relying on shell. In which case, it's no longer limited by shell command maximum size. I guess a future development could be to adapt this mode to `xxhsum`.

t-mat commented

2016-08-25 09:12:39 -05:00

(Migrated from github.com)

Hi, please use find and/or xargs:
http://askubuntu.com/questions/318530/

You can find many similar solutions by googling with "md5sum recursive".

Hi, please use `find` and/or `xargs`: http://askubuntu.com/questions/318530/ You can find many similar solutions by googling with "md5sum recursive".

👍 2

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

gh-Cyan4973/xxHash#87

No description provided.

Rows
Columns