Add argument for recursive hashing

vovcacik commented

2017-08-14 12:54:52 -05:00

(Migrated from github.com)

Not a big deal on Unix, but kinda pita on Windows.

Current workaround:

(it's not that complicated, bit chatty)

@echo off
setlocal enabledelayedexpansion

set EXE=xxhsum.exe

:: This batch adds "-r <folder>" argument for recursive hashing,
:: otherwise acts transparent.
if "%~1"=="-r" (
  :: Strip "-r" and folder name from arguments that we are going to pass to xxhsum.exe
  set ARGS=%*
  set ARGS=!ARGS:%~1=!
  set ARGS=!ARGS:%~2=!

  :: Process root dir
  call :process-dir "%~2" "%~2"

  :: Process subdirs
  for /D /r "%~f2" %%d in (*) do (
    :: Dont feed xxhsum.exe with folders that contain no files
    for /F %%_ in ('dir /b /a:-d "%%d" 2^>nul') do (
      :: Process subdir only once (the above for loop will cycle once for each file in %%d)
      if "%%d" neq "!LAST!" (
        set LAST=%%d
        call :process-dir "%%d" "%~2"
      )
    )
  )
) else (
  :: Be transparent
  %EXE% %*
)

goto :eof

:: Usage: process-dir <the-dir-being-processed> <the-dir-from-r-arg>
:process-dir
  :: Get relative path to dir
  set REL=%~f1
  set REL=!REL:%~f2=%~2!
  %EXE% %ARGS% "%REL%\*"
goto :eof

Not a big deal on Unix, but kinda pita on Windows. ###### Current workaround: *(it's not that complicated, bit chatty)* ``` @echo off setlocal enabledelayedexpansion set EXE=xxhsum.exe :: This batch adds "-r <folder>" argument for recursive hashing, :: otherwise acts transparent. if "%~1"=="-r" ( :: Strip "-r" and folder name from arguments that we are going to pass to xxhsum.exe set ARGS=%* set ARGS=!ARGS:%~1=! set ARGS=!ARGS:%~2=! :: Process root dir call :process-dir "%~2" "%~2" :: Process subdirs for /D /r "%~f2" %%d in (*) do ( :: Dont feed xxhsum.exe with folders that contain no files for /F %%_ in ('dir /b /a:-d "%%d" 2^>nul') do ( :: Process subdir only once (the above for loop will cycle once for each file in %%d) if "%%d" neq "!LAST!" ( set LAST=%%d call :process-dir "%%d" "%~2" ) ) ) ) else ( :: Be transparent %EXE% %* ) goto :eof :: Usage: process-dir <the-dir-being-processed> <the-dir-from-r-arg> :process-dir :: Get relative path to dir set REL=%~f1 set REL=!REL:%~f2=%~2! %EXE% %ARGS% "%REL%\*" goto :eof ```

👍 3 👎 1

vovcacik commented

2017-08-14 12:55:48 -05:00

(Migrated from github.com)

Also see #87

t-mat commented

2017-08-18 16:20:37 -05:00

(Migrated from github.com)

Plan A: Implement "Read filename(s) from stdin" switch.
- If we'll have such functionality, we'll be able to use it by
  dir /s /b **TARGET-DIR** | xxhsum --read-filenames-from-stdin > C:\my-result
- It's not straightforward. But it allows us to cooperate with other shell commands.
- We may hit EOF problem of stdin.
Plan B: Implement opendir()-ish functionality. For example, https://github.com/cxong/tinydir
- It's straightforward.
  xxhsum --show-hash-recursive **TARGET-DIR** > C:\my-result
- Since there's no "perfect" opendir() other than shell, it may cause other problems. e.g Permission, race condition, networking, Unicode/MBCS path name, etc.
Plan C: Do not implement anything.
- Just use
  for /R **TARGET-DIR** %f in (*) do ( xxhsum %f >> C:\my-result )
- cmd.exe must die :finnadie:
- for PowerShell use
  Get-ChildItem **TARGET-DIR** -Recurse | where { !$_.PSIsContainer} | ForEach-Object {xxhsum $_.FullName} | Out-File C:\my-result

Personally, I suppose "Plan C" may satisfy 70% of usage and I believe we should compromise with it.
But "Plan A" also would be nice since it's constructive and easy to implement without platform dependent functionality and/or external dependency.

- Plan A: Implement "Read filename(s) from stdin" switch. - If we'll have such functionality, we'll be able to use it by `dir /s /b **TARGET-DIR** | xxhsum --read-filenames-from-stdin > C:\my-result` - It's not straightforward. But it allows us to cooperate with other shell commands. - We may hit EOF problem of stdin. - Plan B: Implement `opendir()`-ish functionality. For example, https://github.com/cxong/tinydir - It's straightforward. `xxhsum --show-hash-recursive **TARGET-DIR** > C:\my-result` - Since there's no "perfect" `opendir()` other than shell, it may cause other problems. e.g Permission, race condition, networking, Unicode/MBCS path name, etc. - Plan C: Do not implement anything. - Just use `for /R **TARGET-DIR** %f in (*) do ( xxhsum %f >> C:\my-result )` - `cmd.exe` must die :finnadie: - for PowerShell use `Get-ChildItem **TARGET-DIR** -Recurse | where { !$_.PSIsContainer} | ForEach-Object {xxhsum $_.FullName} | Out-File C:\my-result` Personally, I suppose "Plan C" may satisfy 70% of usage and I believe we should compromise with it. But "Plan A" also would be nice since it's constructive and easy to implement without platform dependent functionality and/or external dependency.

data-man commented

2017-08-18 16:33:24 -05:00

(Migrated from github.com)

Plan D (the best :-): add xxHash to RHash. See issue #26.

Plan D (the best :-): add xxHash to [RHash](https://github.com/rhash/RHash). See issue [#26](https://github.com/rhash/RHash/issues/26).

👍 1

t-mat commented

2017-08-18 16:37:51 -05:00

(Migrated from github.com)

@data-man I'm really sorry but I can't read that issue 😢
But RHash looks good.

@data-man I'm really sorry but I can't read that issue 😢 But RHash looks good.

data-man commented

2017-08-18 16:49:08 -05:00

(Migrated from github.com)

@t-mat

Sorry, machine translated:

Please add support for xxHash algorithm is a step into the future, available in 202 lines of code. Judging by the tests, runs faster CRC32, which is useful when calculating checksum *.iso, *.mkv, databases, and similar large files. Today it is in HashOnClick, but, alas, for the $.

@t-mat Sorry, machine translated: > Please add support for xxHash algorithm is a step into the future, available in 202 lines of code. Judging by the tests, runs faster CRC32, which is useful when calculating checksum *.iso, *.mkv, databases, and similar large files. Today it is in HashOnClick, but, alas, for the $$$.

👍 1

Cyan4973 commented

2017-08-18 16:54:07 -05:00

(Migrated from github.com)

Plan A looks interesting, it's also a fairly new feature, existing utilities like md5sum do not support it as far as I know.
The "without platform dependent functionality" tag feels premature to me : we would need to be sure that dir output can be parsed safely, in a universal way, on all systems.
Alternatively, it could be tagged as working on only a few tested systems, and gracefully refuse to work on others.

As stated in #87, another potential solution could be to support a "recursive mode", typically using -r command, including all files in selected and included directories. It's also not supported by md5sum, but it's a well-known capability of compression tools like gzip, xz, zstd, lz4 etc., and we've got some code to make it work. The existing code is "reasonably portable" : it indeed depends on some OS-related libraries for directory discovery, but works well on a lot of OS, and is automatically disabled on unsupported OS.

Plan A looks interesting, it's also a fairly new feature, existing utilities like `md5sum` do not support it as far as I know. The "without platform dependent functionality" tag feels premature to me : we would need to be sure that `dir` output can be parsed safely, in a universal way, on all systems. Alternatively, it could be tagged as working on only a few tested systems, and gracefully refuse to work on others. As stated in #87, another potential solution could be to support a "recursive mode", typically using `-r` command, including all files in selected and included directories. It's also not supported by `md5sum`, but it's a well-known capability of compression tools like `gzip`, `xz`, `zstd`, `lz4` etc., and we've got some code to make it work. The existing code is "reasonably portable" : it indeed depends on some OS-related libraries for directory discovery, but works well on a lot of OS, and is automatically disabled on unsupported OS.

👍 1

vovcacik commented

2017-08-21 09:34:12 -05:00

(Migrated from github.com)

@t-mat
The for /R **TARGET-DIR** %f in (*) do ( xxhsum %f >> C:\my-result ) is fairly straightforward but awfully slow, thats why I am using for /D /r "%~f2" %%d in (*) in my snippet.

@t-mat The `for /R **TARGET-DIR** %f in (*) do ( xxhsum %f >> C:\my-result )` is fairly straightforward but awfully slow, thats why I am using `for /D /r "%~f2" %%d in (*)` in my snippet.

frenchiveruti commented

2020-10-13 12:59:20 -05:00

(Migrated from github.com)

A variant for WLS (using debian terminal on windows 10):

IFS=$'\n'; for file in *; do ./xxhsum.exe $file >> hashes.txt ;done

Assuming:

Terminal is inside the folder with the files to hash.
xxhsum is in same dir as files to hash (that's why ./xxhsum.exe is used)
Be aware that xxhsum hash will make it into the hashes.txt file.
1.IFS=$'\n' makes the newline the only separator, so you can have whitespaces

Another method:
Select files to hash, copy all the routes using "shift+right click", paste into a txt like stdin.txt
remove leading and tailed quotations "" using "search and replace" on your favorite text editor.
save file as unix (LF) format. (windows format CR LF gives errors)

Run this command:

for file in $(cat stdin.txt); do ./xxhsum.exe $file >> hashes.txt; done

Now the hashes file contains just the files that you wanted to hash.

A variant for WLS (using debian terminal on windows 10): ```bash IFS=$'\n'; for file in *; do ./xxhsum.exe $file >> hashes.txt ;done ``` Assuming: 1. Terminal is inside the folder with the files to hash. 1. xxhsum is in same dir as files to hash (that's why ./xxhsum.exe is used) 1. Be aware that xxhsum hash will make it into the hashes.txt file. 1.IFS=$'\n' makes the newline the only separator, so you can have whitespaces Another method: Select files to hash, copy all the routes using "shift+right click", paste into a txt like stdin.txt remove leading and tailed quotations "" using "search and replace" on your favorite text editor. save file as unix (LF) format. (windows format CR LF gives errors) Run this command: ```bash for file in $(cat stdin.txt); do ./xxhsum.exe $file >> hashes.txt; done ``` Now the hashes file contains just the files that you wanted to hash.

👍 1

flawrist commented

2021-03-17 21:34:16 -05:00

(Migrated from github.com)

xxhsum 0.8.0 behaves oddly even without recursive traversal.

$ xxhsum -H128 *.* 
xxhsum: $RECYCLE.BIN: Is a directory                                                                                       
94796d676519ea4c53e10e249f0ef219  2017-03-18 суббота.md                                                             
f91167e2b1937e78a6931748d1df4df3  idea-icon.png                                                                            
xxhsum: Sync: Is a directory                                                                                               
xxhsum: System Volume Information: Is a directory                                                                          
07c9eef2123cff8ed27d1e0a7eaddec1  SYSTEM.LOG1                                                                              
99aa06d3014798d86001c324468d497f  SYSTEM.LOG2                                                                              
df6f507eacafd3219612174a23ac4312  SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TM.blf                                      
18ae6eb42537764911b1248ee9d947a6  SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TMContainer00000000000000000001.regtrans-ms 
0d9f6acee70bd714fd444a4e0c1d4c46  SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TMContainer00000000000000000002.regtrans-ms 
Error: Could not open 'Загрузки': Permission denied.

I asked to calculate hashes of files, so why it mentions dirs in the output and why the Cyrillic-named dir is suddenly not accessible?

xxhsum 0.8.0 behaves oddly even without recursive traversal. ```console $ xxhsum -H128 *.* xxhsum: $RECYCLE.BIN: Is a directory 94796d676519ea4c53e10e249f0ef219 2017-03-18 суббота.md f91167e2b1937e78a6931748d1df4df3 idea-icon.png xxhsum: Sync: Is a directory xxhsum: System Volume Information: Is a directory 07c9eef2123cff8ed27d1e0a7eaddec1 SYSTEM.LOG1 99aa06d3014798d86001c324468d497f SYSTEM.LOG2 df6f507eacafd3219612174a23ac4312 SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TM.blf 18ae6eb42537764911b1248ee9d947a6 SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TMContainer00000000000000000001.regtrans-ms 0d9f6acee70bd714fd444a4e0c1d4c46 SYSTEM{f013ec73-0670-11e9-b9d0-00241dd250d2}.TMContainer00000000000000000002.regtrans-ms Error: Could not open 'Загрузки': Permission denied. ``` I asked to calculate hashes of files, so why it mentions dirs in the output and why the Cyrillic-named dir is suddenly not accessible?

t-mat commented

2021-03-19 06:44:13 -05:00

(Migrated from github.com)

I asked to calculate hashes of files, so why it mentions dirs in the output

It's caused by the shell which you're using. In this case, xxhsum does nothing about wildcard. Usually, *nix flavor shell expands wildcard (globbing) such as *.* to filesystem entries including directory.
For example, the following command shows actual expanded arguments.

echo *.*

why the Cyrillic-named dir is suddenly not accessible?

I have no idea. On my terminal, xxhsum for directory which has Cyrillic name shows the following message as intended.

$ xxhsum *
xxhsum: Загрузки: Is a directory

But since error message suggests issue about permission, you can try some method to figure out what is happening

Use whoami command to make sure you're using your account or not.
Retrieve some infomation about the directory by ls -ld Загрузки.

> I asked to calculate hashes of files, so why it mentions dirs in the output It's caused by the shell which you're using. In this case, `xxhsum` does nothing about wildcard. Usually, *nix flavor shell expands wildcard ([globbing](https://en.wikipedia.org/wiki/Glob_%28programming%29)) such as `*.*` to filesystem entries including directory. For example, the following command shows actual expanded arguments. ``` echo *.* ``` > why the Cyrillic-named dir is suddenly not accessible? I have no idea. On my terminal, `xxhsum` for directory which has Cyrillic name shows the following message [as intended](https://github.com/Cyan4973/xxHash/blob/a9054f397d7f41bc505638df3853b270eb9e7493/xxhsum.c#L661). ``` $ xxhsum * xxhsum: Загрузки: Is a directory ``` But since error message suggests issue about permission, you can try some method to figure out what is happening - Use `whoami` command to make sure you're using your account or not. - Retrieve some infomation about the directory by `ls -ld Загрузки`.

👍 1

flawrist commented

2021-03-19 07:26:20 -05:00

(Migrated from github.com)

@t-mat, I am a Windows 7 administrator with its shell and a few ported UNIX utils, nothing fancy.

$ whoami
...\alexander

$ ls -ld Загрузки
drwxr-xr-x 1 Alexander None 0 мар 19 05:30 Загрузки

Actually, 'denied' issue seems not to be related to permissions as follows

$ mkdir Birds Vögel
$ xxhsum -H128 *.*
...
xxhsum: Birds: Is a directory
Error: Could not open 'Vögel': Permission denied.
                        ^------------------------------------------- the crux, perhaps?

@t-mat, I am a Windows 7 administrator with its shell and a few ported UNIX utils, nothing fancy. ```console $ whoami ...\alexander $ ls -ld Загрузки drwxr-xr-x 1 Alexander None 0 мар 19 05:30 Загрузки ``` Actually, 'denied' issue seems not to be related to permissions as follows ``` console $ mkdir Birds Vögel $ xxhsum -H128 *.* ... xxhsum: Birds: Is a directory Error: Could not open 'Vögel': Permission denied. ^------------------------------------------- the crux, perhaps? ```

Rows
Columns

Add argument for recursive hashing #109

Current workaround: