Incorrect MD5 for a sequence submit

HaoQinC commented

2020-06-09 20:55:34 -05:00

(Migrated from github.com)

Hi, usually I need to calculate the MD5 value of large files which could up to 20 GB or more, so I submit the jobs one by one with 10 MB chunks sequentially for one file. And I noticed that for the same context the new job submit is not allowed before the previous one is completed, so I use this

while (hash_ctx_processing(ctx)) {
            md5_ctx_mgr_flush(mgr);
        }

before each submits, and also before the fetch of the digest. But if the submit size is larger than 63 Bytes the result is incorrect. However, if I put the flush method after the submit method instead of before, there is no such issue. I just wonder what' the difference between before and after, no matter which way I have used before the fetch of digest I have the status HASH_CTX_STS_COMPLETE.
The reason I wanna the flush before the submit because I have a context pool so that I have tried to avoid flush after every submit.

code just like this:
the one has the issue:

while (processed < size) {
    while (hash_ctx_processing(ctx)) {
            md5_ctx_mgr_flush(mgr);
        }
    ctx = md5_ctx_mgr_submit(mgr, ctx, chunk, step, flag);
}
while (hash_ctx_processing(ctx) {
        md5_ctx_mgr_flush(mgr);
 }
Get digest

The one has no such issue:

while (processed < size) {
    ctx = md5_ctx_mgr_submit(mgr, ctx, chunk, step, flag);
    while (hash_ctx_processing(ctx)) {
            md5_ctx_mgr_flush(mgr);
        }
}
Get digest

Hi, usually I need to calculate the MD5 value of large files which could up to 20 GB or more, so I submit the jobs one by one with 10 MB chunks sequentially for one file. And I noticed that for the same context the new job submit is not allowed before the previous one is completed, so I use this ``` while (hash_ctx_processing(ctx)) { md5_ctx_mgr_flush(mgr); } ``` before each submits, and also before the fetch of the digest. But if the submit size is larger than 63 Bytes the result is incorrect. However, if I put the flush method after the submit method instead of before, there is no such issue. I just wonder what' the difference between before and after, no matter which way I have used before the fetch of digest I have the status HASH_CTX_STS_COMPLETE. The reason I wanna the flush before the submit because I have a context pool so that I have tried to avoid flush after every submit. code just like this: the one has the issue: ``` while (processed < size) { while (hash_ctx_processing(ctx)) { md5_ctx_mgr_flush(mgr); } ctx = md5_ctx_mgr_submit(mgr, ctx, chunk, step, flag); } while (hash_ctx_processing(ctx) { md5_ctx_mgr_flush(mgr); } Get digest ``` The one has no such issue: ``` while (processed < size) { ctx = md5_ctx_mgr_submit(mgr, ctx, chunk, step, flag); while (hash_ctx_processing(ctx)) { md5_ctx_mgr_flush(mgr); } } Get digest ```

gbtucker commented

2020-06-10 11:39:33 -05:00

(Migrated from github.com)

Hi @HaoQinC. I have an idea why the first does not work but you really don't want to use multi-buffer hashing like this. Perhaps in the first case it is waiting only until one of the jobs is complete and not until the required one is complete.

Normally you don't want to do singular subit() then flush() as this defeats keeping multiple jobs in flight and the benefit of multi-buffer hashing. For large hashing jobs you might take a look as an example the test https://github.com/intel/isa-l_crypto/blob/master/tests/extended/md5_mb_over_4GB_test.c as this is explicitly testing multiple jobs over 4GB.

Hi @HaoQinC. I have an idea why the first does not work but you really don't want to use multi-buffer hashing like this. Perhaps in the first case it is waiting only until one of the jobs is complete and not until the required one is complete. Normally you don't want to do singular subit() then flush() as this defeats keeping multiple jobs in flight and the benefit of multi-buffer hashing. For large hashing jobs you might take a look as an example the test https://github.com/intel/isa-l_crypto/blob/master/tests/extended/md5_mb_over_4GB_test.c as this is explicitly testing multiple jobs over 4GB.

HaoQinC commented

2020-06-10 12:19:01 -05:00

(Migrated from github.com)

Hi @gbtucker thanks for the response, it's a little confused me as there is only one context I have used in my test, that's mean there is only one job has been submitted, and it works if I limit the submit size under 64 Byte.

gbtucker commented

2020-06-10 13:31:13 -05:00

(Migrated from github.com)

Sorry, I didn't get at first that you are hashing 20GB but at 63 bytes at a time. As these are so small of an update you may have just hit a case where it happens to work. The MD5_BLOCK_SIZE = 64 and we have an internal partial block buffer for the unusual case that updates are not in whole block sizes. You may have just hit extra unintentional delay for 63 byte updates as it has to wait for a complete block to move forward. Certainly in practice I would suggest updates be in larger chunks if possible.

Sorry, I didn't get at first that you are hashing 20GB but at 63 bytes at a time. As these are so small of an update you may have just hit a case where it happens to work. The `MD5_BLOCK_SIZE = 64` and we have an internal partial block buffer for the unusual case that updates are not in whole block sizes. You may have just hit extra unintentional delay for 63 byte updates as it has to wait for a complete block to move forward. Certainly in practice I would suggest updates be in larger chunks if possible.

HaoQinC commented

2020-06-11 17:37:51 -05:00

(Migrated from github.com)

Thanks for your patience, the size problem is just a test I found, I will set the size a couple of MBs, so it's OK. But I still do not understand the difference to flush before submit and after submit. I have created another test, where I only use one context of the manager:

while (processed < size) {
        int step = chunk_size;
        if (processed == 0) {
            flag = HASH_FIRST;
        } else if (size - processed <= chunk_size) {
            flag = HASH_LAST;
            step = size - processed;
        } else {
            flag = HASH_UPDATE;
        }
        char sub[step];
        memcpy(sub, &hello[processed], step);
        ctx = md5_ctx_mgr_submit(mgr, &ctxpool[0], sub, step, flag);
        processed += step;
        while (md5_ctx_mgr_flush(mgr));
    }

This works well, however, if I put flush before the submit, the result is incorrect, and I didn't find any error by checking ctx->error. I thought the order should have no matter as there is only one job submitted in the same time, only one context is in use. Is there any difference in the two orders?

while (processed < size) {
        int step = chunk_size;
        if (processed == 0) {
            flag = HASH_FIRST;
        } else if (size - processed <= chunk_size) {
            flag = HASH_LAST;
            step = size - processed;
        } else {
            flag = HASH_UPDATE;
        }
        char sub[step];
        memcpy(sub, &hello[processed], step);
        while (md5_ctx_mgr_flush(mgr));
        ctx = md5_ctx_mgr_submit(mgr, &ctxpool[0], sub, step, flag);
        processed += step;
    }

The reason why I tried this test because I have tried to use a queue to cache the request and submit the jobs if the related context has the return and is idle, however, the result is incorrect for the same reason as I mentioned above.

I understand that the ctx return order is the same as the submit order, but in the application the coming data order is unpredicted and the chunks' number of the data is not the same. in md5_mb_over_4GB_test.c, the order is restricted to keep same and after the job submit the flush will be called immediately except for the first submit of each contexts. Do we have any other solutions to avoid the submit-flush pair?

Thanks for your patience, the size problem is just a test I found, I will set the size a couple of MBs, so it's OK. But I still do not understand the difference to flush before submit and after submit. I have created another test, where I only use one context of the manager: ``` while (processed < size) { int step = chunk_size; if (processed == 0) { flag = HASH_FIRST; } else if (size - processed <= chunk_size) { flag = HASH_LAST; step = size - processed; } else { flag = HASH_UPDATE; } char sub[step]; memcpy(sub, &hello[processed], step); ctx = md5_ctx_mgr_submit(mgr, &ctxpool[0], sub, step, flag); processed += step; while (md5_ctx_mgr_flush(mgr)); } ``` This works well, however, if I put flush before the submit, the result is incorrect, and I didn't find any error by checking ctx->error. I thought the order should have no matter as there is only one job submitted in the same time, only one context is in use. Is there any difference in the two orders? ``` while (processed < size) { int step = chunk_size; if (processed == 0) { flag = HASH_FIRST; } else if (size - processed <= chunk_size) { flag = HASH_LAST; step = size - processed; } else { flag = HASH_UPDATE; } char sub[step]; memcpy(sub, &hello[processed], step); while (md5_ctx_mgr_flush(mgr)); ctx = md5_ctx_mgr_submit(mgr, &ctxpool[0], sub, step, flag); processed += step; } ``` The reason why I tried this test because I have tried to use a queue to cache the request and submit the jobs if the related context has the return and is idle, however, the result is incorrect for the same reason as I mentioned above. I understand that the ctx return order is the same as the submit order, but in the application the coming data order is unpredicted and the chunks' number of the data is not the same. in md5_mb_over_4GB_test.c, the order is restricted to keep same and after the job submit the flush will be called immediately except for the first submit of each contexts. Do we have any other solutions to avoid the submit-flush pair?

gbtucker commented

2020-06-11 19:51:58 -05:00

(Migrated from github.com)

Hi @HaoQinC, still not sure. The last example is missing a final flush or check, possibly leaving the last update in the queue waiting for more submits to join.

Also to be clear, the ctx return can be in a different order than submit. If a small block is submitted after long one for example, the small block will likely finish and return next.

Hi @HaoQinC, still not sure. The last example is missing a final flush or check, possibly leaving the last update in the queue waiting for more submits to join. Also to be clear, the ctx return can be in a different order than submit. If a small block is submitted after long one for example, the small block will likely finish and return next.

HaoQinC commented

2020-06-12 00:04:49 -05:00

(Migrated from github.com)

Thanks, the order is a key point. Sorry I didn't make it clear, before I print the md5 value I also try the code while (md5_ctx_mgr_flush(mgr)); to guarantee the job is finished. So this is confused me why the flush order is a matter as I only use one context for one job with a couple of sequent submit.

The scenario is just like I have multiple producers that provide different size data, some may need to submit only once with HASH_ENTIRE, and most of them need to submit multiple times, and the coming order is random. With multiple solutions, all of them need to check if the previous job is completed before the next submit for the same context, everything looks good in the process and there is no any error reported, but the final result incorrect comparing with OpenSSL.

Thanks, the order is a key point. Sorry I didn't make it clear, before I print the md5 value I also try the code `while (md5_ctx_mgr_flush(mgr));` to guarantee the job is finished. So this is confused me why the flush order is a matter as I only use one context for one job with a couple of sequent submit. The scenario is just like I have multiple producers that provide different size data, some may need to submit only once with HASH_ENTIRE, and most of them need to submit multiple times, and the coming order is random. With multiple solutions, all of them need to check if the previous job is completed before the next submit for the same context, everything looks good in the process and there is no any error reported, but the final result incorrect comparing with OpenSSL.

Rows
Columns

Incorrect MD5 for a sequence submit #46