MD5 optimization to support Amazon s3 integrity check generation

Rob_B_ — Mon, 19 Oct 2015 15:39:16 GMT

Amazon S3 uses MD5 for an integrity check on its data (note: non-security function so MD5 is ok -- I have no security issues with using MD5 for an integrity check). Since this is their choice, for my on-premise data I am moving to and from Amazon using Intel Servers I am now computing MD5 using software -- no hardware assist. This work is taking up a good portion of my cloud cores. Note that other object storage services like Swift also use MD5 so don't tell me to switch object storage services. I talked with Amazon today and they mentioned that MD5 might be available with SHA-1 function now in SilverLake. I can't seem to see anything like this. Can you please let me know if this is the case? And if it's not the case maybe Amazon and Intel can sit down and agree on an efficient, hardware-assisted integrity check algorithm?

topic MD5 optimization to support Amazon s3 integrity check generation in Software Tuning, Performance Optimization & Platform Monitoring

MD5 optimization to support Amazon s3 integrity check generation