Intel® DevCloud
Help for those needing help starting or connecting to the Intel® DevCloud
1474 Discussions

What are the best practices for transferring files to/from AWS S3?

User01
New Contributor II
539 Views

I see that I am unable to connect to AWS S3 using the AWS CLI on login-2. I am able to connect from a batch node (e.g., s001-n016). However, the transfer rate from the batch node is limited to 2.0 MiB/s, which is too slow to be very useful. In addition, using a batch node to transfer files is an inefficient use of the resource. In other clusters and supercomputers, there is typically a transfer queue separate from the compute queue and specifically designed to handle data transfers.

What is the best practice for efficiently transferring files to/from Amazon Web Services (AWS) S3 using the AWS command-line interface (CLI)?

Labels (2)
0 Kudos
2 Replies
User01
New Contributor II
493 Views

Is any information available for this question?

0 Kudos
User01
New Contributor II
462 Views

This is still a significant limitation to the usability of DevCloud for development and performance testing.

0 Kudos
Reply