Intel® DevCloud
Help for those needing help starting or connecting to the Intel® DevCloud
1644 Discussions

What are the best practices for transferring files to/from AWS S3?

User01
New Contributor II
‎06-20-2022 10:06 AM
756 Views

I see that I am unable to connect to AWS S3 using the AWS CLI on login-2. I am able to connect from a batch node (e.g., s001-n016). However, the transfer rate from the batch node is limited to 2.0 MiB/s, which is too slow to be very useful. In addition, using a batch node to transfer files is an inefficient use of the resource. In other clusters and supercomputers, there is typically a transfer queue separate from the compute queue and specifically designed to handle data transfers.

What is the best practice for efficiently transferring files to/from Amazon Web Services (AWS) S3 using the AWS command-line interface (CLI)?

Labels (2)
Labels:
0 Kudos

Link Copied

2 Replies
User01
New Contributor II
‎06-30-2022 06:53 AM
710 Views

Is any information available for this question?

0 Kudos
Copy link
User01
New Contributor II
‎07-13-2022 12:01 PM
679 Views

This is still a significant limitation to the usability of DevCloud for development and performance testing.

0 Kudos
Copy link
Reply

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.