Using Asynchronous I/O in C

[youn__kihang]

Hello, All

I have a code that creates a lot of small files with fopen in C. 

Because a large number of MPI ranks are written to the same folder, performance degradation occurs in shared storage.

In fortran I have found to use the asynchronous option on open. How do I modify the code to use asynchronous I/O?

Please let me know if there are any options to use with fopen.

Thanks

[jimdempseyatthecove]

fopen (and it's accompanying writes) are not enhanced for asynchronous I/O.

This said, you are free to use a separate thread that uses fopen (and it's accompanying writes) to perform the I/O. Actually the fopen can be performed by the main thread of the rank and writes by an instantiated thread. Note, it will be your responsibility to coordinate the I/O such that the sequence is correct and that you do not destroy a buffer until after it has been written.

Hint, use only 1 thread per file and a FIFO queue from your Rank thread to your additional I/O thread.

Jim Dempsey

[SantoshY_Intel]

Hi,

We need some more information so that we can investigate well. could you provide the below details?

1. Are you trying to "write" a single file using multiple ranks? (or) multiple files using multiple ranks?
2. Can you provide your system environmental details like OS & filesystem?
3. Can you provide the size of your files approximately?

Awaiting your reply.

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

As we didn't hear back from you, Is your issue resolved? If not, please provide the details that have been asked in my previous post.

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

As we have not heard back from you, we are considering that your issue has been resolved. So we will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread. Any further interaction in this thread will be considered community only.

Have a Good day!

Thanks & Regards,

Santosh