at the moment the memory limit of a 64 bit process is out of reach. So you will not get an out of memory exception until all the virtual memory is used completely. Normally the OS starts paging if the physical memory is short and the system behaves really slow.
If you want to avoid that, you have several options. One would be to query the OS about the free physical memory. Check if the amount is below a certain threshold and then throw an out of memory exception.
To check the memory you can use the following APIs:
Linux: Read /proc/meminfo
c:\forum\U84518>icl /EHsc maincpp.cpp
Intel C++ Intel 64 Compiler XE for applications running on Intel 64, Ve
rsion 18.104.22.168 Build 20110719
Copyright (C) 1985-2011 Intel Corporation. All rights reserved.
Microsoft Incremental Linker Version 10.00.40219.01
Copyright (C) Microsoft Corporation. All rights reserved.
Out of memory with i=25742.... 1627Mb/37636Mb
Press any key to continue . . .