- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Hi,

This is probably a simple one, but I am new to MKL and can't figure it out. I have a large amount of time series data for which I am trying to construct a covariance matrix, and I can't seem to get my code (heavily derived from the vslscorrelationmatrix.c example) to work with N > 100k observations. I am running Windows 7, Visual Studio 2012, am building in 64-bit Release mode, and have ILP64 set up per the Link Line Advisor instructions.

When using a large N value (even with a very small number of dimensions in the array), I get a stack overflow error when creating a new task. I have pretty much everything in main() after the task create function commented out for debugging, but please see below. From some tinkering, it looks like the (float*)x argument (representing the observation matrix) in vslsSSNewTask is causing the crash. Could somebody please advise? Thanks in advance.

#include <stdio.h> #include "mkl.h" #include <stdlib.h> #include <malloc.h> #include <stdint.h> #include <string> #include "D:\Program Files (x86)\Intel\Composer XE 2013 SP1\mkl\examples\examples_core\vslc\source\errcheck.inc" //new includes #include "stdafx.h" #include <windows.h> #include <conio.h> #include <tchar.h> #define DIM 3 /* Task dimension */ #define N 100000 /* Number of observations */ double* memmap(std::string ticker); int main(int argc, char **argv) { printf("JPM\n"); double *jpm = memmap("JPM"); for (int i = 0; i <5; i++) { printf("%lf\n", jpm); } printf("C\n"); double *c = memmap("C"); for (int i = 0; i <5; i++) { printf("%lf\n", c); } printf("AAPL\n"); double *aapl = memmap("AAPL"); for (int i = 0; i <5; i++) { printf("%lf\n", aapl); } VSLSSTaskPtr task; MKL_INT64 dim; MKL_INT64 n; MKL_INT64 x_storage; MKL_INT64 cov_storage; MKL_INT64 cor_storage; float x[DIM]; float cov[DIM][DIM], cor[DIM][DIM]; float mean[DIM]; float aTmp, rTmp; int i, j, errcode; int errnums = 0; /***** Initializing parameters for Summary Statistics task *****/ dim = DIM; n = N; x_storage = VSL_SS_MATRIX_STORAGE_COLS; cov_storage = VSL_SS_MATRIX_STORAGE_FULL; cor_storage = VSL_SS_MATRIX_STORAGE_FULL; for(i = 0; i < dim; i++) { mean = 0.0; for(j = 0; j < dim; j++) { cov= 0; cor = 0; } } for(i=0;i<n;i++) { x [0]=jpm; x[1]=c; x[2]=aapl; } printf("%llu",sizeof(x)); getchar(); /***** Create Summary Statistics task *****/ errcode = vslsSSNewTask( &task, &dim, &n, &x_storage, (float*)x, 0, 0 ); CheckVslError(errcode); /***** Initialization of the task parameters using FULL_STORAGE for covariance/correlation matrices *****/ /* errcode = vslsSSEditCovCor( task, mean, (float*)cov, &cov_storage, (float*)cor, &cor_storage ); CheckVslError(errcode); */ // Compute covariance/correlation matrices using FAST method /* errcode = vslsSSCompute( task, VSL_SS_COV|VSL_SS_COR, VSL_SS_METHOD_FAST ); CheckVslError(errcode); */ //enable above /* printf("Task dimension : %d\n", dim); printf("Number of observations : %d\n\n", n); //Print the computed mean, covariance and correlation matrices printf("\nComputed means\n"); for(i = 0; i < dim; i++) { printf("%+lf ", mean); } printf("\n\nComputed covariance matrix "); printf("Computed correlation matrix\n"); for(i = 0; i < dim; i++) { for(j = 0; j < dim; j++) { printf("%+10lf ", cov); } printf(" "); for(j = 0; j < dim; j++) { printf("%+10lf ", cor ); } printf("\n"); } printf("\nPrinting input matrix\n"); printf("%f",x[0][0]); printf("\n"); printf("%f",x[0][1]); printf("\n"); printf("%f",x[0][2]); printf("\n"); //Delete Summary Statistics task errcode = vslSSDeleteTask( &task ); CheckVslError(errcode); MKL_Free_Buffers(); */ printf("completed\n"); getchar(); return 0; } double* memmap(std::string ticker) { LPCTSTR pBuf; using namespace std; const std::string price = "PRICE"; std::string hname = ticker+price; TCHAR *param=new TCHAR[hname.size()+1]; param[hname.size()]=0; std::copy(hname.begin(),hname.end(),param); int dataLen = 10000000; HANDLE hMapFile = NULL; PVOID pView = NULL; // Try to open the named file mapping identified by the map name. hMapFile = OpenFileMapping( FILE_MAP_ALL_ACCESS, // Read access FALSE, // Do not inherit the name param // File mapping name ); pView = MapViewOfFile( hMapFile, // Handle of the map object FILE_MAP_ALL_ACCESS, // Read access 0, // High-order DWORD of the file offset 0, // Low-order DWORD of the file offset dataLen // The number of bytes to map to view ); double *bPoint = (double*) pView; double *TempArr = (double*)pView; return TempArr; }

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

No, I am not talking about chunks now, your array is not really big enough for that.

Array of 3*100000 should be ok, but not allocated on stack.

1) float x

float *x;

x = (float*)malloc(sizeof(float)*DIM*N);

...

free(x);

2) x

Everything else stays as it is now.

Link Copied

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

could you please attach the example as a separate .cpp file in instead of using plain text.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

It looks like main problem is in the way you allocate huge array on stack:

float x

You are probably getting stack overflow because of that. Moving to malloc/free for allocating memory for x should solve that.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Hi Ilya,

Sorry to trouble you, but are you proposing to do this in chunks? Also, I'm new to MKL, but I have been working with other, much larger objects in C++: shouldn't an array of doubles or floats with dimensions 3*100000 this be ok with IPL64?

Would you happen to have an example of malloc/free for this usage? Thank you in advance.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

No, I am not talking about chunks now, your array is not really big enough for that.

Array of 3*100000 should be ok, but not allocated on stack.

1) float x

float *x;

x = (float*)malloc(sizeof(float)*DIM*N);

...

free(x);

2) x

Everything else stays as it is now.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Thank you Ilya, that did it!

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page