Allocating large 3D Arrays in C - Errors

Surrender

I'm trying to allocate around 30 3d arrays of size 300*300*300.

I've tried allocating them statically, i.e.:

float dx[xSize][ySize][zSize];

With this method I get a crash (upon execution) when I allocate the 1st array.

I have better results allocating the arrays dynamically:
float*** dx ;
dx=malloc(xSize*sizeof *dx );//Allocate 'xSize' pointer-to-pointer of int

for(x=0;x<xSize;x++)
{
dx[ x ] = malloc(ySize* sizeof **dx);//Allocate 'ySize' pointers of int
for(y=0;y<ySize;y++)
{
dx[ x ][ y ]= malloc(zSize* sizeof ***dx);//Allocate 'zSize' ints
}
}

With the above I can allocate around 15 arrays before the execution crashes.

I'm using the Eclipse CDT IDe and the GNU compiler.
I have the program coded in Matlab already but I wanted to code it in C to compare speeds.
Has anyone any experience with working with large datasets in C such as this and could they point me in the right direction?

fasty

Have you set a breakpoint and stepped through the code to see the exact point of failure?

It's not a solution to your problem, but as an alternative, allocate a single array of 300*300*300 and write a helper function to retrieve the element you want based on x,y,z coordinates. What you're doing is memory management hell!

Surrender

fasty wrote: »

Have you set a breakpoint and stepped through the code to see the exact point of failure?

It's not a solution to your problem, but as an alternative, allocate a single array of 300*300*300 and write a helper function to retrieve the element you want based on x,y,z coordinates. What you're doing is memory management hell!

I have run through the code with a breakpoint, it occurs after 15 3D array allocations (each the same size as the array I posted(300^3)).

I commented out every allocation bar one and I can access any element I want without a problem.

I'm also running this on Vista64 with 4GB of Ram, is there a stack allocation problem here?

Martyr

its probably a stack overflow.
try declare it as global memory rather than inside a local function?

ianhobo

Surrender wrote: »

I'm trying to allocate around 30 3d arrays of size 300*300*300.

I've tried allocating them statically, i.e.:

float dx[xSize][ySize][zSize];

What are xSize ySize zSide, and how and where are they declared?

Surrender

What are xSize ySize zSide, and how and where are they declared?

They are defined at the start -

#define xSize 300

I've tried allocating them from a file but I get the same error either way.

ianhobo

dx=malloc(xSize*sizeof *dx );//Allocate 'xSize' pointer-to-pointer of int

Whats going on with your malloc's?
Is this your realy code or have you changed it for your post?

sizeof is a function, I wouldn't expect to it to compile at all the way you have used it?

Can you post any warnings that your getting?

Surrender

ianhobo wrote: »

Whats going on with your malloc's?
Is this your realy code or have you changed it for your post?

sizeof is a function, I wouldn't expect to it to compile at all the way you have used it?

Can you post any warnings that your getting?

No warnings, I get an error messgae in a box saying that the .exe has stopped working.

I've used that code before to allocate 3d arrays, that's my actual code, if you have an alternative for 3d allocation will you post it up?

ianhobo

Well, actually, this may be the problem...........

your declaring your array as type float,

float is (most likely) 4bytes, so with your sizes, what you actually want
[300 * 4] * [300 * 4] * [300 * 4]
1200 * 1200 * 1200 bytes
= 1728 000 000 -> Almost two gigs.

I know you said you have 4gigs of RAM, but depending on the type of RAM various things may be going on that are beyond your control in terms of memory control. Any memory you want to allocate comes from the heap, and its unlikely that it will be large enough.

Also, windows may simply be deliberately preventing your request in order to keep the system stable and to prevent massive massive paging.

Change your size #defines to something tiny like 5, a quantity that will definitely succeed. This will allow your to prove that your code is right and that the problem lies with the amount of memory you want

ianhobo

Oh yeah,

and you want 30 of them!!

that's over 50gigs of space

It simply isn't going to happen

Surrender

Ok, This code will run with a smaller 3d array, the code works, this I know for a fact.

It will operate with a 300*300*100 allocation for 30 arrays

That's [300*4]*[300*4]*[100*4] = 576 000 000
Around half a gig, doesn't make sense, 30 times this is 17 GB

I thought it was (300*300*100)*4 Which works out at ~36 000 000 = 36 MB

Is this a c problem or should I use c++ with new and free?

ianhobo

Surrender wrote: »

I'm trying to allocate around 30 3d arrays of size 300*300*300.

Your first quote says 300*300*300, not 300*300*100

Which is correct?

My maths is bases on your first post where each dimensions is 300 in size, but yeah, I jumped jumped into it too quick without thinking sorry

even at 36MB per array, 30 of them is over a gig

ianhobo

Create a new program which just allocates small amounts of memory in a loop, and see how far your program will go until the allocations start to fail

This will tell you how much space you are able to allocate from your programs.

Compare the amount of memory that you can allocate here with the points where your array allocations are crashing in your other program

If they are similar, then you're just running out of memory

Surrender

Sorry, I was just using a different example -
300*300*100 works
300*300*300 crashes

I was using the 300*300*100 because it still would require 17GB (according to your calculation).
I'm chancing the maths myself but I think 17GB would cause it to crash as well.

Whereas [300*300*100]*4 = 36MB
30 Arrays , 30*36MB = 1.08GB

So for the 300^3 array: [300^3]*4 = 108 MB for each array
30*108MB = 3.24GB

I think my compiler won't be able to take 3.24GB.

Surrender

Max allocation size = 1.82GB

You're right, I think I need 3.24GB, Out of memory. None of this carry on in Matlab .

Can I allocate more memory in the compiler? Don't think the GNU compiler will let me.

ianhobo

No no, my calculation was wrong, it wouldn't be that much.

There are quite a few issues here which may all be playing a part in why this is failing.

global or persistent variable declarations get their memory from the stack
memory requests through malloc or new get their memory from the heap.

For one large request, (from what I've been able to read on the net so far) it looks like there has to be the amount of memory requested, free as contiguous, non-fragmented memory.

Also, regardless of how much ram a system has, stack references are limited to 0x7fff ffff in size, (2 gigs)

As for Matlab, it will be using the WinAPI to manage larger virtual memory spaces and other memory mapping techniques which are specific to the operating system

Do you have to use floats? And do the arrays have to be so big?
Are you using every array location?
Could you work with file data instead?

phil

ianhobo wrote: »

your declaring your array as type float,

float is (most likely) 4bytes, so with your sizes, what you actually want
[300 * 4] * [300 * 4] * [300 * 4]
1200 * 1200 * 1200 bytes
= 1728 000 000 -> Almost two gigs.

Hmmmm? Your logic ( & math) is a little off here.

A 2x2 array of floats say, tmp[2][2] holds four values. If those four values are 4 bytes, then the space consumed is 16 bytes.

By your maths, it would be (2x4) * (2x4) (which is 64 btw) instead of (2x2)x4.

To the OP:

Malloc will allocate on the heap. Your original allocation will happen on the stack. The stack isn't infinite in size and is the stack size is normally defined by the program on Windows.

I'm not a Windows programmer, but there are probably utilities out there to increase to edit the stack size of binaries. The proper way to do it of course is at compile time and there are probably some compiler options to increase the stack size limit (whether they be switches or macros).

Phil.

ianhobo

Yes yes I've said a few times that I was wrong
I'm sorry

Surrender

Thanks again folks for the feedback,

I've dug around and I can only increase the stack size in Visual Studio using an FSTACK command.
I can't seem to find a way to get Vista to allocate more stack space.
I tested the available stack on a 32 bit Vista and a Vista Ultimate 64 with 4GB Ram:
32: 1.08GB
64: 1.64GB
Not the huge increase I would have thought.

If all else fails I'll have to use Vis Studio 2005 (even though I detest it).


IanHobo, I have to use floats (either that or doubles). And I need all the arrays, there's no avoiding that, it's just the kind of algorithm involved.

Could I save the arrays to files and use read and write to the file? It'll slow things up dramatically but it would still beat Matlab for pace.

conceited

Would this idea be any use to you.
Declare a fraction of the memory you wanted , use it and take down the stack frame and repeat?

Diom

Do you really have to declare all 30 of them at once?
Edit: Nevermind just saw your reply to Ianhobo above

What are you trying to do? Perhaps people can suggest an alternative methodology. I realise that this would mean changing the Matlab as well if you are going to compare speeds.

Surrender

I don't see how I can change the code.
I'm modelling a 3 Dimensional space with multiple magnetic fields.

ianhobo

sure, but do all 30 arrays have to be in active memory at the one time?

Do the arrays have to be processed in parallel? Or could you complete the computations for say the first array, then move on to the second?

Are the array accesses random? Could any were from [0][0][0] - [300][300][300] be potential accessed, or are the array accesses maybe sequential?

I presume one array models one 3d space, with the other 29 for representing different models? Are these models independent of each other?