Python Memory Error using Numpy only with Ubuntu?
up vote
1
down vote
favorite
I have a strange memory error when using Numpy on an Ubuntu server. The server has 64GB of RAM and 250GB additional swap space and memlock is unlimited.
However, if I type
import numpy as np
x = 1000000
np.zeros((x,x))
I get a memory error. If I do the same on my 16GB Ram Macbook it works fine.
How can this happen? I want to use the server for numerical experiments and it just doesn't work. The experiments run fine on my Macbook, and the Python process uses only 2GB of RAM. Is there a way to prevent the Memory Error on the server?
python numpy out-of-memory ubuntu-18.04
asked Nov 11 at 14:55
TheWaveLad
3371621
add a comment |
up vote
1
down vote
favorite
I have a strange memory error when using Numpy on an Ubuntu server. The server has 64GB of RAM and 250GB additional swap space and memlock is unlimited.
However, if I type
import numpy as np
x = 1000000
np.zeros((x,x))
I get a memory error. If I do the same on my 16GB Ram Macbook it works fine.
How can this happen? I want to use the server for numerical experiments and it just doesn't work. The experiments run fine on my Macbook, and the Python process uses only 2GB of RAM. Is there a way to prevent the Memory Error on the server?
python numpy out-of-memory ubuntu-18.04
asked Nov 11 at 14:55
TheWaveLad
3371621
Given you fill the array with values, it will produce an error on both machines, since here you will need at least 1'000GB of memory to store the values if these are bytes, let alone if these are doubles (then that will result in 8'000GB).
– Willem Van Onsem
Nov 11 at 14:59
So the only reason why this might work here, is because it used a sparse matrix. But from the moment you start "filling" the matrix with values, that will result in trouble.
– Willem Van Onsem
Nov 11 at 15:02
So Mac OS seems to do a different kind of memory allocation? the resulting matrix will indeed be sparse, and it would be better to use sparse matrices from scipy, but the different behavior on the two machines is strange.
– TheWaveLad
Nov 11 at 15:05
"If I do the same on my 16GB Ram Macbook it works fine." It surely does not work as fine as you think. Try, for example,z = np.zeros(1000000, 1000000); z[...] = 1. That is, actually write data to all the elements ofz.
– Warren Weckesser
Nov 11 at 15:29
2
Also, numpy does not create a sparse matrix on Mac OS. Numpy does not have sparse matrices. The difference is in how Mac OS handles memory allocation.
– Warren Weckesser
Nov 11 at 15:33
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have a strange memory error when using Numpy on an Ubuntu server. The server has 64GB of RAM and 250GB additional swap space and memlock is unlimited.
However, if I type
import numpy as np
x = 1000000
np.zeros((x,x))
I get a memory error. If I do the same on my 16GB Ram Macbook it works fine.
How can this happen? I want to use the server for numerical experiments and it just doesn't work. The experiments run fine on my Macbook, and the Python process uses only 2GB of RAM. Is there a way to prevent the Memory Error on the server?
python numpy out-of-memory ubuntu-18.04
asked Nov 11 at 14:55
TheWaveLad
3371621
I have a strange memory error when using Numpy on an Ubuntu server. The server has 64GB of RAM and 250GB additional swap space and memlock is unlimited.
However, if I type
import numpy as np
x = 1000000
np.zeros((x,x))
I get a memory error. If I do the same on my 16GB Ram Macbook it works fine.
How can this happen? I want to use the server for numerical experiments and it just doesn't work. The experiments run fine on my Macbook, and the Python process uses only 2GB of RAM. Is there a way to prevent the Memory Error on the server?
python numpy out-of-memory ubuntu-18.04
python numpy out-of-memory ubuntu-18.04
asked Nov 11 at 14:55
TheWaveLad
3371621
asked Nov 11 at 14:55
TheWaveLad
3371621
asked Nov 11 at 14:55
TheWaveLad
3371621
asked Nov 11 at 14:55
TheWaveLad
3371621
asked Nov 11 at 14:55
TheWaveLad
3371621
3371621
Given you fill the array with values, it will produce an error on both machines, since here you will need at least 1'000GB of memory to store the values if these are bytes, let alone if these are doubles (then that will result in 8'000GB).
– Willem Van Onsem
Nov 11 at 14:59
So the only reason why this might work here, is because it used a sparse matrix. But from the moment you start "filling" the matrix with values, that will result in trouble.
– Willem Van Onsem
Nov 11 at 15:02
So Mac OS seems to do a different kind of memory allocation? the resulting matrix will indeed be sparse, and it would be better to use sparse matrices from scipy, but the different behavior on the two machines is strange.
– TheWaveLad
Nov 11 at 15:05
"If I do the same on my 16GB Ram Macbook it works fine." It surely does not work as fine as you think. Try, for example,z = np.zeros(1000000, 1000000); z[...] = 1. That is, actually write data to all the elements ofz.
– Warren Weckesser
Nov 11 at 15:29
2
Also, numpy does not create a sparse matrix on Mac OS. Numpy does not have sparse matrices. The difference is in how Mac OS handles memory allocation.
– Warren Weckesser
Nov 11 at 15:33
add a comment |
Given you fill the array with values, it will produce an error on both machines, since here you will need at least 1'000GB of memory to store the values if these are bytes, let alone if these are doubles (then that will result in 8'000GB).
– Willem Van Onsem
Nov 11 at 14:59
So the only reason why this might work here, is because it used a sparse matrix. But from the moment you start "filling" the matrix with values, that will result in trouble.
– Willem Van Onsem
Nov 11 at 15:02
So Mac OS seems to do a different kind of memory allocation? the resulting matrix will indeed be sparse, and it would be better to use sparse matrices from scipy, but the different behavior on the two machines is strange.
– TheWaveLad
Nov 11 at 15:05
"If I do the same on my 16GB Ram Macbook it works fine." It surely does not work as fine as you think. Try, for example,z = np.zeros(1000000, 1000000); z[...] = 1. That is, actually write data to all the elements ofz.
– Warren Weckesser
Nov 11 at 15:29
2
Also, numpy does not create a sparse matrix on Mac OS. Numpy does not have sparse matrices. The difference is in how Mac OS handles memory allocation.
– Warren Weckesser
Nov 11 at 15:33
Given you fill the array with values, it will produce an error on both machines, since here you will need at least 1'000GB of memory to store the values if these are bytes, let alone if these are doubles (then that will result in 8'000GB).
– Willem Van Onsem
Nov 11 at 14:59
Given you fill the array with values, it will produce an error on both machines, since here you will need at least 1'000GB of memory to store the values if these are bytes, let alone if these are doubles (then that will result in 8'000GB).
– Willem Van Onsem
Nov 11 at 14:59
So the only reason why this might work here, is because it used a sparse matrix. But from the moment you start "filling" the matrix with values, that will result in trouble.
– Willem Van Onsem
Nov 11 at 15:02
So the only reason why this might work here, is because it used a sparse matrix. But from the moment you start "filling" the matrix with values, that will result in trouble.
– Willem Van Onsem
Nov 11 at 15:02
So Mac OS seems to do a different kind of memory allocation? the resulting matrix will indeed be sparse, and it would be better to use sparse matrices from scipy, but the different behavior on the two machines is strange.
– TheWaveLad
Nov 11 at 15:05
So Mac OS seems to do a different kind of memory allocation? the resulting matrix will indeed be sparse, and it would be better to use sparse matrices from scipy, but the different behavior on the two machines is strange.
– TheWaveLad
Nov 11 at 15:05
"If I do the same on my 16GB Ram Macbook it works fine." It surely does not work as fine as you think. Try, for example,
z = np.zeros(1000000, 1000000); z[...] = 1. That is, actually write data to all the elements of z.– Warren Weckesser
Nov 11 at 15:29
"If I do the same on my 16GB Ram Macbook it works fine." It surely does not work as fine as you think. Try, for example,
z = np.zeros(1000000, 1000000); z[...] = 1. That is, actually write data to all the elements of z.– Warren Weckesser
Nov 11 at 15:29
2
2
Also, numpy does not create a sparse matrix on Mac OS. Numpy does not have sparse matrices. The difference is in how Mac OS handles memory allocation.
– Warren Weckesser
Nov 11 at 15:33
Also, numpy does not create a sparse matrix on Mac OS. Numpy does not have sparse matrices. The difference is in how Mac OS handles memory allocation.
– Warren Weckesser
Nov 11 at 15:33
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53249931%2fpython-memory-error-using-numpy-only-with-ubuntu%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Given you fill the array with values, it will produce an error on both machines, since here you will need at least 1'000GB of memory to store the values if these are bytes, let alone if these are doubles (then that will result in 8'000GB).
– Willem Van Onsem
Nov 11 at 14:59
So the only reason why this might work here, is because it used a sparse matrix. But from the moment you start "filling" the matrix with values, that will result in trouble.
– Willem Van Onsem
Nov 11 at 15:02
So Mac OS seems to do a different kind of memory allocation? the resulting matrix will indeed be sparse, and it would be better to use sparse matrices from scipy, but the different behavior on the two machines is strange.
– TheWaveLad
Nov 11 at 15:05
"If I do the same on my 16GB Ram Macbook it works fine." It surely does not work as fine as you think. Try, for example,
z = np.zeros(1000000, 1000000); z[...] = 1. That is, actually write data to all the elements ofz.– Warren Weckesser
Nov 11 at 15:29
2
Also, numpy does not create a sparse matrix on Mac OS. Numpy does not have sparse matrices. The difference is in how Mac OS handles memory allocation.
– Warren Weckesser
Nov 11 at 15:33