Process MemoryError on Server But Not Local Machine
I have a .Net Console app that spawns a Process to execute a python script. The input is a relatively large input, around 1GB, that goes through some image processing steps. The python script is wrapped in a try/except block that just returns a message for the outcome (hacky but such is life). The problem I am finding is that the file will not process on 2 separate servers, but it will process without error on my local machine.
Server specs: Windows Server 2012 R2 Standard, quad core Intel Xeon, with 32GB RAM -- AWS EC2
Local Machine: Windows 10 Pro, quad core i7, 16GB RAM -- dev machine/laptop
Python is running in a virtual environment with the same requirement packages in both instances, the code is the same, but something is killing the process when running on an AWS EC2 server instance.
I can see the RAM spike in the Task Manager on my local machine, but when I run it on the servers, the python process is short lived enough that I cannot find it nor see it spike RAM in the Task Manager.
I am getting the error message MemoryError back so I know python is at least starting the script and hitting the error. Locally, python spikes to about 10-11GB RAM usage when starting so I wouldn't expect RAM to really be a limiting factor.
Is there some system policy or other protection device in Windows Server software that prevents python allocating the necessary RAM? Nothing is running through an AppPool, it is a simple Console .exe program. What could be causing python to die like that?
Edit:
I have been going line-by-line through the python script I have. The error seems to originate on: someVar = np.zeroes([largeNumber, largeNumber2]).astype('uint8')... From what I gather of numpy's documentation, the astype method creates a copy in memory.
It still doesn't help me figure out why it runs locally and not on a server though.
python .net out-of-memory windows-server-2012-r2
asked Nov 15 '18 at 17:35
BrancoBranco
142115
add a comment |
I have a .Net Console app that spawns a Process to execute a python script. The input is a relatively large input, around 1GB, that goes through some image processing steps. The python script is wrapped in a try/except block that just returns a message for the outcome (hacky but such is life). The problem I am finding is that the file will not process on 2 separate servers, but it will process without error on my local machine.
Server specs: Windows Server 2012 R2 Standard, quad core Intel Xeon, with 32GB RAM -- AWS EC2
Local Machine: Windows 10 Pro, quad core i7, 16GB RAM -- dev machine/laptop
Python is running in a virtual environment with the same requirement packages in both instances, the code is the same, but something is killing the process when running on an AWS EC2 server instance.
I can see the RAM spike in the Task Manager on my local machine, but when I run it on the servers, the python process is short lived enough that I cannot find it nor see it spike RAM in the Task Manager.
I am getting the error message MemoryError back so I know python is at least starting the script and hitting the error. Locally, python spikes to about 10-11GB RAM usage when starting so I wouldn't expect RAM to really be a limiting factor.
Is there some system policy or other protection device in Windows Server software that prevents python allocating the necessary RAM? Nothing is running through an AppPool, it is a simple Console .exe program. What could be causing python to die like that?
Edit:
I have been going line-by-line through the python script I have. The error seems to originate on: someVar = np.zeroes([largeNumber, largeNumber2]).astype('uint8')... From what I gather of numpy's documentation, the astype method creates a copy in memory.
It still doesn't help me figure out why it runs locally and not on a server though.
python .net out-of-memory windows-server-2012-r2
asked Nov 15 '18 at 17:35
BrancoBranco
142115
If you are running on a shared server (i.e. a server you don't control and on which other users also run), there are likely numerous policies in place to prevent programs from using up all of available memory. I'm guessing that's what you're running into (some sort of quota).
– Jonah Bishop
Nov 15 '18 at 17:38
It is a VPS and from what I've seen locally, there should be more than enough allocated RAM for it to process. Also the test one is a local server at our facility and not AWS.
– Branco
Nov 15 '18 at 17:43
add a comment |
I have a .Net Console app that spawns a Process to execute a python script. The input is a relatively large input, around 1GB, that goes through some image processing steps. The python script is wrapped in a try/except block that just returns a message for the outcome (hacky but such is life). The problem I am finding is that the file will not process on 2 separate servers, but it will process without error on my local machine.
Server specs: Windows Server 2012 R2 Standard, quad core Intel Xeon, with 32GB RAM -- AWS EC2
Local Machine: Windows 10 Pro, quad core i7, 16GB RAM -- dev machine/laptop
Python is running in a virtual environment with the same requirement packages in both instances, the code is the same, but something is killing the process when running on an AWS EC2 server instance.
I can see the RAM spike in the Task Manager on my local machine, but when I run it on the servers, the python process is short lived enough that I cannot find it nor see it spike RAM in the Task Manager.
I am getting the error message MemoryError back so I know python is at least starting the script and hitting the error. Locally, python spikes to about 10-11GB RAM usage when starting so I wouldn't expect RAM to really be a limiting factor.
Is there some system policy or other protection device in Windows Server software that prevents python allocating the necessary RAM? Nothing is running through an AppPool, it is a simple Console .exe program. What could be causing python to die like that?
Edit:
I have been going line-by-line through the python script I have. The error seems to originate on: someVar = np.zeroes([largeNumber, largeNumber2]).astype('uint8')... From what I gather of numpy's documentation, the astype method creates a copy in memory.
It still doesn't help me figure out why it runs locally and not on a server though.
python .net out-of-memory windows-server-2012-r2
asked Nov 15 '18 at 17:35
BrancoBranco
142115
I have a .Net Console app that spawns a Process to execute a python script. The input is a relatively large input, around 1GB, that goes through some image processing steps. The python script is wrapped in a try/except block that just returns a message for the outcome (hacky but such is life). The problem I am finding is that the file will not process on 2 separate servers, but it will process without error on my local machine.
Server specs: Windows Server 2012 R2 Standard, quad core Intel Xeon, with 32GB RAM -- AWS EC2
Local Machine: Windows 10 Pro, quad core i7, 16GB RAM -- dev machine/laptop
Python is running in a virtual environment with the same requirement packages in both instances, the code is the same, but something is killing the process when running on an AWS EC2 server instance.
I can see the RAM spike in the Task Manager on my local machine, but when I run it on the servers, the python process is short lived enough that I cannot find it nor see it spike RAM in the Task Manager.
I am getting the error message MemoryError back so I know python is at least starting the script and hitting the error. Locally, python spikes to about 10-11GB RAM usage when starting so I wouldn't expect RAM to really be a limiting factor.
Is there some system policy or other protection device in Windows Server software that prevents python allocating the necessary RAM? Nothing is running through an AppPool, it is a simple Console .exe program. What could be causing python to die like that?
Edit:
I have been going line-by-line through the python script I have. The error seems to originate on: someVar = np.zeroes([largeNumber, largeNumber2]).astype('uint8')... From what I gather of numpy's documentation, the astype method creates a copy in memory.
It still doesn't help me figure out why it runs locally and not on a server though.
python .net out-of-memory windows-server-2012-r2
python .net out-of-memory windows-server-2012-r2
asked Nov 15 '18 at 17:35
BrancoBranco
142115
asked Nov 15 '18 at 17:35
BrancoBranco
142115
edited Nov 15 '18 at 19:29
Branco
asked Nov 15 '18 at 17:35
BrancoBranco
142115
asked Nov 15 '18 at 17:35
BrancoBranco
142115
asked Nov 15 '18 at 17:35
BrancoBranco
142115
142115
If you are running on a shared server (i.e. a server you don't control and on which other users also run), there are likely numerous policies in place to prevent programs from using up all of available memory. I'm guessing that's what you're running into (some sort of quota).
– Jonah Bishop
Nov 15 '18 at 17:38
It is a VPS and from what I've seen locally, there should be more than enough allocated RAM for it to process. Also the test one is a local server at our facility and not AWS.
– Branco
Nov 15 '18 at 17:43
add a comment |
If you are running on a shared server (i.e. a server you don't control and on which other users also run), there are likely numerous policies in place to prevent programs from using up all of available memory. I'm guessing that's what you're running into (some sort of quota).
– Jonah Bishop
Nov 15 '18 at 17:38
It is a VPS and from what I've seen locally, there should be more than enough allocated RAM for it to process. Also the test one is a local server at our facility and not AWS.
– Branco
Nov 15 '18 at 17:43
If you are running on a shared server (i.e. a server you don't control and on which other users also run), there are likely numerous policies in place to prevent programs from using up all of available memory. I'm guessing that's what you're running into (some sort of quota).
– Jonah Bishop
Nov 15 '18 at 17:38
If you are running on a shared server (i.e. a server you don't control and on which other users also run), there are likely numerous policies in place to prevent programs from using up all of available memory. I'm guessing that's what you're running into (some sort of quota).
– Jonah Bishop
Nov 15 '18 at 17:38
It is a VPS and from what I've seen locally, there should be more than enough allocated RAM for it to process. Also the test one is a local server at our facility and not AWS.
– Branco
Nov 15 '18 at 17:43
It is a VPS and from what I've seen locally, there should be more than enough allocated RAM for it to process. Also the test one is a local server at our facility and not AWS.
– Branco
Nov 15 '18 at 17:43
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325033%2fprocess-memoryerror-on-server-but-not-local-machine%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325033%2fprocess-memoryerror-on-server-but-not-local-machine%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If you are running on a shared server (i.e. a server you don't control and on which other users also run), there are likely numerous policies in place to prevent programs from using up all of available memory. I'm guessing that's what you're running into (some sort of quota).
– Jonah Bishop
Nov 15 '18 at 17:38
It is a VPS and from what I've seen locally, there should be more than enough allocated RAM for it to process. Also the test one is a local server at our facility and not AWS.
– Branco
Nov 15 '18 at 17:43