Image inaccessible from python script but accessible in browser
up vote
1
down vote
favorite
I was trying to automatically download some traffic camera photos to play around with some image object recognition scripts and I have found that some links to them will throw a 403: Forbidden error when I try to download them from Python, and yet I can access them in a browser. One such image is at this link: https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg
This code:
urllib.request.urlretrieve("https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg", "traffic.jpg")
Returns a 403 error for me. What gives? I can understand that perhaps these organizations are not keen on having people bog down their servers with automatic downloads and perhaps there are some GDPR-related constraints, but I am actually more curious about how they are able to detect that the request is coming from a script and not from normal use
python-3.x urllib http-status-code-403
asked Nov 11 at 9:19
1saac
3581415
add a comment |
up vote
1
down vote
favorite
I was trying to automatically download some traffic camera photos to play around with some image object recognition scripts and I have found that some links to them will throw a 403: Forbidden error when I try to download them from Python, and yet I can access them in a browser. One such image is at this link: https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg
This code:
urllib.request.urlretrieve("https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg", "traffic.jpg")
Returns a 403 error for me. What gives? I can understand that perhaps these organizations are not keen on having people bog down their servers with automatic downloads and perhaps there are some GDPR-related constraints, but I am actually more curious about how they are able to detect that the request is coming from a script and not from normal use
python-3.x urllib http-status-code-403
asked Nov 11 at 9:19
1saac
3581415
quite a few websites check the user-agent http header and block exotic ones. You can check if it's the case here by setting this header to a well known value (there are lists of common browsers UA on the web).
– bruno desthuilliers
Nov 11 at 9:26
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I was trying to automatically download some traffic camera photos to play around with some image object recognition scripts and I have found that some links to them will throw a 403: Forbidden error when I try to download them from Python, and yet I can access them in a browser. One such image is at this link: https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg
This code:
urllib.request.urlretrieve("https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg", "traffic.jpg")
Returns a 403 error for me. What gives? I can understand that perhaps these organizations are not keen on having people bog down their servers with automatic downloads and perhaps there are some GDPR-related constraints, but I am actually more curious about how they are able to detect that the request is coming from a script and not from normal use
python-3.x urllib http-status-code-403
asked Nov 11 at 9:19
1saac
3581415
I was trying to automatically download some traffic camera photos to play around with some image object recognition scripts and I have found that some links to them will throw a 403: Forbidden error when I try to download them from Python, and yet I can access them in a browser. One such image is at this link: https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg
This code:
urllib.request.urlretrieve("https://www.svz-bw.de/kamera/ftpdata/KA101/KA101_gross.jpg", "traffic.jpg")
Returns a 403 error for me. What gives? I can understand that perhaps these organizations are not keen on having people bog down their servers with automatic downloads and perhaps there are some GDPR-related constraints, but I am actually more curious about how they are able to detect that the request is coming from a script and not from normal use
python-3.x urllib http-status-code-403
python-3.x urllib http-status-code-403
asked Nov 11 at 9:19
1saac
3581415
asked Nov 11 at 9:19
1saac
3581415
asked Nov 11 at 9:19
1saac
3581415
asked Nov 11 at 9:19
1saac
3581415
asked Nov 11 at 9:19
1saac
3581415
3581415
quite a few websites check the user-agent http header and block exotic ones. You can check if it's the case here by setting this header to a well known value (there are lists of common browsers UA on the web).
– bruno desthuilliers
Nov 11 at 9:26
add a comment |
quite a few websites check the user-agent http header and block exotic ones. You can check if it's the case here by setting this header to a well known value (there are lists of common browsers UA on the web).
– bruno desthuilliers
Nov 11 at 9:26
quite a few websites check the user-agent http header and block exotic ones. You can check if it's the case here by setting this header to a well known value (there are lists of common browsers UA on the web).
– bruno desthuilliers
Nov 11 at 9:26
quite a few websites check the user-agent http header and block exotic ones. You can check if it's the case here by setting this header to a well known value (there are lists of common browsers UA on the web).
– bruno desthuilliers
Nov 11 at 9:26
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53247345%2fimage-inaccessible-from-python-script-but-accessible-in-browser%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
quite a few websites check the user-agent http header and block exotic ones. You can check if it's the case here by setting this header to a well known value (there are lists of common browsers UA on the web).
– bruno desthuilliers
Nov 11 at 9:26