Forcing cURL to get a password from the environment
up vote
11
down vote
favorite
This question about using cURL with a username and password has suboptimal answers for me:
curl -u "user:pw" https://example.com
puts the pw in the process listcurl "https://user:pw@example.com"
puts the pw in the process listcurl -u "user:$(cat ~/.passwd)" https://example.com
puts the pw in the process listcurl -u user https://example.com
prompts for the pwcurl --netrc-file ~/.netrc https://example.com
requires a file
#4 is secure, but I might run this command hundreds of times a day, so it's tedious. #5 is close to secure, but that file could be read by somebody with root access.
The cURL man page says (note the bold text):
-u/--user <user:password>
Specify the user name and password to use for server authentication.
Overrides-n/--netrc
and--netrc-optional
.
If you just give the user name (without entering a colon) curl will
prompt for a password.
If you use an SSPI-enabled curl binary and do NTLM authentication, you
can force curl to pick up the user name and password from your environment
by simply specifying a single colon with this option:-u :
.
I've tried setting $USER
and $PASSWORD
(and $CURLOPT_PASSWORD
and others) in the environment, but cURL doesn't pick up either of them when invoked as curl -u : https://example.com
(nor does it work without the -u :
).
I'm not doing NTLM, so this doesn't work. Unless I'm missing something.
Is there a way to pass credentials to curl
solely through the environment?
(Workaround moved to an answer)
bash security curl environment-variables sh
|
show 7 more comments
up vote
11
down vote
favorite
This question about using cURL with a username and password has suboptimal answers for me:
curl -u "user:pw" https://example.com
puts the pw in the process listcurl "https://user:pw@example.com"
puts the pw in the process listcurl -u "user:$(cat ~/.passwd)" https://example.com
puts the pw in the process listcurl -u user https://example.com
prompts for the pwcurl --netrc-file ~/.netrc https://example.com
requires a file
#4 is secure, but I might run this command hundreds of times a day, so it's tedious. #5 is close to secure, but that file could be read by somebody with root access.
The cURL man page says (note the bold text):
-u/--user <user:password>
Specify the user name and password to use for server authentication.
Overrides-n/--netrc
and--netrc-optional
.
If you just give the user name (without entering a colon) curl will
prompt for a password.
If you use an SSPI-enabled curl binary and do NTLM authentication, you
can force curl to pick up the user name and password from your environment
by simply specifying a single colon with this option:-u :
.
I've tried setting $USER
and $PASSWORD
(and $CURLOPT_PASSWORD
and others) in the environment, but cURL doesn't pick up either of them when invoked as curl -u : https://example.com
(nor does it work without the -u :
).
I'm not doing NTLM, so this doesn't work. Unless I'm missing something.
Is there a way to pass credentials to curl
solely through the environment?
(Workaround moved to an answer)
bash security curl environment-variables sh
1
You don't need perl for that you could just a shell snippet there just as well but note that the process environment is available toroot
just as much as a file is (though for a shorter timeframe potentially).
– Etan Reisner
Nov 19 '15 at 3:47
SSPI and NTLM are both Windows technologies. Presumably thecurl
man page is not talking about retrieving the password from a POSIX-style environment.
– chepner
Nov 19 '15 at 14:51
1
I thinkdeclare -p USER | sed 's/^[^=]*="//;s/"[^"]*$//;s/\"/"/'
satisfies but certainly isn't better than the perl, etc. solution. Do heredocs/herestrings show up in the command? (I don't recall offhand.) If not those would work too.
– Etan Reisner
Nov 19 '15 at 18:23
2
It took me 29 seconds to replacecurl
with a wrapper script that did a strace dump on the original binary, circumventing every single of these techniques including #4. By design, you can not hide from local root.
– that other guy
Nov 19 '15 at 19:19
1
Something like--netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD")
possibly? But once you've hit a sub-shell/etc. avoidingsed
isn't really worth much.
– Etan Reisner
Nov 20 '15 at 13:18
|
show 7 more comments
up vote
11
down vote
favorite
up vote
11
down vote
favorite
This question about using cURL with a username and password has suboptimal answers for me:
curl -u "user:pw" https://example.com
puts the pw in the process listcurl "https://user:pw@example.com"
puts the pw in the process listcurl -u "user:$(cat ~/.passwd)" https://example.com
puts the pw in the process listcurl -u user https://example.com
prompts for the pwcurl --netrc-file ~/.netrc https://example.com
requires a file
#4 is secure, but I might run this command hundreds of times a day, so it's tedious. #5 is close to secure, but that file could be read by somebody with root access.
The cURL man page says (note the bold text):
-u/--user <user:password>
Specify the user name and password to use for server authentication.
Overrides-n/--netrc
and--netrc-optional
.
If you just give the user name (without entering a colon) curl will
prompt for a password.
If you use an SSPI-enabled curl binary and do NTLM authentication, you
can force curl to pick up the user name and password from your environment
by simply specifying a single colon with this option:-u :
.
I've tried setting $USER
and $PASSWORD
(and $CURLOPT_PASSWORD
and others) in the environment, but cURL doesn't pick up either of them when invoked as curl -u : https://example.com
(nor does it work without the -u :
).
I'm not doing NTLM, so this doesn't work. Unless I'm missing something.
Is there a way to pass credentials to curl
solely through the environment?
(Workaround moved to an answer)
bash security curl environment-variables sh
This question about using cURL with a username and password has suboptimal answers for me:
curl -u "user:pw" https://example.com
puts the pw in the process listcurl "https://user:pw@example.com"
puts the pw in the process listcurl -u "user:$(cat ~/.passwd)" https://example.com
puts the pw in the process listcurl -u user https://example.com
prompts for the pwcurl --netrc-file ~/.netrc https://example.com
requires a file
#4 is secure, but I might run this command hundreds of times a day, so it's tedious. #5 is close to secure, but that file could be read by somebody with root access.
The cURL man page says (note the bold text):
-u/--user <user:password>
Specify the user name and password to use for server authentication.
Overrides-n/--netrc
and--netrc-optional
.
If you just give the user name (without entering a colon) curl will
prompt for a password.
If you use an SSPI-enabled curl binary and do NTLM authentication, you
can force curl to pick up the user name and password from your environment
by simply specifying a single colon with this option:-u :
.
I've tried setting $USER
and $PASSWORD
(and $CURLOPT_PASSWORD
and others) in the environment, but cURL doesn't pick up either of them when invoked as curl -u : https://example.com
(nor does it work without the -u :
).
I'm not doing NTLM, so this doesn't work. Unless I'm missing something.
Is there a way to pass credentials to curl
solely through the environment?
(Workaround moved to an answer)
bash security curl environment-variables sh
bash security curl environment-variables sh
edited Nov 20 '15 at 4:26
asked Nov 19 '15 at 3:37
Adam Katz
5,55013351
5,55013351
1
You don't need perl for that you could just a shell snippet there just as well but note that the process environment is available toroot
just as much as a file is (though for a shorter timeframe potentially).
– Etan Reisner
Nov 19 '15 at 3:47
SSPI and NTLM are both Windows technologies. Presumably thecurl
man page is not talking about retrieving the password from a POSIX-style environment.
– chepner
Nov 19 '15 at 14:51
1
I thinkdeclare -p USER | sed 's/^[^=]*="//;s/"[^"]*$//;s/\"/"/'
satisfies but certainly isn't better than the perl, etc. solution. Do heredocs/herestrings show up in the command? (I don't recall offhand.) If not those would work too.
– Etan Reisner
Nov 19 '15 at 18:23
2
It took me 29 seconds to replacecurl
with a wrapper script that did a strace dump on the original binary, circumventing every single of these techniques including #4. By design, you can not hide from local root.
– that other guy
Nov 19 '15 at 19:19
1
Something like--netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD")
possibly? But once you've hit a sub-shell/etc. avoidingsed
isn't really worth much.
– Etan Reisner
Nov 20 '15 at 13:18
|
show 7 more comments
1
You don't need perl for that you could just a shell snippet there just as well but note that the process environment is available toroot
just as much as a file is (though for a shorter timeframe potentially).
– Etan Reisner
Nov 19 '15 at 3:47
SSPI and NTLM are both Windows technologies. Presumably thecurl
man page is not talking about retrieving the password from a POSIX-style environment.
– chepner
Nov 19 '15 at 14:51
1
I thinkdeclare -p USER | sed 's/^[^=]*="//;s/"[^"]*$//;s/\"/"/'
satisfies but certainly isn't better than the perl, etc. solution. Do heredocs/herestrings show up in the command? (I don't recall offhand.) If not those would work too.
– Etan Reisner
Nov 19 '15 at 18:23
2
It took me 29 seconds to replacecurl
with a wrapper script that did a strace dump on the original binary, circumventing every single of these techniques including #4. By design, you can not hide from local root.
– that other guy
Nov 19 '15 at 19:19
1
Something like--netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD")
possibly? But once you've hit a sub-shell/etc. avoidingsed
isn't really worth much.
– Etan Reisner
Nov 20 '15 at 13:18
1
1
You don't need perl for that you could just a shell snippet there just as well but note that the process environment is available to
root
just as much as a file is (though for a shorter timeframe potentially).– Etan Reisner
Nov 19 '15 at 3:47
You don't need perl for that you could just a shell snippet there just as well but note that the process environment is available to
root
just as much as a file is (though for a shorter timeframe potentially).– Etan Reisner
Nov 19 '15 at 3:47
SSPI and NTLM are both Windows technologies. Presumably the
curl
man page is not talking about retrieving the password from a POSIX-style environment.– chepner
Nov 19 '15 at 14:51
SSPI and NTLM are both Windows technologies. Presumably the
curl
man page is not talking about retrieving the password from a POSIX-style environment.– chepner
Nov 19 '15 at 14:51
1
1
I think
declare -p USER | sed 's/^[^=]*="//;s/"[^"]*$//;s/\"/"/'
satisfies but certainly isn't better than the perl, etc. solution. Do heredocs/herestrings show up in the command? (I don't recall offhand.) If not those would work too.– Etan Reisner
Nov 19 '15 at 18:23
I think
declare -p USER | sed 's/^[^=]*="//;s/"[^"]*$//;s/\"/"/'
satisfies but certainly isn't better than the perl, etc. solution. Do heredocs/herestrings show up in the command? (I don't recall offhand.) If not those would work too.– Etan Reisner
Nov 19 '15 at 18:23
2
2
It took me 29 seconds to replace
curl
with a wrapper script that did a strace dump on the original binary, circumventing every single of these techniques including #4. By design, you can not hide from local root.– that other guy
Nov 19 '15 at 19:19
It took me 29 seconds to replace
curl
with a wrapper script that did a strace dump on the original binary, circumventing every single of these techniques including #4. By design, you can not hide from local root.– that other guy
Nov 19 '15 at 19:19
1
1
Something like
--netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD")
possibly? But once you've hit a sub-shell/etc. avoiding sed
isn't really worth much.– Etan Reisner
Nov 20 '15 at 13:18
Something like
--netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD")
possibly? But once you've hit a sub-shell/etc. avoiding sed
isn't really worth much.– Etan Reisner
Nov 20 '15 at 13:18
|
show 7 more comments
3 Answers
3
active
oldest
votes
up vote
13
down vote
accepted
This bash
solution appears to best fit my needs. It's decently secure, portable, and fast.
#!/bin/bash
SRV="example.com"
URL="https://$SRV/path"
curl --netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD") "$URL"
This uses process substitution (<( command )
runs command
in a sub-shell to populate a file descriptor to be handed as a "file" to the parent command, which in this case is curl
). The process substitution contains a here-string (cat <<< text
, a variant of echo text
that won't put anything into your process list), creating a file descriptor for the netrc file in order to pass credentials to the remote web server.
The security afforded by process substitution is actually pretty sound: its file descriptor is not a temporary file and is unavailable from even other calls in the same shell instance, so this appears secure in this context; an adversary would have to dig through memory or launch a complicated attack to find its contents. Since the $PASSWORD
environment variable is also in memory, this should not increase the attack surface.
As long as you haven't used export PASSWORD
, a trick like ps ewwp $$
shouldn't reveal the password (as noted in this comment). It'd also be wise to use some less obvious variable name.
Here is a simplified insecure version of the above code that may help explain how it works:
#!/bin/sh
# INSECURE VERSION, DO NOT USE
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp)
printf "machine %s login %s password %sn" "$SRV" "$USER" "$PASSWORD" > "$TMP"
curl --netrc-file "$TMP" "$URL"
rm -f "$TMP"
This insecure version has lots of flaws, all of which are solved in the previous version:
- It stores the password in a file (though that file is only readable to you)
- It very briefly has the password in a command line
- The temporary file remains until after
curl
exits
Ctrl+c will quit without removing the temporary file
Some of that could be solved by:
#!/bin/sh
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp /dev/shm/.XXXXX) # assumes /dev/shm is a ramdisk
trap "rm -f $TMP" 0 18
cat << EOF > "$TMP"
machine $SRV login $USER password $PASSWORD
EOF
(sleep 0.1; rm -f "$TMP") & # queue removing temp file in 0.1 seconds
curl --netrc-file "$TMP" "$URL"
I consider this version to be messy, suboptimal, and possibly less secure (though it is more portable). It also requires a version of sleep
that understands decimals (and 0.1 seconds may be too fast if the system is heavily loaded).
I had originally posted a workaround that included a perl
one-liner in my question, then (with help from Etan Reisner) I worked through a few better methods before settling on this here-string method, which is both lighter-weight (faster) and more portable.
At this point, it's elegant enough that I'd consider it the "answer" rather than an "ugly workaround," so I've migrated it to be this official answer. I've given @ghoti a +1 for his answer, which correctly states that cURL's command line program is incapable of doing what I want on its own, but I'm not "accepting" that answer because it doesn't help solve the issue.
add a comment |
up vote
7
down vote
Is there a way to pass credentials to
curl
solely through the environment?
No, I don't think there is.
The CURLOPT_USERPWD documentation I think describes what you need, but this is an option that would be available using the curl library in some other language. PHP, Perl, C, etc.
The curl binary you run from your shell is just another front end on that library, but the way things like CURLOPT_USERPWD get passed to the library through the curl binary is by use of command line options on the binary.
You could theoretically write your own binary as a front end to the curl library, and write in support for environment variables.
You could alternately hack environment support as you're hoping to see it into the existing curl binary, and compile your own with local functions.
Beware, though, that even environment variables may be leaked by your shell into the process table. (What do you see when you run ps ewwp $$
?)
Perhaps a .netrc file with restricted permissions will be the safest way to go. Perhaps you will need to generate a temporary .netrc file to be used by the --netrc-file
curl option.
I think you either have to pick the least risky solution for your environment, or write something in a real language that does security properly.
1
foo=bar; export foo; ps ewwp $$ |grep -i foo
finds the variable inbash
but interestingly finds nothing inzsh
(my shell of choice). (Withoutexport
,ps ewwp $$
doesn't find the variable inbash
orzsh
, butperl
doesn't pick it up either, making it useless.) The bash/zsh process substitution (<(…)
) in my workaround appears to do a good job of restricting access to that very temporary "file" (though I'm sure it still lives in memory).
– Adam Katz
Nov 19 '15 at 18:53
I've moved my workaround to an answer and alleviated the need forexport
which makes the variable invisible tops ewwp
– Adam Katz
Nov 20 '15 at 4:29
add a comment |
up vote
1
down vote
User "Tom, Bom" provides a decent solution for this here: https://coderwall.com/p/dsfmwa/securely-use-basic-auth-with-curl
curl --config - https://example.com <<< 'user = "username:password"'
This prevents passwords from showing up in the process list, though this does not specifically address the OP's original question:
Is there a way to pass credentials to curl solely through the environment?
I still give points to @ghoti for giving a more comprehensive and informative answer.
add a comment |
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
13
down vote
accepted
This bash
solution appears to best fit my needs. It's decently secure, portable, and fast.
#!/bin/bash
SRV="example.com"
URL="https://$SRV/path"
curl --netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD") "$URL"
This uses process substitution (<( command )
runs command
in a sub-shell to populate a file descriptor to be handed as a "file" to the parent command, which in this case is curl
). The process substitution contains a here-string (cat <<< text
, a variant of echo text
that won't put anything into your process list), creating a file descriptor for the netrc file in order to pass credentials to the remote web server.
The security afforded by process substitution is actually pretty sound: its file descriptor is not a temporary file and is unavailable from even other calls in the same shell instance, so this appears secure in this context; an adversary would have to dig through memory or launch a complicated attack to find its contents. Since the $PASSWORD
environment variable is also in memory, this should not increase the attack surface.
As long as you haven't used export PASSWORD
, a trick like ps ewwp $$
shouldn't reveal the password (as noted in this comment). It'd also be wise to use some less obvious variable name.
Here is a simplified insecure version of the above code that may help explain how it works:
#!/bin/sh
# INSECURE VERSION, DO NOT USE
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp)
printf "machine %s login %s password %sn" "$SRV" "$USER" "$PASSWORD" > "$TMP"
curl --netrc-file "$TMP" "$URL"
rm -f "$TMP"
This insecure version has lots of flaws, all of which are solved in the previous version:
- It stores the password in a file (though that file is only readable to you)
- It very briefly has the password in a command line
- The temporary file remains until after
curl
exits
Ctrl+c will quit without removing the temporary file
Some of that could be solved by:
#!/bin/sh
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp /dev/shm/.XXXXX) # assumes /dev/shm is a ramdisk
trap "rm -f $TMP" 0 18
cat << EOF > "$TMP"
machine $SRV login $USER password $PASSWORD
EOF
(sleep 0.1; rm -f "$TMP") & # queue removing temp file in 0.1 seconds
curl --netrc-file "$TMP" "$URL"
I consider this version to be messy, suboptimal, and possibly less secure (though it is more portable). It also requires a version of sleep
that understands decimals (and 0.1 seconds may be too fast if the system is heavily loaded).
I had originally posted a workaround that included a perl
one-liner in my question, then (with help from Etan Reisner) I worked through a few better methods before settling on this here-string method, which is both lighter-weight (faster) and more portable.
At this point, it's elegant enough that I'd consider it the "answer" rather than an "ugly workaround," so I've migrated it to be this official answer. I've given @ghoti a +1 for his answer, which correctly states that cURL's command line program is incapable of doing what I want on its own, but I'm not "accepting" that answer because it doesn't help solve the issue.
add a comment |
up vote
13
down vote
accepted
This bash
solution appears to best fit my needs. It's decently secure, portable, and fast.
#!/bin/bash
SRV="example.com"
URL="https://$SRV/path"
curl --netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD") "$URL"
This uses process substitution (<( command )
runs command
in a sub-shell to populate a file descriptor to be handed as a "file" to the parent command, which in this case is curl
). The process substitution contains a here-string (cat <<< text
, a variant of echo text
that won't put anything into your process list), creating a file descriptor for the netrc file in order to pass credentials to the remote web server.
The security afforded by process substitution is actually pretty sound: its file descriptor is not a temporary file and is unavailable from even other calls in the same shell instance, so this appears secure in this context; an adversary would have to dig through memory or launch a complicated attack to find its contents. Since the $PASSWORD
environment variable is also in memory, this should not increase the attack surface.
As long as you haven't used export PASSWORD
, a trick like ps ewwp $$
shouldn't reveal the password (as noted in this comment). It'd also be wise to use some less obvious variable name.
Here is a simplified insecure version of the above code that may help explain how it works:
#!/bin/sh
# INSECURE VERSION, DO NOT USE
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp)
printf "machine %s login %s password %sn" "$SRV" "$USER" "$PASSWORD" > "$TMP"
curl --netrc-file "$TMP" "$URL"
rm -f "$TMP"
This insecure version has lots of flaws, all of which are solved in the previous version:
- It stores the password in a file (though that file is only readable to you)
- It very briefly has the password in a command line
- The temporary file remains until after
curl
exits
Ctrl+c will quit without removing the temporary file
Some of that could be solved by:
#!/bin/sh
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp /dev/shm/.XXXXX) # assumes /dev/shm is a ramdisk
trap "rm -f $TMP" 0 18
cat << EOF > "$TMP"
machine $SRV login $USER password $PASSWORD
EOF
(sleep 0.1; rm -f "$TMP") & # queue removing temp file in 0.1 seconds
curl --netrc-file "$TMP" "$URL"
I consider this version to be messy, suboptimal, and possibly less secure (though it is more portable). It also requires a version of sleep
that understands decimals (and 0.1 seconds may be too fast if the system is heavily loaded).
I had originally posted a workaround that included a perl
one-liner in my question, then (with help from Etan Reisner) I worked through a few better methods before settling on this here-string method, which is both lighter-weight (faster) and more portable.
At this point, it's elegant enough that I'd consider it the "answer" rather than an "ugly workaround," so I've migrated it to be this official answer. I've given @ghoti a +1 for his answer, which correctly states that cURL's command line program is incapable of doing what I want on its own, but I'm not "accepting" that answer because it doesn't help solve the issue.
add a comment |
up vote
13
down vote
accepted
up vote
13
down vote
accepted
This bash
solution appears to best fit my needs. It's decently secure, portable, and fast.
#!/bin/bash
SRV="example.com"
URL="https://$SRV/path"
curl --netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD") "$URL"
This uses process substitution (<( command )
runs command
in a sub-shell to populate a file descriptor to be handed as a "file" to the parent command, which in this case is curl
). The process substitution contains a here-string (cat <<< text
, a variant of echo text
that won't put anything into your process list), creating a file descriptor for the netrc file in order to pass credentials to the remote web server.
The security afforded by process substitution is actually pretty sound: its file descriptor is not a temporary file and is unavailable from even other calls in the same shell instance, so this appears secure in this context; an adversary would have to dig through memory or launch a complicated attack to find its contents. Since the $PASSWORD
environment variable is also in memory, this should not increase the attack surface.
As long as you haven't used export PASSWORD
, a trick like ps ewwp $$
shouldn't reveal the password (as noted in this comment). It'd also be wise to use some less obvious variable name.
Here is a simplified insecure version of the above code that may help explain how it works:
#!/bin/sh
# INSECURE VERSION, DO NOT USE
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp)
printf "machine %s login %s password %sn" "$SRV" "$USER" "$PASSWORD" > "$TMP"
curl --netrc-file "$TMP" "$URL"
rm -f "$TMP"
This insecure version has lots of flaws, all of which are solved in the previous version:
- It stores the password in a file (though that file is only readable to you)
- It very briefly has the password in a command line
- The temporary file remains until after
curl
exits
Ctrl+c will quit without removing the temporary file
Some of that could be solved by:
#!/bin/sh
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp /dev/shm/.XXXXX) # assumes /dev/shm is a ramdisk
trap "rm -f $TMP" 0 18
cat << EOF > "$TMP"
machine $SRV login $USER password $PASSWORD
EOF
(sleep 0.1; rm -f "$TMP") & # queue removing temp file in 0.1 seconds
curl --netrc-file "$TMP" "$URL"
I consider this version to be messy, suboptimal, and possibly less secure (though it is more portable). It also requires a version of sleep
that understands decimals (and 0.1 seconds may be too fast if the system is heavily loaded).
I had originally posted a workaround that included a perl
one-liner in my question, then (with help from Etan Reisner) I worked through a few better methods before settling on this here-string method, which is both lighter-weight (faster) and more portable.
At this point, it's elegant enough that I'd consider it the "answer" rather than an "ugly workaround," so I've migrated it to be this official answer. I've given @ghoti a +1 for his answer, which correctly states that cURL's command line program is incapable of doing what I want on its own, but I'm not "accepting" that answer because it doesn't help solve the issue.
This bash
solution appears to best fit my needs. It's decently secure, portable, and fast.
#!/bin/bash
SRV="example.com"
URL="https://$SRV/path"
curl --netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD") "$URL"
This uses process substitution (<( command )
runs command
in a sub-shell to populate a file descriptor to be handed as a "file" to the parent command, which in this case is curl
). The process substitution contains a here-string (cat <<< text
, a variant of echo text
that won't put anything into your process list), creating a file descriptor for the netrc file in order to pass credentials to the remote web server.
The security afforded by process substitution is actually pretty sound: its file descriptor is not a temporary file and is unavailable from even other calls in the same shell instance, so this appears secure in this context; an adversary would have to dig through memory or launch a complicated attack to find its contents. Since the $PASSWORD
environment variable is also in memory, this should not increase the attack surface.
As long as you haven't used export PASSWORD
, a trick like ps ewwp $$
shouldn't reveal the password (as noted in this comment). It'd also be wise to use some less obvious variable name.
Here is a simplified insecure version of the above code that may help explain how it works:
#!/bin/sh
# INSECURE VERSION, DO NOT USE
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp)
printf "machine %s login %s password %sn" "$SRV" "$USER" "$PASSWORD" > "$TMP"
curl --netrc-file "$TMP" "$URL"
rm -f "$TMP"
This insecure version has lots of flaws, all of which are solved in the previous version:
- It stores the password in a file (though that file is only readable to you)
- It very briefly has the password in a command line
- The temporary file remains until after
curl
exits
Ctrl+c will quit without removing the temporary file
Some of that could be solved by:
#!/bin/sh
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp /dev/shm/.XXXXX) # assumes /dev/shm is a ramdisk
trap "rm -f $TMP" 0 18
cat << EOF > "$TMP"
machine $SRV login $USER password $PASSWORD
EOF
(sleep 0.1; rm -f "$TMP") & # queue removing temp file in 0.1 seconds
curl --netrc-file "$TMP" "$URL"
I consider this version to be messy, suboptimal, and possibly less secure (though it is more portable). It also requires a version of sleep
that understands decimals (and 0.1 seconds may be too fast if the system is heavily loaded).
I had originally posted a workaround that included a perl
one-liner in my question, then (with help from Etan Reisner) I worked through a few better methods before settling on this here-string method, which is both lighter-weight (faster) and more portable.
At this point, it's elegant enough that I'd consider it the "answer" rather than an "ugly workaround," so I've migrated it to be this official answer. I've given @ghoti a +1 for his answer, which correctly states that cURL's command line program is incapable of doing what I want on its own, but I'm not "accepting" that answer because it doesn't help solve the issue.
edited Nov 21 '15 at 20:23
answered Nov 20 '15 at 4:22
Adam Katz
5,55013351
5,55013351
add a comment |
add a comment |
up vote
7
down vote
Is there a way to pass credentials to
curl
solely through the environment?
No, I don't think there is.
The CURLOPT_USERPWD documentation I think describes what you need, but this is an option that would be available using the curl library in some other language. PHP, Perl, C, etc.
The curl binary you run from your shell is just another front end on that library, but the way things like CURLOPT_USERPWD get passed to the library through the curl binary is by use of command line options on the binary.
You could theoretically write your own binary as a front end to the curl library, and write in support for environment variables.
You could alternately hack environment support as you're hoping to see it into the existing curl binary, and compile your own with local functions.
Beware, though, that even environment variables may be leaked by your shell into the process table. (What do you see when you run ps ewwp $$
?)
Perhaps a .netrc file with restricted permissions will be the safest way to go. Perhaps you will need to generate a temporary .netrc file to be used by the --netrc-file
curl option.
I think you either have to pick the least risky solution for your environment, or write something in a real language that does security properly.
1
foo=bar; export foo; ps ewwp $$ |grep -i foo
finds the variable inbash
but interestingly finds nothing inzsh
(my shell of choice). (Withoutexport
,ps ewwp $$
doesn't find the variable inbash
orzsh
, butperl
doesn't pick it up either, making it useless.) The bash/zsh process substitution (<(…)
) in my workaround appears to do a good job of restricting access to that very temporary "file" (though I'm sure it still lives in memory).
– Adam Katz
Nov 19 '15 at 18:53
I've moved my workaround to an answer and alleviated the need forexport
which makes the variable invisible tops ewwp
– Adam Katz
Nov 20 '15 at 4:29
add a comment |
up vote
7
down vote
Is there a way to pass credentials to
curl
solely through the environment?
No, I don't think there is.
The CURLOPT_USERPWD documentation I think describes what you need, but this is an option that would be available using the curl library in some other language. PHP, Perl, C, etc.
The curl binary you run from your shell is just another front end on that library, but the way things like CURLOPT_USERPWD get passed to the library through the curl binary is by use of command line options on the binary.
You could theoretically write your own binary as a front end to the curl library, and write in support for environment variables.
You could alternately hack environment support as you're hoping to see it into the existing curl binary, and compile your own with local functions.
Beware, though, that even environment variables may be leaked by your shell into the process table. (What do you see when you run ps ewwp $$
?)
Perhaps a .netrc file with restricted permissions will be the safest way to go. Perhaps you will need to generate a temporary .netrc file to be used by the --netrc-file
curl option.
I think you either have to pick the least risky solution for your environment, or write something in a real language that does security properly.
1
foo=bar; export foo; ps ewwp $$ |grep -i foo
finds the variable inbash
but interestingly finds nothing inzsh
(my shell of choice). (Withoutexport
,ps ewwp $$
doesn't find the variable inbash
orzsh
, butperl
doesn't pick it up either, making it useless.) The bash/zsh process substitution (<(…)
) in my workaround appears to do a good job of restricting access to that very temporary "file" (though I'm sure it still lives in memory).
– Adam Katz
Nov 19 '15 at 18:53
I've moved my workaround to an answer and alleviated the need forexport
which makes the variable invisible tops ewwp
– Adam Katz
Nov 20 '15 at 4:29
add a comment |
up vote
7
down vote
up vote
7
down vote
Is there a way to pass credentials to
curl
solely through the environment?
No, I don't think there is.
The CURLOPT_USERPWD documentation I think describes what you need, but this is an option that would be available using the curl library in some other language. PHP, Perl, C, etc.
The curl binary you run from your shell is just another front end on that library, but the way things like CURLOPT_USERPWD get passed to the library through the curl binary is by use of command line options on the binary.
You could theoretically write your own binary as a front end to the curl library, and write in support for environment variables.
You could alternately hack environment support as you're hoping to see it into the existing curl binary, and compile your own with local functions.
Beware, though, that even environment variables may be leaked by your shell into the process table. (What do you see when you run ps ewwp $$
?)
Perhaps a .netrc file with restricted permissions will be the safest way to go. Perhaps you will need to generate a temporary .netrc file to be used by the --netrc-file
curl option.
I think you either have to pick the least risky solution for your environment, or write something in a real language that does security properly.
Is there a way to pass credentials to
curl
solely through the environment?
No, I don't think there is.
The CURLOPT_USERPWD documentation I think describes what you need, but this is an option that would be available using the curl library in some other language. PHP, Perl, C, etc.
The curl binary you run from your shell is just another front end on that library, but the way things like CURLOPT_USERPWD get passed to the library through the curl binary is by use of command line options on the binary.
You could theoretically write your own binary as a front end to the curl library, and write in support for environment variables.
You could alternately hack environment support as you're hoping to see it into the existing curl binary, and compile your own with local functions.
Beware, though, that even environment variables may be leaked by your shell into the process table. (What do you see when you run ps ewwp $$
?)
Perhaps a .netrc file with restricted permissions will be the safest way to go. Perhaps you will need to generate a temporary .netrc file to be used by the --netrc-file
curl option.
I think you either have to pick the least risky solution for your environment, or write something in a real language that does security properly.
answered Nov 19 '15 at 4:45
ghoti
34.7k74179
34.7k74179
1
foo=bar; export foo; ps ewwp $$ |grep -i foo
finds the variable inbash
but interestingly finds nothing inzsh
(my shell of choice). (Withoutexport
,ps ewwp $$
doesn't find the variable inbash
orzsh
, butperl
doesn't pick it up either, making it useless.) The bash/zsh process substitution (<(…)
) in my workaround appears to do a good job of restricting access to that very temporary "file" (though I'm sure it still lives in memory).
– Adam Katz
Nov 19 '15 at 18:53
I've moved my workaround to an answer and alleviated the need forexport
which makes the variable invisible tops ewwp
– Adam Katz
Nov 20 '15 at 4:29
add a comment |
1
foo=bar; export foo; ps ewwp $$ |grep -i foo
finds the variable inbash
but interestingly finds nothing inzsh
(my shell of choice). (Withoutexport
,ps ewwp $$
doesn't find the variable inbash
orzsh
, butperl
doesn't pick it up either, making it useless.) The bash/zsh process substitution (<(…)
) in my workaround appears to do a good job of restricting access to that very temporary "file" (though I'm sure it still lives in memory).
– Adam Katz
Nov 19 '15 at 18:53
I've moved my workaround to an answer and alleviated the need forexport
which makes the variable invisible tops ewwp
– Adam Katz
Nov 20 '15 at 4:29
1
1
foo=bar; export foo; ps ewwp $$ |grep -i foo
finds the variable in bash
but interestingly finds nothing in zsh
(my shell of choice). (Without export
, ps ewwp $$
doesn't find the variable in bash
or zsh
, but perl
doesn't pick it up either, making it useless.) The bash/zsh process substitution (<(…)
) in my workaround appears to do a good job of restricting access to that very temporary "file" (though I'm sure it still lives in memory).– Adam Katz
Nov 19 '15 at 18:53
foo=bar; export foo; ps ewwp $$ |grep -i foo
finds the variable in bash
but interestingly finds nothing in zsh
(my shell of choice). (Without export
, ps ewwp $$
doesn't find the variable in bash
or zsh
, but perl
doesn't pick it up either, making it useless.) The bash/zsh process substitution (<(…)
) in my workaround appears to do a good job of restricting access to that very temporary "file" (though I'm sure it still lives in memory).– Adam Katz
Nov 19 '15 at 18:53
I've moved my workaround to an answer and alleviated the need for
export
which makes the variable invisible to ps ewwp
– Adam Katz
Nov 20 '15 at 4:29
I've moved my workaround to an answer and alleviated the need for
export
which makes the variable invisible to ps ewwp
– Adam Katz
Nov 20 '15 at 4:29
add a comment |
up vote
1
down vote
User "Tom, Bom" provides a decent solution for this here: https://coderwall.com/p/dsfmwa/securely-use-basic-auth-with-curl
curl --config - https://example.com <<< 'user = "username:password"'
This prevents passwords from showing up in the process list, though this does not specifically address the OP's original question:
Is there a way to pass credentials to curl solely through the environment?
I still give points to @ghoti for giving a more comprehensive and informative answer.
add a comment |
up vote
1
down vote
User "Tom, Bom" provides a decent solution for this here: https://coderwall.com/p/dsfmwa/securely-use-basic-auth-with-curl
curl --config - https://example.com <<< 'user = "username:password"'
This prevents passwords from showing up in the process list, though this does not specifically address the OP's original question:
Is there a way to pass credentials to curl solely through the environment?
I still give points to @ghoti for giving a more comprehensive and informative answer.
add a comment |
up vote
1
down vote
up vote
1
down vote
User "Tom, Bom" provides a decent solution for this here: https://coderwall.com/p/dsfmwa/securely-use-basic-auth-with-curl
curl --config - https://example.com <<< 'user = "username:password"'
This prevents passwords from showing up in the process list, though this does not specifically address the OP's original question:
Is there a way to pass credentials to curl solely through the environment?
I still give points to @ghoti for giving a more comprehensive and informative answer.
User "Tom, Bom" provides a decent solution for this here: https://coderwall.com/p/dsfmwa/securely-use-basic-auth-with-curl
curl --config - https://example.com <<< 'user = "username:password"'
This prevents passwords from showing up in the process list, though this does not specifically address the OP's original question:
Is there a way to pass credentials to curl solely through the environment?
I still give points to @ghoti for giving a more comprehensive and informative answer.
edited Nov 10 at 18:47
answered Nov 10 at 18:42
sfgeorge
864
864
add a comment |
add a comment |
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f33794842%2fforcing-curl-to-get-a-password-from-the-environment%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
You don't need perl for that you could just a shell snippet there just as well but note that the process environment is available to
root
just as much as a file is (though for a shorter timeframe potentially).– Etan Reisner
Nov 19 '15 at 3:47
SSPI and NTLM are both Windows technologies. Presumably the
curl
man page is not talking about retrieving the password from a POSIX-style environment.– chepner
Nov 19 '15 at 14:51
1
I think
declare -p USER | sed 's/^[^=]*="//;s/"[^"]*$//;s/\"/"/'
satisfies but certainly isn't better than the perl, etc. solution. Do heredocs/herestrings show up in the command? (I don't recall offhand.) If not those would work too.– Etan Reisner
Nov 19 '15 at 18:23
2
It took me 29 seconds to replace
curl
with a wrapper script that did a strace dump on the original binary, circumventing every single of these techniques including #4. By design, you can not hide from local root.– that other guy
Nov 19 '15 at 19:19
1
Something like
--netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD")
possibly? But once you've hit a sub-shell/etc. avoidingsed
isn't really worth much.– Etan Reisner
Nov 20 '15 at 13:18