How do you compute and/or apply the gradients of paticular parameters within a variable in TensorFlow?
I have four sets of TensorFlow Variables.
I am using these variables in an autoencoder based on the archtiecture found in this paper: http://users.cecs.anu.edu.au/~u5098633/papers/www15.pdf
Briefly, only the weights that are connected to certain input nodes are important for training. That is, if we have 10 input nodes, I only want to focus on, say, 5 of these nodes, and the weights that are connected to these nodes. My variable that models these connections is:
self.V = tf.Variable(tf.truncated_normal([input_size, hidden_size]), name='V')
I am able to get these particular weights by using:
weights_to_update = tf.gather(self.V, [indices_of_weights]).
Now, to get the particular gradients of these weights, I have tried using
grads_and_vars = tf.Optimizer.compute_gradients(my_loss_function, weights_to_update)
which I thought would calculate the gradients of these particular weights, and ignore all other weights in the self.V variable. I would then use tf.Optimizer.apply_gradients(graqds_and_vars) to perform the update step.
However, the output I get from using compute_gradients with the var_list parameter as the weights I want is:
[(None, <tf.Tensor 'GatherV2:0' shape=(20, 200) dtype=float32>)]
which shows that the shape is correct for what I want, but that this is None values. Subsequently, calling apply_gradients with what this returns leads to an error.
I have searched all over the internet to find out how to compute/apply gradients to a subset of a tf.Variable, but I cannot find out how to do this anywhere.
So, to sum up, for each training step, I would like to know how to train only a subset of parameters from each of my tf.Variable objects in my graph.
python tensorflow neural-network
asked Nov 11 at 1:03
Frederick
306
add a comment |Â
I have four sets of TensorFlow Variables.
I am using these variables in an autoencoder based on the archtiecture found in this paper: http://users.cecs.anu.edu.au/~u5098633/papers/www15.pdf
Briefly, only the weights that are connected to certain input nodes are important for training. That is, if we have 10 input nodes, I only want to focus on, say, 5 of these nodes, and the weights that are connected to these nodes. My variable that models these connections is:
self.V = tf.Variable(tf.truncated_normal([input_size, hidden_size]), name='V')
I am able to get these particular weights by using:
weights_to_update = tf.gather(self.V, [indices_of_weights]).
Now, to get the particular gradients of these weights, I have tried using
grads_and_vars = tf.Optimizer.compute_gradients(my_loss_function, weights_to_update)
which I thought would calculate the gradients of these particular weights, and ignore all other weights in the self.V variable. I would then use tf.Optimizer.apply_gradients(graqds_and_vars) to perform the update step.
However, the output I get from using compute_gradients with the var_list parameter as the weights I want is:
[(None, <tf.Tensor 'GatherV2:0' shape=(20, 200) dtype=float32>)]
which shows that the shape is correct for what I want, but that this is None values. Subsequently, calling apply_gradients with what this returns leads to an error.
I have searched all over the internet to find out how to compute/apply gradients to a subset of a tf.Variable, but I cannot find out how to do this anywhere.
So, to sum up, for each training step, I would like to know how to train only a subset of parameters from each of my tf.Variable objects in my graph.
python tensorflow neural-network
asked Nov 11 at 1:03
Frederick
306
If your gradient is None, there is probably a problem with your network setup. Does simply usingOptimizer.minimize(loss)work (i.e. run) and change all the weights?
– cheersmate
Nov 12 at 8:27
add a comment |Â
I have four sets of TensorFlow Variables.
I am using these variables in an autoencoder based on the archtiecture found in this paper: http://users.cecs.anu.edu.au/~u5098633/papers/www15.pdf
Briefly, only the weights that are connected to certain input nodes are important for training. That is, if we have 10 input nodes, I only want to focus on, say, 5 of these nodes, and the weights that are connected to these nodes. My variable that models these connections is:
self.V = tf.Variable(tf.truncated_normal([input_size, hidden_size]), name='V')
I am able to get these particular weights by using:
weights_to_update = tf.gather(self.V, [indices_of_weights]).
Now, to get the particular gradients of these weights, I have tried using
grads_and_vars = tf.Optimizer.compute_gradients(my_loss_function, weights_to_update)
which I thought would calculate the gradients of these particular weights, and ignore all other weights in the self.V variable. I would then use tf.Optimizer.apply_gradients(graqds_and_vars) to perform the update step.
However, the output I get from using compute_gradients with the var_list parameter as the weights I want is:
[(None, <tf.Tensor 'GatherV2:0' shape=(20, 200) dtype=float32>)]
which shows that the shape is correct for what I want, but that this is None values. Subsequently, calling apply_gradients with what this returns leads to an error.
I have searched all over the internet to find out how to compute/apply gradients to a subset of a tf.Variable, but I cannot find out how to do this anywhere.
So, to sum up, for each training step, I would like to know how to train only a subset of parameters from each of my tf.Variable objects in my graph.
python tensorflow neural-network
asked Nov 11 at 1:03
Frederick
306
I have four sets of TensorFlow Variables.
I am using these variables in an autoencoder based on the archtiecture found in this paper: http://users.cecs.anu.edu.au/~u5098633/papers/www15.pdf
Briefly, only the weights that are connected to certain input nodes are important for training. That is, if we have 10 input nodes, I only want to focus on, say, 5 of these nodes, and the weights that are connected to these nodes. My variable that models these connections is:
self.V = tf.Variable(tf.truncated_normal([input_size, hidden_size]), name='V')
I am able to get these particular weights by using:
weights_to_update = tf.gather(self.V, [indices_of_weights]).
Now, to get the particular gradients of these weights, I have tried using
grads_and_vars = tf.Optimizer.compute_gradients(my_loss_function, weights_to_update)
which I thought would calculate the gradients of these particular weights, and ignore all other weights in the self.V variable. I would then use tf.Optimizer.apply_gradients(graqds_and_vars) to perform the update step.
However, the output I get from using compute_gradients with the var_list parameter as the weights I want is:
[(None, <tf.Tensor 'GatherV2:0' shape=(20, 200) dtype=float32>)]
which shows that the shape is correct for what I want, but that this is None values. Subsequently, calling apply_gradients with what this returns leads to an error.
I have searched all over the internet to find out how to compute/apply gradients to a subset of a tf.Variable, but I cannot find out how to do this anywhere.
So, to sum up, for each training step, I would like to know how to train only a subset of parameters from each of my tf.Variable objects in my graph.
python tensorflow neural-network
python tensorflow neural-network
asked Nov 11 at 1:03
Frederick
306
asked Nov 11 at 1:03
Frederick
306
asked Nov 11 at 1:03
Frederick
306
asked Nov 11 at 1:03
Frederick
306
asked Nov 11 at 1:03
Frederick
306
306
If your gradient is None, there is probably a problem with your network setup. Does simply usingOptimizer.minimize(loss)work (i.e. run) and change all the weights?
– cheersmate
Nov 12 at 8:27
add a comment |Â
If your gradient is None, there is probably a problem with your network setup. Does simply usingOptimizer.minimize(loss)work (i.e. run) and change all the weights?
– cheersmate
Nov 12 at 8:27
If your gradient is None, there is probably a problem with your network setup. Does simply using
Optimizer.minimize(loss) work (i.e. run) and change all the weights?– cheersmate
Nov 12 at 8:27
If your gradient is None, there is probably a problem with your network setup. Does simply using
Optimizer.minimize(loss) work (i.e. run) and change all the weights?– cheersmate
Nov 12 at 8:27
add a comment |Â
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53244934%2fhow-do-you-compute-and-or-apply-the-gradients-of-paticular-parameters-within-a-v%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If your gradient is None, there is probably a problem with your network setup. Does simply using
Optimizer.minimize(loss)work (i.e. run) and change all the weights?– cheersmate
Nov 12 at 8:27