Myujth

In short, when I restore my pretrained model, i wanna change the optimizer to AdamOptimizer for further training. However, it strikes to me that it rise error like below:

NotFoundError (see above for traceback): Key beta1_power not found in checkpoint

 [[Node: save_1/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2/tensor_names, save_1/RestoreV2/shape_and_slices)]]

I just assumed that the corresponding variables could be add to computational graph automatically without human intervention just like tf.get_variable do.

The code I use is below:

# 0. only 1 gpu
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

# 1. define global parameters

args = get_parser()
global_step = tf.Variable(name='global_step', initial_value=0, trainable=False)
inc_op = tf.assign_add(global_step, 1, name='increment_global_step')
images = tf.placeholder(name='img_inputs', shape=[None, *args.image_size, 3], dtype=tf.float32)
labels = tf.placeholder(name='img_labels', shape=[None, ], dtype=tf.int64)
dropout_rate = tf.placeholder(name='dropout_rate', dtype=tf.float32)

# 2 prepare train datasets and test datasets by using tensorflow dataset api
# 2.1 train datasets 

tfrecords_f = os.path.join(args.tfrecords_file_path, 'tran_asia.tfrecords')
dataset = tf.data.TFRecordDataset(tfrecords_f)
dataset = dataset.map(parse_function)
dataset = dataset.shuffle(buffer_size=args.buffer_size)
dataset = dataset.batch(args.batch_size)
iterator = dataset.make_initializable_iterator()
next_element = iterator.get_next()

# 3. define network, loss, optimize method, learning rate schedule, summary writer, saver
# 3.1 inference phase

w_init_method = tf.contrib.layers.xavier_initializer(uniform=False)
net = get_resnet(...)

# 3.2 loss

logit = self_define_loss(embedding=net.outputs, labels=labels, w_init=w_init_method, out_num=args.num_output)
...
# 3.3 calculate loss

infer_loss = ...

# 3.4 optimizer(change after pretrained)

# stage1
# opt = tf.train.MomentumOptimizer(learning_rate=lr, momentum=args.momentum)
# stage2
opt = tf.train.AdamOptimizer(learning_rate=lr)

# 3.5 get train op

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
 train_op = opt.apply_gradients(grads, global_step=global_step)

# 4.restore stage1 model 

# 4.1 saver

saver = tf.train.Saver(max_to_keep=10)

# 4.2 init all variables

sess.run(tf.global_variables_initializer())

# 4.3 restore stage1 model and change optimizer to do further training!

restore_saver = tf.train.Saver()
restore_saver.restore(sess, 'xxx.ckpt')


# Omit training part
...

The Tensorflow version i use is 1.7.0, really appreciate for your kindness help, Thanks you!