How does the multi-input deep learning work in Keras?
up vote
2
down vote
favorite
I have a multi-input convolutional neural network model that inputs 2 images from 2 datasets to give one output which is the class of the two inputs. The two datasets have the same classes. I used 2 vgg16 models and concatenate them to classify the two images.
vgg16_model = keras.applications.vgg16.VGG16()
input_layer1= vgg16_model .input
last_layer1 = vgg16_model.get_layer('fc2').output
vgg16_model2 = keras.applications.vgg16.VGG16()
input_layer2= vgg16_model .input
last_layer2 = vgg16_model.get_layer('fc2').output
con = concatenate([last_layer1, last_layer2]) # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer])
My questions are:
1- Which case from the following represents how the images enter to the model?
One to One
database1-img1 + database2-img1
database1-img2 + database2-img2
database1-img3 + database2-img3
database1-img4 + database2-img4
.........
Many to many
database1-img1 + database2-img1
database1-img1 + database2-img2
database1-img1 + database2-img3
database1-img1 + database2-img4
database1-img2 + database2-img1
database1-img2 + database2-img2
database1-img2 + database2-img3
database1-img2 + database2-img4
.........
2- In general in deep learning, Does the images enter from the two datasets to the model at the same time have the same class (labels) or not?
tensorflow machine-learning keras neural-network deep-learning
asked Nov 10 at 18:31
Noran
1345
add a comment |
up vote
2
down vote
favorite
I have a multi-input convolutional neural network model that inputs 2 images from 2 datasets to give one output which is the class of the two inputs. The two datasets have the same classes. I used 2 vgg16 models and concatenate them to classify the two images.
vgg16_model = keras.applications.vgg16.VGG16()
input_layer1= vgg16_model .input
last_layer1 = vgg16_model.get_layer('fc2').output
vgg16_model2 = keras.applications.vgg16.VGG16()
input_layer2= vgg16_model .input
last_layer2 = vgg16_model.get_layer('fc2').output
con = concatenate([last_layer1, last_layer2]) # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer])
My questions are:
1- Which case from the following represents how the images enter to the model?
One to One
database1-img1 + database2-img1
database1-img2 + database2-img2
database1-img3 + database2-img3
database1-img4 + database2-img4
.........
Many to many
database1-img1 + database2-img1
database1-img1 + database2-img2
database1-img1 + database2-img3
database1-img1 + database2-img4
database1-img2 + database2-img1
database1-img2 + database2-img2
database1-img2 + database2-img3
database1-img2 + database2-img4
.........
2- In general in deep learning, Does the images enter from the two datasets to the model at the same time have the same class (labels) or not?
tensorflow machine-learning keras neural-network deep-learning
asked Nov 10 at 18:31
Noran
1345
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have a multi-input convolutional neural network model that inputs 2 images from 2 datasets to give one output which is the class of the two inputs. The two datasets have the same classes. I used 2 vgg16 models and concatenate them to classify the two images.
vgg16_model = keras.applications.vgg16.VGG16()
input_layer1= vgg16_model .input
last_layer1 = vgg16_model.get_layer('fc2').output
vgg16_model2 = keras.applications.vgg16.VGG16()
input_layer2= vgg16_model .input
last_layer2 = vgg16_model.get_layer('fc2').output
con = concatenate([last_layer1, last_layer2]) # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer])
My questions are:
1- Which case from the following represents how the images enter to the model?
One to One
database1-img1 + database2-img1
database1-img2 + database2-img2
database1-img3 + database2-img3
database1-img4 + database2-img4
.........
Many to many
database1-img1 + database2-img1
database1-img1 + database2-img2
database1-img1 + database2-img3
database1-img1 + database2-img4
database1-img2 + database2-img1
database1-img2 + database2-img2
database1-img2 + database2-img3
database1-img2 + database2-img4
.........
2- In general in deep learning, Does the images enter from the two datasets to the model at the same time have the same class (labels) or not?
tensorflow machine-learning keras neural-network deep-learning
asked Nov 10 at 18:31
Noran
1345
I have a multi-input convolutional neural network model that inputs 2 images from 2 datasets to give one output which is the class of the two inputs. The two datasets have the same classes. I used 2 vgg16 models and concatenate them to classify the two images.
vgg16_model = keras.applications.vgg16.VGG16()
input_layer1= vgg16_model .input
last_layer1 = vgg16_model.get_layer('fc2').output
vgg16_model2 = keras.applications.vgg16.VGG16()
input_layer2= vgg16_model .input
last_layer2 = vgg16_model.get_layer('fc2').output
con = concatenate([last_layer1, last_layer2]) # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer])
My questions are:
1- Which case from the following represents how the images enter to the model?
One to One
database1-img1 + database2-img1
database1-img2 + database2-img2
database1-img3 + database2-img3
database1-img4 + database2-img4
.........
Many to many
database1-img1 + database2-img1
database1-img1 + database2-img2
database1-img1 + database2-img3
database1-img1 + database2-img4
database1-img2 + database2-img1
database1-img2 + database2-img2
database1-img2 + database2-img3
database1-img2 + database2-img4
.........
2- In general in deep learning, Does the images enter from the two datasets to the model at the same time have the same class (labels) or not?
tensorflow machine-learning keras neural-network deep-learning
tensorflow machine-learning keras neural-network deep-learning
asked Nov 10 at 18:31
Noran
1345
asked Nov 10 at 18:31
Noran
1345
asked Nov 10 at 18:31
Noran
1345
asked Nov 10 at 18:31
Noran
1345
asked Nov 10 at 18:31
Noran
1345
1345
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
accepted
- It is a 1:1 mapping, the same should be with multiple outputs as well.
When you have a model such as Model(inputs=[input_layer1, input_layer2], outputs=[output_layer]) or even Model(inputs=[input_layer1, input_layer2], outputs=[output_layer1, output_layer2]) , You must feed it with inputs / output of the same shape.
Assume the other case - You will need to have ds1.shape[0] * ds2.shape[0] different labels, for each possible mix of the 2 datasets, and will need to have them ordered at a certain way. That is not really feasible, at least not simply.
2. Its not as if the same images have the same label, but the Pair of both images have a single label.
answered Nov 10 at 20:32
Or Dinari
640318
Thank you so much for this valuable information. I want to modify the model to combine only the images from the same class to produce the class of the two combined images. How can I do that?
– Noran
Nov 10 at 21:15
1
Sort the datasets according to the classes, such that for all i ds1[i].class == ds2[i].class == labels[i] , After youve done that you can permutate them toghetar.
– Or Dinari
Nov 10 at 21:37
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
- It is a 1:1 mapping, the same should be with multiple outputs as well.
When you have a model such as Model(inputs=[input_layer1, input_layer2], outputs=[output_layer]) or even Model(inputs=[input_layer1, input_layer2], outputs=[output_layer1, output_layer2]) , You must feed it with inputs / output of the same shape.
Assume the other case - You will need to have ds1.shape[0] * ds2.shape[0] different labels, for each possible mix of the 2 datasets, and will need to have them ordered at a certain way. That is not really feasible, at least not simply.
2. Its not as if the same images have the same label, but the Pair of both images have a single label.
answered Nov 10 at 20:32
Or Dinari
640318
Thank you so much for this valuable information. I want to modify the model to combine only the images from the same class to produce the class of the two combined images. How can I do that?
– Noran
Nov 10 at 21:15
1
Sort the datasets according to the classes, such that for all i ds1[i].class == ds2[i].class == labels[i] , After youve done that you can permutate them toghetar.
– Or Dinari
Nov 10 at 21:37
add a comment |
up vote
1
down vote
accepted
- It is a 1:1 mapping, the same should be with multiple outputs as well.
When you have a model such as Model(inputs=[input_layer1, input_layer2], outputs=[output_layer]) or even Model(inputs=[input_layer1, input_layer2], outputs=[output_layer1, output_layer2]) , You must feed it with inputs / output of the same shape.
Assume the other case - You will need to have ds1.shape[0] * ds2.shape[0] different labels, for each possible mix of the 2 datasets, and will need to have them ordered at a certain way. That is not really feasible, at least not simply.
2. Its not as if the same images have the same label, but the Pair of both images have a single label.
answered Nov 10 at 20:32
Or Dinari
640318
Thank you so much for this valuable information. I want to modify the model to combine only the images from the same class to produce the class of the two combined images. How can I do that?
– Noran
Nov 10 at 21:15
1
Sort the datasets according to the classes, such that for all i ds1[i].class == ds2[i].class == labels[i] , After youve done that you can permutate them toghetar.
– Or Dinari
Nov 10 at 21:37
add a comment |
up vote
1
down vote
accepted
up vote
1
down vote
accepted
- It is a 1:1 mapping, the same should be with multiple outputs as well.
When you have a model such as Model(inputs=[input_layer1, input_layer2], outputs=[output_layer]) or even Model(inputs=[input_layer1, input_layer2], outputs=[output_layer1, output_layer2]) , You must feed it with inputs / output of the same shape.
Assume the other case - You will need to have ds1.shape[0] * ds2.shape[0] different labels, for each possible mix of the 2 datasets, and will need to have them ordered at a certain way. That is not really feasible, at least not simply.
2. Its not as if the same images have the same label, but the Pair of both images have a single label.
answered Nov 10 at 20:32
Or Dinari
640318
- It is a 1:1 mapping, the same should be with multiple outputs as well.
When you have a model such as Model(inputs=[input_layer1, input_layer2], outputs=[output_layer]) or even Model(inputs=[input_layer1, input_layer2], outputs=[output_layer1, output_layer2]) , You must feed it with inputs / output of the same shape.
Assume the other case - You will need to have ds1.shape[0] * ds2.shape[0] different labels, for each possible mix of the 2 datasets, and will need to have them ordered at a certain way. That is not really feasible, at least not simply.
2. Its not as if the same images have the same label, but the Pair of both images have a single label.
answered Nov 10 at 20:32
Or Dinari
640318
answered Nov 10 at 20:32
Or Dinari
640318
answered Nov 10 at 20:32
Or Dinari
640318
answered Nov 10 at 20:32
Or Dinari
640318
640318
Thank you so much for this valuable information. I want to modify the model to combine only the images from the same class to produce the class of the two combined images. How can I do that?
– Noran
Nov 10 at 21:15
1
Sort the datasets according to the classes, such that for all i ds1[i].class == ds2[i].class == labels[i] , After youve done that you can permutate them toghetar.
– Or Dinari
Nov 10 at 21:37
add a comment |
Thank you so much for this valuable information. I want to modify the model to combine only the images from the same class to produce the class of the two combined images. How can I do that?
– Noran
Nov 10 at 21:15
1
Sort the datasets according to the classes, such that for all i ds1[i].class == ds2[i].class == labels[i] , After youve done that you can permutate them toghetar.
– Or Dinari
Nov 10 at 21:37
Thank you so much for this valuable information. I want to modify the model to combine only the images from the same class to produce the class of the two combined images. How can I do that?
– Noran
Nov 10 at 21:15
Thank you so much for this valuable information. I want to modify the model to combine only the images from the same class to produce the class of the two combined images. How can I do that?
– Noran
Nov 10 at 21:15
1
1
Sort the datasets according to the classes, such that for all i ds1[i].class == ds2[i].class == labels[i] , After youve done that you can permutate them toghetar.
– Or Dinari
Nov 10 at 21:37
Sort the datasets according to the classes, such that for all i ds1[i].class == ds2[i].class == labels[i] , After youve done that you can permutate them toghetar.
– Or Dinari
Nov 10 at 21:37
add a comment |
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53242146%2fhow-does-the-multi-input-deep-learning-work-in-keras%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
