Does Logstash support Elasticsearch's _update_by_query?
Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html
elasticsearch logstash logstash-configuration
add a comment |
Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html
elasticsearch logstash logstash-configuration
add a comment |
Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html
elasticsearch logstash logstash-configuration
Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html
elasticsearch logstash logstash-configuration
elasticsearch logstash logstash-configuration
asked Nov 16 '18 at 1:35
wangjinhaowangjinhao
185
185
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The elasticsearch
output plugin can only make calls to the _bulk
endpoint, i.e. using the Bulk API.
If you want to call the Update by Query API, you need to use the http
output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.
Note: There's an issue requesting this feature, but it's still open after two years.
UPDATE
So if your input event is "cname":"wang", "cage":11
and you want to update by query all documents with "cname":"wang"
to set "cage":11
, your query needs to look like this:
POST your-index/_update_by_query
"script":
"source": "ctx._source.cage = params.cage",
"lang": "painless",
"params":
"cage": 11
,
"query":
"term":
"cname": "wang"
So your Logstash config should look like this (your input may vary but I used stdin
for testing purposes):
input
stdin
codec => "json"
filter
mutate
add_field =>
"[script][lang]" => "painless"
"[script][source]" => "ctx._source.cage = params.cage"
"[script][params][cage]" => "%cage"
"[query][term][cname]" => "%cname"
remove_field => ["host", "@version", "@timestamp", "cname", "cage"]
output
http
url => "http://localhost:9200/index/doc/_update_by_query"
http_method => "post"
format => "json"
ok, thanks very much
– wangjinhao
Nov 19 '18 at 3:14
Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you
– wangjinhao
Nov 21 '18 at 3:24
See my updated answer
– Val
Nov 21 '18 at 5:15
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330232%2fdoes-logstash-support-elasticsearchs-update-by-query%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The elasticsearch
output plugin can only make calls to the _bulk
endpoint, i.e. using the Bulk API.
If you want to call the Update by Query API, you need to use the http
output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.
Note: There's an issue requesting this feature, but it's still open after two years.
UPDATE
So if your input event is "cname":"wang", "cage":11
and you want to update by query all documents with "cname":"wang"
to set "cage":11
, your query needs to look like this:
POST your-index/_update_by_query
"script":
"source": "ctx._source.cage = params.cage",
"lang": "painless",
"params":
"cage": 11
,
"query":
"term":
"cname": "wang"
So your Logstash config should look like this (your input may vary but I used stdin
for testing purposes):
input
stdin
codec => "json"
filter
mutate
add_field =>
"[script][lang]" => "painless"
"[script][source]" => "ctx._source.cage = params.cage"
"[script][params][cage]" => "%cage"
"[query][term][cname]" => "%cname"
remove_field => ["host", "@version", "@timestamp", "cname", "cage"]
output
http
url => "http://localhost:9200/index/doc/_update_by_query"
http_method => "post"
format => "json"
ok, thanks very much
– wangjinhao
Nov 19 '18 at 3:14
Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you
– wangjinhao
Nov 21 '18 at 3:24
See my updated answer
– Val
Nov 21 '18 at 5:15
add a comment |
The elasticsearch
output plugin can only make calls to the _bulk
endpoint, i.e. using the Bulk API.
If you want to call the Update by Query API, you need to use the http
output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.
Note: There's an issue requesting this feature, but it's still open after two years.
UPDATE
So if your input event is "cname":"wang", "cage":11
and you want to update by query all documents with "cname":"wang"
to set "cage":11
, your query needs to look like this:
POST your-index/_update_by_query
"script":
"source": "ctx._source.cage = params.cage",
"lang": "painless",
"params":
"cage": 11
,
"query":
"term":
"cname": "wang"
So your Logstash config should look like this (your input may vary but I used stdin
for testing purposes):
input
stdin
codec => "json"
filter
mutate
add_field =>
"[script][lang]" => "painless"
"[script][source]" => "ctx._source.cage = params.cage"
"[script][params][cage]" => "%cage"
"[query][term][cname]" => "%cname"
remove_field => ["host", "@version", "@timestamp", "cname", "cage"]
output
http
url => "http://localhost:9200/index/doc/_update_by_query"
http_method => "post"
format => "json"
ok, thanks very much
– wangjinhao
Nov 19 '18 at 3:14
Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you
– wangjinhao
Nov 21 '18 at 3:24
See my updated answer
– Val
Nov 21 '18 at 5:15
add a comment |
The elasticsearch
output plugin can only make calls to the _bulk
endpoint, i.e. using the Bulk API.
If you want to call the Update by Query API, you need to use the http
output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.
Note: There's an issue requesting this feature, but it's still open after two years.
UPDATE
So if your input event is "cname":"wang", "cage":11
and you want to update by query all documents with "cname":"wang"
to set "cage":11
, your query needs to look like this:
POST your-index/_update_by_query
"script":
"source": "ctx._source.cage = params.cage",
"lang": "painless",
"params":
"cage": 11
,
"query":
"term":
"cname": "wang"
So your Logstash config should look like this (your input may vary but I used stdin
for testing purposes):
input
stdin
codec => "json"
filter
mutate
add_field =>
"[script][lang]" => "painless"
"[script][source]" => "ctx._source.cage = params.cage"
"[script][params][cage]" => "%cage"
"[query][term][cname]" => "%cname"
remove_field => ["host", "@version", "@timestamp", "cname", "cage"]
output
http
url => "http://localhost:9200/index/doc/_update_by_query"
http_method => "post"
format => "json"
The elasticsearch
output plugin can only make calls to the _bulk
endpoint, i.e. using the Bulk API.
If you want to call the Update by Query API, you need to use the http
output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.
Note: There's an issue requesting this feature, but it's still open after two years.
UPDATE
So if your input event is "cname":"wang", "cage":11
and you want to update by query all documents with "cname":"wang"
to set "cage":11
, your query needs to look like this:
POST your-index/_update_by_query
"script":
"source": "ctx._source.cage = params.cage",
"lang": "painless",
"params":
"cage": 11
,
"query":
"term":
"cname": "wang"
So your Logstash config should look like this (your input may vary but I used stdin
for testing purposes):
input
stdin
codec => "json"
filter
mutate
add_field =>
"[script][lang]" => "painless"
"[script][source]" => "ctx._source.cage = params.cage"
"[script][params][cage]" => "%cage"
"[query][term][cname]" => "%cname"
remove_field => ["host", "@version", "@timestamp", "cname", "cage"]
output
http
url => "http://localhost:9200/index/doc/_update_by_query"
http_method => "post"
format => "json"
edited Nov 21 '18 at 5:15
answered Nov 16 '18 at 4:51
ValVal
108k6147181
108k6147181
ok, thanks very much
– wangjinhao
Nov 19 '18 at 3:14
Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you
– wangjinhao
Nov 21 '18 at 3:24
See my updated answer
– Val
Nov 21 '18 at 5:15
add a comment |
ok, thanks very much
– wangjinhao
Nov 19 '18 at 3:14
Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you
– wangjinhao
Nov 21 '18 at 3:24
See my updated answer
– Val
Nov 21 '18 at 5:15
ok, thanks very much
– wangjinhao
Nov 19 '18 at 3:14
ok, thanks very much
– wangjinhao
Nov 19 '18 at 3:14
Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you
– wangjinhao
Nov 21 '18 at 3:24
Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you
– wangjinhao
Nov 21 '18 at 3:24
See my updated answer
– Val
Nov 21 '18 at 5:15
See my updated answer
– Val
Nov 21 '18 at 5:15
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330232%2fdoes-logstash-support-elasticsearchs-update-by-query%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown