Does Logstash support Elasticsearch's _update_by_query?










0















Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html










share|improve this question


























    0















    Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
    https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
    https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html










    share|improve this question
























      0












      0








      0








      Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
      https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
      https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html










      share|improve this question














      Does the Elasticsearch output plugin support elasticsearch's _update_by_query?
      https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html
      https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html







      elasticsearch logstash logstash-configuration






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 16 '18 at 1:35









      wangjinhaowangjinhao

      185




      185






















          1 Answer
          1






          active

          oldest

          votes


















          0














          The elasticsearch output plugin can only make calls to the _bulk endpoint, i.e. using the Bulk API.



          If you want to call the Update by Query API, you need to use the http output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.



          Note: There's an issue requesting this feature, but it's still open after two years.



          UPDATE



          So if your input event is "cname":"wang", "cage":11 and you want to update by query all documents with "cname":"wang" to set "cage":11, your query needs to look like this:



          POST your-index/_update_by_query

          "script":
          "source": "ctx._source.cage = params.cage",
          "lang": "painless",
          "params":
          "cage": 11

          ,
          "query":
          "term":
          "cname": "wang"





          So your Logstash config should look like this (your input may vary but I used stdin for testing purposes):



          input 
          stdin
          codec => "json"


          filter
          mutate
          add_field =>
          "[script][lang]" => "painless"
          "[script][source]" => "ctx._source.cage = params.cage"
          "[script][params][cage]" => "%cage"
          "[query][term][cname]" => "%cname"

          remove_field => ["host", "@version", "@timestamp", "cname", "cage"]


          output
          http
          url => "http://localhost:9200/index/doc/_update_by_query"
          http_method => "post"
          format => "json"







          share|improve this answer

























          • ok, thanks very much

            – wangjinhao
            Nov 19 '18 at 3:14











          • Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you

            – wangjinhao
            Nov 21 '18 at 3:24











          • See my updated answer

            – Val
            Nov 21 '18 at 5:15










          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330232%2fdoes-logstash-support-elasticsearchs-update-by-query%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          0














          The elasticsearch output plugin can only make calls to the _bulk endpoint, i.e. using the Bulk API.



          If you want to call the Update by Query API, you need to use the http output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.



          Note: There's an issue requesting this feature, but it's still open after two years.



          UPDATE



          So if your input event is "cname":"wang", "cage":11 and you want to update by query all documents with "cname":"wang" to set "cage":11, your query needs to look like this:



          POST your-index/_update_by_query

          "script":
          "source": "ctx._source.cage = params.cage",
          "lang": "painless",
          "params":
          "cage": 11

          ,
          "query":
          "term":
          "cname": "wang"





          So your Logstash config should look like this (your input may vary but I used stdin for testing purposes):



          input 
          stdin
          codec => "json"


          filter
          mutate
          add_field =>
          "[script][lang]" => "painless"
          "[script][source]" => "ctx._source.cage = params.cage"
          "[script][params][cage]" => "%cage"
          "[query][term][cname]" => "%cname"

          remove_field => ["host", "@version", "@timestamp", "cname", "cage"]


          output
          http
          url => "http://localhost:9200/index/doc/_update_by_query"
          http_method => "post"
          format => "json"







          share|improve this answer

























          • ok, thanks very much

            – wangjinhao
            Nov 19 '18 at 3:14











          • Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you

            – wangjinhao
            Nov 21 '18 at 3:24











          • See my updated answer

            – Val
            Nov 21 '18 at 5:15















          0














          The elasticsearch output plugin can only make calls to the _bulk endpoint, i.e. using the Bulk API.



          If you want to call the Update by Query API, you need to use the http output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.



          Note: There's an issue requesting this feature, but it's still open after two years.



          UPDATE



          So if your input event is "cname":"wang", "cage":11 and you want to update by query all documents with "cname":"wang" to set "cage":11, your query needs to look like this:



          POST your-index/_update_by_query

          "script":
          "source": "ctx._source.cage = params.cage",
          "lang": "painless",
          "params":
          "cage": 11

          ,
          "query":
          "term":
          "cname": "wang"





          So your Logstash config should look like this (your input may vary but I used stdin for testing purposes):



          input 
          stdin
          codec => "json"


          filter
          mutate
          add_field =>
          "[script][lang]" => "painless"
          "[script][source]" => "ctx._source.cage = params.cage"
          "[script][params][cage]" => "%cage"
          "[query][term][cname]" => "%cname"

          remove_field => ["host", "@version", "@timestamp", "cname", "cage"]


          output
          http
          url => "http://localhost:9200/index/doc/_update_by_query"
          http_method => "post"
          format => "json"







          share|improve this answer

























          • ok, thanks very much

            – wangjinhao
            Nov 19 '18 at 3:14











          • Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you

            – wangjinhao
            Nov 21 '18 at 3:24











          • See my updated answer

            – Val
            Nov 21 '18 at 5:15













          0












          0








          0







          The elasticsearch output plugin can only make calls to the _bulk endpoint, i.e. using the Bulk API.



          If you want to call the Update by Query API, you need to use the http output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.



          Note: There's an issue requesting this feature, but it's still open after two years.



          UPDATE



          So if your input event is "cname":"wang", "cage":11 and you want to update by query all documents with "cname":"wang" to set "cage":11, your query needs to look like this:



          POST your-index/_update_by_query

          "script":
          "source": "ctx._source.cage = params.cage",
          "lang": "painless",
          "params":
          "cage": 11

          ,
          "query":
          "term":
          "cname": "wang"





          So your Logstash config should look like this (your input may vary but I used stdin for testing purposes):



          input 
          stdin
          codec => "json"


          filter
          mutate
          add_field =>
          "[script][lang]" => "painless"
          "[script][source]" => "ctx._source.cage = params.cage"
          "[script][params][cage]" => "%cage"
          "[query][term][cname]" => "%cname"

          remove_field => ["host", "@version", "@timestamp", "cname", "cage"]


          output
          http
          url => "http://localhost:9200/index/doc/_update_by_query"
          http_method => "post"
          format => "json"







          share|improve this answer















          The elasticsearch output plugin can only make calls to the _bulk endpoint, i.e. using the Bulk API.



          If you want to call the Update by Query API, you need to use the http output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.



          Note: There's an issue requesting this feature, but it's still open after two years.



          UPDATE



          So if your input event is "cname":"wang", "cage":11 and you want to update by query all documents with "cname":"wang" to set "cage":11, your query needs to look like this:



          POST your-index/_update_by_query

          "script":
          "source": "ctx._source.cage = params.cage",
          "lang": "painless",
          "params":
          "cage": 11

          ,
          "query":
          "term":
          "cname": "wang"





          So your Logstash config should look like this (your input may vary but I used stdin for testing purposes):



          input 
          stdin
          codec => "json"


          filter
          mutate
          add_field =>
          "[script][lang]" => "painless"
          "[script][source]" => "ctx._source.cage = params.cage"
          "[script][params][cage]" => "%cage"
          "[query][term][cname]" => "%cname"

          remove_field => ["host", "@version", "@timestamp", "cname", "cage"]


          output
          http
          url => "http://localhost:9200/index/doc/_update_by_query"
          http_method => "post"
          format => "json"








          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Nov 21 '18 at 5:15

























          answered Nov 16 '18 at 4:51









          ValVal

          108k6147181




          108k6147181












          • ok, thanks very much

            – wangjinhao
            Nov 19 '18 at 3:14











          • Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you

            – wangjinhao
            Nov 21 '18 at 3:24











          • See my updated answer

            – Val
            Nov 21 '18 at 5:15

















          • ok, thanks very much

            – wangjinhao
            Nov 19 '18 at 3:14











          • Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you

            – wangjinhao
            Nov 21 '18 at 3:24











          • See my updated answer

            – Val
            Nov 21 '18 at 5:15
















          ok, thanks very much

          – wangjinhao
          Nov 19 '18 at 3:14





          ok, thanks very much

          – wangjinhao
          Nov 19 '18 at 3:14













          Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you

          – wangjinhao
          Nov 21 '18 at 3:24





          Hi! I have a question. If my logstash input is ""cname":wang, "cage":11" and I want to update all documents with cname=wang and set cage to 11, how should I write the logstash conf file to achieve it? Thank you

          – wangjinhao
          Nov 21 '18 at 3:24













          See my updated answer

          – Val
          Nov 21 '18 at 5:15





          See my updated answer

          – Val
          Nov 21 '18 at 5:15



















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330232%2fdoes-logstash-support-elasticsearchs-update-by-query%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Top Tejano songwriter Luis Silva dead of heart attack at 64

          ReactJS Fetched API data displays live - need Data displayed static

          政党