How many partitions does Kafka broker support
What is the limit of Kafka partitions that a Kafka broker can support? and if there is no limit, how many partition per Broker makes my cluster turn well and fast?
apache-kafka
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
asked Nov 13 '18 at 15:29
CaroleCarole
219112
add a comment |
What is the limit of Kafka partitions that a Kafka broker can support? and if there is no limit, how many partition per Broker makes my cluster turn well and fast?
apache-kafka
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
asked Nov 13 '18 at 15:29
CaroleCarole
219112
add a comment |
What is the limit of Kafka partitions that a Kafka broker can support? and if there is no limit, how many partition per Broker makes my cluster turn well and fast?
apache-kafka
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
asked Nov 13 '18 at 15:29
CaroleCarole
219112
What is the limit of Kafka partitions that a Kafka broker can support? and if there is no limit, how many partition per Broker makes my cluster turn well and fast?
apache-kafka
apache-kafka
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
asked Nov 13 '18 at 15:29
CaroleCarole
219112
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
asked Nov 13 '18 at 15:29
CaroleCarole
219112
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
edited Nov 13 '18 at 15:35
ctrl-alt-delor
4,17832443
4,17832443
asked Nov 13 '18 at 15:29
CaroleCarole
219112
asked Nov 13 '18 at 15:29
CaroleCarole
219112
asked Nov 13 '18 at 15:29
CaroleCarole
219112
219112
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
As of the recent blogs, 200K+ partitions per cluster, but that of course heavily depends on the actual hardware you run, and how well you can maintain Kafka up to that point. I doubt a Raspberry Pi will be able to handle that much load, for example.
There isn't exactly a hard limit per broker, but the old rule of thumb was less than 1000 partitions on average per broker will keep the cluster working optimally
That blog says 4000 is now a good number
As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
add a comment |
Recording to apache blog, the Limit that a Kafka broker can support up to 4,000 partitions and each cluster up to 200,000 partitions.
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53284309%2fhow-many-partitions-does-kafka-broker-support%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
As of the recent blogs, 200K+ partitions per cluster, but that of course heavily depends on the actual hardware you run, and how well you can maintain Kafka up to that point. I doubt a Raspberry Pi will be able to handle that much load, for example.
There isn't exactly a hard limit per broker, but the old rule of thumb was less than 1000 partitions on average per broker will keep the cluster working optimally
That blog says 4000 is now a good number
As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
add a comment |
As of the recent blogs, 200K+ partitions per cluster, but that of course heavily depends on the actual hardware you run, and how well you can maintain Kafka up to that point. I doubt a Raspberry Pi will be able to handle that much load, for example.
There isn't exactly a hard limit per broker, but the old rule of thumb was less than 1000 partitions on average per broker will keep the cluster working optimally
That blog says 4000 is now a good number
As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
add a comment |
As of the recent blogs, 200K+ partitions per cluster, but that of course heavily depends on the actual hardware you run, and how well you can maintain Kafka up to that point. I doubt a Raspberry Pi will be able to handle that much load, for example.
There isn't exactly a hard limit per broker, but the old rule of thumb was less than 1000 partitions on average per broker will keep the cluster working optimally
That blog says 4000 is now a good number
As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
As of the recent blogs, 200K+ partitions per cluster, but that of course heavily depends on the actual hardware you run, and how well you can maintain Kafka up to that point. I doubt a Raspberry Pi will be able to handle that much load, for example.
There isn't exactly a hard limit per broker, but the old rule of thumb was less than 1000 partitions on average per broker will keep the cluster working optimally
That blog says 4000 is now a good number
As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
answered Nov 13 '18 at 15:32
cricket_007cricket_007
80.2k1142110
80.2k1142110
add a comment |
add a comment |
Recording to apache blog, the Limit that a Kafka broker can support up to 4,000 partitions and each cluster up to 200,000 partitions.
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
add a comment |
Recording to apache blog, the Limit that a Kafka broker can support up to 4,000 partitions and each cluster up to 200,000 partitions.
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
add a comment |
Recording to apache blog, the Limit that a Kafka broker can support up to 4,000 partitions and each cluster up to 200,000 partitions.
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
Recording to apache blog, the Limit that a Kafka broker can support up to 4,000 partitions and each cluster up to 200,000 partitions.
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
answered Nov 13 '18 at 15:34
HISIHISI
1,4381922
1,4381922
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53284309%2fhow-many-partitions-does-kafka-broker-support%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
