Extract rows between two identical strings?
Since I have a file which is huge (several GBs), I would not like to load the whole thing in memory and instead use *generators to load line by line. My file is something like this:
# millions of lines
..................
..................
keyw 28899
2233 121 ee 0o90 jjsl
2321 232 qq 0kj9 jksl
keyw 28900
3433 124 rr 8hu9 jkas
4532 343 ww 3ko9 aslk
1098 115 uy oiw8 rekl
keyw 29891
..................
..................
# millions more
So far I have found a similar answer here. But I am lost as how to implement it. Because the ans has specific identifiers Start and Stop, whereas my files have an incremental number with a identical keyword. I would like some help regarding this.
Edit: Generators not iterators
python
asked Nov 14 '18 at 13:28
rNovrNov
3113
add a comment |
Since I have a file which is huge (several GBs), I would not like to load the whole thing in memory and instead use *generators to load line by line. My file is something like this:
# millions of lines
..................
..................
keyw 28899
2233 121 ee 0o90 jjsl
2321 232 qq 0kj9 jksl
keyw 28900
3433 124 rr 8hu9 jkas
4532 343 ww 3ko9 aslk
1098 115 uy oiw8 rekl
keyw 29891
..................
..................
# millions more
So far I have found a similar answer here. But I am lost as how to implement it. Because the ans has specific identifiers Start and Stop, whereas my files have an incremental number with a identical keyword. I would like some help regarding this.
Edit: Generators not iterators
python
asked Nov 14 '18 at 13:28
rNovrNov
3113
Using iterators in python has nothing to do with whether data from disk is loaded to memory or not.
– SilverSlash
Nov 14 '18 at 13:32
I meant generators not iterators
– rNov
Nov 14 '18 at 13:36
add a comment |
Since I have a file which is huge (several GBs), I would not like to load the whole thing in memory and instead use *generators to load line by line. My file is something like this:
# millions of lines
..................
..................
keyw 28899
2233 121 ee 0o90 jjsl
2321 232 qq 0kj9 jksl
keyw 28900
3433 124 rr 8hu9 jkas
4532 343 ww 3ko9 aslk
1098 115 uy oiw8 rekl
keyw 29891
..................
..................
# millions more
So far I have found a similar answer here. But I am lost as how to implement it. Because the ans has specific identifiers Start and Stop, whereas my files have an incremental number with a identical keyword. I would like some help regarding this.
Edit: Generators not iterators
python
asked Nov 14 '18 at 13:28
rNovrNov
3113
Since I have a file which is huge (several GBs), I would not like to load the whole thing in memory and instead use *generators to load line by line. My file is something like this:
# millions of lines
..................
..................
keyw 28899
2233 121 ee 0o90 jjsl
2321 232 qq 0kj9 jksl
keyw 28900
3433 124 rr 8hu9 jkas
4532 343 ww 3ko9 aslk
1098 115 uy oiw8 rekl
keyw 29891
..................
..................
# millions more
So far I have found a similar answer here. But I am lost as how to implement it. Because the ans has specific identifiers Start and Stop, whereas my files have an incremental number with a identical keyword. I would like some help regarding this.
Edit: Generators not iterators
python
python
asked Nov 14 '18 at 13:28
rNovrNov
3113
asked Nov 14 '18 at 13:28
rNovrNov
3113
edited Nov 14 '18 at 13:36
rNov
asked Nov 14 '18 at 13:28
rNovrNov
3113
asked Nov 14 '18 at 13:28
rNovrNov
3113
asked Nov 14 '18 at 13:28
rNovrNov
3113
3113
Using iterators in python has nothing to do with whether data from disk is loaded to memory or not.
– SilverSlash
Nov 14 '18 at 13:32
I meant generators not iterators
– rNov
Nov 14 '18 at 13:36
add a comment |
Using iterators in python has nothing to do with whether data from disk is loaded to memory or not.
– SilverSlash
Nov 14 '18 at 13:32
I meant generators not iterators
– rNov
Nov 14 '18 at 13:36
Using iterators in python has nothing to do with whether data from disk is loaded to memory or not.
– SilverSlash
Nov 14 '18 at 13:32
Using iterators in python has nothing to do with whether data from disk is loaded to memory or not.
– SilverSlash
Nov 14 '18 at 13:32
I meant generators not iterators
– rNov
Nov 14 '18 at 13:36
I meant generators not iterators
– rNov
Nov 14 '18 at 13:36
add a comment |
1 Answer
1
active
oldest
votes
If you want to adapt that answer this may help:
bucket =
for line in infile:
if line.split()[0] == 'keyw':
for strings in bucket:
outfile.write( strings + 'n')
bucket =
continue
bucket.append(line.strip())
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
Damn. That was stupidly easy. My bad, for not sticking a while longer.
– rNov
Nov 14 '18 at 13:44
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53301362%2fextract-rows-between-two-identical-strings%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you want to adapt that answer this may help:
bucket =
for line in infile:
if line.split()[0] == 'keyw':
for strings in bucket:
outfile.write( strings + 'n')
bucket =
continue
bucket.append(line.strip())
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
Damn. That was stupidly easy. My bad, for not sticking a while longer.
– rNov
Nov 14 '18 at 13:44
add a comment |
If you want to adapt that answer this may help:
bucket =
for line in infile:
if line.split()[0] == 'keyw':
for strings in bucket:
outfile.write( strings + 'n')
bucket =
continue
bucket.append(line.strip())
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
Damn. That was stupidly easy. My bad, for not sticking a while longer.
– rNov
Nov 14 '18 at 13:44
add a comment |
If you want to adapt that answer this may help:
bucket =
for line in infile:
if line.split()[0] == 'keyw':
for strings in bucket:
outfile.write( strings + 'n')
bucket =
continue
bucket.append(line.strip())
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
If you want to adapt that answer this may help:
bucket =
for line in infile:
if line.split()[0] == 'keyw':
for strings in bucket:
outfile.write( strings + 'n')
bucket =
continue
bucket.append(line.strip())
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
answered Nov 14 '18 at 13:39
klashxxklashxx
10.1k12438
10.1k12438
Damn. That was stupidly easy. My bad, for not sticking a while longer.
– rNov
Nov 14 '18 at 13:44
add a comment |
Damn. That was stupidly easy. My bad, for not sticking a while longer.
– rNov
Nov 14 '18 at 13:44
Damn. That was stupidly easy. My bad, for not sticking a while longer.
– rNov
Nov 14 '18 at 13:44
Damn. That was stupidly easy. My bad, for not sticking a while longer.
– rNov
Nov 14 '18 at 13:44
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53301362%2fextract-rows-between-two-identical-strings%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Using iterators in python has nothing to do with whether data from disk is loaded to memory or not.
– SilverSlash
Nov 14 '18 at 13:32
I meant generators not iterators
– rNov
Nov 14 '18 at 13:36