Recreating lost seconds + milliseconds in Pandas DatetimeIndex
0
I have data from a gps unit which measures at 10 Hz but for some reason the vendor provides the timestamps up to minute precision. Thus, I end up with multiple replicates.
Is there a simple way to recreate the missing seconds and milliseconds assuming tat the order of the timestamps is correct and time starts at s = 0 and ms = 0?
Test case:
import pandas as pd
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
python pandas datetime
asked Nov 13 '18 at 14:55
ComUserComUser
32
add a comment |
0
I have data from a gps unit which measures at 10 Hz but for some reason the vendor provides the timestamps up to minute precision. Thus, I end up with multiple replicates.
Is there a simple way to recreate the missing seconds and milliseconds assuming tat the order of the timestamps is correct and time starts at s = 0 and ms = 0?
Test case:
import pandas as pd
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
python pandas datetime
asked Nov 13 '18 at 14:55
ComUserComUser
32
You can find the split the blanks using the known values by interpolation.
– MEdwin
Nov 13 '18 at 15:07
Hi, welcome to StackOverflow. I think your question may be a duplicate of this one: stackoverflow.com/questions/30056399/…
– Evan
Nov 13 '18 at 15:12
1
Possible duplicate of Interpolate and fill pandas dataframe with datetime index
– Evan
Nov 13 '18 at 15:12
add a comment |
0
0
0
I have data from a gps unit which measures at 10 Hz but for some reason the vendor provides the timestamps up to minute precision. Thus, I end up with multiple replicates.
Is there a simple way to recreate the missing seconds and milliseconds assuming tat the order of the timestamps is correct and time starts at s = 0 and ms = 0?
Test case:
import pandas as pd
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
python pandas datetime
asked Nov 13 '18 at 14:55
ComUserComUser
32
I have data from a gps unit which measures at 10 Hz but for some reason the vendor provides the timestamps up to minute precision. Thus, I end up with multiple replicates.
Is there a simple way to recreate the missing seconds and milliseconds assuming tat the order of the timestamps is correct and time starts at s = 0 and ms = 0?
Test case:
import pandas as pd
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
python pandas datetime
python pandas datetime
asked Nov 13 '18 at 14:55
ComUserComUser
32
asked Nov 13 '18 at 14:55
ComUserComUser
32
edited Nov 15 '18 at 15:00
ComUser
asked Nov 13 '18 at 14:55
ComUserComUser
32
asked Nov 13 '18 at 14:55
ComUserComUser
32
asked Nov 13 '18 at 14:55
ComUserComUser
32
32
You can find the split the blanks using the known values by interpolation.
– MEdwin
Nov 13 '18 at 15:07
Hi, welcome to StackOverflow. I think your question may be a duplicate of this one: stackoverflow.com/questions/30056399/…
– Evan
Nov 13 '18 at 15:12
1
Possible duplicate of Interpolate and fill pandas dataframe with datetime index
– Evan
Nov 13 '18 at 15:12
add a comment |
You can find the split the blanks using the known values by interpolation.
– MEdwin
Nov 13 '18 at 15:07
Hi, welcome to StackOverflow. I think your question may be a duplicate of this one: stackoverflow.com/questions/30056399/…
– Evan
Nov 13 '18 at 15:12
1
Possible duplicate of Interpolate and fill pandas dataframe with datetime index
– Evan
Nov 13 '18 at 15:12
You can find the split the blanks using the known values by interpolation.
– MEdwin
Nov 13 '18 at 15:07
You can find the split the blanks using the known values by interpolation.
– MEdwin
Nov 13 '18 at 15:07
Hi, welcome to StackOverflow. I think your question may be a duplicate of this one: stackoverflow.com/questions/30056399/…
– Evan
Nov 13 '18 at 15:12
Hi, welcome to StackOverflow. I think your question may be a duplicate of this one: stackoverflow.com/questions/30056399/…
– Evan
Nov 13 '18 at 15:12
1
1
Possible duplicate of Interpolate and fill pandas dataframe with datetime index
– Evan
Nov 13 '18 at 15:12
Possible duplicate of Interpolate and fill pandas dataframe with datetime index
– Evan
Nov 13 '18 at 15:12
add a comment |
1 Answer
1
active
oldest
votes
0
IIUC, you can redefine your dataframe indexing using pd.date_range:
np.random.seed(0)
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
df = pd.DataFrame(np.random.random(1801), index=ive_got)
Input dataframe print(df.head(10)):
0
2018-11-13 12:01:00 0.548814
2018-11-13 12:01:00 0.715189
2018-11-13 12:01:00 0.602763
2018-11-13 12:01:00 0.544883
2018-11-13 12:01:00 0.423655
2018-11-13 12:01:00 0.645894
2018-11-13 12:01:00 0.437587
2018-11-13 12:01:00 0.891773
2018-11-13 12:01:00 0.963663
2018-11-13 12:01:00 0.383442
Redefined your index using pd.date_range with a frequency:
df.index = pd.date_range(df.index.min(), df.index.max(), freq='100ms')
Output print(df.head(10)):
0
2018-11-13 12:01:00.000 0.548814
2018-11-13 12:01:00.100 0.715189
2018-11-13 12:01:00.200 0.602763
2018-11-13 12:01:00.300 0.544883
2018-11-13 12:01:00.400 0.423655
2018-11-13 12:01:00.500 0.645894
2018-11-13 12:01:00.600 0.437587
2018-11-13 12:01:00.700 0.891773
2018-11-13 12:01:00.800 0.963663
2018-11-13 12:01:00.900 0.383442
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53283734%2frecreating-lost-seconds-milliseconds-in-pandas-datetimeindex%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
IIUC, you can redefine your dataframe indexing using pd.date_range:
np.random.seed(0)
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
df = pd.DataFrame(np.random.random(1801), index=ive_got)
Input dataframe print(df.head(10)):
0
2018-11-13 12:01:00 0.548814
2018-11-13 12:01:00 0.715189
2018-11-13 12:01:00 0.602763
2018-11-13 12:01:00 0.544883
2018-11-13 12:01:00 0.423655
2018-11-13 12:01:00 0.645894
2018-11-13 12:01:00 0.437587
2018-11-13 12:01:00 0.891773
2018-11-13 12:01:00 0.963663
2018-11-13 12:01:00 0.383442
Redefined your index using pd.date_range with a frequency:
df.index = pd.date_range(df.index.min(), df.index.max(), freq='100ms')
Output print(df.head(10)):
0
2018-11-13 12:01:00.000 0.548814
2018-11-13 12:01:00.100 0.715189
2018-11-13 12:01:00.200 0.602763
2018-11-13 12:01:00.300 0.544883
2018-11-13 12:01:00.400 0.423655
2018-11-13 12:01:00.500 0.645894
2018-11-13 12:01:00.600 0.437587
2018-11-13 12:01:00.700 0.891773
2018-11-13 12:01:00.800 0.963663
2018-11-13 12:01:00.900 0.383442
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
add a comment |
0
IIUC, you can redefine your dataframe indexing using pd.date_range:
np.random.seed(0)
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
df = pd.DataFrame(np.random.random(1801), index=ive_got)
Input dataframe print(df.head(10)):
0
2018-11-13 12:01:00 0.548814
2018-11-13 12:01:00 0.715189
2018-11-13 12:01:00 0.602763
2018-11-13 12:01:00 0.544883
2018-11-13 12:01:00 0.423655
2018-11-13 12:01:00 0.645894
2018-11-13 12:01:00 0.437587
2018-11-13 12:01:00 0.891773
2018-11-13 12:01:00 0.963663
2018-11-13 12:01:00 0.383442
Redefined your index using pd.date_range with a frequency:
df.index = pd.date_range(df.index.min(), df.index.max(), freq='100ms')
Output print(df.head(10)):
0
2018-11-13 12:01:00.000 0.548814
2018-11-13 12:01:00.100 0.715189
2018-11-13 12:01:00.200 0.602763
2018-11-13 12:01:00.300 0.544883
2018-11-13 12:01:00.400 0.423655
2018-11-13 12:01:00.500 0.645894
2018-11-13 12:01:00.600 0.437587
2018-11-13 12:01:00.700 0.891773
2018-11-13 12:01:00.800 0.963663
2018-11-13 12:01:00.900 0.383442
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
add a comment |
0
0
0
IIUC, you can redefine your dataframe indexing using pd.date_range:
np.random.seed(0)
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
df = pd.DataFrame(np.random.random(1801), index=ive_got)
Input dataframe print(df.head(10)):
0
2018-11-13 12:01:00 0.548814
2018-11-13 12:01:00 0.715189
2018-11-13 12:01:00 0.602763
2018-11-13 12:01:00 0.544883
2018-11-13 12:01:00 0.423655
2018-11-13 12:01:00 0.645894
2018-11-13 12:01:00 0.437587
2018-11-13 12:01:00 0.891773
2018-11-13 12:01:00 0.963663
2018-11-13 12:01:00 0.383442
Redefined your index using pd.date_range with a frequency:
df.index = pd.date_range(df.index.min(), df.index.max(), freq='100ms')
Output print(df.head(10)):
0
2018-11-13 12:01:00.000 0.548814
2018-11-13 12:01:00.100 0.715189
2018-11-13 12:01:00.200 0.602763
2018-11-13 12:01:00.300 0.544883
2018-11-13 12:01:00.400 0.423655
2018-11-13 12:01:00.500 0.645894
2018-11-13 12:01:00.600 0.437587
2018-11-13 12:01:00.700 0.891773
2018-11-13 12:01:00.800 0.963663
2018-11-13 12:01:00.900 0.383442
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
IIUC, you can redefine your dataframe indexing using pd.date_range:
np.random.seed(0)
id_like = pd.date_range(start = '12:01:05', end = '12:04:05', freq='100ms')
ive_got = id_like.floor('1Min')
df = pd.DataFrame(np.random.random(1801), index=ive_got)
Input dataframe print(df.head(10)):
0
2018-11-13 12:01:00 0.548814
2018-11-13 12:01:00 0.715189
2018-11-13 12:01:00 0.602763
2018-11-13 12:01:00 0.544883
2018-11-13 12:01:00 0.423655
2018-11-13 12:01:00 0.645894
2018-11-13 12:01:00 0.437587
2018-11-13 12:01:00 0.891773
2018-11-13 12:01:00 0.963663
2018-11-13 12:01:00 0.383442
Redefined your index using pd.date_range with a frequency:
df.index = pd.date_range(df.index.min(), df.index.max(), freq='100ms')
Output print(df.head(10)):
0
2018-11-13 12:01:00.000 0.548814
2018-11-13 12:01:00.100 0.715189
2018-11-13 12:01:00.200 0.602763
2018-11-13 12:01:00.300 0.544883
2018-11-13 12:01:00.400 0.423655
2018-11-13 12:01:00.500 0.645894
2018-11-13 12:01:00.600 0.437587
2018-11-13 12:01:00.700 0.891773
2018-11-13 12:01:00.800 0.963663
2018-11-13 12:01:00.900 0.383442
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
edited Nov 15 '18 at 20:13
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
answered Nov 13 '18 at 15:13
Scott BostonScott Boston
55.8k73156
55.8k73156
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53283734%2frecreating-lost-seconds-milliseconds-in-pandas-datetimeindex%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You can find the split the blanks using the known values by interpolation.
– MEdwin
Nov 13 '18 at 15:07
Hi, welcome to StackOverflow. I think your question may be a duplicate of this one: stackoverflow.com/questions/30056399/…
– Evan
Nov 13 '18 at 15:12
1
Possible duplicate of Interpolate and fill pandas dataframe with datetime index
– Evan
Nov 13 '18 at 15:12