how to define the number of digits after second in timestamp of spark streaming data?
0
My timestamp in real data would be like
this or as shown below
2018-02-28T00:05:20.3717898Z
2018-02-28T00:05:23.6589778Z
2018-02-28T00:05:23.9119922Z
2018-02-28T00:05:25.4230787Z
2018-02-28T00:05:25.6710929Z
2018-02-28T00:05:26.4271361Z
And I use this code to read the data
userSchema=StructType().add('time','timestamp')
s=spark.readStream.schema(userSchema).csv('xxxx')
The result is like
this
Complete no idea how it happened.
apache-spark
edited Nov 16 '18 at 8:47
user238607
683712
asked Nov 15 '18 at 21:47
ellie ellie
11
add a comment |
0
My timestamp in real data would be like
this or as shown below
2018-02-28T00:05:20.3717898Z
2018-02-28T00:05:23.6589778Z
2018-02-28T00:05:23.9119922Z
2018-02-28T00:05:25.4230787Z
2018-02-28T00:05:25.6710929Z
2018-02-28T00:05:26.4271361Z
And I use this code to read the data
userSchema=StructType().add('time','timestamp')
s=spark.readStream.schema(userSchema).csv('xxxx')
The result is like
this
Complete no idea how it happened.
apache-spark
edited Nov 16 '18 at 8:47
user238607
683712
asked Nov 15 '18 at 21:47
ellie ellie
11
I think spark might be reading it in the correct format. What could be happening is that it is showing you the truncated form. Try to use s.show(10, truncate=false). Here is a question you with exactly the same problem as yours : stackoverflow.com/questions/33742895/…
– user238607
Nov 16 '18 at 7:17
Thanks, your answer is very heuristic. But the streaming object doesn't support shown() function. I tried to modify the timestamp format when read data and use option("truncate", False) for writestream(), the results look much better.
– ellie
Nov 16 '18 at 15:27
add a comment |
0
0
0
My timestamp in real data would be like
this or as shown below
2018-02-28T00:05:20.3717898Z
2018-02-28T00:05:23.6589778Z
2018-02-28T00:05:23.9119922Z
2018-02-28T00:05:25.4230787Z
2018-02-28T00:05:25.6710929Z
2018-02-28T00:05:26.4271361Z
And I use this code to read the data
userSchema=StructType().add('time','timestamp')
s=spark.readStream.schema(userSchema).csv('xxxx')
The result is like
this
Complete no idea how it happened.
apache-spark
edited Nov 16 '18 at 8:47
user238607
683712
asked Nov 15 '18 at 21:47
ellie ellie
11
My timestamp in real data would be like
this or as shown below
2018-02-28T00:05:20.3717898Z
2018-02-28T00:05:23.6589778Z
2018-02-28T00:05:23.9119922Z
2018-02-28T00:05:25.4230787Z
2018-02-28T00:05:25.6710929Z
2018-02-28T00:05:26.4271361Z
And I use this code to read the data
userSchema=StructType().add('time','timestamp')
s=spark.readStream.schema(userSchema).csv('xxxx')
The result is like
this
Complete no idea how it happened.
apache-spark
apache-spark
edited Nov 16 '18 at 8:47
user238607
683712
asked Nov 15 '18 at 21:47
ellie ellie
11
edited Nov 16 '18 at 8:47
user238607
683712
asked Nov 15 '18 at 21:47
ellie ellie
11
edited Nov 16 '18 at 8:47
user238607
683712
edited Nov 16 '18 at 8:47
user238607
683712
edited Nov 16 '18 at 8:47
user238607
683712
683712
asked Nov 15 '18 at 21:47
ellie ellie
11
asked Nov 15 '18 at 21:47
ellie ellie
11
asked Nov 15 '18 at 21:47
ellie ellie
11
11
I think spark might be reading it in the correct format. What could be happening is that it is showing you the truncated form. Try to use s.show(10, truncate=false). Here is a question you with exactly the same problem as yours : stackoverflow.com/questions/33742895/…
– user238607
Nov 16 '18 at 7:17
Thanks, your answer is very heuristic. But the streaming object doesn't support shown() function. I tried to modify the timestamp format when read data and use option("truncate", False) for writestream(), the results look much better.
– ellie
Nov 16 '18 at 15:27
add a comment |
I think spark might be reading it in the correct format. What could be happening is that it is showing you the truncated form. Try to use s.show(10, truncate=false). Here is a question you with exactly the same problem as yours : stackoverflow.com/questions/33742895/…
– user238607
Nov 16 '18 at 7:17
Thanks, your answer is very heuristic. But the streaming object doesn't support shown() function. I tried to modify the timestamp format when read data and use option("truncate", False) for writestream(), the results look much better.
– ellie
Nov 16 '18 at 15:27
I think spark might be reading it in the correct format. What could be happening is that it is showing you the truncated form. Try to use s.show(10, truncate=false). Here is a question you with exactly the same problem as yours : stackoverflow.com/questions/33742895/…
– user238607
Nov 16 '18 at 7:17
I think spark might be reading it in the correct format. What could be happening is that it is showing you the truncated form. Try to use s.show(10, truncate=false). Here is a question you with exactly the same problem as yours : stackoverflow.com/questions/33742895/…
– user238607
Nov 16 '18 at 7:17
Thanks, your answer is very heuristic. But the streaming object doesn't support shown() function. I tried to modify the timestamp format when read data and use option("truncate", False) for writestream(), the results look much better.
– ellie
Nov 16 '18 at 15:27
Thanks, your answer is very heuristic. But the streaming object doesn't support shown() function. I tried to modify the timestamp format when read data and use option("truncate", False) for writestream(), the results look much better.
– ellie
Nov 16 '18 at 15:27
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53328357%2fhow-to-define-the-number-of-digits-after-second-in-timestamp-of-spark-streaming%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53328357%2fhow-to-define-the-number-of-digits-after-second-in-timestamp-of-spark-streaming%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I think spark might be reading it in the correct format. What could be happening is that it is showing you the truncated form. Try to use s.show(10, truncate=false). Here is a question you with exactly the same problem as yours : stackoverflow.com/questions/33742895/…
– user238607
Nov 16 '18 at 7:17
Thanks, your answer is very heuristic. But the streaming object doesn't support shown() function. I tried to modify the timestamp format when read data and use option("truncate", False) for writestream(), the results look much better.
– ellie
Nov 16 '18 at 15:27