Make a hint to SQLite that a particular column is always sorted
up vote
0
down vote
favorite
I have the following table in an SQLite database
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
CREATE INDEX `time_index` ON `log`(`time`);
The index is created because the most frequent query is going to be
SELECT * FROM `log` WHERE `time` BETWEEN ? AND ?
Since the time is going to be always the current time when the new record is added, the index is not really required here. So I would like to "tell" the SQLite engine something like "The lines are going to be added with the 'time' column always having increasing value (similar to AUTO_INCREMENT), and if something goes wrong I will take all responsibility".
Is it possible at all?
sql sqlite sqlite3
add a comment |
up vote
0
down vote
favorite
I have the following table in an SQLite database
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
CREATE INDEX `time_index` ON `log`(`time`);
The index is created because the most frequent query is going to be
SELECT * FROM `log` WHERE `time` BETWEEN ? AND ?
Since the time is going to be always the current time when the new record is added, the index is not really required here. So I would like to "tell" the SQLite engine something like "The lines are going to be added with the 'time' column always having increasing value (similar to AUTO_INCREMENT), and if something goes wrong I will take all responsibility".
Is it possible at all?
sql sqlite sqlite3
Please explain what can go wrong, and what exactly you want to take responsibility for ? And what exactly would SQLite have to do with this information from you?
– krokodilko
Nov 12 at 8:14
When you have an array of sorted values, you can find a particular item in the array by bisecting it rather than making a sequential search. I want to tell SQLite that the rows in the table described above are going to be naturally sorted by the 'time' column. And by 'go wrong' I mean the violation of the "sorted" rule. In this case SQLite might still do bisection even though the lines are not sorted anymore, and I would be completely fine with that.
– Alexander Solovets
Nov 12 at 8:19
One thing: Your have yourtime
column with an affinity ofREAL
, but a default value that is aTEXT
string. Sqlite will happily let you do this, but it looks odd and might mislead people (Including yourself) about the format you intend to use with the column.
– Shawn
Nov 12 at 8:24
Thank you, @Shawn. Actually, the default value in the real code is different, it computes UNIX time using thejulianday
function. I just wanted simplify my example.
– Alexander Solovets
Nov 12 at 8:30
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have the following table in an SQLite database
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
CREATE INDEX `time_index` ON `log`(`time`);
The index is created because the most frequent query is going to be
SELECT * FROM `log` WHERE `time` BETWEEN ? AND ?
Since the time is going to be always the current time when the new record is added, the index is not really required here. So I would like to "tell" the SQLite engine something like "The lines are going to be added with the 'time' column always having increasing value (similar to AUTO_INCREMENT), and if something goes wrong I will take all responsibility".
Is it possible at all?
sql sqlite sqlite3
I have the following table in an SQLite database
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
CREATE INDEX `time_index` ON `log`(`time`);
The index is created because the most frequent query is going to be
SELECT * FROM `log` WHERE `time` BETWEEN ? AND ?
Since the time is going to be always the current time when the new record is added, the index is not really required here. So I would like to "tell" the SQLite engine something like "The lines are going to be added with the 'time' column always having increasing value (similar to AUTO_INCREMENT), and if something goes wrong I will take all responsibility".
Is it possible at all?
sql sqlite sqlite3
sql sqlite sqlite3
asked Nov 12 at 7:34
Alexander Solovets
1,6621020
1,6621020
Please explain what can go wrong, and what exactly you want to take responsibility for ? And what exactly would SQLite have to do with this information from you?
– krokodilko
Nov 12 at 8:14
When you have an array of sorted values, you can find a particular item in the array by bisecting it rather than making a sequential search. I want to tell SQLite that the rows in the table described above are going to be naturally sorted by the 'time' column. And by 'go wrong' I mean the violation of the "sorted" rule. In this case SQLite might still do bisection even though the lines are not sorted anymore, and I would be completely fine with that.
– Alexander Solovets
Nov 12 at 8:19
One thing: Your have yourtime
column with an affinity ofREAL
, but a default value that is aTEXT
string. Sqlite will happily let you do this, but it looks odd and might mislead people (Including yourself) about the format you intend to use with the column.
– Shawn
Nov 12 at 8:24
Thank you, @Shawn. Actually, the default value in the real code is different, it computes UNIX time using thejulianday
function. I just wanted simplify my example.
– Alexander Solovets
Nov 12 at 8:30
add a comment |
Please explain what can go wrong, and what exactly you want to take responsibility for ? And what exactly would SQLite have to do with this information from you?
– krokodilko
Nov 12 at 8:14
When you have an array of sorted values, you can find a particular item in the array by bisecting it rather than making a sequential search. I want to tell SQLite that the rows in the table described above are going to be naturally sorted by the 'time' column. And by 'go wrong' I mean the violation of the "sorted" rule. In this case SQLite might still do bisection even though the lines are not sorted anymore, and I would be completely fine with that.
– Alexander Solovets
Nov 12 at 8:19
One thing: Your have yourtime
column with an affinity ofREAL
, but a default value that is aTEXT
string. Sqlite will happily let you do this, but it looks odd and might mislead people (Including yourself) about the format you intend to use with the column.
– Shawn
Nov 12 at 8:24
Thank you, @Shawn. Actually, the default value in the real code is different, it computes UNIX time using thejulianday
function. I just wanted simplify my example.
– Alexander Solovets
Nov 12 at 8:30
Please explain what can go wrong, and what exactly you want to take responsibility for ? And what exactly would SQLite have to do with this information from you?
– krokodilko
Nov 12 at 8:14
Please explain what can go wrong, and what exactly you want to take responsibility for ? And what exactly would SQLite have to do with this information from you?
– krokodilko
Nov 12 at 8:14
When you have an array of sorted values, you can find a particular item in the array by bisecting it rather than making a sequential search. I want to tell SQLite that the rows in the table described above are going to be naturally sorted by the 'time' column. And by 'go wrong' I mean the violation of the "sorted" rule. In this case SQLite might still do bisection even though the lines are not sorted anymore, and I would be completely fine with that.
– Alexander Solovets
Nov 12 at 8:19
When you have an array of sorted values, you can find a particular item in the array by bisecting it rather than making a sequential search. I want to tell SQLite that the rows in the table described above are going to be naturally sorted by the 'time' column. And by 'go wrong' I mean the violation of the "sorted" rule. In this case SQLite might still do bisection even though the lines are not sorted anymore, and I would be completely fine with that.
– Alexander Solovets
Nov 12 at 8:19
One thing: Your have your
time
column with an affinity of REAL
, but a default value that is a TEXT
string. Sqlite will happily let you do this, but it looks odd and might mislead people (Including yourself) about the format you intend to use with the column.– Shawn
Nov 12 at 8:24
One thing: Your have your
time
column with an affinity of REAL
, but a default value that is a TEXT
string. Sqlite will happily let you do this, but it looks odd and might mislead people (Including yourself) about the format you intend to use with the column.– Shawn
Nov 12 at 8:24
Thank you, @Shawn. Actually, the default value in the real code is different, it computes UNIX time using the
julianday
function. I just wanted simplify my example.– Alexander Solovets
Nov 12 at 8:30
Thank you, @Shawn. Actually, the default value in the real code is different, it computes UNIX time using the
julianday
function. I just wanted simplify my example.– Alexander Solovets
Nov 12 at 8:30
add a comment |
2 Answers
2
active
oldest
votes
up vote
1
down vote
You don't want a separate index. You want to declare the column to be the primary key:
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY,
`data` BLOB NOT NULL
) WITHOUT ROWID;
This creates a single b-tree index for the log
based on the primary key. In other databases, this structure would be called a "clustered index". You have probably already read the documentation but I'm referencing it anyway.
That's the solution that I naturally discovered myself, but the problem is that the precision of REAL is not enough to handle microsecond resolution, and thus two adjacent records may have the sametime
value which would violate the PRIMARY KEY constraint. Is there a technique to work around this problem? Or is having a separate index my only solution?
– Alexander Solovets
Nov 12 at 23:02
add a comment |
up vote
1
down vote
You would have an issue, or not depending upon how you consider that you cannot use :-
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
because :-
Every WITHOUT ROWID table must have a PRIMARY KEY. An error is raised
if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a
PRIMARY KEY.
Clustered Indexes and the WITHOUT ROWID Optimization
So you might as well make the time column the PRIMARY KEY.
but the problem is that the precision of REAL is not enough to handle
microsecond resolution, and thus two adjacent records may have the
same time value which would violate the PRIMARY KEY constraint.
Then you could use a composite PRIMARY KEY where the precision required is satisfied by multiple columns (a second column would likely more than suffice) perhaps along the lines of :-
CREATE TABLE log (
time_datepart INTEGER,
time_microsecondpart,
data BLOB NOt NULL,
PRIMARY KEY (time_datepart,time_microsecondpart)
) WITHOUT ROWID;
The time_microsecondpart column needn't necessarily be microseconds it could be a counter derived from another table similar to how the sqlite_sequence table is utilised when AUTOINCREMENT is utilised (less the need for the column that holds the name of the table that a row is attached to).
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53257624%2fmake-a-hint-to-sqlite-that-a-particular-column-is-always-sorted%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
You don't want a separate index. You want to declare the column to be the primary key:
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY,
`data` BLOB NOT NULL
) WITHOUT ROWID;
This creates a single b-tree index for the log
based on the primary key. In other databases, this structure would be called a "clustered index". You have probably already read the documentation but I'm referencing it anyway.
That's the solution that I naturally discovered myself, but the problem is that the precision of REAL is not enough to handle microsecond resolution, and thus two adjacent records may have the sametime
value which would violate the PRIMARY KEY constraint. Is there a technique to work around this problem? Or is having a separate index my only solution?
– Alexander Solovets
Nov 12 at 23:02
add a comment |
up vote
1
down vote
You don't want a separate index. You want to declare the column to be the primary key:
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY,
`data` BLOB NOT NULL
) WITHOUT ROWID;
This creates a single b-tree index for the log
based on the primary key. In other databases, this structure would be called a "clustered index". You have probably already read the documentation but I'm referencing it anyway.
That's the solution that I naturally discovered myself, but the problem is that the precision of REAL is not enough to handle microsecond resolution, and thus two adjacent records may have the sametime
value which would violate the PRIMARY KEY constraint. Is there a technique to work around this problem? Or is having a separate index my only solution?
– Alexander Solovets
Nov 12 at 23:02
add a comment |
up vote
1
down vote
up vote
1
down vote
You don't want a separate index. You want to declare the column to be the primary key:
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY,
`data` BLOB NOT NULL
) WITHOUT ROWID;
This creates a single b-tree index for the log
based on the primary key. In other databases, this structure would be called a "clustered index". You have probably already read the documentation but I'm referencing it anyway.
You don't want a separate index. You want to declare the column to be the primary key:
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY,
`data` BLOB NOT NULL
) WITHOUT ROWID;
This creates a single b-tree index for the log
based on the primary key. In other databases, this structure would be called a "clustered index". You have probably already read the documentation but I'm referencing it anyway.
answered Nov 12 at 12:38
Gordon Linoff
754k35289397
754k35289397
That's the solution that I naturally discovered myself, but the problem is that the precision of REAL is not enough to handle microsecond resolution, and thus two adjacent records may have the sametime
value which would violate the PRIMARY KEY constraint. Is there a technique to work around this problem? Or is having a separate index my only solution?
– Alexander Solovets
Nov 12 at 23:02
add a comment |
That's the solution that I naturally discovered myself, but the problem is that the precision of REAL is not enough to handle microsecond resolution, and thus two adjacent records may have the sametime
value which would violate the PRIMARY KEY constraint. Is there a technique to work around this problem? Or is having a separate index my only solution?
– Alexander Solovets
Nov 12 at 23:02
That's the solution that I naturally discovered myself, but the problem is that the precision of REAL is not enough to handle microsecond resolution, and thus two adjacent records may have the same
time
value which would violate the PRIMARY KEY constraint. Is there a technique to work around this problem? Or is having a separate index my only solution?– Alexander Solovets
Nov 12 at 23:02
That's the solution that I naturally discovered myself, but the problem is that the precision of REAL is not enough to handle microsecond resolution, and thus two adjacent records may have the same
time
value which would violate the PRIMARY KEY constraint. Is there a technique to work around this problem? Or is having a separate index my only solution?– Alexander Solovets
Nov 12 at 23:02
add a comment |
up vote
1
down vote
You would have an issue, or not depending upon how you consider that you cannot use :-
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
because :-
Every WITHOUT ROWID table must have a PRIMARY KEY. An error is raised
if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a
PRIMARY KEY.
Clustered Indexes and the WITHOUT ROWID Optimization
So you might as well make the time column the PRIMARY KEY.
but the problem is that the precision of REAL is not enough to handle
microsecond resolution, and thus two adjacent records may have the
same time value which would violate the PRIMARY KEY constraint.
Then you could use a composite PRIMARY KEY where the precision required is satisfied by multiple columns (a second column would likely more than suffice) perhaps along the lines of :-
CREATE TABLE log (
time_datepart INTEGER,
time_microsecondpart,
data BLOB NOt NULL,
PRIMARY KEY (time_datepart,time_microsecondpart)
) WITHOUT ROWID;
The time_microsecondpart column needn't necessarily be microseconds it could be a counter derived from another table similar to how the sqlite_sequence table is utilised when AUTOINCREMENT is utilised (less the need for the column that holds the name of the table that a row is attached to).
add a comment |
up vote
1
down vote
You would have an issue, or not depending upon how you consider that you cannot use :-
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
because :-
Every WITHOUT ROWID table must have a PRIMARY KEY. An error is raised
if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a
PRIMARY KEY.
Clustered Indexes and the WITHOUT ROWID Optimization
So you might as well make the time column the PRIMARY KEY.
but the problem is that the precision of REAL is not enough to handle
microsecond resolution, and thus two adjacent records may have the
same time value which would violate the PRIMARY KEY constraint.
Then you could use a composite PRIMARY KEY where the precision required is satisfied by multiple columns (a second column would likely more than suffice) perhaps along the lines of :-
CREATE TABLE log (
time_datepart INTEGER,
time_microsecondpart,
data BLOB NOt NULL,
PRIMARY KEY (time_datepart,time_microsecondpart)
) WITHOUT ROWID;
The time_microsecondpart column needn't necessarily be microseconds it could be a counter derived from another table similar to how the sqlite_sequence table is utilised when AUTOINCREMENT is utilised (less the need for the column that holds the name of the table that a row is attached to).
add a comment |
up vote
1
down vote
up vote
1
down vote
You would have an issue, or not depending upon how you consider that you cannot use :-
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
because :-
Every WITHOUT ROWID table must have a PRIMARY KEY. An error is raised
if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a
PRIMARY KEY.
Clustered Indexes and the WITHOUT ROWID Optimization
So you might as well make the time column the PRIMARY KEY.
but the problem is that the precision of REAL is not enough to handle
microsecond resolution, and thus two adjacent records may have the
same time value which would violate the PRIMARY KEY constraint.
Then you could use a composite PRIMARY KEY where the precision required is satisfied by multiple columns (a second column would likely more than suffice) perhaps along the lines of :-
CREATE TABLE log (
time_datepart INTEGER,
time_microsecondpart,
data BLOB NOt NULL,
PRIMARY KEY (time_datepart,time_microsecondpart)
) WITHOUT ROWID;
The time_microsecondpart column needn't necessarily be microseconds it could be a counter derived from another table similar to how the sqlite_sequence table is utilised when AUTOINCREMENT is utilised (less the need for the column that holds the name of the table that a row is attached to).
You would have an issue, or not depending upon how you consider that you cannot use :-
CREATE TABLE `log` (
`time` REAL NOT NULL DEFAULT CURRENT_TIMESTAMP,
`data` BLOB NOT NULL
) WITHOUT ROWID;
because :-
Every WITHOUT ROWID table must have a PRIMARY KEY. An error is raised
if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a
PRIMARY KEY.
Clustered Indexes and the WITHOUT ROWID Optimization
So you might as well make the time column the PRIMARY KEY.
but the problem is that the precision of REAL is not enough to handle
microsecond resolution, and thus two adjacent records may have the
same time value which would violate the PRIMARY KEY constraint.
Then you could use a composite PRIMARY KEY where the precision required is satisfied by multiple columns (a second column would likely more than suffice) perhaps along the lines of :-
CREATE TABLE log (
time_datepart INTEGER,
time_microsecondpart,
data BLOB NOt NULL,
PRIMARY KEY (time_datepart,time_microsecondpart)
) WITHOUT ROWID;
The time_microsecondpart column needn't necessarily be microseconds it could be a counter derived from another table similar to how the sqlite_sequence table is utilised when AUTOINCREMENT is utilised (less the need for the column that holds the name of the table that a row is attached to).
edited Nov 13 at 1:43
answered Nov 12 at 10:49
MikeT
14.3k102441
14.3k102441
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53257624%2fmake-a-hint-to-sqlite-that-a-particular-column-is-always-sorted%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Please explain what can go wrong, and what exactly you want to take responsibility for ? And what exactly would SQLite have to do with this information from you?
– krokodilko
Nov 12 at 8:14
When you have an array of sorted values, you can find a particular item in the array by bisecting it rather than making a sequential search. I want to tell SQLite that the rows in the table described above are going to be naturally sorted by the 'time' column. And by 'go wrong' I mean the violation of the "sorted" rule. In this case SQLite might still do bisection even though the lines are not sorted anymore, and I would be completely fine with that.
– Alexander Solovets
Nov 12 at 8:19
One thing: Your have your
time
column with an affinity ofREAL
, but a default value that is aTEXT
string. Sqlite will happily let you do this, but it looks odd and might mislead people (Including yourself) about the format you intend to use with the column.– Shawn
Nov 12 at 8:24
Thank you, @Shawn. Actually, the default value in the real code is different, it computes UNIX time using the
julianday
function. I just wanted simplify my example.– Alexander Solovets
Nov 12 at 8:30