Discussion:
Roadmap for SQLite
Shaun Seckman (Firaxis)
2009-11-02 20:39:37 UTC
Permalink
Hello,

Is there any sort of development road map for SQLite?
I'm really interested in finding out what sort of major features or
tweaks are planned to be in upcoming releases (within the next 6
months). The foreign key constraint feature came as a pleasant surprise
but ended up invalidating a good chunk of code written to work around
the lack of checking (lots of trigger generation code). I'm also hoping
that storing prepared statements inside of a database is a planned
feature as well since having a bunch of SQL queries embedded in my C++
leaves me nervous :)



-Shaun
Roger Binns
2009-11-02 21:13:39 UTC
Permalink
Post by Shaun Seckman (Firaxis)
I'm really interested in finding out what sort of major features or
tweaks are planned to be in upcoming releases (within the next 6
months).
You get what you ask for :-) Have a look at these two pages:

http://www.sqlite.org/consortium.html
http://www.hwaci.com/sw/sqlite/prosupport.html
Post by Shaun Seckman (Firaxis)
I'm also hoping
that storing prepared statements inside of a database is a planned
feature as well since having a bunch of SQL queries embedded in my C++
leaves me nervous :)
You can buy an extension that does that (as has been pointed out several
times on this list). However what is wrong with storing the unprepared SQL
queries?

The prepared queries are byte code for the SQLite virtual machine (aka
VDBE). The generated byte code changes with each release (that is how many
features are added and bugs fixed), so you wouldn't want to keep them over
SQLite version changes. The normal use case is for limited environments
where you don't want the memory consumption of the SQLite parser. The
parsing is done ahead of time on the build workstation before putting the
database containing them onto the embedded device. Note that you also can't
change the database schema in any way since that invalidates the
pre-generated byte code.

Roger
Shaun Seckman (Firaxis)
2009-11-02 22:56:45 UTC
Permalink
The SQLite Consortium membership would be great but so far this
newsgroup has answered every single one of my questions within a 2 hour
window of posting..for free! (you guys frickin rock.) So SQLite
development is purely guided by requests from consortium members? There
aren't any planned features that the developers wish to include just for
maintenance or to add to already amazing feature set and polish of
SQLite? As for buying the extension, I remember seeing it in the list
of professional support a few months ago but it currently isn't listed
anymore.

The limitations of the prepared byte code you mention are pretty
acceptable in our use case. As you mentioned, my main reasons for
wanting this sort of feature is to reduce memory consumption and cpu
time from preparing statements and as an added benefit to have a
singular point where all prepared statements are located and could be
tested prior to use (granted, this could be done already via some
architecture and build process tweaks to my project). As for the byte
code not working with future versions of SQLite, we typically generate
the database file on the fly the first time the application is run, then
load it from disk for all subsequent runs. This cached database is
destroyed whenever a newer version of the exe is built (so new version
of SQLite would just require us to invalidate our cache and regenerate
it). The schema changes are a bit of a drag but would be rare and in
that case any prepared statement that is invalidated by them could just
be re-prepared.

Another potential feature that could benefit my usage and possibly
others is if prepared statements could use memory allocated from the
stack rather than the heap or if memory allocated for one statement
could be reused for another one (assuming that previous statement has
been completely finalized). I can understand this being very tricky to
implement since you must clearly define what items have the same
lifetime as the statement and which ones may be passed on later (ie.
strings).

It's not an absolute requirement for me right now though, just a nice to
have. So far I've really been impressed with the performance of SQLite
and the ability to bind parameters to prepared statements has met many
of my needs. In fact, the only major problem I've had with SQLite is
that the amount of polish has spoiled me and often makes me disgusted
when using other third party libraries which I won't name. (If only
other libraries had 100% branch coverage..)

-Shaun


-----Original Message-----
From: sqlite-users-bounces-CzDROfG0BjIdnm+***@public.gmane.org
[mailto:sqlite-users-bounces-CzDROfG0BjIdnm+***@public.gmane.org] On Behalf Of Roger Binns
Sent: Monday, November 02, 2009 4:14 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Roadmap for SQLite

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Post by Shaun Seckman (Firaxis)
I'm really interested in finding out what sort of major features or
tweaks are planned to be in upcoming releases (within the next 6
months).
You get what you ask for :-) Have a look at these two pages:

http://www.sqlite.org/consortium.html
http://www.hwaci.com/sw/sqlite/prosupport.html
Post by Shaun Seckman (Firaxis)
I'm also hoping
that storing prepared statements inside of a database is a planned
feature as well since having a bunch of SQL queries embedded in my C++
leaves me nervous :)
You can buy an extension that does that (as has been pointed out several
times on this list). However what is wrong with storing the unprepared
SQL
queries?

The prepared queries are byte code for the SQLite virtual machine (aka
VDBE). The generated byte code changes with each release (that is how
many
features are added and bugs fixed), so you wouldn't want to keep them
over
SQLite version changes. The normal use case is for limited environments
where you don't want the memory consumption of the SQLite parser. The
parsing is done ahead of time on the build workstation before putting
the
database containing them onto the embedded device. Note that you also
can't
change the database schema in any way since that invalidates the
pre-generated byte code.

Roger
Scott Hess
2009-11-02 23:26:26 UTC
Permalink
On Mon, Nov 2, 2009 at 3:56 PM, Shaun Seckman (Firaxis)
Post by Shaun Seckman (Firaxis)
The SQLite Consortium membership would be great but so far this
newsgroup has answered every single one of my questions within a 2 hour
window of posting..for free!  (you guys frickin rock.)  So SQLite
development is purely guided by requests from consortium members?  There
aren't any planned features that the developers wish to include just for
maintenance or to add to already amazing feature set and polish of
SQLite?
Speaking from past experience with open-source projects, often enough
if you have an explicit roadmap, people continually pester you with
messages like "Is it done yet?" and "Why isn't it done, yet?" and "My
gosh, how long could it possibly take you to implement such a trivial
feature?". Also, it discourages outside developers from doing it
themselves and submitting patches. If your project has a really big
development team, you need such a roadmap just to keep everyone
somewhat on-topic, but I think the set of core SQLite developers isn't
large enough to require that level of formality.

Realistically, if SQLite doesn't have the features you need right now,
it's probably risky to assume that it will acquire the features before
you need them. Either budget to work around the lacking features, or
budget to implement them. [And always remember that it's cheap to ask
- maybe the feature is in there, you just mis-understood something.]

-scott
P Kishor
2009-11-02 23:30:22 UTC
Permalink
Post by Scott Hess
And always remember that it's cheap to ask
- maybe the feature is in there, you just mis-understood something
Whereby "it's cheap to ask" implies that it is "inexpensive to ask"
and not "to ask is cowardly, impolite or deserving of contempt."

Very sound advice.
--
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
-----------------------------------------------------------------------
Assertions are politics; backing up assertions with evidence is science
=======================================================================
Roger Binns
2009-11-03 00:26:00 UTC
Permalink
Post by Shaun Seckman (Firaxis)
The SQLite Consortium membership would be great but so far this
newsgroup has answered every single one of my questions within a 2 hour
window of posting..for free!
The consortium membership is not about answering mailing list questions. It
makes the SQLite team become virtual members of your development team.
Buying support is in the same kind of vein but to a lesser extent.
Post by Shaun Seckman (Firaxis)
So SQLite
development is purely guided by requests from consortium members?
No, they just get highest priority. Look carefully at the links to get the
full list of benefits. Think of scenarios like hitting a critical bug two
days before a product release and not being able to tell if it is your code
or SQLite. Or having a new feature come out in a new SQLite version and
wanting it backported to an older version. Or desperately needing some
feature/functionality right now because some other component has gone over
CPU/memory/disk bandwidth budgets. Or maybe you want the test suites ported
to some new platform.
Post by Shaun Seckman (Firaxis)
As for buying the extension, I remember seeing it in the list
of professional support a few months ago but it currently isn't listed
anymore.
Contact them - HWACI is DRH's company - http://www.hwaci.com/contact.html
Post by Shaun Seckman (Firaxis)
Another potential feature that could benefit my usage and possibly
others is if prepared statements could use memory allocated from the
stack rather than the heap
That sounds rather tricky unless it is stack higher up in the call chain.
Again I'm sure DRH can discuss that with you.
Post by Shaun Seckman (Firaxis)
if memory allocated for one statement
could be reused for another one (assuming that previous statement has
been completely finalized).
All memory from a statement is freed when you finalize. Do you mean when
you reset?

Roger

Continue reading on narkive:
Search results for 'Roadmap for SQLite' (Questions and Answers)
20
replies
What Is FireFox?
started 2007-09-03 22:00:40 UTC
internet
Loading...