Shared in-memory SQLite database in shared memory

[Jaco Breitenbach]

Dear all,

My application consists of several indepent processes that must all access
(read) the same data table during processing. In order to optimise memory
usage I was wondering if it is possible to load an in-memory SQLite
database into shared memory. The database would be maintained by a
separate management process and all other processes would only require read
access. This way only one copy of the database would have to be held in
system memory.

Is this possible with the current implementation of SQLite? Any
suggestions would be welcome.

Best regards,
Jaco

[Simon Slavin]

Post by Jaco Breitenbach
My application consists of several indepent processes that must all access
(read) the same data table during processing. In order to optimise memory
usage I was wondering if it is possible to load an in-memory SQLite
database into shared memory.

Not simply and not without a little strangeness. However, SQLite has a Shared Cache mode which is easy to use, and you can open your database on disk, but declare a really big cache size for it. This will remove the need to use an in-memory database since SQLite will gradually read the whole database in to the cache. Read this:

<http://www.sqlite.org/sharedcache.html>

and this:

<http://www.sqlite.org/pragma.html#pragma_cache_size>

Simon.

[Jaco Breitenbach]

Hi Simon,

In my application I have multiple separate processes, not threads. I
believe the shared cache doesn't work across processes. I essentially want
a single shared read-only database that can be accessed at high speed.
Since the data set can become quite large and with multiple processes
requiring access to the identical data, I wanted to have the database in
shared memory. The database will contain reference information with which
the data I'm processing must be enriched. So once the database has been
loaded in memory, it will remain static and its contents will never change.

Best regards,
Jaco

Post by Jaco Breitenbach

Post by Jaco Breitenbach
My application consists of several indepent processes that must all

access

Post by Jaco Breitenbach
(read) the same data table during processing. In order to optimise

memory

Post by Jaco Breitenbach
usage I was wondering if it is possible to load an in-memory SQLite
database into shared memory.

Not simply and not without a little strangeness. However, SQLite has a
Shared Cache mode which is easy to use, and you can open your database on
disk, but declare a really big cache size for it. This will remove the
need to use an in-memory database since SQLite will gradually read the
<http://www.sqlite.org/sharedcache.html>
<http://www.sqlite.org/pragma.html#pragma_cache_size>
Simon.
_______________________________________________
sqlite-users mailing list
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[Roger Andersson]

I wanted to have the database in shared memory.

Maybe a ram drive?
What operating system are you running?

Cheers
Roger

[Simon Slavin]

Post by Jaco Breitenbach
In my application I have multiple separate processes, not threads.

Sorry, I missed that. You would seem to need some sort of sharing memory between your processes. Don't know how to do that.

Simon.

[Howard Chu]

Post by Jaco Breitenbach
Dear all,
My application consists of several indepent processes that must all access
(read) the same data table during processing. In order to optimise memory
usage I was wondering if it is possible to load an in-memory SQLite
database into shared memory. The database would be maintained by a
separate management process and all other processes would only require read
access. This way only one copy of the database would have to be held in
system memory.
Is this possible with the current implementation of SQLite? Any
suggestions would be welcome.

The OpenLDAP MDB (memory mapped database) library will do exactly what you
want. Since it uses a shared memory map to access the DB, no matter how many
processes access it concurrently there's only one copy of data present in RAM.
It also performs reads faster than anything else, even pure in-memory
databases. Nothing else is anywhere close to as efficient as MDB for reads.

Read more here http://highlandsun.com/hyc/mdb/

The port of SQLite using MDB as its backend is available on gitorious
https://gitorious.org/mdb/