Stephen Chin - 2003-06-30

Logged In: YES
user_id=532847

At least with the latest version of the database, you won't
have the corruption problem.

I've been doing some thinking about memory usage, and I
would really like to be able to have a persistent version of
sheets that never requires a database flush. You should be
able to just suck in an updated project file (or files from
disk), and have it overwrite/merge the existing classes
without having to rebuild... ever.

There are a few problems with this currently:
1. The database never recycles ids, so even if you never run
out of unique ids (which would take a very long time), you
will run out of disk space to remember the state of all the
ids. In most cases about 95% of the ids are "dead".
2. Sheets keeps indexes to do class reference lookups and
the like. These indexes are a single point of "bloat" which
you will be required to load whenever you want to do
anything interesting in sheets (and thus will suck up memory).
3. In general, very little memory profiling has been done on
the database or on sheets to see why so much memory is
needed to edit a sizable volume of code.
4. In the general case it would be really nice to be able to
load an infinite amount of data (stretched out across
multiple network loads). What this means for sheets is that
the queries (including the text query) need to have a
concept of incremental evaluation. In other words, you get
back a set of the query results, and can request/wait-for
more results at your discretion.

Well, a lot of this is pie in the sky, but I am doing some
work on items 1 and 3:
1. I'm almost done implementing an id compaction algorithm
for the database that runs incrementally on every commit.
This will definitely solve the problem of database id bloat
(by my estimate, cutting most databases down to 1/4 their
size), and should pave the road for having a database which
can stay up indefinitely.
3. I did do some profiling on the database, and fixed some
inefficient uses of objects. Now there is precisely one
ArchivalReference per object, and that behaves similar to a
SoftReference in that it keeps a pointer to the object it
references. This eliminates a lot of round-trip lookups
into the database, and reduced the overall size of the
loaded database by abount 20%. (I'll check this in along
with database compaction soon).

Oh, and about this bug... the simple answer is to integrate
a text lookup tool. There are a few nice ones that will
create a compact index of all the textual entries and should
avoid having to load all the objects in the database
one-by-one. ...any volunteers? :)