RFC: Specialized Collection Types for Data Sources

[KyleAMathews]

Summary

This RFC proposes a new package @tanstack/db-collections that introduces specialized collection types for common data sources. By providing dedicated implementations for ElectricSQL and API-based data, we simplify the developer experience and enhance type safety. Instead of manually configuring sync implementations, developers can use purpose-built factory functions that handle the complexity of data synchronization while exposing source-specific features like transaction tracking for ElectricSQL or cache invalidation for API queries.

The proposal maintains the core Collection functionality while adding type-safe specialized methods where they make sense. This approach gives developers the best of both worlds: the familiar optimistic updates of the base Collection, plus streamlined integration with their chosen data source. By separating different collection types into their own classes, we provide better IDE support, clearer error messages, and a more maintainable codebase that can grow to support additional data sources in the future.

Background

1. The optimistic package currently has a base Collection class that provides core functionality for managing data with optimistic updates
2. Users currently create collections directly with new Collection(config)
3. Different data sources (ElectricSQL, API endpoints) require different sync implementations and helper methods
4. The base Collection class doesn't provide specialized methods for specific data sources

Problem

1. Users need to implement their own sync configurations for different data sources
2. Helper methods specific to data sources (e.g., awaitTxId for ElectricSQL) aren't available out of the box
3. There's no type safety around which methods are available for which data source
4. Configuration for different data sources isn't standardized

Proposal

1. Create a new package @tanstack/db-collections that provides specialized collection implementations:

// QueryCollection
interface QueryCollectionConfig<T> extends Omit<CollectionConfig<T>, 'sync'> {
  queryFn: () => Promise<T[]>;
  // Initial subset of TanStack Query options
  refetchInterval?: number;
  enabled?: boolean;
  retry?: boolean | number;
  // More options to be added as we support more TanStack Query features
}

class QueryCollection<T> extends Collection<T> {
  invalidate(): void;
}

function createQueryCollection<T>(config: QueryCollectionConfig<T>): QueryCollection<T>;

// ElectricCollection
interface ElectricCollectionConfig<T> extends Omit<CollectionConfig<T>, 'sync'> {
  streamOptions: ShapeStreamOptions;
  primaryKey: string[];
}

class ElectricCollection<T> extends Collection<T> {
  awaitTxId(txId: number): Promise<void>;
}

function createElectricCollection<T>(config: ElectricCollectionConfig<T>): ElectricCollection<T>;

2. Implementation Strategy:

   a. ElectricCollection:

   • Create internal sync configuration using streamOptions and primaryKey
   • Set up ShapeStream subscription in constructor
   • Track transaction IDs for awaitTxId functionality
   • Generate collection keys using primaryKey configuration
   • Surface ElectricSQL errors directly to developers

   b. QueryCollection:

   • Create internal sync configuration using queryFn and TanStack Query options
   • Initialize TanStack Query instance for data fetching
   • Map query results to collection format using standard schema
   • Expose invalidate method that triggers query refetch
   • Surface query errors directly to developers

3. Package Structure:

@tanstack/db-collections/
├── src/
│   ├── electric.ts       # ElectricCollection implementation
│   ├── query.ts         # QueryCollection implementation
│   └── index.ts         # Public exports

4. Public API:

// Main exports
export { createElectricCollection, createQueryCollection };
export type { ElectricCollection, QueryCollection };
export type { ElectricCollectionConfig, QueryCollectionConfig };

5. Error Handling:

   • Configuration errors (e.g., missing primary key) throw immediately
   • Runtime errors from ElectricSQL or query execution surface to developers
   • No automatic error recovery - developers handle all error cases

6. Type Safety:

   • Distinct types for each collection ensure correct method availability
   • Configuration interfaces extend base CollectionConfig appropriately
   • Factory functions enforce correct option combinations
   • Full IDE support for method discovery and type checking

Definition of Success

1. Developer Experience

   • Developers can create specialized collections with a single factory function call
   • TypeScript provides accurate autocompletion for collection-specific methods
   • Error messages are clear and point to configuration issues
   • No runtime surprises - behavior matches the collection type

2. API Design

   • ElectricCollection seamlessly integrates with ElectricSQL's shape streams
   • QueryCollection provides familiar TanStack Query-like configuration
   • Collection-specific methods (awaitTxId, invalidate) are intuitive and well-typed
   • Base Collection functionality remains unchanged and fully accessible

3. Implementation Quality

   • No memory leaks in subscriptions or event handlers
   • Efficient data synchronization with minimal overhead
   • Proper cleanup when collections are disposed
   • Type-safe without requiring type assertions

4. Extensibility

   • Architecture supports adding new collection types in the future
   • Easy to add more TanStack Query features to QueryCollection
   • Possible to add more ElectricSQL features to ElectricCollection
   • Clear pattern for implementing new collection-specific methods

[KyleAMathews]

One option to explore here is just wrapping tanstack/query — https://tanstack.com/query/latest/docs/reference/QueryClient/#queryclientfetchquery could be used under the hood.

Inside mutationFn, the query object could be exposed to invalidate it after the mutation finishes.

[KyleAMathews]

Ok, plan now is to make a @tanstack/db-collections package which provides various collection types people can use (to start with QueryCollection & ElectricCollection and more to come).

So you can write:

import { createQueryCollection } from "@tanstack/db-collections";

const todosCollection = createQueryCollection({
  queryFn: async () => fetch("/api/todos"),
});

const addTodo = useOptimisticMutation({
  mutationFn: async () => {
    // POST to /api/todos
    await todosCollection.invalidate();
  },
});

[KyleAMathews]

Updated the main body with a detailed proposal

[thruflo]

Makes lots of sense to me 👌