Skip to content

prevent errors when closing SQLite memory URLs in SqlCatalog - #4464

Open
ialarmedalien wants to merge 1 commit into
apache:mainfrom
ialarmedalien:sqlite_memory_gc
Open

ialarmedalien wants to merge 1 commit into
apache:mainfrom
ialarmedalien:sqlite_memory_gc

Conversation

@ialarmedalien

@ialarmedalien ialarmedalien commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Currently, if using an InMemoryCatalog (in-memory sqlite catalog) in a worker thread, closing or garbage collecting in the main thread raises an sqlite3.ProgrammingError. SQLAlchemy's Pool._close_connection() catches the exception, logs an ERROR traceback on sqlalchemy.pool, and leaves the underlying sqlite3.Connection open, which subsequently emits ResourceWarning: unclosed database when garbage collection occurs (Python 3.13+).

This PR sets the check_same_thread connection argument to default to False to allow the connection to be closed and garbage collection to occur.

Are these changes tested?

Yes -- see new tests:

  • close / garbage collection from a different thread succeeds
  • other operations remain single-thread only
  • explicit check_same_thread params are respected

Are there any user-facing changes?

No API changes.

Comment thread tests/catalog/test_sql.py
executor.submit(catalog.list_namespaces).result()

# closing the main thread connection also closes the worker thread's connection
with warnings.catch_warnings(record=True) as caught_warnings:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: these checks, which ensure that cross-thread close() and garbage collection complete successfully, are the same as those performed in the previous test (test_sql_catalog_cross_thread_sqlite_close). Depending on what the policy is on repetition and what a single test should cover, I can delete the previous test / delete the close/garbage collection code in this test / etc.

@Fokko

Fokko commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Thanks for digging into this @ialarmedalien! The traceback on close is annoying, but I'm not sure this is the right place to fix it.

The branch mostly repeats what SQLAlchemy already does. The pysqlite dialect already does pysqlite_opts.setdefault("check_same_thread", not self._is_url_file_db(url)). File-backed databases already get check_same_thread=False with a QueuePool, and a check_same_thread in the URL query is already respected. So the only behavior this changes is for in-memory databases, where SQLAlchemy keeps the check on deliberately. The "check_same_thread" not in url.query guard is only needed because connect_args overrides the URL. That puts the rule partly in our code and partly in the dialect, and the driver-name check misses variants like pysqlite_numeric.

It doesn't fix the underlying problem. With sqlite:///:memory: and SingletonThreadPool, every thread gets its own empty database. test_sql_catalog_sqlite_memory_preserves_thread_isolation actually asserts that a worker thread gets no such table. A catalog created in a worker and used or closed on the main thread was never usable across threads, and this PR only stops the error at close time. I don't think we should lock that behavior in with a test.

If we want InMemoryCatalog to work across threads, I'd scope the fix to InMemoryCatalog and use SQLAlchemy's documented recipe for sharing a memory database:

engine = create_engine("sqlite://", poolclass=StaticPool, connect_args={"check_same_thread": False})

Access has to be serialized, but the InMemoryCatalog docstring already says it doesn't support concurrent access. I tried it locally: a catalog created in a worker thread can be read from both threads and closes cleanly from the main thread, with no pool error or ResourceWarning. One test covering "create in a worker, use and close on the main thread" would be enough.

For SqlCatalog itself, I'd leave the engine construction alone. Users who want different behavior can already pass ?check_same_thread=false in the URI or pass their own engine=.

What do you think?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants