Improve error messages for Postgres query failures

[rkistner]

This improves error messages for Postgres query failures to assist with debugging. Unfortunately we have to wrap every query invocation (or the entire client) to achieve this, but this PR covers most of them.

This introduces two new error codes for Postgres:

/**
 * Error querying the source database.
 */
PSYNC_S1120 = 'PSYNC_S1120',

/**
 * Socket timeout while querying the source database.
 *
 * This can happen if Postgres takes too long to respond to a query, or if there is
 * a network-level issue.
 */
PSYNC_S1121 = 'PSYNC_S1121',

Before:

error: [powertest_184_d1c6] Replication error postgres query late {"stack":"Error: postgres query late\n    at PgConnection.stream (file:///home/ralf/src/powersync-service/node_modules/.pnpm/pgwire@0.8.1/node_modules/pgwire/mod.js:270:13)\n    at stream.next (<anonymous>)\n    at PgResult.fromStream (file:///home/ralf/src/powersync-service/node_modules/.pnpm/pgwire@0.8.1/node_modules/pgwire/mod.js:1063:22)\n    at PgConnection.query (file:///home/ralf/src/powersync-service/node_modules/.pnpm/pgwire@0.8.1/node_modules/pgwire/mod.js:266:21)\n    at WalStream.snapshotTableInTx (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:501:22)\n    at async file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:423:17\n    at async MongoSyncBucketStorageV3.startBatch (file:///home/ralf/src/powersync-service/modules/module-mongodb-storage/dist/storage/implementation/MongoSyncBucketStorage.js:129:9)\n    at async WalStream.initialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:399:30)\n    at async WalStream.startInitialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:395:9)\n    at async WalStream.initReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:793:13)"}
error: [powertest_184_d1c6] cause Socket idle timeout {"stack":"Error: Socket idle timeout\n    at Socket.<anonymous> (file:///home/ralf/src/powersync-service/packages/jpgwire/dist/socket_adapter.js:57:34)\n    at Socket.emit (node:events:509:28)\n    at Socket._onTimeout (node:net:610:8)\n    at listOnTimeout (node:internal/timers:605:17)\n    at process.processTimers (node:internal/timers:541:7)"}

After:

error: [powertest_184_d1c6] Replication error [PSYNC_S1121] Socket timed out while replicating
  cause: postgres query failed {"cause":{},"errorData":{"code":"PSYNC_S1121","description":"Socket timed out while replicating","details":"cause: postgres query failed","name":"DatabaseQueryError","stack":"Error: postgres query failed\n    at PgConnection.stream (file:///home/ralf/src/powersync-service/node_modules/.pnpm/pgwire@0.8.1/node_modules/pgwire/mod.js:334:15)\n    at async rstream (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/rquery.js:22:9)\n    at async ChunkedSnapshotQuery.nextChunk (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/SnapshotQuery.js:111:24)\n    at async WalStream.snapshotTable (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:552:28)\n    at async WalStream.snapshotTableInTx (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:479:13)\n    at async file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:424:17\n    at async MongoSyncBucketStorageV3.startBatch (file:///home/ralf/src/powersync-service/modules/module-mongodb-storage/dist/storage/implementation/MongoSyncBucketStorage.js:129:9)\n    at async WalStream.initialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:400:30)\n    at async WalStream.startInitialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:396:9)\n    at async WalStream.initReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:800:13)","status":500},"is_service_error":true,"name":"DatabaseQueryError","query":"SELECT * FROM \"public\".\"bulk_data\" WHERE \"id\" > $1 ORDER BY \"id\" LIMIT 10000","stack":"Error: postgres query failed\n    at PgConnection.stream (file:///home/ralf/src/powersync-service/node_modules/.pnpm/pgwire@0.8.1/node_modules/pgwire/mod.js:334:15)\n    at async rstream (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/rquery.js:22:9)\n    at async ChunkedSnapshotQuery.nextChunk (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/SnapshotQuery.js:111:24)\n    at async WalStream.snapshotTable (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:552:28)\n    at async WalStream.snapshotTableInTx (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:479:13)\n    at async file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:424:17\n    at async MongoSyncBucketStorageV3.startBatch (file:///home/ralf/src/powersync-service/modules/module-mongodb-storage/dist/storage/implementation/MongoSyncBucketStorage.js:129:9)\n    at async WalStream.initialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:400:30)\n    at async WalStream.startInitialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:396:9)\n    at async WalStream.initReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:800:13)"}
error: [powertest_184_d1c6] cause postgres query failed {"stack":"Error: postgres query failed\n    at PgConnection.stream (file:///home/ralf/src/powersync-service/node_modules/.pnpm/pgwire@0.8.1/node_modules/pgwire/mod.js:334:15)\n    at async rstream (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/rquery.js:22:9)\n    at async ChunkedSnapshotQuery.nextChunk (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/SnapshotQuery.js:111:24)\n    at async WalStream.snapshotTable (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:552:28)\n    at async WalStream.snapshotTableInTx (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:479:13)\n    at async file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:424:17\n    at async MongoSyncBucketStorageV3.startBatch (file:///home/ralf/src/powersync-service/modules/module-mongodb-storage/dist/storage/implementation/MongoSyncBucketStorage.js:129:9)\n    at async WalStream.initialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:400:30)\n    at async WalStream.startInitialReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:396:9)\n    at async WalStream.initReplication (file:///home/ralf/src/powersync-service/modules/module-postgres/dist/replication/WalStream.js:800:13)"}

There are still general log formatting issues here that make the output difficult to read, but at least the details are better:

1. The main error message is more useful: [PSYNC_S1121] Socket timed out while replicating instead of postgres query late
2. Includes the query: SELECT * FROM \"public\".\"bulk_data\" WHERE \"id\" > $1 ORDER BY \"id\" LIMIT 10000
3. Stack trace includes the original error location, rather than the rollback.

[changeset-bot]

🦋 Changeset detected

Latest commit: 3e6a820

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages

Name	Type
@powersync/service-module-postgres	Patch
@powersync/service-errors	Patch
@powersync/service-jpgwire	Patch
@powersync/service-schema	Patch
@powersync/service-image	Patch
@powersync/lib-services-framework	Patch
@powersync/service-module-mssql	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-rsocket-router	Patch
@powersync/service-core	Patch
@powersync/lib-service-mongodb	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-core-tests	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[mintlify]

Docs PR opened: powersync-ja/powersync-docs#416

Added entries for new Postgres replication error codes PSYNC_S1120 and PSYNC_S1121 to the error codes reference page.

[LucDeCaf]

Looks good; I'll mention that there are still a couple raw db.query calls outside the module-postgres/src/replication folder, but I'm guessing those are outside the original intended scope of rquery and rstream. It might be worth expanding the scope of the function to also wrap those errors, but that would make the error codes less specific, i.e. Socket timed out while replicating -> Socket timed out. I could also see a future change adding something like pgquery for generic error messages. Just ideas to play around with.

[rkistner]

Yeah, this focuses on the most important replication queries to start with.

Postgres storage is specifically excluded for now - we'd want different error codes for those. There are also a couple of cases using retriedQuery that are not covered yet. We could add similar functions to cover the other cases later.

But I think this should at least cover the most important cases that users are running into.