Downloads wait when files are put in place more slowly than they are fetched; progress is said after each file

Hakalya commented

2026-10-12 15:03:06 +00:00

Collaborator

FOX-72. Seen on a slow disk with 10 downloads at a time: about 50 fetched files waited in staging (an interruption throws them away) and no progress line came for 11 minutes. Now at most twice --jobs files are fetched or on their way and not yet in place, and the progress line is written after each file. Two new tests, both fail on main (40 staged instead of at most 4; lines only for 5 instead of 1 to 5). Full suite: 181 tests OK. Live check tools/live_jobs_check.py with the change: 8 of 8 PASS, rates as before (6.2 small files/s and 11.1 MiB/s at 10). Not deployed, the pilot on Naomi is untouched. 🤖 Generated with Claude Code

FOX-72. Seen on a slow disk with 10 downloads at a time: about 50 fetched files waited in staging (an interruption throws them away) and no progress line came for 11 minutes. Now at most twice --jobs files are fetched or on their way and not yet in place, and the progress line is written after each file. Two new tests, both fail on main (40 staged instead of at most 4; lines only for 5 instead of 1 to 5). Full suite: 181 tests OK. Live check tools/live_jobs_check.py with the change: 8 of 8 PASS, rates as before (6.2 small files/s and 11.1 MiB/s at 10). Not deployed, the pilot on Naomi is untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Hakalya added 1 commit

2026-10-12 15:03:06 +00:00

Downloads wait when files are put in place more slowly than they are fetched; progress is said after each file 6ff0cbb526

At most twice --jobs files are fetched or on their way and not yet in place.
Before, every download was queued at once: on a slow disk about 50 finished
files waited in the staging folder (thrown away by an interruption) and no
progress line came for 11 minutes, because it was only written between two
rows of finished files (FOX-72).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Hakalya added 1 commit

2026-10-12 15:15:46 +00:00

A pass waits for the index when another process holds it, instead of ending in "database is locked" cfdf96ed3f

Seen in the same run (FOX-72): after 25 minutes the pass ended with
sqlite3.OperationalError: database is locked. Another process had the index
open for longer than the 5 seconds sqlite3 waits by default, on a disk that
was 93% busy. The restart threw 310 fetched files away. The index is now
waited for up to 5 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Hakalya added 1 commit

2026-10-12 15:21:05 +00:00

The index keeps a write-ahead log: reading it no longer ends a pass in "database is locked" 5c0a2bf3c4

With the rollback journal a write cannot be committed while anyone is in the
middle of reading, and sqlite3 gives up after 5 seconds. That ended the pass
on 2026-10-12 (FOX-72). Rebuilt with a read-only query not read to its end:
"database is locked" after 5.02s before, written at once now. Every commit
is still synced, with one sync instead of four. Two writers wait for each
other up to 60 seconds (was 300 in the commit before, as the only remedy).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Hakalya added 2 commits

2026-10-12 15:30:59 +00:00

Revert "The index keeps a write-ahead log: reading it no longer ends a pass in "database is locked"" 37db2184b6

This reverts commit 5c0a2bf3c4.

FOX-75 asks for a sync-status that changes no file and for no change of the
journal mode without measurements on the disk in question. The two do not go
together: measured, a read-only sync-status on an index with a write-ahead
log that no pass has open makes mirror.db-shm and mirror.db-wal and leaves
them there. The index of a running mirror would also be changed at the next
start. What the log gains (a read and a commit do not wait for each other,
one sync per commit instead of four) is kept in docs/PROTOCOL.md 8.2 by the
next commit, so it can be decided on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sync-status only reads the index: nothing is made or changed, a pass is not held up, "busy" after 10 seconds 19b0d6a5fa

sync-status opened the index the way a pass does. Reproduced on cfdf96e with
the tests' service: on an index of an earlier version it added the missing
column, ran an update over all entries, set the file mode; for a folder that
was never mirrored it made a state folder and an index; behind a pass that
was inside a transaction it gave no answer (stopped after 20s).

It now opens the file read-only, reads one committed state within 10 seconds
and closes it before it counts. If a commit is under way for that long it
says "busy" (exit status 75). After a write that was cut off it says so and
leaves the repair to the next pass. sync itself sets the index up as before.

A commit still waits for a read that is under way (rollback journal). For
sync-status that is 10 seconds at most; the 5 minutes of cfdf96e stay for
programs that query the index themselves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

HoroTW merged commit 60c9b16fec into main

2026-10-12 15:32:58 +00:00

HoroTW referenced this pull request from a commit

2026-10-12 15:32:59 +00:00

Merge pull request 'Downloads wait when files are put in place more slowly than they are fetched; progress is said after each file' (#8) from hakalya/fox-72-landing-backlog into main

Rows
Columns

Downloads wait when files are put in place more slowly than they are fetched; progress is said after each file #8