Stream large uploads from disk, drop the 64 MiB upload limit

Hakalya commented

2026-10-12 09:25:32 +00:00

Collaborator

FOX-64. Uploads are read from the open file in 1 MiB pieces and sent as one request with a fixed Content-Length; memory no longer depends on file size and the mirror sets no upload limit by default (--max-upload-bytes adds one). A file that changes while it is sent, or a broken connection, publishes nothing (SAFETY.md M15). Live against the real service in throwaway folders: 128 MiB new/replace/broken upload/download and 4200 MiB new file through sync, read back byte-identical, peak process memory 34 MiB. Backend limits documented: Apple 50 GB per file, web upload is one request with no resume, answer arrives late for large bodies (PROTOCOL.md 1.1 and 6.4). Tests: 148 pass. 🤖 Generated with Claude Code

FOX-64. Uploads are read from the open file in 1 MiB pieces and sent as one request with a fixed Content-Length; memory no longer depends on file size and the mirror sets no upload limit by default (--max-upload-bytes adds one). A file that changes while it is sent, or a broken connection, publishes nothing (SAFETY.md M15). Live against the real service in throwaway folders: 128 MiB new/replace/broken upload/download and 4200 MiB new file through sync, read back byte-identical, peak process memory 34 MiB. Backend limits documented: Apple 50 GB per file, web upload is one request with no resume, answer arrives late for large bodies (PROTOCOL.md 1.1 and 6.4). Tests: 148 pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Hakalya added 1 commit

2026-10-12 09:25:32 +00:00

Stream uploads from disk and drop the 64 MiB upload limit 1ba4d12923

A local file is now read once, in 1 MiB pieces, while its multipart body is
written to the socket with a fixed Content-Length. Memory use no longer
depends on the file size, so the mirror sets no upload limit by default
(--max-upload-bytes adds one). All three upload paths of the mirror (new
file, replace, in-place overwrite) use it.

- transport: body_length sends an iterable body once, never retried;
  per-request timeout
- writer: sources with size/read/check; check runs after the bytes were
  sent and before the commit, so a file that changed meanwhile is not
  published; the wait for the upload answer grows with the size (measured:
  about 80 s after the last byte of a 4.1 GiB file)
- mirror: _Outgoing hashes while sending; the replace journal records
  sha/size/mtime once the content is stored
- docs: how the Windows client handles large files (chunk profiles from
  Apple's public plist, evidence levels), Apple's 50 GB per-file limit,
  live measurements up to 4.1 GiB, no resume on the web API
- tools/live_large_file_test.py: live check in a throwaway folder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

HoroTW merged commit d42feb0a92 into main

2026-10-12 09:28:02 +00:00

HoroTW referenced this pull request from a commit

2026-10-12 09:28:06 +00:00

Merge pull request 'Stream large uploads from disk, drop the 64 MiB upload limit' (#3) from hakalya/fox-64-large-uploads into main

Rows
Columns

Stream large uploads from disk, drop the 64 MiB upload limit #3