Two-way folder mirror (sync) with offline catch-up #2

Merged

HoroTW merged 3 commits from hakalya/fox-60-hydrated-mirror into main

2026-10-12 05:53:51 +00:00

AGit

Hakalya commented

2026-10-12 04:52:35 +00:00

Collaborator

Stacked on PR 1 (6b910a5); the new work is the single commit 892d6f9. Adds sync / sync-status: one remote folder mirrored as ordinary local files in both directions, with a persistent index and change position, catch-up after a restart, fallback to complete listings when the position is refused, quarantine for remote deletions, conflict copies, journalled uploads. Tested at 892d6f9: 120 unit tests green; tools/live_mirror_test.py 18/18 against the real service inside a generated /Hakalya-Sora-Tests/ folder. Limits and guarantees: docs/SAFETY.md (mirror section), design: docs/PROTOCOL.md section 8. Not merged, not deployed. 🤖 Generated with Claude Code

Stacked on PR 1 (6b910a5); the new work is the single commit 892d6f9. Adds `sync` / `sync-status`: one remote folder mirrored as ordinary local files in both directions, with a persistent index and change position, catch-up after a restart, fallback to complete listings when the position is refused, quarantine for remote deletions, conflict copies, journalled uploads. Tested at 892d6f9: 120 unit tests green; tools/live_mirror_test.py 18/18 against the real service inside a generated /Hakalya-Sora-Tests/<run-id> folder. Limits and guarantees: docs/SAFETY.md (mirror section), design: docs/PROTOCOL.md section 8. Not merged, not deployed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Hakalya added 5 commits

2026-10-12 04:52:35 +00:00

Add read-only iCloud Drive web client prototype and protocol note 5e25a416a9

Standard-library Python client: SRP web sign-in with human second
factor, private session store (0700/0600), folder listing, single-file
read, bounded retry, and three read-only probes for the open
change-detection question. Tests run against an in-process fake that
keeps the server half of SRP, and against a local HTTP server.

docs/PROTOCOL.md records what the official Windows client's interface
metadata shows (CloudKit change tokens and APNs do apply to Drive),
the measured result that an app-specific password does not open
iCloud setup, and what is still unverified on the web path.

Refs FOX-58

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Route texted codes to the phone endpoint and resume half-finished sign-ins e69ea920e6

A real sign-in on an account without trusted devices showed that the
second-factor options request answers 201 with mode "sms" and makes
Apple text the code. The client only understood 200, fell back to the
trusted-device endpoint and got 400 / -21669 for a correct code.

- read mode and phone id from the options answer (200 or 201) and
  submit texted codes to verify/phone/securitycode
- a later `login` continues an accepted password at the code step, or
  after it when the code was already accepted; `--restart` starts over
- never fetch the options twice in one run (each fetch is a text)
- fake server models the text-only account; regression tests added
- protocol note: proof and 409 answer now confirmed on the wire

Refs FOX-58

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add Drive change feed (changes, watch) and record live verification 1ef7ff51b1

With one human sign-in the whole web path was exercised on a real
account: session setup, folder listing, a single-file read, and the
question the research was about. The web session can read the Drive
container through CloudKit Web Services, and changes/zone returns the
records changed since a syncToken, deletions included. An idle poll is
one request with an unchanged token.

- changes.py: zones/list + changes/zone, paging with a request cap,
  record parsing (folder, file, content, deletion), read-only
- cli: `changes` (baseline, then delta; position in a 0600 file) and
  `watch` (polling, 30 s floor)
- fake server follows the observed record shapes with invented data
- protocol note: evidence levels raised to "seen on the wire" where
  that is true, and the open points listed: a change made after the
  baseline has not been observed, push is untested

Refs FOX-58

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add minimal writer, safety contract and live write test 6b910a553a

Write tests on a real account were approved for a throwaway folder
(/Hakalya-Sora-Tests/<run-id>). They showed what the web API enforces:
create can be exclusive (409), move-to-trash is a compare-and-swap on
the etag, but content overwrite and rename check no version at all.
Two simultaneous overwrites were both accepted and one was lost.

- writer.py: mkdir, create (exclusive), replace (new document, then
  version-checked trash of the old one, then rename; conflict copy on
  mismatch), overwrite_in_place (documented as unsafe), rename, trash,
  journal and recover. No permanent delete. Optional sandbox folder.
- drive.py: a listing whose entry count differs from the announced
  count raises IncompleteListing instead of being used.
- cli: `cat -o` replaces the local file only after a full download.
- tools/live_write_test.py: 13 live checks incl. races and client-side
  fault injection; content-free report.
- docs/SAFETY.md: the contract, its tests, and what is not guaranteed.
- docs/PROTOCOL.md: write requests and measured server behaviour; a
  change made after the baseline is now observed in the change feed.

Refs FOX-58

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add two-way folder mirror (sync) with offline catch-up 892d6f9db1

sync keeps one remote folder and one local folder of ordinary files equal.
The change feed says where something changed; complete listings of those
folders are the truth. The new position and the new remote picture are
stored in one SQLite transaction after the feed was read to its end, and
the local folder is brought in line from that stored picture, so a pass
that stops anywhere is continued by the next one, also after a restart.

- only a complete listing can make an entry count as gone; a refused or
  expired position falls back to comparing every folder of the mirror
- local files are replaced or removed only when unchanged since the last
  sync; removed ones go to a quarantine outside the mirror; conflicts keep
  both versions
- local create, edit, rename, move and delete are sent with the server's
  checks (exclusive create, version-checked trash and move); a changed
  file is sent as a new document and the old one retired only if unchanged
- writer: move (version-checked by the server), measured live
- protocol note: trash and damaged positions in the feed, empty documents,
  moveItems, mirror design; safety contract for the mirror
- tests against the fake (hydration of 500 files, rg equality and timing
  against a plain copy, delta, restart, interrupted passes, conflicts) and
  tools/live_mirror_test.py for the real service

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Hakalya added 2 commits

2026-10-12 05:51:44 +00:00

Never follow a symlink inside the mirror; recover uploads only where the server still has the folder dd78901596

Local steps walk from the mirrored folder through open directories with
O_NOFOLLOW and act relative to the directory reached. A symlink found by
the scan is skipped with everything below it: not written through, not read,
not taken for a local deletion.

Interrupted uploads are continued after the remote view was read, and only
after every folder from the mirrored one down was found by id in a fresh
listing. A folder that left the mirror is not touched; the operation is
dropped and reported. The writer admits only folders the current picture
still has below the mirrored folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Mirror packages and app libraries (download only); live steps for the two safety fixes f918d63c0c

A package (Pages, Numbers, ...) is fetched through package_token.url, which
answers with a zip archive of the package. It is kept as one local file under
the document's name after checking that the archive is complete and its
members intact. The listed size is the sum of the members, not the length of
the archive, so the two are kept apart and a package is fetched once per
remote version. A local change to a package is never sent.

App libraries (the Pages, Numbers, ... folders) list like folders and are
mirrored now; nothing is sent into them.

tools/live_readonly_check.py mirrors a real folder with a transport that
refuses every request that is not a read. tools/live_mirror_test.py gained
the symlink and the recovery-scope case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>