icloud-drive-linux

Independent Linux iCloud Drive protocol research and Python prototype. Experimental; no Apple affiliation.

Warning

Not a full replacement for Apple's iCloud Drive client. Pages/Numbers/Keynote document packages and documents inside app libraries are download-only. Do not edit their local mirror copies expecting the changes to upload: this client does not sync those edits back. Use the Apple app or iCloud web app for editing the original cloud document.

Package downloads work for both account-owned documents and received shares. Downloaded packages are stored as archives; document can convert them, but opening the converted result in an Apple app has not yet been verified. Conversion does not enable package uploads.

Shared-document identity matters: by default, edits to shared files are held and reported rather than uploaded as replacement documents. The optional --shared-in-place mode preserves identity but cannot guarantee protection against simultaneous edits; do not enable it as a workaround for package uploads. See the safety contract.

Status

Prototype. Standard library only, Python 3.10 or newer.

sync keeps one iCloud Drive folder and one local folder equal, in both directions. The local folder holds ordinary, complete files: rg, editors and everything else work on it at disk speed, also offline and when this program is not running.

Part	State
Sign-in (password proof, texted code, session)	confirmed against the real service, once, on an account without trusted Apple devices
Sign-in with a code from a trusted Apple device	implemented, tested against a fake only
List a folder, read a file	confirmed against the real service
Change feed (changes, watch)	confirmed, including changes made after the baseline, trashed entries and a refused position
Push notifications	not implemented; watch polls
Writing	create, replace, rename, move and trash confirmed against the real service in a throwaway folder, including races. See docs/SAFETY.md for what is and is not guaranteed
Two-way mirror (sync)	confirmed against the real service in a throwaway folder: first download, later changes in both directions, catching up after the program was not running, conflicts, interrupted passes. "The other device" in that test was a second web client, not a Mac
Packages (Pages, Numbers, ...)	downloaded as the zip archive the service sends for a package, under the document's name; verified on account-owned and received-share packages. Download only: the client does not upload modified packages. document writes Apple's single-file form or the package folder from it, with the same members; opening the converted result in an Apple app has not been verified
Shared documents and folders	recognised from the listing. Shared entries are never given a new document id or trashed by the mirror; isolated received-share create, in-place update and rename probes succeeded. Shared deletion remains protected; recovery of a separately deleted probe is unresolved
Folders other accounts share with this one	listed, downloaded and followed in the change feed, confirmed against the real service. Original packages use the owner-qualified download route. Received-folder upload/commit and rename routing is implemented; see write route evidence and limits
App libraries (the "Pages", "Numbers", ... folders)	mirrored, download only: nothing is sent into them
FUSE	not built; not needed for searching (see the protocol note, 8.4)

Use

export PYTHONPATH=src            # or: pip install -e .
python3 -m icloud_drive_linux login          # Apple ID, password, then the code from a trusted device
python3 -m icloud_drive_linux status
python3 -m icloud_drive_linux ls -l /
python3 -m icloud_drive_linux cat /some/file.txt -o file.txt
python3 -m icloud_drive_linux changes        # first call: baseline; later calls: what changed since
python3 -m icloud_drive_linux watch --interval 60
python3 -m icloud_drive_linux logout         # deletes the local session files only

Mirror a folder

python3 -m icloud_drive_linux sync ~/icloud/Notes --remote /Projects/Notes   # first time: downloads everything in it
python3 -m icloud_drive_linux sync ~/icloud/Notes                            # later: what changed, both directions
python3 -m icloud_drive_linux sync ~/icloud/Notes --watch --interval 60      # keep going
python3 -m icloud_drive_linux sync-status ~/icloud/Notes                     # no request, only reads; works offline
rg -n 'some text' ~/icloud/Notes                                             # plain files

• The exit status is 0 only for ready (or ready-with-exclusions: every file the mirror can hold is local; names that cannot exist locally are counted). incomplete, failed and interrupted are 2 or 1; such a folder is never called ready.

• sync-status only reads: it makes no folder and no file and changes nothing in the index, also not in one that an earlier version wrote (sync brings that up to date). It can be asked while a pass runs and then shows what the pass has stored so far; the state is interrupted until the pass has ended. If the index cannot be read within 10 seconds because a pass is writing to it just then, it says busy and the exit status is 75: ask again.

• A pass with nothing to do is one request. After the program, or the machine, was off, the next pass picks up at the stored position: files deleted remotely leave the folder, the rest is applied.

• A local file is only replaced or removed if it still is what the last sync put there. Removed files are moved to a quarantine folder outside the mirror (its path is in sync-status); empty it yourself. If both sides changed a file, both versions are kept, yours as name (conflict <host> <time>).ext.

• Create, edit, rename, move and delete in the local folder are sent on the next pass. An edited file becomes a new document on the server and the old one goes to the iCloud trash; that is the only way this web API can refuse to overwrite a version it has not seen. --in-place keeps the document and gives that protection up. Details: docs/SAFETY.md.

• One folder per local directory, named with --remote; the whole drive only with --whole-drive (tried read-only on a small drive). At most 1 GiB is downloaded per pass unless --max-bytes says otherwise (--max-bytes 0: no limit; needed once for a folder that holds a larger file). --no-upload sends nothing.

• Downloads run 2 at a time; --jobs takes 1 to 10. Many small files gain most (measured: 10 at a time about 6 times as fast as 1), large ones are bound by the line; see docs/PROTOCOL.md, 8.2.

• Large files are sent and fetched as streams from and to disk; memory use does not depend on the file size, and this program sets no upload limit (--max-upload-bytes adds one). Tried against the real service up to 4.1 GiB in one file; Apple's limit is 50 GB per file. A large upload is one request: if the connection breaks, nothing of it becomes visible and the next pass sends the file again from its first byte. There is no resume and no chunk deduplication as in Apple's own clients; see docs/PROTOCOL.md, sections 1.1 and 6.4.

• The index lives in the state directory (mirrors/), not in the mirrored folder, which only gets a hidden .icdl-mirror-id file. Names starting with .icdl- are never synced.

• A Pages, Numbers or Keynote document that is a package arrives as one file, Name.pages: a zip archive with one folder Name.pages/ in it, which is the package (unzip gives it back). The mirror checks that the archive is complete and its members intact. It is fetched once per remote version. Changing it locally sends nothing; renaming, moving and deleting it are sent like for any file.

• document FILE says what a local file is and, for a package archive, writes the other forms beside it; the mirror and the remote document are not touched, and an existing file is never overwritten:

  python3 -m icloud_drive_linux document ~/icloud/Plan.numbers                          # form, members, fingerprint
  python3 -m icloud_drive_linux document ~/icloud/Plan.numbers --single-file ~/Plan.numbers
  python3 -m icloud_drive_linux document ~/icloud/Letter.pages --folder ~/Letter.pages
  

  The single file is the form Pages, Numbers and Keynote call "Single File": one file that can be mailed, opened by tools that read the format, or put into iCloud Drive as a new document. Write it outside the mirror, or under another name: inside the mirror it is sent as a new, ordinary file. Editing stays with Apple's apps and iCloud.com; see docs/PROTOCOL.md, section 9.

• A shared file keeps its document. A share and its link belong to the document, and the safe way of sending a change makes a new one. So a changed shared file is not sent and the pass says so; --shared-in-place sends it under the same document, with the limit of --in-place for these files only. A shared file or folder deleted here is not trashed: it comes back, and sync-status shows the event. Stop sharing it, or delete it on iCloud.com. sync-status counts the shared entries. The first pass after an update from a version without this rule reads every folder once.

• Folders another account shares with this one ("received shares") live in that account's zone. Listing one needs its share, its documents are fetched through another ticket, and its changes are in a feed of their own, which the mirror follows too; a share received later is found by the next complete scan (--full-interval). Such a folder costs the service seconds to list, so a scan keeps up to --jobs listings under way. A package (Pages, Numbers, ...) in it is downloaded as an original package archive through download/by_id/<owner>. A successful batch ticket alone is not sufficient: that alternative route can return an archive URL that answers HTTP 500. See the route probes. Package uploads remain unsupported. Isolated create, in-place content update and rename probes in a received folder succeeded; the writer now routes those requests through the share owner. Shared deletion remains protected. See write route evidence and limits.

• Documents in an app's own folder ("Pages", "Numbers", ...) are mirrored too. Those folders are download only: a file changed or added there is reported and stays local, a file deleted there is fetched again.

• A symlink inside the mirror is never followed. It is reported, and nothing at or below it is read, written or taken for deleted, so a folder that was replaced by a link keeps the pass at incomplete until the link is gone.

• Not handled: changing packages, an ignore list (editor swap files are synced like any file).

Logs

sync writes log lines to stderr; stdout carries only the report of each pass (--json: one JSON object per pass), so both can be read apart. Under systemd every line has its journal priority, so journalctl -p filters by level.

python3 -m icloud_drive_linux sync ~/icloud/Notes --watch --json           # INFO, the default
python3 -m icloud_drive_linux --log-level debug sync ~/icloud/Notes        # the option goes before "sync"
ICLOUD_DRIVE_LOG_LEVEL=debug python3 -m icloud_drive_linux sync ~/icloud/Notes
journalctl --user -u NAME.service -f                                       # follow a service
journalctl --user -u NAME.service -p warning                               # waits, stalls, failures only

For a service, the level is best set in a drop-in (systemctl --user edit NAME.service: [Service], Environment=ICLOUD_DRIVE_LOG_LEVEL=debug); it applies from the next start. Levels: debug, info, warning, error.

What INFO says, and what it does not:

• One line at the start (version, both folders, upload on or off), one at the end of every pass that did something, with the counts of the report.
• A comparison of complete listings (first sync, --full, every --full-interval) says when it starts and ends and, every 30 seconds in between, how many folders are listed so far, how many entries were seen and how many folders are queued. How many folders there are in all is not known before the last one is listed, so there is no percentage. Up to 20 folders are listed by one request, so the count moves in steps. The index (sync-status) is written when that scan has ended; until then it shows the state before it.
• Downloads say how many files and how many listed bytes are due, then every 30 seconds how many files are fetched, how many failed and how many bytes arrived. "Received" counts every byte that arrived, also for a file that is still on its way or whose attempt failed, and a package archive does not have its listed size: the two byte counts are not a ratio. If no file finished and no byte arrived since the last report, the line is a warning that says so.
• Passes with nothing to do are not logged one by one. Every 15 minutes one line, "running, nothing to do", gives the state and how many such passes there were: the program is alive, and nothing changed.
• A failed pass of --watch is one error line with the phase it failed in. A wait of 10 seconds or more that the server asked for is a warning.

DEBUG adds the start and duration of every phase, every pass, every retry and its wait, one line per downloaded file (item id, bytes, seconds), and the text of every problem and event of the report.

Neither level is given a password, a cookie, a session or change token, a request or answer body, or file content, and every URL in a line is cut down to its host. INFO names no file or folder other than the two the mirror was started with. DEBUG names the entries a problem is about (as sync-status does) and identifies everything else by the service's item id. An error message can name the entry or folder it is about.

login needs the real Apple ID password and a second-factor code. An app-specific password does not work for iCloud Drive (measured, see the protocol note). The password is used for one SRP exchange and is neither sent nor stored. The session lives in ~/.local/state/icloud-drive-linux/ (directory 0700, files 0600); choose another place with --state-dir or ICLOUD_DRIVE_STATE_DIR.

Sign-in requests are never repeated automatically. A wrong password ends the command after one attempt; a mistyped code may be entered once more. If a run stops after the password was accepted, the next login continues at the code (or after it) without asking for the password; login --restart starts over.

On an account without trusted Apple devices, Apple texts a new code every time login reaches the code step. Only the newest text is valid.

Tests

PYTHONPATH=src python3 -m unittest discover -s tests -t .

To exercise the writer and the mirror against a real account, inside /Hakalya-Sora-Tests/<run-id> only (they create, change and trash generated files there, and move the run folder to the trash at the end):

PYTHONPATH=src python3 tools/live_write_test.py --state-dir ~/.local/state/icloud-drive-linux
PYTHONPATH=src python3 tools/live_mirror_test.py --state-dir ~/.local/state/icloud-drive-linux --work /tmp/icdl-live

The mirror tests compare rg on a mirror with rg on a plain copy of the same files and are skipped for that part when ripgrep is not installed.

The unit tests need no network and no account. They run the client against a fake that keeps the server half of SRP, and the HTTP transport against a local server on 127.0.0.1.

Documents

• docs/PROTOCOL.md: endpoints, sign-in lifecycle, sync semantics, what is verified and what is not.
• docs/SAFETY.md: the safety contract, its tests, and the limits.
• docs/PROVENANCE.md: where the knowledge and the code come from, licences.