Skip to content

Command Reference#

Global Flags#

  • -c, --config: configuration file path
  • --session: explicit session identifier
  • --owner: owner label override (default: OKDEV_OWNER or local USER)
  • -n, --namespace: namespace override
  • --context: kubeconfig context override. When omitted, spec.kubeContext is used if set; otherwise kubeconfig current-context is used.
  • --output text|json: output format (list, status)
  • --verbose: debug logging
  • NO_COLOR=1 or TERM=dumb: disable ANSI color/styling in interactive terminal output

Exit Codes#

Commands that resolve a session distinguish failure classes so scripts and agents can react without launching a diagnostic chain on every blip:

  • 0: success.
  • non-zero from a remote command (e.g. okdev exec -- false → 1, exit 7 → 7): the remote process's own status, preserved verbatim on single-pod runs. A multi-pod fanout where the command exited non-zero on some pods exits 1 (there is no single status to preserve).
  • 74: the cluster was reachable but the session has no pods — it is genuinely gone.
  • 78: a transient cluster-contact failure (API timeout, connection refused/reset, server overload). okdev already retries once with backoff before surfacing this; on 78 the caller should retry rather than treat the session as dead.
  • Sustained failures escalate the message, not the code: okdev tracks consecutive transient failures per session (any successful contact clears the streak). From the 3rd consecutive failure spanning ≥1 minute, the exit-78 error appends a machine-readable marker line okdev-transient-streak: count=<n> since=<RFC3339> plus guidance that this looks like a sustained network/cluster outage — the caller's cue to stop retrying and alert a human. The exit code stays 78 deliberately: the "78 = retryable" contract is load-bearing for existing retry loops, so the streak marker is the escalation channel. There is no built-in --retry: process-level retry belongs to the caller's loop (until okdev ...; do sleep 30; done), which can now watch stderr for the streak marker as its stop condition.
  • 69: a fanout okdev exec could not run the command on at least one pod (unreachable, timeout, container gone) — a delivery failure, as opposed to the command itself exiting non-zero, which stays exit 1.
  • 1: any other error (configuration, permissions/RBAC, a failed job result, a fanout command that exited non-zero on some pods, etc.).

Commands#

  • okdev version
  • okdev init [--template <name>|<path>|<url>] [--workload-name <name>] [--dev-image <image>] [--set key=value] [--stignore-preset default|python|node|go|rust] [--force]
  • okdev template list [--all]
  • okdev template show <name>
  • okdev validate
  • okdev up [--wait-timeout 10m] [--dry-run] [--wait-hooks] [--workload <name>] [--yes]
  • okdev down [session] [--delete-pvc] [--dry-run] [--wait] [--wait-timeout 2m] [--output json]
  • okdev restart [session] [--pod <name>] [--yes] [--wait-timeout 10m]
  • okdev status [session] [--all] [--all-users] [--details]
  • okdev list [--all-namespaces] [--all-users]
  • okdev use <session>
  • okdev workload list
  • okdev workload use <name>
  • okdev workload show [name]
  • okdev target show
  • okdev target set [--pod <name> | --role <role>]
  • okdev agent list
  • okdev exec [session] [--shell /bin/bash] [--no-tty] [--pod <name> | --role <role> | --label <k=v>] [--exclude <pod>] [--container <name>] [--detach] [--timeout <duration>] [--log-dir <path>] [--no-prefix] [--json] [--require-all] [--gateway <pod>] [--fanout N] [--pkill <pattern> [--signal <sig>]] [--require-sync] [-- command...]
  • okdev jobs list [session] [--job-id <id>] [--container <name>] [--fanout N]
  • okdev jobs logs <job-id> [session] [-f|--follow] [--tail N] [--since <dur|time>] [--pod <name> | --role <role> | --label <k=v>] [--exclude <pod>] [--container <name>] [--fanout N]
  • okdev jobs stop <job-id> [session] [--pod <name> | --role <role> | --label <k=v>] [--exclude <pod>] [--container <name>] [--fanout N]
  • okdev jobs wait <job-id> [session] [--pod <name> | --role <role> | --label <k=v>] [--exclude <pod>] [--container <name>] [--fanout N]
  • okdev exec-jobs [session] [--job-id <id>] [--container <name>] [--fanout N]
  • okdev cp [session] <src> <dst> [--all | --pod <name> | --role <role> | --label <k=v>] [--exclude <pod>] [--container <name>] [--fanout N]
  • okdev logs [session] [--container <name> | --all] [--tail N] [--since 5m] [--follow] [--previous]
  • okdev ssh [session] [--setup-key] [--user root] [--cmd "..."] [--no-tmux] [--forward-agent|--no-forward-agent]
  • okdev ports
  • okdev port-forward [session] <local:remote>... [--address <addr>[,<addr>...]] [--pod <name> | --role <role>] [--ready-only]
  • okdev sync [--mode up|down|bi] [--foreground] [--reset] [--dry-run]
  • okdev sync wait [session] [--timeout 10m]
  • okdev migrate [--template <name>] [--set key=value] [--dry-run] [--yes]
  • okdev upgrade

okdev status [session] [--all] [--all-users] [--details]#

  • Shows session status for the current or selected session.
  • When session is provided, okdev can resolve the saved config from session metadata even outside the repo.
  • --details: prints a single-session diagnostic view with target selection, pod list, pod IP and node placement, mount persistence, sync path semantics, managed SSH state, key local paths, and target pod details.
  • Abnormally terminated containers are surfaced per pod, e.g. container pytorch: OOMKilled (exit 137, finished 2026-07-05T06:32:11Z) — both the current state (container stayed down) and the last termination before a restart (suffixed before last restart). The pod reason column also falls back to the termination reason when no waiting reason applies.
  • Each pod row reports the image its containers are actually running, e.g. images: dev=ghcr.io/org/dev:cuda12 (sha256:1a2b3c4d5e6f), okdev-sidecar=okdev-sidecar:v0.8.0, with the digest shown whenever the runtime reports one. The image comes from the pod status, not the spec, so a mutable tag that moved under a running pod shows the image actually in use. Whether that image matches the one your tests run in is not okdev's business; making it visible is, so a mismatch is noticeable before it turns into a missing dependency inside the pod. Two pods showing different digests for the same tag is the signal.
  • The per-pod table carries an ALIAS column with each pod's short name — the vocabulary --pod, --group, okdev target set --pod and the in-pod /etc/hosts entries all speak. status --details reports it as alias= on the pod line and as alias in --output json.
  • When lifecycle hooks are configured, each pod row reports their progress, e.g. hooks: postSync done, postCreate stale (container restarted; okdev up re-runs it). States: pending (not run yet — e.g. a pod the controller recreated with no okdev up since), running, done, failed, and stale (the hook completed on a previous container instance; an in-place restart wiped its effects and the next okdev up re-runs it). Both hooks run on every session pod, so both are reported on every pod. Scripts should gate launches on done here instead of pod readiness — Ready says the container is up, not that the environment is installed.
  • The sync section always states how the workspace reaches each pod, and why. Either the hub-and-spoke topology with its hub and receivers, or the reason no mesh runs: workspace: shared volume on 2 pod(s), no mesh needed, workspace: single pod, nothing to distribute, workspace: 1 pod(s) run no sidecar and receive nothing. It used to print nothing when there was no mesh, which made "no mesh needed" and "mesh never ran" indistinguishable — start here when a pod does not have your code.
  • Detailed JSON output includes per-pod mount metadata, a hooks array (name, state, at), a containerIssues array (container, reason, exitCode, finishedAt, current), and a pathSemantics section describing which configured sync paths are shared via the workspace sync and which are expected to survive a session restart.
  • --details is only valid when exactly one session is selected.
  • Every status refresh that sees live pods caches a per-session snapshot locally. When the session later disappears without an okdev down (TTL reclaim, quota eviction, admin delete), okdev status no longer stops at No matching sessions found: it prints the last known pod states (phase, reason, container terminations) plus any cluster events that outlived the objects (Killing, Evicted, Preempted, quota errors — events expire ~1h after emission). With --output json the same post-mortem is emitted as a structured object ({"session", "found": false, "lastSeenAt", "workload", "pods", "events"}) so monitoring loops can decide between "reclaimed, safe to recreate" and "evicted for resources, recreating won't help" without scraping text. okdev up and okdev down clear the snapshot — an intentional lifecycle transition is not a death cause.
  • The post-mortem above only answers while the session is still gone. Once it has been recreated, the same question is answered by a one-line report instead: whenever the cached snapshot describes a different run than the live one, okdev up prints previous run <id> ended: <reason> at the end of the recreate, and okdev status keeps printing it for as long as that run is the live one (--details under Previous Run:, --output json as previousRunEnd). The reason is classified — idle-reclaim, evicted-or-preempted, oom-killed, container-failed, deleted, or unknown — with the class-specific mitigation on the second line, because an idle/TTL reclaim and a preemption call for opposite responses (keep the workload busy vs. make the run resumable and chunk it). Which command to use: status on a session that is gone gives the full last-known-state post-mortem; up/status on a session that came back gives this one-liner. Operator-requested recreations (okdev down && up, okdev restart) report nothing, because they clear the snapshot; reused, re-applied and --reconciled workloads keep their run-id and so are not run ends at all.

okdev workload list / use / show / add#

  • Scope: what the current session runs, as opposed to which session commands target. The two are one keystroke apart and are not interchangeable:
    • okdev use <session> — switch which session commands target. Sessions are independent: separate pods, sync channel, ports, SSH alias. Use it to move between parallel environments.
    • okdev workload use <name> — switch what the current session runs. The session name, sync channel, ports and SSH alias all stay; only the underlying workload is replaced, and the old one is deleted.
    • okdev down + edit the config + okdev up — the manual equivalent, and the right move when the change is to shared spec fields (ports, sync, sidecar, volumes) rather than the workload block.

  • Canonical sequence for adding a shape and switching to it:

    $ okdev init --template pytorchjob --workload-name train   # scaffold + declare
$ $EDITOR .okdev/train.yaml                                # fill in image, resources
$ okdev workload use train                                 # pin it
$ okdev up                                                 # apply (confirms the delete)

  • workload list shows PINNED and LIVE as separate columns because they diverge between workload use and okdev up. PINNED is what the next up will create; LIVE is what is running now. Seeing PINNED without LIVE means the switch has not been applied yet.

  • workload use never touches the cluster — it records the choice and reports what the next okdev up will destroy. The destructive step is okdev up, which prompts before deleting a running workload; --yes skips the prompt, and a non-interactive run without --yes fails rather than guessing.
  • okdev up --workload <name> selects and pins in one step. It writes the pin because the session then is running that workload — a stale pin would send the next okdev ssh to a workload whose pods no longer exist.
  • Switching is always explicit. Editing the config, a git pull, or a plain okdev up never replaces a running workload; only workload use and up --workload do. spec.defaultWorkload only decides where a new session starts, so changing it never moves an existing one.
  • Declaring a workload is okdev init's job, not this group's — see okdev init below. This group only inspects and switches.
  • Misuse guard: wanting two shapes alive at once is not what this does — that is two sessions (okdev up --session train-8gpu), not two profiles in one session.

okdev target show#

  • Prints the pinned interactive target for the current session and shows the session pod set.
  • The selected row is marked with *.

okdev target set [--pod <name> | --role <role>]#

  • Explicitly repins the interactive target used by ssh, exec, ports, and sync.
  • --pod selects one concrete session pod.
  • --role selects the highest-priority eligible pod with the matching okdev.io/workload-role.
  • When attachable pods are defined, repinning is restricted to those pods.

okdev agent list#

  • Shows configured coding agents, whether their CLI binary is installed, and whether auth is staged in the current session container.
  • okdev up performs best-effort install checks for configured agents, bootstraps a modern Node/npm runtime via nvm when supported, and then installs missing CLIs.
  • okdev up also stages local auth files for dedicated sessions when a configured local auth file exists.
  • Codex uses ~/.codex/auth.json by default, but spec.agents[].auth.localPath can point at a different local auth file.
  • okdev down removes staged agent auth symlinks/runtime files when it can still reach the target container.
  • okdev does not own agent process launch; users run codex, claude, gemini, opencode, or similar CLIs manually after connecting.

okdev exec [session] [-- command...]#

  • Opens a shell or runs a command in one or more session pods.
  • Without -- command..., opens an interactive shell on the pinned target pod.
  • With -- command... and no pod selector, runs the command on the target pod only — like kubectl exec, and the safe default for state-mutating commands (a forgotten flag can no longer write on every pod). Fanout is opt-in via --all, --workers, --pod, --role, --label, --group, or --exclude. Whenever the session has more pods than the run targets, a one-line stderr notice carries the denominator — notice: running on 1 of 3 session pod(s): target pod master-0 only; use --all, --workers, --role, or --pod to fan out — for scripts and agents as well as terminals, because a partial deploy on a multi-node job produces no error, only divergence later. Single-pod sessions print nothing. Breaking change: earlier versions fanned out to all pods by default — add --all to commands that relied on that.
  • Multi-pod output is prefixed by short pod name on a TTY. When stdout is piped or redirected (scripts, CI, agents), the prefix is auto-suppressed so captured output is clean; pass --no-prefix=false to force it back on.
  • --all targets all session pods.
  • --workers targets only worker-role pods.
  • --pod accepts the full pod name, the short name in the ALIAS column of okdev status / okdev target (e.g. master-0, worker-1), or any unique -<name> suffix — no need to carry the full hash name across pod recreations. Unknown or ambiguous names error out (listing the available names) instead of silently matching nothing. --exclude accepts the same aliases (unknown exclusions are tolerated). The same aliases work for okdev cp --pod, okdev jobs ... --pod, and okdev target set --pod.
  • --group <pod-a,pod-b> targets an explicit pod group. Repeat --group to define multiple groups, and use either full pod names or the short names shown in okdev status. A pod may appear in at most one group.
  • Declared groups run one after another by default.
  • --sequential runs selected pods one-by-one when no groups are declared (output and --log-dir layout are unchanged), or makes the default group-after-group ordering explicit when groups are declared.
  • --parallel runs declared groups in parallel. --fanout caps total concurrent pod executions across all groups.
  • --script <file> uploads a local script file and runs it across the targeted pods. In v1 this is local-file-only; --script - is not supported.
  • With --script, everything after -- is passed to the script as positional arguments ($1, $2, … / "$@"): okdev exec --script ./bench.sh -- --steps 100 --warmup 10 runs bench.sh on each pod with $1=--steps $2=100 $3=--warmup $4=10. Arguments containing spaces are preserved.
  • Failure framing for -- command... fanout: when every failing pod merely reported a non-zero exit code from the command itself, the summary is headed COMMAND EXITED NON-ZERO: (followed by an explicit clarifier that the command was delivered and ran — the exit codes are the command's own results) and okdev exits with the remote status on a single-pod run (exit 3 → 3) or 1 on a multi-pod run — either way okdev exec -- cmd && next still short-circuits && chains. When any pod could not run the command at all (unreachable, timeout, container gone), the summary is headed FAILED: and okdev exits 69, letting scripts tell "my command failed" apart from "delivery failed" without parsing output.
  • Cleanup/kill commands: a bare pkill/pgrep exits 1 when nothing matched — the desired outcome when the processes are already gone, but it reads as a failure in an && chain (okdev prints a signpost hint for this case). Pick one of:
  • okdev exec --pkill '<pattern>' — the first-class path: 0 matched / 1 none, never matches okdev's own exec machinery (no bracket trick needed);
  • okdev exec --all -- sh -c 'pkill -9 -f <pattern> || true' — when the raw command shape is needed;
  • okdev exec --all --json -- pkill -f <pattern> — exit codes become per-pod data, okdev stays exit 0.
  • --kill-group-on-exit (with --detach on multiple pods): the moment any member exits — crash or clean finish — its wrapper broadcasts a SIGKILL of the whole cross-pod job group to every peer over interpod SSH, so surviving ranks cannot hang in rendezvous/NCCL holding GPU memory and starving the next round. Opt-in per launch; requires spec.ssh.interPod: true. Peer kills verify the process group still belongs to the job (same PGID-reuse guard as jobs stop) and leave a <job>.cascaded marker beside the job metadata for forensics. Boundary: if a member's whole container dies (cgroup OOM kill), its wrapper dies with it and cannot broadcast — jobs list shows the group as degraded(...) for exactly this case, and okdev exec --reset-gpu remains the manual cleanup.
  • okdev jobs list group state now distinguishes degraded(N running, M failed of T) — members died while others still run, the stranded-rank shape — from plain running(N/T) (clean partial completion).
  • --reset-gpu: the between-rounds cleanup primitive — kills every process holding GPU compute in the targeted pods (SIGKILL by process group, so hung rank trees go down whole) and then polls nvidia-smi for up to 30s until compute-apps is empty. Exit 0 = GPUs verified clear (including "already clear" — reset is idempotent), 3 = no nvidia-smi in the container, 4 = still busy after the window (leftovers listed). PIDs nvidia-smi reports that are not visible in the container's PID namespace are called out explicitly — they cannot be killed from inside. Replaces the hand-rolled pkill -9 -f "[W]orker..." && nvidia-smi --query-compute-apps loop; okdev's own exec machinery is excluded by construction. Cannot be combined with a command, --script, --shell, --detach, or --pkill. Related but distinct from cross-pod cascade-on-exit (#179): --reset-gpu is an explicit manual reset.
  • The -- command... path is non-interactive fanout execution, not a terminal session. TTY-dependent programs such as watch, top, htop, or full-screen TUIs are not expected to work there; use okdev ssh or okdev exec without -- command... to open an interactive shell first.
  • -- command... preserves raw argv to the remote process. If you choose bash -lc '...', that remote shell layer is still yours.
  • --script avoids the quoting/stitching problem for non-trivial shell pipelines by uploading the file and running it remotely instead of embedding the body in a one-liner.
  • --pod: target specific pods by name (repeatable or comma-separated).
  • --role: target pods by okdev.io/workload-role label (case-insensitive).
  • --label: target pods by arbitrary label key=value (repeatable, AND logic).
  • --exclude: exclude specific pods from the selected set (repeatable or comma-separated). Cannot be used with --pod.
  • --group is mutually exclusive with --all, --workers, --pod, --role, --label, and --exclude.
  • --parallel and --sequential are mutually exclusive.
  • --container: override which container to exec into (default: session target container). Works in both modes.
  • --script executes the uploaded file directly when it has a shebang; otherwise it falls back to sh <file> ....
  • --shell cannot be combined with --script.
  • --detach: launches the command in the background and returns immediately. Prints per-pod launch details including job id, pid, log path, and metadata path. The command runs in its own process group (via setsid) so okdev jobs stop can terminate the entire process tree, not just the leader.
  • When --log-dir is used with multiple declared groups, logs are written into per-group subdirectories under the requested path. --log-dir has no effect with --detach (detached jobs log inside the pod).
  • Detached exec preserves argv to the remote process; it is not flattened back into a single shell string. Detached --script uploads a per-pod temp file, launches it, and removes that temp file after completion.
  • Detached jobs store combined stdout/stderr and JSON metadata under /var/okdev/exec/ on the shared okdev-runtime volume, so logs and metadata survive a target-container crash (e.g. OOMKill) and remain readable through the sidecar. If that directory cannot be created or written by the container user (e.g. a user-supplied root-owned volume with a non-root container), the launcher falls back to the legacy /tmp/okdev-exec/ with a notice on stderr — same lifetime semantics as before. For large outputs (training logs, profiles, dumps), prefer --log-dir on the caller or redirect inside your command to a session volume; the runtime volume is a node-local emptyDir and heavy writers can fill it.
  • The pid returned from --detach is the pid of your command itself (okdev re-parents the launcher so $! is your program's pid after execve). kill <pid> or okdev exec --pod <pod> -- kill <pid> therefore targets your command, not a wrapper shell.
  • When using --detach, pass the command you want okdev to launch directly. Do not add an extra nohup ... & inside the command string.
  • --timeout: per-pod command timeout (e.g., 30s, 5m). Pods exceeding the timeout are cancelled and reported as failed.
  • --require-sync: refuse to run unless background sync is healthy and every mapping has fully converged (an inline okdev sync wait). Applies to both foreground commands and --detach — running pre-edit code does not care whether the process is detached, and a foreground experiment that does so returns a plausible-looking result instead of crashing. Without the flag, a run on a dead, paused or unhealthy channel still proceeds but prints a stderr warning naming what is at risk (the command / the detached job). Requires the session to have sync mappings, and a command to gate: okdev exec --require-sync with no command after -- is rejected, since an interactive shell has nothing to gate.
  • --pkill <pattern>: kill processes whose full cmdline matches the extended regex, on the selected pods. Unlike a raw pkill -f, it can never match okdev's own exec machinery (the helper excludes itself and its whole ancestor chain), so no bracket trick (pkill -f 'patter[n]') is needed. Prints one killed <pid> <cmdline> line per signalled process and follows the pkill exit convention: 0 when at least one process matched, 1 otherwise. --signal <name|number> (default TERM) selects the signal. Cannot be combined with a command after --, --script, --shell, or --detach. For detached jobs prefer okdev jobs stop, which terminates the whole process group.
  • --log-dir: write per-pod output to <dir>/<short-name>.log. Streaming to stdout still happens.
  • --no-prefix: suppress the pod name prefix in output. Auto-enabled when stdout is not a terminal (pipe/redirect/CI); pass --no-prefix=false to keep the prefix in that case.
  • --fanout N: maximum concurrent pod executions (default 16).
  • --pod, --role, and --label are mutually exclusive.
  • --json: capture each pod's result into a JSON envelope instead of streaming. Output is always a JSON array with one envelope per pod in selection order — a single resolved pod still emits a length-1 array, so callers never branch on object-vs-array by pod count. Each envelope is {"pod": "<name>", "exit": N, "stdout": "...", "stderr": "...", "status": "..."}, with an "error" field added when the command could not be run or completed (in which case exit is -1).
  • status makes per-pod delivery explicit: responded (the command ran and reported an exit code — even a non-zero one), unreachable (the pod could not be reached; the command did not run), timeout, error (outcome unknown), or missing (no result came back for the pod). Use it to tell "responded with empty output" apart from "never responded".
  • --require-all (requires --json): exit non-zero after emitting the JSON document unless every targeted pod's status is responded, listing the missing pods. A responded pod with a non-zero exit code does not fail --require-all — its exit code is data.
  • With spec.exec.fanoutMode: auto|gateway and interpod SSH enabled, multi-pod --json runs route through an in-cluster gateway pod: one apiserver exec, fanout over the pod network (see spec.exec in the config manifest). --gateway <pod> picks the gateway explicitly (and enables the gateway path even below the auto threshold). If the gateway driver is unavailable (older sidecar image), okdev prints a notice on stderr and falls back to direct per-pod exec.
  • The remote command's own exit code is reported as data in exit, so a non-zero remote exit (e.g. pgrep returning 1 for no match) does not make okdev exit non-zero. okdev exits 0 whenever the JSON document is produced; inspect each envelope's exit/error to react. Pre-flight failures (session not found, cluster unreachable) still use stderr and the dedicated exit codes (74/78) rather than JSON.
  • --json requires a command (after -- or via --script) and cannot be combined with --detach, --group, --parallel, --sequential, or --log-dir. The [<pod>] prefix is never applied in JSON mode.

okdev jobs list [session]#

  • Lists detached okdev exec --detach jobs across the session's running pods.
  • Reads job metadata from /var/okdev/exec/*.json in the target container (plus the legacy /tmp/okdev-exec/ location for jobs launched by older okdev versions).
  • If the target container is gone (e.g. OOMKilled), listing and jobs logs automatically retry through the okdev-sidecar container, which mounts the same runtime volume.
  • When a command fails because the container is gone, the error carries the termination cause when it can be determined, e.g. container "pytorch" terminated: OOMKilled (exit 137, finished 2026-07-05T06:32:11Z). The same hint is appended to okdev exec FAILED lines and single-pod exec errors.
  • Text output is grouped by logical job id, with one row per detached launch showing job id, summarized state, pod count, earliest start time, and original command. When any job is still running, a footer points at the blocking primitive: hint: `okdev jobs wait <id>` blocks until it finishes; add --grep PATTERN to return when the log matches.
  • On a terminal the COMMAND column is truncated (with …) to the remaining terminal width so long training commands don't wrap rows; piped/redirected output and --json always carry the full command.
  • JSON output includes both the logical job summary and the per-pod podStates records (with pgid and groupLive — the count of live processes still in the job's process group — when available).
  • State values: running (wrapper alive and user command in flight), exited (user command finished; exit code is recorded in podStates / JSON), and orphaned (metadata still says running but the pid has exited or been recycled - typically the wrapper was SIGKILLed or the container was restarted before the completion metadata could be written). Grouped text summaries render forms such as running(1/2), exited(2/2), and failed(1/2).
  • When any pod fails to list its jobs, the command still prints the jobs it was able to collect from the healthy pods and reports the failures in a FAILED: footer (or an errors array in --json output); exit status is non-zero so scripts can detect partial failures.
  • With spec.exec.fanoutMode: auto|gateway and interpod SSH enabled, the per-pod listing routes through the in-cluster fanout gateway (one apiserver exec instead of one per pod). The listing is read-only, so any gateway problem silently falls back to direct per-pod queries.
  • --pod: target specific pods by name (repeatable or comma-separated).
  • --role: target pods by okdev.io/workload-role label (case-insensitive).
  • --label: target pods by arbitrary label key=value (repeatable, AND logic).
  • --exclude: exclude specific pods from the selected set (repeatable or comma-separated). Cannot be used with --pod.
  • --container: override which container to inspect (default: session target container).
  • --job-id: filter to a specific detached exec job id.
  • --fanout N: maximum concurrent pod queries (default 16).
  • --ready-only: inspect only pods that are already running.

okdev jobs logs <job-id> [session]#

  • Streams the detached job's combined stdout/stderr aggregated across all pods in the logical job.
  • Each line is prefixed by the pod short name so multi-pod output stays attributable.
  • --pod: stream logs from specific pods by name (repeatable or comma-separated).
  • --role: stream logs from pods with the matching workload role.
  • --label: stream logs from pods matching label selectors.
  • --exclude: exclude specific pods from the selected set. Cannot be used with --pod.
  • -f / --follow: keep following until every pod in the job reaches a terminal state.
  • --tail N: show only the last N lines of each pod's log (-1 = all, the default). Combines with --follow (tail -n N -f).
  • --since <dur|RFC3339>: skip pods whose log file has not changed since the cutoff (e.g. --since 90s in a poll loop transfers nothing from idle pods). Job logs carry no per-line timestamps, so this is a file-level activity gate — when a file has changed, the (tail-limited) current content is shown, not just the lines after the cutoff. Cannot be combined with --follow.
  • --grep <regex>: keep only lines matching the extended regex, applied pod-side before --tail — --grep 'reward=' --tail 3 returns the latest 3 matching lines and transfers nothing else. Cannot be combined with --follow.
  • --dedup: fold runs of consecutive identical lines into one copy plus a [repeated Nx] count — a multi-rank crash printing the same exception per worker collapses to a single copy. Applied after --grep, before --tail. Cannot be combined with --follow.
  • Snapshot reads normalize carriage returns: tqdm-style progress bars that rewrite one line with \r come back as separate lines, and CRLF endings are stripped — no more single-line blobs defeating --tail. (--follow streams the raw file.)
  • Reads are size-verified: the reader reports the exact byte count of the selection and okdev retries when the received stream is truncated or empty-by-drop, so jobs logs never silently presents a dropped exec stream as the job's output.
  • If the job's container is gone (e.g. OOMKilled), logs are read through the okdev-sidecar container instead — job logs live on the shared runtime volume and survive the container.
  • If some pod logs are unavailable, okdev still streams the logs it can read and reports the missing pods in a FAILED: footer before returning non-zero.

okdev jobs wait <job-id> [session] [--grep <regex>] [--tail N]#

  • Polls the detached job until all pod-local records are terminal.
  • Returns success only when every pod exits cleanly.
  • Returns non-zero when any pod exits non-zero, becomes orphaned, cannot be queried, or the job id does not exist.
  • --grep <regex>: return as soon as a matching line appears in any pod's log instead of waiting for the job to finish — the block-until-log-pattern primitive for monitoring loops (--grep 'step 2|Error' returns when training reaches step 2 or the first error shows up). The matching line(s) are printed pod-prefixed; --tail N bounds how many are shown (default 1, -1 = all matches). The pattern is checked before the terminal state, so a job that already exited (even non-zero) with the pattern in its log still satisfies the wait. If the job ends without ever matching, jobs wait fails with a distinct error.
  • jobs wait vs hand-rolled polling: prefer okdev jobs wait <id> over sleep/jobs list loops for "block until done", and jobs wait --grep over sleep/jobs logs --grep loops for "block until the log says X" — one blocking call, no per-iteration token cost. Use a jobs logs --tail --since loop only when you need the ongoing output itself, not just a completion/progress signal.

okdev jobs stop <job-id> [session]#

  • Stops every still-running pod in the logical job by sending SIGTERM, waiting 10 seconds, then sending SIGKILL to survivors.
  • Scope: a fanned-out --detach shares one job id across every pod it launched on, so the default stops the job on all of them. --pod, --role, --label and --exclude narrow it to some ranks and leave the others running — the same selector set as okdev jobs list/logs. Which to use: no selector to end the whole distributed run, --pod worker-1 to kill one misbehaving rank. Note the default here is deliberately not the target-pod-only default that okdev exec uses: a job id identifies one logical job, and stopping half a distributed run by accident is the worse failure. Say the narrow intent explicitly.
  • Signals the job's whole process group when available: detached jobs are launched via setsid, so children the command forked (e.g. torchrun workers) are signaled too — including stragglers whose leader already exited. Stop only reports success once no live group member remains, so a clean return means the process tree (and its GPU memory) is gone.
  • Group membership is verified via the job's OKDEV_JOB_ID environment marker before signaling, so recycled pids/pgids are never signaled by mistake. Jobs launched by older okdev versions (or in containers without setsid) fall back to leader-only signaling.
  • Prints which pods were signaled (pgid=N for group signals, pid=N for the fallback) during the stop flow.
  • Returns non-zero when any pod could not be queried or signaled, or if processes are still alive after the stop attempt.

okdev jobs wait <job-id> [session]#

  • Waits for the job to finish (or, with --grep, for a log pattern to appear) across the selected pods. Accepts the same --pod/--role/--label/--exclude selectors as stop, so a launcher can block on one rank instead of the whole group. Read-only.

okdev exec-jobs [session]#

  • Compatibility alias for okdev jobs list [session].
  • Uses the same grouped logical-job output and flags as okdev jobs list.

okdev cp [session] <src> <dst>#

  • Copies files or directories between the local machine and session pods.
  • Prefix the remote path with : (e.g., :/workspace/data). The other argument is a local path.
  • Single-pod mode (default): copies to/from the pinned target pod.
  • Multi-pod upload (--all, --pod, --role, --label): fans out the same local source to all matched pods in parallel.
  • Multi-pod download: downloads from each matched pod into <dest>/<short-pod-name>/ subdirectories.
  • Files are streamed via cat pipes. Directories are tar-streamed automatically.
  • Single-file uploads are size-verified and atomic: the pod receives into a temp file next to the destination, the byte count is checked against the local size, and the file is renamed into place — readers never observe a partial file, and a dropped exec stream is retried (up to 3 attempts) instead of silently leaving a missing or truncated file. Directory uploads likewise retry until the pod acknowledges a completed extraction.
  • Single-file downloads resume from an undersized <dest> or a saved <dest>.okdev-part from a previous attempt.
  • If a previous attempt finished streaming but failed before the final rename, rerunning okdev cp promotes the completed <dest>.okdev-part without redownloading.
  • --verify verifies single-file download SHA-256 after copy. It is not supported for uploads or directory downloads.
  • With --verify, resumed downloads hash the existing local bytes once and continue hashing the streamed remainder, so success does not require a second full local reread.
  • --verify currently requires python3 or python in the target container to emit the remote SHA-256 during the download stream.
  • Directory and multi-pod downloads do not yet support resume.
  • On a TTY, an in-place progress line shows transferred bytes, transfer rate, and elapsed time (after a few seconds). For multi-pod copies the line aggregates across pods (e.g. Copying to 8 pods · 3/8 done · 5 in flight · 1.2 GiB · 28.0 MiB/s · 00:42) and surfaces a noticeably slow pod inline. Progress is suppressed on non-TTY writers (pipes, redirects, CI), so machine-readable output is unaffected.
  • --all: target all running pods in the session.
  • --pod: target specific pods by name (repeatable or comma-separated).
  • --role: target pods by okdev.io/workload-role label (case-insensitive).
  • --label: target pods by arbitrary label key=value (repeatable, AND logic).
  • --exclude: exclude specific pods from the selected set (repeatable or comma-separated). Cannot be used with --pod.
  • --container: override which container to copy to/from (default: session target container).
  • --fanout N: maximum concurrent pod transfers (default 16).
  • --verify: verify single-file download SHA-256 after copy.
  • --all, --pod, --role, and --label are mutually exclusive.

okdev init [--template <name>|<path>|<url>] [--workload-name <name>] [--set key=value] [--stignore-preset default|python|node|go|rust] [--force]#

  • Writes a starter config at .okdev/okdev.yaml plus the workload's manifest beside it — .okdev/pod.yaml for the default pod workload, .okdev/job.yaml for --template job, and so on. Every workload type, pod included, is a manifest file.

  • On a project that already has a config, okdev init adds a workload to it rather than refusing. That is the additive mode, and it requires --workload-name:

    $ okdev init --template pytorchjob --workload-name train
  Wrote .okdev/train.yaml
  Declared workload "train" in .okdev/okdev.yaml

    --template selects the shape being added, and works with a template of your own as readily as a built-in. Only its workload block is taken; the template's project-level content is rendered and discarded, because adding a workload changes nothing project-wide.

    The manifest is named after the workload, not its type, so two workloads of the same type never collide. A pod workload added this way gets the same starter manifest okdev init scaffolds, so it is a file to edit rather than a blank.

    Additive mode never prompts for project-level settings and never changes them. Passing a project-level flag (--name, --namespace, --context, --dev-image, --sidecar-image, --sync-local, --sync-remote, --ssh-user, --shell, --stignore-preset) is refused rather than ignored, so one flag never means two things; edit the config to change those. --force still means "rewrite the whole config", and giving it together with --workload-name is refused because they state opposite intents.

    The template's variables are a different matter, because adding a workload instantiates a template just as creating a config does. --set works here for the same reason, and on a terminal you are prompted for each declared variable with its default offered as a hint — identical to a fresh okdev init. --set wins over a prompt, so scripted and interactive use compose.

    $ okdev init --template pytorchjob --workload-name train
  ? Number of worker replicas (workerReplicas): (1) 4
  Wrote .okdev/train.yaml

    With --yes it never prompts: --set first, then the frontmatter defaults, and a variable with no default and no --set is an error naming the variable. Without a terminal and without --yes there is nobody to answer, so it refuses rather than silently taking defaults — the same refusal a fresh init gives.

    $ okdev init --template pytorchjob --workload-name train --set workerReplicas=4
  Wrote .okdev/train.yaml
  Declared workload "train" in .okdev/okdev.yaml

    A --set key the template does not declare is reported as a warning and dropped, matching fresh init. The resolved values render the added workload's manifest but are not persisted to spec.template.vars: that records how the project's config was generated, and adding a workload does not regenerate it.

    Nothing is written unless the resulting config validates: a rejected addition leaves the config and .okdev/ byte-identical. An existing file at the target manifest path is an error naming the file — --force does not clobber manifests. - Configs written before okdev always used the folder — a flat .okdev.yaml, or a bare okdev.yaml — keep working and are never migrated. Workloads added to them still land in .okdev/. - --workload and --generic-preset are removed. The shape to scaffold is a template now, so --template selects it. Passing the old flags is an error naming the replacement rather than a silent no-op — asking for a Job and getting a pod is a failure you would only notice on the cluster.

    was now
    --workload job --template job
    --workload pytorchjob --template pytorchjob
    --workload generic --generic-preset deployment --template deployment
    --workload generic --manifest-path X --inject-path Y --template generic --manifest-path X --inject-path Y
    (omitted) (omitted — basic, which is the pod template)
    - --dev-image: sets the dev container image in the scaffolded .okdev/pod.yaml. The generated manifest gives the dev container equal CPU/memory requests and limits, as the config does for the sidecar, so the starter pod is eligible for Kubernetes Guaranteed QoS.
    - --template: accepts a project template from .okdev/templates/<name>.yaml.tmpl, a user template from ~/.okdev/templates/<name>.yaml.tmpl, built-in basic, a file path, or a URL. Run okdev template list to see available names.
    - --set: sets a frontmatter-declared template variable. Repeat it for multiple variables, for example --set numWorkers=4 --set baseImage=pytorch:latest.
    - Templates can declare string, int, and bool variables in YAML frontmatter. Resolved values are available as .Vars.<name> during rendering and are persisted under spec.template.vars.
    - .Vars holds exactly the declared variables, so referencing a .Vars.<name> the frontmatter never declared is an error naming that variable. Go's default would render it as the literal <no value>, which wrote a broken config for a string variable and failed much later with invalid type for comparison for one used in a numeric comparison. The usual cause is a typo, or frontmatter that was not parsed at all — it is only recognized when the file starts with ---, so a leading blank line silently turns the whole block into body text. okdev template show <name> lists the variables okdev actually parsed.
    - Templates can also declare companion files in frontmatter. Each file has a rendered path and a template path resolved relative to the selected template, which lets a PyTorch template render both okdev.yaml and a matching pytorchjob.yaml.
    - For built-in templates, it also writes a starter local .stignore file for the initialized sync root. An existing .stignore is never replaced, not even with --force — it accumulates hand-written rules, and --force regenerates the config, not your ignore list. okdev init reports Kept existing <path> so it is clear the starter patterns were not applied.
    - The built-in templates are pod (aliased as basic, the default), job, pytorchjob, deployment and generic. Each declares its own starter manifest and .stignore preset in frontmatter — exactly as your own templates do — so a template of yours that shadows one of those names simply replaces it and inherits none of its behavior. generic declares no manifest: it is the bring-your-own shape, and needs --manifest-path plus --inject-path.
    - Generated manifestPath values are bare filenames (job.yaml). okdev resolves a manifest from the config's own directory, then the project root, then .okdev/ — so the same value works whether the config is .okdev/okdev.yaml or a flat .okdev.yaml beside .okdev/.
    - --stignore-preset: override the starter .stignore patterns with a project-oriented preset.
    - When --stignore-preset is omitted, okdev init tries to detect a preset from common repo markers like go.mod, package.json, Cargo.toml, and pyproject.toml.

okdev template list [--all]#

  • Lists templates from project, user, and built-in sources in resolution order.
  • Project templates in .okdev/templates/ shadow user templates, and user templates shadow built-ins.
  • --all includes shadowed lower-priority entries.

okdev template show <name>#

  • Prints a template's resolved source, description, declared companion files, and declared variables.
  • Description and variables come from optional YAML frontmatter at the top of the template.

okdev migrate [--template <name>] [--set key=value] [--dry-run] [--yes]#

  • Without --template, migrates older config schema fields to the current schema.
  • Moves spec.volumes into the workload manifests. Volumes belong beside the containers that mount them; the config held the sources while the manifest held the mounts, and a name declared in both went to the manifest — so a workspace PVC in the config silently became whatever the manifest said. Every workload's manifest gets a copy, a name the manifest already declares is left alone and named in a warning, and a manifest okdev did not write is backed up before being edited.
  • Gives every pod workload a manifest. spec.podTemplate is removed, so a config still carrying one is extracted into .okdev/pod.yaml and the workload is pointed at it. A config that never declared one ran on a container okdev injected by default — that gets written out as a real manifest too, with a warning, since it is a file you should review rather than a hidden default. An existing file at the target path is an error naming it, never an overwrite; move it aside and re-run.
  • After that extraction the next okdev up reports workload drift and offers to recreate. The Pod it builds is unchanged — only the config's description of it moved, and the snapshot it is compared against now hashes the manifest instead of the inline block.
  • With --template, re-renders the selected template, overlays existing config values so local edits win, writes the merged config, and regenerates any template-declared companion files. Unlike the config, companion files are fully overwritten from the re-rendered template; local edits to those files will be lost (the .bak backup is your recovery path).
  • Existing spec.template.vars seed template variables during migration; --set overrides stored values.
  • Unless --no-backup is set, okdev migrate --template writes .bak backups for the config and any rewritten companion files before overwriting them.
  • --dry-run prints the merged config and lists the companion files that would be rewritten, without writing. --yes skips confirmation when writing.

okdev up [--wait-timeout 10m] [--dry-run] [--wait-hooks]#

  • Reconciles Pod/PVC resources, updates SSH config, initializes managed forwarding/sync, then exits.
  • If the session workload already exists, okdev up reuses it and only reruns setup.
  • Workload resources are named per run, for example okdev-<session>-<run-id>. Every recreate is a new run: okdev down && okdev up, okdev restart, okdev up --reconcile and a drift-triggered recreate all mint a fresh run-id and workload name, so a new object never steps back into the name of the one just deleted — some control planes reject or misbehave on rapid name reuse, and reusing it makes the dead run and its replacement indistinguishable in discovery labels, events and logs. Reuse and re-apply keep the run-id: a workload that was patched in place (a controller-managed Deployment under --reconcile) is the same run, because nothing was destroyed.
  • When the run being replaced ended on its own (reclaim, eviction, OOM, external delete), the successful up ends with previous run <id> ended: <reason> plus the mitigation for that class — see the okdev status section for the full contract. A successful up also caches a snapshot of the run it just started, so the next end can be explained even if okdev status is never run.
  • If a Job or PyTorchJob created or recreated by this okdev up reaches a failed pod state before readiness, okdev stops waiting, deletes that failed workload, and clears local session state. Reused existing workloads are not auto-deleted.
  • tmux-backed persistent interactive shells are enabled by default.
  • --tmux: explicitly enable tmux mode in the dev container.
  • --no-tmux: disable tmux mode for this pod.
  • When sync.engine=syncthing, okdev up refreshes the session's local Syncthing processes, starts background sync in bidirectional mode by default, and waits for the initial sync to converge before exiting.
  • Lifecycle hooks run synchronously inside okdev up and are tracked per pod: okdev up && launch is already hook-safe for pods that existed when the hook fanout started. A hook whose done-marker predates the container's current start (an in-place restart wiped its effects) reads as stale and is re-run by the next okdev up.
  • --wait-hooks: keep converging postSync until every session pod — including pods the controller created after the initial fanout (operator recreations, late-scheduled workers) — has completed it. Use this in automation that launches work across all pods right after up; without it, a pod that appeared mid-up completes its hooks only on the next okdev up.
  • On multi-pod sessions, okdev up writes a managed block into every pod's /etc/hosts mapping the short-name aliases (master-0, worker-1, …) to current pod IPs — launch scripts can hardcode MASTER_ADDR=master-0 once and survive recreations. The block is rewritten on every up/restart (including restart --pod). A pod the controller recreated on its own (OOM kill, eviction, node drain, operator restart) no longer leaves the map wrong until the next okdev up: okdev records what it wrote, and okdev exec compares that record against the live pods and rewrites the block before running anything when they disagree — the recreated pod keeps its name but gets a new UID and IP, so the comparison is on UID, not name. okdev status reports the same drift but never repairs it, because a read command must not mutate the cluster. Boundary: only the aliases self-heal. Lifecycle hooks and interpod SSH keys on a controller-recreated pod still need an okdev up, so after an external recreation that remains the complete repair. Best-effort: pods whose image cannot write /etc/hosts warn instead of failing, and a write that lands nowhere is now caught by a readback rather than being reported as success. A partial write names the pods that did not get the block — short names do not resolve inside those.
  • The ready card's next: list is extended with the primitives that session is most likely to need — at most four lines, and only the ones that apply to its shape: okdev sync wait / exec --require-sync when the session has sync mappings, spec.lifecycle.postCreate / postSync when no lifecycle hook is configured (or up --wait-hooks when one is and the session is multi-pod), and fanout targeting plus the in-pod short-name aliases when there is more than one pod. Not TTY-gated: scripts and agents are the callers most likely to hand-roll a substitute for a primitive that already exists.
  • spec.ports is materialized as SSH LocalForward or RemoteForward based on direction.

okdev down [session] [--delete-pvc] [--dry-run] [--wait] [--wait-timeout 2m] [--output json]#

  • Deletes the current session workload and cleans up local SSH/sync metadata.
  • When session is provided, okdev can resolve the saved config from session metadata even outside the repo.
  • Prompts for confirmation by default; use --yes in scripts or non-interactive environments.
  • --dry-run: previews what would be deleted without removing cluster or local state.
  • --wait: waits for the workload object to disappear and then for any remaining session pods to terminate before returning.
  • --wait-timeout: caps the total time spent waiting for workload/pod termination when --wait is enabled.
  • --output json: emits a machine-readable summary of the planned or completed deletion and local cleanup steps.
  • --delete-pvc remains accepted for compatibility but is ignored; okdev no longer manages PVC lifecycle automatically.

okdev env-diff [session] [--pod <name>]#

  • Shows packages installed by hand since the session came up: diffs the pod's current inventory (dpkg/apk + pip) against a baseline okdev up captures right after lifecycle hooks finish. Added/changed/removed packages are listed, and additions come with a ready-to-paste spec.lifecycle.postCreate draft — the persistence path for changes that would otherwise vanish silently on the next pod recreation (the worst case being a missing runtime lib causing a silent perf regression, e.g. NCCL falling back to TCP).
  • The baseline lives on the pod's runtime volume: it survives in-place container restarts and dies with the pod — the same lifetime as the environment it describes. okdev up refreshes it whenever hooks actually ran (hook installs belong in the baseline) and otherwise only writes it where missing, so a resumed up cannot swallow manual drift.
  • Defaults to the target pod (where interactive debugging happens); --pod selects others.
  • Relatedly, okdev exec prints a one-line reminder when the command itself is an install (apt-get install, pip install, conda install, apk add, …): the change lands in the container overlay and needs a lifecycle hook to survive.

okdev restart [session] [--pod <name>] [--yes] [--wait-timeout 10m]#

  • One-command recovery when a container died and the restart policy will not bring it back: deletes the session workload, waits for termination, resets local per-session sync state, and runs the full up flow against the current config.
  • PVCs and other resources the manifest references are untouched; auto-provisioned sync volumes are recreated.
  • Lifecycle hooks (postCreate/postSync) re-run automatically on the fresh pods — configure them for setup that must survive recreation (installed tools, editable installs).
  • Pod names change on recreation; use the short-name aliases (master-0, worker-1) with --pod so scripts survive restarts.
  • Prompts for confirmation; --yes for scripts. --wait-timeout caps both the deletion wait and pod readiness.

Full restart vs --pod — use --pod when one pod of a multi-pod session is broken (e.g. a worker OOMKilled on a bad node); use the full restart when the workload spec changed or the whole session is wedged. A full restart of a 10-pod session pays ~full-recreate time, re-runs hooks everywhere, invalidates per-pod caches, and kills every in-flight detached job; --pod pays that cost only on the named pods.

  • --pod <name> (repeatable, short names ok): deletes only the named pods and lets the workload controller recreate them in place. Other pods keep running — their caches and detached jobs are untouched. Lifecycle hooks are tracked per pod, so postSync/postCreate re-run only on the fresh pods; sync re-bootstraps only if the sync hub pod itself was recreated.
  • Requires a controller-backed workload (job, pytorchjob, generic); for a single-pod pod workload use the full okdev restart.
  • PyTorchJob replicas must have restartPolicy: OnFailure (the scaffold default) — with Never, the training-operator marks the whole job Failed instead of recreating a deleted member pod, so restart --pod refuses up front and explains the fix (okdev up --reconcile after changing the policy).
  • Canonical dead-worker sequence: okdev status (spot the broken pod) → okdev restart --pod worker-3 → relaunch the affected rank/job.
  • --wait-hooks: same as on okdev up — after the restart, keep converging postSync until every session pod (including controller-created late arrivals) has completed it.

okdev ssh [session] [--setup-key] [--user root] [--cmd "..."] [--no-tmux] [--forward-agent|--no-forward-agent]#

  • Targets okdev-sshd in the resolved interactive container.
  • The default interactive container name is dev; override it with spec.workload.attach.container when your workload uses a different main container name.
  • When session is provided, okdev can resolve the saved config from session metadata even outside the repo.
  • Maintains managed host alias in ~/.ssh/config as okdev-<session>.
  • --no-tmux: bypass tmux for this SSH session when tmux mode is enabled.
  • --forward-agent: forwards the local SSH_AUTH_SOCK into the live SSH session so Git/SSH inside the workload can use your local agent.
  • --no-forward-agent: disables forwarding for this SSH session even when spec.ssh.forwardAgent: true.
  • Agent forwarding only applies to the live okdev ssh connection; keys are not copied into the workload.
  • Requires a local SSH agent with identities already loaded, for example eval "$(ssh-agent -s)" and ssh-add ~/.ssh/id_ed25519.

okdev logs [session] [--container <name> | --all] [--tail N] [--since 5m] [--follow] [--previous]#

  • Streams logs from the resolved target pod for the current session.
  • When session is provided, okdev can resolve the saved config from session metadata even outside the repo.
  • Defaults to the pinned target container when --container is omitted.
  • The default target container name is dev unless spec.workload.attach.container selects a different interactive container.
  • --all: streams all regular containers in the target pod and prefixes each line with [container].
  • --follow: follows logs by default; use --follow=false for a bounded dump.
  • --tail and --since mirror the usual Kubernetes log filters.
  • --previous: reads the previous instance logs for the selected container(s).

okdev ports#

  • Advanced/recovery command. Rebuilds managed SSH LocalForward / RemoteForward state from spec.ports after disconnects or local port changes.
  • No-op when managed forwards are already healthy and config is unchanged.
  • --dry-run: previews the SSH alias and port-forward actions without updating SSH config or starting/stopping managed forwards.

okdev port-forward [session] <local:remote>...#

  • Runs direct foreground Kubernetes port-forwarding to one selected session pod.
  • Uses the current session target pod by default.
  • --address: local listen addresses (default: localhost). Accepts comma-separated values or repeated flags, for example --address localhost,0.0.0.0.
  • --pod selects one explicit pod by name.
  • --role selects one pod by workload role. Ambiguity is an error.
  • --ready-only: restricts selection to already-running pods.
  • Only LOCAL:REMOTE mappings are supported (kubectl-style :REMOTE or bare PORT are rejected).
  • The command stays attached until interrupted.

okdev sync [--mode up|down|bi] [--foreground] [--reset] [--dry-run]#

  • Advanced command. Starts detached background sync by default; use --foreground for sync debugging, or explicit one-way sync (up/down).
  • Scope: okdev sync manages the channel (start/repair the transport). A successful return does not mean pending changes have finished propagating — to guarantee your latest edits are on the pod, use okdev sync wait (the data question). The two are complementary, not interchangeable, and okdev sync now says so in its own success output rather than leaving the distinction to the docs.
  • By default each configured mapping syncs in its own direction (spec.sync.paths[].direction, falling back to bi); an explicit --mode forces that mode onto every mapping for the invocation. See the config manifest for choosing persistent directions — down makes the pod the authority so local writes can never clobber pod-generated results.
  • With multiple mappings, each becomes its own syncthing folder; the first (primary) mapping is the one shared to mesh receivers, and sync reset-remote clears only the primary remote.
  • A mapping's local root may nest inside the primary root: okdev maintains a managed block in the primary root's .stignore excluding it from the primary folder (written before the sync daemon starts). Removing a mapping retains the entry as a tombstone so the subtree never silently joins the primary folder; sync start prints a notice and status --details lists active/retained excludes.
  • Without an explicit --mode, no-op when background sync is already active for the session.
  • Liveness is judged by health, not just process existence: a sync process that is alive but unhealthy (e.g. peer disconnected after a network drop) is never reported as "already running" — okdev sync repairs it in place (reset local state + restart), so a plain okdev sync recovers a dead or stale channel without --reset. okdev sync and okdev sync wait share this health check and can no longer contradict each other about the same channel.
  • --background: explicitly request detached background mode.
  • --reset: check local-to-hub sync and mesh receiver health, then reset only what is broken. Skips the local sync teardown when the primary sync is already healthy. For sessions with mesh receivers, probes each receiver and re-runs mesh setup only when broken or disconnected receivers are found.
  • --reset --force / --reset -f: unconditionally reset without health checks.
  • --reset --local: scope reset to local-to-hub sync only (skip mesh).
  • --reset --mesh: scope reset to mesh receivers only (skip local sync).
  • --reset --force --local / --reset --force --mesh: force reset a specific component.
  • okdev init writes the starter config and, for built-in templates, a starter local .stignore file for the initialized sync root.

okdev sync status [session] [--top 5] [--output json]#

  • Scope: the inventory question — what is still moving, and why is it taking so long? Complementary to okdev sync (the channel) and okdev sync wait (has it finished): use status when a transfer is slow or stuck and you need the cause, wait when you just need it to be done. Canonical sequence when sync will not converge: okdev sync status → add the offending path to .stignore → okdev sync → okdev sync wait.
  • Reports per mapping: pending bytes and file count, the largest actually pending files with each one's share of the transfer and its direction (local->remote / remote->local), and the exclude patterns in force — noting whether they come from a .stignore file or from okdev's built-in defaults when no file exists.
  • The pending set comes from syncthing itself (/rest/db/need), not from a scan of local file sizes: an already-synced file never appears, and a file pending from the pod does. This is the difference from the large-file warning printed during okdev up, which is a local size heuristic — that warning now names its top entries inline and points here for the authoritative answer.
  • --top N changes how many pending files are named per mapping (default 5). One file at a 90% share is the common case, and it is a one-line .stignore fix; repeatedly running okdev sync --reset against it never converges, because the cause is still there.

okdev sync wait [session] [--timeout 10m]#

  • Scope: the data question — has everything propagated? — complementary to okdev sync, which manages the channel. Blocks until every configured sync mapping has zero pending bytes in both directions, then returns — the edit-run loop guarantee: vim train.py && okdev sync wait && okdev exec -- python train.py.

okdev sync pause / okdev sync resume#

  • Freezes the sync channel in both directions so a local high-risk operation — git checkout, a rebase, anything that swaps working-tree files — cannot propagate onto a pod under a running job. Canonical sequence: okdev sync pause → switch branches / rebase → okdev sync resume → okdev sync wait before the next run.
  • A pause is an explicit state, not a fault: okdev sync wait fails fast pointing at resume (pending changes cannot converge by design), plain okdev sync refuses to override it, detached launches warn (and --require-sync refuses), and okdev status --details reports health: paused. Only okdev sync resume — or the next okdev up, which rebuilds sync from config as part of the intentional session lifecycle — ends the pause.
  • Implemented as the syncthing folder-level paused flag on the local daemon (the hub), so nothing is torn down: resume picks up deltas without a fresh rescan cycle.
  • Triggers an immediate rescan on both sides before waiting, so files written moments earlier are picked up now instead of after the filesystem-watcher delay.
  • Purely a wait: it does not start or repair sync. It fails fast with a state-accurate report: "not running" when the background process is gone (start it with okdev sync), or "running but unhealthy (…)" when the process is alive but the channel is broken (repair with okdev sync, which self-heals, or okdev sync --reset).
  • Prints pending-byte progress while waiting; exits non-zero if convergence is not reached within --timeout.

okdev upgrade#

  • Checks the latest GitHub release and upgrades the okdev binary in place.
  • Downloads the correct archive for the current OS/architecture, verifies the SHA256 checksum, and atomically replaces the running binary.
  • No-op when already on the latest version.
  • After okdev up completes successfully, a non-blocking version check runs (cached for 24 hours) and prints a reminder to stderr if a newer version is available.