Skip to content

salvus.flow.ng.telemetry

salvus.flow.ng.telemetry salvus flow ng telemetry

Run telemetry for in-process runs: a TelemetrySnapshot captured at solve_forward_through_sessions’s progress_callback chunk boundaries, a TelemetryCollector holding the latest snapshot plus a bounded in-memory history, and sinks that publish it. FileSink writes <output_folder>/progress.json — a strict superset of command-line solver Salvus’ own format (status/last_written/tasks keep their exact meaning), so salvus.flow.executors.salvus_job.SalvusJob. update_status() parses a solver run’s progress through the exact same code path a remote command-line solver job’s status already goes through.

T2 increment of the design in notes/ng-run-telemetry-design.md (control-project repo): adds the live, fileless ZmqSink — a local REP-socket server thread a separate Python process can query (snapshot/history/cancel/ping) via the one handshake file it writes, <output_folder>/telemetry_endpoint.json. NGInProcessJob. telemetry()/telemetry_stream()/cancel() (in in_process_job.py) use it too, though a same-process caller reads the TelemetryCollector it already holds directly rather than round-tripping through the socket.

Threading model: this module is written to be correct under BOTH the GIL (Python 3.11) and free-threaded Python (3.14t, no GIL) — nothing here relies on the GIL for thread-safety. TelemetryCollector guards its latest-snapshot slot and its bounded history with one explicit threading.Lock; every reader (latest, history) copies the reference/tuple out while holding that lock rather than assuming a plain attribute read or deque append is atomic. The cancel flag is a threading.Event, not a bare bool. This is now exercised by more than one thread for real: ZmqSink runs its request loop on its own daemon thread, reading collector.latest/collector.history and setting collector.cancel_requested concurrently with the solve loop’s own thread writing them through the same lock-guarded paths. On the native side, the session bindings release the GIL for the duration of advance/run_segment (see session_binding.hpp) so this thread can actually run while a chunk is in flight; the session’s own steps_completed counter is NOT read from here or anywhere else mid-advance (it is a plain, non-atomic C++ counter) — this module’s snapshots are built entirely from the Python-side progress_callback arguments and psutil, never from native session state read cross-thread.

Functions

current_process_resources()

def current_process_resources(peak_rss_bytes: int) -> TelemetryResources: ...

Read the current process’ resource usage via psutil.

Parameters
  • peak_rss_bytes int — The running peak RSS the caller has tracked so far — folded into the returned peak with the freshly-read current RSS, since this function has no memory of its own across calls.
Returns TelemetryResources

live_timer_snapshot()

def live_timer_snapshot(timer: typing.Any) -> dict[str, typing.Any] | None: ...

Serialize a Timer tree that may still be running, tolerating unstopped nodes — unlike in_process_job._serialize_timer, which requires the whole tree to already be stopped (Timer.duration raises ValueError on a running timer with no accumulated durations, per its own docstring).

A still-running node’s duration is estimated as elapsed wall time so far (time.perf_counter_ns() - t._start_time_ns); a stopped node uses its own real duration unchanged. Never raises into the solve loop: if reading a node’s timing fails for any reason, that node’s duration/percentage are omitted rather than propagating the failure (a telemetry snapshot is best-effort, and must never be able to take the actual solve down with it).

Parameters
  • timer typing.Any — The root Timer, running or stopped.
Returns dict[str, typing.Any] | None — The same recursive shape _serialize_timer produces, or None if even the root’s own timing could not be read.

make_progress_callback()

def make_progress_callback(
    collector: TelemetryCollector,
    sink: TelemetrySink | None,
    timer: typing.Any,
    estimator: RunningDurationEstimator,
    job_id: str,
    backend: str,
    dtype: str,
    solution_order: int | None,
    num_ranks: int,
    num_threads: int,
    start_time_wall: float,
    start_time_str: str,
    dt: float,
) -> typing.Callable[[int, int], None]: ...

Build a solve_forward_through_sessions-shaped progress_callback that builds a fresh TelemetrySnapshot on every chunk boundary and feeds it to collector.update() and, if given, sink.publish().

Every snapshot this callback builds reports status="running" — the mid-loop callback has no way to know whether whatever comes after the time-stepping loop returns (output copying, etc.) will itself succeed, so "finished"/"failed" are always the caller’s own explicit final collector.update()/sink.close() call, made after (or instead of) this callback’s own last firing.

Parameters
  • collector TelemetryCollector — Updated with a new snapshot on every call.
  • sink TelemetrySink | None — Published to on every call, if given.
  • timer typing.Any — The run’s root Timer, still running — serialized via live_timer_snapshot. None if no timer exists yet.
  • estimator RunningDurationEstimator — Supplies eta_seconds/steps_per_second; None values are reported for both until it has timed at least one post-warm-up step.
  • job_id str — TelemetryRun.job_id.
  • backend str — TelemetryRun.backend.
  • dtype str — TelemetryRun.dtype.
  • solution_order int | None — TelemetryRun.solution_order.
  • num_ranks int — TelemetryRun.num_ranks.
  • num_threads int — TelemetryRun.num_threads.
  • start_time_wall float — time.time() at run start, for TelemetryProgress.wall_elapsed.
  • start_time_str str — TelemetryRun.start_time.
  • dt float — The time axis’ own step size, for TelemetryProgress. sim_time = step * dt.
Returns typing.Callable[[int, int], None]

Classes

CompositeSink

class CompositeSink(builtins.object):
    def __init__(self, sinks: typing.Sequence[TelemetrySink]) -> None: ...

Fan publish/close out to several sinks (e.g. a FileSink and a ZmqSink together), each independently — one sink’s own exception (already supposed to be impossible per TelemetrySink’s contract, but never trusted blindly here) can never stop the others from being called.

Parameters
  • sinks typing.Sequence[TelemetrySink] — Published to, in order, on every publish/close call.
Methods
close()
def close(self, final_snapshot: TelemetrySnapshot) -> None: ...

Close every sink in sinks with final_snapshot.

Parameters
  • final_snapshot TelemetrySnapshot — The run’s final (terminal-status) snapshot.
Returns None
publish()
def publish(self, snapshot: TelemetrySnapshot) -> None: ...

Publish snapshot to every sink in sinks.

Parameters
  • snapshot TelemetrySnapshot — The snapshot to publish.
Returns None

FileSink

class FileSink(builtins.object):
    def __init__(
        self,
        output_folder: pathlib.Path,
        interval: float = 2.0,
        history_path: pathlib.Path | None = None,
    ) -> None: ...

Writes each published snapshot to <output_folder>/progress.json, at most every interval seconds — except a run’s very first publish and any publish whose snapshot.run.status is terminal (finished/failed/cancelled), both of which always write regardless of interval.

The written JSON is a strict SUPERSET of SalvusCompute’s own progress.json shape: status (one of TelemetryRun.status’s values), last_written (formatted exactly like command-line solver’s own, _LAST_WRITTEN_FORMAT — SalvusJob.update_status() parses this with a fixed arrow format list and does not accept an otherwise-valid ISO-8601 string in the wrong shape), and tasks (one task_type: "progress" task tracking the time-stepping loop, empty while n_steps is not yet known during setup) — so SalvusJob.update_status() and the tqdm-based progress-callback renderer (salvus.flow.utils.progress_callback) both work on an The solver run’s progress.json unchanged. Everything solver-specific lives under one additional top-level telemetry key (TelemetrySnapshot.to_dict()), so it can never collide with or be mistaken for a command-line solver field.

This is a purely local-filesystem sink: NGInProcessJob (the only consumer in T1) reads its own collector directly, in-process, so the write cadence here has no correctness requirement of its own yet. A future remote the solver site (T3) that relies on SalvusJob. update_status()’s own staleness check will need a cadence at least as tight as 2 * site.update_interval_in_seconds (0.5s for a local site, 2.0s for ssh, 60s for slurm, per BaseExecutor.update_interval_in_seconds) or that check discards the snapshot as stale — not a concern this increment’s own consumer hits, since it never goes through that remote-IO path.

Parameters
  • output_folder pathlib.Path — Directory progress.json is written into. Created if missing.
  • interval float = 2.0 — Minimum seconds between two non-terminal publishes.
  • history_path pathlib.Path | None = None — If given, one JSON line per publish is appended here (history.jsonl-shaped) for post-mortem plots. Off (None) by default.
Attributes
progress_json_path pathlib.Path

Where the progress file is written.

Methods
close()
def close(self, final_snapshot: TelemetrySnapshot) -> None: ...

Force-write final_snapshot, bypassing interval.

Parameters
  • final_snapshot TelemetrySnapshot — The run’s final (terminal-status) snapshot.
Returns None
publish()
def publish(self, snapshot: TelemetrySnapshot) -> None: ...

Write snapshot, subject to interval throttling.

Parameters
  • snapshot TelemetrySnapshot — The snapshot to write.
Returns None

TelemetryCollector

class TelemetryCollector(builtins.object):
    def __init__(self, history_size: int = 1000) -> None: ...

Owns the latest TelemetrySnapshot and a bounded in-memory history; updated from the solve loop’s own progress callback.

See this module’s own docstring for the threading model: every access below is under one threading.Lock, correct with or without the GIL.

Parameters
  • history_size int = 1000 — Maximum number of past snapshots kept in memory (oldest dropped first). ~1000 snapshots is a few hundred KB at this dataclass’ size.
Attributes
history tuple[TelemetrySnapshot, ...]

A point-in-time copy of the bounded history, oldest first.

latest TelemetrySnapshot | None

The most recently update()-d snapshot, or None.

Methods
update()
def update(self, snapshot: TelemetrySnapshot) -> None: ...

Swap in a new latest snapshot and append it to the history.

Parameters
  • snapshot TelemetrySnapshot — The new latest snapshot.
Returns None

TelemetryOutput

class TelemetryOutput(builtins.object):
    def __init__(self, count: int, bytes_written: int) -> None: ...

Generic, physics-blind bookkeeping for one output kind.

Parameters
  • count int — Number of items of this kind produced so far (e.g. 1 for a receivers file, however many test masses for Newtonian-noise — whatever the registering caller counts as one item of this kind).
  • bytes_written int — Bytes written to disk for this kind so far, 0 until the corresponding file actually exists.
Methods
to_dict()
def to_dict(self) -> dict[str, int]: ...

JSON-safe representation.

Returns dict[str, int]

TelemetryProgress

class TelemetryProgress(builtins.object):
    def __init__(
        self,
        step: int,
        n_steps: int,
        sim_time: float,
        wall_elapsed: float,
        eta_seconds: float | None,
        steps_per_second: float | None,
    ) -> None: ...

Where the run is in its time-stepping loop.

Parameters
  • step int — Steps completed so far.
  • n_steps int — Total steps this run will take. 0 before the time axis is known (the setup phase).
  • sim_time float — Simulated time elapsed, in seconds.
  • wall_elapsed float — Wall-clock time elapsed since the run started, in seconds.
  • eta_seconds float | None — Predicted wall-clock time to completion, from RunningDurationEstimator. None before any post-warm-up step has been timed.
  • steps_per_second float | None — Recent mean throughput (steps timed so far divided by the wall-clock time RunningDurationEstimator attributes to them). None under the same condition as eta_seconds.
Methods
to_dict()
def to_dict(self) -> dict[str, typing.Any]: ...

JSON-safe representation.

Returns dict[str, typing.Any]

TelemetryResources

class TelemetryResources(builtins.object):
    def __init__(
        self,
        rss_bytes: int,
        peak_rss_bytes: int,
        cpu_percent: float,
        num_threads: int,
    ) -> None: ...

Process resource usage at the moment a snapshot was built.

Parameters
  • rss_bytes int — Current resident set size.
  • peak_rss_bytes int — Largest RSS observed by this collector so far (not a true OS-level high-water mark, which psutil does not expose portably — a running max over every snapshot taken).
  • cpu_percent float — psutil.Process.cpu_percent(), measured since the previous call on the same process handle (0.0 on the very first call in a process, which is a psutil convention, not a real “no CPU used” reading).
  • num_threads int — OS-level thread count of the current process (psutil.Process.num_threads()) — independent of TelemetryRun.num_threads, which is the solver’s own requested execution_policy thread count.
Methods
to_dict()
def to_dict(self) -> dict[str, typing.Any]: ...

JSON-safe representation.

Returns dict[str, typing.Any]

TelemetryRun

class TelemetryRun(builtins.object):
    def __init__(
        self,
        job_id: str,
        backend: str,
        dtype: str,
        solution_order: int | None,
        num_ranks: int,
        num_threads: int,
        start_time: str,
        status: RunStatus,
        error: str | None = None,
    ) -> None: ...

Static-ish facts about the run itself.

Parameters
  • job_id str — Stable identifier for this run, shared with NGInProcessJob.job_name when one exists.
  • backend str — solver.backend ("python", "cpu", or "cuda").
  • dtype str — solver.dtype ("float32" or "float64").
  • solution_order int | None — solver.solution_order, if resolved yet; None during setup.
  • num_ranks int — solver.execution_policy.num_ranks.
  • num_threads int — solver.execution_policy.num_threads_per_rank.
  • start_time str — When the run started, formatted like last_written (_LAST_WRITTEN_FORMAT).
  • status RunStatus — One of setup, running, finished, failed, cancelled.
  • error str | None = None — The failing exception’s type name (e.g. "ValueError"), when status == "failed"; None otherwise.
Methods
to_dict()
def to_dict(self) -> dict[str, typing.Any]: ...

JSON-safe representation.

Returns dict[str, typing.Any]

TelemetrySink

class TelemetrySink(typing.Protocol):
    def __init__(self):
        ...

Something a TelemetrySnapshot can be published to.

publish must never raise into the solve loop — implementations are responsible for catching and swallowing their own IO failures, since a telemetry sink must never be able to take a real solve down with it.

Methods
close()
def close(self, final_snapshot: TelemetrySnapshot) -> None: ...

Publish final_snapshot unconditionally and release resources.

Parameters
  • final_snapshot TelemetrySnapshot — The run’s final (terminal-status) snapshot.
Returns None
publish()
def publish(self, snapshot: TelemetrySnapshot) -> None: ...

Publish one snapshot, subject to the sink’s own cadence.

Parameters
  • snapshot TelemetrySnapshot — The snapshot to publish.
Returns None

TelemetrySnapshot

class TelemetrySnapshot(builtins.object):
    def __init__(
        self,
        progress: TelemetryProgress,
        run: TelemetryRun,
        timings: dict[str, typing.Any] | None,
        resources: TelemetryResources,
        outputs: dict[str, TelemetryOutput],
        source: str = "in_process",
        version: int = 2,
    ) -> None: ...

One immutable point-in-time view of a running (or finished) the solver job.

Parameters
  • progress TelemetryProgress — Time-stepping progress.
  • run TelemetryRun — Static-ish run facts, including status.
  • timings dict[str, typing.Any] | None — The live Timer tree, in the same shape as salvus.flow.ng.in_process_job._serialize_timer produces for the final, stopped tree (label/duration/ duration_unit/percentage/optional children) — see live_timer_snapshot below for how a still-running node’s duration is estimated. None before any timer exists (immediately at setup).
  • resources TelemetryResources — Process resource usage.
  • outputs dict[str, TelemetryOutput] — Per output kind (e.g. "receivers", "newtonian_noise"), keyed by a short lowercase name.
  • source str = 'in_process' — Where this snapshot came from; always "in_process" for T1 (the only transport that exists yet).
  • version int = 2 — TELEMETRY_SNAPSHOT_VERSION at construction time.
Methods
to_dict()
def to_dict(self) -> dict[str, typing.Any]: ...

JSON-safe representation, nested plain dicts throughout.

Returns dict[str, typing.Any]

ZmqSink

class ZmqSink(builtins.object):
    def __init__(
        self,
        collector: TelemetryCollector,
        output_folder: pathlib.Path,
        bind_address: str = "127.0.0.1",
        poll_timeout_ms: int = 200,
    ) -> None: ...

The live, fileless telemetry channel: one daemon thread owning a zmq.REP socket, bound to a random local port, answering snapshot/history/cancel/ping requests — JSON in, JSON out, authenticated by a random per-run token no request can guess.

Unlike FileSink, this sink never actively writes a snapshot anywhere on publish() — the socket answers on demand, reading straight from the TelemetryCollector it was constructed with (already updated by the solve loop’s own collector.update() call, made right alongside sink.publish()). publish() is therefore a no-op; only close() does real work (stopping the thread and tearing down the endpoint file).

The one handshake a separate process needs is <output_folder>/telemetry_endpoint.json ({host, port, token, pid, started_at, schema_version}), written once at bind time and removed at close().

Parameters
  • collector TelemetryCollector — Read directly by every request; never mutated here except cancel_requested.set() for a "cancel" request.
  • output_folder pathlib.Path — Where the endpoint descriptor is written.
  • bind_address str = '127.0.0.1' — The interface to bind to. Left at its default ("127.0.0.1", localhost-only) unless a caller explicitly opts into "0.0.0.0" — this socket has no encryption, only a shared-secret token, so exposing it beyond localhost is a deliberate choice, not this class’ own default.
  • poll_timeout_ms int = 200 — How often the server thread’s zmq.Poller wakes up to check close()’s stop signal even with no request pending — bounds shutdown latency, not a correctness knob.
Attributes
port int

The bound TCP port (useful for tests; not part of the stable API a consumer should rely on — read telemetry_endpoint.json instead).

Methods
close()
def close(self, final_snapshot: TelemetrySnapshot) -> None: ...

Stop the server thread, close the socket, and remove the endpoint file — final_snapshot is unused (it is already the collector’s own latest by the time close() is ever called, matching every other TelemetrySink.close() in this module), kept only to satisfy the shared TelemetrySink protocol.

Parameters
  • final_snapshot TelemetrySnapshot — Unused.
Returns None
publish()
def publish(self, snapshot: TelemetrySnapshot) -> None: ...

No-op: requests are answered on demand from the collector.

Parameters
  • snapshot TelemetrySnapshot — Unused.
Returns None