# Source plugins and collection (https://powercontext.oceanbase.io/en/rfcs/2617-source-plugins/)

- Proposal Name: `source_plugins_and_collection`
- Start Date: 2026-11-12
- RFC PR: [oceanbase/powercontext#2617](https://github.com/oceanbase/powercontext/pull/2617)

# Summary [#summary]

Building on the [plugin interfaces RFC #2549](https://github.com/oceanbase/powercontext/pull/2549), this proposal defines
standalone Source plugins: a plugin provides Source types and, when needed, collects continuously from external systems.
Collection can run on the central Server or on devices such as robots and edge machines, and results are always reported to
the central Server.

- **Source plugins**: a directory with `plugin.py`, which assigns a `SourcePlugin` declaration to `plugin`. A plugin that
  provides only `definitions` extends the Source types; a plugin that declares `collect` reads from external systems per
  collection target.
- **Collection runtimes (runners)**: the processes that run `collect`. The central Server has a built-in runner named
  `server`; a process started on a device with `powercontext source-agent` is also a runner. It registers with the
  central Server before it syncs: the Server assigns its name from a suggestion, or registers the fixed name it asks
  for, and records the principal (type and identifier) that registered it as the owner.
- **Targets live centrally**: targets are stored on the central Server in `pc_source_collect_targets`. Each belongs to a
  Scope and names the runner and the plugin that collect it. Users add, change, and remove them through the Source
  management APIs or the Dashboard without logging in to the device.
- **Pull sync**: runners sync with the central Server on a schedule, fetching every target assigned to them and reporting
  the plugins they loaded with their versions. The central Server never pushes to devices; there is no reverse channel.
- **Deduplication**: a plugin declares which parameters form a target's unique key and how they are normalized, so the
  central Server recognizes duplicates as targets are added.
- **Results**: Sources and checkpoints go through core's existing remote ingestion APIs; each target's latest status is
  reported together with its checkpoint.
- **Versions**: a Source plugin declares its version and its minimum compatible version explicitly. The version loaded on
  the central Server is the specified version; the central Server accepts a device's requests, and hands them to its own
  copy of the plugin, only when the device's plugin version is within the range the central version declares. Data
  upgrades use the `upgrade(ctx)` hook from RFC #2579 and run only on the central Server.
- **Large content**: the framework does not prescribe a transfer method; the same Source plugin can also run on the central
  Server, provide a route that receives files, and read them back in its Projections.

This proposal replaces the plugin contracts and management mechanisms of the
[Source plugin framework RFC #2012](https://powercontext.oceanbase.io/en/llms.mdx/rfcs/2012-source-plugin-system/content.md): the management service, instance registration, reverse
channel, and `source load` command are removed, and `pc_source_write_plugins` is dropped when it holds no data. Section 6
of the reference-level explanation covers upgrading deployments that used the previous framework.

This document describes the target design; it does not mean the mainline or a release already provides it. Interface
names and code examples illustrate the contract; the implementation determines the final syntax.

# Motivation [#motivation]

The old Source plugin framework (#2012) carries a lot of management for plugins: a separate management service with its own
SQLite (instances, capabilities, targets, desired state, commit records), one Runtime process per plugin instance, poll /
reply reverse channels and capability catalog sync between the service and Server, plus deployment commands and instance
start/stop tools. Deployers run several processes and maintain two databases; plugin authors must understand instances,
channels, and the deployment flow.

RFC #2549 changes Artifact plugins to a model where plugins implement interfaces and core calls them when needed; Source
plugins should work the same way. One requirement is supported by the old framework but not by RFC #2549's initial edge
design:

- **Scenarios such as embodied AI configure collection for a fleet of devices centrally.** Devices usually sit behind NAT and
  cannot be reached from outside, but they can reach the central Server. Operators need to decide on a central panel which
  device collects what into which Scope, instead of logging in to each device.
- **Configuring on a panel needs immediate deduplication and parameter validation**: when a target is added, users need to
  know whether it already exists and whether its parameters are valid, not find out after the device runs it.

Goals:

1. A plugin author writes only `plugin.py`, without learning instances, channels, or the deployment flow.
2. Targets are stored centrally; users manage any runner's targets on the central Server or the Dashboard, and adding or
   removing targets restarts nothing.
3. Devices only need to reach the central Server; the central Server never connects to devices.
4. Disconnects, restarts, and resubmission are idempotent and produce no duplicate Sources.
5. Deployers run only Server, plus one collection process per device when collecting on devices.

# Guide-level explanation [#guide-level-explanation]

## 1. Plugin shape [#1-plugin-shape]

Source plugins live in the same plugin directory as Artifact plugins and have the same shape:

```python
plugin = SourcePlugin(version="1.2.0", definitions=(CameraEventDefinition(),), collect=Collect(...), target=..., setup=setup)
```

| Interface | Required | Description |
| --- | --- | --- |
| `version` | Yes | The plugin version, compared under PEP 440 |
| `min_compatible_version` | No | The lowest device version the central Server still accepts while running this version; defaults to `version` |
| `definitions` | Yes | A sequence of `SourceDefinition`, the same as a plugin's own Source types in RFC #2549 |
| `target` | For collection | The Pydantic model of target parameters and the identity expression; see below |
| `setup(ctx) -> SourceCollector` | For collection | The factory, returning an object that implements `collect` |
| `collect(request) -> CollectResult` | For collection | Reads external data per target and checkpoint, returning Sources and new checkpoints |

A plugin that provides only `definitions` extends the Source types: callers submit through the existing Source write API,
and core runs `resolve` and Projections in the central host process, as for a plugin's own Source types in RFC #2549.

## 2. Example: robot camera events [#2-example-robot-camera-events]

```python
from pydantic import BaseModel, Field

from powercontext.plugins import (
    Collect, CollectedItem, CollectRequest, CollectResult, Interval, SourcePlugin, TargetSpec,
)


class CameraTarget(BaseModel):
    camera: str = Field(description="The camera name on the device, such as front")
    min_interval_seconds: int = Field(default=5, ge=1)


class CameraCollector:
    def __init__(self, ctx):
        self.ctx = ctx

    async def collect(self, request: CollectRequest) -> CollectResult:
        result = CollectResult()
        for target in request.targets:
            since = (target.checkpoint or {}).get("since", 0)
            events = await read_local_events(target.parameters["camera"], since)
            items = [CollectedItem(definition="camera-event", value=event) for event in events]
            result.add(target, items, checkpoint={"since": events[-1].sequence if events else since})
        return result


async def setup(ctx):
    return CameraCollector(ctx)


plugin = SourcePlugin(
    version="1.2.0",
    min_compatible_version="1.0.0",
    definitions=(CameraEventDefinition(),),
    collect=Collect(schedule=Interval(seconds=30)),
    target=TargetSpec(
        description="Collect the device camera's events",
        parameters=CameraTarget,
        identity={"camera": ("strip", "lower")},
    ),
    setup=setup,
)
```

The plugin name comes from its directory name (assumed here to be `camera-events`) and is also the name of this collection
capability.

## 3. Collection runtimes [#3-collection-runtimes]

- **Server**: the central Server has a built-in runner named `server`, which runs the collection plugins in its plugin
  directory in Workers on the Supervisor's schedule.
- **Edge**: on a device, run

  ```text
  powercontext source-agent --server https://memory.example.com
  ```

  It loads only the Source plugins in the plugin directory and connects to the central Server with its credential.
- **Runner names**: a name identifies a runner across the whole deployment, and targets, status, and checkpoints refer to
  runners by name. The central Server decides the name, which never changes afterwards; a separate display name, which can
  change and need not be unique, is shown in interfaces. The name `server` is reserved for the central Server's own runner.
- **Registration**: a runner registers with the central Server when it first starts, and only then syncs, in one of two
  ways:
  - **Suggested name** (the default): the runner generates a random registration ID and registers with it and a suggested
    name (`--suggest-name`, the host name by default). When nobody has the suggested name, the central Server uses it;
    otherwise it assigns another one (such as `robot-7-2`) and returns the decided name. The runner keeps the registration
    ID and name in a local state file (`--state-dir`, in the user data directory by default) and reuses them after a
    restart; it exits with an error when the state directory is not writable. The same credential subject registering
    again with the same registration ID gets the same name, so retries after a network failure are safe; another subject
    holding the registration ID does not get the name. A suggested name never takes over an existing name.
  - **Fixed name**: `--name robot-7` sets the name explicitly, for containers and CI whose state directory does not
    persist. A name that does not exist is registered; one that exists and belongs to the same credential subject (same
    type and ID) reuses its record; one that belongs to another subject fails the start. This way reads and writes no
    state file, so it also runs on a read-only root file system.
  - One credential can register several runners, but each device should have a credential of its own. Any authenticated
    subject can register, with no limit on the number; registering only adds the runner to the list, and submitting data
    to a Scope still needs `scope.contribute` on that Scope.
- **Sync**: a runner calls the central sync API with its decided name on a schedule (every 30 seconds by default); the
  central Server accepts only names that are registered and belong to the current credential subject, and a sync never
  creates a runner record. A sync does three things:
  1. report the plugins it loaded, each plugin's version and load error, and when it was last seen;
  2. fetch every target assigned to it, across all Scopes;
  3. compare them with the targets it is running, start new ones, and stop removed or disabled ones.
- A runner needs no Scope list: the targets assigned to it carry their Scopes. After a new Scope is created and a target is
  assigned, collection starts at the next sync.

## 4. Managing targets [#4-managing-targets]

Callers use the Source management APIs (HTTP and MCP):

```text
pc_source_runners()
pc_source_add(scope_id="robot-7-data", runner="robot-7", capability="camera-events",
              parameters={"camera": "front"})
pc_source_status(scope_id="robot-7-data")
pc_source_target_set_enabled(scope_id="robot-7-data", target_id="...", enabled=false)
pc_source_target_delete(scope_id="robot-7-data", target_id="...")
```

- **Choosing a runner and capability**: `pc_source_runners` lists every registered runner with its display name, the
  plugins and versions it loaded, whether those versions are compatible with the central Server, and when it was last
  seen. Targets can be assigned only to registered runners (`server` always exists); creating a target for a runner that
  does not exist is refused. A capability's description,
  parameter format, and identity expression come from the plugin loaded on the central Server. The Dashboard builds its
  form from them: choose a Scope, then a runner, then a capability, then fill in the parameters; a capability has the same
  form on every runner.
- **Validation and deduplication**: the central Server validates `parameters` against the format of the plugin version it
  loaded and computes the target key from the identity expression. When a target with the same key already exists for the
  same Scope, runner, and capability, the existing target is returned; the same key with different parameters is a
  conflict.
- **Status**: each target's latest attempt time, latest success time, consecutive failures, and latest error are reported
  by the runner together with its checkpoint. A `last_seen_at` later than the target's `updated_at` means the runner has
  fetched the latest configuration.
- **Configuration version**: each target has a configuration version that increases when its parameters change, when it
  is enabled or disabled, when it is transferred, and when its checkpoint is cleared; a sync delivers it to the runner. A
  runner submits results with the configuration version it collected under, and the central Server refuses results
  collected under an older configuration (see section 5).
- **Changing**: fields of `parameters` outside the identity expression can change, and targets can be enabled or
  disabled; fields in the identity expression cannot change. To collect another object, add another target, so a
  checkpoint always belongs to the same external object.
- **Disabling and deleting**: a disabled target leaves the sync results while its checkpoint is kept, and resuming
  continues from where it stopped; deleting also deletes its checkpoint, while collected Sources are kept.
- **Transferring**: a target can be reassigned to another registered runner. The target ID stays the same and the
  checkpoint follows the target by default, for example when a reinstalled device registers under a new name; when the
  new device's data differs from the old one's, the checkpoint can be cleared at the same time. A target with the same key
  already on the destination runner is a conflict.
- **Permissions**:
  - A device runner collects with the device's credential, network position, and local files, so its owner decides
    what it collects: creating a target, changing its parameters, and enabling it again need, besides Source write
    permission on the Scope, a caller that is the principal that registered the runner or has `server.admin`. Otherwise
    a user with write permission on some Scope could make another user's device read files or reach addresses with
    parameters of their choosing. Disabling and deleting a target only make a runner collect less and need Source write
    permission on the Scope. A device can submit results only to Scopes where its credential has `scope.contribute`.
  - Adding, changing, and removing targets of the `server` runner needs `server.admin`: the server runner reaches external
    systems with credentials from the Server environment, so deciding what it collects is an operator decision that
    cannot be left to a user with write permission on some Scope. The server runner submits results inside the Server and
    needs no further permission.
  - Transferring a target is creating it on the destination runner, with the same checks: Source write permission on the
    Scope, a registered destination runner, a caller that registered the destination runner or has `server.admin`, and
    `server.admin` when the destination is `server`; when the source runner is `server`, `server.admin` is needed as for
    changing a `server` target. A user with only Scope write permission therefore cannot create a target on their own
    device and then transfer it to `server` or to another user's device.
  - Listing runners needs only authentication; a runner can only sync and report its own targets; changing a display
    name needs `server.admin`.
  - Deleting a runner record: the subject that registered a runner can delete its own records, and `server.admin` can
    delete any record. Deleting a runner that still has targets is a conflict until they are transferred or deleted.

## 5. Submitting results and recovering from disconnects [#5-submitting-results-and-recovering-from-disconnects]

A runner collects, per Scope, the enabled targets assigned to it:

- On the Server, core saves the returned Sources and each target's new checkpoint in one transaction.
- On a device, the runner first submits Sources through `POST /v1/source-observations`, then submits the new checkpoint
  through `POST /v1/connector-checkpoints/commit` with the old checkpoint as the expected value, together with the target's
  status and the configuration version it collected under. If anything fails midway, the checkpoint does not advance and
  the next call collects again from the old checkpoint; resubmitted Sources have the same identity and content, and the
  central Server deduplicates them.
- **Targets are validated at commit**: the checkpoint and the target status are written in one transaction. In that
  transaction the central Server first confirms that the target still exists, is enabled, belongs to the submitting
  runner, and has the configuration version given in the request; if any of these fails, the whole commit is refused and
  neither the checkpoint nor the status changes. Commits by the server runner pass the same validation. After a target is
  deleted, disabled, transferred, has its checkpoint cleared, or has its parameters changed, results collected under the
  old configuration therefore cannot advance the checkpoint, and a late commit cannot recreate the checkpoint of a
  deleted target. A runner whose commit is refused discards that result and collects again under the latest
  configuration after its next sync.
- **Validation does not depend on request fields**: the central Server looks up by `(scope_id, target_id)` whether a
  checkpoint commit belongs to a collection target, whichever optional fields the request carries. A collection target's
  checkpoint can be committed only by the runner the target belongs to; other callers are refused even with write
  permission on the Scope. Checkpoints of collection targets use the reserved connector version `target`; a commit that
  names it, or carries a configuration version, for a target that no longer exists is refused as well, and is not
  stored as an ordinary connector checkpoint.
- **Sources already submitted**: Sources are submitted by a separate request before the checkpoint. When the checkpoint
  commit is refused, those Sources are already saved and are kept: they are real observations of the external object,
  their identity comes from the plugin, and the central Server deduplicates them when the same object is collected again
  under the new configuration.
- Every device request carries the plugin version; the central Server checks it as described in section 8 and hands the
  request to the same plugin loaded on the central Server.

## 6. Large content [#6-large-content]

How images, audio, and other large content reach the central Server is up to the plugin; the framework supports three
approaches:

| Approach | How | Suits |
| --- | --- | --- |
| Inline | Encode it into the Source content, bounded by the Server's request body limit | Thumbnails, short audio |
| The plugin's object storage | The runner uploads to object storage the plugin names; the Source records the URI and digest | Deployments that already have object storage |
| Through the central plugin | The same Source plugin also loads on the central Server, receives files through RFC #2579 plugin routes, and stores them in its own storage; the Source records a reference | Devices that should not reach object storage directly |

In the third approach one plugin package has two roles: on the device it collects and uploads; on the central Server it
receives files and runs `resolve` and Projections. The plugin on the device calls its own central routes through
`ctx.center.call_plugin(...)`; the runner supplies the credential and address, which the plugin never sees. The device
uploads the file first, then submits the Source that references it; the central Server runs the Projections when it
writes the Source, so they can read the file from plugin storage, and their results are saved with the Source like any
other Projection. Delivery to Artifact plugins reads only the saved results and never reruns Projections, so cleaning up
a file later cannot fail a consumer's window. Files are stored by content digest and never change once written, so
Projection results are deterministic.

## 7. Operations [#7-operations]

- Every Source plugin a device uses must also be deployed in the central Server's plugin directory. The central Server
  registers a plugin's definition manifests (the declarations of Source types and Projections) when it loads the plugin,
  as in RFC #2549, with no separate registration command. Devices need only `scope.contribute` on the Scopes they serve.
- When a runner goes offline, its record and targets remain; whether it is online is judged from `last_seen_at`. When a
  device is retired, users delete its targets and record.
- Stateless deployments (containers, CI) either mount the state directory on a persistent volume or use a fixed name.
  With a suggested name and a state directory that does not persist, every start registers another runner.
- After the local state is lost (a reinstall, a cleared directory), starting with a suggested name registers a new name,
  and the old targets are transferred to the new runner; starting with the same credential and a fixed name keeps the old
  name and targets instead. Delete the state file before cloning a machine image, or two devices hold the same name.
- For a malicious or leaked credential, disable it in credential management and delete the runners it registered;
  registration has no separate defense.
- When a device moves to a credential of another subject (for example after its service account is disabled), its old
  name belongs to the old subject and cannot be reused: register a new runner with the new credential, transfer the
  targets to it, and have an administrator delete the old record.

## 8. Versions and upgrades [#8-versions-and-upgrades]

The plugin version loaded on the central Server is the specified version for its capability, and device versions are
compared with it; the plugin is responsible for data format compatibility.

- **Compatible range**: when a device's plugin version satisfies
  `central min_compatible_version <= device version <= central version`, the central Server accepts its sync, Source
  submission, checkpoint commit, and `call_plugin` requests and hands them to its own plugin; otherwise it rejects them
  with `plugin_version_incompatible`. At sync, targets of an incompatible plugin are not delivered, while the runner's other
  plugins sync as usual; the runner list shows which devices need upgrading. Requests for a plugin the central Server has
  not loaded are rejected too.
- **Upgrade the central Server first, then devices**: a device version newer than the central one is also rejected,
  because the central Server cannot judge whether newer code is compatible. After the central upgrade, older devices
  within the range keep working and can upgrade one by one. With several central instances, each checks against the
  version it loaded, so during a rolling upgrade one device's requests may be accepted by one instance and rejected by
  another; finish upgrading every central instance before upgrading devices. Runners treat
  `plugin_version_incompatible` as retryable and retry at the next sync.
- **What compatibility means**: by declaring `min_compatible_version`, a plugin promises that the new central version can
  process Sources submitted by devices within the range; a new version that still needs to read an old version's Sources
  keeps declaring the old definitions (RFC #2579). The only thing a device reads from the central Server is target
  parameters. They are filled in under the central version's format and may carry fields an older device does not know, so
  device-side parameter models must ignore unknown fields (Pydantic's default; do not set `extra="forbid"`). When a new
  parameter must take effect on every device, raise `min_compatible_version`. A checkpoint is normally written and read
  by the runner its target is assigned to. A transfer keeps the checkpoint by default, also between a device runner and
  the `server` runner, which is right when the destination runs a plugin version that can read it; otherwise transfer
  with `reset_checkpoint` so that the destination starts from the beginning.
- **Data upgrades**: the central Server calls `upgrade(ctx)` each time it loads a plugin version, under the rules of RFC
  #2579. Devices do not call `upgrade`: core keeps no plugin data on devices, and a plugin checks and migrates its own
  local files in `setup(ctx)`.

# Reference-level explanation [#reference-level-explanation]

## 1. Declarations [#1-declarations]

```python
class TargetSpec(BaseModel):
    description: str                        # the capability description shown on the panel
    parameters: type[BaseModel]             # target parameters; core takes their JSON Schema
    identity: dict[str, tuple[Normalizer, ...]]   # the parameters forming the unique key and their normalization steps

class Collect(BaseModel):
    schedule: Interval | Cron
    max_targets: int = 50                   # maximum targets per invocation
    max_items: int = 200                    # maximum Sources returned per invocation
    timeout_seconds: float = 120
    max_workers: int = 1

class SourcePlugin(BaseModel):
    version: str                            # PEP 440
    min_compatible_version: str | None = None   # defaults to version
    definitions: Sequence[SourceDefinition]
    collect: Collect | None = None
    target: TargetSpec | None = None        # required when collect is set
    scopes: tuple[str, ...] | None = None   # Scope allowlist of the Server runner
    setup: Callable[[PluginContext], Awaitable[SourceCollector]] | None = None
```

- The plugin name is the capability name and must match `^[a-z][a-z0-9-]{0,63}$`.
- `version` and `min_compatible_version` must be valid PEP 440 versions with `min_compatible_version <= version`;
  otherwise the plugin fails to load.
- Normalization steps come from a fixed set: `strip`, `lower`, `collapse_spaces`, `remove_prefix:<value>`, and
  `remove_suffix:<value>`. The central Server and runners share one implementation. The target key is the SHA-256 of the
  capability name plus each parameter's normalized value, ordered by parameter name.
- Parameters in the identity expression must be string fields of `parameters`; parameters not listed do not take part in
  identity.

## 2. Collection contract [#2-collection-contract]

```python
class CollectTarget(BaseModel):
    target_id: str
    parameters: dict[str, JsonValue]
    checkpoint: JsonValue | None            # the target's last committed checkpoint; None the first time; read-only

class CollectRequest(BaseModel):
    scope_id: str
    reason: Literal["schedule", "target_added"]
    targets: Sequence[CollectTarget]
    max_items: int

class CollectedItem(BaseModel):
    definition: str                         # the name of a definition in definitions
    value: BaseModel                        # an instance of that definition's input_class

class CollectResult(BaseModel):
    def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None,
            has_more: bool = False) -> None: ...
    def fail(self, target: CollectTarget, *, code: str, message: str) -> None: ...   # only this target failed

class SourceCollector(Protocol):
    async def collect(self, request: CollectRequest, /) -> CollectResult: ...
```

- Each invocation covers one Scope and at most `max_targets` enabled targets; with more, targets rotate, and targets that
  returned `has_more` go first in the next invocation.
- Returning more than `max_items` items in total, or a target absent from the request, is a failure; nothing is truncated.
  Targets absent from the result keep their checkpoint.
- Source identity is determined by `resolve`. Repeated collection of the same external state with the same identity creates
  no duplicate Source; the same identity with different content is a conflict, and the invocation fails.
- A plugin reports a single target's failure with `fail`: that target's checkpoint stays, its status columns record the
  failure, and the other targets' results in the same invocation commit as usual. When `collect` raises, returns an
  invalid result, or the submission fails, the whole invocation fails and enters the backoff of the Supervisor (on the
  Server) or the runner (on a device).

## 3. Persistence [#3-persistence]

| Data | Handling |
| --- | --- |
| `pc_source_runners` | Added: `runner` (primary key, the name), `display_name`, the registering subject `principal_type` and `principal_id`, `registration_id` (the registration ID of a suggested-name registration, unique on `(principal_type, principal_id, registration_id)`; empty for a fixed name), `registered_at`, `capabilities` (canonical JSON: each plugin's name, version, and load error), `capabilities_digest`, `last_seen_at` |
| `pc_source_collect_targets` | Added: primary key `(scope_id, target_id)`; `runner` (indexed), `capability`, `target_key`, `parameters` (canonical JSON), `enabled`, `config_version` (the configuration version, starting at 1), `updated_at`, `updated_by`; unique `(scope_id, runner, capability, target_key)`; written by the runner: `last_attempt_at`, `last_success_at`, `consecutive_failures`, `last_error_code`, `last_error_message` |
| `pc_connector_checkpoints` | Kept; target checkpoints, keyed by `(scope_id, target_id)` |
| `pc_sources`, Projections, the Source journal, `pc_source_definition_manifests` | Kept |
| `pc_source_write_plugins` | Dropped; plugins that provide only definitions come from the plugin directory |
| The old Source plugin management service's SQLite | Removed with the management service, not migrated |

- Write frequency: `capabilities` is written only when its digest changes; `last_seen_at` once per runner per sync; status
  columns at each collection, in the same transaction as the checkpoint (on a device, in the checkpoint commit request).
  User changes to targets write only the user columns and never conflict with runner writes.
- The two new tables and the drop ship as one new migration revision; the migration resources of released versions stay
  unchanged. The migration preflight refuses to upgrade while `pc_source_write_plugins` still holds data and leaves the
  database unchanged (see section 6).
- Export and restore include both new tables; `last_seen_at` in `pc_source_runners` and the status columns of
  `pc_source_collect_targets` are not exported and restore empty.

## 4. APIs [#4-apis]

| API | Permission | Description |
| --- | --- | --- |
| `GET /v1/source-runners` (MCP `pc_source_runners`) | Authenticated | Every registered runner, its display name, its capabilities, whether their versions are compatible with the central Server, and when it was last seen; capability descriptions and parameter formats come from the central Server's plugins |
| `POST /v1/source-runners` | Authenticated | Registers: a suggested-name registration carries `registration_id` and an optional `suggested_name`, a fixed-name one carries `name`; returns the decided name. See section 3 of the guide-level explanation |
| `PATCH /v1/source-runners/{runner}` | `server.admin` | Changes the display name |
| `POST /v1/source-runners/{runner}/sync` | The runner itself | Accepts only registered names; the request carries the capability list (plugin versions and load errors); the response returns every enabled target assigned to the runner whose version is compatible, each with its configuration version and checkpoint |
| `DELETE /v1/source-runners/{runner}` | The subject that registered the runner, or `server.admin` | Deletes a runner record; a conflict while it still has targets |
| `POST /v1/scopes/{scope_id}/source-targets` (MCP `pc_source_add`) | Source write on the Scope, and the principal that registered the runner or `server.admin`; `server.admin` when the runner is `server` | Creates a target: `runner`, `capability`, `parameters`; validated and deduplicated |
| `GET /v1/scopes/{scope_id}/source-targets` (MCP `pc_source_status`) | Scope read | Targets, their configuration, and status |
| `PATCH /v1/scopes/{scope_id}/source-targets/{target_id}` (MCP `pc_source_target_set_enabled`) | As for creation to change parameters or enable; to disable, Source write on the Scope, and `server.admin` for a `server` target | Changes fields of `parameters` outside the identity expression, or enables and disables |
| `POST /v1/scopes/{scope_id}/source-targets/{target_id}/transfer` (MCP `pc_source_target_transfer`) | As for creating it on the destination runner; `server.admin` when the source runner is `server` | Reassigns the target to another registered runner and increases its configuration version; optional `reset_checkpoint` |
| `DELETE /v1/scopes/{scope_id}/source-targets/{target_id}` (MCP `pc_source_target_delete`) | Source write on the Scope; `server.admin` for a `server` target | Deletes a target and its checkpoint |
| `POST /v1/source-observations` | Existing | Requests from a runner gain the plugin name and version, checked against the compatible range |
| `POST /v1/connector-checkpoints/commit` | Existing; a collection target's checkpoint is accepted only from the runner the target belongs to | The request gains an optional `status` (latest attempt, success, error category, and message) and `config_version`; requests from a runner gain the plugin name and version, checked against the compatible range. The central Server decides by `(scope_id, target_id)` whether the commit belongs to a collection target; if it does, it validates the target's existence, enabled state, ownership, and configuration version in the same transaction that writes the checkpoint and status, and returns a conflict otherwise |

"The runner itself" means the authenticated subject is the runner record's registering subject, with the same type and
ID.

Requests sent through `call_plugin` carry the plugin version too.

Removed: the instance, configuration, channel, and write-plugin loading APIs under `/v1/source-management/*`; MCP
`pc_source_instances` and `pc_source_instance_set_enabled`; the `source load` command; and the `powercontext.yml` /
`write.yml` manifests. The central Server registers definitions when it loads a plugin; there is no separate definition
registration command.

## 5. Device-side context [#5-device-side-context]

On a device, `PluginContext` adds `center`:

```python
class PluginCenter(Protocol):
    async def call_plugin(self, plugin: str, method: str, path: str, /, *, json: JsonValue | None = None,
                          content: bytes | None = None, content_type: str | None = None) -> PluginResponse: ...
```

It calls central plugin routes (`/v1/plugins/{plugin}/api{path}`) with the runner's credential and can reach only the
plugin's own routes. A central Source plugin may declare RFC #2579 routes, and its Projections may read plugin storage
through `ctx.storage` when the Source is written (section 6).

On devices, core keeps no plugin data, so `ctx.storage` is unavailable and raises when accessed; the plugin manages its
own local files. `ctx.center` exists only on devices, which lets a plugin tell which side it runs on.

## 6. Mapping from the old framework [#6-mapping-from-the-old-framework]

| Old framework | This proposal |
| --- | --- |
| The management service and its SQLite | Two central tables |
| Instance registration (by an administrator) | Runners register themselves and the central Server decides their names; they report capabilities when they sync |
| Reverse poll / reply channels | Runners pull and sync |
| Instance `scope_ids` | Targets carry their Scope |
| `default_capabilities` default instances | Not provided; users choose the runner on the panel |
| `identify` calls on the device | A declarative identity expression computed centrally |
| The built-in `content.inline` capability | The existing Source write API |
| Write plugins (`pc_source_write_plugins`) | Source plugins that provide only `definitions` |
| The tickets, ticket_chat, and images examples in `examples/source_plugins` | Rewritten as Source plugin examples |

**Upgrading deployments that used the previous framework**:

- Databases that never loaded a write plugin and never ran the previous management service are unaffected: the migration
  only adds two empty tables and drops the empty `pc_source_write_plugins`.
- While `pc_source_write_plugins` holds records (after `source load --kind write`), the migration refuses with
  `unsupported_plugin_data` and leaves the database unchanged. The previous version has no operation that removes these
  registrations, so such a database currently has no supported in-place upgrade path: keep using the previous version,
  or move to a new database; do not delete the records by hand to bypass the check.
- The previous management service's own SQLite (instances, configuration, and channels) is not migrated. After the
  upgrade, rewrite the old plugins as the Source plugins of this proposal and configure the collection targets again
  through the target APIs.
- Collected Sources, Projections, and the checkpoint table are kept. The definition manifests of Source types
  (`pc_source_definition_manifests`) are kept too; a rewritten plugin must keep declaring definitions of the same name
  and version for the saved Sources to stay readable.

# Drawbacks [#drawbacks]

- **Two more tables**: migrations, export and restore, and both database backends must support them.
- **Limited normalization**: identity uses only fixed normalization steps, so complex rules such as URL normalization cannot
  be expressed; plugins can merge identical objects themselves while collecting.
- **Configuration takes effect with a delay**: up to the sync interval, 30 seconds by default.
- **Only its owner and administrators assign targets to a shared runner**: on a runner run under a service account for a
  team, members cannot create targets themselves; the service account or `server.admin` does. Letting a runner accept
  each assignment would be more flexible and is not part of this RFC.
- **Runner identity rests on the credential**: a name is not a secret, so whoever holds a credential can sync under any
  name that credential registered, consistent with the trust model of a shared credential. Two runners that fix the same
  name under one credential, or that copied the same state file, count as one runner and get the same targets, which
  core cannot tell apart.
- **A fixed name may be taken**: when another subject already registered the fixed name, the runner fails to start until
  an administrator deletes that record.
- **The central Server must be able to load every Source plugin on devices**: libraries used only on devices cannot be
  imported at the top of `plugin.py`, or the central load fails; splitting dependencies by side is left to the later
  plugin installation and dependency proposal.
- **The central Server must upgrade first**: when devices upgrade before it, requests from the newer devices are rejected
  until the central Server upgrades.

# Rationale and alternatives [#rationale-and-alternatives]

| Option | Trade-off |
| --- | --- |
| Runners choose their names, bound to the credential subject at the first sync | No registration API; but a sync creates the record as a side effect, uniqueness depends on the devices, and targets can point at runners that never appeared |
| Targets stored locally on the device (RFC #2549's initial design) | No new table, but the central Server can neither see nor manage device targets; embodied scenarios would need logging in to each device |
| Targets stored as Artifacts of a built-in family (like Scope Prompt management) | No new table, with revisions, restore, and export ready-made; but finding a device's targets means searching across Scopes without an index |
| Each device configures a home Scope that lists the Scopes it serves | No new table, but assigning a new Scope to a device means first finding its home Scope, which still needs a scan |
| The central Server pushes configuration through a reverse channel (the old framework) | Configuration applies immediately, but it needs long-lived connections, leases, and instance registration, the main source of the old framework's complexity |
| Targets with a revision and an "applied revision" | Helps concurrency control and showing what is applied; but runners fetch everything on each sync and time-based checks are enough, which also avoids clock and incremental-fetch problems |
| `identify_target` written as plugin code | More expressive, but the central Server cannot run it for a plugin on a device, so duplicates are found only afterwards |
| Each runner reports its own parameter format and targets are validated against their runner's format | The central Server need not upgrade first; but a capability has different forms on different devices, targets must be revalidated one by one after a format change, and there is no way to judge whether a device version is compatible |
| The plugin provides a parameter migration function and core rewrites target parameters | Incompatible formats could upgrade automatically; but runners would write back parameters users filled in, conflicting with user edits, and raising `min_compatible_version` already constrains device versions |
| Devices also call `upgrade(ctx)` | Symmetric with the central Server; but core keeps no plugin data on devices, and running the same `upgrade` on both sides would require the plugin to tell which side it is on |
| No deduplication | Duplicate targets only collect twice, and Sources are still deduplicated by identity; but panel configuration needs an immediate hint, and lists and statistics need unique targets |

# Prior art [#prior-art]

- **AWS IoT Device Shadow and Azure IoT Hub Device Twin**: the cloud stores the desired state, devices report the actual
  state, and devices connect to the cloud to sync.
- **Kubernetes spec and status**: users write the desired state, controllers write observations, and the two are stored
  separately.
- **The Airbyte connector protocol**: a connector receives the previous state and emits records and a new state, which the
  platform stores.
- **Existing PowerContext foundations**: the remote ingestion APIs `POST /v1/source-observations` and
  `/v1/connector-checkpoints/*`, definition manifest registration, RFC #2549's plugin host processes and Supervisor
  scheduling, and RFC #2579's plugin routes.

# Unresolved questions [#unresolved-questions]

| Question | Notes |
| --- | --- |
| Visibility of the runner list | Every authenticated user can see the runner list and capabilities today. Whether it should be limited by Scope |
| Targets per runner | Fetching everything on each sync suits a few to a few dozen targets per device; whether a Server runner collecting hundreds or thousands of accounts needs paging or incremental sync |
| Default runners | Whether plugins or deployers may name a default runner per capability to reduce choices on the panel |

# Future possibilities [#future-possibilities]

- Plugin installation and dependencies: installing from PyPI, a Git URL, or a wheel, with plugins on devices downloading the
  version the central Server specifies during sync, and dependencies declared separately for the central Server and devices.
- Push notification: an optional long-lived connection besides sync, so configuration applies sooner.
