Configuring overload and loss per feed¶

This guide is for teams that run a market maker, or any trading application, on Mongoose. It shows how to choose and configure what each feed does when a subscriber falls behind. The design and its guarantees are in Overload and loss policy (design); this page is the practical how-to.

1. The choice per feed¶

Every feed and pipe has an overload policy. You make the choice once per feed, and optionally override it for individual subscriber groups (processor group names).

Policy	What happens when a subscriber queue is full	Use for
`DROP_AFTER_SPIN` (default for push feeds)	Spin for the spin bound (10 ms by default), then drop the item for that queue only. Counted, and the queue's health turns DEGRADED.	Replaceable data, where the next update fully supersedes the last: top-of-book and quote snapshots, telemetry.
`PAUSE` (default for file feeds, from 1.1.3)	The source does not publish the item. It keeps the item and its read position, reads nothing further, and offers the same item again on its next duty cycle. Nothing waits and nothing is dropped.	Sources that can re-read: a `FileEventSource`. The source slows to the pace of its slowest subscriber.
`BLOCK_WITH_DEADLINE`	Wait up to the deadline for room, without holding any lock. Then deliver to every blocking subscriber, or to none, and throw `PublishRefusedException` to the publisher. Subscribers given a drop-tolerant override may still drop their copy, by design. A loss latches the queue until an operator clears it.	Data that must not be lost silently: venue session and health events, fills, order acknowledgements, credit and limit updates, hedge instructions.

If you set nothing, a feed uses its source's default and logs at boot: feed 'X' has no explicit overload policy: using the source's default, PAUSE for a re-readable source (FileEventSource), else DROP_AFTER_SPIN (spin 10 ms, then drop for that queue). Set a policy on every feed so the choice is explicit.

Changed in 1.1.3: file feeds pause by default

A FileEventSource feed with no overload block now uses PAUSE. Before 1.1.3 it used DROP_AFTER_SPIN, so a slow processor could lose lines from a large file. Now the file feed slows to the processor's pace and loses nothing; feed.status reports policy: PAUSE and counts pauses. To keep the old behaviour, set overload.policy: DROP_AFTER_SPIN on the feed. An explicit policy always wins, and so does the deprecated slowConsumerStrategy: BACKOFF, which means an explicit DROP_AFTER_SPIN on every feed, file feeds included. Every other source keeps DROP_AFTER_SPIN as its default.

2. Recommended policies for a market maker¶

Feed	Policy	Deadline	Notes
Venue session and health (logon, logout, disconnect, kill switch)	`BLOCK_WITH_DEADLINE`	e.g. 250–1000 ms	Never drop. Publish from the venue connector's own thread, not an agent thread.
Fills and execution reports	`BLOCK_WITH_DEADLINE`	e.g. 250 ms	Never drop. A refusal must reach the connector, which rejects or re-requests upstream. Turn on `killSwitch`.
Order acknowledgements and cancels	`BLOCK_WITH_DEADLINE`	e.g. 100–250 ms	As fills. Cancel priority is not implemented yet.
Credit and limit updates, hedge instructions	`BLOCK_WITH_DEADLINE`	e.g. 250 ms	Never drop.
Replaceable market data (top-of-book, quote snapshots)	`DROP_AFTER_SPIN`	none (spin 0–10 ms)	A loss costs staleness: the next snapshot repairs it. Conflation is not implemented yet; a stale queue keeps old snapshots and drops the newest.
Incremental market data (order-book deltas, trades)	`BLOCK_WITH_DEADLINE`, or `DROP_AFTER_SPIN` only with sequence-gap detection and resynchronisation	e.g. 50–250 ms	A missed delta or trade can permanently corrupt derived state; the next update does not repair it. Either refuse explicitly and handle the refusal, or detect the gap from the venue's sequence numbers and resynchronise (snapshot and replay).
Telemetry and analytics subscribers on any feed	`DROP_AFTER_SPIN`, as a per-group override, with `spinBoundMillis: 0`	none	Stops a slow telemetry consumer from slowing, or refusing, the fills that a risk group needs.

The deadlines above are illustrations, not recommendations: choose them per deployment from your queue sizes, consumer rates and how long a connector can safely be held. Keep them short. A deadline is how long a publisher may be held waiting for queue capacity by one wedged subscriber. It bounds only that wait: the time to run the data mapper or the journal codec, or to acquire the publish lock behind another publisher, is not included. Deadlines above 60 s are warned at boot.

3. Configuration¶

YAML¶

eventFeeds:
  - name: venue-session
    instance: !!com.example.VenueSessionSource { }
    overload:
      policy: BLOCK_WITH_DEADLINE
      blockDeadlineMillis: 500
      killSwitch: true

  - name: fills
    instance: !!com.example.FillSource { }
    overload:
      policy: BLOCK_WITH_DEADLINE
      blockDeadlineMillis: 250
      killSwitch: true
      subscribers:
        telemetryGroup:              # processor group (agentName) of the telemetry processors
          policy: DROP_AFTER_SPIN
          spinBoundMillis: 0

  - name: prices
    instance: !!com.example.PriceSource { }
    overload:
      policy: DROP_AFTER_SPIN
      spinBoundMillis: 1
      quietPeriodMillis: 30000       # DEGRADED clears after 30 s without loss (minimum 1000)

Keys¶

Key	Where	Default	Rule
`policy`	feed, subscriber	required in a block	`DROP_AFTER_SPIN`, `BLOCK_WITH_DEADLINE` or `PAUSE`
`blockDeadlineMillis`	feed, subscriber	none	Required for, and only for, `BLOCK_WITH_DEADLINE`; > 0; warned above 60000. Rejected with `PAUSE`
`spinBoundMillis`	feed, subscriber	10	≥ 0. With `PAUSE`, used only when a push publish falls back to drop
`subscribers`	feed	none	Keys are processor group names; an unknown name is warned at boot and after start
`quietPeriodMillis`	feed	60000	≥ 1000; recovery of drop-tolerant queues
`killSwitch`	feed	false	Calls the server's kill-switch hook when a never-drop queue latches

slowConsumerStrategy is deprecated. BACKOFF means DROP_AFTER_SPIN, and DISCONNECT and EXIT_PROCESS fail boot.

Programmatic¶

EventFeedConfig.builder().name("fills").instance(fillSource)
    .overload(OverloadPolicyConfig.blockWithDeadline(Duration.ofMillis(250))
        .killSwitch(true)
        .subscriber("telemetryGroup", OverloadPolicyConfig.dropAfterSpin(Duration.ZERO)))
    .build();

4. Publishing into a never-drop feed¶

Publish from a non-agent thread. A publish waits for room only on a connector's or session's own thread. On an agent thread (any processor group or service agent group), or nested inside another publish, a full queue falls back to drop-after-spin. That publish is counted in downgradedCount(), and the queue latches. This is why a HandlerPipe on BLOCK_WITH_DEADLINE protects only its non-agent publishers, and it is warned at boot.
Handle PublishRefusedException. It means no subscriber received the item and no sequence number was used, so a retry does not duplicate. Reject the order upstream, alarm, or retry.
Pooled items: copy the payload before you publish if you may retry. The refused instance is back in its pool and has been reset by the time you see the exception. Acquire a fresh one and fill it from your copy.
Pre-start caches are not protected on drop or block feeds. A feed that caches events before start (cacheEventLog) replays them with drop semantics. This is warned. On a PAUSE feed the replay pauses at a full queue and resumes in order instead (section 5).

5. PAUSE: no-drop backpressure for re-readable sources¶

PAUSE (new in 1.1.3) is the lossless choice for a source that can read the same data again. When a subscriber's queue under PAUSE is full, the publish does not happen at all: no sequence number, no journal entry, no queue holds the item. The source keeps the item and offers it again on its next duty cycle. Nothing waits, so it is safe on an agent thread, which is where a FileEventSource publishes from. (BLOCK_WITH_DEADLINE cannot wait on an agent thread and falls back to drop there; that is why a file feed had no lossless setting before 1.1.3.)

eventFeeds:
  - name: orders
    instance: !!com.telamin.mongoose.connector.file.FileEventSource
      filename: data/orders.csv
    overload:
      policy: PAUSE                 # the file default; DROP_AFTER_SPIN to keep the old behaviour
      spinBoundMillis: 10           # optional: used only by a push publish's fallback
      subscribers:
        telemetryGroup:             # a lossy subscriber that must not slow the others
          policy: DROP_AFTER_SPIN
          spinBoundMillis: 0

Programmatic: OverloadPolicyConfig.pause().

What to expect:

The whole feed slows to its slowest PAUSE subscriber. That is the price of losing nothing. To stop a slow, drop-tolerant subscriber from holding the others back, give its group a DROP_AFTER_SPIN override, as above. Only PAUSE (and BLOCK_WITH_DEADLINE) queues are checked for room; a DROP_AFTER_SPIN queue of the same feed still drops after its spin.
A file source on a pause keeps the line and the rest of its buffer, reads nothing further from the file, and stays BEHIND in feed.status. It reports CAUGHT_UP only once it has published everything to the end of the file.
BLOCK_WITH_DEADLINE on a file feed pauses too, instead of falling back to drop on the agent thread.
Push publishes (a source that cannot pause: in-memory feeds, pipes, most connectors) into a full PAUSE queue fall back to DROP_AFTER_SPIN with the spin bound. The drop is counted in dropped, the fallback in downgraded, and it is warned once. Use PAUSE only on sources that support it: today, FileEventSource.
Cache replay (cacheEventLog) pauses at a full queue and resumes in order.
Health. A pause is not a loss and changes no health state. PAUSE is a never-drop policy, so a loss on it (only possible through the push fallback) latches the queue like a BLOCK_WITH_DEADLINE loss.
Fenced or dead groups. On a feed that commits positions (a COMMITED file feed) a fenced or dead consumer pauses the whole feed until the fence lifts. See Process a file feed at least once.

feed.status shows the policy and the pauses per feed: policy, paused (now), pauses (episodes), and dropped, refused, downgraded, lossless. See Wait until a feed has caught up.

A pause that never ends (a wedged subscriber) is visible only in feed.status today: the feed stays paused and does not settle. There is no health check for it yet.

6. Wiring the kill switch¶

The hook is off by default. To use it, set killSwitch: true on the feeds that should trigger it, and register a hook once:

server.setOverloadKillSwitch(loss -> {
    // runs on mongoose-overload-kill-switch-<feed>, never under a publish lock: it may block or call the server
    log.error("never-drop loss on {}/{} ({}): {}", loss.feed(), loss.queue(), loss.kind(), loss.reason());
    riskControls.haltQuoting("overload loss on " + loss.feed());
});

It fires once per latch, the first loss since the queue was last cleared.
Once the server begins to stop, no hook is started, including one already queued. A hook that is already running is not interrupted: write hooks that return promptly.
It does fire for a queue that latched before the hook was set.
A hook that runs longer than 5 s is warned. Each feed has its own hook thread, so a slow hook never delays another feed's.

7. Operator runbook: LATCHED and clear¶

You see	Meaning	Do
Health `queue-drops:<path>` DEGRADED, "N item(s) dropped by queue"	A drop-tolerant queue lost items	Check consumer capacity. It returns to UP after the feed's quiet period with no further loss. A feed with no configured policy stays DEGRADED, as in earlier releases.
Health DEGRADED, "never-drop queue … latched until cleared"	A never-drop queue refused or dropped an item	Treat as an incident: find what was lost (below), reconcile with the venue, then clear.
Log `overload state: … to=LATCHED`	The latch happened. The reason says refused, dropped or downgraded.	As above.
Log or audit `from=LATCHED to=REMOVED`	The latched queue was unsubscribed. The latch is kept.	Still clear it. The health check stays until you do.

Commands, on the admin interface:

overload.status              # every feed, queue and state; a queue removed while latched shows as "<name> [removed]: LATCHED"
overload.clear <feed>        # operator clear: LATCHED and DEGRADED queues of the feed return to UP

The same are available on the server as MongooseServer.overloadStates() and clearOverloadState(feed). A clear writes one record per queue and does not reset counters.

What was lost. Look at these, by feed and queue:

the OverloadState records (publisher state audit, and the group control audit for never-drop queues);
the first-loss QueueDrop record;
the counters queue.<path>.refused, .dropped, .blocked, .blockedNanos and .blockedMaxNanos, and feed.<name>.downgraded.

In RECORD mode the processor's recording holds a Dropped entry at the point of loss. Drops past the 1024 pending marks are not placed individually; they are counted into the first entry after the overflow. A replay reports it (GroupReplayer.losses, lossless()) and carries on.

8. Checklist¶

Every feed and pipe has an explicit overload.policy, and no boot warning says otherwise.
File feeds that must not lose lines use PAUSE; slow, drop-tolerant subscribers on them have a DROP_AFTER_SPIN override.
Your launcher fails a run that reaches settled with lossless: false in feed.status.
Every never-drop feed is published from non-agent threads, and its publishers handle PublishRefusedException.
Telemetry and other slow subscribers on never-drop feeds have a DROP_AFTER_SPIN override.
Every subscriber override names a real processor group: no warning at boot or after start.
Never-drop feeds that should halt trading have killSwitch: true, and a hook is registered.
Alerting covers DEGRADED health for queue-drops:* and the overload state log lines.
The operator runbook includes overload.status and overload.clear.

Combined 1.1.0 failure and overload contract¶

See the 1.1.0 release notes and client migration checklist for the complete breaking-change and operator acceptance checklist. PublishRefusedException carries FailureKind and EvidenceStatus; overload transition records carry OVERLOAD_LOSS (or NONE on recovery/clear) and the evidence status at observation. Group status JSON includes overload states, including retained REMOVED latches; the group's overload health check reports loss.

A blocked publish retains its intended bindings while waiting. Group death or quarantine cancels that admission before commit, even if its target queue has since been removed; ordinary unsubscribe re-evaluates remaining targets. The refusal is all-or-nothing and consumes the pooled reference as documented in the overload guide. Cancellation is TARGET_UNAVAILABLE / REFUSED_STATE. The six cases in FailureOverloadSeamTest check metadata, group health/JSON, removed latches, dead control, fatal CONTAIN, and quarantine of a generated processor. Direct calls from external threads are outside this fence. NOT_DEGRADED never proves an export complete.