A position-sizing calculation blew up. Not a wrong number — no number at all. The account had money in it, the broker knew the balance, and the process whose entire job was holding that balance answered nil.

Where this fits

The last two posts were about accounting for trades — P&L when a position spans sessions and the reconciliation that fixes it. This one is about the thing those trades happen inside: the account. It's the part of the system I assumed would be boring, and it has produced more distinct production failures than the order logic it supports.

An account sounds like the simplest object in the system

It's a row. An id, a name, a balance, a link to whichever credential found it. You sync it once, you read it when you need to size a position, and you get on with the interesting work.

That description is accurate right up until the account is attached to something live. Then "the balance" stops being a value and becomes a claim about a remote machine — one with a timestamp, a failure mode, and at least three processes that each believe they hold the current version.

Three layers, and confusing them is where the bugs come from

There are three separate representations of account state, and they do not agree by default.

The database row is a cache, not truth. It carries cash_balance, realized_pnl, last_synced_at, sync_error. It is written by a user-triggered sync that fans out over every {broker, credential, environment} combination the user holds, and it exists so the connections page has something to render before anything is running.

The broker session — TSSession, one per broker credential rather than one per trading session — polls on a timer and broadcasts over PubSub. It writes nothing to the database at all, which follows from the sharing: a process several sessions attach to has no single account row to own. The interval on that poll has a comment attached that is really an incident report:

  # Balances change in response to fills, so a 30-second cadence is plenty
  # responsive for the UI/portfolio-heat path. Polling every 5 seconds turned
  # this handler into the highest-frequency cascade-entry point — when a
  # brief token-refresh gap opens at the broker, the 5s cycle generates an
  # immediate 401 storm before TokenStore can catch up. On error, back off to
  # 60s for that single cycle to give TokenStore room to refresh; the next
  # success resets to the 30s cadence.
  @accounts_poll_interval_ms 30_000
  @accounts_poll_backoff_ms 60_000

This isn't an accounts problem. Any process polling a credentialed endpoint on a short fixed interval is a queued outage waiting for its trigger, because the failure it's most likely to hit — an expired token — is exactly the one that makes retrying faster actively harmful.

The per-session AccountManager is a GenServer holding the live view: orders, position, balance. It's the one the trading logic actually reads. I've called it "the executor" in these posts, because submitting orders is the half that comes up; this is the other half.

It takes two API calls to produce one balance

TradeStation returns account identity from one endpoint and account money from another. That split seeds most of what follows, because the struct gets built first with the money missing:

  defp to_account_balance(account, ts_user_id) do
    %AccountBalance{
      account_id: account["AccountID"],
      name: account["AccountID"],
      nickname: account["Alias"],
      user_id: ts_user_id,
      cash_balance: nil,
      realized_pnl: nil,
      type: account["AccountType"]
    }
  end

A second call fetches balances for the joined account ids and merges them in. Which means a perfectly valid AccountBalance exists, in a normal code path, with cash_balance: nil. It isn't an error state. It's a stage.

Tradovate has the identical two-call shape and makes the opposite choice about the gap — an account with no matching balance row is kept with zeroed figures rather than dropped. Two brokers, one problem, two defensible answers. The adapter layer has to be honest that both exist, which in practice means the rest of the system cannot assume a balance is populated just because an account was returned.

Failure one: the balance that was never there

The trader (RealtimeTrader) and its account manager boot concurrently. The trader reads account state exactly once during startup, and the manager may not have resolved its account yet at that instant. So the read legitimately returns nil.

Nothing re-read it. The nil persisted until an unrelated balance broadcast happened to land — and on a quiet account those can be hours apart. In between, every fixed-fraction sizing decision blew up inside the strategy sandbox.

The fix has two halves worth showing. First, a definition of "resolved" that is more careful than a nil check, because three different shapes all mean the same thing:

  @doc false
  # A balance is only usable once it actually carries a cash figure. `nil` (the account manager
  # hadn't resolved an account yet), `%{}` (the `State` struct default) and a populated struct
  # whose `cash_balance` is still nil all mean "not resolved" for position-sizing purposes.
  # Public so the retry behaviour is directly testable.
  def account_balance_resolved?(balance) when is_map(balance),
    do: not is_nil(Map.get(balance, :cash_balance))

  def account_balance_resolved?(_balance), do: false

A populated struct whose cash_balance is nil is the one that gets you. It passes every "did we get an account?" check and still can't size a trade.

Second, a retry schedule that deliberately never gives up:

  # Backoff schedule (in ms) for re-reading the account balance when the account manager hadn't
  # resolved one yet at trader startup. Front-loaded so the common case — the manager wins its
  # own account lookup a beat after we asked — heals in about a second; the tail keeps polling
  # at `@account_balance_retry_max_ms` indefinitely rather than giving up, because the only
  # other thing that can ever fill the balance in is an inbound `:account_balances` broadcast,
  # and on a quiet account those can be hours apart.
  @account_balance_retry_backoff_ms [1_000, 2_000, 5_000, 10_000, 30_000]
  @account_balance_retry_max_ms 60_000

"Retry forever" is usually a smell. It's correct here for a specific reason: giving up would leave the session running and permanently unable to size a trade, which is strictly worse than a log line every sixty seconds. The warning is throttled to every tenth attempt so a manager that never resolves doesn't write a warning a minute for the life of the process.

The re-fetch also has to survive the manager being dead, because a GenServer call into a missing process exits the caller by default. A failed re-read must never take the trader with it — it just means we try again on the next tick.

Failure two: a live trader with a dead account manager

That last guard is where this gets interesting, because it caused the next bug.

A session's supervisor groups its children :one_for_all — the per-session SessionSupervisor the broker-agnostic post landed — with :transient workers. The supervision tree post established :transient: it restarts a process that terminates abnormally — anything other than :normal or :shutdown — and leaves alone one that stops cleanly. That distinction is the entire bug:

  # ── Dead account manager sweep ──────────────────────────────────────────────
  #
  # A session's `SessionSupervisor` tree is grouped `:one_for_all`, but an account manager that
  # stops with reason `:normal` (its `restart: :transient`, e.g. account resolution exhaustion)
  # doesn't trigger a group restart — and since RealtimeTrader now degrades gracefully instead of
  # crashing when its account manager is gone, nothing else in the tree crashes either. The
  # session is left running-but-broken: RealtimeTrader alive, account manager gone,
  # PnL/position/order reconciliation failing silently forever.

This is the kind of thing that only shows up in production. Hardening the trader so it wouldn't crash when its account manager disappeared removed an accidental self-heal. Previously the trader crashed, the crash was abnormal, :one_for_all tore down the group, and everything came back.

The crash was load-bearing. Making the system more robust at one layer made it permanently broken at another, and nothing alerted, because from the outside the session looked fine.

The recovery went into EnsureSessionsAreRunning — the process that already brings sessions back after a node reboot. When I introduced it I was careful to say it wasn't a workaround for a supervision problem. It is now, a little. It watches for one specific shape — trader registered, account manager not, for longer than a grace period — and cycles the session, on a 30-second tick with a three-minute grace window: long enough to ride out a broker hiccup, short enough that a session doesn't sit silently stale.

The detection pass is a pure function taking the sessions, a map of when each manager was first seen missing, and the current time, returning what to cycle and the pruned tracking map. That pruning is load-bearing too — a session that was just cycled drops out of the map so it gets a fresh grace window instead of re-triggering on the next tick. Splitting the decision from the action is what makes a grace period testable without standing up a supervision tree, and elapsed time is measured with System.monotonic_time/1, not wall clock.

Failure three: the broker changed the account id

Then there's the one I didn't design for at all. Tradovate can hand a login a new account id. The stored id now matches nothing, and the session is dead in the water through no fault of anything local.

Recovery has to guess, and the interesting part is that the guess is gated on cardinality:

  # Account resolution exhausted its retries and the stored account_id still matches nothing.
  # If Tradovate is returning exactly one account for this login, that's high-confidence
  # evidence the account was reissued a new id under us (a known Tradovate behavior)
  # rather than a genuine misconfiguration, so we auto-adopt it. With more than one
  # candidate, guessing risks trading on the wrong account (wrong margin/buying power/exposure),
  # so behavior is unchanged: log clearly and stop.

One candidate: adopt it, persist the correction, alert loudly. More than one: refuse and stop. The asymmetry is the whole design. Automation that recovers is worth a great deal; automation that recovers onto the wrong account is worth less than no automation, because now real capital is moving somewhere nobody is looking.

That fix then needed two more rounds of fixes. The one worth showing is the round where the recovery started persisting broker-supplied names: the broker frequently returns a nil nickname, one of the target columns is allow_nil? false, and the persist runs on every balance broadcast. A naive write retried a failing database write for the life of the session. The repair is three clauses:

  defp maybe_put_change(params, _key, _old, nil), do: params
  defp maybe_put_change(params, _key, same, same), do: params
  defp maybe_put_change(params, key, _old, new), do: Map.put(params, key, new)

Skip nil, skip unchanged, otherwise write. Pattern matching earns its keep in places like this — same, same as a change detector is the kind of thing that reads as obvious only after you've written the version with the if.

The same round added a third name column. The recovery had been writing the broker's nickname straight onto the user-editable one, silently clobbering whatever the user had typed. The account now carries the broker's name, the user's nickname, and the broker's nickname as separate fields, because two of those are owned by different parties and neither gets to win.

What this system deliberately doesn't do

Account management is a phrase that implies more than what's here.

There is no buying power field anywhere. There is no margin call handling, no maintenance-margin monitoring, no liquidation logic. Margin isn't account state at all — it's fetched from the broker on demand, per symbol, and folded into the trader's contract details when a feed is built, never stored on the account row. There are no pattern-day-trade rules, because these are futures and PDT is an equities rule that doesn't apply. Positions aren't stored on the account row either — they're fetched per session and reconciled on a timer.

Currency is the one to be careful about. Margin figures carry their own currency and get converted to USD where they're summed. Cash balances carry no currency and are added together directly. That's fine exactly as long as every account is USD-denominated, which the code assumes and does not check — a latent bug with a trigger I haven't pulled yet.

And the uniqueness rule that stops two rows claiming one account id is enforced in application code, not by a database constraint. The check is a SELECT immediately before the write, which is the kind of uniqueness rule that works until two code paths want the same id at once.

The part that turned out to be interesting

Every failure here is the same failure wearing a different hat. An account isn't a value the system owns; it's a cached assertion about somebody else's database, and every layer that caches it has to be explicit about what it means when the answer isn't there yet.

The balance can be missing because the second API call hasn't happened. The holder of the balance can be gone while everything around it looks healthy. The identity the balance is keyed on can change underneath a running process. None of those are exotic distributed-systems problems.

The one thing I'd carry to any system with an external account behind it: decide early what "we don't know the balance yet" means, and make it a state you can name and test. Every bug above is what happens when that state exists in reality but not in the code.

The next post is what the system does once it does know the balance: portfolio heat, and enforcing a real-time risk limit without anything that deserves to be called a risk engine.