Part of the IBKR Saga

The Python port was working. Orders were flowing. The system was live on IBKR, trading real capital with real stops, and the whole pipeline — from signal generation through the Elixir-to-Python bridge and into IBKR's API — was holding up. After the encryption ordeal, this felt like the finish line.

It wasn't.

What "working" looked like

The system was submitting bracket orders through IBKR's API: three legs — entry, profit target, stop loss. The stop loss is the one that matters most. It's the insurance policy. Without it, a losing trade has no floor.

The order flow looked correct. The system would submit a bracket, IBKR would acknowledge it, and the dashboard showed all three legs as :submitted. The primary would fill. The profit target and stop loss would appear as working orders. Everything looked right.

Except stop orders submitted via the API weren't executing.

The shape of the problem

Not a crash, no error message, no exception — just an acknowledged order doing nothing. The API returns a success response. The order appears in the working orders list. The system's internal state says the stop is live.

The order sits there, acknowledged by the broker, visible in the UI, completely inert. When price hits the stop level, nothing happens. The position stays open. The loss grows. The system doesn't know anything is wrong because, from its perspective, the stop order exists and is working.

Not a crash. Not an error. A lie.

How I found it

That's when it stopped being a debugging problem and started being a trust problem.

I was watching a position that should have been stopped out, and it wasn't. Price had blown through the stop level and kept going. The system was still showing an open position with a working stop order at a price that had already been hit.

I checked the logs (price was correct), verified the order type (stop, not stop-limit), and pulled status directly from IBKR's API, bypassing the event stream. The order was showing as live on their side. Live. At a price already traded through.

Reproducing it

Once I suspected the API, reproducing it was straightforward. Stop orders submitted programmatically — through my system, through paper trading, through a second platform to rule out my code entirely — all failed the same way. The order appeared live. The price hit the stop level. Nothing happened.

Stop orders placed manually through IBKR's desktop app or mobile app never failed. Same account, same contracts, same stop prices.

That's not an intermittent bug. That's a clear line: API submissions fail, manual submissions work. The failure wasn't in my code, wasn't in my order parameters, wasn't in the broker's execution engine for orders it owned. It was specifically in how IBKR handled stop orders that arrived via their API.

This is the worst kind of production bug: not intermittent at all once you understand it, but invisible until you think to check the submission path.

What hardened after this

At the time, the order submission path was simple. The strategy makes a decision, the MyApp.DecisionHandler persists the order, the account manager submits it to the broker. The broker acknowledges it. Done.

The system now polls the broker independently of the event stream every 30 seconds (positions) and every 5 minutes (orders). If they diverge from internal state, it logs a warning immediately. It doesn't auto-correct — placing orders to fix a divergence you don't understand is how you make things worse — but it makes the problem visible.

def handle_info(:reconcile_position, state) do
  Process.send_after(self(), :reconcile_position, 30_000)

  case state.account_manager.module.get_current_position(state.session.id) do
    {:ok, position} when not is_nil(position) ->
      reconcile_with_position(state, position)

    {:ok, nil} ->
      reconcile_flat(state)

    {:error, error} ->
      error("Failed to reconcile position: #{inspect(error)}")
      {:noreply, state}
  end
end

But here's the thing: this reconciliation wouldn't have caught the IBKR bug. When a stop order silently fails, the broker still reports the position you expect — both sides agree. No divergence. These loops catch different failures: dropped events, stale state. Real problems, but not this one.

The order lifecycle that matters

The Order resource now has explicit lifecycle states:

attribute :status, :atom do
  constraints one_of: [
    :pre_sent,
    :pending,
    :submitted,
    :filled,
    :partially_filled,
    :cancelled,
    :rejected
  ]

  default :pre_sent
  allow_nil? false
  public? true
end

The :pre_sent state is the key addition — a signal that we've created the order locally but haven't yet confirmed the broker received it. If an order stays in :pre_sent for too long, that's a problem you can detect.

There are also explicit timestamps for each transition:

attribute :sent_at, :utc_datetime_usec do
  public? true
end

attribute :submitted_at, :utc_datetime_usec do
  public? true
end

attribute :filled_at, :utc_datetime_usec do
  public? true
end

attribute :cancelled_at, :utc_datetime_usec do
  public? true
end

Every state transition is recorded. If an order goes from :submitted to some indeterminate state, you can see exactly when it last had a confirmed status. This is the kind of telemetry you don't think you need until a stop order ghosts you.

The real lesson: you can't unit test trust

You can't unit test trust. I could verify bracket structure, stop prices, position sizing — none of that catches a broker silently dropping stops.

My code was correct. The order was submitted with the right parameters through the right API. The broker accepted it and said it was working. It just... wasn't.

The only real defense would be reconciling against ground truth: what did price actually do? Did it cross the stop level? Then why are we still holding? But I shouldn't have to build that. Cross-referencing candles against working orders is defensive engineering for a problem that shouldn't exist. The answer: stop trusting the broker.

Where this left me

API submissions fail, manual submissions work, paper trading fails the same way. There wasn't much left to investigate on my end. I looked for prior reports. They exist: the IBKR developer mailing list has threads about stops being missed via the API, orders disappearing silently. No official acknowledgment, no fix, no timeline.

That's when the question changed. It wasn't "how do I fix this bug?" It was "do I want to run a production trading system on a broker whose stop orders silently fail?" A stop order is the most basic safety mechanism in trading. If you can't trust that, you can't trust the broker — and the answer to a trust problem isn't better error handling.

Six months of work getting onto IBKR. Two weeks to decide to leave.

Next: Why I Fired My Broker (and What I Replaced Them With) — the decision to leave IBKR, what I looked for in a replacement, and what rebuilding on TradeStation actually involved.

The IBKR Saga