Icebox: Rewriting a Turbine OPC-UA–MQTT Bridge in Rust

This is the Icebox work on my resume at GE Vernova. Icebox is an OPC-UA to MQTT adapter that ships as part of the software customers buy with a turbine. Some of the edge applications on that site speak MQTT rather than OPC-UA, so they need a process that subscribes to live tags on the turbine and republishes them on the bus those applications already read.

The 2018 version was Java Spring Boot with a commercial OPC-UA library. I rewrote it as Icebox 2.0 because we no longer wanted to pay for that commercial library. The adapter also deploys to an air-gapped site — no internet, no on-call patch cycle — and is expected to run for years, so we needed more safety than a service you patch when it breaks. A bug you would otherwise find in the field has to fail the build.

We also had to know the new binary was cheap enough to leave on the edge box, so we measured Java Icebox and 2.0 on the same load-test configuration. A signal here is one tag Icebox is moving — one temperature, one pressure, etc. The load was about 2,000 signals to comfortably cover anything we had yet deployed to the field. Field counts were expected to start in the hundreds, so 2k was a comfortable over-test, not an idle case. On that run, classic Java Icebox sat at ~15% CPU and 310 MiB RAM. 2.0 sits at 0.45% CPU and 46 MiB, about 33× less CPU and ~7× less RAM. That comparison is Java versus the 2.0 build that actually shipped, not a claim that Rust is automatically faster.

If you have never had to leave software next to a turbine, the rest of this needs a little context. I will start with what the adapter actually does, then the design I took too long to challenge, and then how I knew later changes were real.

What the adapter actually needs to do

A site has measurements — temperatures, pressures, valve positions — that live on the turbine as tags on OPC-UA (Open Platform Communications Unified Architecture), a binary industrial protocol built around sessions, certificates, and subscriptions rather than HTTP. A client opens a session, asks to be notified when a tag changes, and gets a stream of updates.

Some edge applications on that site do not speak OPC-UA. They read MQTT (Message Queuing Telemetry Transport), a lightweight publish/subscribe bus, and Icebox is how those applications get plant data: it subscribes on OPC-UA and publishes on MQTT.

Two failure modes sit behind most of the design. If the MQTT broker is gone, Icebox cannot deliver, so it logs and waits to reconnect; there is no local buffer that keeps the plant running in the meantime. If the OPC-UA session is dead, publishing the last value you saw would leave downstream applications acting on stale data, so the adapter has to go quiet instead of looking healthy.

The tags also belong to physical gas turbines, and multiple turbines at site do not necessarily share fate. Publish rates and other settings can differ per machine, and if one set of signals is missing or sick, it must not take the others with it.

Why Rust, and why actors

Java Icebox had been running since about 2018. It was a Spring Boot service with one large object that owned the MQTT client, the OPC-UA client, and all of the signals, and it used read-write locks whenever that shared state had to change. At a hundred tags that is a reasonable program. The rewrite was driven by the paid library, not by a performance crisis: we needed a client we could ship and maintain.

We chose Rust for a small footprint on an edge box, and, more importantly for this product, for compile-time guarantees. Ownership and the type system catch use-after-transition, double-subscribe, and publish-while-offline before the binary is copied onto the air gap. This meant that, for the most part, if it compiled, it worked.

Compile-time safety does not, by itself, tell you how to structure 15k signals. Tokio is Rust’s asynchronous runtime, and sharing mutable state across Tokio tasks means Arc<Mutex<_>> or equivalent — the same shape as that Java object and its read-write locks, with everyone contending on the clients and the signal map. An actor is a task plus channels, with each piece of state on one task and everyone else sending a message, so we traded lock fights for channel design. That is a good reason to use actors, though it still leaves open how many of them you create.

I implemented 2.0. I did not invent the actor shape alone: a peer who knew that pattern better than I did advised on the design.

The grain I should have challenged

His first cut was one actor per signal. Every tag is an object, so every tag is a task that subscribes, holds a value, and publishes — very abstract, and very much how you would model this in Java. At ~100 signals it was fine. As counts climbed into the thousands, the runtime spent its life switching, and memory was the other tell: on the order of 5× the OPC-UA server that already held those tags. Around 15k signals the application choked, from scheduler cost rather than a deadlock.

We ran that design far enough to feel it. I did not yet have the judgment to reject the abstraction up front. A more experienced engineer had proposed it, it mapped cleanly onto “a signal is an object,” and I treated that as settled. Actors were still the right concurrency model; the mistake was letting object-modeling choose the grain.

What replaced it was one actor per gas turbine — a group. On the order of ~10 groups per Icebox instance, so ten event loops instead of fifteen thousand. After that, 15k signals was light work.

A group is a better boundary than one global manager that owns every tag in a map because failure domains and config are already per turbine. Different machines can have different publish rates and settings, and one machine’s tags can vanish while the others keep publishing. That is an operations requirement expressed as an actor boundary, not an argument that a single map would have been slower.

The process is then three kinds of actor. An OPC-UA connection actor owns the session and, once that session is live, hands the OPC-UA client to the turbine actors. An MQTT client actor owns the broker connection. Each turbine actor owns that machine’s signals: it uses the client it received to set up its own OPC-UA subscriptions, takes the data those subscriptions deliver, and publishes through MQTT. Two turbines are enough to see the shape; a real instance has on the order of ten.

%%{init: {"flowchart": {"useMaxWidth": false}}}%%
flowchart TB
    Conn[OPC-UA connection actor]
    Mqtt[MQTT client actor]

    subgraph TurbA[Turbine A actor]
        direction LR
        A1[signal]
        A2[signal]
        A3[signal]
    end

    subgraph TurbB[Turbine B actor]
        direction LR
        B1[signal]
        B2[signal]
        B3[signal]
    end

    Conn -->|live OPC-UA client| TurbA
    Conn -->|live OPC-UA client| TurbB
    TurbA -->|publish| Mqtt
    TurbB -->|publish| Mqtt

Each turbine subgraph is the signal set that actor owns. The arrows from the connection actor are the live client, not the tag values; the values arrive on the subscriptions the turbine set up with that client. When the OPC-UA session drops, the connection actor warns the groups, and each group decides its own offline path. The isolation idea survived from the first design; what changed was the granularity.

Actors only work if the OPC-UA client can actually be shared across them. We started on Rust bindings to open62541, a C stack, which was not a lightweight cloneable client we could hand to more than one actor. Wrapping a single C session in a mutex puts you back in the lock fight the actors were meant to avoid, so we switched to async-opcua: pure Rust, forked from an older opc-ua crate, and actively maintained for about a year after the fork when we chose it. We accepted the later-unmaintained risk. The alternatives were an un-shareable C client or the paid library we were leaving.

Publish-while-offline should not compile

Once the grain was right, the remaining class of field bug was doing the legal-looking thing at the wrong time: publishing after the session is dead, subscribing twice, or treating “TCP handshake in flight” as “we are up.” Those show up on an air-gapped box when the logs say the adapter is healthy and the values are a minute old. The type system is how we made those states unrepresentable.

Each actor is a typestate machine whose states are concrete Rust types. The event handler takes self by value and returns a transition, so illegal methods do not exist on that type. In the sketch below, connect and subscribe exist only on the disconnected type, and check_connection_status exists only on the connected type, so a connected actor cannot subscribe again. The compiler’s refusal is ordinary Rust work rather than a war story.

The actor itself is a wrapper with a run method. That method starts a CurrentState enum whose variants each hold one concrete state struct — for example Disconnected(DisconnectedState). Those structs implement a shared RunState trait. next_event is a Tokio select! over interval ticks: whichever timer fires first is the event. The run loop matches on the enum only to call run_state on whichever struct is inside; that match is dispatch, not a permission check. run_state waits for that state’s events, then consumes the struct in handle_event and returns the next CurrentState. Subscribe is legal or not because of which type you are in, not because of a branch in run. The connection actor and the group actors are the same shape, with different state structs inside.

Play the loop once before reading the sketch.

The reconnect tick is the only arrow that calls connect and subscribe, and those methods only exist on DisconnectedState. The check tick is the only arrow that calls check_connection_status, and that method only exists on ConnectedState. Two states are enough to see the idea; the real connection actor has more.

The loop looks like this — a sketch of the idea, not the site crate:

enum ActorError { /* fatal; the actor dies */ }

enum CurrentState {
    Disconnected(DisconnectedState),
    Connected(ConnectedState),
}

enum DisconnectedEvent {
    Heartbeat,
    ReconnectTimer,
}

enum ConnectedEvent {
    Heartbeat,
    ConnectionLost,
}

struct DisconnectedState {
    heartbeat: tokio::time::Interval,
    reconnect: tokio::time::Interval,
}

struct ConnectedState {
    heartbeat: tokio::time::Interval,
    check: tokio::time::Interval,
}

impl DisconnectedState {
    fn connect(&self) {
        // open the session — only this type has this method
    }

    fn subscribe(&self) {
        // set up OPC-UA subscriptions — only this type has this method
    }
}

impl ConnectedState {
    fn check_connection_status(&self) {
        // only this type has this method
    }
}

trait RunState: Sized {
    type Event;
    type Next: Into<CurrentState>;
    type Error;

    async fn next_event(&mut self) -> Result<Self::Event, Self::Error>;

    async fn handle_event(self, event: Self::Event) -> Result<Self::Next, Self::Error>;

    async fn run_state(mut self) -> Result<CurrentState, Self::Error> {
        let event = self.next_event().await?;
        self.handle_event(event).await.map(Into::into)
    }
}

impl RunState for DisconnectedState {
    type Event = DisconnectedEvent;
    type Next = CurrentState;
    type Error = ActorError;

    async fn next_event(&mut self) -> Result<Self::Event, Self::Error> {
        tokio::select! {
            _ = self.heartbeat.tick() => Ok(DisconnectedEvent::Heartbeat),
            _ = self.reconnect.tick() => Ok(DisconnectedEvent::ReconnectTimer),
        }
    }

    async fn handle_event(self, event: Self::Event) -> Result<Self::Next, Self::Error> {
        match event {
            DisconnectedEvent::Heartbeat => {
                // still disconnected; send health, stay here
                Ok(CurrentState::Disconnected(self))
            }
            DisconnectedEvent::ReconnectTimer => {
                self.connect();
                self.subscribe();
                Ok(CurrentState::Connected(ConnectedState {
                    heartbeat: tokio::time::interval(std::time::Duration::from_secs(1)),
                    check: tokio::time::interval(std::time::Duration::from_secs(15)),
                }))
            }
        }
    }
}

impl RunState for ConnectedState {
    type Event = ConnectedEvent;
    type Next = CurrentState;
    type Error = ActorError;

    async fn next_event(&mut self) -> Result<Self::Event, Self::Error> {
        tokio::select! {
            _ = self.heartbeat.tick() => Ok(ConnectedEvent::Heartbeat),
            _ = self.check.tick() => Ok(ConnectedEvent::ConnectionLost),
        }
    }

    async fn handle_event(self, event: Self::Event) -> Result<Self::Next, Self::Error> {
        match event {
            ConnectedEvent::Heartbeat => Ok(CurrentState::Connected(self)),
            ConnectedEvent::ConnectionLost => {
                self.check_connection_status();
                Ok(CurrentState::Disconnected(DisconnectedState {
                    heartbeat: tokio::time::interval(std::time::Duration::from_secs(1)),
                    reconnect: tokio::time::interval(std::time::Duration::from_secs(5)),
                }))
            }
        }
    }
}

struct ConnectionActor { /* ... */ }

impl ConnectionActor {
    async fn run(self) -> Result<(), ActorError> {
        let mut current_state = CurrentState::Disconnected(DisconnectedState {
            heartbeat: tokio::time::interval(std::time::Duration::from_secs(1)),
            reconnect: tokio::time::interval(std::time::Duration::from_secs(5)),
        });
        loop {
            current_state = match current_state {
                CurrentState::Disconnected(state) => state.run_state().await?,
                CurrentState::Connected(state) => state.run_state().await?,
            };
        }
    }
}

next_event borrows the struct and waits on whichever interval ticks first. handle_event consumes it, so the only way forward is a CurrentState variant. connect and subscribe are methods on DisconnectedState only; check_connection_status is a method on ConnectedState only. A double subscribe is a compile error rather than a runtime guard. A fatal ActorError kills the actor rather than letting it limp.

That contract is also how reconnect works, which is the path that would otherwise produce publish-while-offline. I implemented it as exponential backoff with jitter. The connection actor walks a small set of types that exist because the failure modes are different: waiting to attempt, building a client, waiting for the session, and connected. Heartbeat ticks still send health while disconnected, so the rest of the system sees that we are not up rather than silence. Building a client is separate from a live session so a failed build is not confused with a drop, and waiting for the session is separate from connected so a handshake in flight is not treated as an event loop that is already running. Loss is an explicit notification or a check timer that notices the event loop has finished.

When that actor loses the session — immediately if the socket dies, ≤15 s if the heartbeat stops — it warns the groups. A group goes silent on MQTT (last-known would be a lie), then into a setup state where subscribe is the only legal operation, then back to publishing. If the MQTT broker is the side that disappeared, Icebox logs loudly and waits to reconnect; there is no store-and-forward.

How I knew a change helped

After the scheduler was no longer the story, I still needed to know whether a later change actually helped. A flamegraph answers where the time went, not how much wall-clock CPU the process used. perf / cargo flamegraph report relative slices of whatever the process spent: a function can drop from 7% to 1% of the pie while the pie itself grew. Every iteration needed both numbers: the flamegraph, to see if the bump moved, and container CPU % and memory under the same ~2k-signal load, to see if the box got cheaper.

I wired that into a just load-test recipe: build the image, start compose, run Icebox at ~2k signals. A Python script took those parameters plus a list of git commit hashes, ran the recipe on each commit, and printed absolute and percent deltas, which is how the next three cuts got numbers instead of arguments.

A run of that script looked like this:

Commit	CPU	Δ CPU	RSS	Δ RSS
Baseline (e7c4a19)	0.84%	—	51 MiB	—
1 (12f90bb)	0.84%	0%	51 MiB	0%
2 (88a1c3e)	0.76%	−9.5%	51 MiB	0%
3 (3d0e5aa)	0.65%	−22.6%	47 MiB	−7.8%

Each row is one hash rebuilt and run at the same ~2k-signal load. Absolute CPU and RSS are what the container used; the delta columns are that row versus Baseline, so every comparison uses the same starting hash. Commit 1 did not move the box, so I would not call it an optimization even if a flamegraph shifted — that is the case the relative-only view gets wrong. Commit 2 dropped CPU and left memory alone, so it is a CPU cut: less work per flush, not a smaller working set. Commit 3 dropped both, so the box actually got cheaper. I kept a commit when those absolute columns moved the right way by more than run-to-run noise. The hashes and values here are an example of the report, not the log from the plant; the measured Java versus 2.0 endpoints are the table later in this section.

The first of those cuts was JSON. Reflective JSON — serde_json walking a value tree — sat at about 7% of runtime. Most of each signal object never changes: id, unit, range, key names. At startup, each signal’s static JSON is baked into byte slices, and the flush loop copies those slices into a reused buffer and formats only the live value. Serialization dropped to ~1%. You can emit invalid JSON that way, so tests parse the buffer back through serde_json. That was worth the complexity because a flamegraph pointed here, not because splicing is a default style.

The same profiles showed ~10% of runtime creating Tokio tracing spans. We left the instrumentation in and made those spans debug-level only, so production does not pay for them.

The last of those measured cuts was the container image. We first shipped scratch: Icebox compiled for x86_64-unknown-linux-musl, fully static, tiny, and easy to copy onto an air gap. musl is a small C library used for static Linux binaries; glibc is what most distributions ship. Load tests said the musl binary was about 15% slower than the same code in a Google distroless image linked against glibc, which is still no shell and still small. rustc built both; the libc changed.

Those cuts, on top of the group actors, are the 2.0 row below, on the same ~2k-signal job as the opening:

Build	Signals	CPU	RAM (RSS)
Classic Icebox (Java / Spring Boot)	~2k	~15%	310 MiB
Icebox 2.0 (groups, glibc, splice, debug spans)	~2k	0.45%	46 MiB
Rust, one actor per signal	~2k	~12%	~250 MiB

System tests against a field-shaped OPC-UA server

Typestate catches illegal calls at compile time. It does not prove the assembled actor does the right thing when the OPC-UA server disappears. I built a full system test environment using Docker Compose: a custom OPC-UA server, an MQTT broker, and Icebox. The mock server is built with the same crate and is shaped like the field server, including known quirks such as mislabeled fields. A mock that only speaks a clean spec would miss those.

The crate also has about 99% coverage on testable application code. Coverage tools still flag the typestate and other trait definitions as untested, because there is no instance to execute, and that warning is ignored on purpose. Each state’s impl is tested; the assembled actor is tested in compose.

The system test suite is about 20 cases, each a scenario: OPC-UA down, MQTT broker down, the OPC-UA connection fluttering up/down/up, high load, etc. The test process parses Icebox container stdout and checks that the logs show the expected actions — offline, backoff, no stale publish, resubscribe. The same stack ran in CI. Treating stdout as the oracle is brittle if logs are free text, so json structured logs were implemented using tokio-tracing’s log formatter.

Combined with typestate, that is the field story so far: zero bugs released.

The CPU and memory numbers are real, but they are not the reason the rewrite looks the way it does. I got the concurrency grain wrong, ran it until 15k signals made that obvious, and then put the actor boundary where the plant already had one. After the scheduler was fine, a commit-by-commit harness is how JSON splicing, debug-only spans, and glibc instead of musl became numbers instead of taste. The type system is why publish-while-offline never made it onto the box, and the tests are why “it compiled” is not the only reason we have not shipped a field bug.

What I learned

I learned the actor pattern by shipping it: a task plus channels, state on one owner, messages instead of locks. I also learned that the grain matters more than the pattern. Isolation survived; one task per signal did not.

Resource use is a design smell. Memory on the order of 5× the OPC-UA server and a box that choked at 15k signals were not a reason to buy a bigger edge box. They were the scheduler telling us the abstraction was wrong. Container CPU % and RSS are how you notice that before a customer does, so I keep watching them.

A justfile that can stand up the environment from scratch — build, compose, load-test — is how someone else, or I six months later, reproduces the work. If it is not in the recipe, it did not happen. That recipe also has to be replayable on an old git hash: just load-test has to be idempotent and must not depend on files that never got committed, local secrets, untracked configs, or an environment variable I always export by hand. Checkout an old hash, run the same recipe, get a comparable CPU and RSS pair. That is git hygiene as a measurement tool. The same hygiene is what lets you find a regression with git bisect or race two commits on the same load.

cargo fmt and Clippy at pedantic catch a lot of “is this idiomatic Rust?” before a reviewer does. Pedantic is annoying until it is the cheapest way to write the language you chose for compile-time safety. Pre-commit hooks ran those checks, and whatever else we required, before a push, so CI was not the first time they ran. Fail on the laptop, not after a pipeline queue.

If I did it again

I would still use actors. I would not wait until 15k signals to find out that one task per tag was too fine. A more experienced engineer proposed that grain, it mapped onto “a signal is an object,” and I treated it as settled. The next time we left a hundred-signal load, I would measure task count and scheduler cost then, while there was still room to change the boundary without first watching the box choke.

The other change is how the connection actor learns that the session is gone. We moved to async-opcua after a decent chunk of Icebox already existed, so that actor never subscribed to the events the stack will give you: session and status, channel lifecycle, and so on. We inferred loss from heartbeats, a check timer, and explicit drop notifications. Driving the actor off those events would make it more reactive, and it would have a better picture of what is actually happening to the session, instead of polling whether the event loop is still running.