Single Gateway Failure
When a single CME Gateway fails during trading hours, updated Order Book information is not available until the CME Gateway reconnects to the exchange, and traders cannot submit orders via the CME Gateway. All orders currently in the market remain as working orders. If traders want to cancel their orders, they must call the exchange. Clients (e.g., X_TRADER® workstations) can remain connected to the CME Gateway while the gateway reconnects to the exchange.
Gateway Recovery: Fills
After the CME Gateway begins accepting fills again, X_TRADER® refreshes the Fill window by downloading the fill information stored in the *.bof file (located in <root drive>:ttdatfiles). Additionally, while the CME Gateway is down, CME Group queues any fills that occur. When the gateway reconnects, CME Group sends these fills.
Gateway Recovery: Prices
After it is restarted after a failure, the Price Server goes through the normal price feed connection process as described in Price Server Data Flow.
Gateway Recovery: Order Routers
Order Routers exist on the CME Gateway as threads in the Order Server. Thus, if an Order Router loses its connection, the Order Server restarts it.
When an Order Router thread loses its connection to CME Group, it begins the recovery process detailed below. If at anytime the Order Router regains its connection, it begins communicating with the exchange as normal and the recovery process stops:
- The Order Router attempts to reconnect to the IP address and port set in the PrimaryIP and PrimaryPort parameters
- The Order Router continues to reconnect using the primary IP address.
While Order Routers cycle, they appear in Guardian in a “feed down” state.
Gateway Recovery: Orders
When it reconnects to the exchange, the CME Gateway automatically requests all Order Book information from the exchange host. Because the exchange host queues all messages that occur while the CME Gateway is disconnected, no order or fill information is lost.
When the Order Server starts, it determines the current Order Book from the *_mode_orders.tbl file, which contains a record of active orders (i.e., working or partially filled orders). Because the Order Server records all order activity to the *_mode_orders.tbl file, the size of this file depends upon the volume of trading activity prior to the failure (i.e., the more the trading activity, the larger the file).
Because the *_mode_orders.tbl file contains a record of all orders, do not delete it! Doing so destroys the CME Gateway’s record regarding order book information. While orders remain active in the market, traders cannot access them through the CME Gateway to change or delete them.
On the CME Gateway, if the Order Server is unable to start up after recovering from the failure (i.e., if it continually restarts), use the -r command line parameter to start the Order Server.
Starting the Order Server in this manner can result in the loss of fill information.
Gateway Recovery: Hardware
After you replace any failed hardware and restart the CME Gateway, recovery processes are identical to those listed above for Gateway Software Recovery.
However, if the hard drive is replaced, traders are unable to access any of their order or fill information as the relevant log and table files are permanently lost. Traders must call the exchange and cancel their working orders. If the trader is on a position and must close it, he must do so through a broker.
Example Scenario 1 - Single CME Gateway
Quick Financial Services (QFS) has one CME Gateway. During trading hours, their gateway experiences a drive head crash, and immediately, all traders lose their connection to the exchange. The exchange displays red in Guardian, prices stop updating, and traders stop receiving fills. Even if they have X_TRADER® open, traders lose the ability to trade on CME Group. Before closing out of their workspaces, traders document their positions. Traders, who must cancel orders, call the exchange.
QFS installs and starts up another CME Gateway that same day. After startup, Jim (a trader) logs into the new CME Gateway. Jim opens up his workspaces and grids. He then begins trading at current market prices.