Single Gateway Failure
When a single MX Gateway fails during trading hours, updated Order Book information is not available until the MX Gateway reconnects to the exchange, and traders cannot submit orders via the MX Gateway. All orders currently in the market remain as working orders. If traders want to cancel their orders, they must call the exchange.
Gateway Recovery: Prices
After it is restarted after a failure, the Price Server goes through the normal price feed connection process as described in Price Server Data Flow.
Gateway Recovery: Order Routers
Order Routers exist on the MX Gateway as threads in the Order Server. If an Order Router loses its connection, the Order Server restarts it.
When an Order Router thread loses its connection to MX, it begins the recovery process detailed below. If at anytime the Order Router regains its connection, it begins communicating with the exchange as normal and the recovery process stops:
- The Order Router attempts to reconnect to the IP address and port set in the ExchangeIP and ExchangePort parameters
- The Order Router continues the reconnect attempts until the connection is established.
While Order Routers cycle, they appear in Guardian in a “feed down” state.
Gateway Recovery: Orders
When X_TRADER® reconnects to the MX Gateway, it automatically requests all Order Book information. Since the MX host queues all messages that occur while the MX Gateway is disconnected, no order or fill information is lost. However, MX only sends order and fill information for the current session -- orders and fills from previous sessions are not sent to the MX Gateway.
When the Order Server starts, it determines the current Order Book from the *_Mode_orders.tbl files. After reading into memory the contents of these files, the Order Server deletes them and creates a new set of *_Mode_orders.tbl files. Only active orders (i.e., working or partially filled orders) are included in the new files.
Because the Order Server logs all order activity to the *_Mode_orders.tbl files, file size depends upon the volume of trading activity prior to the failure (i.e., the more the trading activity, the larger the file). Large *_Mode_orders.tbl files can slow Order Server startup by up to several minutes.
Because *_Mode_orders.tbl files contain a log of all Order Server activity, do not delete them! Doing so destroys the MX Gateway’s records regarding order book information (i.e., while orders remain active in the market, traders cannot access them through the MX Gateway to change or delete them).
Gateway Recovery: Fills
After the MX Gateway begins accepting fills again, X_TRADER® refreshes the Fill window by downloading the fill information stored in the *.bof file (located in <root drive>:ttdatfiles). Additionally, while the MX Gateway is down, MX queues any fills that occur. When the gateway reconnects, MX sends these fills.
If you lose the Order Book on the Order Server for the current session, any missed fills for those orders are reported to the Administrator. However, fills for orders entered from previous sessions (e.g., GTC orders) are not recorded by the gateway, because MX only provides order information for the current session affected by the recovery. In other words, the gateway cannot verify if orders from a previous session are filled.
Gateway Recovery: Hardware
After you replace any failed hardware and restart the MX Gateway, recovery processes are identical to those listed for Gateway Software Recovery.
However, if the hard drive is replaced, traders are unable to access any of their order or fill information as the relevant log and table files are permanently lost. Traders must call the exchange and cancel their working orders. If the trader has an open position and must close it, he must do so through a broker.
Example Scenario 1 - Single MX Gateway
Quick Financial Services (QFS) has one MX Gateway. During trading hours, their gateway experiences a drive head crash, and immediately, all traders lose their connection to the exchange. The exchange displays red in Guardian, prices stop updating, and traders stop receiving fills. Even if they have X_TRADER® open, traders lose the ability to trade on MX. Before closing out of their workspaces, traders document their positions. Traders, who must cancel orders, call the exchange.
QFS installs and starts up another MX Gateway that same day. After startup, traders log into the new MX Gateway, open up workspaces and grids, and begin trading at current market prices.