Exchange Host Failure and Recovery
Several types of failures can occur between the customer’s network and the exchange. Such failures can involve network devices, the CME Group price and order environments, and the exchange matching exchange. Of these failures, this section discusses three primary types:
- Exchange Host Failure
- MDP Server Failure
- MDP Price Delivery Failure
- iLink Server Failure
Exchange Host Failure
In the event that only the CME Group exchange host fails, all traders on the TT System experience the following:
- Orders: CME Group rejects all submitted orders.
- Prices: Prices go stale.
However, the TT Gateway continues to run and remains connected to the CME Group’s iLink and MDP servers.
When the exchange host again goes live, the TT Gateway receives the appropriate fix messages. Traders can again begin trading normally and prices begin updating as normal.
MDP Server Failure
When an MDP server fails in the CME Group’s environment, any Price Servers connected to it shutdown after the number of seconds set in the HeartbeatTimeout parameter of the [price_session] section in the hostinfo.cfg file. These Price Servers appear red (inactive) in Guardian. During the MDP server failure, all prices remain stale.
MDP Price Delivery Failure
This section highlights gateway behavior during price transmission failure. The CME Group provides price data along two separate yet identical feeds. The data packets are transmitted with sequence numbers to help identify missing packets. If the TT Gateway discovers a missing sequence number, the Price Server updates with correct information after receiving the next price update.
When waiting for the next price update, the CME Gateway may temporarily remove market data. This information automatically repopulates when a new price update is received.
You must verify that the price feeds are receiving data. For further information, refer to Verifying Connectivity.
iLink Server Failure
When a CME Group iLink Server fails, all Order Routers that connect to that server lose their connections. Normally, the Order Routers on the TT Gateway cycle to the backup iLink server whose connection information is included in the hostinfo.cfg file.
However, in the case that both iLink servers fail, Order Routers on the CME Gateway begin cycling by attempting to connect first to the Primary and then to the Backup IP addresses listed in the hostinfo.cfg file.
The CME Group exchange host may or may not retain orders that reached the exchange prior to the failure. Traders must call CME Group for an accurate accounting of their orders.
For details on Order Router cycling and recovery, refer to Gateway Recovery: Order Routers.
CME Group Recovery Processes
Depending upon the particular type of failure, the following items occur (reconnection does not occur in any particular order):
- If the MDP server failed, the Price Server on the CME Gateway reconnects to the CME Group exchange host. Prices go live again.
- If the iLink Server failed, the Order Routers on the CME Gateway reconnect to the CME Group exchange host. Traders can now trade.
- Fills: Unless fills were dropped by CME Group, traders have access to their fills for the entire trading day. Traders see their fills for the day re-downloading.
- Orders: Due to the possibility of orders being dropped by CME Group during an iLink server or CME Group exchange host crash, traders must call the exchange and verify their working orders.
- Position: Because the CME Group exchange host may drop
fills or orders during a failure, upon reconnection, traders may
have incorrectly calculated positions. To determine correct positions,
traders must call CME Group and inquire upon all orders in their
Additionally, in a crash situation it is possible that Cancel on Disconnect functionality can cause stale orders. Traders should inquire on their orders with the CME Group to verify the accuracy of all information.
If the Cancel on Disconnect functionality creates stale orders, they will be deleted from the Order Book if inquired in X_TRADER® using the Inq button in the Order book.
If you subscribe to CME Group’s Cancel on Disconnect functionality, recovery occurs as described in Cancel on Disconnect.
Cancel on Disconnect
CME Gateways support the exchange’s Cancel on Disconnect functionality. To enable this functionality, you must subscribe for the free service through the Globex Control Center (GCC). The exchange enables this functionality based on registered session ID and trader ID pairs.
The Cancel on Disconnect service monitors for involuntary disconnects. Once CME Group detects a disconnect, the exchange cancels all day orders except for GTDate and GTC orders.
In certain situations, some orders may not be cancelled by the exchange. Following a disconnect, customers should contact the GCC to determine their correct positions and inquire on all orders in their order book.
For more information on the CME Group’s Cancel on Disconnect functionality, refer to http://www.cmegroup.com/globex/files/CancelOnDisconnect.pdfhttp://www.cmegroup.com/globex/files/CancelOnDisconnect.pdf.