Disaster Recovery Documentation
Single TT Gateway Scenario
When the client has only one TT Gateway, and it fails, all traders currently trading through the TT Gateway immediately lose their connections. Unless the traders have access to third-party trading software (e.g., exchange-provided front-end screens, or another ISV), they cannot trade on that exchange until the TT Gateway fully recovers.
To prevent these sorts of problems, TT recommends that you install two or more TT Gateways, each with one or more third-party devices (if possible).
For further details, refer to the section called Redundant TT Gateways.
The following list provides specific information on what happens to trading information (such as fills and orders) when a single TT Gateway fails during trading hours:
- Native Orders: Updated Order Book information is not available until the TT Gateway reconnects to the exchange. Orders cannot be submitted via the TT Gateway. Whether orders remain active in the market depends on the exchange.
- Synthetic Orders: The TT Gateway retains any synthetic orders (and their statuses) submitted before the TT Gateway failure. Additional synthetic orders cannot be submitted via the TT Gateway. Whether the working order components of synthetic orders remain active in the market depends on the exchange. Updated order information is not available until the TT Gateway connects to the exchange.
- Fills: All traders stop receiving fill information from the exchange.
- Prices: Current prices do not update.
In most cases if traders want or need to trade, they must call the exchange. Certain exchanges do not allow direct trading via phone except to cancel orders (e.g., CME). On these exchanges, traders must use another brokerage firm or trading house.
At the time of failure, TT recommends that traders document their positions because:
- This information is useful for recovery and failover purposes.
- Traders do not have access to their current position until the failed TT Gateway comes back online. They can refer to their documented positions if needed.
Gateway Software Recovery
Recovery in this scenario occurs when the failed TT Gateway is fully operable again. After a failure, and upon successful restart of a TT Gateway, the following gateway activities occur:
- Fills: The TT Gateway begins accepting fills again. X_TRADER® refreshes the Fill Window by downloading all fill information stored in the *.bof files on the TT Gateway. Generally, TT Gateways request all fill information that occurred since the last recorded sequence number. However, certain exchanges might not retain this information. In these cases, traders should reconcile their Order Book with the exchange via the phone.
- Orders: When a trader reconnects, X_TRADER® re-requests all Order Book information (working orders in market) from the TT Gateway. Certain exchanges cancel orders or put traders' orders on hold when a disconnect occurs. Refer to the appropriate disaster recovery chapter in the SAM for the specific Gateways.
- Synthetic Orders: The setting of the parameter called MustBeConnected in aconfig_local.xml determines how the TT Gateway behaves in regards to synthetic orders when recovering from a failure. However, both the TT LIFFE Gateway and TT MISS Gateway have unique processing requirements. In addition to the information presented here, refer to the appropriate SAM for behavioral exceptions.
- Prices: Prices become current after traders log in and re-subscribe to specific products by opening their grids and workspaces.
After reconnecting to the exchange, TT recommends that traders verify their current X_TRADER® position against their documented positions (see the previous section for information on documenting positions). If a discrepancy exists between the two, the trader must call the appropriate exchange to determine why the discrepancy exists.
Gateway Hardware Recovery
TT Gateways that suffer hardware failure are unable to go back online until the situation is rectified via replacement of the effected components. When failed hardware is replaced or corrected, and the TT Gateway is restarted, recovery processes are identical to those listed in the section titled Gateway Software Recovery.
However, if the hard drive is replaced, traders may be unable to access their fill information and in some cases their order information, as the relevant log and table files are permanently lost. Refer to the appropriate disaster recovery chapter in the SAM for the specific gateway.