Disconnects, Crashes, and Restarts
What Happens to Algorithm Definitions
When you need to restart an Algo SE Server, either due to a crash or as part of general maintenance, Algo SE loads all persisted algorithm definitions and makes them available to traders. All algorithm definitions that were not specified as persistent when they were deployed will not be available to traders.
The most recent X_TRADER to perform an action on an algorithm is referred to as the instance owner. For example, when a user runs an algorithm, their X_TRADER is the instance owner. If another user with whom the instance owner shares an order book on both Algo SE and the destination gateway modifies the running algorithm, the X_TRADER of this other user becomes the instance owner.
What Happens to Running Algorithms
The following scenarios explain what happens to algorithms and child orders when connectivity is lost in different situations, including when:
- Algo SE stops normally
- Algo SE crashes
- X_TRADER stops normally
- X_TRADER crashes
- Algo SE and X_TRADER lose connectivity with each other
- Algo SE loses connectivity with a TT Gateway
Scenario 1: Algo SE Stops Normally
If you stop an Algo SE Server as part of normal operations, the Algo SE Server performs the following actions for each running or paused algorithm:
- Stops all running algorithms.
- Determines which of the following actions (if any) to
take on child orders for which the corresponding TT Gateway is available:
- Submit delete requests for all orders that are children of the ASE Block.
- Submit update requests to remove the OMA indicator, i.e. the related ID field of the child order controlled by an OMA on the server, from all child orders of OMA algorithms.
- Submit update requests to remove the OMA indicator, i.e. the related ID field of the child order controlled by an OMA on the server, from all child orders of non-OMA algorithms whose Leave Order On Cancel setting is set to true.
- Submit delete requests for all child orders of non-OMA algorithms whose Leave Order On Cancel setting is set to false.
- Deletes all parent algorithms regardless of the success or failure of the deletion and/or update requests on their underlying child orders.
OMA is an abbreviation for Order Management Algorithm and is defined as any algorithm that takes a TT Order Key as an input variable. See the ADL User Guide for more information.
Scenario 2: Algo SE Crashes
If Algo SE crashes, X_TRADER detects that it has lost connectivity to Algo SE, which can take as long as 30 seconds.
If the instance owner of an algorithm was connected to Algo SE when it crashed, the instance owner’s X_TRADER deletes all non-ASE child orders if either of the following are true:
- The Client Disconnect Action parameter of the parent algorithm is set to either Pause and the child order’s Leave Orders on Pause parameter is set to false.
- The Client Disconnect Action parameter of the parent algorithm is set to either Cancel and the child order’s Leave Orders on Cancel parameter is set to false.
The instance owner’s X_TRADER will also delete all ASE child orders if the Client Disconnect Action parameter of the parent algorithm is set to either Pause or Delete.
If the instance owner of an algorithm was not connected to Algo SE when it went down, when the Algo SE Server restarts, it downloads all child orders from all TT Gateways, then performs the cleanup actions required of the server as if it were being shut down cleanly.
Scenario 3: X_TRADER Stops Normally
If a trader is the instance owner of running algorithms and tries to close X_TRADER, X_TRADER displays the following message:
If the trader chooses to exit anyway, the Algo SE Server will continue running these algorithms, but only if they do not rely on Microsoft Excel links. It will pause all algorithms that rely on Microsoft Excel links.
Scenario 4: X_TRADER Crashes
If XT crashes, Algo SE detects that it lost connectivity to X_TRADER, which can take as long as 30 seconds.
Once the Algo SE Server detects that it lost connectivity to an X_TRADER client which is the instance owner of running algorithms, Algo SE examines the value of each algorithm’s Client Disconnect Action and responds as shown in the following table.
Algo SE leaves the algorithm running.
Algo SE pauses the running algorithm, then checks the value of the Leave Orders on Pause parameter for each child order of this algorithm.
Note: It is not permissible to enable the Leave Orders on Pause setting for ASE orders. In other words, the Algo SE Server always deletes ASE child orders when the parent algorithm is paused.
Algo SE stops the running algorithm puts it into Failed state, and then checks the value of the Leave Orders on Cancel parameter for each child order of this algorithm.
Note: It is not permissible to enable the Leave Orders on Cancel parameter for ASE orders. In other words, the Algo SE Server always deletes ASE child orders when the parent algorithm is cancelled.
Algo SE always cancels running algorithms whose input parameters contain Microsoft Excel links.
Scenario 5: Connection Between X_TRADER and Algo SE Is Lost (X_TRADER and Algo SE are up)
If X_TRADER and an Algo SE Server lose connectivity with each other, it can take up 30 seconds for either to detect it.
- If Algo SE detects that it has lost connectivity to an X_TRADER client that is the instance owner of running algorithms, it performs the steps detailed in Scenario 4.
- If X_TRADER detects that it has lost connectivity to an Algo SE Server on which it has running algorithms, it will perform the steps listed in Scenario 2.
Scenario 6: Algo SE Loses Connectivity to a TT Gateway
When an Algo SE Server loses connectivity to a TT Gateway, it pauses all algorithms that use that TT Gateway, whether or not an algorithm has any currently-working orders on the Gateway. These algorithms cannot be resumed until the Algo SE Server reconnects to the TT Gateway.
Market State Changes
During the design process, an ADL developer can specify the behavior of an algorithm when one of the markets through which it is routing orders or listening for prices closes. For each algorithm, the developer can enable or disable the Ignore Market State configuration option to instruct Algo SE whether to pause the algorithm if any of the markets for which it has an instrument closes. For more information, refer to the ADL User Guide.
When Disconnections Between Algo SE and FIX Sessions Occur
When connections between FIX sessions and Algo SE are interrupted, the following actions can occur:
- If all FIX sessions are disconnected, and the Order Server shows as down in Guardian, Algo SE pauses all algorithms that use that TT Gateway, whether or not an algorithm has any currently-working orders on the Gateway. These algorithms cannot be resumed until the Algo SE Server reconnects to the TT Gateway.
- If a FIX session is disconnected, but the Order Server
remains active in Guardian, the Order Server rejects new orders
and in the following order:
- If order action is an add request, the algo pauses.
- If order action is a reject to external modification, the reject is ignored and the algo keeps running.
- If order action is a change or cancel-replace and the working quantity is greater than 0, Algo SE retries the action once and then pauses it if the retry is rejected.