-
Notifications
You must be signed in to change notification settings - Fork 16
6 Sailfish Test Scripts basics
This chapter provides a brief overview of Sailfish test scripts and basic instructions about their support and update.
In simple terms, a Sailfish test script is a sequence of actions aimed either at sending a message to the system under test or at listening to the system’s response. Typically, these actions go in pairs: an inbound test message is sent to the system and the corresponding outbound message is expected from the system. In addition to these input-output action pairs, there are auxiliary SF actions aimed at introducing pauses, sleep periods and counts.
Below is a brief overview of some of the available actions.
Example:
| #Description | #action | #message_type | Side | OrderQty | Price |
|---|---|---|---|---|---|
| test case start | |||||
| Sending a 'NewOrderSingle' client-initiated message to server according to the dictionary set up in the service or according to the value of #dictionary field. | send | NewOrderSingle | BUY | 1000 | %{Price} |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'New'. | receive | ExecutionReport | ${buyside_order.Side} | ${buyside_order.OrderQty} | ${buyside_order.Price} |
| Sending a 'NewOrderSingle' client-initiated message to server according to the dictionary set up in the service or according to the value of #dictionary field. | send | NewOrderSingle | SELL | 1000 | %{Price} |
| Receiving an 'ExecutionReport' server-initiated message by server to client with status 'New'. | receive | ExecutionReport | ${sellside_order.Side} | ${sellside_order.OrderQty} | ${sellside_order.Price} |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'Filled'. | receive | ExecutionReport | ${buyside_order.Side} | ${buyside_order.OrderQty} | ${buyside_order.Price} |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'Filled'. | receive | ExecutionReport | ${sellside_order.Side} | ${sellside_order.OrderQty} | ${sellside_order.Price} |
| test case end |
The ‘Send’ action is used to send an inbound test message into the system under test.
- The message will be constructed according to the rules of the selected interface dictionary;
- The message will be routed to the server via an interface channel and on behalf of the user according to the selected Sailfish service;
- The message will contain all fields and values that were selected for this operation.
The ‘Receive’ action is used to define the parameters of an outbound system message so that it could be filtered and found among the others. This is a verification message which can have a Passed or a Failed result. To Pass this action, the outbound system message should comply with all three conditions listed below:
-
The message should be received via the service and by the user that were defined for this verification;
-
The message should be received within a defined timeout;
-
The message should contain exactly the same pairs of fields/values that were defined for this verification (However, the incoming messages may contain more optional fields than was defined in the matrix).
If any one of the above conditions was not met, the ‘Receive’ action will fail.
Actually, the ‘Receive’ action defines the parameters of an outbound system message, which is expected as a response to an inbound (‘Send’ action) message.
The ‘Retrieve’ action works the same way as the ‘Receive’ action does, except for one difference:
- For the ‘Receive’ action, Sailfish will be searching for a message with predefined parameters in the Sailfish’s memory (RAM storage; which contains only the messages that correspond to the current test case) and the search range will be limited by the action timeout and checkpoint;
- For the ‘Retrieve’ action, Sailfish will be searching for a message with predefined parameters in the Sailfish’s storage (HDD storage, which contain all the messages received by the service), and the search range is not restricted.
The ‘Count’ action works like the ‘Receive’ action, with an extra condition: to Pass the action, the number of received messages should be equal to the parameter that was defined for this verification.
There is a number of auxiliary and service actions that can be used to improve test script usability and performance. The most popular actions are:
-
GetCheckPoint - to set the time marker in memory of Sailfish where all outbound system messages for all enabled services are stored during test case execution; the marker can be further used by the ‘Receive’ action to start the limit of the range in Sailfish cache in which the expected message should be searched;
-
Sleep - to set delays – where and if needed - in milliseconds before other actions;
-
AskForContinue - is used to pause the test case. It offers the user to "stop" or "continue" test execution;
-
SetVariables - is used to define variables. It should be used with the following columns:
· #reference — variable name;
· with any other columns (tags) that contain values of the declared variable;
- SetStatic - to pass the parameters between test cases. It uses the following columns:
· #reference — static variable name;
· #static_type — variable type; Java simple types and BigDecimal, LocalDate, LocalTime, LocalDateTime are supported.
· #static_value — static variable value.
A static variable can be set by a constant (e.g. 100 or VALUE) or by reference to a previously defined ${order.field_name} field. A static variable cannot be set by reference to the #{function_name} function. A static variable can be referenced from an action using the following syntax: %{static_variable_name}
For a complete list of available actions, refer to Sailfish HELP.
Every test script consists of one or more test cases.
Every test case consists of one or more test actions.
Every test case should begin with “Test case start” and should end with “Test case end” statements in the #Action column. The rest of the test case’s content should be placed between these two operators; it completely depends on the testing scenario and may contain:
- Repeating groups;
| #description | #reference | #action | #message_type | TradingParty | Side | OrderQty | Price |
|---|---|---|---|---|---|---|---|
| Start of test case definition. Basic type of blocks. Send Limit orders: buy 1000@105 + sell 1000@105 -> trade (Test case status - Passed). | test case start | ||||||
| Repeating group | TradingParty_ref | [NoPartyIDs_ref] | |||||
| Get a checkpoint to limit the search scope to the Messages received after this checkpoint. | check1 | GetCheckPoint | |||||
| Sending a 'NewOrderSingle' client-initiated message to server according to the dictionary set up in the service or according to the value of the #dictionary field. | buyside_order | send | NewOrderSingle | [TradingParty_ref] | BUY | 1000 | 1000 |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'New'. | exr_buy | receive | ExecutionReport | [TradingParty_ref] | ${buyside_order.Side} | ${buyside_order.OrderQty} | ${buyside_order.Price} |
| End of test case definition. | test case end |
- Static Variables;
| #reference | #action | #message_type | #static_type | #static_value | SecurityID | Price |
|---|---|---|---|---|---|---|
| test case start | ||||||
| SecurityID | SetStatic | String | ABC | |||
| Price | SetStatic | BigDecimal | 1000 | |||
| buyside_order | send | NewOrderSingle | %{SecurityID} | %{Price} | ||
| exr_buy | receive | ExecutionReport | ${buyside_order.SecurityID} | ${buyside_order.Price} | ||
| sellside_order | send | NewOrderSingle | %{SecurityID} | %{Price} | ||
| exr_sell | receive | ExecutionReport | ${sellside_order.SecurityID} | ${sellside_order.Price} | ||
| test case end |
- Checkpoints;
| #description | #reference | #check_point | #action |
|---|---|---|---|
| test case start | |||
| Get a checkpoint to limit the search scope to the Messages received after this checkpoint. | check1 | GetCheckPoint | |
| Sending a 'NewOrderSingle' client-initiated message to server according to the dictionary set up in the service or according to the value of the #dictionary field. | buyside_order | send | |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'New'. | exr_buy | check1 | receive |
| Get a checkpoint to limit the search scope by only Messages received after this checkpoint. | !check2 | GetCheckPoint | |
| Sending a 'NewOrderSingle' client-initiated message to server according to the dictionary set up in the service or according to the value of the #dictionary field. | sellside_order | send | |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'New'. | exr_sell | !check2 | receive |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'Filled'. | exr_buy_f1 | !check2 | receive |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'Filled'. | exr_buy_f2 | !check2 | receive |
| test case end |
(*) – When a reference for checkpoint starts with ! - that means it is a SmartCheckpoint which automatically moves to the position after the found message upon a passed received action.
- Functions;
| #action | #message_type | TransactTime |
|---|---|---|
| test case start | ||
| send | NewOrderSingle | #{TransactTime("")} |
| receive | ExecutionReport | * |
| test case end |
- Actions;
| #description | #action |
|---|---|
| test case start | |
| Sending a 'NewOrderSingle' client-initiated message to server according to the dictionary set up in the service or according to the value of the #dictionary field. | send |
| Receiving an 'ExecutionReport' server-initiated message to client with status 'New'. | receive |
| test case end |
Repeating groups can be specified by using the following syntax:
[reference_1, reference_2]
[${reference_1}, ${reference_2}]
Static Variables can be referenced from the action’s fields using the following syntax: %{static_variable_name}.
References can be specified by using the following syntax:
- ${reference} – this is a reference to a message. This syntax is used for specifying a repeating group;
- ${reference.field} – this is a reference to a field of a message or a sub-message.
Functions can be specified by using the following syntax: #{function_name}
Sailfish Test Scripts have flexible syntax with just a few mandatory rules, which are listed below:
- “Test case start” / “Test case end” statements should be set for every test case;
- In the first row, the test script (.csv/.xls/.xlsx file) should contain a Header.
It is also recommended that the Sailfish system fields (marked with #) precede optional interface tags.
Please see the descriptions of Sailfish service fields in the test script Header below:
- #execute – an optional flag to control the execution of a current action or test case. It is possible to switch off the execution of a particular test case/cases by specifying “N”. Valid values: “Y” and “N”, blank value defaults to “Y”;
- #comment – an option field used for the comments or the scenario description. If the value of the column is 'y', then other columns may have any text. Sailfish ignores similar lines.
- #description – used to describe a row. It is displayed in the report;
- #reference – allows to refer to parameters of a certain row later in the test script;
- #service_name – the name of a service which should send or receive the message;
- #timeout – the maximum time for waiting for the expected message in milliseconds (messages are expected in the received actions); the next action will not be processed until this time is elapsed or the expected message is received;
- #check_point – defines a starting point for waiting for the expected message. Together with the #timeout, the value defines the period for waiting for the expected message;
- #action – action name (case sensitive);
- #message_type – defines the type of message from the dictionary specified in the service;
- #add_to_report – an optional flag to control the appearance of an action in the report. Valid values: “Y” and “N”, blank value defaults to “Y”.
Sailfish Test Script formulae may include the following components and their combinations:
- arithmetic operations, such as +; -; *; /;
- numbers (e.g. 1000);
- static variable such as %{static_variable_name};
- references to a previously defined fields such as ${reference.field_name} e.g. ${ord1.OrderQty};
- functions such as #{function_name}.
Note: The operation becomes calculable only having ${...}, %{...}, #{...} before them.
E.g.
{ord0.OrderQuantity}-${ord2.OrderQuantity}-${ord1.OrderQuantity}; ${ord0.OrderQuantity}-1000; ${itch_es_1.Turnover}+100000*103)/(${itch_es_1.Volume}+100000) etc.