$ cd liquibook $ . ./env.sh $ $MPC_ROOT/mwc.pl -type make liquibook.mwc $ make depend $ make all### Output from build * The Liquibook test and example libraries will be in $LIQUIBOOK_ROOT/lib * The Liquibook example programs will be in $LIQUIBOOK_ROOT/bin * The Liquibook test programs will be in $LIQUIBOOK_ROOT/bin/test ## Building Liquibook examples and test programs with Visual Studio Use the following commands to set up the build environment and create Visual Studio project and solution files. Note if you are using MinGW or other linux-on-Windows techniques, follow the Linux instructions; however, OCI does not normally test this.
> cd liquibook
> copy winenv.bat w.bat #optional if you want to keep the original
# note that single character batch file names are ignored in
# .getignore so the customized
# file will not be checked into the git repository (a good thing.)
> edit w.bat # edit is your choice of text editor
# follow the instructions in the file itself.
> w.bat # sets and verifies environment variables
> mpc.bat # generate the visual studio solution and project files.
Then:
* Start Visual Studio from the command line by typing liquibook.sln or
* Start Visual Studio from the Windows menu and use the menu File|Open|Project or Solution to load liquibook.sln.
## For any platform
Liquibook should work on any platform with a modern C++ compiler (supporting at least C++11.)
The MPC program used to create the build files and the Boost library used in the tests and some of the examples support a wide variety of platforms.
See the [MPC documentation](https://github.com/objectcomputing/MPC) for details about using MPC in your enviornment.
See the [Boost website](http://www.boost.org/) for details about using Boost in your environment.
================================================
FILE: README_ORDER_ENTRY.md
================================================
# Manual Order Entry -- Liquibook example program
This program accepts requests typed from the console or read from a script file and applies them to
one or more order books. It displays the results of the callbacks generated by Liquibook in
response to those requests.
## The Command Line
The mt_order_entry command accepts two command line options. Both are optional.
Parameter:
* script_file_name
The name of a script file.
* The script file contains a series of requests -- one per line. See below for the syntax of these requests.
* If you don't want a script file, but you want to specify a log file, use a single hyphen (-) for the script file name. The commands will be read from the console.
* log_file_name
The name of a file to which output should be written.
* Prompts (if any) will still be written to the console.
## Request syntax
Requests are read from the console or from a script file.
An empty line or a line beginning with a hash (#) is considered a comment and will be ignored.
Other lines read are in the format:
REQUEST [parameters]* [;]
That is: a one word request followed by parameters as necessary and terminated in a semicolon.
The semicolon may be optional for some commands, but it can always be included if desired.
The stand-alone word "END" may be substituted for a semicolon.
If a request accepts optional parameters, or if not all required parameters are provided, the
console user will be prompted for additional parameters until a semicolon is entered.
This prompting will happen even if the input is being read from a file, so the requests coming
in from a script file should be complete and should be terminated with a semicolon.
####For example:
> BUY 100 GOOG 850 AON;
ADDING order: [#2 BUY 100 GOOG $850 AON Last Event:{Submitted BUY 100 GOOG @850}]
> BUY (incomplete request)
Quantity: 100 (prompt for parameters)
Symbol: GOOG
Limit Price or MKT: 800
AON, or IOC, or STOP, or END: ;
ADDING order: [#3 BUY 100 GOOG $800 Last Event:{Submitted BUY 100 GOOG @800}]
## Requests
Most requests can be spelled out or abbreviated to their first character.
Parameters accepted by each request will be explained later in this document.
Requests are not case sensitive. Even though they are shown here in ALL CAPS
a BUY request may be entered as BUY, buy, b, or even Buy.
* BUY or B : Enter a new order to buy a security. * SELL or S : Enter a new order to sell a security. * CANCEL or C : Request that an existing order be canceled. * MODIFY or M : Request that an existing order be modified. * DISPLAY or D : Display information about existing orders * FILE or F : Open or close a script file. * ? : Display help for requests * QUIT : Exit the program.## Symbols Liquibook itself imposes no limitations on symbol. It accepts an arbitrary string of characters (actually of 8 bit octets so UTF-8 or other encodings are supported with no special effort.) The manual entry program has a few restrictions on the symbols to ease parsing: * The symbol may not include spaces. * The symbol may not begin with a plus sign (+) or exclamation mark(!) * The symbol should not be ALL. * The symbol should not be a single asterisk (*) It is important to stress that these restrictions apply only to the example program, not to Liquibook itself. Disallowing spaces obviously makes parsing the incoming request easier. Using "ALL" and "*" would confict with the DISPLAY command parsing. The restriction against leading plus signs and exclamation marks requires more explanation. Liquibook uses a separate order book for each security. When a new symbol is encountered (representing a new security) the manual entry program will create a new order book for the symbol. It needs to know whether to create a simple order book, or a depth book. * If the new symbol is prefixed by a plus sign(+), then a simple order book is created. * If it is prefixed by an exclamation mark (!) then a depth book is created. * If neither of these prefixes is present then manual entry will prompt the console user to determine what type of book to create. For example:
> buy 100 ibm 50;
New Symbol IBM.
Add [S]imple book, or [D]epth book, or 'N' to cancel request.
[SDN}: d
Create new depth order book for IBM
ADDING order: [#1 BUY 100 IBM $50 Last Event:{Submitted BUY 100 IBM @50}]
Accepted: [#1 BUY 100 IBM $50 Open: 100 Last Event:{Accepted }]
Book Change: IBM
Depth Change: IBM Changed Change Id: 1 Published: 0
BIDS:
Price 50 Count: 1 Quantity: 100 Change id#: 1
BBO Change: IBM Changed Change Id: 1 Published: 0
> buy 200 t 78;
New Symbol T.
Add [S]imple book, or [D]epth book, or 'N' to cancel request.
[SDN}: s
Create new order book for T
ADDING order: [#2 BUY 200 T $78 Last Event:{Submitted BUY 200 T @78}]
Accepted: [#2 BUY 200 T $78 Open: 200 Last Event:{Accepted }]
Book Change: T
> buy 50 r 99;
New Symbol R.
Add [S]imple book, or [D]epth book, or 'N' to cancel request.
[SDN}: n
Request ignored
Cannot process command BUY 50 R 99 ;
Since console prompting does not play well with scripting, it is important that symbols used in scripts
have the appropriate prefix the first time they appear.
It is acceptable to have a + or ! prefix on a symbol for which a book already exists. The prefix will be ignored
and the existing book will be used.
## Prices
Liquibook uses integers for prices. It does care what currency unit is actually being used as long as all orders for a
particular security use the same units.
Because Liquibook uses integers the prices must be expressed in terms of the "atomic currency unit" For example
if prices are in US Dollars, then the atomic currency unit is often a penny, so fifteen dollars would be expressed to
Liquibook as 1500.
In most cases where a price is needed, the word MARKET (or the abbreviation MKT) is accepted indicating that no limit price is specified for this order, and a trade can be generated at the price specified for the counter order, or at the current market price for the security if both sides of the trade specify MARKET price.
## Order Identity
Liquibook uses the memory address of the applications order object as an identifier for the order. [This may be changed in the future because it
imposes some odd restrictions on object lifetime.] The manual entry program needs a way for the console user (or script)
to identify orders, and the memory address is obviously not available for this purpose.
Thus the order entry program assigns an order id to each order. Whenever an order is displayed the order ID appears after a hash sign(#).
So for example if a new order is added, and the response is:
ADDING order: [#1 BUY 100 IBM $50 Last Event:{Submitted BUY 100 IBM @50}]
This order may be referred to in following commands by using the order id 1. When an order is read from the console or from a script,
an optional # can prefix the order id. Thus: "Cancel #1" and "CANCEL 1" refer to the same order.
Order ID's may also be entered with a leading minus sign. These order IDs are relative to the NEXT order id that will be used.
Thus "Cancel -1" would cancel the most recent order entered. "Cancel -2" would cancel the order before that, and so on.
## Details of specific Requests
### BUY or B
Enter a new order to buy a security.
Syntax: BUY quantity symbol price [AON | IOC | STOP price]* ;
Parameters:
* *quantity* is number of shares (or other tradable units) to buy.
* *symbol* is an arbitrary character string.
* See the **Symbol** section above for information about this string.
* *price* is the highest price the trader is willing to pay for each tradable unit of the security, or MARKET to accept any trade.
* See the **Price** section above for details about expressing prices.
* AON This optional parameter sets the All-Or-None condition for this order.
* IOC This optional parameter sets the Immediate-Or-Cancel condition for this order.
* STOP price This optional parameter sets a STOP LOSS price for this order using the same currency units as the price parameter.
The traling semicolon (or the word END) is required for a BUY request.
Zero or more trades may be triggered by entering a BUY order.
### SELL or S
Enter a new order to sell a security.
Syntax: SELL quantity symbol price [AON | IOC | STOP price]* ;
Parameters:
* *quantity* is number of shares (or other tradable units) to be sold.
* *symbol* is an arbitrary character string.
* See the **Symbol** section above for information about this string.
* *price* is the lowest price the trader is willing to accept for each tradable unit of the security, or MARKET to accept any trade.
* See the **Price** section above for details about expressing prices.
* AON This optional parameter sets the All-Or-None condition for this order.
* IOC This optional parameter sets the Immediate-Or-Cancel condition for this order.
* STOP price This optional parameter sets a STOP LOSS price for this order using the same currency units as the price parameter.
The traling semicolon (or the word END) is required for a SELL request.
Zero or more trades may be triggered by entering a SELL order.
### CANCEL or C
Request that an existing order be canceled.
Parameter:
* orderid, or #orderid, or -relative_order_id.
* See the **Order Identity** section above for details.
No trades will be triggered by entering a CANCEL request.
The traling semicolon (or the word END) is optional for a CANCEL request. The request is considered complete after one required parameter.
The request may be rejected if the order has already been filled.
### MODIFY or M
Request that an existing order be modified.
Parameters:
* orderid, or #orderid, or -relative_order_id.
* See the **Order Identity** section above for details.
* PRICE new_price
* See the **Price** section above for details about expressing prices.
* Prices may be increased or decreased.
* PRICE is optional. If it is not specified the existing price will be unchanged.
* QUANTITY new_initial_quantity
* Note this is NOT the new quantity to be open (that's a moving target). Nor is it a delta to quantity (although that's how Liquibook expresses it internally.)
* QUANTITY is optional. If it is not specified the existing quantity will be unchanged.
The traling semicolon (or the word END) is required for a MODIFY request.
Zero or more trades may be triggered by entering a MODIFY request.
The request may be rejected if the order has already been filled, or if no changes are requested.
### DISPLAY or D
Display information about existing orders
Parameters:
* * this optional first parameter (a single asterisk(*)) requests a more verbose display.
* order_id or symbol or ALL.
* One of these must appear.
* See the **Order Identity** section above for a description of the ways in which order ID can be expressed.
* A symbol will display all orders related to a specific symbol.
* The literal word ALL will display information about all known symbols.
### FILE or F
Open or close a script file.
Parameter:
* If this command appears in a script file, no parameter is allowed. This is considered to be a request to close the script file.
* filename If this command is entered from the console, the filename is the name of a script file from which future commands should com.
Commands will be read from the script file until:
* The "FILE" command appears in the script file,
* End-of-file on the script file.
* The "QUIT" command appears in the script file.
If the script file was originally opened by a FILE command entered from the console, and the script exits because of-
a FILE command or end-of-file, then control returs to the console.
If the script file was opened from the command line (see the **The Command Line** section), or the QUIT command appears in the script file, then the program exits.
### HELP or ?
Display help for requests
Displays a brief form of the information in this document.
### QUIT
Exit the program.
================================================
FILE: doc/settAug2013/liquibook_sett.html
================================================
Building a Market Data Feed with Liquibook
Introduction
The financial industry relies heavily on systems to disseminate and interpret market data. Both trading decisions and analytics are based on the current state of the market for a particular security. A typical trading exchange involves multiple parties, and in a conceptual view looks like this:
Building such an exchange is a complex undertaking, involving order taking, matching, serialization and compression to a feed, dissemination, deserialization, decompression, and interpretation. This paper will demonstrate how to build such a system using only open-source components, including Liquibook for limit order book matching, QuickFAST for compression and decompression, and Boost ASIO for distribution.
Liquibook Background
Liquibook is a C++ open source limit order book matching engine. It is a component library that you can integrate into your project for implementing matching algorithms. Liquibook includes components for an order book, aggregate depth tracking, BBO tracking, and a trade feed.
QuickFAST Background
The FAST protocol is a compression and encoding mechanism for streaming data. QuickFAST is an implementation of the FAST protocol in C++ for encoding and decoding message streams using the FAST protocol.
Boost ASIO Background
Boost ASIO is a cross-platform C++ library for asynchronous network programming.
Example System
The example project for this paper will take on a more advanced use case: simulating an exchange producing a 5-level depth feed - aggregated by price, and a trade feed. This example involves a number of steps:
Accept Orders
The example exchange generates orders, simulating the receipt of orders from traders. A true exchange takes orders from traders by means of an API - often via the FIX protocol. For the purposes of this paper, internally generated orders will suffice.
Maintain a Limit Order Book for Each Security
The example exchange must be ready to match unfilled orders against inbound orders. This is done by means of a limit order book. The limit order book keeps these orders sorted so that when multiple existing orders can match a new inbound order, the matching is performed in a sequence determined by established exchange rules. The example exchange uses Liquibook's DepthOrderBook class to maintain the limit order book.
Match Buy and Sell Orders
The example exchange must match new inbound orders against those in the limit order book. The exchange must detect a match, and fill the appropriate orders. Liquibook's order book classes naturally includes matching capability.
Update Trading Client Order Status
An exchange must always notify traders of changes in their order status. This includes accepting, filling, and rejecting orders, but also accepting or rejecting order cancellation and replacement requests. In the example exchange, there are no trading clients (the orders are randomly generated), so this step will be skipped. Liquibook does, however, provide an interface and make callbacks for changes in order status, which can easily be turned into notifications sent to trading clients.
Aggregate the Top 5 Price Levels into Depth
While a full order book feed is useful, some exchanges choose to send aggregated forms of data instead, known as depth. Rather than representing individual orders, depth models price levels as a whole. An exchange will also limit the number of levels distributed - top 5 and top 10 are common limits.
The example exchange goes through the extra step and complexity of building 5-level depth. Liquibook provides the DepthOrderBook class, which manages the order book and depth, and the DepthListener class for being notified of changes to the top few depth levels.
Determine Changes to Depth in Response to an Event
It is not enough to maintain the depth. In order to produce an incremental feed, the example exchange must also be able to determine which levels of the depth have changed. A change can happen to one or more levels in the depth, on both the buy and sell sides, in response to a single order event. Liquibook has an interface for interrogating the various levels of depth and finding out if it has changed since the last published change. This makes it trivial to build an incremental feed - sending only changed levels to feed clients.
Compress and Distributing Market Data in a Feed
In order to effectively trade, feed clients have a need to understand order flow in the exchange. Feeds in which clients are notified of all possible order detail are known as "quote" feeds or "full order book" feeds, while those which aggregate orders by price are called "depth" feeds. A "BBO" feed gives only the top-level quote for a security, and a "trade" feed reflects all trades, or matched orders. Liquibook has support for building all of these feeds.
The example exchange builds both a trade and depth feed, then compresses the trade and depth messages using QuickFAST and sends them to clients using Boost ASIO.
Handle the Feed in a Client Program
Finally, the example includes feed clients that connect to the feed, decode the updates to recreate the trade events and reproduce the market depth. The clients use QuickFAST for decoding, Boost ASIO to handle network I/O, and Liquibook to reproduce the depth.
Getting Started
If you intend to build this project, you will need to download and build:
- Boost version 1.53 or later
- Xerces version 3.1.1 or later
- QuickFAST version 1.5.0 or later
- Liquibook version 1.1.0 or later
The project is found in total within the directory
examples/depth_feed_publisher of Liquibook.
The Example Exchange
The example exchange consists of two applications, a publisher and a subscriber. The figure below illustrates the flow of data in the example exchange.
Publisher Application
The publisher implements the exchange in the example project. The publisher generates orders and adds them to the proper order book, handles the trade and depth events through listeners, encodes these using QuickFAST, and sends them using Boost ASIO.
The Order Book
As an exchange, the publisher must implement an order book. Liquibook comes
with an order book class - two actually: book::OrderBook for
managing an order book only, and a derived class
book::DepthOrderBook for adding depth aggregation to that order
book. Since the publisher builds a depth feed, it uses
book::DepthOrderBook. The relevant parts of the
book::OrderBook class are:
namespace liquibook { namespace book {
template <class OrderPtr = Order*>
class OrderBook {
public:
typedef OrderBook<OrderPtr > MyClass;
typedef TradeListener<MyClass > TypedTradeListener;
/// @brief set the trade listener
void set_trade_listener(TypedTradeListener* listener);
/// @brief add an order to book
/// @param order the order to add
/// @param conditions special conditions on the order
/// @return true if the add resulted in a fill
virtual bool add(const OrderPtr& order, OrderConditions conditions = 0);
/// @brief perform all callbacks in the queue
virtual void perform_callbacks();
};
} }
Note that the book::OrderBook class is a template class,
allowing the user to define not only the order class used in the order book,
but the style of pointer used. This can be a regular pointer or a smart
pointer.
The first method of interest allows a setting of a listener for trades. This
listener, shown later, gets notified when a trade occurs. The example
exchange needs this notification to build a trade feed, and thus sets the
trade listener. The book::OrderBook class also includes
listeners for all order status updates, for providing updates back to the
trade clients, and a listener for all changes to the order book, to build a
full order book feed.
Next is a method to add an order to the order book. add()
accepts an order pointer, and condition flags for special conditions, like
an immediate or cancel order. Note that there are also methods (not used in
the example project) to cancel an order and to replace an order.
Finally, there is a method to perform the callbacks on the order book.
Liquibook gives the client code the responsibility to execute this method,
so that it can be done in the calling thread, or a background thread.
By default, this method calls perform_callback() for each
callback in the queue. Naturally, perform_callback() can be
overridden, but that is not necessary in the publisher. The default
implementation of this method issues callbacks for to the trade listener,
order listener, and order book listener, if present.
The derived book::DepthOrderBook class is simpler:
namespace liquibook { namespace book {
/// @brief Implementation of order book child class, that incorporates
/// aggregate depth tracking. Overrides perform_callback() method to
// track depth aggregated by price.
template
class DepthOrderBook : public OrderBook {
public:
typedef Depth DepthTracker;
typedef BboListenerTypedBboListener;
typedef DepthListenerTypedDepthListener;
typedef Callback DobCallback;
/// @brief construct
DepthOrderBook();
/// @brief set the BBO listener
void set_bbo_listener(TypedBboListener* bbo_listener);
/// @brief set the depth listener
void set_depth_listener(TypedDepthListener* depth_listener);
/// @brief handle a single callback
virtual void perform_callback(DobCallback& cb);
// @brief access the depth tracker
DepthTracker& depth();
// @brief access the depth tracker
const DepthTracker& depth() const;
private:
DepthTracker depth_;
TypedBboListener* bbo_listener_;
TypedDepthListener* depth_listener_;
};
book::DepthOrderBook overrides perform_callback()
to update its depth, which is accessible through the depth()
accessor methods. In addition, book::DepthOrderBook adds two
new listeners - a BBO listener, for tracking only changes to the best bid
and best offer, and a depth listener, for tracking all depth changes.
To build the exchange, the publisher must create a
book::DepthOrderBook, and it set the trade listener and depth
listener. While handling these callbacks, the publisher will need to
update the feed clients with trade and depth update messages.
Note that in some of the callbacks, a pointer to the order book is provided, while a pointer to the order is provided in others. This gives the publisher two opportunities to provide additional fields and logic through inheritance. The publisher takes advantage of this, providing access to the symbol of the order book's security:
namespace liquibook { namespace examples {
typedef boost::shared_ptr OrderPtr;
class ExampleOrderBook : public book::DepthOrderBook<OrderPtr> {
public:
ExampleOrderBook(const std::string& symbol);
const std::string& symbol() const;
private:
std::string symbol_;
};
} } // End namespace
Here the reader will note that the ExampleOrderBook
class specializes book::DepthOrderBook and binds a shared
pointer of Order to its parent's template argument.
The Order class is shown below. The trivial
implementation of the ExampleOrderBook is omitted
from this paper. The class diagram for order books looks like this:
The Exchange
The ExampleOrderBook class manages the order book and
depth for a single security. Since exchanges handle multiple securities,
somewhere the publisher needs to create an order book for each security, and
maintain a mapping from a symbol to its order book. The publisher does this
in the Exchange class:
namespace liquibook { namespace examples {
class Exchange {
public:
Exchange(ExampleOrderBook::TypedDepthListener* depth_listener,
ExampleOrderBook::TypedTradeListener* trade_listener);
// Permanently add an order book to the exchange
void add_order_book(const std::string& symbol);
// Handle an incoming order
void add_order(const std::string& symbol, OrderPtr& order);
private:
typedef std::map<std::string, ExampleOrderBook> OrderBookMap;
OrderBookMap order_books_;
ExampleOrderBook::TypedDepthListener* depth_listener_;
ExampleOrderBook::TypedTradeListener* trade_listener_;
};
} }
The Exchange class has only three methods: a
constructor, a method to add an order book, and a method to handle an order.
As expected, it maintains a map of order books. It also maintains pointers
to the depth listener and trade listener. The constructor does exactly that,
in order to add the listeners to future order books:
namespace liquibook { namespace examples {
Exchange::Exchange(ExampleOrderBook::TypedDepthListener* depth_listener,
ExampleOrderBook::TypedTradeListener* trade_listener)
: depth_listener_(depth_listener),
trade_listener_(trade_listener)
{
}
The add_order_book() method creates a new order book, sets the
listeners on the order book, and adds a mapping from the given symbol to the
order book:
void
Exchange::add_order_book(const std::string& sym)
{
std::pair result;
result = order_books_.insert(std::make_pair(sym, ExampleOrderBook(sym)));
result.first->second.set_depth_listener(depth_listener_);
result.first->second.set_trade_listener(trade_listener_);
}
Finally, the add_order() method finds the correct order book,
adds the order to the order book, and then triggers callbacks by calling
the perform_callbacks() method:
void
Exchange::add_order(const std::string& symbol, OrderPtr& order)
{
OrderBookMap::iterator order_book = order_books_.find(symbol);
if (order_book != order_books_.end()) {
order_book->second.add(order);
order_book->second.perform_callbacks();
}
}
} } // End namespace
The call to perform_callbacks() will trigger any callbacks to
the provided listeners. This way, a single thread is manipulating the order
books. It is also possible to issue perform_callbacks() calls
in a background thread, but the publisher does it this way for simplicity.
Order Class
Liquibook requires that the application define an order class to represent
an order to match. Liquibook requires that it meet the interface defined
in the class src/book/order.h:
namespace liquibook { namespace book {
class Order {
public:
/// @brief is this a limit order?
bool is_limit() const;
/// @brief is this order a buy?
virtual bool is_buy() const = 0;
/// @brief get the price of this order, or 0 if a market order
virtual Price price() const = 0;
/// @brief get the quantity of this order
virtual Quantity order_qty() const = 0;
};
} }
The publisher defines Order to implement the
book::Order interface:
namespace liquibook { namespace examples {
class Order : public book::Order {
public:
Order(bool buy,
const double& price,
book::Quantity qty);
virtual bool is_buy() const;
virtual book::Quantity order_qty() const;
virtual book::Price price() const;
static const uint8_t precision_;
private:
bool is_buy_;
double price_;
book::Quantity qty_;
};
} }
Note that as a template class, Liquibook's book::OrderBook does
not strictly require inheriting from the book::Order class,
just that the interface be implemented.
The example's Order class implements book::Order,
and adds a static field called precision_ for price conversion.
The Order constructor simply copies the arguments passed to it:
namespace liquibook { namespace examples {
const uint8_t Order::precision_(100);
Order::Order(bool buy, const double& price, book::Quantity qty)
: is_buy_(buy),
price_(price),
qty_(qty)
{
}
Note the presence of the static initializer of precision_.
The is_buy() and order_qty() are simple accessors:
bool
Order::is_buy() const
{
return is_buy_;
}
book::Quantity
Order::order_qty() const
{
return qty_;
}
The final method is used to provide a price to the order book. For
performance reasons, the Liquibook requires this to be in integer format,
so Order converts the price by multiplying by the precision:
book::Price
Order::price() const
{
return price_ * precision_;
}
There are two implications - first, that the publisher is limited to penny precision. This is suitable for purposes of the example, but may need to be increased for other use cases. The second implication is that all securities in the publisher have the same precision - since a static variable is used to set precision. Again, in other use cases, this may not be sufficient.
QuickFAST Templates
To transmit messages using the FAST protocol, one must first decide on the messages to transmit on the feed. This exchange will produce a trade feed and an incremental depth feed. A simplified view of a trade consists of a symbol, a quantity, and a trade price. The publisher alters this slightly by providing the trade cost (price times quantity), from which price per share can be derived.
QuickFAST uses XML files, called templates to describe the messages in the
feed protocol. The example's templates can be found in the file
examples/depth_feed_publisher/depth.xml. The trade message
template looks like this:
<template name="Trade" id="1">
<uInt16 name="MessageType" id="100">
<constant value="22"/>
</uInt16>
<uInt32 name="SequenceNumber" id="200">
<increment/>
</uInt32>
<uInt32 name="Timestamp" id="300">
<copy/>
</uInt32>
<string name="Symbol" id="400">
<copy/>
</string>
<uInt32 name="Quantity" id="604">
<copy/>
</uInt32>
<uInt32 name="Cost" id="603">
<copy/>
</uInt32>
</template>
Note that the tag names within the <template> tag
indicate the type of the field. The reader may be surprised to see a
currency field (Cost) represented as an integer (uint32). This, however is
consistent with Liquibook's internal storage of prices as as integer.
Liquibook's design decision is thus carried forward to the feed protocol,
requiring the clients to convert these prices back to their decimal format.
In addition to the expected trade fields, the publisher provides a message type, to indicate to the subscriber that the message is for a trade, a sequence number, so the subscriber can be confident it has received all messages, and processed them in the correct order, and a timestamp, so the subscriber can be confident the message has been received in a timely manner.
The depth update template is more complicated:
<template name="Depth" id="2">
<uInt16 name="MessageType" id="100">
<constant value="11"/>
</uInt16>
<uInt32 name="SequenceNumber" id="200">
<increment/>
</uInt32>
<uInt32 name="Timestamp" id="300">
<copy/>
</uInt32>
<string name="Symbol" id="400">
<copy/>
</string>
<sequence name="Bids" id="500">
<uInt8 name="LevelNum" id="501">
<copy/>
</uInt8>
<uInt32 name="OrderCount" id="502">
<copy/>
</uInt32>
<uInt32 name="Price" id="503">
<copy/>
</uInt32>
<uInt32 name="AggregateQty" id="504">
<copy/>
</uInt32>
</sequence>
<sequence name="Asks" id="600">
<uInt8 name="LevelNum" id="601">
<copy/>
</uInt8>
<uInt32 name="OrderCount" id="602">
<copy/>
</uInt32>
<uInt32 name="Price" id="603">
<copy/>
</uInt32>
<uInt32 name="AggregateQty" id="604">
<copy/>
</uInt32>
</sequence>
</template>
The depth message begins with the same 4 fields as the trade message: message type, sequence number, timestamp, and symbol. The depth message also includes two sequences, or variable-length lists. These sequences represent the changed bid and ask levels of the depth for the message security. The sequences themselves contain a set of fields, although in this case both sequences are of the same type.
Each changed depth level begins with a level number, indicating which level has changed. If the exchange were to produce all depth levels on every change, this would not be necessary. Because in an incremental feed the sequence element could represent any of the 5 levels, it is required.
Order count indicates the number of orders which were aggregated at this level. If the order count is 0, it indicates a deleted level. Price is the price common to all the aggregated orders at this level. As in the trade message, it is represented by an integer. Finally, aggregate quantity shows the sum of all the order quantities at this level.
Using QuickFAST Template
With the template in place, the publisher (and subscriber) must use
QuickFAST to put the templates to use for encoding (and decoding). The
publisher (and subscriber) do this by means of the
TemplateConsumer class:
namespace liquibook { namespace examples {
class TemplateConsumer {
public:
static QuickFAST::Codecs::TemplateRegistryPtr
parse_templates(const std::string& template_filename);
// Trade field identities
static const QuickFAST::Messages::FieldIdentity id_qty_;
static const QuickFAST::Messages::FieldIdentity id_cost_;
// Common field identities
static const QuickFAST::Messages::FieldIdentity id_seq_num_;
static const QuickFAST::Messages::FieldIdentity id_msg_type_;
static const QuickFAST::Messages::FieldIdentity id_timestamp_;
static const QuickFAST::Messages::FieldIdentity id_symbol_;
// Depth field identities
static const QuickFAST::Messages::FieldIdentity id_bids_length_;
static const QuickFAST::Messages::FieldIdentity id_bids_;
static const QuickFAST::Messages::FieldIdentity id_asks_length_;
static const QuickFAST::Messages::FieldIdentity id_asks_;
static const QuickFAST::Messages::FieldIdentity id_level_num_;
static const QuickFAST::Messages::FieldIdentity id_order_count_;
static const QuickFAST::Messages::FieldIdentity id_price_;
static const QuickFAST::Messages::FieldIdentity id_size_;
};
} }
The TemplateConsumer class also has a static
parse_templates() method for parsing of templates. It also
includes several static members of type
const QuickFAST::Messages::FieldIdentityCPtr. These members
are used by QuickFAST to establish identities of fields within the various
messages. The reader will notice that the field identifiers correspond to
fields in the template definition, with two exceptions:
id_bids_length_, and id_asks_length_. This is
because the FAST protocol specifies that a sequence length is required for
each sequence, but can be implicitly declared.
These members are initialized in a manner consistent with the template definition:
namespace liquibook { namespace examples {
using namespace QuickFAST::Messages;
const FieldIdentity TemplateConsumer::id_seq_num_("SequenceNumber");
const FieldIdentity TemplateConsumer::id_msg_type_("MessageType");
const FieldIdentity TemplateConsumer::id_timestamp_("Timestamp");
// more like this...
The remaining members are initialized in a similar manner, and thus are not
included in this paper. Finally, there is the parse_templates()
method which parses the QuickFAST templates:
QuickFAST::Codecs::TemplateRegistryPtr
TemplateConsumer::parse_templates(const std::string& template_filename)
{
std::ifstream template_stream(template_filename.c_str());
QuickFAST::Codecs::XMLTemplateParser parser;
return parser.parse(template_stream);
}
} }
The parse_templates() method opens an input file stream using
the filename passed to it. Then it creates a
QuickFAST::Codecs::XMLTemplateParser and calls the
parse() method on it, passing it the input file stream. This
parses the template file, and produces the
QuickFAST::Codecs::TemplateRegistryPtr which preserves the
parsed form of the template file.
Connecting the Publisher and Subscriber
The publisher and subscriber communicate over TCP/IP using boost::ASIO. The
communication between publisher and subscriber is one way - the publisher
only sends messages, and the subscriber only receives them. The
communication is also one-to-many, as multiple subscribers connect to a
single publisher. Since QuickFAST encodes a stream of data, uniquely
compressed for each recipient, the publisher will need to maintain a
QuickFAST state for each subscriber's encoder. This is done through a class
called DepthFeedSession.
The publisher also uses the DepthFeedConnection to listen for connections from subscribers. The DepthFeedConnection will establish many DepthFeedSession instances.
namespace liquibook { namespace examples {
typedef boost::shared_ptr WorkingBufferPtr;
typedef std::deque WorkingBuffers;
typedef boost::array Buffer;
typedef boost::shared_ptr BufferPtr;
typedef boost::function MessageHandler;
typedef boost::function ResetHandler;
typedef boost::function SendHandler;
typedef boost::function RecvHandler;
class DepthFeedConnection;
// Session between a publisher and one subscriber
class DepthFeedSession : boost::noncopyable {
public:
DepthFeedSession(boost::asio::io_service& ios,
DepthFeedConnection* connection,
QuickFAST::Codecs::TemplateRegistryPtr& templates);
In addition to declaring some typedefs and a forward declaration, this
snippet of code declares DepthFeedSession's constructor. This
constructor accepts (1) an IO Service object through which it can create a
TCP socket, (2) the DepthFeedConnection and (3) the parsed
QuickFAST templates, so it can encode outgoing messages.
Next are an accessor and setter for the session's connection status, and an accessor for the session's TCP socket:
// Is this session connected?
bool connected() const { return connected_; }
// Mark this session as connected
void set_connected() { connected_ = true; }
// Get the socket for this session
boost::asio::ip::tcp::socket& socket() { return socket_; }
The real meat of the class is next, where messages are sent to the
subscriber, in send_trade() for trades, and both
send_incr_update()and send_full_update() for
depth updates.
// Send a trade messsage to all clients
void send_trade(QuickFAST::Messages::FieldSet& message);
// Send an incremental update - if this client has handled this symbol
// return true if handled
bool send_incr_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message);
// Send a full update - if the client has not yet received for this symbol
void send_full_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message);
The reader may wonder why there are two methods for sending depth updates. This is because the publisher produces an incremental feed for depth updates, allowing only the changed depth levels to be published to the subscriber, rather than all 5 levels of depth. This is particularly effective because of the nature of this data - most changes to depth are at the top one or two levels of the limit order book.
Note that the trade feed is not an incremental feed - each trade stands on its own. This could be different if the trade carried accumulating values, such as volume for the day, or average traded price. In this case, an incremental feed may be suitable for trades as well.
For the depth feed, two update methods are provided, for sending an incremental update, and for sending a full update. Even though the depth feed is an incremental feed, it is necessary to send full updates on occasion. A full update is required, for example, when the subscriber does not yet have the depth state for a security, such as when it first connects to the publisher.
The DepthFeedSession class includes a number of private members:
private:
bool connected_;
uint64_t seq_num_;
boost::asio::io_service& ios_;
boost::asio::ip::tcp::socket socket_;
The first member indicates the connection status of the session. Next, is a sequence number counter for the session. This is followed by two low-level ASIO members: a reference to the IO service, and a socket for the session.
This is followed by a pointer to the one DepthFeedConnection of
the publisher, and the session's QuickFAST encoder:
DepthFeedConnection* connection_;
QuickFAST::Codecs::Encoder encoder_;
Next, is a set to track which securities have had depth
information published, in order to track which securities still need a full
update:
typedef std::set<std::string> StringSet;
StringSet sent_symbols_;
Next are two static members that store the template IDS for the trade and depth messages:
static QuickFAST::template_id_t TID_TRADE_MESSAGE;
static QuickFAST::template_id_t TID_DEPTH_MESSAGE;
Finally, there are two private methods, to set the sequence number on a message, and to handle a sent message:
void set_sequence_num(QuickFAST::Messages::FieldSet& message);
void on_send(WorkingBufferPtr wb,
const boost::system::error_code& error,
std::size_t bytes_transferred);
};
The DepthFeedSession implementation begins with the declarations
of its two static members, TID_TRADE_MESSAGE and
TID_DEPTH_MESSAGE.
using namespace boost::asio::ip;
namespace liquibook { namespace examples {
QuickFAST::template_id_t DepthFeedSession::TID_TRADE_MESSAGE(1);
QuickFAST::template_id_t DepthFeedSession::TID_DEPTH_MESSAGE(2);
Next is the class constructor, which sets the connected status to false, sets the session sequence number to 0, initializes the socket, saves the IO service and connection, and provides the templates to the QuickFAST encoder:
DepthFeedSession::DepthFeedSession(
boost::asio::io_service& ios,
DepthFeedConnection* connection,
QuickFAST::Codecs::TemplateRegistryPtr& templates)
: connected_(false),
seq_num_(0),
ios_(ios),
socket_(ios),
connection_(connection),
encoder_(templates)
{
}
The next method sends a trade to the subscriber:
void
DepthFeedSession::send_trade(QuickFAST::Messages::FieldSet& message)
{
// Add or update sequence number in message
set_sequence_num(message);
std::cout && "sending trade message with " && message.size() && " fields" && std::endl;
QuickFAST::Codecs::DataDestination dest;
encoder_.encodeMessage(dest, TID_TRADE_MESSAGE, message);
WorkingBufferPtr wb = connection_->reserve_send_buffer();
dest.toWorkingBuffer(*wb);
// Perform the send
SendHandler send_handler = boost::bind(&DepthFeedSession::on_send,
this, wb, _1, _2);
boost::asio::const_buffers_1 buffer(
boost::asio::buffer(wb->begin(), wb->size()));
socket_.async_send(buffer, 0, send_handler);
}
The send_trade() accepts an argument, reference to type
QuickFAST::Messages::FieldSet, which contains the method in a
protocol-neutral format. The method starts by setting the sequence number
of the message by passing the field set to set_sequence_num().
The caller is responsible for filling out the other fields of the message,
while the session sets the sequence number.
Next, the method declares a data destination variable, which holds an encoded message. This message is then encoded into the destination by the QuickFAST encoder, using the template ID for trade messages.
Next a QuickFAST working buffer is reserved for sending the message, and the
encoded message is serialized into the buffer. To send the message, the
method first creates a send handler for the buffer, so that
DepthFeedSession::on_send is called after the send. Next, it
sets up an ASIO buffer to send the working buffer's contents, and calls the
socket's async_send() method.
The send_incr_update method is similar, with some key
differences:
bool
DepthFeedSession::send_incr_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message)
{
bool sent = false;
// If the session has been started for this symbol
if (sent_symbols_.find(symbol) != sent_symbols_.end()) {
QuickFAST::Codecs::DataDestination dest;
// Add or update sequence number in message
set_sequence_num(message);
encoder_.encodeMessage(dest, TID_DEPTH_MESSAGE, message);
WorkingBufferPtr wb = connection_->reserve_send_buffer();
dest.toWorkingBuffer(*wb);
SendHandler send_handler = boost::bind(&DepthFeedSession::on_send,
this, wb, _1, _2);
boost::asio::const_buffers_1 buffer(
boost::asio::buffer(wb->begin(), wb->size()));
socket_.async_send(buffer, 0, send_handler);
sent = true;
}
return sent;
}
First, the reader will notice that send_incr_update() returns a
bool value. This value indicates whether the message was sent
or not. If not, the caller is responsible for calling
send_full_update() with the message.
The method also checks the sent_symbols_ for the symbol of the
message (provided as an argument) to ensure the security has been sent to the
subscriber previously. Should this be true, execution is similar to that of
send_trade(), where the sequence number is set, the message is
encoded to a data destination, a working buffer is reserved, the destination
serialized to the working buffer, and the working buffer contents is sent.
send_full_update() method is similar to
send_incr_update, without the return value. In this case, the
method updates the sent_symbols_ map to be true. As a form of
protection, it does not send the full update if the map already had a value
for that security.
void
DepthFeedSession::send_full_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message)
{
// Mark this symbols as sent
std::pair result = sent_symbols_.insert(symbol);
// If this symbol is new for the session
if (result.second) {
QuickFAST::Codecs::DataDestination dest;
// Add or update sequence number in message
set_sequence_num(message);
encoder_.encodeMessage(dest, TID_DEPTH_MESSAGE, message);
WorkingBufferPtr wb = connection_->reserve_send_buffer();
dest.toWorkingBuffer(*wb);
// Perform the send
SendHandler send_handler = boost::bind(&DepthFeedSession::on_send,
this, wb, _1, _2);
boost::asio::const_buffers_1 buffer(
boost::asio::buffer(wb->begin(), wb->size()));
socket_.async_send(buffer, 0, send_handler);
}
}
The first private method, set_sequence_num() is used to set
the sequence number field in the outgoing message. The sequence number is
unique to each session, so it must be encoded for each copy of the outbound
message:
void
DepthFeedSession::set_sequence_num(QuickFAST::Messages::FieldSet& message)
{
// Create the field
QuickFAST::Messages::FieldCPtr value =
QuickFAST::Messages::FieldUInt32::create(++seq_num_);
// Update the sequence number
if (!message.replaceField(TemplateConsumer::id_seq_num_, value)) {
// Not found, add the sequence number
message.addField(TemplateConsumer::id_seq_num_, value);
}
}
The method begins by creating a QUICKFAST field for the incremented sequence
number. Since this code resides within a session, there is a copy of the
member variable seq_num_ for each subscriber. The first message
will have a sequence number of 1.
The method attempts to update the sequence number field in the message, and if the sequence number is not yet present in the message, the field is added. Using this technique, the message can be built once, and then have the sequence number customized for each subscriber, without having to build the message from scratch for each subscriber.
The other private method, on_send() handles a sent message,
logging any error code, and calling the DepthFeedConnection
version of the method:
void
DepthFeedSession::on_send(WorkingBufferPtr wb,
const boost::system::error_code& error,
std::size_t bytes_transferred)
{
if (error) {
std::cout << "Error " << error << " sending message" << std::endl;
connected_ = false;
}
// Keep buffer for later
connection_->on_send(wb, error, bytes_transferred);
}
The DepthFeedConnection class is used by both the publisher and
the subscriber. It has a constructor that accepts the command line
arguments:
typedef boost::shared_ptrSessionPtr; class DepthFeedConnection : boost::noncopyable { public: DepthFeedConnection(int argc, const char* argv[]);
Next are an accessor for the templates parsed by the connection, and two
methods for connecting the publisher and subscribers. The publisher calls
accept() and the subscriber calls connect():
// Get the template registry
const QuickFAST::Codecs::TemplateRegistryPtr&
get_templates() { return templates_; }
// Connect to publisher
void connect();
// Accept connection from subscriber
void accept();
The next method, run() issues all the ASIO callbacks. Both the
publisher and subscriber must cal run().
// Let the IO service run
void run();
The next two methods set up handlers for events:
// Set a callback to handle a message
void set_message_handler(MessageHandler msg_handler);
// Set a callback to handle a reset connection
void set_reset_handler(ResetHandler reset_handler);
The first, set_message_handler(), handles incoming messages in
the subscriber. The second, set_reset_handler() within the
publisher handles disconnects within the subscriber to the publisher.
The next two methods are for buffer management:
// Reserve a buffer for receiving a message
BufferPtr reserve_recv_buffer();
// Reserve a buffer for sending a message
WorkingBufferPtr reserve_send_buffer();
Rather than constantly allocating and deallocating buffers, the publisher
and subscriber have a pool of buffers ready to be used again and again.
The publisher and subscriber will reserve these buffers from
DepthFeedConnection, bind them to an ASIO asynchronous callback,
and then restore the buffers back to the DepthFeedConnection.
The first, reserve_recv_buffer() is called by the subscriber for
a buffer to receive a message. Likewise, reserve_send_buffer()
reserves a buffer to serialize an encoded QuickFAST message, for the
publisher to send to the subscriber.
Next, are three methods for the publisher to send a message to each subscriber:
// Send a trade messsage to all clients
void send_trade(QuickFAST::Messages::FieldSet& message);
// Send an incremental update
// return true if all sessions could handle an incremental update
bool send_incr_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message);
// Send a full update to those which have not yet received for this symbol
void send_full_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message);
These three methods call the corresponding method on
DepthFeedSession for each subscriber. The public interface ends
with a number of event handlers:
// Handle a connection
void on_connect(const boost::system::error_code& error);
// Handle an accepted connection
void on_accept(SessionPtr session,
const boost::system::error_code& error);
// Handle a received message
void on_receive(BufferPtr bp,
const boost::system::error_code& error,
std::size_t bytes_transferred);
// Handle a sent message
void on_send(WorkingBufferPtr wb,
const boost::system::error_code& error,
std::size_t bytes_transferred);
These methods handle, the result of a subscriber connecting to the publisher, a publisher's accepted connection from a subscriber, a message received from the publisher, and a message sent to the subscriber, respectively.
The private members of DepthFeedConnection start with some
configuration date: the name of the file containing the FAST templates, the
host for the subscriber to connect to, and the connection port. These are
followed by the event handlers, the parsed QuickFAST templates, buffer pools,
the DepthFeedSession instances, and some boost ASIO helpers.
The private section ends with some methods, to issue a read, and parse the command line arguments:
private:
typedef std::deque<BufferPtr> Buffers;
typedef std::vector<SessionPtr> Sessions;
const char* template_filename_;
const char* host_;
int port_;
MessageHandler msg_handler_;
ResetHandler reset_handler_;
QuickFAST::Codecs::TemplateRegistryPtr templates_;
Buffers unused_recv_buffers_;
WorkingBuffers unused_send_buffers_;
Sessions sessions_;
boost::shared_ptr<boost::asio::ip::tcp::acceptor> acceptor_;
boost::asio::io_service ios_;
boost::asio::ip::tcp::socket socket_;
boost::shared_ptr<boost::asio::io_service::work> work_ptr_;
void issue_read();
static const char* template_file_from_args(int argc, const char* argv[]);
static const char* host_from_args(int argc, const char* argv[]);
static int port_from_args(int argc, const char* argv[]);
};
} } // End namespace
The DepthFeedConnection constructor derives the configuration
variables by passing the command-line arguments to the
*_from_args() methods:
DepthFeedConnection::DepthFeedConnection(int argc, const char* argv[])
: template_filename_(template_file_from_args(argc, argv)),
host_(host_from_args(argc, argv)),
port_(port_from_args(argc, argv)),
templates_(TemplateConsumer::parse_templates(template_filename_)),
socket_(ios_)
{
}
It also invokes parse_templates() to parse the QuickFAST
templates, and initializes the subscriber socket. Next,
connect() does a standard ASIO asynchronous connection, using
on_connect() as its callback:
void
DepthFeedConnection::connect()
{
std::cout << "Connecting to feed" << std::endl;
tcp::endpoint endpoint(address::from_string(host_), port_);
socket_.async_connect(endpoint, boost::bind(&DepthFeedConnection::on_connect,
this, _1));
}
The accept() method implements the publisher's side of ASIO
connectivity. The first time accept() id called a TCP acceptor
is created and initialized. For each call, a DepthFeedSession
is created, and an asynchronous accept is started, using to
on_accept() as its callback:
void
DepthFeedConnection::accept()
{
if (!acceptor_) {
acceptor_.reset(new tcp::acceptor(ios_));
tcp::endpoint endpoint(tcp::v4(), port_);
acceptor_->open(endpoint.protocol());
boost::system::error_code ec;
acceptor_->set_option(boost::asio::socket_base::reuse_address(true), ec);
acceptor_->bind(endpoint);
acceptor_->listen();
}
SessionPtr session(new DepthFeedSession(ios_, this, templates_));
acceptor_->async_accept(
session->socket(),
boost::bind(&DepthFeedConnection::on_accept, this, session, _1));
}
The run() method invokes the run() method on the IO
service object. Usually run() will exit when there are no more
callbacks to make. Using the io_service::work object ensures
run() does not exit:
void
DepthFeedConnection::run()
{
std::cout << "DepthFeedConnection::run()" << std::endl;
// Keep on running
work_ptr_.reset(new boost::asio::io_service::work(ios_));
ios_.run();
}
The next two methods simply save off the two message handlers:
void
DepthFeedConnection::set_message_handler(MessageHandler handler)
{
msg_handler_ = handler;
}
void
DepthFeedConnection::set_reset_handler(ResetHandler handler)
{
reset_handler_ = handler;
}
As stated above, the DepthFeedConnection uses buffer pools to
reduce heap allocations and deallocations. The following two methods
provide unused buffers to the caller:
BufferPtr
DepthFeedConnection::reserve_recv_buffer()
{
if (unused_recv_buffers_.empty()) {
return BufferPtr(new Buffer());
} else {
BufferPtr bp = unused_recv_buffers_.front();
unused_recv_buffers_.pop_front();
return bp;
}
}
WorkingBufferPtr
DepthFeedConnection::reserve_send_buffer()
{
if (unused_send_buffers_.empty()) {
return WorkingBufferPtr(new QuickFAST::WorkingBuffer());
} else {
WorkingBufferPtr wb = unused_send_buffers_.front();
unused_send_buffers_.pop_front();
return wb;
}
}
In both methods, an unused buffer is returned, and one is created if none are available.
The send_trade(), send_incr_update() and
send_full_update() methods send a message to all subscribers by
iterating through the sessions and invoking the corresponding method on each:
void
DepthFeedConnection::send_trade(QuickFAST::Messages::FieldSet& message)
{
// For each session
Sessions::iterator session;
for (session = sessions_.begin(); session != sessions_.end(); ) {
// If the session is connected
if ((*session)->connected()) {
// conditionally send on that session
(*session)->send_trade(message);
++session;
} else {
// Remove the session
session = sessions_.erase(session);
}
}
}
bool
DepthFeedConnection::send_incr_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message)
{
bool none_new = true;
// For each session
Sessions::iterator session;
for (session = sessions_.begin(); session != sessions_.end(); ) {
// If the session is connected
if ((*session)->connected()) {
// send on that session
if (!(*session)->send_incr_update(symbol, message)) {
none_new = false;
}
++session;
} else {
// Remove the session
session = sessions_.erase(session);
}
}
return none_new;
}
void
DepthFeedConnection::send_full_update(const std::string& symbol,
QuickFAST::Messages::FieldSet& message)
{
// For each session
Sessions::iterator session;
for (session = sessions_.begin(); session != sessions_.end(); ) {
// If the session is connected
if ((*session)->connected()) {
// conditionally send on that session
(*session)->send_full_update(symbol, message);
++session;
} else {
// Remove the session
session = sessions_.erase(session);
}
}
}
In any of the cases, if the session is no longer connected, the session
is removed. There is one important difference in
send_incr_update() - the method's return value indicates if any
of the sessions gave back a return value of false.
The next method handles a connection within the subscriber:
void
DepthFeedConnection::on_connect(const boost::system::error_code& error)
{
if (!error) {
std::cout << "connected to feed" << std::endl;
reset_handler_();
issue_read();
} else {
std::cout << "on_connect, error=" << error << std::endl;
socket_.close();
sleep(3);
// Try again
connect();
}
}
When on_connect() is called, the method tries again, in case of
failure, or calls the reset handler and then begins to read on the socket, in
case of success.
The next handler is the counterpart in the publisher:
void
DepthFeedConnection::on_accept(SessionPtr session,
const boost::system::error_code& error)
{
if (!error) {
std::cout << "accepted client connection" << std::endl;
sessions_.push_back(session);
session->set_connected();
} else {
std::cout << "on_accept, error=" << error << std::endl;
session.reset();
sleep(2);
}
// accept again
accept();
}
In the case of on_accept(), the publisher always restarts the
accept cycle - so that another subscriber may connect. The subscriber, on
the other hand, only wants one connection.
In the case of a successful connection, the publisher saves off the session, and marks it as connected, so that messages may be sent to it. If there is failure, the session is deleted (by resetting the shared pointer) and the publisher waits momentarily before starting again.
The next event handler handles a message received by the subscriber:
void
DepthFeedConnection::on_receive(BufferPtr bp,
const boost::system::error_code& error,
std::size_t bytes_transferred)
{
if (!error) {
// Next read
issue_read();
// Handle the buffer
if (!msg_handler_(bp, bytes_transferred)) {
socket_.close();
}
} else {
std::cout << "Error " << error << " receiving message" << std::endl;
socket_.close();
sleep(3);
connect();
}
// Restore buffer
unused_recv_buffers_.push_back(bp);
}
When the receive is handled, if the read was successful, the subscriber first issues a new read, and then calls the receive handler, which parses the message. Should this parsing fail, the socket will close (and the next read will then fail).
Should the read fail, the socket is closed, and the connection cycle starts anew. In all cases, the buffer is restored back to the pool.
The next handler is called after a send of a message to the subscriber:
void
DepthFeedConnection::on_send(WorkingBufferPtr wb,
const boost::system::error_code& error,
std::size_t bytes_transferred)
{
// Keep buffer for later
unused_send_buffers_.push_back(wb);
}
Recall that this method is called from within
DepthFeedSession::on_send() which has already examined the
error code. What remains is to restore the buffer back to the pool.
The next method starts an ASIO read on the subscriber's socket:
void
DepthFeedConnection::issue_read()
{
BufferPtr bp = reserve_recv_buffer();
RecvHandler recv_handler = boost::bind(&DepthFeedConnection::on_receive,
this, bp, _1, _2);
boost::asio::mutable_buffers_1 buffer(
boost::asio::buffer(*bp, bp->size()));
socket_.async_receive(buffer, 0, recv_handler);
}
issue_read() reserves a buffer, binds the buffer to the
on_receive() method, and issues the asynchronous read.
The next three methods look through the command line arguments for configuration variables:
const char*
DepthFeedConnection::template_file_from_args(int argc, const char* argv[])
{
bool next_is_name = false;
for (int i = 0; i < argc; ++i) {
if (next_is_name) {
return argv[i];
} else if (strcmp(argv[i], "-t") == 0) {
next_is_name = true;
}
}
return "./templates/depth.xml";
}
const char*
DepthFeedConnection::host_from_args(int argc, const char* argv[])
{
bool next_is_host = false;
for (int i = 0; i < argc; ++i) {
if (next_is_host) {
return argv[i];
} else if (strcmp(argv[i], "-h") == 0) {
next_is_host = true;
}
}
return "127.0.0.1";
}
int
DepthFeedConnection::port_from_args(int argc, const char* argv[])
{
bool next_is_port = false;
for (int i = 0; i < argc; ++i) {
if (next_is_port) {
return atoi(argv[i]);
} else if (strcmp(argv[i], "-p") == 0) {
next_is_port = true;
}
}
return 10003;
}
} } // End namespace
The first method looks for a -t followed by a template file name. The second looks foe a -h followed by a host name. The last one looks for a -p followed by a port number. All three return default values.
Handling Liquibook Events
At this point, the publisher is able to accept subscriber connections and
route QuickFAST messages to the subscriber. What is missing is handling
Liquibook events, and creating the QuickFAST messages to send. As shown in
figure 2, the publisher needs a properly-typed
TradeListener for trade events, and a properly-typed
DepthListener for depth events.
The TradeListener interface has a single method to implement:
namespace liquibook { namespace book {
/// @brief listener of trade events. Implement to build a trade feed.
template <class OrderBook >
class TradeListener {
public:
/// @brief callback for a trade
/// @param book the order book of the fill (not defined whether this is before
/// or after fill)
/// @param qty the quantity of this fill
/// @param cost the cost of this fill (qty * price)
virtual void on_trade(const OrderBook* book,
Quantity qty,
Cost cost) = 0;
};
} }
Likewise, the DepthListener interface has a single method to
implement:
namespace liquibook { namespace book {
/// @brief listener of depth events. Implement to build an aggregate depth
/// feed.
template <class OrderBook >
class DepthListener {
public:
/// @brief callback for change in tracked aggregated depth
virtual void on_depth_change(
const OrderBook* book,
const typename OrderBook::DepthTracker* depth) = 0;
};
} }
The publisher implements these interfaces in the
DepthFeedPublisher class:
namespace liquibook { namespace examples {
class DepthFeedPublisher : public ExampleOrderBook::TypedDepthListener,
public ExampleOrderBook::TypedTradeListener,
public TemplateConsumer {
The DepthFeedPublisher class implements both the
ExampleOrderBook::TypedDepthListener and the
ExampleOrderBook::TypedTradeListener interfaces, which bind the
book::DepthListener and the book::TradeListener
templates to those used in ExampleOrderBook.
DepthFeedPublisher also inherits from
TemplateConsumer, but this is simply a convenience (to reduce
the need for name qualifiers), as TemplateConsumer's data and
methods are entirely static.
The DepthFeedPublisher has a simple constructor, for
initializing its pointer.
public: DepthFeedPublisher();
The DepthFeedPublisher keeps a pointer to a
DepthFeedConnection, and receives that pointer through a setter
method:
void set_connection(DepthFeedConnection* connection);
Next are the two event handlers to implement the two interfaces:
virtual void on_trade(
const book::OrderBook<OrderPtr>* order_book,
book::Quantity qty,
book::Cost cost);
virtual void on_depth_change(
const book::DepthOrderBook<OrderPtr>* order_book,
const book::DepthOrderBook<OrderPtr>::DepthTracker* tracker);
The private section has a single member - the
DepthFeedConnection pointer:
private: DepthFeedConnection* connection_;
Next are methods to build the feed messages:
// Build an trade message
void build_trade_message(
QuickFAST::Messages::FieldSet& message,
const std::string& symbol,
book::Quantity qty,
book::Cost cost);
// Build an incremental depth message
void build_depth_message(
QuickFAST::Messages::FieldSet& message,
const std::string& symbol,
const book::DepthOrderBook<OrderPtr>::DepthTracker* tracker,
bool full_message);
void build_depth_level(
QuickFAST::Messages::SequencePtr& level_seq,
const book::DepthLevel* level,
int level_index);
And finally a method to generate a time stamp for the messages:
uint32_t time_stamp(); }; } } // End namespace
The DepthFeedPublisher constructor initializes its connection
pointer, which is set by a separate setter method:
namespace liquibook { namespace examples {
using namespace QuickFAST::Messages;
DepthFeedPublisher::DepthFeedPublisher()
: connection_(NULL)
{
}
void
DepthFeedPublisher::set_connection(DepthFeedConnection* connection)
{
connection_ = connection;
}
The DepthFeedPublisher's primary responsibility is to build
feed messages, in response to events. The first event handler is
on_trade():
void
DepthFeedPublisher::on_trade(
const book::OrderBook<OrderPtr>* order_book,
book::Quantity qty,
book::Cost cost)
{
// Publish trade
QuickFAST::Messages::FieldSet message(20);
const ExampleOrderBook* exob =
dynamic_cast<const ExampleOrderBook*>(order_book);
std::cout << "Got trade for " << exob->symbol()
<< " qty " << qty
<< " cost " << cost << std::endl;
build_trade_message(message, exob->symbol(), qty, cost);
connection_->send_trade(message);
}
on_trade() starts by casting the order book to the derived type.
The reader will recall that the derived order book is used to store the
symbol of the order book. The method then builds the trade message,
through a call to build_trade_message(), and distributes the
message through the DepthFeedConnection.
The next event handler handles depth change events:
void
DepthFeedPublisher::on_depth_change(
const book::DepthOrderBook<OrderPtr>* order_book,
const book::DepthOrderBook<OrderPtr>::DepthTracker* tracker)
{
// Publish changed levels of order book
QuickFAST::Messages::FieldSet message(20);
const ExampleOrderBook* exob =
dynamic_cast<const ExampleOrderBook*>(order_book);
build_depth_message(message, exob->symbol(), tracker, false);
if (!connection_->send_incr_update(exob->symbol(), message)) {
// Publish all levels of order book
QuickFAST::Messages::FieldSet full_message(20);
build_depth_message(full_message, exob->symbol(), tracker, true);
connection_->send_full_update(exob->symbol(), full_message);
}
}
Similar to the trade event handler, the depth change event handler casts the
order book to the derived class, builds a feed message, and sends it to the
clients through the DepthFeedConnection. What is different in
on_depth_change() is that since the depth feed is incremental,
as noted above, the clients must be known to be in a valid state for the
security of the update to handle the next message.
The reader will recall that the call to
DepthFeedConnection::send_incr_update() returns true if and only
if all of the clients could handle the message. If not, false is returned,
and on_depth_change() builds and sends a full update instead.
In the steady state, only the incremental message need by built.
The build_trade_message() method is used by
on_trade() to build a trade message:
void
DepthFeedPublisher::build_trade_message(
QuickFAST::Messages::FieldSet& message,
const std::string& symbol,
book::Quantity qty,
book::Cost cost)
{
message.addField(id_timestamp_, FieldUInt32::create(time_stamp()));
message.addField(id_symbol_, FieldString::create(symbol));
message.addField(id_qty_, FieldUInt32::create(qty));
message.addField(id_cost_, FieldUInt32::create(cost));
}
Building a QuickFAST message is relatively simple. Each field is added to
the message through the addField() call. The sequence number,
added later, is particular to a specific client.
The next message builder, build_depth_message(), builds
repeating sequences of data, and is more complex:
void
DepthFeedPublisher::build_depth_message(
QuickFAST::Messages::FieldSet& message,
const std::string& symbol,
const book::DepthOrderBook<OrderPtr>::DepthTracker* tracker,
bool full_message)
{
size_t bid_count(0), ask_count(0);
message.addField(id_timestamp_, FieldUInt32::create(time_stamp()));
message.addField(id_symbol_, FieldString::create(symbol));
// Build the changed levels
book::ChangeId last_published_change = tracker->last_published_change();
// Build changed bids
{
SequencePtr bids(new Sequence(id_bids_length_, 1));
int index = 0;
const book::DepthLevel* bid = tracker->bids();
// Create sequence of bids
while (true) {
if (full_message || bid->changed_since(last_published_change)) {
build_depth_level(bids, bid, index);
++bid_count;
}
++index;
if (bid == tracker->last_bid_level()) {
break;
} else {
++bid;
}
}
message.addField(id_bids_, FieldSequence::create(bids));
}
// Build changed asks
{
SequencePtr asks(new Sequence(id_asks_length_, 1));
int index = 0;
const book::DepthLevel* ask = tracker->asks();
// Create sequence of asks
while (true) {
if (full_message || ask->changed_since(last_published_change)) {
build_depth_level(asks, ask, index);
++ask_count;
}
++index;
if (ask == tracker->last_ask_level()) {
break;
} else {
++ask;
}
}
message.addField(id_asks_, FieldSequence::create(asks));
}
std::cout << "Encoding " << (full_message ? "full" : "incr")
<< " depth message for symbol " << symbol
<< " with " << bid_count << " bids, "
<< ask_count << " asks" << std::endl;
}
This method starts like add_trade(), adding the timestamp and
symbol fields. Next, two sequences are built, for the changed bid and ask
levels. To support an incremental feed, Liquibook tracks a change number
(called ChangeId) on each depth level, and the ID of the last
published change in the depth, which is set after the callback completes.
To determine the levels to publish, the publisher needs to compare the last
change of each level to the last published change of the depth as a whole.
In build_depth_message(), a bid sequence is created, and an
associated length is attached. The changed bid levels are each added to the
sequence through calls to build_depth_level(). The iteration
logic here must be careful to not move beyond the last bid level. Finally,
the sequence is added to the message. The same is done for changed ask
levels.
By comparison, build_depth_level() is simple:
void
DepthFeedPublisher::build_depth_level(
QuickFAST::Messages::SequencePtr& level_seq,
const book::DepthLevel* level,
int level_index)
{
FieldSetPtr level_fields(new FieldSet(4));
level_fields->addField(id_level_num_, FieldUInt8::create(level_index));
level_fields->addField(id_order_count_,
FieldUInt32::create(level->order_count()));
level_fields->addField(id_price_,
FieldUInt32::create(level->price()));
level_fields->addField(id_size_,
FieldUInt32::create(level->aggregate_qty()));
level_seq->addEntry(level_fields);
}
This method shows how to add an entry to a QuickFAST sequence - by creating
a FieldSet, adding fields to it, and adding the
FieldSet to the sequence.
The final method is a helper, to create an integer timestamp for a message:
uint32_t
DepthFeedPublisher::time_stamp()
{
time_t now;
time(&now);
return now;
}
} } // End namespace
Publisher Initialization
At this point the publisher has all the necessary parts to publish a trade feed and an incremental depth feed for the exchange. The only things that remains are to create, initialize, and associate the various parts, and to generate the random orders.
The exchange is initialized in the file publisher_main.cpp.
The main() function starts by establishing the securities to
"trade" in the exchange - in this case the NASDAQ 100 - taken from a
snapshot in April 2013. main() records the symbol and a base
price in a structure:
struct SecurityInfo {
std::string symbol;
double ref_price;
SecurityInfo(const char* sym, double price)
: symbol(sym),
ref_price(price)
{
}
};
typedef std::vector SecurityVector;
The base price serves as a basis for generating random prices for the
example exchange. main() keeps this security information in a
SecurityVector and populates it in the
create_securities() method, later in main().
The main() function begins by creating the
DepthFeedConnection, and uses Boost's thread library to
create a background thread to accept connections from subscribers:
int main(int argc, const char* argv[])
{
// Feed connection
examples::DepthFeedConnection connection(argc, argv);
// Open connection in background thread
connection.accept();
boost::function acceptor(
boost::bind(&examples::DepthFeedConnection::run, &connection));
boost::thread acceptor_thread(acceptor);
Next, the DepthFeedPublisher and the Exchange are
created, and theDepthFeedPublisher is made aware of the
DepthFeedConnection:
// Create feed publisher examples::DepthFeedPublisher feed; feed.set_connection(&connection); // Create exchange examples::Exchange exchange(&feed, &feed);
Note that the exchange constructor requires a TradeListener and
a DepthListener, both of which the
DepthFeedPublisher implements.
Finally, the securities are created and used to populate the exchange, and orders are generated.
// Create securities SecurityVector securities; create_securities(securities); // Populate exchange with securities populate_exchange(exchange, securities); // Generate random orders generate_orders(exchange, securities); return 0; }
The final function call, to generate_orders() is an infinite
loop, so there is no need to worry about the main() method
returning.
The helper function create_securities() adds 100 securities to
the SecurityVector:
void
create_securities(SecurityVector& securities) {
securities.push_back(SecurityInfo("AAPL", 436.36));
securities.push_back(SecurityInfo("ADBE", 45.06));
securities.push_back(SecurityInfo("ADI", 43.93));
// Repeated for 100 securities...
}
Due to the mercy of the author, the other 97 securities were omitted from
this paper. The next helper function, populate_exchange() adds
these securities to the exchange:
void
populate_exchange(examples::Exchange& exchange, const SecurityVector& securities) {
SecurityVector::const_iterator sec;
for (sec = securities.begin(); sec != securities.end(); ++sec) {
exchange.add_order_book(sec->symbol);
}
}
The final helper function, generate_orders() creates random
orders and adds them to the exchange:
void
generate_orders(examples::Exchange& exchange, const SecurityVector& securities) {
time_t now;
time(&now);
std::srand(now);
size_t num_securities = securities.size();
while (true) {
// which security
size_t index = std::rand() % num_securities;
const SecurityInfo& sec = securities[index];
// side
bool is_buy = std::rand() % 2;
// price
uint32_t price_base = sec.ref_price * 100;
uint32_t delta_range = price_base / 50; // +/- 2% of base
int32_t delta = std::rand() % delta_range;
delta -= (delta_range / 2);
double price = double (price_base + delta) / 100;
// qty
book::Quantity qty = (std::rand() % 10 + 1) * 100;
// order
examples::OrderPtr order(new examples::Order(is_buy, price, qty));
// add order
exchange.add_order(sec.symbol, order);
// Wait for eyes to read
sleep(1);
}
}
The implementation of generate_orders() is somewhat complex, but
the details are not relevant. There is a call to sleep()
inside the loop, so that the data stream is produced at a somewhat readable
pace. At this point, the publisher is complete.
Subscriber Application
With the publisher in place, focus turns to the subscriber. The subscriber
must connect to the publisher, decode the FAST feed, recreate the depth
state for each security and display the results. As noted earlier, the
connection is handled by the DepthFeedConnection class. The
decoding and display of messages is handled by a the
DepthFeedSubscriber class:
namespace liquibook { namespace examples {
class DepthFeedSubscriber : public TemplateConsumer {
public:
DepthFeedSubscriber(
const QuickFAST::Codecs::TemplateRegistryPtr& templates);
// Handle a reset of the connection
void handle_reset();
// Handle a message
// return false if failure
bool handle_message(BufferPtr& bp, size_t bytes_transferred);
The DepthFeedSubscriber class has a number of private members:
private:
QuickFAST::Codecs::Decoder decoder_;
typedef std::map<std::string, book::Depth<5> > DepthMap;
DepthMap depth_map_;
uint64_t expected_seq_;
static const uint64_t MSG_TYPE_DEPTH;
static const uint64_t MSG_TYPE_TRADE;
These members include a QuickFAST decoder, an order book for each security
(kept in a std::map), and the expected sequence number. The
sequence number is tracked to validate the feed, so that the subscriber is
sure that every message from the publisher has been handled.
Finally, there are two constants for determining message type. The reader will note that these message type fields appear in the message, but the publisher nowhere encoded them. This is because of their field instructions, found in the FAST template, of constant.
In addition, there are a few private methods, to process incoming methods:
void log_depth(book::Depth<5>& depth);
bool handle_trade_message(const std::string& symbol,
uint64_t& seq_num,
uint64_t& timestamp,
QuickFAST::Messages::Message& msg);
bool handle_depth_message(const std::string& symbol,
uint64_t& seq_num,
uint64_t& timestamp,
QuickFAST::Messages::Message& msg);
};
} }
After initializing the static members comes the class constructor:
namespace liquibook { namespace examples {
const uint64_t DepthFeedSubscriber::MSG_TYPE_DEPTH(11);
const uint64_t DepthFeedSubscriber::MSG_TYPE_TRADE(22);
using QuickFAST::ValueType;
DepthFeedSubscriber::DepthFeedSubscriber(
const QuickFAST::Codecs::TemplateRegistryPtr& templates)
: decoder_(templates),
expected_seq_(1)
{
}
The constructor passes the templates to the decoder, and initializes the
expected sequence number. Next is the handle_reset() method,
which is called when the connection to the publisher is reset:
void
DepthFeedSubscriber::handle_reset()
{
expected_seq_ = 1;
}
This simple method just resets the expected sequence number to one. Next, is
the meaty method handle_message():
bool
DepthFeedSubscriber::handle_message(BufferPtr& bp, size_t bytes_transferred)
{
// Decode the message
QuickFAST::Codecs::DataSourceBuffer source(bp->c_array(), bytes_transferred);
QuickFAST::Codecs::SingleMessageConsumer consumer;
QuickFAST::Codecs::GenericMessageBuilder builder(consumer);
decoder_.decodeMessage(source, builder);
QuickFAST::Messages::Message& msg(consumer.message());
// Examine message contents
uint64_t seq_num, msg_type, timestamp;
const QuickFAST::StringBuffer* string_buffer;
size_t bids_length, asks_length;
std::string symbol;
if (!msg.getUnsignedInteger(id_seq_num_, ValueType::UINT32, seq_num)) {
std::cout << "Could not get seq num from msg" << std::endl;
return false;
}
if (seq_num != expected_seq_) {
std::cout << "ERROR: Got Seq num " << seq_num << ", expected "
<< expected_seq_ << std::endl;
return false;
}
if (!msg.getUnsignedInteger(id_msg_type_, ValueType::UINT32, msg_type)) {
std::cout << "Could not get msg type from msg" << std::endl;
return false;
}
if (!msg.getString(id_symbol_, ValueType::ASCII, string_buffer)) {
std::cout << "Could not get symbol from msg" << std::endl;
return false;
}
if (!msg.getUnsignedInteger(id_timestamp_, ValueType::UINT32, timestamp)) {
std::cout << "Could not get timestamp from msg" << std::endl;
return false;
}
bool result = false;
symbol = (std::string)*string_buffer;
switch (msg_type) {
case MSG_TYPE_DEPTH:
result = handle_depth_message(symbol, seq_num, timestamp, msg);
break;
case MSG_TYPE_TRADE:
result = handle_trade_message(symbol, seq_num, timestamp, msg);
break;
default:
std::cout << "ERROR: Unknown message type " << msg_type
<< " seq num " << seq_num << std::endl;
return false;
}
++expected_seq_;
return result;
}
This method first decodes the FAST message. This is done by stuffing the message contents into a QuickFAST buffer specific for decoding. Next, a message builder and consumer are created and associated, and the message is decoded into the builder. After this, the message is available from the consumer.
The common fields are then checked, by using the type-specific extractors on
the QuickFAST::Messages::Message class, such as
getUnsignedInteger(). This starts with the sequence number,
which is validated against the expected sequence number. Next the message
type, the symbol, and the timestamp are extracted. If any of these fail, the
method exits with an error value.
Finally, since the message type is known, the proper message-type-specific handler is called.
The first helper method logs the contents of the depth for a security:
void
DepthFeedSubscriber::log_depth(book::Depth<5>& depth)
{
book::DepthLevel* bid = depth.bids();
book::DepthLevel* ask = depth.asks();
printf("----------BID---------- ----------ASK----------\n");
while (bid || ask) {
if (bid && bid->order_count()) {
printf("%8.2f %9d [%2d]",
(double)bid->price() / Order::precision_,
bid->aggregate_qty(), bid->order_count());
if (bid == depth.last_bid_level()) {
bid = NULL;
} else {
++bid;
}
} else {
// Blank lines
printf(" ");
bid = NULL;
}
if (ask && ask->order_count()) {
printf(" %8.2f %9d [%2d]\n",
(double)ask->price() / Order::precision_,
ask->aggregate_qty(), ask->order_count());
if (ask == depth.last_ask_level()) {
ask = NULL;
} else {
++ask;
}
} else {
// Newline
printf("\n");
ask = NULL;
}
}
}
This method is complex, because it logs both bid and ask on the same line. In its loop, there could be a bid and an ask, only a bid, or only an ask.
The next helper handles a depth message:
bool
DepthFeedSubscriber::handle_depth_message(
const std::string& symbol,
uint64_t& seq_num,
uint64_t& timestamp,
QuickFAST::Messages::Message& msg)
{
size_t bids_length, asks_length;
std::cout << timestamp
<< " Got depth msg " << seq_num
<< " for symbol " << symbol << std::endl;
// Create or find depth
std::pair<DepthMap::iterator, bool> results = depth_map_.insert(
std::make_pair(symbol, book::Depth<5>()));
book::Depth<5>& depth = results.first->second;
if (msg.getSequenceLength(id_bids_, bids_length)) {
for (size_t i = 0; i < bids_length; ++i) {
const QuickFAST::Messages::MessageAccessor* accessor;
if (msg.getSequenceEntry(id_bids_, i, accessor)) {
uint64_t level_num, price, order_count, aggregate_qty;
if (!accessor->getUnsignedInteger(id_level_num_, ValueType::UINT8,
level_num)) {
std::cout << "Could not get Bid level from depth msg" << std::endl;
return false;
}
if (!accessor->getUnsignedInteger(id_price_, ValueType::UINT32,
price)) {
std::cout << "Could not get Bid price from depth msg" << std::endl;
return false;
}
if (!accessor->getUnsignedInteger(id_order_count_, ValueType::UINT32,
order_count)) {
std::cout << "Could not get Bid count from depth msg" << std::endl;
return false;
}
if (!accessor->getUnsignedInteger(id_size_, ValueType::UINT32,
aggregate_qty)) {
std::cout << "Could not get Bid agg qty from depth msg" << std::endl;
return false;
}
book::DepthLevel& level = depth.bids()[level_num];
level.set(price, aggregate_qty, order_count);
} else {
std::cout << "Failed to get bid " << i << std::endl;
return false;
}
msg.endSequenceEntry(id_bids_, i, accessor);
}
}
if (msg.getSequenceLength(id_asks_, asks_length)) {
for (size_t i = 0; i < asks_length; ++i) {
const QuickFAST::Messages::MessageAccessor* accessor;
if (msg.getSequenceEntry(id_asks_, i, accessor)) {
uint64_t level_num, price, order_count, aggregate_qty;
if (!accessor->getUnsignedInteger(id_level_num_, ValueType::UINT8,
level_num)) {
std::cout << "Could not get Ask level from depth msg " << std::endl;
return false;
}
if (!accessor->getUnsignedInteger(id_price_, ValueType::UINT32,
price)) {
std::cout << "Could not get Ask price from depth msg" << std::endl;
return false;
}
if (!accessor->getUnsignedInteger(id_order_count_, ValueType::UINT32,
order_count)) {
std::cout << "Could not get Ask count from depth msg " << std::endl;
return false;
}
if (!accessor->getUnsignedInteger(id_size_, ValueType::UINT32,
aggregate_qty)) {
std::cout << "Could not get Ask agg qty from depth msg " << std::endl;
return false;
}
book::DepthLevel& level = depth.asks()[level_num];
level.set(price, aggregate_qty, order_count);
} else {
std::cout << "Failed to get ask " << i << std::endl;
return false;
}
msg.endSequenceEntry(id_asks_, i, accessor);
}
}
log_depth(depth);
return true;
}
This method has the common fields passed to it, and logs the symbol of the message. It then finds the proper depth object for the security, or creates one if this is the first depth message for the security.
Next the method must iterate through the sequence of changed bids. To do this, the sequence length is first extracted, and each entry accessed through an accessor.
Each bid field is then accessed using type-specific extractors, including the
level number, the price, the number of orders, and the aggregate quantity.
These updated values are then used to update the depth level for that
security through a call to set(). Similar logic is performed
for changed ask levels, and the resulting depth is logged using
log_depth().
The final helper method handles a trade message:
bool
DepthFeedSubscriber::handle_trade_message(
const std::string& symbol,
uint64_t& seq_num,
uint64_t& timestamp,
QuickFAST::Messages::Message& msg)
{
uint64_t qty, cost;
// Get trade fields
if (!msg.getUnsignedInteger(id_qty_, ValueType::UINT32, qty)) {
std::cout << "Could not qty from trade msg" << std::endl;
return false;
}
if (!msg.getUnsignedInteger(id_cost_, ValueType::UINT32, cost)) {
std::cout << "Could not get cost from trade msg" << std::endl;
return false;
}
double price = (double) cost / (qty * Order::precision_);
std::cout << timestamp
<< " Got trade msg " << seq_num
<< " for symbol " << symbol
<< ": " << qty << "@" << price
<< std::endl;
return true;
}
} }
Like handle_depth_message(), handle_trade_message()
is passed the common fields. It extracts the trade quantity and cost, from
which it calculates the price of the trade (after adjusting for the
precision). The result is then logged.
Subscriber Initialization
Finally, the subscriber can create, initialize, and associate its components.
The main() function of subscriber starts by creating the
DepthFeedConnection and the DepthFeedSubscriber:
int main(int argc, const char* argv[])
{
// Create the connection
liquibook::examples::DepthFeedConnection connection(argc, argv);
// Create feed subscriber
liquibook::examples::DepthFeedSubscriber feed(connection.get_templates());
// Set up handlers
liquibook::examples::MessageHandler msg_handler =
boost::bind(&liquibook::examples::DepthFeedSubscriber::handle_message,
&feed, _1, _2);
liquibook::examples::ResetHandler reset_handler =
boost::bind(&liquibook::examples::DepthFeedSubscriber::handle_reset,
&feed);
connection.set_message_handler(msg_handler);
connection.set_reset_handler(reset_handler);
// Connect to server
connection.connect();
connection.run();
return 0;
}
Next, the handlers for messages and connection resets are set up, and finally, the subscriber connects to the publisher. This completes the subscriber application.
Running the Example
By default the publisher and subscriber connect on port 10003 on localhost. To run on a different port, add the -p <port> to each command line. The subscriber may also specify a remote host with the -h <host> option. To specify a different template file, or template file location, use the -t <filename> option.
For example:
$ ./depth_feed_publisher -p 9999 $ ./depth_feed_subscriber -p 9999
The subscriber produces output like this:
1369167639 Got depth msg 8182 for symbol GILD
----------BID---------- ----------ASK----------
49.93 400 [ 1] 50.12 500 [ 1]
49.66 300 [ 1] 50.13 500 [ 1]
49.59 1000 [ 3] 50.14 500 [ 1]
49.58 700 [ 1] 50.19 500 [ 1]
50.26 800 [ 1]
1369167640 Got depth msg 8183 for symbol XLNX
----------BID---------- ----------ASK----------
37.40 400 [ 1] 37.60 1000 [ 2]
37.36 400 [ 1] 37.61 1900 [ 3]
37.35 1000 [ 1] 37.65 900 [ 3]
37.28 500 [ 2] 37.70 600 [ 1]
37.25 1700 [ 2] 37.71 1000 [ 2]
1369167641 Got depth msg 8184 for symbol GOLD
----------BID---------- ----------ASK----------
78.30 300 [ 1] 78.43 300 [ 1]
78.18 300 [ 1] 78.48 700 [ 1]
78.17 900 [ 1] 78.64 600 [ 1]
78.09 200 [ 1] 78.70 600 [ 1]
78.06 900 [ 1] 79.12 500 [ 1]
1369167642 Got trade msg 8185 for symbol HSIC: 600@89.47
1369167642 Got trade msg 8186 for symbol HSIC: 200@89.49
1369167642 Got depth msg 8187 for symbol HSIC
----------BID---------- ----------ASK----------
89.32 700 [ 1] 89.49 500 [ 1]
89.09 500 [ 1] 89.52 200 [ 1]
88.96 600 [ 1] 89.68 500 [ 1]
88.82 200 [ 1] 89.76 900 [ 1]
88.59 100 [ 1] 89.91 100 [ 1]
These updates show the full depth for each security updated, and not just the changed levels. The first three updates shows the depth for various securities, followed by some trades and a depth update for a fourth security. The trades go hand-in-hand with a depth update. Some depth updates cause trades, and some do not.
Note that a very good test of the incremental feed is to start two subscribers at different times, so that the first subscriber has handled a good number of messages before the second subscriber is started. The two should still be perfectly in sync, and produce identical output.
Summary
Although much code has been walked through in this example, very little custom code was related to the order book. Most of the code was either general networking code using ASIO, or QuickFAST code. In addition, the complexities of order book maintenance and depth-building are contained - most of the code shown was simple, or even trivial.
Liquibook makes it very simple to implement a limit order book matching engine, whether you are building an exchange, or are a trader simulating an exchange's matching capability.
References
- Liquibook - https://github.com/objectcomputing/liquibook
- QuickFAST - http://quickfast.org
- Boost - http://www.boost.org/
- Xerces C - http://xerces.apache.org/xerces-c/
- FIX protocol - http://www.fixprotocol.org
- FAST protocol - http://www.fixprotocol.org/fast
- NASDAQ 100 - http://www.nasdaq.com/markets/indices/nasdaq-100.aspx
![[Valid RSS]](images/valid-rss-rogers.png "Validate my RSS feed")