Technology Tips to Help Investment Firms Reduce Market Data Costs

Since the online revolution, we’re collectively flooding the world with unprecedented amounts of digital data. Several years ago it was estimated that by 2020 the amount of information in the digital realm would reach 44 zettabytes. That’s 1,000,000,000,000,000,000,000 bytes, by the way. A truly astronomical figure, seeing as it’s also forty times more than the number of stars in the observable universe.

Another sobering statistic is that each day we produce an incredible 2.5 quintillion bytes of data. This means that 90% of all the world’s data came into existence in the past two years alone.

Data in the online trading industry

With trading now being an almost exclusively online activity, you’d expect the same broad trend to be present in this industry. However, online trading actually finds itself at the nexus of a number of related trends that are amplifying the effects of this deluge of data.

This industry is also a beneficiary of mobile, market access, and mainstream adoption trends that are having significant impacts on the cost of data procurement, management and storage.

One of the significant shifts is the adoption of cryptocurrencies. New asset classes mean thousands of new symbols to be integrated, transmitted and stored. If you’ve been around for long enough, you’ll recall the cost of market data once being so negligible that it hardly factored into the business logic of most outfits. Nowadays, an entire industry has sprung up to assist market participants in organising this most crucial area of their operations.

As an example, Devexperts’ market data subsidiary, dxFeed, currently stores petabytes of market data history and this is only going to grow as markets continue to evolve. To this end, I’d like to provide a few insights for companies that are just now starting to think about how to optimise their market data use.


Audit your market data use regularly

This should involve performing regular audits so as to pin down which vendors’ feeds are business-critical, how they have performed, and which ones have raised their prices. Additionally, do your vendors overlap in what they provide? Are you paying for packages that you’re currently not using and have no plans to in the immediate future? Finally, are long-term contracts really saving you money? How flexible are they? Are you having to meet the needs of an evolving market by sourcing different additions to your available assets elsewhere?

Only this type of regular audit process will provide you with the insights necessary to identify inefficiencies and then be able to do something about them by negotiating with existing providers or securing a better deal for your business elsewhere.


Make the most of what you’re paying for

With your market data use being regularly audited, you should be able to make cuts to account for needless redundancies and for price feeds that are not currently being used. After that, the next step is to make the very most of the market data that you do want to keep.

If you’re currently subscribing to any data feeds that are composed of unrefined market data, you can do a lot worse than employing compression algorithms to smooth out price spikes that are unique to the feed being used and thus not representative of the broader market. This is done by comparing different feeds and filtering out spikes that occur within a given percentage.

This has the effect of adding value to the data you’re already paying for, which will be much appreciated by the traders whose positions are affected by said spikes.


Manage your subscriptions

While adding value is great, the largest savings to your business are probably to be made in paying closer attention to the costs of your subscriptions. In our experience, this is where third parties (dxFeed included!) have been extremely disruptive and can often offer much more cost effective solutions than you may currently be aware of.

This is because third parties are equipped to provide data that approximates the original real-time exchange streams. The effect of this is to massively reduce exchange costs because the derived data that’s employed cannot be used to reconstitute original tick data.
Third parties also allow for more customisation over the data that you’re subscribed to. The big players in market data provision can be inflexible, requiring you to subscribe to certain collections of symbols by default, many of which may not be of interest to your traders. By contrast, third party market data providers are more likely to allow their clients to select only the specific feeds the business requires.


Consider storage costs

With so much emphasis being placed on accurate, to-the-second price data, the importance of detailed histories can easily be overlooked.

How far back should brokers go? At what resolution? What exactly needs to be preserved? This can sometimes come down to IT budgeting constraints, whereas a solid third party provider can offer options that remove the guesswork and can be tailored to the needs of their traders. These include the on-demand streaming of historical price data, as well as detailed market replay data that’s crucial to advanced clients who wish to perform backtests in order to optimise their trading strategies.


Reduce cost of transmission

Retail brokerages are sometimes limited by their platform providers in how they distribute data, but there are a number of hacks that can be used in order to optimise for transmission costs.

For one, quote conflation and removal algorithms can be employed to sift through the tick data and only transmit non-duplicate ticks, as well as removing quotes that are no longer relevant. The result is a stream that looks identical to the underlying market data, but with costly duplicates and redundant data having been filtered-out. This frees up bandwidth and reduces latency, particularly in times of peak traffic.

Delta-encoding is another way to reduce bandwidth and storage costs by transmitting just the differences between sequential values instead of transmitting the whole stream including redundancies. So, instead of a stream of price ticks that appear as follows: “411, 414, 416, 413, 415”, after delta-encoding they would look like this: 1, 3, 2, -3, 2. By only sending or storing the deltas, this method reduces the number of bits required to store and transmit historical market data.


Final thoughts

The above are but a few introductory ideas and optimisations that we at both Devexperts and dxFeed have put into practice over the years while working closely with brokers to overhaul their market data infrastructures.

Each brokerage business is unique, with differing inefficiencies that have to be identified and addressed. But if there’s one thing that we’ve found to be almost universally true, it’s that all market data infrastructures could use at least some optimisation. In our experience, there’s always money to be saved and efficiencies to be made. After all, we’re in the earliest innings of the exponential data trend we began this article with, and you may be surprised at how many optimisations there are already to be made under all that data.