- S-Network Global Indexes
- Posted September 8, 2017
Not All Backtests Are Created Equal
The efficacy of the Smart Beta backtesting process has come under criticism from some corners of the market recently. Some of this criticism is valid, but much of it depends on gross generalizations.
A well-constructed backtest provides investors with important information about the performance characteristics of a specific methodology. That said, the fact remains that not all Smart Beta backtests are created equal.
Here are some key factors to look for in evaluating backtested results:
Having a clearly stated and transparent methodology is at the core of any sound backtest. If the index rule set can be applied to fully replicate the index, then you know the back test is grounded in reality and will be valid. Complete transparency is a hallmark of all S-Network branded Smart Beta indexes; all of our rules are published on public-facing websites. In fact, we encourage regulators, gatekeepers and sophisticated investors to replicate our indexes independently.
It’s critical to select the Smart Beta stocks or re-weight the Smart Beta stocks from an appropriate universe of stocks. At S-Network, we've developed a family of benchmark indexes that provide daily index files starting on December 31, 1999. Data from inception has been fully cleansed for these benchmarks, which have been relived on a daily basis from inception, reflecting bankruptcies, delistings, mergers, acquisitions, spinoffs and other types of corporate events. By using a well-developed universe, the practice of "garbage in/garbage out" is largely eliminated.
Although trading stocks in recent years has become increasingly more frictionless, pitfalls remain in defining implementation costs. To begin with, all financial products based on an index will underperform the index by the amount of the total expense ratio of the financial product. Beyond that, trading costs can compromise real life performance compared to backtested performance. We take steps to avoid some common pitfalls that may compromise the liquidity of an index. For example, an index’s investability is constrained by the liquidity of its least liquid stock. Accordingly, we avoid mixing large cap and small cap stocks or developed market and emerging market stocks because of potentially sharp divergences in liquidity characteristics. By the same token, we always apply liquidity screens to our eligibility rules.
By examining these three key considerations carefully, one can be reasonably assured of the efficacy of a backtest. A well-conceived and implemented backtest covering an extended period provides an astute investor with critical information that would ordinarily be unavailable. At S-Network, most of our backtests originate on December 31, 1999 and therefore demonstrate performance characteristics in both good and bad markets. For example, it is revelatory to see how an index performed over the 2001 tech crash or the 2008 real estate crash.
Finally, it is important to note that a Smart Beta Index will seldom outperform the market in all conditions. At the risk of using a hackneyed expression, "Past Performance Is No Guarantee of Future Results!" So far this year, most Smart Beta Indexes have underperformed the market, regardless of whether their histories are live or backtested. The reason being Apple, Microsoft, Amazon, Alphabet and Facebook have accounted for a disproportionate percentage of the stock market rise. If these stocks did not dominate the Smart Beta index, as they do in most capitalization-weighted large cap indexes like the S&P 500, the Smart Beta Index will have underperformed the market.
A lot of garbage can make its way into the market based on flawed backtests. However, S-Network branded indexes as well as many of those branded by other major providers are based on sound investment principles, specific diversification policies and are screened to assure liquidity up to a certain AuM threshold.
Ultimately, it pays to evaluate backtests carefully and be able to take advantage of the myriad benefits they provide.