These chronic performance shortfalls have led many clients to rethink their hedge fund investments. In 2014, the California Public Employees’ Retirement System (CalPERS), the largest

Continue reading Which hedge funds actually beat the market?

]]>The last few years have been difficult times for hedge funds. For the majority of these funds, performance has lagged market averages, certainly not in keeping with the exalted fees charged by the fund managers. For example, as of 1 July 2017, the HFRI Fund Weighted Composite Index is up 3.28% year-to-date, and 4.79% annualized gain for the previous 5 years. The corresponding figures for the S&P500 Index (including dividends) are 9.34% and 13.6%.

These chronic performance shortfalls have led many clients to rethink their hedge fund investments. In 2014, the California Public Employees’ Retirement System (CalPERS), the largest U.S. pension fund, announced that it would liquidate its USD$4 billion investment in hedge funds. In April 2016, the New York City public pension fund announced that it will also liquidate all hedge fund investments.

Others are questioning hedge fund fees, typically a 2% annual fee, plus a performance fee of 20% on any profits. These fees are, of course, far higher than the fees typically charged by conventional mutual funds, not to mention the rock-bottom fees (as low as 0.05%) of popular index-tracking exchange-traded funds (ETFs).

As a result of these difficulties, in 2016 total hedge fund holdings declined by USD$70 billion, only the third net loss in history.

Each year, *Institutional Investor’s Alpha* publishes its annual report on the performance of the top 100 hedge funds.

In their latest report, numerous large funds suffered sizable asset declines. These include Paulson & Co., which at one time was the world’s third-largest hedge fund (recall they made billions on their 2007 bet that the U.S. home mortgage market would collapse). In the wake of recent double-digit losses in several of its mainline funds, Paulson now ranks #69.

Other funds with large declines include Lone Pine Capital (now #19), Tudor Investments (#65), Capital Management (#29), Oct-Ziff Capital Management Group (#9) and others. Perry Corp., in the wake of several years of losses, no longer ranks in the top 100.

In spite of these liquidations and closures, however, there are still over 10,000 hedge funds in operation.

So in the midst of all this gloom-and-doom, are there any hedge fund firms that actually make good money, achieving returns that significantly beat market averages, and are actually increasing in assets? Yes, as it turns out. Here are some:

- Renaissance Technologies (ranked #4). They now manage $42 billion in assets, up a whopping 42% from the previous year. According to IIA, Renaissance is arguably the most successful hedge fund firm of all time.
- AQR Capital Management (ranked #2). AQR’s assets increased by 48%, to $69.7 billion.
- Two Sigma (ranked #11). Their assets increased 28%.
- Bridgewater Associates (ranked #1). Bridgewater manages USD$122.3 billion, up 17% from 2015. Its Pure Alpha fund has achieved a 14.6% annualized return for the past five years.

In general, according to Institutional Investor’s Alpha,

Five of the six largest firms in this year’s ranking rely all or mostly on computers and algorithms to make their investment decisions, a theme that has increasingly played a role in the top-100 ranking over the past few years. And all five posted asset increases last year.

So it really is true that geeks are ruling the world? It sure seems so. At the least, these results are consistent with the “efficient market hypothesis,” in the sense of the difficulty of guessing markets based on present or past data. At the least, only very sophisticated mathematical techniques are likely to achieve “positive alpha” returns.

As we have pointed out in a previous Blog, markets by definition incorporate the collective judgments of many thousands of highly trained market analysts worldwide, who employ sophisticated mathematical algorithms running on powerful supercomputers to ferret out any regularities or correlations, and then use high-frequency trading algorithms to act on these phenomena in real time (not to mention millions of other corporate and individual investors). The efforts of these many players largely cancel each other out, leaving a time series that is little more than a random walk.

Not so long ago, discretionary traders dismissed the idea that one day quantitative funds would rule the space. One argument was that quant funds were “black boxes”, models that recommend counter-intuitive trades, bets that nobody can understand. Ironically, that may be the reason for today’s **quant hegemony**: A counter-intuitive good bet is better than an intuitive one, because the former is less likely to be crowded. In other words, a black box does not have to share the profits with millions of individuals, all reading the same Wall Street Journal articles. Furthermore, the number of possible black-box models is combinatorially enormous, thanks to the explosion of data sources, Big data, machine learning and high-performance computing.

In any event, we should not be surprised that even relatively sophisticated hedge funds have difficulty achieving consistently above-market returns. After all, they are betting against a signal that is almost entirely random noise, and then charging a premium fee.

]]>A typical scenario is that in the midst of a market downturn, investors panic and sell out, with the intent of waiting for the market to “bottom out” before reinvesting. Some investors believe that they can

Continue reading Gender, marital status and investment performance

]]>A recent DALBAR study found that over a 30-year period, the average self-directed equity mutual fund investor earned only 3.7 percent, compared with 10.3 percent that could be obtained by simply investing in a S&P500 index fund. Much of this huge shortfall is due to panic selling during market downturns, or attempts to time the market.

A typical scenario is that in the midst of a market downturn, investors panic and sell out, with the intent of waiting for the market to “bottom out” before reinvesting. Some investors believe that they can actually earn a tidy profit in the process. Sadly, the all-too-frequent experience is that the sell orders come near the market bottom, and by the time investors reinvest, the market has rallied significantly, leaving the investors with a permanent haircut on their investment portfolio compared to a simple buy-and-hold approach (not to mention the fees paid in the process).

In general, individual investors’ attempts at market timing are not only futile, they are measurably counterproductive. According to a Morningstar analysis, as recently summarized by Marketwatch, individual U.S. equity mutual fund investors earn 0.79% less annual return than their underlying funds. Individual U.S. fixed income fund investors earn 0.73% less. This may not sound like much, but over a 40-year investment horizon a 0.79% annual reduction from an 8% average annual return balloons to a 26% permanent portfolio loss.

As Morningstar notes, almost all of this reduction is due to amateurish attempts at market timing. If even professional market analysts and highly sophisticated computer programs struggle with market timing, what realistic chance is there that an individual investor, armed only with charts and graphs, can do better?

Active traders do significantly more poorly than those who trade infrequently or not at all (although it may be hard to convince such persons that their efforts are detrimental). According to a study by Brad Barber and Terrance Odeon of U.C. Berkeley, the most active U.S. equity traders achieved only about 62% of the return of those who trade infrequently. If we again assume an average 8% annual return for buy-and-hold investors and a 4.9% return for active traders, over a 40-year time horizon the buy-and-hold portfolio would be 3.2 times larger! As Barber and Odeon conclude, “trading is hazardous to your wealth.”

In a separate study, Barber and Odeon found that men trade 45 percent more than women, and that this trading reduced their overall rate of return by 14.2%, as opposed to “only” a 9.1% reduction for women. Again, over a 40-year time horizon, these losses are 35% of portfolio value for men and 24% for women. If we compare the average amount of “churning” (portfolio turnover), the figures are 77% for men and 53% for women.

These losses are even more pronounced for singles — the average annual portfolio churn for single men was 90%, compared with 50% for single women.

Why is there such a difference between the sexes? As Barber and Odeon observe,

Psychological research has established that men are more prone to overconfidence than women, particularly so in male-dominated realms such as finance. … Models of investor overconfidence predict that, since men are more overconfident than women, men will trade more and perform worse than women. Our empirical tests provide strong support for the behavioral finance model.

The two studies mentioned above by Barber and Odeon are now 17 and 19 years old. But from anecdotal exchanges with people in the industry, the situation is little changed — men are more likely to trade frequently, more likely to be overconfident in their investing, and more likely to achieve subpar rates of return.

The phenomenon of excessive trading (and corresponding drops in performance) is exacerbated with the ease of online trading. In an earlier era, if one wanted to sell (or buy) some shares, one would need to call one’s broker, discuss the trade, and then wait for the broker to have the trade executed. Even though that was not a desirable situation, and fees were much higher, it did require time for more careful reflection and one-on-one discussion with the broker.

But now it is all too easy to trade. Major brokerages are currently engaged in a price war. For as little as USD$4.95 one can buy or sell a large block of stocks, bonds or exchange-traded fund shares. For certain exchange-traded funds operated by the brokerages, there is no fee at all (except for a small exchange fee).

Indeed, the rise of exchange-traded funds, which can be bought and sold anytime during the trading day, has prompted calls of concern that they are merely facilitating unwise investor behavior. Vanguard founder Jack Bogle, for instance, recently called (paywall) for politicians to re-examine ETFs. He noted that the returns achieved by Vanguard investors in their large mutual funds exceeds that achieved by investors in the equivalent Vanguard ETFs by 1.6%, almost certainly due to attempts by investors to time the market.

Still others wonder if the whole U.S. retirement system needs to be rethought, given its increasing reliance on user-directed 401Ks. Paul Merriman, citing a DALBAR report, assessed the situation in these terms:

The numbers change from year to year, but every DALBAR report comes to the same conclusion: Investors’ emotion-based trading is counterproductive… [Despite] enormous efforts by thousands of industry experts to educate millions of investors, imprudent action continues to be widespread. … The belief that investors will make prudent decisions after education and disclosure has been totally discredited.”

In the meantime, all individual investors can do is to keep focused on consistent long-term investment, preferably in very-low-fee index funds or the like, not attempting to time or second-guess the market. And given the results mentioned above for gender and marital status, maybe we all should try harder to maintain effective communication with our spouse or significant other, in order to avoid hasty and unwise investment decisions.

]]>In a recent WSJ article, Jason Zweig brilliantly summarizes the unbearable hype and hubris exhibited by some self-titled “quants”:

BlackRock, the giant asset manager, recently announced it will rely more heavily on computers to pick stocks. Rob Arnott, a leading advocate of mechanical investing approaches, said this past week that it’s “actually relatively easy to beat the market” if you get the math right.

Mr. Zweig is of

Continue reading Pseudo-quants

]]>*As the old joke says, “math is what mathematicians do.” Somehow this simple tautology is lost in the dishonest world of finance*

In a recent WSJ article, Jason Zweig brilliantly summarizes the unbearable hype and hubris exhibited by some self-titled “quants”:

BlackRock, the giant asset manager, recently announced it will rely more heavily on computers to pick stocks. Rob Arnott, a leading advocate of mechanical investing approaches, said this past week that it’s “actually relatively easy to beat the market” if you get the math right.

Mr. Zweig is of course a great professional and a polite journalist. Some of us would have mentioned, in passing, that Rob Arnott’s funds have been divested by PIMCO after years of underperformance. So it seems beating the market is not “actually relatively easy” after all. Many other “factor investing” and “smart beta” zero-humility funds will follow.

*Some call it “factor investing”, others call it “smart beta”. Whatever this “dumb alpha” is, it does not perform like real quant investments*

Like any buzzword, the term *quant* has been abused to mean everything. Historically, a quant has been a PhD in a STEM fields who applies her or his scientific knowledge to finance. Perhaps the best known example is Emanuel Derman, a PhD in physics who in the early 2000s wrote a bestseller titled *“My Life as a Quant.”* Some of the most successful funds in history are led by quants, with no background in academic finance: RenTec, Two Sigma, DE Shaw, CFM, etc. Moreover, these quant funds do not hire candidates with a PhD in finance. Clearly, the most successful quant funds do not consider academic finance as part of the quant world. Let us understand why.

*RenTec was founded by the eminent mathematician James H. Simons, who had no prior financial background. The firm has managed to produce average returns of 35% over more than two decades, without ever employing a PhD in finance as quant researcher*

Given the success of these firms, and the term quant, soon financial academics saw profitable to call themselves “quants” too. They must have thought: “Forget about real math, financial models use numbers and econometrics, right?” In the uncertain world of finance, it is good marketing to attach a scientific aura to a financial product. Here is a little problem: Finance is not a science according to accepted epistemological definitions. In order to be considered a science, finance must overcome three hurdles:

- It must discover immutable positive laws.
- Experiments must be independently reproducible.
- Predictions must be accurate.

Finance is the result of changing human institutions, agents, laws. Unlike physical phenomena, financial markets are an adaptive system. There are no positive laws in finance. Competition means that every edge is doomed to disappear. At best academics may find something that used to work, as the opportunity will be arbitraged away following its publication.

The only financial laboratory is the market, and that is too busy solving real-life problems. Academics cannot repeat experiments, hence their discoveries cannot be independently validated. We cannot go back to 6 May 2010 and repeat the events of the flash crash by removing some actors, in order to derive a precise cause-effect mechanism. Instead, academics pretend that historical simulations (backtests) can replace true experiments. Even if that were true, virtually no study controls for multiple testing, a practice the American Statistical Association considers *“highly misleading”*. As the current President of the American Finance Association has acknowledged, this means that the great majority of the discoveries published in journals are likely false, due to selection bias. The reason these pseudo-scientific financial theories are taught in classrooms is because they are unfalsifiable in a Popperian sense.

*The scientific method requires experimentation, not historical simulation. Finance is an art, and there is nothing wrong with that*

Where are the billionaire financial academics? There are no billionaire Nobel Prize winners in Economics. If you could make accurate financial predictions, would you not act on them, if for no other reason to prove a point? No academic financial theory has generated substantial investment returns for their authors. Of course, some academics have enriched themselves by charging outrageous management fees to their investors, but that has not made their investors wealthier. Useless predictions means finance is not a science in a Lakatosian sense.

*Left: The original Nobel medal, given for Physics, Chemistry and Medicine. Right: The medal funded by the Bank of Sweden since 1968, against the will of the Nobel family. Yes, they look very similar, like quants vs. pseudo-quants, but do not be fooled by appearances: What would you call a “science” that cannot predict anything?*

Many investors have been misled to believe that financial products originated in academic finance are scientific. Pension allocators have poured hundreds of billions of dollars in so-called “factor investments” and “smart beta funds”, not because they perform well, but because they have a good academic pedigree. Remember, these asset allocators are not rewarded for success, and their decision is informed primarily by risk management. That is, managing *their* risk of being fired! When fund ABC fails, they will point out that ABC invested in an idea by Nobel Prize XYZ, and that they are not going to second-guess science. The term con-art seems more appropriate than science, as these funds abuse people’s trust in science.

Many of these products, known as “factor investments” or “smart beta funds”, take the form of ETFs that offer immediate liquidity. That may have been alright when the industry was $100 billion, but assets keep growing steadily approaching $1 trillion, propelled by the “Fed’s put”. Because they rely on the same principles, these funds are highly correlated. Sooner or later the Fed will cease to suppress volatility, and let market forces dictate prices. When customers realize that factor investments underperform the market (especially given the hefty fees!), they will pull their money. One possibility is that instant liquidity may enable scenarios like the flash crash.

A more likely scenario is that investors will steadily lose money… bled by fees and arbitrageurs. Here is how:

- Companies will adjust their accounting to become attractive to these publicly-known factor models.
- A small number of securities favored by the same academic models will receive large inflows.
- Prices will rise beyond their fair value, not because these theories are right but because the academic marketing machine keeps inflating their prices in a predictable manner.
- Quant funds will prey on pseudo-quant funds.

And that’s a key flaw of academic financial models, that they are public: Humans will always find a way to turn a purported financial “law” on its head, and profit from those gullible enough to believe in it.

]]>Some of these forecasts are optimistic. For example, on 3 January 2015 Thomas Lee predicted that the S&P 500 index would be at 2325 one year hence. (The S&P 500 ranged between 1867 and 2122 during this period, closing at 2012 on 4 January 2016, well short of the goal.)

Some are

Continue reading How accurate are market forecasters?

]]>Many investors, individual as well as institutional, rely on market experts and forecasters when making investment decisions. Needless to say, some of these forecasts tend to be more accurate than others. How can one decide which of these forecasts, if any, to take seriously?

Some of these forecasts are optimistic. For example, on 3 January 2015 Thomas Lee predicted that the S&P 500 index would be at 2325 one year hence. (The S&P 500 ranged between 1867 and 2122 during this period, closing at 2012 on 4 January 2016, well short of the goal.)

Some are pessimistic. In July 2013, Jerry Burnham predicted that the Dow Jones Industrial Average (DJIA) would drop to 5,000 before it topped 20,000. He repeated this forecast on the PBS News Hour in May 2014. (The DJIA exceeded 20,000 on 25 January 2017, having never dropped below 14,700 during the period 1 July 2013 through 25 January 2017.)

There have been several previous analyses of forecaster accuracy, both in academic literature and also in the financial press.

As a single example, recently Nir Kaissar analyzed a set of predictions by market forecasters over a 17-year period from 1999 through 2016. He found that although there was a reasonably high correlation between the average forecast and the year-end price of the S&P 500 index for the given year, these predictions were surprisingly unreliable during major shifts in the market.

For example, Kaissar found that the strategists overestimated the S&P 500’s year-end price by 26.2 percent on average during the three recession years 2000 through 2002, yet they underestimated the index’s level by 10.6 percent for the initial recovery year 2003. A similar phenomenon was seen in 2008, when strategists overestimated the S&P 500’s year-end level by a whopping 64.3 percent in 2008, but then underestimated the index by 10.9 percent for the first half of 2009.

In other words, as Kaissar lamented, the forecasts were least useful when they mattered most.

Two of the present bloggers (Bailey and Lopez de Prado), together with Amir Salehipour of the University of Newcastle, have completed a new study of market forecasters. For this study, we expanded on a 2013 study conducted by the CXO Advisory Group, which ranked 68 forecasters.

For our study, we further analyzed these 68 forecasters based on two additional factors:

- The time frame of the forecast. Forecasts are categorized as up to one month, up to three months, up to nine months or beyond nine months (e.g., two or three years).
- The importance and specificity of the forecast. For example, a forecast that states “the market will be volatile in the next few days” is not a very specific forecast, but the forecast “the market will experience a correction” is more specific and thus more important.

The results of our analysis are available in a preprint manuscript. We found, perhaps not too surprisingly, that most of these forecasts did not perform significantly different than a chance forecast.

However, a few did remarkably well. The top-ranking forecaster was 78.7% accurate by our metric. The next three had 72.5%, 71.8% and 70.5% accuracy scores. A total of 11 of the 68 had accuracy scores exceeding 60%. At the other end of our ranking, two had accuracy scores near 17%; three others had scores 25% or lower. A total of 18 had accuracy scores less than 40%.

Full details are presented in our preprint manuscript.

One question that remains from this study is to what extent our analysis, or those of any other similar study, are biased by the simple fact that unsuccessful forecasters tend not to remain in the business for a long period of time. Thus long-term accuracy scores and rankings (the only ones that are statistically significant) necessarily omit those forecasters who have dropped out. We cannot answer this question; we merely list it as a concern. But it does mean that all rankings and scorings may tend to be optimistic and must be read carefully.

As we have pointed out in several previous blogs (see, for example, Blog A), we should not be surprised that even the best professionals in the business have difficulty consistently beating the market over the long term. Since markets by definition incorporate the collective judgments of many thousands of participants worldwide, including organizations who employ sophisticated mathematical algorithms, powerful computer systems and high-frequency trading facilities, it follows that most major markets are reduced to time series that exhibit many of the statistical characteristics of a random walk. And one key characteristic of a random walk is unpredictability — the impossibility of prediction, based on the past history of the time series, of the future course the time series will take.

Obviously a few market professionals have indeed been able to beat the market averages over a sustained period of time. Warren Buffett is frequently cited as an example, and there is no doubt his record is distinguished — over 20% compounded per annum return for 40 years running.

On the other hand, almost all of his outsize gains were made in the 1970s and 1980s. If we look at the record, say, of Buffett’s Berkshire Hathaway-B stock from its inception in May 1996 through March 2017, the average compounded annual gain was 9.9%, which is greater than the S&P500 index (8.3%), but not dramatically so. What’s more, over the past nine years BRK-B’s return has been only 6.2%, underperforming the S&P500 (7.1%) by nearly one percentage point. So was Buffett’s performance in the 1970s and 1980s merely a statistical outlier?

We do not suggest that value analyses and forecasting are vain. After all, detailed analyses of fundamental value and prospects for the future for individual securities are essential for a well-functioning marketplace, in our own day as in years past. But forecasters, fund managers and other major market players do need to be rigorously and impartially evaluated from time to time. And all investors, big and small, should be advised not to rely on market forecasters and fund managers whose record is poor or whose record has not been rigorously vetted through careful statistical analysis.

]]>So what advice does Buffett have for the rest of us mere mortals, individual investors as well as institutional investors?

His answer is quite arresting: Invest mainly in

Continue reading Investment advice from the world’s most successful stock picker

]]>So what advice does Buffett have for the rest of us mere mortals, individual investors as well as institutional investors?

His answer is quite arresting: Invest mainly in broad-market index-tracking funds with very low fees.

In his 2017 letter to stockholders, released 25 February 2017, Buffett mentioned a bet he made nearly 10 years ago, where he challenged an asset manager for the hedge fund Protege Partners that a low-cost S&P 500 index-tracking fund would outperform a basket of hedge funds. In his letter, Buffett provided an update: the basket of hedge funds had average compound annual returns of 2.2% through 2016; the index fund has returned 7.1%, more than three times as much.

Buffett’s bet, which he almost certainly will win, ends on 31 December 2017. He has promised that his winnings will be donated to charity.

Buffett points his finger at the large fees assessed by the hedge funds — he estimates that approximately 60% of the gains achieved by the hedge funds were lost to management fees over the past ten years. He writes, “When trillions of dollars are managed by Wall Streeters charging high fees, it will usually be the managers who reap outsized profits, not the clients.” He added “Both large and small investors should stick with low-cost index funds.”

Buffett recently calculated that over the past ten years, wealthy investors, pension funds and other institutional investors have lost roughly USD$100 billion to fees charged by hedge funds and other high-fee investment vehicles. He adds, “my calculation of the aggregate shortfall is conservative.”

Buffett also emphasized that these high fees have consequences:

Much of the financial damage befell pension funds for public employees. … Many of these funds are woefully underfunded, in part because they have suffered a double whammy: poor investment performance accompanied by huge fees. The resulting shortfalls in their assets will for decades have to be made up by local taxpayers.

Buffett continued by heaping praise on Jack Bogle, the founder of Vanguard Group, which pioneered very low-cost index-fund investment.

Along this line, Buffett clearly practices what he preaches, at least for that portion of his personal fortune that is to be inherited by his wife and heirs. He has instructed the trustee of his estate, after he dies, to invest at least 90% of the money in low-cost S&P 500 index-tracking funds, and the remainder in government bonds.

One question that is frequently raised is whether a large-scale movement to passive indexed investing would threaten the health of the system.

There are certainly some legitimate grounds for concern here. After all, the investing world will always require at least some investors who address fundamentals, who read quarterly reports and who continually reassess the value of a particular firm’s stock and bond offerings in light of current news, both about the firm itself and also the ever-changing landscape of economic environment and competitors in the firm’s sector. Clearly if 90% or more of a market were passively indexed, that market would be quite unstable, subject to wrenching volatility when, for example, some trader accidentally sells far too many shares, as in the 2010 flash crash. Even, say, 70% would be cause for concern.

However, we are still quite some distance from this level of passive investment. A December 2014 tabulation of U.S. market data found a total equity capitalization of USD$21.7 trillion, of which USD$2.6 trillion, or 12.1% was passively indexed. So U.S. equity markets have a long ways to go before passive investment becomes a significant stability issue. European and Asian markets are even further from the level of concern.

As we have pointed out in several previous blogs (see, for example, Blog A and Blog B), we should not be terribly surprised at the fact that even the best hedge funds and other highly professional investment organizations have difficulty consistently beating the market. Since markets by definition incorporate the collective judgments of many thousands of participants worldwide, including quite a few organizations who employ very sophisticated mathematical algorithms and high-frequency trading software (often canceling out each others’ efforts), it follows that equity markets are reduced to time series that exhibit many of the statistical characteristics of a random walk (including unpredictability).

A few organizations, such as the Renaissance Medallion Fund, and a few individual managers, such as Warren Buffet, have indeed been able to beat the market averages over a sustained period of time. But replicating this success is, by definition, very difficult, and even more difficult to prove statistically.

According to an ancient account, when Pharaoh Ptolemy I of Egypt grew frustrated at the degree of effort required to master geometry, he asked his tutor Euclid whether there was some easier path. Euclid is reputed to have replied, “There is no royal road to geometry.” Indeed. And there is no royal road to investment either.

]]>The basic idea behind smart beta is to observe that traditional capitalization-weighted investments (such as S&P 500 index funds) tend to be heavily weighted in favor of securities from large, stable firms. Thus the smart beta community

Continue reading Backtest overfitting in smart beta investments

]]>In the past few years “smart beta” (also known as “alternative beta” or “strategic beta”) investments have grown rapidly in popularity. As of the current date (January 2017), assets in these investment categories have grown to over USD$500 billion, and are expected to reach USD$1 trillion by 2020. More than 844 exchange-traded funds employing a smart beta strategy are now in operation.

The basic idea behind smart beta is to observe that traditional capitalization-weighted investments (such as S&P 500 index funds) tend to be heavily weighted in favor of securities from large, stable firms. Thus the smart beta community believes that by adopting a different weighting system, such as one that magnifies smaller, higher-risk securities, one can achieve superior long-term growth. The smart beta movement had its roots in tests of the efficient market hypothesis in the academic finance community during the 1970s, 1980s and 1990s.

The general “smart beta” philosophy encompasses many specific strategies. If one invests equally, say, in a S&P500 index fund and a mid-cap or small-cap index fund, one is using a “smart beta” strategy, by this definition. One of the earliest commercial smart beta funds assigned all S&P 500 index components equal weights. Today, the smart beta world encompasses a diverse range of strategies, ranging from novel systems of weights to systematic trading algorithms, derivatives and multi-asset investments.

Even within the realm of weighting-based smart beta strategies, there are numerous variations, as given in the following taxonomy:

*Return-oriented*: Dividend-screened, dividend-weighted, value, growth, fundamentals, multi-factor, size, momentum, buyback/shareholder yield, earnings weighted, quality, expected returns or revenue weighted.*Risk-oriented*: Minimum volatility/variance, low/high beta or risk-weighted.*Other*: Non-traditional commodity, equal-weighted, non-traditional fixed income or multi-asset.

Many are concerned, however, that the original concept of “smart beta” has been extended to such a large collection of sophisticated strategies that the simplicity and elegance of the original concept has been lost. What’s more, as the complexity of these strategies has increased, so has the likelihood that they will suffer from (or be invalidated by) backtest overfitting.

By backtest overfitting here, we mean the usage of historical market data to develop an investment strategy, where many parameter variations (quite possibly millions or billions of variations) have been searched by computer to find an optimal strategy. The present authors, among others, have developed statistical tests to help one avoid backtest overfitting; see, for example, our paper on the deflated Sharpe ratio, which corrects for two leading sources of performance inflation, namely selection bias under multiple testing and non-normally distributed returns.

A new paper on the topic of backtest overfitting in smart beta strategies has appeared in the Journal of Portfolio Management (see also preprint).

These authors (Antti Suhonen, Matthias Lennkh and Fabrice Perez) began with a database of approximately 3000 alternative beta strategies, from which they selected 215 unique strategies with sufficient data for their analysis. The average backtest period of their test set was 10.7 years, and the average live operation time was 4.6 years. These 215 strategies included commodity, equity, fixed income, foreign exchange and multi-asset schemes.

Among their tabulated statistics is the Sharpe ratio of the strategy during the backtest period, the Sharpe ratio during the live period and the “realized haircut” — the percentage reduction in Sharpe ratio between the backtest and the live periods. Over all tested strategies, they found a surprisingly large median haircut of 72%; for equity strategies it was 80%. It is also notable that only 18 of the 215 strategies had a live Sharpe ratio greater than the backtest Sharpe ratio; 65 of the 215 strategies had a negative Sharpe ratio over the live period.

Suhonen, Lennkh and Perez recognized that given the 2008-2009 worldwide market crash, their results might be dismissed as an artifact of that unfortunate episode. But when they restricted their data to just a 3-year pre/post time window, thus avoiding the 2008-2009 period, their results were little changed — the median haircut dropped from 72% to 62% (which is still a very large drop-off), but the same number of strategies as before (65) exhibited a negative Sharpe ratio over the live period.

In another interesting analysis, the Suhonen-Lennkh-Perez paper categorized the 215 strategies by complexity, and then rated their performance accordingly to this categorization. Indeed, as some have previously feared, they found that the more complex strategies suffered larger haircuts — in particular, the most complex strategies suffered 30 percentage points more haircut than the simplest strategies.

The findings on complexity are arguably the most significant of the Suhonen-Lennkh-Perez paper. What they found, in summary, is that while the “smart beta” approach may have some merit for very simple strategies, such as merely balancing a large-cap exchange-traded fund with a mid- or small-cap exchange-traded fund, more sophisticated strategies of this class (which are typically the result of large-scale computer-based searches for an optimal parameter selection) fall prey to backtest overfitting.

These overall results are entirely consistent with the results of a paper by the present authors, which has found that backtest overfitting is remarkably easy to occur in any investment strategy that was designed using computer-based searches over a large parameter space to find an “optimal” design (as is certainly done in many smart beta strategies).

These results are also consistent with an separate paper by the present authors on stock fund weighting schemes. We demonstrated that it is relatively easy to design a stock portfolio, consisting only of weighted S&P 500 index stocks, that will achieve virtually any desired performance profile, based on backtests. Such portfolios, however, typically do much worse on new out-of-sample data, a symptom of serious backtest overfitting.

Backtest overfitting is not a minor flaw. If an overfit strategy is implemented, it may well result in disappointing returns or even catastrophic loss of capital. Thus it behooves all who design such strategies to ensure that their design is not overfit.

]]>For most other hedge funds, it has been a different story, namely year after year of subpar performance. For example, the HFRI Equity Hedge (Total) Index of U.S. equity

Continue reading How much of hedge fund profits are taken by management?

]]>The past few years have been rough on hedge funds. Some have done very well, such as the Medallion Fund, a highly mathematical quant fund operated by Renaissance Technologies. It has produced returns averaging over 30% since its founding in 1988, totaling approximately $55 billion in profits, making multimillionaires of many of its very fortunate investors, who for the most part are professional mathematicians and others employed by Renaissance.

For most other hedge funds, it has been a different story, namely year after year of subpar performance. For example, the HFRI Equity Hedge (Total) Index of U.S. equity hedge funds was up only 4.85% as of the end of November 2016 (from 1 January 2016), with an annualized return over the past 5 years of only 5.15%. This compares to a 12.06% annualized return for the S&P500 index (14.30% with reinvested dividends) over the same five-year period ending 30 November 2016.

Not surprisingly, many hedge fund clients are not happy. In 2014, the California Public Employees’ Retirement System (CalPERS), which is the largest U.S. pension fund, announced that it would completely exit its USD$4 billion investment in hedge funds. In April 2016, the New York City public pension fund announced that it will also exit all hedge fund investments. Finally, in December 2016 the New Jersey Investment Council voted to cut in half its USD$9 billion hedge fund holdings.

In total, over USD$51 billion was withdrawn from U.S. hedge funds over the first nine months of 2016. Similar troubles have beset hedge funds in Europe and Asia.

In the wake of these withdrawals, many hedge funds have closed. In the first quarter of 2016, more U.S. hedge funds were liquidated than started. Tudor Investments, led by billionaire Paul Tudor Jones, laid off 15% of its workforce, in the wake of over USD$2 billion of investor withdrawals. Howard Fischer, head of Basso Capital Management, lamented “It’s miserable, miserable.” George Papamarkakis, head of London’s North Asset Management (which declined roughly 10% in 2016), lamented “There’s gloom everywhere.”

Not surprisingly, hedge fund managers’ woes have elicited little sympathy from clients. As New York City’s Public Advocate Letitia James told members the board, “Let them sell their summer homes and jets, and return those fees to their investors.”

Many are questioning the traditional hedge fund fee structure, typically a “2-20” rule: a 2% annual fee, plus a performance fee of 20% on any profits. These fees are, of course, far higher than the fees typically charged by conventional mutual funds, not to mention the rock-bottom fees of popular index-tracking mutual funds and exchange-traded funds (ETFs). BlackRock’s iShares IEUR ETF, which tracks the MSCI Europe IMI index, charges an annual fee of 0.10%. Vanguard’s VOO ETF, which tracks the S&P500 index, charges 0.05%. Neither ETF has a performance fee.

But as a recent article in the New York Times points out, effective hedge fund fees are often even higher than the 2-20 rule would lead one to believe, due to quirks of accounting. For example, the NY Times article cited a letter to investors by Bill Ackman, who disclosed that in the four years since his Pershing Square Holdings fund was formed, it had gained a total of 20.5% (decidedly mediocre given that the S&P500 gained 67% over this period) before fees, but only 5.7% after fees. In other words, over the four-year period, the fund management kept approximately 72% of the fund’s profits.

The difficulties can be illustrated by considering the fate of a USD$1 million investment in a fund that generates 10% in the first two years, and then loses 5% in the next two years. Before fees, the investment would be USD$1.092 million at the end of the fourth year, for a 9% gain. But after deducting a 20% performance fee in the first two years (and zero performance fee for the last two years), the investment would be at USD$1.053 million, or in other words a total return of just 5.3%.

But even this calculation, which was cited in the New York Times, does not include the 2% annual fee assessed by many hedge funds. If one recalculates the above with a 2% annual fee deducted after each year’s close, then the result is as follows: $1.058 million after year one; $1.119 million after year two, $1.042 million after year three, and $0.970 million after year four. In other words, although the investment gained a profit of 10% before fees, the 2-20 fees erased all gains and left the client with a 3% loss.

In light of these experiences, many, both within and without the hedge fund industry, are questioning the traditional 2-20 fee structure. Some funds have already reduced the 2% annual fee to as low as 0.7%. Others in the industry are arguing that a hedge fund should not assess any fee whatsoever in years that it loses money, and only a minimal fee if it fails to meet its benchmark. One way or another, the hedge fund industry is changing before our eyes.

As we have pointed out in a previous Blog, these results should not come as a surprise. After all, markets by definition incorporate the collective judgments of many thousands of highly trained market analysts worldwide, who employ sophisticated mathematical algorithms running on powerful supercomputers to ferret out any regularities or correlations, and then use high-frequency trading algorithms to act on these phenomena in real time (not to mention millions of other corporate and individuals investors). The efforts of these many players largely cancel each other out, leaving a time series that is little more than a random walk.

It could be argued that some pockets of opportunity existed in the 1980s and 1990s, when computer technologies were scarce and some agents were more informed than others. But in today’s world, those informational asymmetries are largely absent, and so are most of the easily exploitable opportunities. As George Papamarkakis of London’s North Asset Management explains, “There are only so many market inefficiencies out there to profit from.”

Along this line, it should also not come as a surprise that the few remaining hedge funds that do make a respectable gain above the larger markets tend to rely on highly sophisticated mathematical algorithms, along with massive computer databases and processing power. The Renaissance Medallion Fund, mentioned above, reportedly employs very sophisticated techniques, although the details, as one might image, are carefully guarded secrets. A few other funds also do well, but these might be the effect of survivorship and selection bias, rather than management skill.

In any event, let us not be surprised that even relatively sophisticated hedge funds have difficulty achieving consistently above-market returns. After all, they are betting against a signal that is almost entirely random noise, and then charging a premium fee.

]]>A Mathematician’s Apology (1941). G.H. Hardy (1877-1947)

Mathematics as the great equalizerWhat do many of the most successful (and richest) hedge fund managers have in common with a life-long homeless person? That they have worked together, on equal terms, in solving some of the hardest mathematical questions.

Yes, in a stratified

Continue reading Erdős Numbers: A True “Prince and the Pauper” story

]]>*I still say to myself when I am depressed and find myself forced to listen to pompous and tiresome people: “Well, I have done one thing you could never have done, and that is to have collaborated with Littlewood and Ramanujan on something like equal terms.” *

*A Mathematician’s Apology (1941). **G.H. Hardy (1877-1947)*

What do many of the most successful (and richest) hedge fund managers have in common with a life-long homeless person? That they have worked together, on equal terms, in solving some of the hardest mathematical questions.

Yes, in a stratified world with increasing wealth inequality, gated communities and elite clubs, where the ultra-wealthy rarely interact with the poor, many of the richest people on Earth are mysteriously connected to a vagabond who played a critical role in their careers. But in this story the pauper didn’t want to become a prince. In fact, the pauper methodically donated all the money he ever received for his achievements.

Paul Erdős (1913-1996) was one of the most prolific mathematicians in history. Over his lifetime, he published a staggering 1,525 journal articles and collaborated with 511 colleagues, all without institutional support or holding an academic position. Paul carried two half-empty suitcases around the world, which contained all his earthly possessions. He would stay in a co-author’s house for a few weeks per visit, just enough time to make a new mathematical breakthrough, before moving on to the next fortunate host. One of his mottos was “another roof, another proof.”

*Paul Erdős inspired generations of mathematicians, of all ages. In this 1985 photo, Erdős explains a problem to Terence Tao, who was only 10 years old at the time. Tao received the Fields Medal in 2006*

Paul Erdős made important contributions to a wide range of subjects, and was awarded the prestigious Wolf Prize (1983/84) among a long list of honors. However his place in mathematical history is well secured for another exceptional reason. His most important “proof” is perhaps that mathematicians can work effectively through vast collaboration networks. Until recently, mathematics was essentially a “lone wolf” quest. Collaboration was rare, and authors tended to hide zealously their discoveries until they were ripe for publication (think of Newton and Leibniz). Today, mathematicians are interconnected, coordinate efforts and borrow ideas from each other, sometimes devoting hundreds of brains to proving the same conjecture. Can dozens of good mathematicians work together like a great one? Surprisingly, the answer is yes! Visit the Polymath project for astonishing examples of discoveries involving hundreds of co-authors.

Erdős was involved in so many different mathematical endeavors, that some colleagues humorously wondered, who is not somehow working with him? Hence the idea of “Erdős number” (E#) was born. E# is the length of the shortest collaboration path between a mathematical author and Erdős. If David has co-authored a paper with Erdős, David has E#1. If Mark has co-authored a paper with David, Mark has E#2 (or E#1, if Mark co-authored directly with Erdős), and so on. If there is no collaboration path between Peter and Erdős, Peter’s E# is infinite. The American Mathematical Society offers a free online tool that computes E#, and Oakland University’s Erdős Number Project hosts a wealth of resources.

This notion of E# first started in the year 1969 as a tribute to Paul Erdős by colleagues and friends, without envisioning it as a research tool. However, over the decades academic studies on E# distributions have found interesting features about how mathematical discoveries take place. Out of the ~401,000 mathematical authors (since the 1930s) listed in MathSciNet, over 67% have a finite E#. Being listed in that authors’ catalog is in itself an achievement, because it means that the researcher made a mathematical discovery that was peer-reviewed and published in a journal indexed by MathSciNet. Also, note that ~401,000 authors over roughly 90 years is a relatively small number, considering the millions of people who receive a STEM degree *every year*. The median E# across these accomplished mathematicians is 5, with an upper bound of 13. Every winner of the Fields Medal has a finite E#, with a median of 3, a statistically significant divergence from the overall population. Every winner of the Abel Prize has a finite E#, with a median of 3 and an average of 2.94. Every winner of the Nevanlinna Prize has a finite E#, with a median of 2 and an average of 2.4.

About 80% of Nobel Laureates in Physics also have finite E#, including Albert Einstein (E#2), Enrico Fermi (E#3), Wolfgang Pauli (E#3), Max Born (E#3), Richard Feynman (E#3), Hans Bethe (E#3), etc. Srinivasa Ramanujan has an E#3, albeit Erdős was only 7 years old when the Indian genius tragically died. That is because G.H. Hardy, Ramanujan’s mentor and co-author, has an E#2. Good mathematicians tend to collaborate with other good mathematicians. Like in sports, the better a “player” becomes, the more he is pursued by colleagues who want him to join an existing team, and work together on a challenging problem. Strong teams do not chase weak players, and strong players do not join weak teams. As the collaboration network becomes denser around great mathematicians, the chances to connect with the center (E#0) grow dramatically.

*Good mathematicians tend to work with other good mathematicians. There is often a collaboration path between any two great mathematicians, regardless of age, nationality, ethnicity or social background… even spanning centuries. The figure above shows a path connecting Sir Andrew J. Wiles (E#3) with Ramanujan (E#3), through an all-star sequence of Fields medalists and Fellows of the Royal Society*

Does this mean that E# measures mathematical prowess? Not necessarily. If we search hard enough, we may find highly-cited mathematicians who have made breakthroughs on their own. It is fair to say that E# measures a combination of these two ingredients: *Proficiency *and* collaborative skills*. A skillful mathematician working alone will have an infinite E#, and a very social but mediocre mathematician will have a hard time publishing in a MathSciNet-indexed journal, plus finding co-authors in the proximity of Paul Erdős.

A large portion of the most successful investment funds in history are led by mathematicians. Many of those “quant” firms have been either founded or managed by mathematicians with low E#. The following table provides an incomplete list.

Name |
Firm |
Position |
E# |

Elwyn R. Berlekamp | Axcom Trading Advisors | President | 2 |

Peter J. Weinberger | Renaissance Technologies | Managing Director | 2 |

James B. Ax | Axcom Trading Advisors | Founder, CEO | 3 |

Leonard E. Baum | Axcom Trading Advisors | Chief Scientist, Founder | 3 |

Jean-Philippe Bouchaud | Capital Fund Management | Chairman | 3 |

James H. Simons | Renaissance Technologies | Founder, CEO | 3 |

Vincent Della Pietra | Renaissance Technologies | Managing Director | 3 |

Glen T. Whitney | Renaissance Technologies | Managing Director | 3 |

Marc Potters | Capital Fund Management | Co-CEO | 4 |

David E. Shaw | D.E. Shaw | Founder, CEO | 4 |

Anthony W. Ledford | Man AHL | Chief Scientist | 4 |

Stephen Della Pietra | Renaissance Technologies | Managing Director | 4 |

Lalit R. Bahl | Renaissance Technologies | Senior Researcher | 4 |

Alexey V. Kononenko | Renaissance Technologies | Managing Director | 4 |

Alexander Astashkevich | Renaissance Technologies | Senior Researcher | 4 |

Alkes L. Price | Renaissance Technologies | Managing Director | 4 |

Nicholas Patterson | Renaissance Technologies | Managing Director | 4 |

Peter Brown | Renaissance Technologies | Co-CEO | 4 |

John Overdeck | Two Sigma | Founder, Co-CEO | 4 |

David J. Hand | Winton Capital Management | Chief Scientist | 4 |

Neil A. Chriss | Hutchin Hill | Founder, CEO | 5 |

Robert B. Litterman | Kepos Capital | Chairman | 5 |

Alexander Belopolsky | Renaissance Technologies | Senior Researcher | 5 |

Robert J. Frey | Renaissance Technologies | Managing Director | 5 |

Of course one reason for this is, sophisticated mathematical models help beat the collective wisdom of the markets. But as explained earlier, E# measures more than mathematical prowess. It also accounts for collaborative skills. Mathematicians that are used to work as part of a research network exhibit behavioral traits that should be useful in the corporate world. Cracking the market’s fortune formulas is definitely a complex team effort that requires dozens of individuals working together, complementing each other. This combination of collaborative skills with mathematical proficiency, as measured by E#, is the stronger predictor of investment performance.

Collaborating effectively is not easy in mathematics, and should not be discounted as a common skill. It requires being able to share complex ideas in a respectful, objective and rigorous way. If understanding advanced math is hard enough, understanding cutting-edge in-the-making math is quite a feat. In addition, brilliant people often prefer to pursue their own ideas rather than embrace someone else’s in a collaborative effort. The ability to lead and persuade others to join an obscure intellectual struggle is rather uncommon.

As it is often the case in mathematics, the concept of E# was proposed without ever suspecting its future applications. The inventor of E# could not have envisioned that this notion would connect some of the richest people in the world, the princes of Finance, with good old Paul Erdős. As for the pauper of this story, he had the traits that make a great prince … if only he cared about earthly matters.

]]>Lopez de Prado argues that while economics is arguably one the most mathematical of the social sciences, the mathematical methods of economists may not be up to the task of modeling the complexity of the social institutions and the business/finance world. Outdated and inappropriate statistical methods are of particular concern, with economists and econometricians often drawing very dubious conclusions from the available data.

The

Continue reading Mathematics and economics: A reality check

]]>Lopez de Prado argues that while economics is arguably one the most mathematical of the social sciences, the mathematical methods of economists may not be up to the task of modeling the complexity of the social institutions and the business/finance world. Outdated and inappropriate statistical methods are of particular concern, with economists and econometricians often drawing very dubious conclusions from the available data.

The author suggests that graph theory, topology and even information theory and signal processing may be significantly more appropriate for these models. Machine learning methods may also be useful here.

Another issue is the inappropriate utilization of experimental methods, in particular the backtest. As we have demonstrated in several recent papers, backtest overfitting, selection bias and other errors are increasingly common in the field. In fact, in a time when one can write simple computer programs to explore millions or billions of variations of a proposed financial strategy, testing each based on historical backtests, and then only selecting the very best option, then that optimal variation is virtually certain to be statistically overfit. And if such a strategy is actually deployed, then the results could be disastrous.

Lopez de Prado then suggests that perhaps financial academics should commit a portion of their salaries to a validation of their proposed strategies and analysis — a documented track record…

Full details are in the published article.

]]>It is worth taking a brief look at the mathematics behind blockchain. The following is based in part on an article by Eric Rykwalder, one of the founders of Chain.com, a startup blockchain software firm in San Francisco.

The elliptic curve digital signature algorithmBlockchain is basically a publicly available ledger where participants enter data

Continue reading The mathematics behind blockchain

]]>In a previous Math Investor blog, we described the emerging world of blockchain, emphasizing how it might impact the financial services and investment world. Already numerous firms, including several startup organizations, are pursuing blockchain to facilitate and streamline many types of financial transactions.

It is worth taking a brief look at the mathematics behind blockchain. The following is based in part on an article by Eric Rykwalder, one of the founders of Chain.com, a startup blockchain software firm in San Francisco.

Blockchain is basically a publicly available ledger where participants enter data and certify their acceptance of the transaction via an elliptic curve digital signature algorithm (ECDSA). An elliptic curve is an equation such as y^{2} = x^{3} + a x + b. In Bitcoin and most other implementations, a = 0 and b = 7, so this is simply y^{2} = x^{3} + 7 (see graph). Elliptic curves have numerous interesting properties, such as the fact that a nonvertical line intersecting two nontangent points will always intersect a third point on the curve. Indeed, one can define “addition” on the curve as finding that third point corresponding to two given points. This is basically what is done in ECDSA, except that the operations are performed modulo some large prime number M.

In particular, in ECDSA, addition of two points (p1,p2) and (q1,q2), and the doubling of (p1,p2), are performed as follows:

Addition of (p1,p2) and (q1,q2):

c = (q2 – p2) / (q1 – p1) mod M

r1 = c^{2} – p1 – q1 mod M

r2 = c (p1 – r1) – p2 mod M

Doubling of (p1,p2):

c = (3 p1^{2}) / (2 p2) mod M

r1 = c^{2} – 2p1

r2 = c (p1 – r1) – p2

Some readers will note that “division” is indicated in the first line of each algorithm. What this means is the product, modulo M, of the expression to the left of the slash by the multiplicative inverse of the expression to the right of the slash. Since M is a prime, every nonzero integer from 1 to M-1 has a multiplicative inverse. For example, the multiplicative inverse of 5 mod 17 is 7, because 5*7 = 35 = 1 mod 17; in other words, 5^{-1} mod 17 = 7. In practice, these inverses are rapidly calculated by means of the Euclidean algorithm, where one accumulates the divisors in a particular way. See a note by Nick Korevaar for some examples.

One other preliminary detail is how to “multiply” in this algebraic structure, in particular to calculate an expression such as m * (p1,p2) for some integer m. This can be done by first doubling the input (p1,p2), and then using the addition algorithm repeatedly until m copies of (p1,p2) have been added, but this of course is not practical when m and M are very large, as they are in real blockchain applications. Instead, such “multiply” operations are typically done using the binary algorithm for multiplication, which we will sketch here for ordinary integers but which can be easily adapted to ECDSA:

To compute r = n * b mod M: First set t to be the largest power of two such that t is less than or equal to n, and set r = 1. Then perform:

A: If n is greater than or equal to t, set r = b + r mod M, and set n = n – t; else set t = t / 2.

B: If t is greater than or equal to 1 set r = 2 * r mod M, and go to A (if t < 1, then we are done).

Computer scientists will immediately recognize this as almost identical to the binary algorithm for exponentiation modulo M, except that we are adding and doubling instead of multiplying and squaring. It is implied, for example, when one writes 3^{17} mod 10 = ((((3^{2} mod 10)^{2} mod 10)^{2} mod 10)^{2} mod 10) * 3 mod 10 = 3, thus performing the exponentiation in only five multiplications mod 10 instead of 16 or 17.

Now we may state the ECDSA algorithm (except that we omit some relatively minor details that apply mainly to real-world implementations):

First, select a modulus M, a “base point” (p1,p2), and a private key k1 (integer between 1 and M-1). These are typically selected such that the order of the base point (namely the maximum number of times (p1,p2) can be added to itself before the addition formula above fails due to zero divide) is prime and at least as large as M (this is not required but is normally done, and with this assumption the algorithm below is simpler). This often takes some experimentation, although practical applications can do this very rapidly.

As a concrete example, let us take M = 199 (which is prime), and the base point (p1,p2) = (2,24). For this M and (p1,p2), one can calculate that the order n = 211. Then let us select as our private key k1 = 151. We first need to calculate the public key (r1,r2) corresponding to the private key. This is done by multiplication:

(r1,r2) = k1 * (p1,p2)

where again the multiplication is done either by repeated summation or by the binary algorithm above. If we do this, we find that the public key (r1,r2) = (64,80).

Now select some data z1, say z1 = 104. We shall construct a digital signature of the data. This is done as follows:

1. Choose some integer k2 between 1 and n-1, where n is the order.

2. Calculate (s1,s2) = k2 * (p1,p2). If s1 = 0, return to step 1.

3. Calculate s2 = (z1 + s1 * k1) / k2 mod n. If s2 = 0, return to step 1.

Then the digital signature is (s1,s2). In our specific case, if we select k2 = 115, we calculate (s1,s2) = (99,52).

Now we can test the digital signature, as a third party might to verify that the transaction (which in this example we presume is coded in the data z1 = 104) is valid. This is done as follows:

1. Calculate u1 = s2^{-1} mod n

2. Calculate u2 = z1 * u1 mod n

3. Calculate u3 = s1 * u1 mod n

4. Calculate (t1,t2) = u2 * (p1,p2) + u3 * (r1,r2)

5. Verify that t1 = s1.

In our case, we find that the result of step 4 is (t1,t2) = (99,44). Since t1 = 99 = s1 (see above), the validity of the signature is confirmed (it is not necessary for t2 to equal s2).

We should emphasize that our example involves extremely modest-sized integers. In a real Bitcoin or blockchain application, these integers are typically 256 bits long, dramatically increasing the cost of performing the above operations, but, on the other hand, very dramatically increasing the cost required for someone to “break” the system, such as by computationally attempting to recover the private key from the public key.

So what conclusions can we draw from this exercise? First of all, it should be clear that the mathematics involved is not trivial, and the necessary computations to implement this scheme certainly are not trivial. Nonetheless what we have is an effective one-way function: it is relatively easy to verify a signature, but it is very difficult to work back from publicly available data, such as the public key, to obtain the critical private key.

ECDSA is the essence of how both Bitcoin and other blockchain applications work. The scheme has resisted some rather extensive testing for weaknesses, both mathematically and computationally. The few failures that have occurred in practice have generally been because users were not careful in protecting their private keys, or else they used a fairly standard pseudorandom number generator to produce the private keys, which attackers then exploited.

As with all the technology we rely on in our digital age, the weakest links are users who are not careful.

]]>