Saturday, March 14, 2009

Survivorship Bias - Is the Index indicative ?

It was with much interest that I read about the recent revamp of the Straits Times Index (FTSTI) in March 2009 and that the Index would be reviewed and revamped twice a year in March and September to ensure a fair and true representation of stocks listed on the Stock Exchange of Singapore. The latest revision removed 2 property companies - Yanlord Land and Keppel Land; while adding two transport operators - ComfortDelgro and SMRT. Interestingly enough, the reason given was that property was in a slump and that the market capitalization of Yanlord and Keppel Land was lower than that of the 2 transport companies, thus removing them from the index would ensure a better representation as transportation was seen as a more "resilient" industry.

This brings us to the very obvious conclusion that the Index seems to be "revamped" every other time such that the components are switched for other companies. The issue of survivorship bias will certainly be raised - this bias can be explained by the fact that the index continues to persist and climb higher over time not because shares on the whole keep rising, but because the components in the Index are substituted with stronger companies over time, while leaving out the laggard companies which do not perform. This is especially pervasive if you consider the fact that bellweather companies such as Creative used to make up the index until they were dropped due to declining sales and profits.

To cut a long story short, what I am trying to say is that if one uses an Index as a benchmark for stock market performance, one is likely to get results which are highly skewed due to this bias. When weak companies are dropped in favour of good ones, isn't it very obvious even to the layman that the index will continue to rise in future instead of stagnating or even falling behind over time ? Thus, the many articles written about stock markets always returning about 6-8% per annum may not be totally 100% true after all, if we account for this bias and the fact that the index is simply NOT representative from one period to another. For simplicity sake, I will confine my discussion to the Singapore Stock Market as some may point out that the Dow Jones Industrial Index (DJIA) had so many switches in its components since its inception that the Index is almost a totally different animal today than it once was 70 years ago.

This begs the question - how representative is an Index of the broad market and should investors rely on such indices to gauge market sentiment and relative price levels ? Some points to ponder on:-

1) The high weightage of the three local banks (DBS, UOB and OCBC) plus SingTel skews the Index towards these 4 companies. In fact, the rise and fall of just these 4 companies can have a drastic and sharp effect on the Index even though other index components ma not fluctuate much. My argument here is that the weightage is skewed too much towards financial stocks currently.

2) The usage of just 30 stocks to represent a universe of about 700+ companies seems to undermine the representative strength of the index. In the USA, though the DJIA is the bellweather Index which is still used by a broad array of investors, there are also two other important indices - the NASDAQ and the S&P 500. Of the 2, the S&P 500 is considered to be the most representative index because it includes 500 companies in it - this obviously makes it more representative than the DJIA which only includes 30 blue-chip companies (some of which are not so "blue-chip" any longer). My suggestion is for Singapore to have such an index incorporating about 40-50% of the companies on the 2 Boards, in order to show a more representative sample.

3) There are many other indices as mentioned by SGX in their press release some days ago, such as the FTSE ST Mid-Cap Index, Small-Cap Index, Fledgling Index and even an ST China Index. However, all these indices are not prominent enough and are usually not even mentioned by commentators or news reports (with the exception of the China Index since S-Shares seem to be taking a severe battering these couple of weeks). I was hoping SGX could come up with more useful indices which are better recognized rather than the existing ones which seem (to me) to be rather defunct.

4) How are dividends accounted for in terms of computing the returns on an Index ? Besides the fact that an index suffers from survivorship bias, another problem is also that of dividends which are declared and paid out by index constituents over the years. Are these accounted for by the Index ? I doubt so. There are researchers, however, who assume the dividends are reinvested into the Index and compute a median return for that. Suffice to say that such literature remains buried amongst the scholarly and those who study the stock market for a living, and has yet to be mentioned in a mainstream media article or given due attention.

All the above seek to highlight the problems with STI and indices in general. I would appreciate opinions on this issue as well as it is also quite a new area for me to blog on, and I will approach this subject again in the near future should I discover new insights.


Cheng said...

Indexs are good benchmarks to measure performance. They kick out the weak and include the strong. Most investors should do that, to me it's prudent investing in this way of allocating capital.

That is why Warren Buffett said that most investors would be better off buying indexs if they are not willing to sacrifice time, effort and money to learn how to invest prudently and intelligently.

Investing need lots of patience and careful research.


la papillion said...

Hi mw,

I would take the re-constitution of the sti to include the stronger companies as a plus point. I say that because index is the only instrument that I can safely say that one can buy and hold forever. Isn't it fantastic that we can invest in such an instrument that revamps itself from time to time, presumably stronger every time it changes?

Another thing about the're right. It's hard to account for it, since there's nothing that I read before that talks about the returns after taking into account the dividends declared.

floatingrok said...

I agree that STI Index doesn't reflect well the movement of the whole market, and the index is schewed highly to certian big shares.

1) The rationale for the existence and purpose of an index is that it is representive of the market. To achieve that, the change of components is necessary, but the change needs to be minimum and reflect the capitalisation of that industry of the whole stock market. If banking and real eastate accounted for 20% and 30% of whole market cap, it is not right to have banking shares in the index that represent 30% (just an example, not the exact number), and real eastate companies were taken out just because of the poor performance. However, after taking out Yanlord and so on, real eastate comanies still have sufficent weights to represent the share of this industry, then should be righ for such change, but the reason to take them out is unacceptable.

2) Another point, it is better to have an index that has a long history and is relatively stable for its components. It is not likely / right to change suddenly from 30 compoents to 100 components.

3)I believe that the dividends have been taken into account into the calculation. Don't be wrroied about this. Just take a look of the mannual and formula of any index. It will tell.

musicwhiz said...

Hi Cheng,

I agree with you that we should weed out the weak and retain the strong in our own portfolios, but this doesn't answer the question of whether the Index is representative due to survivorship bias. I am talking about comparability across periods.

As with WB's quote, yes I do agree with him too. I tried to research and invest on my own and it's been tough, to say the least. I have yet to experience a positive return though of course my time frame thus far has been less than 3 years since I started value investing.


musicwhiz said...

Hi LP,

You've got a good point there ! That's assuming you buy the index - it's sort of like a never-die kind of thing. Reminds me of Jason in Friday the 13th though. Eerie. Haha...

As for the dividends, floatingrok has come out to say that the Index does take dividends into account, so question answered !


musicwhiz said...

Hi floatingrok,

1) Noted and I agree with you. We should see the weightage for industries in the Index as well.

2) I am not proposing replacing the STI, more like augmenting its usefulness by coming up with a separate index which includes more companies. An example is DJIA and S&P 500 co-existing side by side.

3) Thanks for clarifying on this point !


8percentpa said...


Dividends are accounted for when calculating index returns. Over a 5-10 yr period, dividend yield is about 3%. We know that stocks on average give 10% annual return, so dividend makes up 30% of this 10% return. Not a small amount!

Yes survivorship bias is definitely in indices. That is why they are so hard to beat. And that's why it makes sense for most retail investors to invest in them. They earn better return without paying more (through index funds or ETFs which will track and follow all changes)

If you look at LT charts of S&P and Dow, they look quite similar. ie 30 stocks actually do give a good representation of 500 stocks. To borrow the term from Nassim Taleb, this is an effect of Extremistan e.g. Top 1% of human population accounts for 90% of the world's wealth, or top 20 best-selling books' sales accounts for 60% of the revenue of all books sold etc.

Hope this helps! Cheers!

musicwhiz said...

Hi 8percentpa,

I see, thanks a lot for the info. I really needed that to bolster my knowledge on this area.

Take Care and Regards,

BL Trading Team said...

True, seems like a fund manager managing the index.

Anyway my financial model tells me that STI will touch 1200, only a matter of time. So I will not touch stocks for now.

musicwhiz said...

To Brenden Lee,

OK, let's both wait for STI 1,200 then. I have no problem doing that !


WY said...

Hi 8percentpa,

Thx for the info.

Some questions on the following,
1. "Dividends are accounted for when calculating index returns."
How do they do that ? is it by considering total dividends be reinvested into the index ? at which period's price ?

2. "Over a 5-10 yr period, dividend yield is about 3%. We know that stocks on average give 10% annual return"
Is this data for STI ?

musicwhiz said...

Hi WY,

I guess they have ways of incorporating the data by using some (arcane) formula. Don't under-estimate the power of mathematicians and number crunchers haha, I've seen a number of them on forums and they scare me to bits !

I think stocks on average should give a return of 5-8%, dividends included. Anything more than that is supernormal and is Buffett-worthy !


WY said...

Thx musicwhiz.

It can be done. Shouldn't be difficult but rather whether it's done or not and how's it's done.

For example, let's say receives dividend from SembCorp on 31 March, from DBS on 31 July.

Is it to reinvest the SembCorp dividend by buying proportionate index component stocks on 31 March & likewise for DBS's dividend on 31 July ?

Or accumulate the dividends until end of the Fiscal Year, then the lump sum used to buy component stocks ?

Curious to know the industry practice :)

musicwhiz said...

Hi WY,

Not too sure about that, sorry. Perhaps another reader may wish to help you with that.


8percentpa said...


The price charts that we usually get from Yahoo or Bloomberg etc does not factor in dividends.

However, when calculating returns of indices, most studies would have taken dividends into account.

The 10% return and 3% yield is for US mkt (ie S&P). Singapore's stock mkt history is about 30 yrs, which is relatively short. Depending on which 5 yr period or 10 yr period you take, the results are vastly different.

I have not done a thorough study though.

8percentpa said...

Just realized I havent fully answered the qns.

Say I wanted to calculate the return of the STI index from 2003 to 2008. It would be

STI price in 2008 - STI price in 2003 + dividends received from 2003 to 2008 (adjusted to same denominator for STI price)

And calculate the CAGR for the whole thing to get the annual rate of return.