Monte Carlo Sensitivity Analysis - Probability of success

Post questions about how to use the planner, user inputs, how the planner works, and comments and suggestions.
Post Reply
herbsims
Posts: 5
Joined: Mon Sep 17, 2018 10:25 am

Monte Carlo Sensitivity Analysis - Probability of success

Post by herbsims »

OK, I am not sure why the simulations are doing this.

When I run the sensitivity analysis with the output being the Probability of success goes DOWN as my Portfolio RETURNS go UP? In my mind, if I have a return of 2% then my probability of success is 100%, but if my return is 6% then I should have a greater probability of success since I am earning more interest and thus impacting my 401(k) less. But the heat map shows LESS probability of success for 6% than when I have 2%. I've attached a screenshot of the heat map for clarity.

What am I setting up wrong to get an "inverted" heat map? I've attached a typical heat map so show what I am talking about.

Thanks in advance!
Attachments
Screen Shot 2018-09-17 at 9.32.15 AM.jpg
jimr
Posts: 821
Joined: Thu Feb 28, 2008 6:48 pm

Re: Monte Carlo Sensitivity Analysis - Probability of success

Post by jimr »

I may be missing something, but if this graph is telling the story I think it is, it's a fantastic example of the downside of taking on more risk than needed in your retirement portfolio.

The key to this is the fact that higher return goes hand-in-hand with higher volatility of returns. This higher volatility increases the fatness of the returns distribution, which that means many more 'bad luck' outcomes on the left tail of the returns distribution.

If you look at the individual runs, you'll probably find that the runs on the right side of the heatmap have a much higher median ending portfolio value compared to the runs to the left (at the same spending level).

So even though the probability of success is going down as you move from left to right, the median ending portfolio value is probably going up. What this means is that a higher return on average (really median) yields a better result, but that higher median result comes at a cost of a greater potential for bad luck at the tails.

It might be interesting to set a fixed spending amount and run the sensitivity analysis with the range of returns and standard deviations specified separately. You'll probably notice that as long as you keep the max standard deviation down pretty low, the probability of success won't decline much. But as you increase the max standard deviation, all else equal, you'll start to see lower and lower probabilities of success as the standard deviation of returns (eg volatility) increases by more than the benefit from the higher return.
herbsims
Posts: 5
Joined: Mon Sep 17, 2018 10:25 am

Re: Monte Carlo Sensitivity Analysis - Probability of success

Post by herbsims »

"It might be interesting to set a fixed spending amount and run the sensitivity analysis with the range of returns and standard deviations specified separately. "

So what typical values would be input for the returns and SD? I am assuming a 2-4% return (conservative), so would I also be conservative on the SD as well?

How is this done?
jimr
Posts: 821
Joined: Thu Feb 28, 2008 6:48 pm

Re: Monte Carlo Sensitivity Analysis - Probability of success

Post by jimr »

OK. I missed that you changed the default max return in sensitivity analysis from 12% down to 6% without adjusting the max standard deviation downward. That's why the "volatility" impact I mentioned above (as the return increases) was so strong and looked inverted to you.

The way sensitivity analysis works with the parameters you were using, the returns and standard deviations are varied together between their min values and max values on the x-axis while the spending amount is varied on the y axis.

So for example, in your screenshot, the std dev that got matched with 3% return was 2% and the std dev that got matched to the 6% return was 18%. The return/std dev pairs used for the rest of the x axis between these two endpoints are just interpolated between these extremes.

So what I said above is still true, it's just that you magnified this impact by adjusting the return down without adjusting the max standard deviation (volatility) down.

If you cut the max standard deviation from your screenshot above in half and run it again, you'll likely see the same trend but it probably would be much less dramatic.

Of course, my main point above still holds. We only have history as a guide for which standard deviations we should assume for a given rate of return. The future might be better or it might be worse.

So in general it's safest to take only the amount of risk you need for your plan to work. If you take extra risk trying to get a higher return, that may result in excessive volatility which would negatively impact your chances for success.

Also, your question above about which standard deviation value you should map to a given return value is a tough one to answer. People usually just use historical results as a best guess, but it's important to understand that's all it is - a guess.

Jim
herbsims
Posts: 5
Joined: Mon Sep 17, 2018 10:25 am

Re: Monte Carlo Sensitivity Analysis - Probability of success

Post by herbsims »

To comment on the standard deviation. One sigma only captures 65% of the random possible samples, 2 sigma captures 95%, and 3 sigma captures 99.7% of the possible data.

So why wouldn't I want to use a higher SD to make the simulation more "real world"?

For example, the funds that I am tracking has a the following historic values:
Avg 4.77% 6.87% 7.14% 7.90% 12.25%
1 Sigma (65%) 3.37% 9.91% 12.05% 13.78% 8.52%
2 Sigma (95%) 6.73% 19.83% 24.10% 27.55% 17.05%
3 Sigma (99.7%) 10.10% 29.74% 36.14% 41.33% 25.57%

So to really get a good handle on the funds why wouldn't I use the 3 sigma value?

Thanks,

Herb
Last edited by herbsims on Wed Sep 19, 2018 9:16 am, edited 1 time in total.
jimr
Posts: 821
Joined: Thu Feb 28, 2008 6:48 pm

Re: Monte Carlo Sensitivity Analysis - Probability of success

Post by jimr »

We might be getting terms confused.

The SD value that you enter along with the return always represents a 1 sigma value. Associating a higher SD with a given return is basically making the return distribution wider (fatter) which means that there's a greater absolute amount of variation around the mean. A higher SD doesn't mean a higher sigma level in terms of the confidence in your estimate. Specifying the average return and standard deviation just defines the return distribution for the simulation model to use.

On the other hand, the probability of success that the simulation produces can be interpreted in terms of sigma. Most people would not be comfortable with a plan that offers just 1 sigma of confidence (eg probability of success at 67%). Instead they'd like something closer to 2 or even 3 sigma.

That said, it's important to keep in mind that the results produced by the simulation have a lot of error in them that's unavoidable because of the uncertainty in the inputs. No one really knows what the distribution of investment returns really is/will be. So just that one estimate is riddled with errors that can't be reduced. To make things worse, we're stuck with big estimation errors in inflation, tax changes, planned expenses, and so forth.

Some experts go as far as saying that the inherent error in a Monte Carlo or any other retirement planning approach is at least 20%. That means the difference between an 80% result and a 100% result is not statistically significant.
jimr
Posts: 821
Joined: Thu Feb 28, 2008 6:48 pm

Re: Monte Carlo Sensitivity Analysis - Probability of success

Post by jimr »

I think we might have been each editing our posts at the same time. I think the first paragraph above is probably most responsive to your question.

The key is that the simulation assumes a normal distribution of returns. To fully define a normal distribution requires just two parameters, the average return and the SD. Those two parameters precisely define the shape of the bell curve that represents the return distribution that will be fed into the simulation.

The simulation then uses that information to run 10,000 paths or iterations through your retirement plan. For each year of each path, the code randomly chooses a return based on the distribution that you specified. During most of the paths through the simulation, it's very likely that the code will choose several 3 sigma or greater return values (eg randomly selected return is 3x sigma less than the average return).
Post Reply

Who is online

Users browsing this forum: No registered users and 27 guests