Sensitivity analysis results vs. main simulation results

[Cincy_1988]

Hello

I'm delving into FRP (which by the way is really GREAT!) as I'll retire in May 2025. This is my second question since joining the group yesterday!

The main simulation results indicate my retirement is adequate (100% success with a green traffic light & sufficient funds remaining at the end of my life). However, sensitivity analysis results 0% chance of success (annual retirement spending vs. portfolio return (std dev) plot) even though the minimum & maximum spending levels are much lower than what I've used for the main simulation. For sensitivity analysis, I've set the min. & max. returns & standard of deviations equal to what I used for the main simulation (I'm using the same values for both min. & max.) thinking this makes the comparisons apples to apples.

I think I understand the difference between the two simulations, but I don't understand why the simulation results are so vastly different - the main simulation shows 100% success & sensitivity simulation shows 0% chance of success. Which one should I use for planning purposes?

Thanks,
Bahram

[jimr]

The sensitivity analysis feature uses the regular simulation code, so the issue has to be something to do with planner inputs.

Do you have anything set up in Additional Inputs? Additional Inputs settings get layered in on top of whatever settings are used in the main sensitivity analysis window. That means if you entered expenses in additional inputs (instead of on the main planner window) they may be getting counted twice.

In sensitivity analysis, you can see the full details of any specific run by clicking the cell in the heat map that corresponds to a given run. That will highlight the row in the details table below that corresponds to that run. Once you have a specific run highlighted in the lower table, you can right-click in that row and select show details to see all the year-by-year data for that run. That will show the amounts used for retirement income and retirement expenses for each year. This is probably the best way to verify that the planner is doing what you think it's doing.

[Cincy_1988]

Thanks for your prompt and clear reply.

Yes, I had expenses & incomes under "additional inputs" (with $0 on the main window). By moving the values from additional inputs to the main window, I get the same conclusions!

However, I really need to enter the data as additional inputs because (a) my pension doesn't have COLA, (b) I've selected travel expense to end at 85, and (c) I've specified my small social security benefit to start at 70.5. I really can't enter these details on the main window. So, it seems I should just use the results from the main window and don't run sensitivity analysis. Is my interpretation correct?

Thanks again!

[jimr]

You can use additional inputs with sensitivity analysis, but you just have to take into account any conflicts between what you've set up in additional inputs and the parameters sensitivity analysis is varying.

Based on what you've described, I think you could leave all the income cash flows as is in additional inputs (pension, ss), and just move the base amount of your planned expenses to the main input window. Then create an additional inputs cash flow for that extra vacation spending that only continues to 85.

In that case, sensitivity analysis would be varying the base spending amount. The vacation spending (up to age 85) would still get added to the base amount, but the results may still be interesting. Again, take a look at the details of some of the individual results from the lower table to make sure the planner is doing what you expect, especially what it shows for planned expenses.

[Cincy_1988]

Thank you! As best as I can tell, sensitivity calculations do account for stopping travel expenses at 85, which is great!

For sensitivity analysis, should I set BOTH the max. & min. rate of return (& std. dev.) equal to what I've in the main window? The default values don't seem to be related to the main window values.

The trickier part is for inflation rate because the main window allows data for average & std. dev. whereas only min. & max. values can be specified for sensitivity analysis. Should min. = average (from main window) - 1 std. dev. (from main window) but not less than zero & max. = average (from main window) + 1 std. dev. (from main window)? FYI, I've downloaded historical inflation data, and I'm using the average & std. dev. for various years in the main window.

Thanks for helping me understand this GREAT too!

[jimr]

The general idea with sensitivity analysis is to test out wide variations inn two of your planner input parameters. The default is to vary expenses and rate of return. The upper and lower bounds you set for each of these two varying inputs is up to you depending on how you're thinking about your plan. Most people select ranges for both input parameters such that the value they think is most likely for that input (eg spending) is in the middle of the range. So for example, if you expect retirement spending to be $75k, you might like to set the lower bound to $50k and the upper bound to $100k. But really, it's totally up to you. For example, if your plan still had a 100% prob. success with $100k spending, you might try a much higher limit to see where it starts to become less of a slam dunk.

For the return/std deviation parameter, most people set the range there to represent the least risky portfolio at the low end (eg lower return and lower standard deviation), and the most risky portfolio at the upper limit (highest return/std deviation). Keep in mind that the return/std deviation parameter is a special case where the planner varies the values for these in lock-step from lowest to highest, such that the portfolio with the lowest return has the lowest std deviation and the portfolio with the highest return has the highest standard deviation.

If this all seems confusing, the best way to really see what's happening with sensitivity analysis is to click on the individual cells of the heat map and look at the right hand panel to see the input parameters that were used to generate that cell. Also, as mentioned before, you can also right-click on the row in the lower table associated with that run (eg cell you clicked on) to see even more details about what the assumptions/inputs were that generated that particular cell in the heat map.

Finally, I recommend leaving the inflation standard deviation input at 0 in all cases. That ability to set an inflation std deviation was added based on a user request years ago and with hindsight, I no longer think it's particularly useful to set that to anything other than 0. Setting inflation std deviation to a nonzero value causes the simulation to vary that input stochastically, the same way it varies return. The issue is that this likely adds more error than signal in the model, since the relationship between inflation and return is not clearly understood and likely varies over time in ways that the simulation doesn't/can't emulate. I think all this just makes things more confusing and noisy than if the inflation std deviation is just left at zero.

[Cincy_1988]

Thank you. I suppose I'm trying to be unrealistically precise!

I went back and looked at keeping travel expense (that ends @ 85) under additional inputs and moving the other expenses to the main window. I was wrong, sensitivity analysis double counts the travel expense. These're the numbers I get for some made up incomes and expenses:

Main window expense = $49,800
Travel expense (ending @ 85) under additional inputs = $14,400
Main window results @ 95 for 5.4% (3.5%): $873,583 (median), $629,574 (bottom 10%), $1,171,280 (top 90%)
Sensitivity analysis for 5.4% (3.5%): $363,025 (median), $119,899 (bottom 10%), $646,639 (top 90%)

Move travel (ending @ 95) to main window, so the total expense under the main window = $64,200
Main window results @ 95 for 5.4% (3.5%): $720,052 (median), $486,666 (bottom 10%), $1,023,009 (top 90%)
Sensitivity analysis for 5.4% (3.5%): $723,933 (median), $486,724 (bottom 10%), $1,026,515 (top 90%)

When you get a chance, please look into your program to see if there's a way to avoid double counting of expenses entered under additional inputs for sensitivity analysis. (I haven't checked, but I believe income under additional income is also doubled counted for sensitivity analysis.)

Thank you so much for patiently and thoroughly answering my many questions.

[jimr]

Do you have any entries in the top window of additional inputs?

Either way, I'm happy to take a look at what's going on with your setup if you email the .frp file to info@flexibleretirementplanner.com.

IMPORTANT: PLEASE DON'T EMAIL A .FRP FILE THAT CONTAINS YOUR PERSONAL FINANCIAL INFORMATION. BEFORE SENDING THE FILE, USE SAVE-AS TO RENAME THE FILE, THEN CHANGE AROUND THE AMOUNTS SUCH THAT THE PROBLEM STILL EXISTS, BUT YOUR PRIVATE FINANCIAL INFORMATION IS PROTECTED. THEN SAVE AGAIN AND SEND THE MODIFIED FILE WITH "FAKE" AMOUNTS.