I Asked AI to Plan My Retirement — Then I Caught It Lying

[00:00:00] Hey, welcome to another episode of Retirement Made Simple. I'm your host, Kevin Lum. I'm a certified financial planner based in Los Angeles, and this podcast is dedicated to helping a million people retire without worry. As a quick reminder, every episode here comes straight from our YouTube channel. So this is just the audio, so you can listen while you're walking, driving, or living your life. Let's dive in.

[00:00:26] Every few days, I'll read a new article about how AI is going to put financial advisors out of business. And honestly, based on the experiences I've heard about from many of you, a lot of financial advisors at risk. But today, I want to answer the question, can I use AI for financial advice, and more specifically, for retirement advice? And so my answer in short is probably.

[00:00:50] So today, I'm going to tell you where AI excels for financial planning, where it fails, and how you can use it most successfully. And for fun, I'm going to tell you what I learned about retirement planning from building a home entertainment system. More than half of American adults have already asked AI for financial advice. That's a significantly larger number than a set across the desk from a human advisor.

[00:01:16] So let me say this clearly. The debate about whether people should or will use AI to plan their financial future, that debate is over. They're already doing it. In fact, I'm guessing that many of you are already doing it. That leaves the question, is it any good? And here's what I found after digging through some research on this topic, particularly from a researcher at MIT.

[00:01:40] The answer is, it's incredible. And it's awful. AI gives you better retirement advice than you might expect, and it fails in some really significant ways. So let's start with the good. Researchers at MIT Sloan School of Management ran one of the most thorough tests that I've discovered.

[00:01:58] They built a simulation of an entire human financial life from age 22 to age 90, looking at earnings and savings and investing and paying taxes and getting hit with random life events, you know, spending shocks along the way. And then they had real people write prompts to chat GPT and Gemini, and they ran those AI answers through thousands of simulated lifetimes to see what actually happened to the money.

[00:02:28] And what they found was the advice that these AI systems gave was sound. It consistently told people to do the things that economists would actually recommend. Save more during your working years, draw it down in retirement, stay heavily diversified in stock funds when you're young, right? Invest more in stocks when you're young, and then dial back the risk as you age. And under that guidance, most of the people in the simulations, they increased and built real wealth and real savings.

[00:02:57] That ultimately wasn't the main reason or the main design behind these AI chat bots, right? It's better at programming and things, but it wasn't really designed initially to do financial planning. But the researcher said he was surprised at how good the advice was. But it gets even more interesting because the AI didn't just answer the questions that people asked. It expanded the conversation, which is something a great advisor does.

[00:03:22] Only 6% of people asked about liquidity, having the cash you can spend when you need to spend it, right? Your emergency savings or whatever it might be. But the AI brought it up in 83% of its answers. In other words, it raised things that people should have asked about even when they didn't know to ask it. And there's also been another really interesting change as these AI tools have emerged and developed. Emotional intelligence.

[00:03:48] One of the MIT professors who is part of this study has been testing these tools since 2022. And he said when he first started in 2022, he had asked an older version of ChatGPT what to do after losing 25% of his life savings. And it gave him neutral, reasonable advice, right? Kind of like what a robot would give you. When he asked the newer versions the same question, it opened with a completely different framing. He said, I'm really sorry. You're not alone in this.

[00:04:16] A loss like this can feel gut-wrenching. Let's slow down and make it manageable. It didn't lead with advice. It led with empathy. There are some financial advisors out there who could learn from that. And for those of you watching the channel, most of you already know this, but the worst financial decisions almost always get made out of fear. And those of you who are watching this channel, you know the importance not only of looking at the spreadsheet of the numbers, but you also know the importance of the behavioral side of financial planning.

[00:04:45] And what ultimately the researchers discovered was that AI was an incredible thought partner. But it also had some serious downside. So we've talked about the good, let's talk about the bad. As most of you probably know, AI is bad at math, like terrible at math. These systems don't calculate. They predict. They're guessing the next most likely word based on the patterns they've seen. So they're not actually running numbers or running calculations.

[00:05:14] But your retirement plan, it has thousands of moving calculations. And what's happening is these models are essentially guessing at the story around them. Now, these tools are using plugins and integrations to help fill some of those gaps, but it's still a constant challenge. Let me just show you a quick example. Sure, go for it. I'm ready to see what you've got. How many E's are in the word 17? Sure. The word 17 has exactly five E's in it. Are you sure about that?

[00:05:45] Yes, I'm sure. 17 has three E's. One in the first syllable and two in the second. I've got you covered. Wait, I thought you just said it has five E's. Now you're saying it has three E's? You're right to catch that. I misspoke earlier. The word 17 definitely has three E's. Just three. Thanks for sticking with me. Okay, just to verify, because I get confused really easily. It only has three E's. No problem at all. Yes, it's only three E's and 17.

[00:06:15] You've got it. For those of you following at home, 17 has four E's. That is ChatGPT. That is the latest model, and I pay for the pro edition. So where does it break down? It breaks down on tax optimization and withdrawal sequencing and calculating IRMA surcharges in relationship to everything else in the plan. One of the researchers called out that it's really bad at social security calculations because

[00:06:43] it said social security is based on 22,000 pages of rules. He said AI just gets social security projections wrong. It's not great at multivariable retirement planning. The knowledge is there. These systems are great at being able to tell you exactly what a backdoor Roth is or being able to explain to you what a Roth conversion is, or even be able to explain to you how your social security benefit gets calculated. It's phenomenal at that. So it's a very great knowledge partner.

[00:07:12] But when multivariable calculations are involved, in fact, calculations are at all are involved. It struggles to put the knowledge together. I spent weeks trying to build a calculation engine using the various AI tools, and I would compare the calculations against what I knew to be true, and it would be off again and again. And when I would tell it, look, you're wrong. You're missing something. The number isn't right.

[00:07:38] It would often say, oops, I forgot to take into account this other variable. And one of the things that makes retirement planning so complex, and you know this, those of you who have been watching the channel, is the interplay between all of the various components. It's a giant puzzle, and you need to fit all the puzzle pieces together just right, right? And you've got this golden window to do Roth conversions in when your income is lower,

[00:08:03] and you have to think about how is this going to impact, you know, Irma surcharges, and how is this going to impact the taxability of social security? All these variables that interplay together. And these models currently just don't do that sort of planning well. They're great at the knowledge side. They're great at helping you understand what the different Irma cliff levels are, but it's not great at the multivariable calculations. Actually, just not great at calculations in general.

[00:08:29] AI approaches retirement planning a little like how I approach putting a puzzle together after I've been sitting there too long. I don't have the patience to do puzzles. My wife and her family, they will, over holidays, will sit there for days putting together this massive puzzle. I last for about three minutes, and then I start trying to force pieces into place. AI, it feels, does similar things. It makes leaps in assumptions which are often wrong once it becomes too complex.

[00:08:58] So to reinforce this point, I want to just pause and tell you about a home project I'm working on. I promise this is relevant to the topic at hand. Recently, I was served an ad on Instagram for this incredible home stereo. It was the perfect home audio system, but it was $20,000, right? It promised to allow me to listen to Coltrane the way it was meant to be heard. So I decided I'm going to build this for like one-tenth the cost. So I found a cabinet maker who would build beautiful cabinets that the stereo could go inside.

[00:09:28] And then I started doing research on the perfect record player and tube amplifiers and speakers that really allow the warmth to shine through. And I did a ton of research using AI, and it was amazing, right? I felt like a genius. I put together this amazing system, and I was very clear about what I was trying to achieve. At the same time, I decided to read some Reddit forums, and I found a couple of amazing tube amplifiers that everybody on there said were great. And so I decided to ask AI about it.

[00:09:57] And the next thing I know, it was telling me to forget everything it said previously. My new Reddit source system was the perfect system for me. And then I reminded it that I wanted to install inside of a stereo cabinet. And then it said, no, no, no, all of this is wrong. The initial system is wrong. This new system is wrong. You should build your own speakers. And before long, it devolved into a chaotic mess.

[00:10:22] It was a brilliant thought partner, but it also failed me miserably because I ended up more confused than when I started. And the same thing can happen in retirement planning. Because there are so many routes you can go down as you optimize your plan, and the deeper you go, the more chaos that will unfold. And for those of you who are optimizers and planners, you're going to find yourself in utter paralysis.

[00:10:48] I literally stopped working on my stereo project because I became so overwhelmed at the options, and an AI just giving me such conflicting information if I changed anything about the inputs. At first, it felt like a superpower. Like I had this insane audiophile knowledge. I felt so smart. But then that knowledge quickly devolved into chaos. I imagine that it's something akin to the story of the Tower of Babel.

[00:11:17] At first, they thought they were all powerful and had all this incredible knowledge. And then before long, it evolves into chaos. But now I want to talk about the ugly side of AI, the thing that almost nobody is talking about. So the same MIT study that I mentioned earlier found something really disturbing. The quality of the advice depends heavily on who is asking, right? People with low financial literacy based on how they worded the questions ended up with less money at the end of the simulation, right?

[00:11:45] So the more financial literate you were as you went in, the better output. People who knew less about finance based on how they worded their questions, they ended up in these studies with less money than people with more financial literacy, right? So your ability to be able to communicate with these systems in a financially literate way helped guide your outcome, which isn't that surprising. But it does require a baseline of knowledge to get these tools to behave in the way that you want. But then there was something that really surprised me.

[00:12:16] Prompts written by women in the studies led to less wealth than prompts written by men. Now, part of this came from how the prompts were phrased. Women tended to use words like family and groceries and pay. Men tended to use words like strategy and crypto and growth, right? So it leaned more into growth in their plan. But part of it also came from the AI itself. When researchers took an identical prompt and simply added the words,

[00:12:44] I am a woman, the model recommended less money in stocks. Same question. One word changed the advice. Now, maybe AI thinks women are more risk averse. I'm sure it consumed some paper at some point that said as much. So let's assume for a moment that's true. Let's assume that women are less risk averse than men. AI doesn't just reflect your blind spots. It can quietly amplify them.

[00:13:11] It assumes based on your gender or your demographic that you're less risk averse. And then it leans into that stereotype instead of pushing back on it. It often feels as if it reflects what I want to hear and then amplifies it even more. And one of the things I've talked about in prior videos is that a great financial advisor often tells you things you don't want to hear, right? To make this plan work, you're going to need to take a bit more risk.

[00:13:39] You're being a little too risk averse or you're being a bit too risky. But AI in these studies seems to tend to lean into whatever our bias is. Now, listen, I am not anti-AI. I think it's transformative and it can be an incredible thought part. But the question becomes, how do we use this tool well? And ultimately, the tool isn't the problem. So what I've done is I've created a few rules to help you use these tools better. And in fact, I will put a link in the description.

[00:14:09] So I put together a quick PDF with some of these rules. So you'll have them to reference back to. But a few rules. Number one, don't ask the chat bots if you're right. Ask them why you're wrong. So let me give you an example. Instead of, is real estate a safe investment? Ask it, am I wrong to think real estate is a safe investment? How so? That forces it to find the holes in your reasoning and in your thinking instead of just flattering you.

[00:14:37] Number two, make it state its assumptions. Let me show you why this is so important. A reporter gave three different AI tools, kind of a scenario, and asked, can it retire at age 65? And all three said, yeah, it's tight, but it's doable. Then the reporter asked one simple follow-up question. He said, what assumption did you make? And the AI admitted it had assumed that she died at 90.

[00:15:01] It wasn't modeling taxes accurately, and it hadn't priced in any long-term care at all. One of them, Claude, then walked back its own answer, right? Once they pushed back and asked, what assumptions did you use? And I've seen something similar a lot from these tools. It went from, it's tight but doable, to it's meaningfully underfunded without course correction. So the reassuring answer came first.

[00:15:27] The honest answer only showed up when someone pushed back. And I see this all the time in my own personal use. My wife is using it to program a piece of software, and she always ends a programming session by saying, what would a good programmer do at this point? And what mistakes would it discover? She told me it never ceases to make her code better, which she's like, why didn't it just do this thing in the first place?

[00:15:56] But by asking it, what would a great coder do? And to find the mistakes, it somehow continues to improve the codes. Which leads to the third thing that I do. I would ask it to be an adversarial financial advisor. So once you built a plan using whatever your favorite chatbot is, ask it to review your plan for holes. And use the word, I want you to be an adversarial financial advisor.

[00:16:20] And you will be surprised the mistakes it'll find in the plan that it just got done writing. But even better, run it through another model and ask for an adversarial review. Push it. Ask what's missing. Run the same question through two different platforms and let them argue with each other. I still find very basic mistakes. The other day, I tried running a sample plan through Claude and it came back with wrong contribution limits for Roth IRAs.

[00:16:49] When I pushed back, it said, you're right. I discovered my mistake and it corrected it. But it's often helpful to kind of compare and contrast a couple different models and to push back to say, can you review your work? Are there any mistakes that you've made? And it'll often, to its credit, will find its own mistakes. But at first pass, because it's often just spitting out data that's consumed and there's so much competing data on the Internet. And a lot of it's old, but it sometimes gets a bit confused. But if you push back, it tends to find its own mistakes and to learn from them.

[00:17:19] You understand the basics, right? The better your foundation of financial literacy is, the better the answers you're going to get from these systems, right? The better the answers that you're going to receive. But remember, again, I'm going to feel a bit like a broken record, but there are limits to AI, right? It can't run a truly smart tax withdrawal strategy, at least today. It can't reliably navigate the edge cases of Social Security. And it doesn't bear any responsibility when it's wrong, right?

[00:17:45] There's no fiduciary standard for chatbots. They're just a machine that takes no responsibility when it's wrong. But here's what I want you to hear from me as we kind of wrap this up. For most people, the biggest retirement problem is not going to be a knowledge gap. The viewers of this channel are incredibly smart people, and you have incredibly great tools at your fingertips. A planner that was quoted in one of the studies I read said it perfectly.

[00:18:14] He said, the obstacle is rarely behavioral. It's not technical. It's the fear of investing. It's the fear of spending the money that you saved. And at the end of the day, AI can hand you a flawless spreadsheet and get all the calculations right and still not move you one inch past the anxiety that's actually running your decisions. The most common failure in retirement is not running out of money.

[00:18:40] It's that the wealthiest generation in history is spending less than it can because it's afraid to spend, right? They can't figure out how to take this pile of money and actually turn it into something they can use. And as of today, no chat bot has fixed that. So here's where I land. Ultimately, use these tools as a thought partner, something that teaches you and maybe can even pressure test your assumptions.

[00:19:06] AI, in my opinion, is a genuine gift that we didn't have a few years ago. But at the end of the day, all the tools and the calculators, no matter how great they are, have limits because so much of retirement planning is part art and part science. And even though I hate to admit it because I love data and I love research, but the best retirement planners know this, that the tools are important.

[00:19:33] They're incredibly important, but the real power of a financial professional is that they've run the same play over and over and over again, which has allowed them to become incredible probabilistic thinkers, which allows them to guide you through a myriad of complex decisions in a way that feels simple and manageable.

[00:19:54] Personally, if I didn't ever want to hire a financial advisor, I would use AI in conjunction with a more traditional planning tool. I think great capital is great if you want to get your numbers directionally right and you're going to eventually hire a financial advisor. But if you just want to do it all yourself, you never want to talk to an advisor, I think Bolden is as good as any tool that's out there.

[00:20:16] And so they've done a great job of combining a traditional retirement planning kind of engine with AI to support that, which I think is really important. I'll put an affiliate link in the description for Bolden and I'll also put a link in the description for that PDF I put on how you can use AI well. I had some prompt suggestions and just kind of a recap of some of the rules that we've talked about and I expand on them just a bit. Hey, thanks for listening.

[00:20:41] If you enjoyed this content, if you do me a favor and just leave a review on whatever podcast app you're using, Apple or Google or Spotify. And also you can find us on YouTube. Just search Foundry Financial or Retirement Made Simple. You should be able to find us by searching both. And then you can find our website at foundryfinancial.org. Thanks for listening.