Chat GPT, CFA?

While some are worried that Chat GPT could soon be replacing every job from journalist to juggling coach, one group of people that don’t have to worry – yet – are Wall Street’s CFAs.

Recently a team at JP Morgan paired with academic researchers to see if ChatGPT could pass the first two levels of the coveted CFA exam, which usually takes humans four years to complete in total, according to Bloomberg.

At least for now, the AI bot isn’t able to pass the test. 

In an 11 page report written after the trial, the researchers wrote: “Based on estimated pass rates and average self-reported scores, we concluded that ChatGPT would likely not be able to pass the CFA Level I and Level II under all tested settings. GPT-4 would have a decent chance of passing the CFA Level I and Level II if prompted.”

Researchers, comprising scholars and six individuals from JPMorgan’s AI Research group, such as Sameena Shah and Antony Papadimitriou, are at the forefront of this study. In its commitment to equipping finance professionals with current knowledge, the CFA Institute has overhauled its examination syllabus. As early as 2017, the institute declared its intention to incorporate queries on artificial intelligence and big data analysis techniques into its examinations.

Chris Wiese, managing director for education at the CFA Institute, told Bloomberg: “While multiple choice exams and essay questions remain excellent ways to assess learning and understanding in a secure proctored environment, the day-to-day in finance does not present itself only as a series of short, standalone questions.”

He added: “This is why to become a CFA charterholder, we also require 4,000 hours of qualifying work experience, a minimum of two references, a strong moral compass, and, coming soon, the completion of hands-on practical skills modules.”

The company is exploring large-language model technology to aid CFA candidates. Thousands take the three-tiered test regularly, with recipients often dedicating over 300 hours of study per level. Recent years have seen a dip in pass rates, with Level 1’s average at 37% this August, down from 43% in 2018.

Level I of the CFA has 180 multiple choice questions, whereas Level II combines case studies with 88 questions. Both language models found Level II tougher. In Level I, ChatGPT and GPT-4 excelled in areas like derivatives and equity, but faltered in financial reporting and portfolio management.

In Level II, ChatGPT lagged behind GPT-4 in alternative investments and fixed income but surpassed in portfolio management and economics. ChatGPT’s main errors were knowledge-related, while GPT-4 often made calculation mistakes, the Bloomberg article notes.

The report concluded: “The one error type that GPT-4 makes more frequently than ChatGPT was reasoning errors. It would seem that, along with GPT-4’s greater ability to reason, it has a greater chance of ‘talking itself’ into incorrect lines of reasoning.”

We’ll see how long this lasts. Our guess is Chat GPT has replaced half of Wall Street’s already useless sell side salespeople analysts by 2025.