Wall Street breathed a sigh of relief: ChatGPT wasn't smart enough to be a financial analyst!

Source: Finance Associated Press

Author: Huang Junzhi

Wall Street analysts who fear that artificial intelligence (AI) will take their jobs can now breathe a sigh of relief.

ChatGPT fails to pass the Chartered Financial Analyst (CFA) exam – the benchmark qualification for many financial professionals on Wall Street, according to a recent study. In the study, conducted by a team of JPMorgan AI researchers and university academics, ChatGPT and GPT-4 were asked CFA-related questions to test whether they were able to perform complex financial reasoning.

"Based on the estimated pass rate and average self-reported scores, we conclude that ChatGPT may not be able to pass CFA Level 1 and Level 2 exams in all testing settings," the researchers wrote. ”

The CFA is a rigorous three-part test designed to measure knowledge of securities analysis and portfolio management. It takes years for professionals to complete all three levels of exams, and candidates often put in hundreds of hours of preparation time. Once certified, it is considered a mark of distinction and can provide more opportunities for the recipient throughout their career.

Many people who took the exam failed. According to the CFA Institute, only 38% of candidates who took the Level 1 exam in February 2023 passed the exam. Fifty-two per cent passed the Level 2 exam in May. The pass rate for the most recent Level 3 exam in August was 47 per cent.

According to the study, ChatGPT made a number of "knowledge-based mistakes" when asked about simulated level 1 questions that involved a large number of multiple-choice questions, which the researchers defined as "errors in which the model lacks the critical knowledge needed to answer questions." GPT-4 makes the same knowledge-based mistakes, but they're not as bad as ChatGPT.

Level 1 tests are usually more difficult than Level 2, but these models perform worse in Level 2 tests, which require more analysis and interpretation of information on things like equity investment and financial reporting.

Overall, GPT-4 outperforms ChatGPT on both levels. In fact, if the prompts are adequate enough, it performs well enough to pass the first two levels of the CFA exam. "GPT-4 will have a considerable chance of passing the CFA Level 1 and Level 2 exams if prompted," the researchers wrote. ”

To be sure, natural language processing, or the ability of software to understand and manipulate human language, has found applications in several areas of finance, such as sentiment analysis or summary reporting.

But with the development of AI large language models such as ChatGPT, the issue of AI replacing human financial professionals has become a hot topic of discussion on Wall Street. Earlier this year, ChatGPT caused a huge stir due to its human-like response.

Fortunately, this situation seems far from a reality at the moment.