Human error and internal control failures led to a US$62M fine for Citigroup.

The circumstances surrounding the error, failures, and fines include many takeaways for IT, business units, and senior management as they design information and technology systems and procedures and manage risk.

What happened

In May 2024, the UK’s Financial Conduct Authority (FCA) and Prudential Regulation Authority imposed fines and penalties totalling about £62M on Citigroup. See FCA fines CGML £27,766,200 for failures in its trading systems and controls, and The Prudential Regulation Authority (PRA) fines Citigroup Global Markets Limited (CGML) £33,880,000 for failures in its trading systems and controls.

The human error was commonplace. A Citigroup trader wanted to sell a basket of 349 stocks with a total value of $58M. Instead, he entered 58M in the field for quantities, not value. That error directed the trading system to sell stocks valued at US$444 billion, not US$58M. The flood of stocks caused a massive sell-off that disrupted stock markets.

The FCA said it expected entities “to have effective systems and controls in place to stop errors like this occurring” and “firms to look at their own controls and ensure that they are appropriate given the speed and complexity of financial markets.”

The FCA referred to its investigations and the failings in Citigroup’s systems and controls spanning 2018 to 2022. The FCA stated that although Citigroup had undertaken some remedial work, control weaknesses persisted. The absence of preventative controls known as “hard blocks” and “the inappropriate calibration of other controls” were unacceptable to the PRA.

Many entities are in different sectors from Citigroup, are not subject to the same regulatory standards, and do not process anything near these dollar values. Nonetheless, there are important lessons and takeaways from Citigroup’s experience regardless of organizational size.

Takeaways

A key takeaway is the importance of designing effective application controls. Application controls are the mechanisms and procedures that regulate input, processing, and output functions to ensure data integrity, confidentiality, and availability. Application controls prevent or detect input errors and other threats to data integrity.

Application controls should be a mix of automated and manual controls working in harmony.

News reports and regulators’ websites describe precisely what went wrong at Citigroup. For example, read articles at Citi Trader Got 711 Warning Messages Before Sparking Flash Crash, and the Financial Times print edition on July 13 and 14, 2024, entitled Counting the cost of my ‘major keying error’ (online version here).

Based on news reports, Citigroup had experienced at least two critical failures in application controls. First, the trader received a popup error message with over 711 alerts. Then, he was able to manually override the popup, which allowed the system to process part of his transaction. The trader himself later realized his error and cancelled the trades that he could. But although only minutes had elapsed between initiation and attempted cancellation of the transaction, trades totalling over US$1.4BN had been processed and havoc wreaked on stock markets.

It seems excessive that the system generated 711 alerts in the popup error message. Perhaps some warnings were unnecessary. Furthermore, the trader could manually override popups without scrolling down to review all the alerts before overriding.

Any entity relying on error messaging should streamline messages so that they are not excessive. Streamlining could include redefining what qualifies as an alert or the thresholds for a warning.

Readers of this blog may recall that a Citigroup entity, albeit in the US, had a well-publicized error related to weaknesses in system design and other controls, when it mistakenly transferred $900M of its own money to entities and had difficulty recouping some of the overpayment from entities that refused to return the money. See First Reference Talks blog entitled A $900M error, poor system design, and failed internal controls.

Both errors in the US and UK highlight that design goals include simplicity and effectiveness so that users are more likely to meaningfully engage with alerts instead of mindlessly racing through them because they are overwhelming “noise” or sometimes “cry wolf.”

Use manual overrides with caution. Strategies including the following reduce the likelihood that overrides result in fraud or errors:

1. Preventing manual overrides for certain types of alerts or alerts above a certain threshold. In Citigroup parlance, include hard blocks. The magnitude of the trader’s error makes it one that should have had hard blocks or controls to prevent the system from processing the entire transaction based on reasonableness checks, logic or relational checks, and other validation controls. If a US$444BN trade does not qualify as such a transaction, it is hard to imagine what would.
2. Including system designs that require users to scroll through and read or acknowledge alerts before the system permits dismissal of the alert.
3. Making each alert standalone as opposed to a series of sub-alerts under one popup, so the user must manually click to resolve each alert before proceeding.
4. Requiring a supervisor or other individual to sign off or click a popup to authorize an override. The corollary is a system designed to prevent a user from dismissing certain of their own alerts.

Also based on news reports and statements from regulators, the above and other control failures coalesced into a perfect stew for the trader’s error to foment.

First, an automated system that the trader would have otherwise used was unavailable to him, causing him to build the trade manually. When system unavailability leads to changes in processes, there should be heightened vigilance against likely errors, given the atypical processing, the inability to rely on automated controls in the unavailable system, and other factors.

Additionally, there were planned staff absences in the department that would have typically performed real-time monitoring as an additional control, so it transferred some of those responsibilities to another department. That overflow department failed to escalate alerts that it received. Segregation of duties can prevent errors and improve controls because there is a second set of eyes focussed only on reviewing and monitoring and not processing. However, poor staff scheduling or staffing disruptions are risk factors because they lead to inadequate resources to effect segregation of duties as a control.

It is unclear why the backup department did not escalate alerts it received. However, backup staff may sometimes lack the training and practice to carry out their overflow roles properly. Control procedures must ensure adequate and trained staff for both regular and backup purposes.

Furthermore, Citigroup had temporarily changed thresholds for certain hard and soft blocks two years before the error to accommodate volatility arising from the pandemic. It had failed to revisit its decision in subsequent years.

Unlike Citigroup UK, Citigroup US reportedly had controls in place since 2013, which would have stopped all the trades from going through. In fact, it was some of Citigroup UK’s hard blocks—which do not permit overrides—that prevented some of the trades from being processed. Thus, it would seem that there are systems and procedures technically capable of avoiding the trader’s error, and Citigroup UK had merely to implement the necessary controls that were in place elsewhere in the group.

Ultimately, IT, business units, data owners, and senior management should remember that ongoing monitoring and improvement of controls are essential to respond to changes in operating and regulatory environments. For example, what was acceptable during a pandemic may be ineffective to deal with post-pandemic risks. Furthermore, if external auditors or regulators raise material concerns about internal controls, swift and in-depth action and post-action assessments are critical.

Meeting your duty of care

While you may not be a Citigroup with billions of dollars at stake, the takeaways from Citigroup’s error apply equally. Perform a risk assessment of existing procedures and implement commensurate application security controls and other internal controls. Review the Information and Technology database in PolicyPro, including SPP IT 9.04 – Application Security Controls and incorporate effective controls from the system design and acquisition phases, followed by continual monitoring and improvement.

Policies and procedures are essential, but the work required to create and maintain them can seem daunting. The Finance and Accounting, Operations and Marketing, Not-for-Profit, and Information Technology databases in PolicyPro, co-marketed by First Reference and Chartered Professional Accountants Canada (CPA Canada), contain sample policies, procedures, checklists and other tools, plus authoritative commentary to save you time and effort in establishing and updating your internal controls and policies. Not a subscriber? Request free 30–day trials of Finance and Accounting, Not-for-Profit, Operations and Marketing, and Information Technology databases in PolicyPro here.