RBS failed to properly report 44.8 million transactions between November 2007 and February 2013 and failed altogether to report 804,000 transactions between November 2007 and February 2012.
This represents 37 per cent of relevant transactions carried out by RBS in this period and breaches FCA rules on transaction reporting and its requirements for firms to have adequate management and controls.
Many of the problems with RBS' own systems were compounded by the takeover of ABN Amro Bank N.V. in October 2007.
The FCA said that, given the considerable resources available to RBS, it felt RBS should have been able to overcome these challenges and ensure adequate systems and controls were in place.
{desktop}{/desktop}{mobile}{/mobile}
A transaction report includes identifiers of the financial instrument, the firm that undertook the transaction, the counterparty to the transaction and other factors such as price and quantity.
Tracey McDermott, the FCA's director of enforcement and financial crime, said: "Effective market surveillance depends on accurate and timely reporting of transactions. We have set out clear guidance on transaction reporting, backed up by extensive market monitoring, and we expect firms to get it right.
"As well as a financial penalty, firms can expect to incur the cost of resubmitting historically incorrect reports. We will continue to take appropriate action against any firm that fails to meet our requirements."
RBS agreed to settle at an early stage which entitled it to a 30 per cent discount. Without this the fine would have been £8m.

• Want to receive a free weekly summary of the best news stories from our website? Just go to home page and submit your name and email address. If you are already logged in you will need to log out to see the e-newsletter sign up. You can then log in again.