Bats Global Markets has warned traders that its systems have carried out hundreds of thousands of transactions over the last four years at prices that were not the best available.
The operator's computers allowed some trades on its two US equity exchanges and its options exchange to be processed at prices equal to or less than the national best bid or offer price, in violation of SEC rules.
The problem dates back to October 2008 and has affected nearly 450,000 trades, costing customers $420,360 in inferior prices, a spokesman Randy Williams told Bloomberg.
In a notice on its Web site, the operator says it is working to have the issue fixed by 25 January.
Last year Bats was forced to pull its planned initial public offering after a humiliating technical breakdown on the opening day of trading.