The number of adviser firms has fallen by 15% since 2021 although the number of advisers overall has remained steady at 31,000.