Former contestants from the reality series have attacked Simon Cowell after he dismissed duty of care failures.