When you choose to switch, the factor purchases the invoices that the previous factoring company had already purchased. This means that in addition to any fees previously incurred, the new factor is now obligated to pay whatever the former factor has advanced. Once the move is made, the factor will accumulate those costs from the client (you) via bills you send.
The old company may impose a fee for early termination if you change before the end of the contract time. The buyout cost for transferring the invoices from the previous factor may also be assessed by the new factor.