If you decide to drive to Fermont, it can be reached via Quebec Highway 389 from Baie Comeau. The road is 581 kilometers long (360 miles). It offers a mix of driving conditions, with about half of the distance paved and the rest being a good condition unpaved road with a few rough patches. Driving the whole distance at a safe speed requires approximately 8-9 hours.
It’s worth mentioning that you are driving through the Manic 5 hydro-electric megaproject that provides power to most of Quebec and the northeast United States. It is located at about the half-way point. If you can spare the time, it’s really worth a visit (free tours are arranged by Hydro-Quebec during the summer). There’s even a decent motel just before Manic 5 if you decide to split up your driving in 2 days.