It's faster if you take the ferry from Sibulan and get on the aircon bus waiting at the port in Santander.
Agree, my experience when travelling by the Ceres bus from Cebu South terminal to Lilo An pier, leaving @9 or 10 am is generally 4 hours. Much depend on the traffic congestions between Cebu city and CarCar, that could increase travel time.