A small amount of mini-charged dark matter could cool the baryons in the early Universe