Optimization of Modulation Constrained Digital Transmission Systems