The pressure regulator is plumbed to the intake manifold. So that manifold pressure is applied to the regulator.
Prior to start, the manifold pressure will be as high as it will ever be. As will be the fuel pressure. Upon starting the motor, the manifold pressure goes down and so too does the fuel pressure.
If the motor won't run with the pressure regulator being plumbed to the intake manifold, but it will when the regulator is not plumbed to the intake manifold, (resulting in an excessively high fuel pressure) I might guess that the fuel injectors are somewhat clogged. At the normal fuel pressure they won't allow enough fuel flow for the motor to operate. But with an excess fuel pressure they might.
When were the fuel injectors serviced?