If the regulator and pump does not leak down when shut off, it will hold pressure.
The bikes with the pump and regulator inside the tank can hold some much pressure for days, that it is nearly impossible to unclip the fuel line. You have to unplug the pump, and crank the motor to fire the injectors, to relieve the pressure just to got the fuel line unclipped.
Yes, the pump does have a check valve that can hold the pressure.