Edit system pulse audio config file.
editor /etc/pulse/default.pa
Comment the next line out like this:
### Automatically suspend sinks/sources that become idle for too long
# Removed to fix delay in sound on display port.
#load-module module-suspend-on-idle
To make it have effect without restarting just kill the pulse daemon
which is restarted by itself.
pulseaudio --kill