Berwick required extra-time to reach the second round of the Ramsdens Cup as they beat 10-man Livingston 3-2.
The hosts took the lead just before the half-hour mark when new loan signing Owen Ronald's corner was headed home by Gray.
Marc McNulty had a golden opportunity to level things up from the penalty spot but blazed his kick over the bar on 77 minutes.
But the Livi striker made amends at the death when he ensured extra-time would follow after heading home Martin Scott's cross.
The visitors were reduced to 10 men four minutes after the restart when Jason Talbot was dismissed for foul and abusive language.
Rangers made the most of the advantage on 99 minutes when Josh Morris thumped home a fine strike from the edge of the box.
Ronald celebrated his debut with a fizzing strike into top corner minutes later before Simon Mensing converted a consolation penalty after Dean Hoskins' challenge on McNulty.