Scott Dalziel struck in the dying stages to seal a 2-0 win for Berwick at Albion.
The hosts had the better of the first half but could not find a way past Paul Grant, who produced two great saves to ensure Berwick were on level terms at the break.
The away side were much improved after the restart. Lee Currie had a shot well saved by Neil Parry and the resulting corner was cleared off the line, before Steven Notman hit the bar.
Berwick did take the lead on 62 minutes as Kenny O'Brien found the net from a tight angle.
Dalziel's fourth league goal of the season from close range gave Berwick back-to-back wins which leave them second in League Two.