Scott Chaplain grabbed a double as Annan scored three in five minutes to come from behind and beat Berwick 3-2.
Philip Addison tapped in from six yards to give Berwick the lead after 14 minutes.
Fraser McLaren then scored from close range in the 25th minute to give Rangers a two-goal cushion going into half-time.
But Annan flew out of the blocks after the break to stun their opponents.
First Chaplain converted from the spot after Chris Pownsley handled in the area, and he headed in his second three minutes later.
Ally Love scored from six yards in the 61st -minute to complete the comeback.