Sebastien Faure claimed his first goal for the club as Rangers fought back from behind to secure a 3-1 win at Berwick.
The home side opened the scoring after just four minutes through an own goal from Anestis Argyriou.
- Gow: Rangers grind out win with a moment of class
- VIDEO: Berwick Rangers 1-3 Rangers Analysis
- Wallace and Little: We rode the storm
But the ScottishThird Division leaders levelled thanks to a Dean Shiels penalty, before Andy Little netted his fifth goal against the English side this season.
Faure then came off the bench and netted the final goal on a day when the action on the park was marred by offensive chanting by a section of the Rangers support off it.
Rangers made one change from last week's win over Clyde for the clash at Berwick.
Emilson Cribari missed out through illness, paving the way for Ross Perry to slot into the heart of defence after recovering from injury.
Rangers had been far from impressive on their last visit to Shielfield, scraping an unconvincing 1-1 draw at the start of the season.
And they endured the worst possible start to this encounter when they found themselves trailing after just four minutes.
The visitors failed to deal with a Neil Janczyk corner into the box and the ball hit off Argyriou - under pressure from Darren Lavery - and into the back of his own net.
Rangers were back on level terms five minutes later when referee Calum Murray pointed to the spot after Shiels went to ground under a Dougie Brydon challenge.
The penalty award may have been harsh on the home side but the Light Blues restored parity when Shiels dusted himself off to rifle straight down the middle from 12 yards.
Rangers could have taken the lead when Barrie McKay tried his luck with a well-struck shot that was deflected just wide.
Berwick then appealed for a penalty of their own when Dylan Easton claimed he was tripped by Argyriou but the referee was unimpressed.
Little had the ball in the net but the offside flag was already raised.
Berwick had the chance to pile on the pressure with a free-kick out on the left and Neil Alexander did well to smother a dangerous, curling effort from Lee Currie.
The Rangers number one was called into action again to block a point blank shot from Lavery, who should have had the net bulging.
With half-time approaching, the home side squandered another chance when Easton latched onto a throw-in and fizzed across the face of goal from the tightest of angles.
Instead, it was Rangers who edged in front in injury time. Lee Wallace supplied the cross and Little sent a lovely left-foot volley beyond the grasp of Marc McCallum.
Both sides made a change at the break. Kal Naismith replaced McKay and Easton made way for Fraser McLaren.
Perry had suffered a blow to the face in the first half and he left the pitch after 53 minutes, allowing for the introduction of Faure.
The Frenchman opened his account for Rangers after 66 minutes. A David Templeton corner into the box fell for Naismith, his header was blocked, and Faure pounced to rifle home the rebound.
Substitute Dean Carse saw a decent effort blocked by Faure and Kenny O'Brien also passed up a decent chance to haul the Wee Rangers back into the game as the Glasgow side held on for the win.