The opening game of this series almost remained scoreless through the first two periods until Derek Stepan gave the Rangers a 1-0 lead with 13 seconds left in the second period.
Tampa Bay's Ondrej Palat tied the game with just under 15 minutes to play in regulation.
Dominic Moore scored his first goal of the playoffs - a crucial one, as he drove to Ben Bishop's net and had a pass attempt ricochet off his shin pad and into the net to give the Rangers a 2-1 lead with under three minutes to play.