An injury-time penalty from Jordan Hugill – his first goal in Norwich City colours – gave the Canaries all three points at the New York Stadium.
City fell behind to an early goal from Rotherham's Freddie Ladapo and then had to rely on a penalty save from Tim Krul to keep the scoreline at 1-0 going into the break.
However, in the second half a lovely flowing one-touch move got City back on level terms, with an own goal, and then Hugill tucked away a late penalty after the hosts had also been reduced to ten men.
Daniel Farke made three changes to the starting line-up from the defeat to Derby County last time out. Captain Christoph Zimmermann and Onel Hernandez missed out through injury and were replaced by Adam Idah and Grant Hanley. While Jordan Hugill was given his first league start up front at the expense of Teemu Pukki who dropped to the bench.
Rotherham made three changes to their line-up as well, with Clark Robertson, Daniel Barlaser and Mickel Miller coming into the starting eleven for Joe Mattock, Kieran Sadlier and George Hirst.
It was Rotherham who came out of the blocks flying as they closed down a ball out from Krul in the third minute. It allowed them to set up a quick attack as the ball way played forward to striker Ladapo, who got between the two centre halves and fired low into the bottom corner.
Rotherham then had a shot from Daniel Barlaser blocked on the edge of the area before a good break forward from Oliver Skipp gave Marco Steipermann as sight of goal, but his shot was well saved by Jamal Blackman. Blackman then saved again from Adam Idah moments later in what was a frantic opening 15 minutes.
Rotherham were then given a good chance to make it 2-0 when Robertson, grappling with Idah on his way out of the City penalty area went to ground and referee David Webb rather generously pointed to the spot. However, Krul made sure that justice was done as he dived and got a strong hand to Ladapo’s penalty to keep the scoreline at 1-0.
Idah and Emi Buendia then sent in efforts from range but Blackman superbly saved the first shot before Buendia was left appealing for penalty of his own as his strike was blocked by a hand of a Rotherham defender.
In what was an end-to-end and very open first half, Rotherham then missed a guilt-edged chance as Wes Harding got crossed for Mickel Miller to head over from four yards out.
The second half started with Idah hitting over from the edge of the box, but it took a while for Norwich to get going with Marco Stiepermann putting the ball into the net on the hour mark, only to be correctly ruled offside.
That seemed to lift Norwich though and five minutes later they put together a flowing one-touch move that saw Stiepermann and Buendia work the ball out to Aarons on the right where his instant cross was destined for the head of Hugill before Michael Ihiekwe diverted it beyond his own keeper and into the net.
Then on 71 minutes the game took a massive swing into Norwich's favour as Angus MacDonald, having lost possesion, lunged into a challenge on Oliver Skipp and was shown a straight red card.
That set up and end-to-end last 15 minutes as Rotherham had chances through Robertson and Matt Crooks, the latter superbly blocked by Krul, while City piled on the pressure.
Farke made a couple of changes brining on Pukki and Mario Vrancic for Stiepermann and Skipp. Hugill then almost got Norwich level in the final minute of normal time as he fired low to the bottom corner, but Blackman saved well and then got up to deny Pukki's rebound effort.
However, Norwich's pressure was rewarded in the third minute of stoppage time when Buendia broke into the box and was brought down by a late challenge from Florian Jozefroon.
That allowed Hugill to step up and send the keeper the wrong way from 12 yards out and spark wild celebrations from the Norwich City players as they secured all three points for the Canaries.
Norwich City: Krul; Aarons, Hanley, Gibson, Quintilla; Idah, Skipp (Vrancic, 80), Stiepermann (Pukki, 80), Rupp, Buendia (McLean, 90+6); Hugill.
Subs not used: McGovern, Vrancic, Placheta, Martin, Tettey.
Goals: Ihiekwe (OG) (66), Hugill (P) (90+3)
Yellows: Hanley (55)
Rotherham United: Blackman; Harding, Ihiekwe, A MacDonald, Robertson; Wiles, Crooks, S MacDonald (Wood, 72), Barlaser, Miller (Jozefzoon, 67), Ladapo (Smith, 76). Subs not used: Jahansson, Wood, Lindsay, Sadlier, Hirst, Smith.
Goals: Ladapo (3)
Yellows: Crooks (90+4)
Red: A MacDonald (71)
Referee: David Webb