The question of what car was the first muscle car will get different answers depending on the person being asked. Sure, there are some that will come up more often, but getting all the historians and aficionados to agree on one is not going to happen. Here are a few notable candidates, though.
1949 Oldsmobile Rocket 88
This one is probably the most frequently mentioned by experts when asked what the first muscle car was. Its 303 cubic inch V-8 could only muscle out 135 horse power, making it underpowered by later standards. It was the first time an American mainstream automaker had put a high compression, dual overhead cam equipped engine into a car for sale, though. It was also a new concept in that the engine, by contemporary auto design theory, would have been considered too large for the smallish car. It really was the catalyst (due to its success) that led to the escalation of engine power that would follow. For all these reasons, the 1949 Rocket 88 is named by many as the first muscle car.
1964 Pontiac GTO
Some point to the 1964 Pontiac GTO as the first muscle car. It was born when Pontiac put a 389 cubic inch engine (used in the full size Pontiac Bonneville) into a mid-sized Tempest LeMans and dubbed it "GTO". Note that, in this instance, the GTO was actually a type of Tempest LeMans. It would later become its own nameplate. The combination was proclaimed by many auto magazines as the hottest muscle on American roads, thus connecting it to the term "muscle car". The 389 (four barrel carburetor) came standard with the vehicle and delivered 325 horse power. Optionally you could upgrade to add 3 two barrel carburetors, bringing the horsepower up to almost 350. That type of power is more in keeping with what people consider a "muscle car" and for that reason, labeling this as the first American muscle car would make sense.
1964 1/2 Ford Mustang
It doesn't seem likely, since it came out after the 1964 GTO, that you could really consider the Ford Mustang the first muscle car (or first American muscle car). That said, it was definitely the first "pony car". That's an undisputed fact as the name and the whole trend were based on the Mustang. Ford's vision for the Mustang was to produce a sporty four-seater that was fairly short and lightweight and could sell for less than $2,500. There were four engine options available, but the ones that really delivered in terms of power were the 289 cubic inch V-8's. Both came with a four barrel carburetor, but one had a compression ratio of 9 to 1 while the high performance version had 10.5 to 1. This allowed the two engines to deliver 210 and 271 horsepower, respectively.
While you might even get some different answers as to what the first muscle car was, the three listed here are important enough to make most lists. Even if you don't consider the Mustang to be the first muscle car, it was definitely the first pony car.