The man who went toe-to-toe with Adrian Peterson at a poker game last month tells TMZ Sports it was actually the NFL legend's mouth that caused the fists to fly.
Joe "Baby Joe" Castaneda said on Tuesday his May 27 brawl with A.P. at a social club in Houston, Texas all went down because the future Hall of Famer hurled an insult at one of his buddies.

Castaneda -- a professional poker player -- couldn't recall exactly what quip Peterson fired off .