Alfred Bluford “Rube” Walker was born on May 16, 1926, in Lenoir North Carolina. He was the son of Alfred and Buelah Walker, the oldest of three boys. He was raised in a typical middle-class family of the era. His father was a semi-pro catcher in his younger days. He would bring string home from his job at the cotton mill to wrap around golf balls until they were as large as a baseball. Once they were wrapped in black tape, the boys would have balls to play with.
He got his nickname while he was a batboy for the Lenoir team of the Class-D Carolina League.