I see three major factors leading to his exit:
Homesickness: Going back several months, there were whispers that Baker missed his family and the West Coast. After all, he moved from Riverside, California, just outside Los Angeles, to a city 2,000 miles away with the most inconsistent weather on the planet (and most important, no In-N-Out). I don’t think he was actually like miserable with homesickness, but kept in the back of his mind that he would like to be a little closer to home if a situation popped up. On top of that, while most of his teammates’ parents were able to attend almost every home game, that’s impossible to ask of Baker’s family.