I made the Server it's own class, and I made the Game it's own class, and kept them separate.
But then I had the Game as a class within the Server, and the Player as a class within the Game.
So separate them. Why is the Game class "within the Server?" Does that mean it's a nested class. like Server.Game? Same with Game.Player, so that it's Server.Game.Player? Why would you want it that way? Maybe I misunderstood what "class X within class Y" means.
--- Update ---
IMO, yes. Nested classes are typically for the sole use of the enclosing class, like listeners, GUI elements, etc., not for general use by other classes in the project. I imagine you'll have errors to overcome during the separation effort, but the overall success of your client/server project requires breaking them apart. Variables from the Game class required by the Player objects should be obtained through appropriate interfaces.
Further, I suggest that Player objects shouldn't communicate with Game objects directly but rather through a server object which directs and controls game play. Client players would logon and request a game from the server, the server would match players and assign them to a game which it creates and initializes, and then the server (or a service/class it creates and directs) would provide the necessary game control and player interaction to play a game to its conclusion.
A stepping stone to a better understanding of this separation of user interface, program data, and program control would be to build a GUI (or several) using an MVC design.