I think you may find
BoxLayout to be helpful for you.
Also, something you may have to consider is using multiple panels on your main JFrame. For example, you may have to create a panel to hold all the cards, a panel to hold the scores, and a panel to hold the Solve and Reset buttons.
How I could see this working is by you using a BoxLayout across the PAGE_AXIS, where objects are placed like this...
Object A
Object B
Object C
...along with the LINE_AXIS version of BoxLayout, which places objects like this...
Object A Object B Object C
...so you can get some sort of result like this...
PAGE_AXIS LINE_AXIS --->
| Object A.a Object A.b Object A.c
| Object B.a Object B.b Object B.c
v Object C.a Object C.b Object C.c
Hopefully this would get you a grid effect without the components filling to the entire size.
Hopefully this helps!