A few things about your code. First, if you wish to change the size of the font in the JButton, you need to change the actual font of the JButton. Second, its better practice to add components to a JFrames content pane as opposed to the JFrame container. Both of these aspects are encorporated in the code snippet below:
JButton button = new JButton("BUTTON");
Font f = button.getFont();/*Be sure to import the java.awt package*/
button.setFont(f.deriveFont(8.0f));
frame.getContentPane().add(button);