I don't have a specific answer to your question, but I have often found that the way to avoid the "infinite loop of resizing" you talk about is to defer your call to resize the window:
boolean correctAspectRatio = ...; // check to see if new size meets aspect ratio requirements
if (!correctAspectRatio) {
final Frame _frame = this;
final int _width = ...; // calculated width
final int _height = ...; // calculated height
SwingUtilities.invokeLater(new Runnable(){
public void run() {
_frame.setSize(_width, _height);
}
});
}