I’ve noticed in v0.79 that when closing the Preferences dialog after having changed a connection related setting (e.g. changing max upload limit from 7 to 5) while there are downloads in progress, occasionaly the dialog won’t close and the GUI stops responding (but downloads continue).
The freeze may be temporary, and cure itself after some time. You should, however, downgrade to the latest stable version of BitComet and stay with stable versions or expect strangenesses like this. For your particular problem it may help to uninstall and reinstall your video drivers. (Take the opportunity to check for upgrades, natch.)
There are some pretty nice features upcoming in the latest releases, but also (alas) some vital features that aren’t there. But everything’s in flux, so we won’t know what will actually make it into the next stable release.