I've done a few minor hacks in the installation a while ago, and will have to review and back up that stuff, so I can replicate it later, or just as a diary of my activities :-)), before I upgrade.
You might also check to see if there are now hooks available that weren't available before, so you don't have to edit the WordPress core. That way, your changes will carry over every time you update.
Before resorting to edit the core, always feel free to post your specific goals here and see if the community here maybe knows of an alternative way.
Speaking of "safe", I am surprised there is no WP forum dedicated to security... (correct me if i'm wrong)
Yes, there is no dedicated forum for that. From what I can tell, most people do come here with security issues or when they've been hacked.
How else can 3.4 be unsafe?
The easiest way to view updates with regard to security is to look at the WordPress blog filtered by security.
In addition, since those are unsupported versions, there may be issues that haven't been identified, because nobody is watching those versions anymore.
Other than security and support, I forgot to mention bug fixes. That's another good reason to keep up to date.