FYI
I've been deactivating plugins on the website I thought was the problem and running P3 after every deactivation.
Seems like "Query Monitor" or "itheme sync" plugin was the one causing the memory build up, ever since I've deactivated those 2, the ram seems to be stable. Maybe there's a conflict with something else...
I'll be monitoring everything for the next few hours/days.
Thanks for that suggestion!