It wouldnt let me edit my post so having to post again...
I seemed to have fixed the problem and narrowed it down to one of two plugins I had.
So I updated to workpress 3.9 today and my site didnt like it. LayerSlider bugged out and would no longer show, and my tmeplate hasnt updated yet.
I also noticed there was an update for Better Security, so I updated that. My site definitely didnt like that update as the whole site bugged out. Couldnt even reach my dashboard.
So I went into my FTP and removed Better Security and LayerSlider since I couldnt use it at the moment.. then after that, admin-ajax doesnt even show up on pingdom anymore.
Soooo.. I guess the problem is fixed?