Ok so I was looking around, and I found two other prxs, which have to do with the psplink, so I don't think I need those, but I had them on, I also found an alternate remote joy prx, which was smaller, and I assume older....
With these it worked... except the game just kinda stops working when you try to do stuff... It doesn't freeze, you can adjust the screen brightness, press the home button... but the buttons don't work..

I'll keep looking, it's a start...