I'm really the only one here with a Vita, so I'll answer your questions, kind citizen.
Assuming you don't already have any UMDs, you can download and play PSP games off of the PSN store. There are quite a lot of games available, and they usually add more every week. Unfortunately, not every game is available, like Kingdom Hearts: Birth By Sleep for example. This is usually due to a publisher, sadly. But otherwise, pretty much 95% of the greatest PSP games are fully available for download.
I'm not sure what you mean by "download play" unless you just mean remote play. I've never actually used this feature since I don't have a PSP, but from what I hear, it's alright. Basically, you can use the Vita to turn on the PS3 locally or over wifi (meaning anywhere in the world) and access media and some games (Battlefield 3, for instance). Copyrighted media can't be loaded, and only some games work, again, due to publishers. If you have a hacked PS3, you can edit some file that lets you run any game over Remote Play. Also, some games, like LittleBigPlanet 2 and PSASBR support using the Vita as a controller. From what I know, you don't need a copy of the game on both systems to do this. (Also, if you by PSASBR on PS3, you get a digital Vita copy for free.)
There are a lot of PSP RPGs to look into, like Persona (1-3) and Monster Hunter Freedom Unite. Mostly JRPGs, so I hope you're not looking for any WRPGs. As for the Vita itself, Persona 4: The Golden was recently released and an action/RPG from Mega Man creator Kenji Inafune called Soul Sacrifice is coming early next year. There's also been a few good RPG releases on PSN. That's not all that's available though, just some good mentions.
Over a PSP, I would definitely recommend the Vita over it. I've gone back to my PSP to demo a PSP game before I bought it and the system just felt cheap and the screen looked ugly compared to the Vita. I don't own a 3DS, so I can't say one way or another, but most of the games on the 3DS don't interest me, and the Vita's hardware has a bit more potential for better looking games (Uncharted: Golden Abyss) and games with creative control schemes (Tearaway).