If it boots you out every time you try to open it, my first thought would be that you installed it incorrectly, or are missing some files. I've never modded a PSVita, nor do I even own one, but sometimes, homebrew ports of games to different consoles require that you provide certain files yourself. I'd look into that. Check to see if all of the files between your PSVita install and your PC install match.
Beyond that, I don't think I can be much help.