I have never written anything (other than technical papers) - but I do know that there is nothing wrong with contacting publishers directly - and that 5 attempted contacts of agents in a year is pitiful
It's a bit like complaining you didn't get a job even though you sent your resume off to five companies. I've been passed over for jobs I really thought were mine and for which I was uniquely qualified and, yeah it sucks to be rejected, but that's all part of life. I have one from a company telling me there was nothing suitable for me and one dated two weeks later from the same company offering me a job - after I had met my potential manager in person through a mutual acquaintance.
A good friend is a would-be writer, she worked as a freelance journalist, got articles published in magazines and made a comfortable living. She never broke through with her book but she is doing what she enjoys and it allows her to live in the country, away from the high costs of the city.
So stop feeling sorry for yourself and do some self promotion and networking. Complain when you've got three walls papered with rejection letters.