...on my own I had to pay 0.25% of the loan amount ($580) fee. What can I do for them to drop this ridiculous fee?

Wow! what an awsome agent you have, your trying to save money by paying taxes and homeowners insurance yourself. Personally when I provide a mortgage I show them the benifits of paying them theirself. In which over 30 years it could mean tens of thousands of $$$ to themand their family. I would say you should find someone else to do business with. There is no point of punishing someone trying to do the right thing.