Absolutely. You should have paid back the loan, paid enough premiums to keep the policy from lapsing, or prepared yourself for the taxes. If the agent who sold you the new policy which caused a change in your existing policy (this is called "replacement") did not go over this with you, they don't seem very professional.

The reason you might want to borrow the money is because a loan is not taxable income. Because it is a loan, you are able to receive money from your policy without paying taxes on it as long as your policy is still inforce. Once the policy lapsed, there is no way to pay the loan back and any money you have borrowed is considered income in the year that the life policy lapsed. Any agent or customer service rep worth 2 cents should know this and be able to explain it to you sufficiently.

BTW, you could have pulled the money out directly (instead of borrowing it) in which case you would have to pay taxes on any gain just like any other investment because it IS your money.