There are several issues here, lets break it down one at a time:
1. Your agreement means nothing. Federal tax law determines who can claim a child in any given situation and you don't get to just overrule the law by making an agreement with someone. That's why I can't just let a random stranger claim my kids to get a bigger tax benefit than what I get.
2. Although your agreement is worthless in this matter, your situation (unmarried, cohabitating parents) is one of the unique scenarios where both parents actually qualify to claim the child. The IRS will allow either ONE of you to claim the child as long as you don't both claim the child the same tax year.
3. Note that the fact that you paid a majority of the child's expenses is irrelevant because that is NOT one of the requirements of claiming a child. The requirement is that the child is related to you, passes an age test, lives with you, and the child does not pay their OWN expenses. Under these rules a parent who lives with a child can claim that child even if that parent didn't pay the child's expenses.
4. Because your ex legally qualifies to claim the child (point #3), neither she nor her accountant are committing any wrongdoing by filing an amendment to trigger an investigation so that the IRS can settle the matter according to the tax law. So no matter what happens, don't expect compensation from either of them.
5. In this situation where there's a duplicate claim (two people claim the same child) where both claims are valid the IRS has tiebreaker rules to apply to determine who gets the claim. Those tiebreakers determine who gets the tax benefits (not your agreement)
6. The first tiebreaker is that a parent always beats a non-parent. That doesn't apply here since both of you are the child's parents. In this case the next tiebreaker is that the parent with the higher AGI (Adjusted Gross Income) wins the claim, and the parent with the lower AGI gets nothing.
7a. Since you mention that you paid most of the bills, I would assume you had the higher income for most (or all) of the years in question. If my assumption is correct then you'll win the claim, keep the refund you already received, and your Ex's request for additional refund via an amended return will be denied.
7b. However, if the IRS sides with her because she had a higher AGI then you will have to repay any refund that was based on claiming the child, there will also be interest charges applied. So do not just ignore the IRS letters and think this will all go away on its own.
8. The IRS is not (yet) asking you to repay your refund, they're only asking that you verify what you claimed on your tax returns. That likely means providing a birth certificate to prove that you are the biological parent and then address records showing that you lived with the child during the year's in question. Your address should be easy to verify with things like employment records, voter registration or old utility bills with your name and address on them. To prove that the child lived with you look for doctor's records, school records or possibly copies of a lease listing your daughter as a resident or anything else that ties your daughter to living at the same address. It might also help in the dispute if you can prove that your ex was also living there, so that she can't try to claim she was living elsewhere with your child.
9. If both you and your ex are able to verify your relationship to the child and that you lived with the child, then the IRS will automatically apply the tiebreaker of the AGI, you don't need to provide anything for that, since its on both of your tax returns.
10. The IRS will never discuss your ex's returns or amendments with you. They will only tell you whether or not they have verified your claim or if they expect you to repay anything. They won't say what they determined about her or whether or not they applied any consequences to her for filing an invalid claim.
Good luck, this could get messy, but whatever you do, do NOT ignore the letters from the IRS.