Your mother made that decision about a relationship that was hers. You simply respected her decision, along with her right to maintain the end of her relationship with that other family member. This was a choice your mom had every right to make. We owe no one a deathbed reconciliation.
Your concerns about your ongoing relationships with the rest of your family are also legitimate. I’ve seen families crumble after a death and it only compounds the grief.
The scorned relative had their own role in their relationship with your mom. They could have, at any time, attempted to mend things without the prompting of her impending death. They chose not to, and that’s not on you. Now imagine if you went against your mom and told them anyway, and they called her to gloat? Or demanded she apologize? Or pretended nothing happened? Or offered a hollow apology? I’m not saying they would have, just that we don’t know—Hollywood deathbed forgiveness scenes are fictional. Humans are emotionally messy creatures.
I wouldn’t respect the wish to have a pet put down. A dog can grieve but that doesn’t mean it should be killed to be spared the possibility. And the dead won’t know anyway.
Maybe “the line” is to respect their wishes and choices while they’re dying but after death, give your own integrity more weight. It may feel like her decision pushed you out of your integrity, but the boundary is that it really was her choice to make. You respected that boundary.
I really think you made the right choice in this situation. With your mom’s apparent anger issues, I imagine you’re experiencing complex grief. Go easy on yourself.