When is Significant Self-Sacrifice Obligatory?

A popular way to deny Singer's conclusion that we ought to donate all that we can afford to the most effective international aid charities is to posit an agential prerogative, or 'moral liberty', to pursue our own projects. Requirements to aid in one-off rescue situations (e.g. drowning children) are compatible with this, but ongoing requirements to reshape the world are not.

This sounds like an appealing result. But I wonder if it can be sustained on further examination: Could it really be true that we are never morally required to sacrifice our control over the general shape of our lives?

Perhaps the most obvious counterexample would be if we were sometimes required to sacrifice our life itself. And it does seem clear that this can be required: I may not steal the cure to a deadly disease from a chemist who is about to mass-produce it to save millions, even if I'm sure to die without immediate treatment.

Less dramatically, we may also be required to sacrifice our long-term liberty and/or living standards: A convicted felon, sentenced to life in prison, may not murder his guards for the sake of escaping to freedom. A wealthy slave-owner, whose wealth and way of life depends on the ongoing exploitation of his slaves, is obliged to set his slaves free even if this would spell the end of his life of leisure.

If morality can demand all this, is it really so incredible to think that it might also require the wealthy to sacrifice their lifestyle to free those who are enslaved, not by people, but by poverty? Indeed, would it not be more incredible to think that it would demand one but not the other?

The best hope for the opponents of Singer may be to combine the above approach with the (notoriously difficult to pin down) distinction between harming and failure to benefit. That is: we can be required even to benefit, in one-off rescue cases, but life-shaping demands can only be negative: requiring us not to harm, rather than to positively benefit.

This hybrid response is certainly stronger than taking either route in isolation. But I still see two major hurdles for it:

(1) Is it really plausible that we're never required to make life-shaping sacrifices to help (and not merely avoid harming) others? Suppose another agent is unjustly imposing great harms on others, and I could successfully fight this evil if I dedicated my life to doing so. Isn't it clear that I should? But now what difference does it make if the harms are imposed by circumstance, rather than agents?

(2) I'm not at all confident that the putative distinction between harming and failure to benefit can be metaphysically vindicated (cf. Kagan, Unger, etc.). Note, for example, that the doing/allowing distinction won't help if your bank account is set up to donate your savings to charity by default. Reducing your charitable donation is then something you do with each purchase, but intuitively should still count as "failure to benefit". But doesn't this intuition just rest on our question-begging presumption that our money is rightly ours to spend, rather than any natural (pre-moral) distinction? Perhaps one could appeal to the global counterfactual of whether others would be better off on net had you never existed? If 'yes', then you qualify as harming them, and if not, then you merely fail to benefit. But this seems messy and vulnerable to counterexamples of its own (e.g. saving someone's life and subsequently enslaving them so that they're only slightly better than dead).

Any suggestions?