We too have been downsizing but it's really tough when both of us are natural packrats. We do have some minor issues when he thinks my stuff is expendable and his isn't and vice versa.
We could make do with a smaller place--we're a tad over 2000 sq ft--but other than one bedroom that is strictly a guest bedroom that we don't want to give up because we do get out of town company, we do live in the whole house. Admittedly we do spend most of our time in the office during the day--the living room at night. But I do use my kitchen a lot and the enclosed back porch is essential as a pantry and storage.
But when one of us goes, the other will have to downsize to a smaller property or make other arrangements.
By the way did your brother/family get over the difficulties or. . .