There's a gap between charity evaluators and charity workers. Look at any article on effective charity, and you're likely to see responses from NGO staff voicing their frustration with the evaluation methods:
"You don't understand how hard it is to persuade donors to fund evaluation."
"We're reaching a population that no one else is reaching."
"The type of work we do just isn't measurable by randomized controlled trials."
"It's hard for really small organizations to get noticed."
The thing that worries me about these responses is that it's hard to be objective about your own project. I've been there. When I worked for a nonprofit doing global development and relief work, I was rooting for them. I knew their values, I knew their staff, and I loved both. You see this in devoted donors, too - once you've helped with a project, you want to see it continue. It's only natural, once you feel like part of the group.
But which charity you support is not the point. The point is to help people who need help. Right? That's why we're doing this, remember?
The flaws in any evaluation (and there will be flaws) are bad not because they shortchange charities, but because they shortchange the people who should be getting better help. The question we should be asking is not, "Is the evaluation fair to these organizations?" but "How can donors help the most?"
My home team isn't in an office in Boston. My home team is in a village somewhere. I'm rooting for them.