Premature optimisation vs just doing it right

I often hear developers in the tech community quoting the XP slogan 'You Aren't Gonna Need It' (aka YAGNI). 'Why are you doing that? You're optimising too early,' they say, bringing in a reference to premature optimisation as well.

I've seen some particularly heated arguments in the Rails and Java communities around this recently. It's also been the pretext to some of the most frustrating conversations I've had, during my career.

I take to heart the principles behind YAGNI and premature optimisation. There's been many occasions where I've reflected on my own work, and realised I didn't pick the simplest solution or I wasted time on something that wasn't required. In these cases, I would have loved someone to have challenged me with those quotes, it would have been perfect timing.

I also take to heart the mantra that 'quality is free.' This is summed up in a classic book by Phillip B Crosby called 'Quality is free: the art of making quality certain'. The book describes how improving quality can actually improve a business' profitability, by reducing the the costs associated with poor quality and preventing defects. A key message from the book that stays with me is 'focus on improving quality' and other measurements will improve too.

One of the ways I act on the 'quality is free' mindset is by continuously trying to improve the code I write, and never being satisfied with 'hacky', unclear, or poorly designed solutions that are likely to cause pain in the future. It's not just about what you do, but how you do it, a Software Craftsman might say.

Where my frustration comes in is when quality improvements are met with the response 'You Aren't Gonna Need It', or 'you're optimising too early.' This, IMO, is a fundamental misunderstanding of what premature optimisation and YAGNI are about.

To quote Ron Jeffries book, 'Extreme Programming Installed':

YAGNI: "You Aren't Gonna Need It." This slogan, one of XP's most famous and controversial, reminds us always to work on the story we have, not something we think we're going to need. Even if we know we're going to need it.
What you won't find here is mention of doing things badly, and only improving them when it hurts you. In fact elsewhere in the book Ron describes how defects can hurt a project, as well as the importance of coding standards and clear, intention-revealing code amongst other things.

How about Donald Knuth's discussions of premature optimisation?

That also doesn't support this view, if you take the time to understand it. One of the key points Donald Knuth makes is, in fact, premature performance optimisations reduce the clarity of code, introduce unnecessary complications and increase the chance of incorrectness. His focus is on avoiding 'micro-optimisations', whose impact is likely negligible anyway.

The idea that these quotes should be used to justify not refactoring or tidying up code seems completely counter to their intention.

Embracing YAGNI and avoiding premature optimisations doesn't mean you should approach software development in a completely reactive way, not giving any consideration to risk or impact until things go wrong. And even if you do consider these things, you shouldn't over-estimate your ability to predict all potential consequences of poor quality work.

Well designed, clear and clean code is probably one of the best protections from future unknowns. It's generally easier to understand, less likely to be hiding defects and easier to change. Continuously writing poor quality software and taking short-cuts is a recipe for chaos.

The reason these conversations are so frustrating is if someone is particularly entrenched in this stance, it's hard to influence them until something has gone wrong. By that point, it can be too late.