This is hard, so I might take a couple of whacks at it, but here's one pass....
What Users Want
My main point is that what users want varies a lot, and that making a standard that was good for "users" involves making tradeoffs.
- Even a single user wants different and somewhat contradictory things. A single user wants both innovation and stability -- for new features, but also for old features to continue to work somehow. Just optimizing that alone involves a lot of tradeoffs.
- Second, there are different categories or roles of users. I'm a writer-user publishing this blog, and you (dear reader) are a reader-user. Although both roles may desire communication, sometimes the desires of the writer and the reader conflict. For example, writers often want readers to also see ads, while readers would just as soon skip the ads. To benefit both the needs and wants of both roles users requires some trade-offs.
- Third, even within a single role, there are variations. For example, among writer-users, some care about ads, some care about exact presentation, some care only about readers who have particular technology, or are running the latest software. Some care about users with accessibility needs and, unfortunately, others don't.
- Fourth, there are other applications, software, and work-flows those users might engage in (for example, copying web pages into email, reposting them in news feeds) and the user needs and wants may vary based on interoperability needs outside of the context of the user role in this individual action.
- Fifth, there are vendors who have different subsets of users who are important to them, based on their business model, and what other activities those vendors also support. As the business model varies (e.g., between selling advertising linked with uniquely provided services, closed hardware platforms combined with content monetization, or tooling and infrastructure), this influences which users, workflows, roles, and associated compatibility issues are important to the vendors who are implementing the standards. And vendors don't want impossible implementation goals in order to meet their user's needs. While you might not care about the desires of the vendors, motivating them is what causes standards to be implemented.
- Finally, it's important to think not only about needs now but also in the future. Do the standards allow sufficient evolution that new reader technology will work well with old content, while old readers continue to work with new content? Has the standard provided for a clean path for future extensibility?
The standards, the specifications, and the discussions around them require a very difficult balance between all of these ultimately user-driven factors.
Given all of this, a standard for the web should try to meet all these needs, not just some. While some end users indeed want "a browser that work on any web site they pick", that would also mean supporting ActiveX, Java, QuickTime, and the five-letter F word.
Those who are designing and coding web sites have complex needs (some motivated by "money"), butthe link to technical requirements is not very simple.
A publisher can't depend on anything being broadly implemented just because some spec says a browser MUST do something. A MUST in a specification isn't a law; it provides no push. The only role a MUST in standard actually has is to provide a check-box for implementations; if the vendor of the implementation says "I implement standard X", they mean, among other things, "I follow every MUST in the spec, and I also follow every SHOULD except when I have a good reason not to, which I can explain". That's it. That's all the standard really does, is give you something to measure against.
A good standard is one where MUST and SHOULD are used as sparingly as possible; only those things that are really necessary to accomplish interoperability in the simple sense that the roles (readers and writers, intermediaries, editors, transformers) can function with expected results. Not necessarily the same results, but within the range of expectation.