08 December 2008

About The Wisdom of Crowds

In his book “The Wisdom of Crowds – Why the many are smarter than the few”, James Surowiecki makes - indirectly but nonetheless powerfully - a very good case for Knowledge Management or the leverage of individual and collective knowledge.

Simply put this way, that the many are smarter than the few is hardly a contentious statement. After all, a croud of say 1000 individuals should be smarter than only 500 of this same croud most of the times. You have more minds available to solve a problem/find an answer. However, what Surowiecki means is that a croud of 1000 can be – with the right conditions – much smarter than the sum of its parts even when it acts/decides in a completely uncoordinated way (meaning each individual acts/decides in isolation from the others). In fact, such a group can be (and Surowiecki gives plenty of examples) smarter than the even best experts in a particular field!

The three conditions for this group wisdom to materialise according to Surowiecki, are that it must be diverse, independent and decentralized.

On diversity, Surowiecki writes (chapter 2, part III):
<<The fact that cognitive diversity matters does not mean that if you assemble a group of diverse but thoroughly uninformed people, their collective wisdom will be smarter than an expert’s. But if you can assemble a diverse group of people who possess varying degrees of knowledge and insight, you’re better off entrusting it with major decisions rather than leaving them in the hands of one or two people, no matter how smart those people are.>>This can be hard to believe but Surowiecki then makes the case for this point very well and I cannot find any reason to disagree with him.

On independence, he writes (chapter 3, part I):
<< First, [independence] keeps the mistakes that people make from becoming correlated.[..] One of the quickest way to make people’s judgments systematically biased is to make them dependent on each other for information. Second independent individuals are more likely to have new information rather than the same old data everyone is already familiar with. The smartest groups , then, are made up of people with diverse perspectives who are able to stay independent of each other. >>
I would think that this condition is in theory much less contentious than the first one on diversity. However, the problem with true independence is that in practice, it is rather difficult to obtain. Often, decisions in a croud are made sequentially with each individual influenced by his/her predecessors.Therefore, Surowiecki advises that <<If you want to improve an organization’s or an economy’s decision making, one of the best things you can do is make sure, as much as possible, that decisions are made simultaneously (or close to it) rather than one after the other.>>

On decentraization, he writes (chapter 4, part II):
<< [..] if you set a croud of self-interested, independent people to work in a decentralized way on the same problem, instead of trying to direct their efforts from the top down, their collective solution is likely to be better than any other solution you can come up with. [..] Decentralization’s great strength is that it encourages independence and specialization on the one hand while still allowing people to coordinate their activities and solve difficult problems on the other.>>
However, Surowiecki then cautions that : << decentralization’s great weakness is that there’s no guarantee that valuable information which is uncovered in one part of the system will find its way through the rest of the system.>> He then asserts that for a crowd of any kinds to allow << individuals to specialize and to acquire local knowledge [..] while also being able to aggregate that local knowledge and private information into a collective whole, [..] [it] needs to find the right balance between the two imperatives: making individual knowledge globally and collectively useful (as we know it can be), while still allowing it to remain resolutely specific and local. >>

Well, well, isn’t this where/when Knowledge Management should come in? In fact, for all intent and purposes, this is a definition of KM I am satisfied to work with in an organizational setting: any intentional and managed changes or activities with a conscious objective to facilitate/enable what is highlighted in blue above. But it then highlights a fundamental reason for organizational KM to have so often failed to deliver: the lack of management recognition that collective knowledge in practice is indeed always valuable, with the potential to be very often correct and effective. Leveraging knowledge is then not just about realizing (and doing something about it) that each employee’s knowledge is valuable (and that’s already hard enough for most senior managements) but that the collective knowledge of the whole or groups of employees is even more valuable.

I think that a cultural shift is needed here for this realisation to become the norm rather than the exception. This shift has already started with the ubiquitous nature and global reach of the World Wide Web enabling huge crowds to influence decisions directly or indirectly (eg. Obama’s election). This shift now needs to enter the board rooms en masse. According to Malcolm Gladwell, “the tipping point” (see his book with this title) should be reached when between 10 and 15% of board rooms will have formally acknowledged the value and power of individual and collective knowledge. I can safely predict this will happen even if I cannot say when.