One important area that might be worth investigating is academic research. Time was, a researcher never knew what anyone else was up to until he arrived at the conference or received the journal. Now, you can often download preprints of research papers directly from the authors. (Making it possible, for instance, to assemble the entire proceedings of an upcoming conference.) The feedback that the authors receive can help them prepare their presentations, and the early distribution helps pipeline further research.

Some authors even offer up early (pre-submission) drafts of their papers online for the world to critique. As you might expect, there can be some tension here regarding secrecy, but it's different than the analogous tension in the business world, because the reward systems in academia and industry are so different. (Academics *want* others to steal their ideas, as long as they are recognized for them!) It might be a fun dichotomy to explore in your article.

Another new form of academic transparency lies with datasets and tools. It is becoming expected that researchers will publish their raw data online. This serves the dual purpose of keeping the researchers honest as well as allowing others to directly build upon their work. Similarly, some researchers post their custom software tools. In terms of replicability, downloading someone's source code is a far cry from reading two paragraphs describing their "experimental apparatus".

Yes, the academic and scientific world have long led the way in terms of practical applications of transparency -- such that you can already see how these axioms work out in them. (As you note, they *want* their ideas to be reused by others -- the open-sharing part of the equation -- so long as they get credit for those ideas, which is the reputation part.)

I didn't realize that there was a trend towards the open publishing of one's data, though. Very cool! When did this first happen? Was there a well-known Patient Zero who voluntarily did this, or has it just sort of been a slow, glacial transition?

I don't know the history, although I'd be interested in it. I'm sure you know that the sharing of scientific data was Berners-Lee's impetus for creating this ol' web in the first place.

I can show you an example, if you like. Here's a paper. The journal has a page where you can download the "supplemental data". The researchers have also set up an online database for their results, and put some of their tools on SourceForge.

Now, that's transparent!

My short attention span means that the last point you raised is the one sticking with me: Reputation is everything. The idea that if you don’t talk about yourself, other people will … that’s a strong one. What if you don’t have the time to talk about yourself though? How does the phenomenon affect those ‘power-rich, time-poor’ people you blogged about recently (drawing the Es on their foreheads)? Also, how does this notion map to groups/organisations rather than individuals? A corporation hires a PR agency to talk about them because they need the attention and need to control what is said. Could that collapse in on itself and lead to individuals needing help promoting themselves? British sci-fi author Jeff Noon dug into that idea a lot in a novel called Nymphomation in 1997.

I guess this question of ‘what if you’re time-poor’ applies to your second section as well: Tap the Hivemind. Will you fall behind your peers if you have no time to crowdsource and need to rely on your own gut instincts instead? Will tools arise to enable easy and fast access to groupthink? Are they already here, in nascent form (Digg, maybe?)

One other question I’d like to raise: Is the digital divide being widened exponentially by this sort of group activity? Is access to the hivemind helping the digital ‘haves’ outpace the digital ‘have-nots’ at an accelerated pace?

Minus ten points for repetition of the word ‘pace’ at the end there. Ooops.

Previous comments have rightly pointed out that one of the most interesting applications of this is in science, where radical transparency can go some way towards realising Robert Merton's idealised picture of science as normatively open. The PLOS journals are really interesting in this regard.

Another area we've been thinking about in terms of access to scientific knowledge is in medicine. Medicine's traditional, paternalistic relationship with the public has been predicated on restricted access to knowledge - through certain books or certain sorts of expertise. Now, anyone can self-diagnose, rightly or wrongly, which changes the way doctors have to think about their patients. We've written a bit about this (see chapter 3 of http://www.demos.co.uk/files/receivedwisdom.pdf)

Many doctors stil see this form of radical openness as a threat. But, if tapped, it could help revolutionise healthcare.

Secrecy won't be dead. It will simply hide in plain sight.

The hyperconnectivity and transparency of this kind of world accelerates the flow of information, creating incentives to hijack the process to push particular memes, including disinformation.

There is already a considerable interest in the potential of "IO" strategies to shift public attitudes among national government intelligence agencies, political parties, NGO's and commercial entities. The techniques for manipulating information flows will only become more sophisticated over time as the world becomes increasingly connected.

I've recently been party to the development of a scientific vision document for the Kaleidoscope network of excellence (http://www.noe-kaleidoscope.org/). Unfortunately, the document itself is not public yet, but I can tell you that one of the central ideas in it is the notion of 'Open Research'. In fact, the process by which the document was developed was pretty much an open-source process, albeit internal to the network (have to take it one step at a time). I can tell you a few amusing stories about that.

Two examples from the network of the concept in action are:
http://telearn.eu/ an Open Archive of publications on Technology Enhanced Learning

http://lp.noe-kaleidoscope.org/ The Learning Patterns project, where we were pretty radical about the idea of 'publish early, publish often'.

Of course, there are more established initiatives, such as:
http://www.dspace.org/

Tap the Hivemind:

two problems:
1. why should they care? So you have a problem, why should I waste my energy solving it for you?
2. those who do bother are often the ones you'd prefer wouldn't.

Just being the devil's advocate. Not that I agree, but worth reading Lanier:
http://www.edge.org/discourse/digital_maoism.html

Last comment in this batch. The place where we most desperately need radical openness is government. e.g., open source the voting machines.
http://www.openvotingconsortium.org/

Another refreshing example:
http://www.theyworkforyou.com/

Of course, all this maps very nicely to Ito's Emergent Democracy:
http://joiwiki.ito.com/joiwiki/index.cgi?emergent_democracy

As our society adapts to open-ness in business and social interaction, education (particularly learning and testing) is going to be impacted mightily. If a precocious 10yr-old can start in on first year lecture notes from MIT, whither (wither? LOL) curriculum guidelines that are age-based? And for those who need remedial work, they'll more and more reach for Google than they will try to ask the teacher for help, esp if the interfaces and material that is Google-sourced is of a higher calibre than is the teacher. As to testing, if open collaboration is such a key skill, does one try to test that skillset in isolation from any other subject or are all subjects tested allowing it? Interesting times indeed.

Tap the Hivemind:

Regarding yish's comments (playing devil's advocate) on the second idea of interest, there are ways to frame your project and engage fellow experts that will improve the collaboration.

I look for the like-minded (homophily) and I think tapping established affiliation networks is one way you don't have to reinvent the wheel.

I'm trying to organize a domestic diaspora network, people from Pittsburgh, to solve problems in the Pittsburgh Region. Scotland has a similar project which is successful, GlobalScot. The idea is to figure out a cause that will activate your target demographic.

Plans for economic development are remarkably opaque. At the very least, there is a high barrier of entry into the discussion. Investment nepotism and backroom deals exclude a range of fresh ideas, effectively stifling innovation.

My expectation is that the radical transparency approach will outperform the Good Old Boys Network, breaking the political gridlock that is hobbling Pittsburgh. The rub is getting the alpha intellectual captial that left Pittsburgh to embrace radical transparency, instead of building another Good Old Boys Network to challenge the established guard in Pittsburgh.

I started just 6 months ago targetting blogs I admire. Some of the bloggers were interested in Pittsburgh and others were interested in the various concepts I was exploring at my blog. We barter help and ideas, so there is an exchange instead of unidirectional support.

Informally, blog networks of knowledge production are developing. I would hypothesize that the progress of this network is moving the fastest along the lines of bloggers who most embrace radical transparency and are committed to this emerging form of ideation.

What are being brought to bear on a particular problem are the expansive networks of a small group of people I highly value and respect. My network helps me sort through all the information noise in order to find the signal that keeps my project moving forward.

So, I don't think "Tapping the Hivemind" is about throwing the masses at a problem, the wisdom of crowds. Instead, we see a network of a small group of people who embrace the tenets of radical transparency and online collaboration.

Bret, awesome! Thanks for those links.

Tony, that point about PR -- i.e. companies hiring people to talk about them, which is basically pre-Internet reputation-hacking ... a great way of looking at it. And of course, in a networked world where your consumers can trade notes on your marketing, overtly lame astroturfing and blatant lies get debunked more quickly than before. The point about time-poverty is well taken also, and, indeed, quite personal for me, because I regularly get swamped by work and can't blog for a month or so, and I wind up getting frustrated for three related reasons: a) it's inherently fun and I miss it; b) I have all this excellent stuff people keep sending me that's time-sensitive and which I'm losing out on; and c) as I lie in bed and stare at the ceiling, I can feel the Google juice draining from my reputation, heh.

Jack, yes, I was just talking to Don Tapscott -- who just published the really excellent Wikinomics -- and he made a similar point: Doctors have begun to adapt to the open availability of medical info online. Whereas they used to react in horror when patients came in saying they'd "done some research online" about their illness -- under the (correct) theory that a little knowledge is a dangerous thing -- many doctors now actively offer solid, accredited medical web sites to their patients and encourage them to read up.

Zenpundit: Precisely. Old-school secrecy was about hiding something so nobody could find it. New-school secrecy utilizes info-overload to hide the truth in plain sight. David Shenk pointed this out in his insanely prescient 1997 book Data Smog. Even back then, he could see that the new way that corporations were hiding unflattering information -- about their polluting records, financial malfeasance, etc. -- is not by locking it away but by releasing such a torrent of information publicly, most of mundane and meaningless, that any scrutineers are drowned in the flow and never notice the incriminating stuff.

Yish, great references -- thanks! As to the question of why should the hivemind care about your problem? Great question ... I don't know the answer entirely. Which is not to say that the hivemind doesn't respond; actually, it does, far more often and more passionately and more usefully than a cynical reading of human nature would ever predict. The bigger problem is, yes, with lame-ass contributions. But I also think it's a problem of garbage in, garbage out: Present the world with your boring, stupid crap and you'll get boring, stupid crap. Present the world with useful, interesting stuff and you'll get back in kind. Every time people bitch about how the conversation on blogs is snarky and pointless and bitchy, I kind of think, what are they talking about? Most of the blogs I go to have really amazing, rich conversations. But then I realize that when people use the word "blog" they tend to mean "political, gossip, or personality-driven blogs" -- and sure, the tenor of conversation on those is almost completely wretched. Go to a blog devoted to almost anything else -- neuroscience, knitting, guns, dog breeding -- and the conversations are often fantastic.

I couldn't agree more about the need for open voting machines, and indeed, I wrote an essay about that for the New York Times Magazine back in May 2004!

I agree with the posit that there is little privacy left.
With blogging and the like, I have moved a good chunk of my networking online both for my business ('New Jersey Concierges') and my blog ('Serge the Concierge').
My daily work puts me in contact with people near and far and it is enriching to get these diverse perspectives and ideas.
On the blog side, I decided to start a second blog where people will share food, wine and travel related topics as seen through the eyes of the native (or the expat).
I am currently inviting people to participate.
The 'transparency' issue has surfaced recently as related to advertising and writing about products and being paid for it.
Definitely a 'paid review' should be disclosed as such.

Have a good day

Serge
Biz:
http://www.njconcierges.com
Blog:
http://www.sergetheconcierge.com

wintermute, yeah, I love that idea: If global collaboration becomes a central part of learning, how do you test one's ability to do so?

Jim, that Pittsburgh project sounds incredibly cool. I've just emailed you to hear more!

An area that we've been working to make transparent is planning - garden variety, everyday planning. Everyone makes plans and most of the time we're at the mercy of our wits and experience to make the best plan for any particular effort.

So, why not openly share plans? How much IP is really in most plans?

Late last summer we released a free planning tool as well as a public plan server that allows people to openly share their plans. So far (with virtually no marketing) we have over 6000 people utilizing the tool and over 1000 projects have been shared - each number is growing daily. The variety of projects is astounding - for example, a plan for making a documentary in Afghanistan, house construction, nanobeam design. All plans are public, shared, and freely available for reuse and extension at http://www.sharedplan.net .

Beyond reuse in the realm of planning we also see this as raising the level of abstraction in open source. What are plans but a process that when executed are expected to lead to a desired outcome - sounds like a program to me. You can read more about this notion in a post on my blog.

We're in the early stages of this effort and have much more work to do but we are increasingly convinced that radical transparency as applied to planning is an idea that an increasing segment of the population groks and welcomes.

I would quibble with a single word in your "secrecy is dead" section and that word is information. I think you mean that DATA is plentiful. INFORMATION is not but the appearance of it is. We are awash in massive amounts of data, which makes the synthesizing of that data much harder. From there, secrecy reappears. For example, information about the ill health of Enron was plentiful (their 2000 annual report, for example, showed a company in very ill health) but few reporters dug beyond the pronouncements from the executive management and parroted lines about amazing growth. As a result, the public was misled.



Secrecy is no longer about hiding but about flooding. Drop mounts of useless data and bury a couple of pearls in it. Few will do the hard work to find the work. I always liked the tagline on Adam Curry's weblog (http://www.curry.com): "There Are No Secrets, Only Information You Don't Yet Have".



On tapping the hivemind, have you considered the difference between the "wisdom of crowds" and "tyranny of the majority". It is something that I haven't heard too many people talk about but what happens when the majority is wrong. Witness, for example, the invasion of Iraq, which was then supported by a majority of the American public. The situation has now changed but, at the time, the minority was called "unpatriotic" (or worse) for daring to ask simple questions about the validity and daring to ask that the reasons for war be debated BEFORE engaging troops on the ground. On a less politically charged end, look at the rise and fall of stocks during the Dotcom bubble and bust: a few cassandras were asking people to be more careful but (as is the curse of cassandra) few listened. The hive mind can be misleading at times so examining that facet could be interesting.



Reputation is everything taps into the two previous concepts: If reputation (and hence reputation management) is everything, are we created a new class system where those who can (either because they have the time and/or money) manage their reputation will be given an advantage over those who can't?

I've never seen an example of this, but I've always felt that political campaigns ought to be totally transparent. There ought to be embedded cameras and journalists who have 100% access to all meetings, with all content being posted on the Web. That would distill legitimate dialogue from the spin very quickly.

The Conservative Party here in Canada ran on a campaign of accountability and authenticity, but was there any more access than usual into their electoral tactics? Not a bit.

That's a pipe dream, of course, but the first candidate to do it would make major noise, and might win by turning a process traditionally cloaked in secrecy on its head.

Collision Detection

As your title suggests transparency's main contribution is that it keeps momentum flowing by minimizing collisions.

Everything, with a past and a future, moves by momentum. When you try to hide information between the past and future, with secrecy, you lose a certain amount of momentum. Each time someone tries to follow your reality and is stopped by that which is hidden, momentum is lost. I suppose we could say that there is a collision between what is real and your reality.

The down side of transparency is that much of that which is hidden doesn’t need to be available. As my fellow commentators suggest much of the information available we are not interested in and, if not made aware, can remain hidden in plain site. In that case transparency can lead to distraction. Transparency can let us see multiple realities besides our own.

What is really needed is trust. We don’t really need transparency, but trust that all we need to see is before us. Or perhaps more important, we need to know that there are enough connections available for us to find what we need to know.

To obtain the trust that is needed to keep transparency’s distractions from affecting our reality, we need connections that we can trust. With connections that we can trust we don’t need to lose momentum by distractions or secrecy.

Second that Dbarefoot, something I have been thinking about for sometime. How are we to trust politicians if we don't have the information, we can't.

I think an important point is, sure tap the hivemind but don't loose the ability to consider the personal instinct on something or else we loose the discoveries that are made when a maverick goes against consensus and takes the lone path.

Does it make it harder to follow your instincts when everyone is telling you you're wrong (on a scale never seen before)?

Also, when applied to transparency and wiki style contribution in writing, do narratives tend towards being more bland when others are involved, despite the principle authors best efforts to avoid it etc?

Hey Clive!

I think the ultimate counterexample of the moment is the iPhone. You can't imagine something like that coming out of an open, deliberative process. I think there is a difference between sort of general efficiency & utility and real excellence, and often it's dark secrets and small teams in locked rooms that still deliver the goods on the latter.

More here --

http://snarkmarket.com/blog/snarkives/media_galaxy/the_iphone_secrecy_and_excellence/

Robin -

That sounds almost like an intelligent design argument. You assume the iPhone is excellence, you assume if it is an outcome of a closed process it can only be that way, and therefore you conclude that excellence needs to be produced in dark rooms.

A tad circular.

By the way, have a look at Greenphone -
http://www.trolltech.com/products/qtopia/greenphone

No, it doesn't have the sex appeal of the iPhone, in fact its quite a different creature. But its interesting in its own right.

Back to tapping the hivemind.

Times I've tried it, including the kal vision project I mentioned above, it seemed like most input came from people I knew and approached personally. In that particular case, we even tried to build in incentives. The main problem is, most interesting people are already over-committed. Often the way to get them to do something is get them to do it for *you* or because they *care*, or preferably both.

I wonder Clive - how many of the commentators here are your 'regulars'?

Then, on the other hand, there's always the long tail (btw - http://www.longtail.com/the_long_tail/2007/01/totally_transpa.html), the occasional unexpected contribution which just might change it all around.

A couple of short disconnected comments:

1. It all reminds me of a friend who recently died. He was a writer but not particularly computer literate. When learning to use the computer he came down to breakfast one morning and announced he had learned how to use google and had found himself on the internet. He then sat there pensively for a moment with a worried look on his face and said "How do you get yourself off the internet?"
2. If politicians can make announcements directly on their own sites (as Barak Obama) they can no longer be controlled by the mass media's manipulation, interpretation or complete suppression of what they say. The mass media is reduced to the same status as any blogger, just another commenter.
3. I have always wondered how there could be any possible basis for details in government contracts being kept secret on a "commercial in confidence" basis. What happens to business in general if all commercial arrangements are public?
4. If the logic of this is that those who share, teach and give of themselves most will triumph (so to speak), does this not ultimately undermine the most basic logic of capitalist fundamentalism, ie that greed, scarcity and control triumphs. Isn't this the classic left/right dialectic, co-operation versus self interest, arising again in a situation of crisis (climate change) where game theory would suggest that anything other than co-operation will lead to certain disaster?
5. Does complete exposure result in the dissolution of the sense of self or the construction of a mask of self?

Hey Clive,

I went through almost all the comments on this post – very well made points.

It is yet a very ‘west’ perspective. That is not to say that here in India, we’re sitting atop elephants and surfing – but there’s still a quantum difference on the ‘Radical Transparency’ bit.

India’s a confusing, even perplexing mix of developed and developing economy and we’re still wrestling old economy with new.

Here’s a few thoughts from this side:

Secrecy is dead – from a business scenario point of view – secrecy would prove perhaps folly – small or new entrepreneurs have a constant fear of being eaten up by the Davids. An idea can easily be replicated – and this part of the world is so good at doing a better replication – open source could be scary. There’s still the Haves and the Have Nots – with a yawning gap between. Hell, even our star technology organizations (read outsourcing) don’t believe in open sources. They don’t even allow a flash drive to pass through their gates without a class Z security acting upon it first.


Reputation is everything – this is the happier part maybe. Among a small but growing community – knowledge counts. And keeping it out there – makes you well, special (for the lack of a better word). You talk, blog, present (during random brain-storming sessions in office) about the new, your own ideas, concepts – and you start getting referred to. It builds you. You’re the king. And this in turn leads to everything – from attention from the Executive President for a junior executive to new business leads.


On a separate lead all together – I’d really like to see the governments taking to this. Darn it – I’d like to know where 33% of my salary goes every year. I’ve been working in Bombay for 8 years – paying my dues to the city – but quality of roads, trains, water, and life has mostly sloped downwards. Yeah, lets push Radical Transparency to the government – right from pre-election – to tax-usage balance sheets.


Cheers,

Arti

Hey Yish -- I guess I just don't see any counterexamples from the open, transparent world that achieve that level of focus and artistry.

A good apples-to-apples comparison might be OpenMoko, a phone designed on open standards that looks super, super cool... but even so, it's a far cry from the iPhone.

I think 'excellence' is the wrong word. As I think about it I'm talking more about focus, design, & vision.

Serge, how does the online networking work in your case? Do people find you via links, searches, word of mouth?

Roger, that Shared Plan project sounds incredibly cool. I'm going to email you for more details!

Tristan great points. Yep, Shenk's book nailed that whole problem of massive data flows burying information -- or more importantly, wisdom. Though interestingly, the crowdsourcing approach to problems tends to fight this negative trend, because while it may be hard for one person to sift through a mountain of crap data to find the pearl of wisdom, it's a lot easier for 10,000 loosely-connected people to do so. It's all about the incentive. If the shareholders of Enron had felt that it was an important reponsibility of theirs to scrutizine Enron's copious filings (and it's kind of an interesting structural critique of the capital markets that they clearly didn't feel this was their bailiwick), then a few hundred thousand scrutineers cooperating on a wiki might have spotted some fascinating stuff long before the collapse. But, like I said, they didn't seem to have the right incentive to do so. Compare that with ridiculously complex video games, crammed full of totally opaque easter eggs, which are basically unplayable by any one person, but which are cracked open quickly by the tip-sharing culture online, including places like Gamefaqs. Because it's fun to do so, the teenagers groupthink the games really quickly; the problem is hard but the incentive is strong.

As to whether you get a caste system in an online reputation economy -- a privileging of the time-rich and in some cases the money-rich -- yeah, I think you do, and I think it's one of the system's biggest flaws.

dbarefoot, heh, politics is likely to be the arena of human endeavour most resistant to transparency. Teenagers totally get it; commercial enterprises are paranoid about their secrets but a lot of them still get it because they can see how it can, in many situations, make them more money. Political culture in the US is a lot more resistant to change.

Larry, good point about trust. Is it possible to build trust without at least some transparency? This is part of what Andri's asking.

Badluckchild, as to your point about whether creativity gets dumbed down by collaboration -- this is a really superb question, because isn't this the big complaint of so many scriptwriters and bands? That their publishers/labels/studios groupthink their ideas and water them down? I think that the hivemind is really good at solving problems -- but finding the spark of creativity that leads to a song/book/script isn't really a problem. I've no doubt there are good ways to harness collaboration to inspire creativity -- a lot of songwriters prefer to work in bands precisely because the collaboration produces the sort of emotional friction that leads to great art. Whether that scales up to a Net-wide approach is a really interesting question.

Robin: This is precisely what I think about your iPhone example. Apple relies on a very artistic, solitary process in creating its products, and it works incredibly well. It's also really culturally fun to have the iPhone be a big secret, in the same way that 24 is fun. That said, their level of secrecy is already biting them in the butt a bit. Had they explained to their users somewhere along the way that the iPhone was going to be locked down -- i.e. you can't instal any other third-party software -- they would have gotten an earful about that. Even superb industrial design, a very artistic process, can benefit from feedback.

Yish, to answer your question, I'd say about half the contributors to this discussion are the "regulars" at Collision Detection. But sure, hivemind projects only work if people care. As Clay Shirky points out, the vast majority of social-software projects die a quick death because the group dynamics implode, or nobody cares and nobody shows up.

Ian, that's a lovely story about your friend! Also: "Does complete exposure result in the dissolution of the sense of self or the construction of a mask of self?" Yes, one of the most interesting philosophical aspects that lurks beneath all the MySpacing, blogging, Wiki-ing, etc., is what individuality feels like in these deeply connected situations. I don't really have anything more interesting yet to think about that, though ... but it strikes me as a fascinating area. Plenty of folks who've studied the open-source movement have commented on the way it screws with the stuff that is considered atomic and unchangeable in traditional Marx/Smith/etc. political theory: Individuals, groups, the nature of property, etc. Yochai Benkler's The Wealth of Networks is probably the best book-length meditation on this.

Arti, really cool to hear your perspective from the business community in India. Speaking to the point that "opening up your process means people can steal your ideas" -- sure, this is precisely why most of my editors wouldn't want me openly blogging about my stories while I do them. But then again, not all industries work this way. The fashion industry is rife with piracy: No sooner does a top designer show her works off on the catwalk than knockoff copies start rolling out of the factories in China. But the thing is, most fashion houses know that you don't make money by prosecuting copiers -- indeed, that soaks up way too much money and time. No, you make money by going back to the drawing board and designing the next season's clothes. Basically, you have to be more creative, and it's that challenge that scares the crap out of a lot of industries, perhaps most notably Hollywood and the big record labels. Then again, the counterargument is that not all industries have a quick enough churn rate that you can fight copying by quickly turning around and issuing a new product line. Pharmaceutical drugs take a long time to develop and test and gain governmental approval, for example.

Clive,

I have a comment that tends to make use of both your "Tap The Hivemind" and "Reputation Is Everything" outlines. This comment relates to information distributors or what I would more accurately classify as news distributors.

It would seem to me that radical transparency will have an immediate success when it is adopted by news organizations. The evolution of this process will invariably bring about an independently derived open source framework which these organization will have to adhere to in order to be seen as relevant to the discussion. This is not to say that they will be forced by this future framework into being non-biased but that they will be forced into revealing their bias if they do exist.

Information today moves through many minds before it makes its way to the consumer. At the present time there doesn't exist a framework that makes this process transparent for those who wish to judge the validity of the output. A fundamental criteria for this open source information framework will be to prevent secret hiders from gaming the system by manipulating the information flow.

Over time an adherence to this framework will become an essential foundation for a news distributor to be regarded as legitimate. This open source legitimacy, over time, will then become a requirement for the average information consumer.

It seems to me that the end game in the battle to maintain secrets will eventually prove to be the polar opposite of what we used to hold as true. Anything electronically generated will be at risk, whereas anything generated by hand will be secure. If you want to keep something secret, pick up a pencil and write it on a piece of paper.

Jim, yeah, newsrooms worldwide are beginning to wonder how and whether to open themselves up more. The interesting thing is that the most radical experiment so far -- the LA Times' ill-fated attempt at the Wikitorial -- failed miserably because opinion-writing doesn't benefit from collaborative work (in the same way, probably, that many art forms don't benefit from collaborative work ... imagine a sonnet that 49 people have haggled over the construction of). What the Times should have done, and what smarter papers are doing now, is using openness not to generate opinions but to solve problems -- information-gathering and analysis problems.

As for being open about one's biases, that's another interesting question. I'm personally less worried about news organizations stating their biases up front, in part because I think most people, myself included, have such relatively well-honed b.s.-detectors that we can easily grok the political biases of any piece of media we're consuming. What I'm more interested in is seeing evidence from a media outlet that they're actually working to ensure balance -- i.e. recognizing their own bias and actively attempting to compensate for it in their coverage. The really big problem is not that journalists and bloggers don't reveal their biases -- it's that they don't think have have biases in their first place, or if they do it's because they're just, well, "right", and so there's no need to pursue balance because the facts they're presenting are the only ones worth considering. But the solution to that begins much earlier, probably, in education; this country could do with a lot more comparative education (in politics, religion, art) so that kids grow up understanding that there are more than one way to see the world.

Scott, heh, totally true.

Nothing like big, fascinating topics to generate bracing discussion, and, unfortunately, disjointed observations like those I've assembled here. Thank goodness blogs & their replies are not obliged to honor limits of time or space, and reconsideration is ever possible.

I come to this topic from a daily absorption with the tangle labeled 'the US healthcare system', a topic that stirs up big talk of its own. So I'll start with a couple of observations on healthcare aspects of Clive's 'research ideas'.

As long as 'healthcare' is concieved of as 'sickness care' (re-read Jack Stilgoe's passage: that's the orientation of his piece, regardless his intent) by ANY participant - whether patient, or doctor, or financier (employer, insurer, 'friend of patient'), healthcare will NOT be 'revolutionised' by infusions of information no longer kept secret, however much it may change treatment of maladies.

But let's set that aside for the moment; more pertinent here is the relationship of radical information openness to improvement in dealing with health matters. Tristan's caution - that while data is plentiful, reliable, actionable information is scarce - is critical in healthcare.

The pace at which doctors are adapting to information sharing is glacial, and piecemeal, and may possibly in the end be almost beside the point, because our individual and collective health will benefit most from a mass revisiting of what it means to be healthy - to be 'not-ill'.

re: Larry Dunbar's observation "As your title suggests transparency's main contribution is that it keeps momentum flowing by minimizing collisions."

I took the implication of Clive's blog title another way: that most of everything is open space, and that collisions are where interesting, measurable stuff - change - happens. Tracing, and, if we're lucky, anticipating, the loci of collisions are what makes the technologically enabled collaborative enterprise the more exciting.

'Secrecy is dead': Clive has perhaps inadvertently juxtaposed 'Secrets' with 'lies' here - a perilous juxtaposition....and while secrecy may be dead, uncertainty is very much alive and vital when it comes to deciding what to do with information.

'Tap the hivemind': the 'hivemind' is a sort of marketplace, having no more monopoly on truth, necessarily, than conventional 'secret-keepers'. In healthcare, as in other areas of interest, drawing inferences from agglomerations of anecdotes in the hope of slouching towards comprehension is certainly perilous, for, to further tweak Keats, "the wisest lack all incentive, while the least informed are full of passionate intensity." Tap the hivemind, but seek trustworthy filters to screen the sediment.

'Reputation is everything' is the slipperiest and least persuasive by far of the 3 ideas, for what is 'reputation' but selected truths - or selected lies? Reputation - "the estimation in which a person or thing is held, esp. by the community or the public generally" amounts to little else. George Washington was a terrific manager of reputation, and a startlingly poor field general. You probably wouldn't want Washington making your army's operational decisions - but if reputation is everything, he may become your unanimous choice.

Reputation is more political than practical, and so may form a starting point for decisionmaking, for action - but hardly amounts to 'everything'.

Secrecy is dead:

Was it ever there? Given enough money or political power, you could always find out almost everything on almost anyone. I find great comfort in secrecy is dead. Now I can look the man in the eye. I know almost as much about him as he about me.

Does radical transparency also lead to tricky catch 22 type situations?

If my design company is openly blogging as we go about our latest design and the audience is saying "No, you're doing it wrong" what do you do?

If you disagree with them and press on you risk alienating your audience and getting a big "we told you so" kick in the guts if you fail.

If you disagree with them but go with their suggestions to appease them then suceed or fail you've stop doing what you believe in and I dont think good things come of that.

Radical transparency definately changes (complicates?) the way you make decisions doesn't it?

Does saying to your potential customers "look we hear what you're saying but we disagree and we're not going to do that" work in the real world?

Yes, indeed, "trust comes from tansparency."

In my opinion, the simple and pragmatic journalism maxim commonly known as the Five Ws - Who, What, Where, When, Why, and How. (The Five Ws as a term and practice includes an unnamed H.) should be applied to any website or blog in order to be perceived as authoritaitve and trustworthy.

Who: Name the people or person in charge and provide their bios, with photos and an overview, as well as links to, downloadable files and/or web pages of past working experiences and client samples. (Some would disagree about the photos, saying that people make rash judgements based on mug shots.) Make it easy for your visitor to know who you are and be able to contact you or a representative of your business. There’s an oversupply of websites that don’t clearly tell you who runs the business, or these sites have a anonymous “contact-us” form that does not clearly identify who the form is actually being delivered to. This is one of the biggest credibility failures of many websites. If you can’t tell people who you are, your readers might think you are hiding from something. There’s more to the “who” function of any website. Explaining who you are should also include your version, or someone else’s version, of what gives you the authority to present information that is credible and trustworthy.

What: Any website should describe the nature of its owner’s business in easy-to-understand terms. The other part of “what” concerns quality of content and design. Is the content well written? Is the website graphically consistent and easy to navigate through? The Internet industry term for this is “usability.” In September 2005, the man known as “the king of usability” by Internet Magazine and “the guru of web page usability” by the New York Times, Jakob Nielsen, explained the irony of the web, noting succinctly that, while the primary purpose of the web is to provide information, it is also loaded with “bad content and [a] lack of information people need - either because it is not provided at all or because it is written in a poor, impenetrable style.”2 All you need is one search engine response to a query to realize that this still holds true today.

Where: Something is definitely amiss if there are not working e-mail addresses, a real physical address of where the business is located, and working telephone numbers listed in a easy-to-find spot on a website. Are you located in Silicon Valley, Silicon Alley (the Manhattan version) or Bath, England? Are directions to your place of business provided? Are you hiding?

When: Today, when it comes to information about the Internet, World Wide Web, and communications and information technology, in general, timeliness is vital, as everything changes so quickly. At the same time, we web surfers often fall into the trap of depending too much on the very latest information out there, when, in fact, there’s plenty of valid and important, but older, information available online about any given topic, dating back as far as the web will take you. The important thing is that anything posted on a website should have some kind of time-stamp on it, so the reader understands the currency of the information being displayed and can then discern its applicability or non applicability to the task at hand.

Why: What are the motives behind the website owner’s content? Is it clearly spelled out that the purpose of the content is to sell you something? Is the cost clearly noted, or is that information buried someplace at the end of a shopping-cart function? Is the content geared toward only providing information to its visitors in the spirit of sharing, or is there some other, not-so-evident, ulterior motive.

How: How was the information presented on the website discovered or created, and is it consistent with other information from other reliable sources?

from "SurfingThroughNoise: Riding the Online Knowledge Wave"

George Lorenzo
george@edpath.com
http://www.edpath.com/stn.htm

Ethan Zukermann reports from the Sunlight/Berkman conference on 'Ten Democracy Projects in Seventy Minutes'.

http://www.worldchanging.com/archives/005836.html

I see three themes there (my analysis): radical transparency, community empowerment, and massive participation.

Radical transparency: you should have access to any knowledge that affects your life. Governments and corporations should be fully exposed.

Community empowerment: communities need the access to technological means to organize and mobilize themselves.

Massive participation: social technology can allow the wide masses to collaboratively shape the democratic process.

gjudd: "Tap the hivemind, but seek trustworthy filters to screen the sediment." Yes, I was just emailing with someone about this point, and wrote this:

"It's sort of the way Wikipedia works by default, I think! Sure, *anyone* can contribute info. But in practise, only about 4,000 hard-core people actually police the entries, clean up poorly-written contributions, fact-check stuff, remove stuff that was added without without sourcing, etc. Sure, the mob *could* constantly fight back and re-enter their crappy/poor/flat-out-wrong information, and "edit wars" do indeed erupt. But they're far less common than you'd think, largely because of the sage work of this smaller core of volunteers.

The way to get a good hivemind is to create a structure that maximizes the good of a hivemind (i.e. the infinite-monkeys cosmos of labor and knowledge, people's deire to contribute) and reduces the downside (i.e. errors, stupid ideas, etc.) Wikipedia and Linux seem to have found great frameworks, but plenty of other social-software projects haven't. Indeed, as Clay Shirky points out, the vast majority of social software projects fail miserably!

In my case, I'm using the hivemind mostly to a) generate interesting leads that I wouldn't come up with on my own, b) generate interesting ideas that I wouldn't come up with on my own. I have a structure that keeps chaos from emerging, because ultimately it's only me who will write the piece. My incentive to try and maximize the hivemind's utility is to offer credit in Wired to anyone whose idea or work or words I use. So in a sense, I'm doing precisely what you suggest: I'm acting as my own fact-checker on all these contributions. "

As to the freaky downsides of the reputation world, yes indeed. I was also just writing an email to someone else who raised the same concern, and I wrote back:

"Yes, precisely -- to me, the most worrisome aspect of what I'm writing about is the reputation part. Good reputations tend to cascade upwards online; if people like what you do/think/are/say, it tends to attract more likeminded appreciators, and their combined google juice drives the people who hate your guts so far down on a Google search that their views don't matter -- no one ever finds them. But a bad reputation can cascade in precisely the same way -- worsening and worsening in a sort of rabid pile-on. A couple of very smart sci-fi writers (Cory Doctorow, in particular) have done superb jobs of illustrating what a freakshow dystopia *that* can be. At the moment, the only people who've experienced this personally are certain politicians, celebrities and corporations who slip up and are piled upon, and the pile-ons build up their own forward inertia until they finally exhaust their own kinetic emotional force and die out ... but a reputation can lie in tatters before that happens. But with blogs and online life, it's starting to happen to everyday people."

Yish, yes, this epiphany -- that if secrecy dies and you know as much about someone else as they know about you, it's sort of comforting, or at least stabilizing -- this was the central conceit of David Brin's The Transparent Society, which I'm reading right now. I interviewed Brin for this piece and he was incredibly smart.

badluckchild: "Does saying to your potential customers 'look we hear what you're saying but we disagree and we're not going to do that' work in the real world?" Yep, that's precisely the conundrum, eh? I'm still unsure how often companies actually try this. It seems like, more often, they pick what they're going to do and then simply pretend that this is what they "asked for", when it's patently obvious that they designed their product or service poorly because of legal concerns, bad engineering, an inability to satisfy their customer base's contradictory and mutually exclusive desires, or some combination thereof. Would it be refreshing to at least hear 'em admit, "oh, yeah, we totally hobbled your MP3 player with DRM because we're basically deathly afraid of Hollywood and the record lablels -- sorry about that, but we need to stay in business"? Or would that turn people away from their iPods/etc all the more?

George, that's a really lovely set of points about transparency in web site design. I can't tell you how many times, when I'm trying to contact someone, they refuse to cough up any contact info at all, so I wind up having to run a WHOIS search on their domain to find the phone number of whatever poor code monkey they got to register their domain name. I call him, he bumps me to the CEO or whoever's in charge. It's crazy.

Yish, great links ... checking now!

This idea works wherever it's been tried, or stumbled upon. The first thing I thought of was the implications of Web 3.0, specifically the
Semantic Web W3C is envisioning. It'll be here faster than we think.

The only counterpoint I think to the Truth and Transparency in the hivemind is the Microsoft Zune site www.comingzune.com. Of course, it was a teaser site and all the signs pointed to it being a Microsoft site when you clicked on the copyrights page, but it was a veiled marketing campaign...like much of viral marketing--be in the abjected horrible (sony's all-i-want-4-christmast-is-a-psp) or the very well done Zune sites. So with some things maybe a veiled transparency? A cel-layered transparency?

Clive,

I've not had time to read all the comments, so forgive me if this is redundant. I write about this kind of thing from a postmodern perspective, so let me give you a few thoughts.

Transparency changes the nature of authority, so don't expect high fives from the institutional hierarchy. I view this as a good thing, however, because all institutions exist first for self-preservation. No matter how altruistic their origin or calling, ultimately, they become self-serving. The best example I can use is the American Medical Association's lobbying arm for online medical information. The AMA needs to actually keep information from people in order to carry on.

The most interesting aspect of the work you're doing is that radical transparency forces everybody into the golden rule (the old one). If secrecy is history, the hivemind has more oomph than the "experts," and reputation is everything, then what we're really talking about is a governor that far exceeds anything humanity has ever known. I mean, our culture is a seething hive of secretive and "unique" individuals who care only about themselves, so what we're talking about here is nothing less that world-changing.

And here's the thing from my view. This will continue whether anybody likes it or not -- AS LONG AS THE WEB REMAINS FREE.

You made a reference to the "modern world" in your invitation above, but what you're really referencing is the postmodern world.

Premodern: I believe, therefore I understand.
Modern: I think and reason, therefore I understand.
Postmodern: I participate, therefore I understand.

One doesn't entirely supplant the other, but this is the cultural drift of the west. The structure of the web itself continues the movement, because clicking on referential links is known as as "deconstruction," and that will not produce people who are inclined towards blind faith in authority.

I agree Terry it certainly seems world changing.

The net and the transparency that comes with it is making us face a world where nobody is perfect.

However, unfortunately a lot of people are slow to catch on to that fact or exploit it when it suits them. As you said Clive, good or bad reputations can cascade and your one mistake can spread in meme form like never before.

I agree entirely George that trust comes through transparency, I just see many unfortunate and unlucky casualties of transparancy due to the fact that the audience is not always able to forgive.

Before you have a chance to make amends they'll move to the next entity that they believe to be perfect until that illusion is shattered and then on to the next etc...

I think the number of people who are able to do the transparency thing well is limited but the power will be with them.

I'm always trying to see the 'big picture' aspects and the biggest one for me is that we'll all change as a result of the reality of the three dimensions you raise, Clive. Particularly as the Mobile Web explodes via cell phones, the factors you cite become much more important by orders of magnitude. If there is a significant probability that any of our acts will be public and linked to us, then this forces most of us to much higher standards of behavior. Even those who would aim to deny this transparency will find their lives much more difficult if they try to swim against the tide.

Clive, you suggest that: "The way to get a good hivemind is to create a structure that maximizes the good of a hivemind (i.e. the infinite-monkeys cosmos of labor and knowledge, people's deire to contribute) and reduces the downside (i.e. errors, stupid ideas, etc.)", and note that "Clay Shirky points out...the vast majority of social software projects fail miserably!"

Just my luck that there's no finite-monkeys-proof cookbook for such projects....because my feeling is that something wikipedia-like would be a superb means of developing models for -and testing the quality of, and reasoning behind - a wide range of healthcare IT standards & policy initiatives. If nothing else, it might provide a means of springing decision leaders from SAPS ("siloed analysis/paralysis syndrome").

Say, if Clay has determined that the vast majority fail miserably, he must have come up with some definition of 'miserable failure'. My own quick search did not turn up anything definitive under his name - any tips on a reference?

(REPOSTING EMAILS SENT TO CLIVE...IS THIS THING ON? HEH)
I see both tremendous opportunity and some very scary situations arising from this, most of which would seem is already out of our control. Saying "Secrecy is dead" translates to "Privacy is dead" to me. And while information dissemination about ideas and processes could be beneficial, "private" information about a person should still be kept private. But as you point out, now you potentially have to "defend" yourself in the public's eyes when it comes to reputation. The problem really is the burden on individuals to be honest while posting their information as well as a burden on the consumers of that information to try to determine if that information is correct. How do we extract what's relevant?

On top of that, what prevents malicious obfuscation? I already spend a tremendous amount time playing "Where's Waldo" when trying to research anything. First I have to eliminate most product placement (unless I'm looking for products of course, but I still have to use this process) then I have to aggregate commonalities in results, then pick out relevant biases and rank those. Now I have some information that I can start to make an opinion on. And the whole time I have to wonder what motivated the person who put this info out there. The good news is that I can generally figure out what's really going on when pulling data from numerous sources. What we need is better sorting processes as well as weighting and suppression options based on our biases or objectives. I'd be interested to know what kind of data reduction people propose, or if those are clear opportunity areas (non-advertising and intellligent search engines?)

A hive works as a cohesive unit, and could be labeled as a super-individual (much like the human body could be viewed the same), massive amounts of uncorrelated data is more a bunch of angry bees than a hive.

I think highlighting initiatives from schools like MIT and Berkeley (and many others) which open their courses is extremely important. These programs have structure, reliable information and a valued reputation or brand. How do we build other highly valued information name brands? How do we keep them open, with minimal bureaucracy. Or is that a busted model?

In the end, having a movement for just more information just adds to our ADD society. Also, just giving up our rights to privacy because we feel nothing can be done about it is irresponsible. Information security and validation is going to be increasingly important in the coming years. Especially when information from us or about us has far reaching impact.

It'll make for an interesting article for sure, but if it's to begin a movement, lets make sure we're moving in the right direction...;)

(SECOND EMAIL EXCHANGE)

As for movements and whether they happen due to Wired articles, I'd be careful as well. I've been reading that magazine for over 13 years now and have definitely seen ideas published there come to fruition. Some are definitely discarded, but when you add more straw to the camel, well, you might just break his back. Malcom Gladwell's tipping point has almost become cliche lately but there's definitely something to it. Also, our love for media guided direction is almost ingrained in our society these days. People want to be told what to do and well formed ideas in the proper places can be very powerful.

If you're going to add fuel to the fire, pour it strategically.

Ironically, just that effect is what you're talking about when you mention snowballing reputations (upward and downward)

I totally appreciate your concerns as I've been privvy to online communities for 15 years or so now. I've seen micro cases of this behavior on BBS', Friendster and now especially on MySpace. I've seen people flayed on Slashdot and other similar types of "professional" gathering spots, and praised as well. Generally speaking however, most communities work like markets and those who put their reputations on the line, but back it up with worthwhile information are lauded by their peers. Personally, I see nothing wrong with being put down a rung or two or twenty if the information you provide isn't useful, or just plain wrong. In fact, this touches on what I called an "information brand" and is necessary, in my mind, to create reliable information sources.

For instance, if I think I have an idea that's worthwhile and want the world to know about it, it's simply not enough to just put it out there. As you pointed out, googleweight pretty much dictates who's going to be seen or not seen when others are searching for information. So some of my options of getting a better googleweight are to publish my information in a forum that already has some momentum (slashdot, technorati, youtube...etc), do some kind of viral campaign (in some cases just paying for it) or hack the weighting algorithms.

If I publish where established subject matter experts can give a candid review and they accept it, then it's highly more likely I'll gain googleweight, but also become a valued member of the brand I'm approaching. If I become part of the "inner circle" due to ideas, but lack community skills, my ideas still pervade, but I may not be welcome in the community anymore. Once an idea is out there and has relevance, it's almost inconsequential who came up with it (unless you're silly and actually want credit for your ideas...heh). Now we're talking hive behavior and even concepts like exaption. Those objects that are fit to survive will. But the greater whole, in theory, will survive even when its constituent parts are no longer present.

I don't think the branding and popularity contests (which you most likely won't see go away, regardless of openess of ideas...we're talking humans here) are the issue though, I think the motivations to become transparent are the hurdles. If it's enough to be part of a recognized brand, fine, but I have to want that to be enough. However, we all know Linus Torvalds (or should anyway), and he's a great example of transparency gone right (I'd argue one of the ultimate examples).

Consider using something more "Wired" like Secrecy is Tired as opposed to Dead. As in keeping secrets for information that should be open is oldthink. I still think it's important to try to keep our lives as private as we see fit, even if it is an illusion of privacy. Moreso, I encourage people to lock up their valuable information. You don't leave your car unlocked in a shady part of town, why leave your valuable information unlocked? I'm not a conspiracy guy, but I don't trust our government and I don't mind them knowing it when they read this..;) Worse though are the corporations that can use your information against you. It's very disturbing when my insurance company sends me health prevention information for recent ailments...isn't that what doctors are for? And now I'm ranting, but hopefully you see where I'm coming from with regards to secrecy/privacy and the struggle to hold onto it.

em333 -- well, I think anyone who does a viral marketing campaign that pretends to be "real" probably has to build into it the fact that it will eventually be exposed, and that this exposition ought to be funny or interesting or clever. All "alternative reality games" basically work this way, right? They start off looking like a suspiciously "real" web site, but the game is quickly exposed, at which point the audience is in on it and the fun is in tracking the development of the fictional narrative, told via fake web sites, actors, phone calls, etc.

Terry -- really excellent stuff! I like that tripartite formulation.

Barry, yes, that's one of the big questions: Does haveing more and more scrutiny of what you're doing and thinking make you act better, smarter, less like a jerk? Or does it just make you paranoid and cautious -- an emotional version of the Sarbanes-Oxley disclosure laws, as it were?

gjudd, if you want an example of Shirky discussing social-software failure, try this.

run_w_xcors -- really superb emails, and here was my reply:

"Awesome email, sir! Yes, precisely -- to me, the most worrisome aspect of what I'm writing about is the reputation part. Good reputations tend to cascade upwards online; if people like what you do/think/are/say, it tends to attract more likeminded appreciators, and their combined google juice drives the people who hate your guts so far down on a Google search that their views don't matter -- no one ever finds them. But a bad reputation can cascade in precisely the same way -- worsening and worsening in a sort of rabid pile-on. A couple of very smart sci-fi writers (Cory Doctorow, in particular) have done superb jobs of illustrating what a freakshow dystopia *that* can be. At the moment, the only people who've experienced this personally are certain politicians, celebrities and corporations who slip up and are piled upon, and the pile-ons build up their own forward inertia until they finally exhaust their own kinetic emotional force and die out ... but a reputation can lie in tatters before that happens. But with blogs and online life, it's starting to happen to everyday people.

For the record, I'm not so much trying to start a movement -- I'm not even sure a magazine article has the power to do that, anyway -- as to merely describe something that is already happening, whether or not we like it ... and to try and extract the good that we can maximize and identify the harm we need to minimize. But it's entirely possible that the harm will outweigh the good!"

Clive,

thank you for the Shirky reference, but I should have mentioned that I had read that very piece - & noted he had not gotten very specific about what 'miserable failure' means for a social network (of course it may merely mean that there's no society there, but that doesn't say much about the 'why not?' of it...).

here're a couple thoughts from worldchanging (and Charlie Stross)

Panopticon Singularity
http://www.worldchanging.com/archives/000353.html

The Rise of the Participatory Panopticon
http://www.worldchanging.com/archives/002651.html

btw there's also a long literature in economics on 'perfect information'

http://en.wikipedia.org/wiki/Perfect_information

e.g. "the market for lemons" won a Nobel prize

http://nobelprize.org/nobel_prizes/economics/laureates/2001/public.html
http://nobelprize.org/nobel_prizes/economics/articles/akerlof/index.html

for me i think privacy is a function of normalcy, of which there are different axes, levels, layers, or what have you that relate to an individuals' relationship to society and their (multiple) identities on a personal and group level. in general, the further outside the 'norm' one is, on all these axes, levels and layers, i think, the more privacy is valued; unless, of course, the norms of society are sufficiently tolerant and open enough toward a wide range of acceptable behaviours that can flexibly incorporate ranges of identity. the more monotonous the culture, however, the less need for privacy because, afterall, if you're not doing anything 'wrong', you shouldn't have to worry about anyone watching... and thus deviants (persons who deviate) are cast into the _outer dark_

Chris, increasing transparency definitely makes online reputation an interesting theme. Reputation management, buzz monitoring, Internet sentiment analysis can be defined as the collision of intersecting disciplines to transform unstructured online dialog (blogs, chats, boards, ...) into marketing insights to better manage brands, promotions and product strategies. Including consumer behavior and generated media (CGM), Old fashion public relations, Search engine optimization (SEO) and Search engine marketing (SEM), Data mining and computational linguistic such as natural language processing and semantic categorization.

With easy to publish tools like blogs and RSS content feeds, everyone can be an author, sharing subjectivity about something, about themselves. Users are online expressing their opinions. 44% of internet users are content creators according to Pew, whether it’s attack or flame sites, consumer opinions, blogs and forums. 39% of the top 100 results are consumer generated media such as blogs, according Elixir.

As Jennifer Laycock from Search Engine Guide puts it, as the world shifts more and more focus toward online research, the importance of knowing what other people are saying about your company increases exponentially. These problems aren't limited to big business either. With consumer-driven review sites like Amazon, TripAdvisor and the new Yahoo! Local populated with reviews from customers, it's more important than ever for even small businesses to keep an eye on their online reputation.

-arnaud

(copied from my blog)

I've always thought of open content / social technologies as democratic, or democratizing. But maybe what we're seeing is the rise of a new form of social structure: Reputocracy: governance by reputation. There's only one rule to this regime: the more you invest, the more your voice counts. Its a market based system, with a single currency - reputation. This is a tricky currency, because it does not obey conservation laws. Therefore, investment is calculated by the amount of capital you have times the risk you take.

For example, if Jimmy Wales writes "I hate french fries" in an email to a college friend, he's not really risking much. Hence that expression does not go far as an investment. If he writes "I hate MicroSoft" in his bio entry, he's in for a hell of a ride. So anything Jimbo says on wikipedia will have a huge impact. Never mind his formal role in the system. On the other hand, I can say whatever I want where ever I want and few people would give a hoot. I simply don't have that much reputation capital to begin with.

As with any economy, some investments pay off, others don't. And as with money economy, if I invest in you and you come good, we're both better off.

The other issue that came to mind is intellectual property. That's always a pain in open-source / open-content environments. Also, always a pain to figure out why people contribute where they can't capitalize. Perhaps, in a an extremely transparent system, we're more concerned about intellectual persona. Property is a funny thing with goods that are expensive to produce and cheep to replicate. Ask the music industry. But what if we can verify where a meme started? Then claiming possession of someone else's ideas is just, uh, too embarrassing. Let's say I post a great idea on my blog. Let's say you copy it, develop it, and get a noble prize for it. Let's say you forgot to mention where you first read it. All I need to do is post a link to digg. Now who's looking stupid?

Radical transparency must be a unilateral movement, meaning that you cannot wait everybody to join in simultaneously. Your competitors will not enter at the same time as you. That is the beauty of the risk of transparency. Scooping out the rival newsrooms, once the leit motiv of hard news journalism, will have to find news ways for those like you, Clive, who dares to ask everybody for ideas for a possible cover article. One thing is helping journalists to change traditional means: we are increasingly competing not to other reporters and media but with billions of people, with the best distribution and ubiquity in the world, armed with ever more powerful PDAs, having new media ready to receive their feeds, with rigurosity and precision in a second level of importance.
I think that the answer is to improve analysis and to learn how to read the signs of potential future news while they are still in their process of gestation, something I've been working in, which I call news pregnancy. But maybe open, radical transparency can be a tool to envigorate a journalism activity - hard news reporting, that is- slowly going into extinction. In any case, I don't see how it can do more harm.

For me this dialogue is the beginning of radical transparency as an economic model. Whence we collectively get that we are responsible for the destruction of our planet by our own actions/non actions, the desire for accountability at all levels will ensue. I wonder on the possibilities that in the near future data collection will allow all financial transactions by anyone open to public scrutiny. Some of us may start to budget!

But in my mind this model will stop any repeat of the current and previous century wars. People just wont buy it!We will have very different calibre of leadership...

The other thoughts are that the youth are ahead of the teachers in many schooling models
and that transparency is essential for transfer of learning(esp collaborative learning). Particularly right now when teachers have so much to learn from their students and the need for them to "get" social media tools (let alone the Net/Mobiles)is so great!AND transparency surrounding the dumbing down of education to provide a dependent workforce for the multinational colonisers brings me full circle to accountability for our planet. Thanks.

Firstly, I very much enjoyed reading the post and all the comments; some very clever people have added their voices here. So, timidly, I'd like to add this for your consideration: If the "new" web is not about content but about transactions, then transparency is only possible if those transactions have a manifestation in the physical world.

In the digital world the common man is unable to distinguish an actual transaction to a "generated" one. Even if currently there are still technical limitations that will make these transactions detectable, those limitations are fast disappearing. And, soon enough, without trust and transparency, "the hive mind" will be indistinguishable from an algorithm developed and employed by the entity that provides you with access to the system. Not intended to sound paranoid - just hoping that the rise of "honesty" and "transparency" in the world will penetrate the cell walls of the entities that have to tell themselves: "Don't be evil".

First observation: funny that the concept of “transparency” should be used to describe things which are now coming in to view. When something possess the quality of transparency, by definition, it cannot be seen, but allows all else to be visible, like air. A lot of the information about business, personal lives, etc. that were once transparent, was because the medium in which they existed, whatever that was, did not allow them to be visible. You could have written up things relating to your daily life and passed them out on a street corner, but I’m betting that this never occurred to single person, pre-blog era. Fifty years ago, there was no analog to blogging. Now that we have that medium, a lot, indeed, comes in to view. Seeing that we’re so much like everyone else, we can all afford to disarm unilaterally. Blogs make this possible!. The great irony is that all of those closely guarded secrets now make other people yawn if they bother to read at all!

What’s truly strange to me is how these new developments in visibility create their own, weird, obscurity. For lack of a better term, I will call it distant intimacy. I’ll use you as an example. Clive, there is an awful lot that I know about you. Have you ever considered this? If I were to list everything, so you could concretely grasp all of it, you might feel a bit strange. I know you’re married, I know what you look like, I know you have a baby, and much more. This creates a weird kind of intimacy, and all the more so since we’ve exchanged a few emails. For many who have created online avatars whether intentionally, or accidentally, I often wonder whether these people think about how these two worlds might blend. If I were to read the Live Journal blog of someone who posted copiously about all details of life, would they be put off if I found them at a party and began asking very personal questions about their public postings? I’m guessing they would feel a bit strange. If, on the other hand, I posted the same questions on the Live Journal forum where they originally appeared, they might be treated as a welcome indulgence. Why the dichotomy? Perhaps people are able to put each world, the virtual and the real, things in seperate compartments. For both the developers and consumers of such content, it’s hard to know which is which. Actual people are not as invisible as they likely would have been in the past to everyone outside of family and friends, but neither are they visible. Rather a state in between, the attributes of which might be compared to that which is spectral. One can now possess at the same time visibility and transparency. But what we get isn’t really real as we’ve come to understand real. So what happens if we fall for the fake? And perhaps I use the word “fake” inadvisably. Maybe we can create new, digitally re-mastered versions of ourselves which are preferable. The standards that we now use for determining what is real will become irrelevant. Thus, I would argue that we cannot at once reveal who we are without also fundamentally changing who we are. At the present moment, there is a disjunction between the various states of “real”.

If what we accept as real is bending, somewhat, then the new real has a strange feel, at least to the old-real (born before 1986). The human brain is so complex because its ultimate purpose is to hack other human brains. In a highly social world, an advantage is conferred upon the brain that knows what other brains are feeling and thinking. Language is the tool we use primarily to hack other brains, but built under language is a different hack which even animals use. But it only works when we can actually see and hear each other. Though it may be used by animals, it makes verbal language seem primitive by comparison. It is what we communicate by moving a single muscle fiber around our mouth, or by changing the intonation in our voice ever so slightly, or by the lifting of ones eyebrow. The emoticons employed in the new “real” fall somewhat short. Lacking the important cues, the sub-language, if you will, which made old-fashioned, in-person communication rich, seamless, and meaningful, I’ve observed that emails, instant messaging, and bulletin boards often make communication more difficult if what they attempt to communicate goes beyond something very simple, like, “let’s meet at 5 for coffee”. Without facial expression, voice, and posture as markers, the message becomes either misunderstood, and garbled. As a result, when people resort to computer mediated communication, they often communicate on what seem to be on the extremes of emotion. Anonymity, of course, only adds to this. This is not to say that messages don’t get mixed in in-person communication. The difference is that when they do, it is often deliberate, not accidental. Thus, radical visibility not only obscures, it distorts.

After having thought about it, it’s seems hard to know what people intend by putting the personal, intimate details of their lives on the internet. It seems like in many cases it is just an opportunity to give vent to ones feelings. The larger consequences of such acts often do not seem to be very well thought out. Perhaps one day the world will be clearly stratified by people who are able to keep secrets about themselves (perhaps only because they never had a computer or access to the internet) and those unable, with the former occupying the higher rungs. One might hope that one would have control over the commingling of the compartment of reality and the compartment of partial reality, but there really is no control. What you put online you may as well assume is before the eyes of literally the entire world forever. Unless you come to terms with what’s out there about you, you’ll never know how well every stranger might know you.

Hello Clive,

Let me first thank you and Wired editor-in-chief Chris Anderson for starting a movement rightly expected from a modern mind.

All of us know that the way we are leading our life today can never be sustainable for long .We are to find out the alternative. Your work is a positive step towards the solution. Allow me to contribute my feelings to make your research a success.
Clive , i feel we must try to find out the root from where we must start our work on "radical transparency" . My belief and experience is that the transparency starts with a SIMPLE MIND . A simple mind is such a mind which is not burdened by the complexities of the modern world. Our mind has become very complex (intentionaly or unintentionaly) due to our education, beliefs, likes , dislikes, prejudices and we have lost the power to see 'what is'.If we can approach our life (personal,social,commercial etc.) from a simple mind , our life will be a much more graceful experience.But to develop a simple mind we need to accept 'what is' rather than only what we like or more precisely what we do not like to face. In my blog--http://supriyod.blogspot.com i have started working on this point and i invite your suggestion and support for the work.

Regarding , practical experiences let me share my practical experiments. I am a professional accountant and run an accounting organisation involving number of young professionals. Whenever any problem arises (in most of the cases re. human factors) i arrange a meeting of all concerned and help everybody present their point transparantly without keeping in mind the possible outcome (which may seem to be grave at times).But i always found that the result becomes ultimately positive and relieves everybody from long term mental sufferings although the process sometimes become painful.

I hope i have made my point clear and SIMPLE.

ALL BE HAPPY

Supriyo Datta
Kolkata, India

Clive;

What an interesting experiment... one that I will be blogging about shortlyI

Our explorations of Trust and Transparency are akin in their radical nature. I am a marketer in toronto who has been blogging about the notion of radical trust for several months now. The foundation of such, sounds much like the notion you are researching.

Have a look at some of my postings, they may be of some use to you...
http://www.radicaltrust.ca/category/rules-to-live-by/

all the best
collin douma
radicaltrust.ca

well done everyone who wants to see open source government it is the future for democracy but first we need to rack the collective brain on a way to untangle a monarchy an aristocracy a democracy and a european union that cannot agree on a constitution - A Constitution is the free software that must underpin an open government in a society increasingly enamoured with liberty and anarchy fuelled by the mistrust of closed government while insulted and disilluusioned by a miriad of invasive petty lawmaking. for an open government we must each own the fundimental laws - and a constitution will probably need a revolution but I digress a little.
In all practical attempts to establish a society of openness and trust in a large population the result has been catastrophic because someone will always keep a secret that is in their best interest and the almost total dependace we place on the internet, for information, serves only to make the keeping of secrets easier. The communist Ideal does not work in practise because it is not in human nature to be that open and all the information posted in public has and will continue to be placed only for selfish reasons. this is ideal to create a platform where vested interests may come togather because mutual vested interests can move more powerfully if they move togather, and goes some way to explaining the boom of e-marketing.
But trust is being lost on the internet as opinion is more highly valued, in this space, than fact.
Radical trust is lunacy and to claim that you have it or that you are deserving of it is disolusionment at best or an attempt to disguise your real motives.
The Gold mine did not show radical trust - they just posted a very open job offer with a fixed one-off salary (prize) this is clever marketing and nothing to do with trust.