Product Whitepaper

Introduction

Product priorities are informed by our five-year plan (click to download PDF version)

The objective of this whitepaper is to provide the facts and analysis required to determine how to prioritize WMF product development efforts. This document will:

• Start with the strategic priorities defined in Wikimedia's five-year plan.
• Provide yearly business targets for each of the Priorities.
• Put these goals within the updated context of (a) the state of Wikimedia projects as of late 2010/early 2011, and (b) some research on the broader web landscape.
• Use the yearly targets and updated context/learnings to establish a Product Roadmap for WMF development efforts.

"Product" here means the entirety of technology through which people receive and develop Wikimedia content, whether that be through the regular website, a mobile gateway, an offline copy, etc. It does not attempt to capture engineering and research processes, code review/QA, or implementation details. Nor does it include purely community-focused programs and activities.

Strategic Plan and Business Goals

The Strategic Plan established Movement Priorities for 2010-2015. The Movement Priorities consist of the following goals: Increase Reach, Improve Content Quality, Increase Participation, Stabilize Infrastructure, and Encourage Innovation.

Movement Priorities – Five Year Targets

In October 2010, the Board passed a resolution approving the following targets for July 2015:

• Reach:
  • 1B unique visitors per month (up from ~424M in July 2010)
• Quality:
  • Metric TBD, but target of 25% increase in quality relative to July 2010 has been established
• Participation:
  • Number of articles: 50M (up from 16.1M in July 2010)
  • Active editors: 200,000 (up from ~85,000 in July 2010)
• Diversity: double percentage of Global South (18.5% to 37%) and female (12.6% to 25%) contributors

These are “big hairy audacious goals” (BHAG), intended to inspire and rally Wikimedia’s collective energy. They are understood to be highly ambitious, and perhaps impossible to achieve. For the purposes of our planning, we will treat them as stretch targets and seek to align our programmatic work so as to reach them. They deliberately omit “infrastructure” and “innovation” (which are among the larger movement priorities) because these areas are seen as internal rather than external measures of success. That said, within the product context below, we do include innovation as an explicit priority, because it relates to certain product investments.

The five-year targets set by the Board will be complemented by additional key performance indicators developed and monitored by the staff on an ongoing basis. These KPIs will establish baselines and targets for:

• Broken-out reach indicators, such as geographically broken-down unique visitors, desktop/mobile/offline usage stats, time-on-site
• Financials, such as number of individual donors, financial reserves, and percentage of restricted funds
• Infrastructure, such as uptime of key services, site load time in different parts of the world, time-to-interactivity, availability of secure off-site copies, public content snapshots
• Content, such as number of media files, content objects in other Wikimedia projects
• Community health, such as editor retention and satisfaction

For the purposes of determining product development priorities, we are focusing first and foremost on the five-year targets set by the Board. Additional key performance indicators (especially those related to content and community) will also inform product development.

Movement Priorities: Business Targets for 2011

For planning purposes, the five-year targets need to be broken down on a yearly basis. In order to do so, we significantly need to refine our planning as to which program activities are likely to help us achieve these targets, as well as improve our forecasting.

For example, we may determine that in order to meet our 2015 target of 200k contributors we need to grow our contributor base to 100k contributors in 2011. This target would require WP projects to add 15k new contributors (net) over the next year. We should be able to assign projects and estimate each project’s contribution to this target. While both Community and Technology projects will be included in the yearly plan, the focus of this document is on Technology projects that help the movement achieve its growth goals.

The business targets should also reflect expectations of growth from mature projects and emerging projects. The drivers of growth are likely to vary based on where a project is in its lifecycle.

Framework for Strategic Product Analysis

The diagram above represents the overall framework for the Product Planning process. As stated, we start off with the five-year movement priorities and strategic goals to establish the business targets. We then put these targets within the context of:

• Community Trends: What are the major trends within the community? How are the different user groups evolving? How are their needs changing? Is there a healthy flow of users going from one stage in their lifecycle to another? Is there a healthy inflow of new contributors? How are projects retaining valuable contributors? How do these dynamics vary across project?
• External Trends: What are the developments in the overall landscape of the web that could affect how users interact with WMF projects? Are there broad behavior patterns that could impact the likelihood of new users contributing/tenured users continuing to contribute? How have user expectations been shifting and what could be the likely impact on WMF projects? Are there other disruptive technologies we need to anticipate?

Imagine each area of product development as a card that has written on it a description of that feature or product area (e.g. "slideshows for images shown in articles") and a hypothesis how that feature will impact the strategic priorities (e.g. "will increase likelihood that readers fully explore the content of an article").

The stack of "feature cards" could easily contain hundreds of individual cards: How do we sort them? The answer is the validation of hypotheses by experimentation ("Let's build a prototype slideshow gadget and see whether it increases usage"), but also by checking them against the established priorities, the trends in the community, and the surrounding context. This validation process ultimately needs to result in a prioritization of product areas. We can reach conclusions relevant to our short term decision-making, and posit questions that need to be answered to support longer term planning.

Product areas relevant to Wikimedia and impact hypotheses are described in further detail later in the document.

Basic User Model

There are limitless ways to get involved with and to contribute to Wikimedia projects. The Contribution Taxonomy Project is an effort to create a very granular view of these different roles.

For the purposes of this document, we are using a simplified user model which does not attempt to account for the full breadth and depth of contribution types: readers are users who participate passively in the Wikimedia projects. They can potentially be converted into new editors. After intense contribution over an extended period of time, they may become advanced editors, which often goes along with taking on more specialized functions in the Wikimedia community and acquiring additional access privileges, such as administrator status.

Community Trends: Reach

Readership Growth

Readership of WMF projects continues its historically strong growth. As of December 2010, WMF projects have approximately 395,000 unique visitors, which represents an approximate 14% increase from December 2009. As of December 2010, WMF sites generated 13.9B page views, a 22% increase from December 2009.

Source: comScore Media Metrix; Wikimedia server logs (pageviews)

Indexing the unique visitors per region to 100, we can more clearly see which regions show the strongest growth in unique visitors. India is specifically broken out because it represents a strategic priority for WMF. The three strategic priority regions identified by WMF – India, Brazil, and the Middle East/North Africa – clearly correspond to the strong growth trends in these regions.

Source: comScore Media Metrix

There are a number of possible explanations for the continued readership growth, including:

• Internet growth: The global Internet-connected population continues to expand.
• Content: WMF projects have increasing breadth of content at a requisite level of quality.
• Search: The content on WMF sites is quickly indexed and highly ranked in all key search engines.
• Lack of direct competition: For many of our larger projects, there are no significant direct competitors (e.g., online encyclopedias), though there are numerous other ways users may obtain information.

Wikimedia Ranking

WMF projects are currently ranked 5th in worldwide usage, measured by Unique Visitors per month. WMF projects have been ranked 4th overall for most of 2008 and the first half of 2009. In July 2009, Facebook overtook WMF projects as 4th overall.

Wikimedia's world-wide ranking can in large part be attributed to its availability in more than 250 languages, whereas many other top websites are only available in one country's language(s), or a set of languages widely spoken by Internet users. In the United States, WMF projects are only ranked 12th largest.

Community Trends: Contributors

(Note: Since the combined Wikipedia language editions have by far the largest contributor base of all Wikimedia projects, we are focusing on them. That being said, trends in other Wikimedia projects require further study. For example, Wikimedia Commons has more than 6,000 active contributors, and has experienced strong contributor growth through 2010.)

Every month, Wikipedia projects have a stable base of approximately 80-90k Active Editors and 10k Very Active Editors contributing to the projects (Active Editors are defined as editors who make at least 5 edits in a given month; Very Active Editors are defined as editors who make at least 100 edits in a given month). In addition, approximately 17-20k users become “New Wikipedians” every month (editors who have completed their first 10 cumulative edits in a given month). While the number of “New Wikipedians” is certainly small compared to the overall Unique Visitor count of 411M, it represents approximately 20% of the Active Contributor count, suggesting that there is, on a relative basis, a material inflow of new contributors to Wikipedia projects every month. While the inflow of new contributors is material, there are two points to note:

1. Even though there is a significant number of New Wikipedians every month, the total number of Active Editors has not grown. This suggests that there are a sizeable number of users that “leave” Wikipedia (the definition of “leave” is a complex one, which we will leave open for now). At present, we do not know whether these New Wikipedians are the ones who are leaving, or whether Wikipedia is losing its more experienced editors.
2. Over time, the Editor growth (or lack thereof) has become more and more disjointed from the Reader growth. With Readers growing at a 20+% rate and Editors slightly declining, the number and fraction of Readers that choose not to edit is increasing (see related analysis and discussion in Mako Hill's blog).

These numbers have been slightly declining over the past few years, though June and July have shown an uncharacteristic dip. When the data are examined on a project-by-project basis, we find that the English Wikipedia shows a larger decline than other Wikipedias.

Overall Editorship has been stagnant/declining

As previously mentioned, the number of Active Editors on Wikipedia has remained stagnant. We can approximate a trendline:

The pattern of stagnation and, in some cases, slow decline is not limited to a few Wikipedias. It has been observed across many different projects. Here are the Active Editor trends for a few projects (de=German, fr=French, ru=Russian, ja=Japanese, pt=Portuguese, zh=Chinese, es=Spanish, he=Hebrew):

Fewer people join the community every month

While the trend across all Wikipedia languages combined is one of stagnation, when looking at the number of people completing their first 10 edits in any given month, a much more noticeable decline can be seen in the aggregate count. The decline of "New Editors" is even more pronounced for some individual projects like the English Wikipedia. In March 2007 (the peak month), 14,734 completed their first 10 edits; in September 2010, only 6,677 people did so.

Editor Trends Study

In October 2010, WMF commissioned the Editor Trends Study to investigate trends within the Wikipedia’s editor community. As previously mentioned, the Active Editor base in the Wikipedia projects has been declining slightly over the past several years even though several thousand New Wikipedians join the project each month. This dynamic suggests that editors are leaving the project faster than new editors are joining, but the existing data does not tell anything about who is leaving. The central questions the Editor Trends Study aims to answer are:

If new editors keep joining Wikipedia, why isn’t the number of active editors growing? Are the new editors leaving as quickly as they join? Or are the more experienced editors the ones that are leaving?

Here are the main findings of the study (the entire paper may be viewed here).

(Note: the study does the most in-depth analysis on the English Wikipedia, with comparisons to the German, French, Spanish, Japanese, and Russian Wikipedias. As part of the project, there is a software toolkit that researchers may use to investigate other projects).

Wikipedia communities are aging.

By looking at the age composition of editors who contribute in a given year, we find that Wikipedia communities are aging. i.e., new users are becoming a smaller and smaller portion of the editing population. The following chart shows the age composition, in wiki-years, of editors that made at least 50 yearly edits to the English Wikipedia:

The data show the age composition of the English Wikipedia is clearly shifting, most dramatically between 2006 and 2007. We see a similar pattern across the larger Wikipedia projects, with the German Wikipedia having the lowest percentage of “young” Wikipedians and the Russian Wikipedia with the highest.

Some of this aging is to be expected. As a community matures, we should expect to see more “veteran” community members (assuming that these community members are retained at a reasonable rate). On the other hand, developing communities have higher percentage of new members (e.g., the Russian Wikipedia).

The retention of New Wikipedians dropped dramatically from mid-2005 to early 2007, and has since remained at a record low.

The following chart shows the change in retention rate (defined as percentage of New Wikipedians still active one year after making their 10th edit) on the English Wikipedia. The chart also shows the Active Editor growth.

We see from the graph that there is a noticeable difference in retention of editors that joined during 2004-2005 compared to editors that joined after 2007. The data show that new editors in recent years are leaving at faster rates than ever before. Of the editors that joined in 2004 and 2005, approximately 35-40% of them were still editing one year later (see paper for full results). Of the editors that joined in 2009, only 12-15% of them were editing a year later. Not only are fewer users becoming New Wikipedians, those who do cross the 10-edit threshold are abandoning at very high rates.

While there is not a definitive explanation for the decrease in retention yet, it is likely the English Wikipedia experienced an “Eternal September” effect from mid-2005 to early 2007. Under this notion, the influx of users created a situation where the existing community tried to accommodate the new users while at the same time accomplishing its work of writing a high-quality encyclopedia. In order to cope with the torrent of new editors joining, existing editors established defensive mechanisms to protect the encyclopedia (e.g., vandalism fighting tools, increased requirements for acceptable edits, etc.). These changes resulted in policies that, intentionally or not, made it more difficult for new editors to become acculturated within Wikipedia.

More in-depth analysis of individual cohorts of users reveals some other interesting trends. The following chart shows the retention patterns of editors that became New Wikipedians the Januaries of various years.

Editor retention has not worsened significantly over the past several years. While 2006-2007 saw a downward step-change in editor retention, the change in retention rate since 2007 has been relatively small, albeit at a lower level. This trend is shown by the close clustering of the retention curves of users joining between 2007-2010 and is consistent with the initial retention chart where the retention rate appears to bottom out around 2007.

Retention rate of veteran editors appears to be stable. We know that editors, even very active ones, leave the project for a variety of reasons. Some take wiki-breaks and others leave permanently. A certain amount of churn is to be expected. The recent activity of veteran editors suggests that they are continuing to edit Wikipedia at reasonable rates. This result is not entirely surprising as these veteran editors are the “survivors.”

Why Do Editors Stop Editing Wikipedia?

We know from evidence that the community can be harsh in its treatment of New Editors (Newbies). There is also evidence that suggests Newbie treatment (e.g., reversions with little or no explanation, hostile editors) impacts these users’ decisions to either continue editing Wikipedia or find some other online pursuit.

Reversion and Newbie Treatment

We have quantitative data that indicate New Editors are reverted more frequently than experienced editors. The following graphs from Erik Zachte show the difference in reversion rates for edits made by anonymous users, registered users, and bots for the English, Dutch, Spanish, and Portuguese Wikipedias:

Erik Zachte’s revert analysis shows that edits made by anonymous users are about five times more likely than edits made by registered users to be reverted. Anonymous edits have ~25% chance of being reverted while registered edits have less than 5% chance of being reverted. Moreover, the reversion rate of anonymous edits has dramatically increased over time while the reversion rate of registered edits has remained relatively stable.

Analysis from Ed Chi et al. describes a similar dynamic. In his 2009 paper, he shows that editors who make fewer edits per month get reverted at higher rates than editors who make more edits per month.

Each line in the above graph shows the reversion rate for a class of users (e.g., users who make 1 edit/month, between 2-9 edits/month, 10-99 edits/month, etc.).

This reversion data suggests a strategic question: How do reversions impact editors' decisions to continue editing Wikipedia?

To further dig into this data, it would be useful to assess how the percentage of vandalism has changed over time. If vandalism has increased together with reverts, the revert trend would be less potentially problematic. While vandalism cannot be reliably detected, some types of destructive editing (such as page blanking or replacement of a long page with a very short text) can be. Qualitative sampling-based research could complement this assessment. Using this data, we could better understand to what extent editors making good faith edits are more likely to be reverted now than five years ago.

Former Contributors Survey

Qualitative data is needed to help us understand how these revert trends impact a user’s decision to continue editing Wikipedia.

In January 20