The Wikipedia.org Portal and Ukrainian Wikipedia

Background

For the past several months, Discovery’s the Wikipedia.org Portal team has been working on redesigning the (wikipedia.org) page to have a cleaner, more welcoming, and less overwhelming design and user experience. On 16 August 2016, the team deployed a major patch wherein the links to Wikipedia in 200+ languages have been put into a drop-down modal that is hidden by default and is only visible when the user intentionally clicks the “Read in your language” button (see Figure 1a and 1b below).

Figure 1 (a): Default experience when visiting the Wikipedia.org Portal with Russian as the preferred language. Our deployment on August 16th, 2016, collapsed the language links into a hidden-by-default modal in an effort to make the page’s design cleaner and less overwhelming to new visitors.

Figure 1 (b): Link to Ukrainian Wikipedia is only visible by clicking the “Read in your language” button to reveal the modal containing links to Wikipedia in various languages.

Figure 1 (c): How the Wikipedia.org Portal looks to a user who has set their language preferences to Ukrainian (first) and Russian (second). The links to Ukrainian and Russian Wikipedias are now the first links around the globe.

Per some comments in a thread on mediawiki.org, there is a suspicion that the deployment of the collapsed language links has decreased the page views to the Ukrainian Wikipedia as a result of some users’ browsers being configured to Russian, thereby burying the link to the Ukrainian Wikipedia (shown in Figure 1). The goal of this analysis is to test the hypothesis that the deployment has had a negative impact on visits to Ukrainian Wikipedia’s main page from the Wikipedia.org Portal. That is, this analysis is not concerned with Ukrainian Wikipedia overall pageviews or pageviews from somebody searching on the Wikipedia.org Portal and going to a specific article or pageviews from search engines and other sources.

Figure 2: Pageviews (on a log scale) to Ukrainian Wikipedia main page and other pages from users with Ukrainian IP addresses and users with non-Ukrainian IP addresses.

Methods

We used Bayesian structural time series (BSTS) to model Ukrainian Wikipedia main page pageviews from the Wikipedia.org Portal, using the R package “bsts” (Scott et al., 2016). See [1], [2], and [7] for more details.

We tried multiple models, including different combinations of:

• seasonality (weekly & monthly) components,
• a one-back autoregressive (AR1) component,
• specific pageview time series as control time series (e.g. Russian Wikipedia Main Page pageviews from the Wikipedia.org Portal, Ukrainian Wikipedia Main Page pageviews not from the Wikipedia.org Portal),
• dynamic time warping-matched pageview time series (“markets”) as control time series, such as Ukrainian Wikitionary (Main Page) pageviews from a Wikimedia project/tool/bot (which had the 2^nd highest similarity).
• a “mix of markets” that included similar markets and markets with the highest posterior inclusion probability via Bayesian variable selection.

When including control time series (e.g. Ukrainian Wikipedia pageviews from a Wikimedia site but not the Wikipedia.org Portal), we used static regression (the coefficients were assumed to not vary over time). Specifically, we set \(Z_t = \beta^T X_t\) and \(\alpha_t = 1\), so that each \(p\)-th covariate had a coefficient \(\beta_p\). All models included an indicator variable \(x_{\text{deployment},t}\) of the Wikipedia.org Portal secondary link collapse patch deployment as a covariate, meaning that \(x_{\text{deployment},t} = 0\) when \(t\) is before 16 August 2016, and \(x_{\text{deployment},t} = 1\) when \(t\) is 16 August 2016 or after, so that \(\beta_{\text{deployment}}\) is the effect of the deployment on Ukrainian Wikipedia main page pageviews from the Wikipedia.org Portal after accounting for variance in pageviews using various control time series.

Figure 3: Plots of the 28 BSTS models’ predictions (in black and gray) with actual pageviews (blue circles) for comparison. (Full resolution version)

Twenty-eight different models (but only twelve dealt with Russian-but-not-Ukrainian-speaking populations) were considered (see Figure 3) and nine were chosen according to the models’ R² (proportion by which the residual variance is less than the variance of the original observations).

Results

Table 1: Top 20 most similar pageviews to Ukrainian Wikipedia main page pageviews from the Wikipedia.org Portal. These are the time series used as controls in models that use “matched markets”.

In Tables 2 and 3 above, we list several well-fitting models (high R²) and show the estimated effect of the deployment. Also included are the 95% credible intervals, which can be interpreted as “there is a 95% probability that the true value of the effect is inside this interval”. Whether looking at models of Ukranian Wikipedia Main Page pageviews from Russian-but-not-Ukrainian-speaking visitors to the Wikipedia.org Portal or the other models, the effect of the deployment was estimated to be positive in some models and negative in others, and not statistically significant (95% credible interval included 0) in all of them, meaning that we do not have evidence to say that the deployment had a reliably positive or a negative impact.

References