Research:MoodBar/UI design

MoodBar	Pilot: Early data (July 2011 - September 2011)	Stage 1: Usage and UX (September 2011 - May 2012)	Stage 2: Impact on editor engagement (May 2012 - September 2012)
MoodBar Feedback Dashboard Mark as Helpful	First month of activity	Time to first feedback Response time Effects of UI changes Email confirmation	Observational study Controlled experiment design Controlled experiment results Response team call to action

This report discusses how design decisions regarding the UI of MoodBar affect the rate of adoptions of the extension by Wikipedia editors. We discuss findings from other reports and propose actions aimed at increasing the number of newly-registered editors reporting feedback about editing (and the Wikipedia UX in general).

We first review the finding from previous reports and then discuss a certain number of proposals.

Summary of Findings

1. The most effective invitation link text is “Editing Wikipedia made me…”
2. The saliency of MoodBar is temporary

We discuss each in a separate section.

The most effective invitation link text is “Editing Wikipedia made me…”

Figure 1:
Weighted number of feedback by bucket category

The most effective invitation link text was tested by means of A/B testing. Three buckets with the following messages were set up, with the following weights:

• Feedback about editing (weight: 80%, bucket: feedback)
• Editing Wikipedia made me...(weight 10%, bucket: editing)
• Share your experience (weight 10%, bucket: share)

The percentages within parentheses refer to the weight of each bucket. The A/B test with those weights ran between July 25, 2011, and November 30, 2011. It collected a total of 8960 feedbacks divided as:

bucket	messages
editing	1480
feedback	6643
share	846

Figure 1 show the total number of messages weighted according to the importance of each bucket. The editing invitation text elicited the highest number of feedbacks per user.

We can actually estimate from the above table the odds of sending feedback under two particular choices; these are just the proportions between the heights of the bars in Figure 1. The results are:

editing vs feedback	share vs feedback	editing vs share
1.785553	1.020661	1.749409

The above table should be read as follows: the first column means that the odds of sending a message given that the user saw the editing invitation text are 1.79 : 1 against those of sending a message given feedback. Thus, we expect that the number of feedbacks should increase by roughly 79% if we switched from feedback to editing.

The saliency of MoodBar is temporary

Figure 2.
A mock-up of the MoodBar notification tooltip. From the MoodBar design document.

Figure 3.
Estimated hazard rate, without tooltip notification.

Figure 4.
Estimated hazard rate, with tooltip notification

What could be the cause for this? Upon activation MoodBar notifies the user with a tooltip (see Figure 2). Is it possible that the behavior of users could be attributable to this feature? Luckily there is a way to test this question.

The tooltip feature was in fact introduced a few months after the deployment of MoodBar, in the effort of increasing the saliency of MoodBar within the general Mediawiki UI. Because the tooltip appears only once, all users who activated MoodBar before that date have never seen it, while the majority of the other users saw the tooltip only once. So, if the increased likelihood of reporting feedback is due to the increased saliency of the MoodBar link, for this earlier group we should see a flatter hazard.

We thus divided our dataset of users in two groups, depending on when they had MoodBar activated. The first group (without the tooltip) consists of users who had MoodBar activated before the tooltip notification feature was introduced (9,961 users), while the latter consists of users who had MoodBar activated after this date of 79,744. We estimated again the hazard rate using the same technique, and show them in Figs 2 and 3. A comparison of the two figures confirms our hypothesis: before the introduction of the tooltip the hazard peaks at a lower value, and takes longer to reach the baseline. Moreover, for the pre-tooltip group the baseline hazard after 50 days is higher than in the post-tooltip group.

A note about the pre-tooltip period

A flatter hazard rate means that moods are sent more uniformly in time, and this is certainly a good thing because we would like to hear from editors also at later stages of their participation into Wikipedia. So, does this mean that before the tooltip we were better off? Luckily for us this is not the case, and here's why.

As can be seen from the MoodBar data dashboard (in particular the second plot), the number of users who sent a mood before the introduction of both the icon and the tooltip is relatively much lower than after. As we can expect, the introduction of the tooltip notification – and to some extent of the icon before it—increase the saliency of the MoodBar link within the MediaWiki UI. However, this has two distinct effects on user adoption: the first is that because more people are aware of the possibility to report their mood, we receive (unsurprisingly!) more feedbacks. The second is that because a one-off notification eventually goes forgotten, the vast majority of the feedbacks we get are sent only in the first few days after the tooltip is shown.

Design Enhancement Proposals

Invitation text

The current bucket for all users is feedback. However, the present analysis shows that the odds of sending a message with the invitation text associated to the editing bucket are higher (1.79:1). We thus propose to switch the invitation text accordingly.

Overall MoodBar saliency

Users tend to report their mood only immediately after the activation of MoodBar; after the first four days since its activation, the chances of sending feedback become in fact tenuous (14x lower hazard on day 10 compared to day 1). We impute this behavior to the effect of the tooltip notification that MoodBar shows upon its activation. A comparison of the hazard rate, before and after the introduction of these features, seems to support this conclusion.

Of course this result has to be read in the general context of the usage of MoodBar. The tooltip notification increases the prominence of the MoodBar UI on the screen and, in fact, since its introduction the number of unique daily posts has increased by roughly 50%. Thus, what our analysis says is that the beneficial effects of the tooltip reminder on user adoption of MoodBar endure only for a few days. Right now only 3% of all users who have had MoodBar activated ever sent feedback. The present analysis suggests that the overall adoption of MoodBar could be improved if, instead of only once, users were reminded of the possibility to send feedbacks multiple times—for example by showing the tooltip after the initial 4 days period.