Research talk:Task recommendations/Work log

Work log

Archive

Thursday, July 17th -- Simulating recommendations for qualitative analysis. Thursday, July 24th -- Gathering simulated recommendations Friday, July 25th -- Subsampling from recommendation lists for hand-coding Friday, August 1st -- Subsampling from recommendation lists for hand-coding Monday, September 22nd -- Gathering cohorts Wednesday, September 24th -- Productivity stats Monday, September 29th -- Funnel flow datasets Tuesday, September 30th -- User pages edited stats dataset Thursday, October 2nd -- Funnel flow stats

Thursday, October 2nd

Given the low number of observations, I'm using a en:beta distribution to approximate the error around proportion measurements. This results in some unusual-looking error bars where the observed proportion can appear outside of them. This is actually more honest to the underlying uncertainty though, so I'll risk the confusion.

At what rate do users see recommendations?

Impressed proportion. The proportion of experimental users who saw at least one recommendation is blotted with error bars based on the central mass of a Beta.

Here, we can see a clear difference between the flyout-only condition and the conditions that contain post-edit. There's even a significant difference between the rate at which users were impressed in the both vs. post-edit condition for large wikis. This is likely due to users clicking on the "Recommendations" link before they have edited an article.

At what rate do users accept recommendations?

Accepting proportion. The proportion of experimental users who accepted a recommendation (by clicking on it) is plotted with error bars based on the central mass of a Beta.

Here, we see roughly the same trends that we saw with impressions. This is probably because seeing a recommendation is required for accepting it. Let's look at the conditional proportion next.

At what rate do users who see recommendations accept (at least one) recommendation?

Accepting proportion of impressed users. The proportion of experimental users who both saw a recommendation and accepted one (by clicking on it) is plotted with error bars based on the central mass of a Beta.

Here, we can see "flyout" shining in the way that we suspected. For large wikis, the flyout condition see a substantially larger proportion of impressed users clicking (accepting) at least one recommendation. There's two potential explanations that I can think of:

1. (Ready for recommendation) When users ask for recommendations, they are in a better position to accept them.
2. (Propensity) Users who notice the "recommendations" link on the personal bar are simply more likely to accept a recommendations.

At what rate do users edit recommended articles?

Editing proportion. The proportion of experimental users who edited an accepted article recommendation is plotted with error bars based on the central mass of a Beta.

Like we saw when looking at the raw impression rate and acceptance rate, we see post-edit conditions resulting in a higher rate of edited recommended articles. I don't think this is telling us much.

At what rate do users who accepted recommendations edit at least one recommendation?

Editing proportion of accepting users. The proportion of experimental users who both accept a recommendation (by clicking on it) and edited the recommended article is plotted with error bars based on the central mass of a Beta.

Here, we're looking at conditional probabilities again. It looks like editors were not significantly more likely to edit an article recommendation that they accepted the flyout condition. This runs counter to my hypothesis -- that users would actually have better recommendations and would therefore be more likely to follow-through with an edit in the flyout condition.

End

That's all for today folks. --Halfak (WMF) (talk) 22:24, 2 October 2014 (UTC)