Research talk:Teahouse long term new editor retention/Work log/2015-10-20
Tuesday, October 20, 2015
editToday I'm extending my analysis to include multivariate regressions. I'll be using a en:logistic regression to look for differences in the proportion of surviving newcomers. I'll be including the pre-invite statistics I worked on last time to control for random effects around the effect of invitation. I expect the effect invitation to become more prominent after controlling for these random effects. I also expect to see some interactions between either initial investment (# of edits pre-invite)/negative feedback and the invitation condition. A positive relationship would suggest that newcomers who are highly invested and get negative feedback gain more "survivalness" from the invite.
Checking for bucketing bias
editThe following list of wilcox and Chi^2 tests check for significant differences between the pre-invite predictors between conditions. Scalars are noted by quantiles (0%, 25%, 50%, 75% and 100%). Logicals by their proportion.
- edits control=5-6-7-11-215 invited=5-6-7-12-241 W=18158354.5 p=0.597
- main_edits control=0-4-6-9-212 invited=0-4-6-9-241 W=18327891 p=0.181
- talk_edits control=0-0-0-0-35 invited=0-0-0-0-37 W=17914856.5 p=0.188
- user_edits control=0-0-0-1-74 invited=0-0-0-1-232 W=17823485.5 p=0.163
- user_talk_edits control=0-0-0-0-24 invited=0-0-0-0-90 W=17958249 p=0.444
- wp_edits control=0-0-0-0-19 invited=0-0-0-0-79 W=18091557 p=0.581
- other_edits control=0-0-0-0-96 invited=0-0-0-0-125 W=18104004.5 p=0.552
- vandal_warning control=0.121 invited=0.123 X-squared=0.153 p=0.696
- spam_warning control=0.027 invited=0.026 X-squared=0.081 p=0.776
- copyright_warning control=0.003 invited=0.003 X-squared=0.304 p=0.582
- general_warning control=0.221 invited=0.214 X-squared=0.57 p=0.45
- block control=0.001 invited=0.002 X-squared=0.729 p=0.393
- welcome control=0 invited=0 X-squared=0 p=1
- csd control=0.035 invited=0.031 X-squared=1.234 p=0.267
- deletion control=0.051 invited=0.047 X-squared=0.696 p=0.404
- afc control=0 invited=0 X-squared=NaN p=NaN
- teahouse control=0 invited=0 X-squared=NaN p=NaN
No significant differences here.
Predicting 1+ edits
editNow to build some logistic models that account for these pre-invite predictors.
- 3 to 4 weeks
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -3.40157 0.28915 -11.764 < 2e-16 ***
grpinvited -0.45260 0.31556 -1.434 0.151491
log(edits + 1) 0.36207 0.12951 2.796 0.005181 **
log(main_edits + 1) 0.09796 0.04883 2.006 0.044832 *
log(talk_edits + 1) 0.19515 0.07423 2.629 0.008563 **
log(user_edits + 1) 0.02817 0.04907 0.574 0.565918
log(user_talk_edits + 1) 0.22418 0.06657 3.368 0.000758 ***
log(wp_edits + 1) 0.13143 0.09053 1.452 0.146561
general_warningTRUE -0.52963 0.19096 -2.773 0.005547 **
csdTRUE -1.52286 0.72156 -2.111 0.034813 *
deletionTRUE -0.41958 0.37541 -1.118 0.263717
grpinvited:log(edits + 1) 0.23479 0.12669 1.853 0.063855 .
grpinvited:general_warningTRUE 0.04394 0.21149 0.208 0.835434
grpinvited:csdTRUE 1.09061 0.75507 1.444 0.148633
grpinvited:deletionTRUE -0.21884 0.42406 -0.516 0.605816
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 8869.9 on 14765 degrees of freedom
Residual deviance: 8564.5 on 14751 degrees of freedom
AIC: 8594.5
Number of Fisher Scoring iterations: 6
First, the obvious effects. We see the usual suspects here. The more edits you do -- overall, but especially talking -- the more likely you are to be retained. We also see some substantially negative effects of warning messages and CSD notifications.
We see a negative effect of invitation here, but it looks like the combined effect of grpinvited:log(edits + 1) counteracts that for editors who saved (log(x+1)=2, x=6) edits or more when the invite was posted. For any editor who saved more than 6 edits (highly motivated), it looks like the invite might be substantially improving retention at scale with how much editing they are doing. But the effect remains insignificant (marginal @ 0.064).
Counter to my suspicions, I don't think we're seeing solid evidence of an interaction between being invited to the teahouse and surviving despite negative feedback (csd & warning). It could be that this is due to too low of observations.
Just for the sake of making sure that my previous analysis wasn't totally off, let's try the model with just the invite as a predictor.
- 3 to 4 weeks (single predictor)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.44393 0.06633 -36.843 <2e-16 ***
grpinvited 0.14830 0.07369 2.012 0.0442 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 8869.9 on 14765 degrees of freedom
Residual deviance: 8865.7 on 14764 degrees of freedom
AIC: 8869.7
Number of Fisher Scoring iterations: 5
Sure enough. Getting the invite seems to look significant on its own. OK! Now to try the long-term retention outcomes.
- 1 to 2 months
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -3.38622 0.24595 -13.768 < 2e-16 ***
grpinvited 0.06769 0.27003 0.251 0.802051
log(edits + 1) 0.50057 0.11032 4.537 5.7e-06 ***
log(main_edits + 1) 0.14552 0.04364 3.335 0.000854 ***
log(talk_edits + 1) 0.17592 0.06710 2.622 0.008744 **
log(user_edits + 1) 0.05333 0.04358 1.224 0.220999
log(user_talk_edits + 1) 0.11024 0.06179 1.784 0.074370 .
log(wp_edits + 1) 0.07868 0.08374 0.940 0.347399
general_warningTRUE -0.50829 0.15991 -3.179 0.001480 **
csdTRUE -0.95765 0.46844 -2.044 0.040918 *
deletionTRUE -0.76776 0.35457 -2.165 0.030360 *
grpinvited:log(edits + 1) 0.01426 0.10815 0.132 0.895065
grpinvited:general_warningTRUE -0.10018 0.17865 -0.561 0.574961
grpinvited:csdTRUE 0.57811 0.50508 1.145 0.252384
grpinvited:deletionTRUE 0.23804 0.39010 0.610 0.541725
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 11229 on 14765 degrees of freedom
Residual deviance: 10845 on 14751 degrees of freedom
AIC: 10875
Number of Fisher Scoring iterations: 5
- 2 to 6 months
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -3.19216 0.23718 -13.459 < 2e-16 ***
grpinvited 0.08069 0.26070 0.310 0.75694
log(edits + 1) 0.43819 0.10755 4.074 4.61e-05 ***
log(main_edits + 1) 0.18349 0.04316 4.252 2.12e-05 ***
log(talk_edits + 1) 0.21654 0.06491 3.336 0.00085 ***
log(user_edits + 1) 0.03663 0.04303 0.851 0.39460
log(user_talk_edits + 1) 0.05446 0.06177 0.882 0.37797
log(wp_edits + 1) 0.11562 0.08142 1.420 0.15563
general_warningTRUE -0.53228 0.15243 -3.492 0.00048 ***
csdTRUE -0.55369 0.37929 -1.460 0.14435
deletionTRUE -0.69683 0.32396 -2.151 0.03148 *
grpinvited:log(edits + 1) 0.01059 0.10513 0.101 0.91977
grpinvited:general_warningTRUE -0.10878 0.17073 -0.637 0.52403
grpinvited:csdTRUE 0.04731 0.42470 0.111 0.91130
grpinvited:deletionTRUE 0.14502 0.36032 0.402 0.68732
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 11889 on 14765 degrees of freedom
Residual deviance: 11496 on 14751 degrees of freedom
AIC: 11526
Number of Fisher Scoring iterations: 5
Similar story here, but it doesn't seem like the effect of the invite isn't even marginally significant. Onto the 5+ measures.
Predicting 5+ edits
editSame story as above except survival only counts when there's 5+ edits in the survival period.
- 3 to 4 weeks
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -4.65842 0.39320 -11.847 < 2e-16 ***
grpinvited -0.61702 0.42874 -1.439 0.15011
log(edits + 1) 0.44489 0.17306 2.571 0.01015 *
log(main_edits + 1) 0.20667 0.06855 3.015 0.00257 **
log(talk_edits + 1) 0.16150 0.10187 1.585 0.11289
log(user_edits + 1) 0.17094 0.06630 2.578 0.00993 **
log(user_talk_edits + 1) 0.21420 0.08913 2.403 0.01625 *
log(wp_edits + 1) 0.19019 0.11490 1.655 0.09787 .
general_warningTRUE -0.78168 0.30218 -2.587 0.00969 **
csdTRUE -1.42153 1.01750 -1.397 0.16239
deletionTRUE -0.29195 0.52487 -0.556 0.57806
grpinvited:log(edits + 1) 0.26699 0.16513 1.617 0.10591
grpinvited:general_warningTRUE 0.22127 0.33150 0.667 0.50447
grpinvited:csdTRUE 1.26795 1.05635 1.200 0.23001
grpinvited:deletionTRUE -0.48490 0.60548 -0.801 0.42322
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 5101.0 on 14765 degrees of freedom
Residual deviance: 4814.7 on 14751 degrees of freedom
AIC: 4844.7
Number of Fisher Scoring iterations: 7
Again, we see a lack of significant independent effect for the invitation. Again, we also see the marginially significant interaction with log(edits + 1) suggesting that the invitation might be more effective for newcomers who save a lot of edits before getting the invitation.
Onto the long-term outcomes:
- 1 to 2 months
Regression with multicollinearity problem
|
|---|
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -4.07573 0.32035 -12.723 < 2e-16 ***
grpinvited -0.40777 0.34891 -1.169 0.24253
log(edits + 1) 0.39092 0.14264 2.741 0.00613 **
log(main_edits + 1) 0.24877 0.05703 4.362 1.29e-05 ***
log(talk_edits + 1) 0.20831 0.08312 2.506 0.01221 *
log(user_edits + 1) 0.16667 0.05522 3.018 0.00254 **
log(user_talk_edits + 1) 0.18157 0.07557 2.403 0.01627 *
log(wp_edits + 1) 0.15650 0.09935 1.575 0.11521
general_warningTRUE -0.55499 0.22216 -2.498 0.01249 *
csdTRUE -12.75334 135.48352 -0.094 0.92500
deletionTRUE -1.16388 0.59382 -1.960 0.05000 *
grpinvited:log(edits + 1) 0.21278 0.13657 1.558 0.11923
grpinvited:general_warningTRUE -0.07856 0.24671 -0.318 0.75015
grpinvited:csdTRUE 12.36522 135.48374 0.091 0.92728
grpinvited:deletionTRUE 0.43928 0.63691 0.690 0.49038
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 7482.1 on 14765 degrees of freedom
Residual deviance: 7085.4 on 14751 degrees of freedom
AIC: 7115.4
Number of Fisher Scoring iterations: 14
|
Yikes! here, we're seeing too much correlation between getting a 'csd' message and being invited. Going to need to drop the predictor.
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -4.07257 0.31851 -12.786 < 2e-16 ***
grpinvited -0.41388 0.34713 -1.192 0.23315
log(edits + 1) 0.39039 0.14205 2.748 0.00599 **
log(main_edits + 1) 0.24097 0.05671 4.249 2.14e-05 ***
log(talk_edits + 1) 0.18318 0.08288 2.210 0.02710 *
log(user_edits + 1) 0.15665 0.05482 2.858 0.00427 **
log(user_talk_edits + 1) 0.17481 0.07540 2.318 0.02043 *
log(wp_edits + 1) 0.15835 0.09922 1.596 0.11051
general_warningTRUE -0.62899 0.22163 -2.838 0.00454 **
deletionTRUE -1.28957 0.59291 -2.175 0.02963 *
grpinvited:log(edits + 1) 0.22144 0.13593 1.629 0.10329
grpinvited:general_warningTRUE -0.02041 0.24601 -0.083 0.93387
grpinvited:deletionTRUE 0.52833 0.63556 0.831 0.40581
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 7482.1 on 14765 degrees of freedom
Residual deviance: 7100.9 on 14753 degrees of freedom
AIC: 7126.9
Number of Fisher Scoring iterations: 6
- 2 to 6 months
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -4.313717 0.293284 -14.708 < 2e-16 ***
grpinvited 0.363524 0.320604 1.134 0.256848
log(edits + 1) 0.672387 0.128262 5.242 1.59e-07 ***
log(main_edits + 1) 0.160990 0.051039 3.154 0.001609 **
log(talk_edits + 1) 0.249009 0.074702 3.333 0.000858 ***
log(user_edits + 1) -0.002232 0.051036 -0.044 0.965120
log(user_talk_edits + 1) 0.047366 0.073726 0.642 0.520577
log(wp_edits + 1) 0.166464 0.092988 1.790 0.073429 .
general_warningTRUE -0.731064 0.209830 -3.484 0.000494 ***
deletionTRUE -0.977154 0.467432 -2.090 0.036575 *
grpinvited:log(edits + 1) -0.089354 0.124897 -0.715 0.474347
grpinvited:general_warningTRUE -0.033336 0.232662 -0.143 0.886070
grpinvited:deletionTRUE 0.286314 0.510743 0.561 0.575081
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 8502.0 on 14765 degrees of freedom
Residual deviance: 8129.1 on 14753 degrees of freedom
AIC: 8155.1
Number of Fisher Scoring iterations: 6
Well, the direction and scale of the coefs don't change. We don't see independent significance in the effect of the invitation or it's interaction with previous activity.
Again, just to check my sanity, let's try the 2 to 6 month regression with the bucket as the single predictor.
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.52590 0.06867 -36.78 <2e-16 ***
grpinvited 0.16681 0.07617 2.19 0.0285 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 8502 on 14765 degrees of freedom
Residual deviance: 8497 on 14764 degrees of freedom
AIC: 8501
Number of Fisher Scoring iterations: 5
Sure enough, there's the significant effect I saw in the simple Chi^2 test. --Halfak (WMF) (talk) 18:54, 20 October 2015 (UTC)