Talk:Wikipedia.org Portal Improvements

The portal improvements page has been moved to Mediawiki.org to be more consistent in how we present the teams within the department (like maps, search, WQDS, etc.)

Deployment of updates (was "Changes won't show up on the site as quickly.")

I just want to comment on this section of the page on deploying of the updates to the portal. We've (various members of RelEng and Discovery and other interested parties like Opsen and non-WMFers or WMFers-acting-as-a-volunteers) been talking about this in IRC (in -releng and -operations, depending on where the conversation started), on phab, and in the office for the past couple of weeks.

One option not in this plan explicitly is simply treating the portal(s) as part of the "weekly train" (see: wikitech:Deployments and wikitech:Deployments/One_week). That means, in short, the portal(s) would update to whatever is in HEAD in master (with or without an automatic build process, which is supported by scap) on a weekly basis. We can decide on which group/day for it to go out with/on (Tues, Wed, or Thurs). Of course, as needed, updates could be done in any scheduled deploy window by someone with deploy rights, or if someone who has commit rights in the portal(s) repo but not deployment rights, it could go out during a SWAT. - Greg (WMF) (talk) 01:37, 6 November 2015 (UTC)

Right. For now we decided it would be simpler/better to not ride the weekly train, and instead we will simply have the portal deployed on demand, as needed. --KSmith (WMF) (talk) 00:10, 10 November 2015 (UTC)

The current implementation of www.wikipedia.org reads from a wiki page (www.wikipedia.org template) and outputs the contents at essentially www.wikipedia.org/index.html. Can you please explain why a similar setup with Git is seemingly so daunting to re-create? We already have this functionality and the major difference that you and your team want to introduce here, as I understand it, is changing the source from meta.wikimedia.org to gerrit.wikimedia.org. Why does this affect "deployments"? --MZMcBride (talk) 01:02, 10 November 2015 (UTC)

I'm also curious about this. There would be two types of updates coming to the portals on a regular basis: any changes resulting from the Discovery team's development work, plus routine updates to wiki listings and article counts. It might not be problematic that folks have to wait a full week for routine updates to go live, but I'm still unclear on why that wait would be technically necessary when the wiki main pages update at least once a day. – Minh Nguyễn ^💬 01:24, 10 November 2015 (UTC)

Meta pages as a sandbox

It would require only a few lines of code in a shell script to keep a Meta page such as www.wikipedia.org template/temp synchronized with the actual live page. So if desired, we can keep such a page open to edits by community members, so that only the maintainers have to learn how to use git and gerrit. – Minh Nguyễn ^💬 01:27, 10 November 2015 (UTC)

To this effect, I've got a quick and dirty workflow for keeping the Git repository up-to-date with Meta in gerrit:252910. It only updates a checkout's contents; manual effort is still required to start a code review, merge, and deploy. Unfortunately, this workflow can't be used on the wikipedia.org portal, which requires manual editing every time something changes due to the many changes the Discovery team has already made to that page. (One step forward, three steps back. Hopefully we'll get that reversed in short order.) – Minh Nguyễn ^💬 11:25, 13 November 2015 (UTC)

Schedule, and returning the portal to Meta

If I'm understanding this correctly, the main portal was moved to gerrit so that these tests and changes could be done more efficiently/easily by the development team. If that's the case, how long is this expected to take, approximately? Can we reasonably assume that there's a plan to properly move the portal back to the community-held space over here once this current effort is finished? --Yair rand (talk) 20:43, 15 February 2016 (UTC)

You are partially correct Yair rand. The portal was moved to gerrit for many reasons, none of which are temporary or merely to make WMF developers lives easier. Sorry for any miscommunication that insinuated otherwise. The largest reason was to bring it inline with our practices for the other wikimedia project portals and make improvements to the usability and infrastructure around the page.

The decision to move the portal to a code management system was made in October of last year with the volunteer maintainers involved in the decision making process. Particularly user Mxn who has been, and continues to thankfully be, helpful in the maintenance of the portal. The investigation into moving to gerrit can be found as a public phabricator task.

I understand change can be scary, and concerns over control are a key concern of the movement. The process is still open, suggestions continue to be made from community members, and work continues to improve the quality of the portal. CKoerner (WMF) (talk) 21:29, 15 February 2016 (UTC)

@Ckoerner: That's quite different from the explanation given on the page, but I see that it does say it's just a draft...

So, what is the plan for this? Is the portal going to be locked away indefinitely? Is there some system in the works to have automatic two-way synchronization between here and gerrit? We can't reasonably count on all future editors being able to use git, and it wouldn't fix the more serious issues even if they did. As a general rule, there aren't any developers. Mxn may be a git-capable developer now, but I have suspicion that wasn't the case when he first starting editing these portal pages eleven years ago, and who knows whether any of us will still be active eleven years from now, and whether the next potential maintainer will ever even come to start maintaining it if it's uneditable without git for some reason. (Okay, git didn't exist back in January 2005, and I have no idea whether Mxn would have known how to use it if it had existed, but still.) Temporary lockdown isn't a disaster, but long-term, the portal needs regular editing, and that's certainly not a job that can be left only to git-using developers, even if it wasn't an offence to basic wiki ethos. --Yair rand (talk) 22:45, 15 February 2016 (UTC)

"Locked away". Oh, you. I'm dashing out, but wanted to let you know I saw your reply. CKoerner (WMF) (talk) 23:13, 15 February 2016 (UTC)

@Ckoerner:: This statement is not correct: «The largest reason was to bring it inline with our practices for the other wikimedia project portals». The other portals were moved under (what I understand to be a) protest when the promoters of the change were requested to keep all portals in synch and under the same maintenance mode to avoid complicating the live of the collaborators that work (used to work?) on them. The concerns related to this change have nothing to do with fear. Things like changing the globe based on location have been requested before. The people who participated on the original conversation was supportive of the change on the expectation that it would allow fully automated maintenance (it did not happen) and on the condition that the existing functionality would be matched (it didn't happen either). And not just that, some contributors also wasted valuable time updating the pages in Meta because this was not communicated properly. --Lsanabria (talk) 01:06, 20 February 2016 (UTC)

There isn't any real benefit in abandoning the usual wiki system. It's also unbelievable that some WMF persons seized a Wikimedia property without even opening a discussion in the global community. I want to believe that you're right, Yair rand, and this was only a brief test. The test has proven unsuccessful, by the way: the portals suffered a quality regression because WMF is unable to keep them up to date. Nemo 07:43, 16 February 2016 (UTC)

Hey Nemo. "The test has proven unsuccessful" is not the case; the tests were formal A/B tests and have, actually, shown the changes are a substantial improvement! On quality, the link you provide shows that the page was updated so I'm not sure what the regression is here.

I'll leave answering the wider questions to Chris and Deb. Ironholds (talk) 08:17, 16 February 2016 (UTC)

As your own link shows: « I see no update to the stats ever since it was moved to Gerrit.». The portal was moved to Gerrit on 1/26, the complain about the lack of update was filled on 2/2 and resolved on 2/8 (if I can read Phab correctly). That is 13 days without updates. That fits my definition of a regression. --Lsanabria (talk) 17:15, 19 February 2016 (UTC)

I've requested that the Phab ticket to be worked on during the week of Feb 22, 2016 to update the stats on the Portal page. This does bring up a good question though - how often should the stats on the Portal page be updated manually, until we can get it automated? --deb (talk) 22:47, 19 February 2016 (UTC)

To quote mxn's comment on this very page on 11/10/2015:

It might not be problematic that folks have to wait a full week for routine updates to go live, but I'm still unclear on why that wait would be technically necessary when the wiki main pages update at least once a day.

I am not the one that can say if weekly updates are fine or not, but personally, I think they are the minimum. --Lsanabria (talk) 23:25, 19 February 2016 (UTC)

I noticed the other portals have not been updated either. I added a note to Phabricator. --Lsanabria (talk) 01:13, 20 February 2016 (UTC)

• To clarify, the WMF has moved www.wikipedia.org as this is where the new Knowledge Engine is to live. They from what I understand wished to have greater control as this is a much more tech heavy project. And they want to be able to carry out testing of the new things they are building. As mentioned this page already gets 200 to 300 M pageviews a month so it is a nice start for something new.[1]
• User:CKoerner (WMF) this sort of language is not helpful "I understand change can be scary". What the community is pissed off about is the secrecy surrounding what has been occurring since April 2015, it is not change. The community is actually requesting a lot of stuff be changed.[2] We know that some at the WMF see the community as "change averse" and that this is likely part of the reason for the degree of secrecy. What we are requesting is collaborative change rather than the WMF going off and taking 10s of millions of movement funds for a "moon shot" created by a couple of people behind closed doors. Doc James (talk · contribs · email) 03:37, 19 February 2016 (UTC)

Hi User:Doc James: The page that you're posting these comments to is our Portal Improvements page, where we list the things we want to try, the things we've improved and the overall general direction we'd like to go with the Portal page so that everyone in the community benefits. The Knowledge Engine is not a real thing and the Portal page is just the Portal page - one of many entrances to a world of knowledge and sharing. Yes, I agree that many (WMF and Community) are probably change-adverse, that's why we have this Portal Improvements page - to publicly talk about the Portal and the improvements we want to do with the community's full knowledge. We have no secrecy on this page or within our work on the Portal. --deb (talk) 21:55, 19 February 2016 (UTC)

Sure "the Knowledge Engine is not a real thing" yet. It is a future plans at this point. I hope you have the "improvement page" to help learn about all the improvements / changes that are wanted. Not sure what you mean by "we have no secrecy on this page or within our work on the Portal"? If this is where the Knowledge engine is going there is indeed and continues to be secrecy surrounding it. Doc James (talk · contribs · email) 00:01, 20 February 2016 (UTC)

Doc James, sometimes folks come to talk pages about WMF efforts with language that indicates to me that they are frustrated. It was a earnest attempt to diffuse any strong emotions. Hard on the issues and soft on the person as it were. We haven’t met (well briefly at your Wiki Project Med Meetup at Wikimania this past summer) but I hope you agree that I don’t know Yair and can’t assume they’ve been aware of every discussion and decisions that’s been made. I was just trying to explain how we got here and why.

If you’ll humor me I have a few clarifying points I’d like to address from your comment.

We are not calling any of our work Knowledge Engine. Continuing to conflate the current work of the Discovery team with a term used in the grant does not further the conversion. We’re not building a “knowledge engine”, a general search engine (like Google) for the entire Internet, nor any robots to replace editors. We’re making search better on-wiki and on the portals.

I love the community wishlist - I personally hope that gets expanded in the future. Creating simple to digest checklists can be very helpful in prioritizing what to work on.

Closed doors, yuck. As others (Brion) myself (Liam’s blog post) have said, the early discussion came from a single individual within the WMF, who left shortly after, and leadership carried on - at a much reduced scale. Lila admitted herself that this was a mistake to not bring forward earlier.

The secrecy is crap, I agree. Some staff see members of the community as aggressive and unkind on top of the secrecy. I think both, and leadership issues within the foundation are contributing to folks not feeling comfortable sharing information. I’m still new, and filled with optimism.

The budget for Discovery, people, technology, etc. is not in the 10s of millions as far as I know. We’re smaller than the Editing and Reading teams (insert objection to their staff size here). As a long time volunteer, perhaps you have more insight that I have, so I apologize in advance if I have my numbers wrong.

The grant was for 250,000 a year for up to 6 years. The foundation is only pursuing the first year at the moment, and will engage with Knight Foundation in I believe August for review. (see correction below) I got a feeling that we might not go any farther than this initial year given the headache this is all causing for everyone.

Again, no “moon shot”. Not even a “hole-in-one”! What we are doing is a series of iterative improvements to search, done in public, with community engagement (like this conversation - seriously!).

Here’s what I can do. I’ve reached out to the staff in charge of the portal project. I’ve asked them what are plans our for making sure the community has a clear and easy (non gerrit/Phabricator) way to submit updates to the portal. In the past it was a talk page on Meta if I recall. I personally think that should not change, but I don't get to make that call. That’s a first step to address some of the concerns I'm hearing. CKoerner (WMF) (talk) 22:07, 19 February 2016 (UTC)

Sure we can call it the "the project formally known as the Knowledge Engine". Except from what I understand it is no longer to be called a "project" either.

We were proposing to build a general search engine and were once worried about Google competing with us. I am not sure what we are doing now. But I have seen a presentation loss the words "search" before being presented to me and afterwards seeing those words reappeared in a later version. So apologies if I am a little skeptical.

With respect to "bots to replace editors" or to put it in other words, creating articles on the fly from Wikidata, I think it is an interesting idea and a useful discussion to have. We have already created hundreds if not millions of articles by bot already. That we are still "officially" denying this idea exists maintains my concerns.

That Lila did not bring this forwards sooner was not a simple oversight. And no we are not "blaming" all this on Damion. Lila's name is on the bottom of the grant agreement which was submitted after Damion left. Yes the idea may have originally been his but Lila took it over and continued on with it with greater secrecy if anything.

This idea was created by the WMF. I would imagine that you have not seen the recent official cost estimates? The grant was only ever to pay for a small amount of it. We asked for 6M and were given 0.25M from Knight. The rest of the 32M over 6 years was to come from general funds / other grants. P.S. "moon shot" was a term repeatedly used to describe this project by the way. And of course the term does not describe "a series of iterative improvements" Doc James (talk · contribs · email) 13:06, 20 February 2016 (UTC)

I'm somewhat confused as to why this is the subject of discussion on the portal improvement page. Ironholds (talk) 04:48, 21 February 2016 (UTC)

It was pointed out to me that I misread the grant. It is only for $250,000. I misspoke about a presumed extension. I apologize for the confusion. CKoerner (WMF) (talk) 22:04, 22 February 2016 (UTC)

Problems

Collapsing non-top-10-language Wiki links is of course unacceptable. The trending articles feature lets you add pretty pictures but is asking for trouble given some of the things that have trended over the years.Geni (talk) 08:06, 16 February 2016 (UTC)

Can you explain beyond "unacceptable"? It's worth noting that we're experimenting with language detection work to avoid obscuring the less-prominent languages for speakers of those languages; agreed that simply hiding all but the top 10 would introduce a substantial bias. And, yeah, that was my worry with trending content too. Ironholds (talk) 08:19, 16 February 2016 (UTC)

If we wish to keep the claim that we care about the global south somewhat credible then hiding their languages is not a good start. The eternal fight for 10th place is also not going to be worth the hassle.Geni (talk) 16:22, 18 February 2016 (UTC)

Hi Geni: We do want to keep the Global South credible on the Portal. We want to conduct a test to be sure we can accurately determine the visitor's language preferences when they visit the Portal and display the page in that language. That way, if the visitor's preferred language is not one of the top 10, we'll adjust the page as we can to ensure the user can view it in their preferred language. We have no intention of minimizing any language that isn't in the top 10. On a side note, the trending articles idea is just that at this point - an idea of what we might be able to do in the future. But it'll need a lot of testing and vetting with the community before it ever goes live on the Portal. --deb (talk) 21:54, 19 February 2016 (UTC)