User talk:Beetstra/Archives 2021

Active discussions
Archive This is an archive of past discussions. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page.

linkwatchers ugh

The linkwatchers have been failing miserably and not particularly talking in IRC. In my inexpert way I have done some restarts and have some responsiveness back. though looking at the syslogs, I do see the troubling (replace user with xxxxx in case it needs to be quiet)

  • LINKANALYSER 000006 @ 2021-1-16 @ 21:44:30 - Dying painfully because of faulty liwa mysql handle - error Can't connect to MySQL: User 'xxxxxx' has exceeded the 'max_user_connections' resource (current value: 10) at LinkAnalyser.pl line 508.

I think that I have it back and running; I am see it generate a report or two. Maybe it was just db issues, <shrug> my problem-solving/understanding of what exactly was wrong is still substandard.  — billinghurst sDrewth 21:52, 16 January 2021 (UTC)

@Billinghurst: Hi, good morning. Yes, there is a cap to the number of connections the bot can have into the linkwatcher database. It once started to eat too many away, and that has consequences on the database. However, sometimes processes do not die cleanly and leave a connection open, upon which the bot reaches the cap. You'll have to kill everything completely, and then restart the bot. And I think that both LiWa3 and COIBot eat from the cap of 10, so one bot may give problems on the other. --Dirk Beetstra T C (en: U, T) 05:53, 17 January 2021 (UTC)
kkkk Probably had a quite a few caught as there was an absolute swag of processes, and had no idea of what was a normal/expected count. Not certain that I am aware of a graceful means to kill everything completely, so I did a number of ugly things that worked in the end. ¯\_(ツ)_/¯  — billinghurst sDrewth 11:14, 17 January 2021 (UTC)
@Billinghurst: I wouldn't know either, I think I just kept killing processes until it was silent ... then restart linkwatcher. --Dirk Beetstra T C (en: U, T) 12:53, 17 January 2021 (UTC)
Wow, that is my magic. "killall -9 perl" had a good impact after the shell was stopped.  — billinghurst sDrewth 12:56, 17 January 2021 (UTC)
@Billinghurst: you did not tell me that that also did not solve your problem. I have applied my ultimate motto: "If violence did not solve your problem, you did not use enough" ... I have restarted the whole instance, and started the script clean. --Dirk Beetstra T C (en: U, T) 18:56, 19 January 2021 (UTC)

  Comment killed all processes coibot, xlinkbot, liwa3; restarted processes. After a day and a half of the analysis of the same couple of domains, I decided that it was a boring loop. I am guessing that it was a database issue, but couldn't tell you. Diversity is back. If there was a simpler set of restarts then please let me which. Size 12 fire boots seemed best.  — billinghurst sDrewth 12:37, 27 January 2021 (UTC)

@Billinghurst: There seems to be something wrong with some forms of access, I notice a lot of join and leaves going on. Maybe a reset of the instances is again needed (strange, just did that ...). --Dirk Beetstra T C (en: U, T) 06:11, 31 January 2021 (UTC)

LiWa3 loop <= kicked; and new WPs

Seems that we had the linkwatchers cycling the same reports, so I have stopped and restarted coibot and liwa3. Got a big list of backlogged reports. Went to add a linkanalyser and COIBOT told me to FRO. Thought that it might processing the backlog, but not at this point. Will look at it again on my tomorrow, though probably cannot do a lot of kicking for 18-20 hours. We seem to be out of the loop anyway.

Do we need to add new WPs to the settings? I can see that the linkwatchers are in the feeds of new WPs like skr.wikipedia so I have no idea what adding them to the settings actually does. Thanks.  — billinghurst sDrewth 13:35, 17 February 2021 (UTC)

@Billinghurst: I don't know what causes that first issue. Seems to be happening over and over. Last time I tried to kill most processes on linkwatcher, it looked like there were 2 or 3 instances running at the same time (but .. why?).
Linkwatcher, on boot, reads a special page (I think the api version of Special:SiteMatrix), and from that creates a list of all wikis (the regular ones, it ignores the 'special' ones in the sitematrix, many of which are not accessible anyway). It adds to that list the ones in the variable special in the settings. Then it takes some aside (because of volume; set in owndiffreader - these wikis get their individual diffreader, currently en.wikipedia and wikidata), and divides the rest over 10 (? number seems to be hardcoded) DiffReaders (it saves files, 'channellist#' which are 'feed' lists for each the DiffReaders). Then the DiffReaders start (1 for en.wiki, one for wikidata, and the 10 for the 'rest'). Except for the ownwiki-ones, all have about 85 wikis to read (most of them rather silent). So any that are not in the sitematrix need to be put into special and in principle need a restart of the whole bot to read those variables at start (but you can manually add one to one of the channellist-files in the , and then reboot only that diffreader that reads that channellist ...). If one DiffReader keeps reconnecting it likely means that one wiki is too high volume (which is then likely es.wikipedia or de.wikipedia??) and that one then probably should move into ownwiki followed by a restart of the bot. --Dirk Beetstra T C (en: U, T) 11:07, 18 February 2021 (UTC)

Seems COIBot account is not AUTHing for freenode

Seeing both COIBot and Xlinkbot come and go in IRC. Tried lots of things including a reboot, to no success. Zppix believes that the account is not AUTHing, and I have no idea how to resolve that aspect, nor even how to problem solve or check that is the case, so I am handing that bit back to you. Happy to be further educated on what I should be doing. Regards arms and legs  — billinghurst sDrewth 23:53, 26 February 2021 (UTC)

Something worked automagically at some point. No idea why, and after liwa3s had disappeared. All back and functioning.  — billinghurst sDrewth 04:34, 27 February 2021 (UTC)

Sigh, bots have issues. Needs a break? COVID-19? Maybemaybe I have a bit of time end of this week to keep an eye on it and see if I can figure out what the issue is. --Dirk Beetstra T C (en: U, T) 05:40, 28 February 2021 (UTC)

bd808 asks ...

Whilst I was seeking some help today BDavis asked two things re COIBOT. I can do the second, but not the first due to access.

<bd808> it would probably also be a good idea to clean up some/all of tools.coibot and add a README there saying that it is all on another Cloud VPS project now

and I went to do it and found that I don't have the access to the tool's group on toolforge to do it.

<bd808> billinghurst: write some system admin procedure docs for this project pretty please. :)
...
<bd808> billinghurst: the most basic info would be something like https://wikitech.wikimedia.org/wiki/Tool:Stashbot#Maintenance

which I will do.  — billinghurst sDrewth 11:32, 27 February 2021 (UTC)

@Billinghurst: you should get access to tools.coibot ... if only I knew how to give you that. --Dirk Beetstra T C (en: U, T) 05:42, 28 February 2021 (UTC)
https://toolsadmin.wikimedia.org/tools/id/coibot login and you can maintain maintainers  — billinghurst sDrewth 11:23, 28 February 2021 (UTC)
Return to the user page of "Beetstra/Archives 2021".