Research:Investigate frequency of section titles in 5 large Wikipedias

Created

13:07, 20 October 2016 (UTC)

Contact

Zareen Farooqui

Wikimedia Foundation

Collaborators

Tilman Bayer

Wikimedia Foundation

Duration: 2016-10 – 2016-12

Open source
via github.com

Open data
via figshare.com

Research:Projects

This page documents a completed research project.

Tracked in Phabricator:
Task T148260

Wikipedia article pages are separated by section headings to organize the content of each page. This project aims to investigate the 100 most frequent section headings in 5 large Wikipedias and produce general insights around section headings.

Methods

This research utilizes the “Articles, templates, media/file descriptions, and primary meta-pages” data dump to generate a dataset consisting of all page headings (article section titles) for the English, French, German, Italian and Spanish Wikipedias on PAWS. The page id, page title, page namespace, heading level and heading text of each article page in that language edition is obtained using Aaron Halfaker's method of extracting headings, using the mwparserfromhell library. The data dumps contain page information for all namespaces, but this research project aims to explore only the main/article namespace (0 for all languages), so only pages with namespace 0 are added to the dataset.

To ensure data integrity, there are a variety of data quality checks on the page headings dataset. These checks confirm the dataset contains all article pages with headings from the data dump, all sections headings in any given article page, and that the sections headings are in sequential order as they appear in the article. This was done by pulling out the last 500 lines of the data dump and confirming all article pages with headings in these 500 lines were in the headers dataset. It is important to note that the code to extract headings does not always parse through the entire data dump and many revisions of the headers dataset were generated before ensuring data integrity. The random article generator is used to spot check articles, ensuring that they are present with their correct heading sequence in the derived dataset. The total article count is derived to calculate the percentage of articles in which each heading appears. The total article count includes all pages from the data dump with namespace 0 and no redirects.

In some cases, the wikitext for headings is formatted as e.g. “== See also ==”, but in other cases it appears as “==See also==”. The former method produces the heading “ See also ”, while the latter produces “See also”. For this analysis, these can be regarded as the same, so all leading and trailing whitespace in the heading text is removed to avoid duplicate titles.

Due to the large datasets and memory limitations in PAWS (1 GB), there are many workarounds in the python code for this analysis. For example, the section headers tsv files are read in small chunks of 100,000 rows, then concatenated together into one pandas dataframe to avoid reading the entire file into memory at once. Also, three columns of the dataframe are converted from the standard np.int64 to np.int32/16/8 data types to conserve memory. Still, the English, French, and German languages produce header files which are too large to analyze in PAWS, so the results for these were completed on a personal laptop. The PAWS notebooks for these languages contain commented out code which one can run if and when the PAWS memory limits are increased.

Results

Top 10 Section Headings
	en	en %	fr	fr %	de	de %	it	it %	es	es %
1	References	78.19	Notes and references Notes et références	43.85	Web links Weblinks	63.55	External links Collegamenti esterni	52.42	References Referencias	66.58
2	External links	44.33	External links Liens externes	40.48	Individual citations Einzelnachweise	46.61	Notes Note	44.73	External links Enlaces externos	56.81
3	See also	21.51	See also / External links Voir aussi	23.63	Literature Literatur	28.19	Other projects Altri progetti	31.43	See also Véase también	21.89
4	History	10.11	References Références	18.6	History Geschichte	14.92	See also Voci correlate	18.8	Bibliography Bibliografía	11.49
5	Notes	5.37	See also Articles connexes	18.17	Life Leben	13.1	Bibliography Bibliografia	16.86	History Historia	10.68
6	Career	3.34	Biography Biographie	16.0	See also Siehe auch	9.16	Biography Biografia	10.63	Demography Demografía	9.13
7	Biography	2.89	History Histoire	12.82	Sources Quellen	3.92	Plot Trama	8.91	Biography Biografía	7.62
8	Further reading	2.81	Bibliography Bibliographie	12.75	Awards Auszeichnungen	2.99	History Storia	5.95	Geography Geografía	6.62
9	Track listing	2.75	External link Lien externe	8.47	Career Karriere	2.93	Career Carriera	5.17	Geographic Distribution Distribución geográfica	4.32
10	Bibliography	2.32	Geography Géographie	6.6	Geography Geographie	2.87	Achievements Palmarès	4.64	Notes Notas	3.39

There are a few headings which are applicable for any subjects such as “References”, “External Links”, and “See Also” which appear frequently in all languages analyzed. After that, the headings are more specific to the topic of the article, e.g. if it’s an article about a person then “Career”, “Biography”, or “Life”.

All analyzed language editions have some version of “References” and "External Links" as the top 2 frequent heading titles. Italian and German language don’t use an exact equivalence of the “References” section heading. In Italian, "Notes" ("Note") and “Bibliography” ("Bibliografia") sections serve the same purpose as "References" (but "Bibliografia" also contains what is called "Further reading" on English Wikipedia, and "Note" also contains non-reference footnotes and remarks). In German, “Individual citations” ("Hilfe:Einzelnachweise") is used for references, but they also appear in "Literature" ("Literatur") sections alongside further reading material. Both of these languages use “External links” ("Collegamenti esterni"/"Weblinks") more than those counterparts of “References”. In French, there is a “Notes and References” ("Notes et références") header which is more popular than the similar “References” ("Références") heading.

In the French language, both "Articles connexes" and "Voir aussi" serve as "See also" headings, but "Voir aussi" can be used for a section encompassing both "External links" and "See also" (example article).

There are comparable, but different headings more frequently used in different language editions. For example, in German, “Life” (“Leben”) appears in 13% of articles (presumably most of these are articles about people) and is the 5th most frequent heading. However, in English Wikipedia “Life” appears in less than 1% of articles and is only the 27th most frequent heading. Instead, “Biography” is used more frequently in 3% of English articles and is the 7th most frequent heading. In German, “Biography” (“Biografie”) is the 32nd most frequent heading and appears in under 1% of articles.

The headers datasets below contain the following columns:

page_id: identifier of the page
page_title: title of the page
page_ns: namespace of page
heading_level: level of heading on page
heading_text: text of heading

The headings for article pages are in sequential order, but not always in consecutive order. So while all headings are in the dataset in the order in which they appear on their article page, there may be other article pages mixed in before one completes.

English

	number_of_articles	section_title	article_percentage
1	4125018	References	78.19
2	2338348	External links	44.33
3	1134624	See also	21.51
4	533444	History	10.11
5	283206	Notes	5.37
6	176458	Career	3.34
7	152442	Biography	2.89
8	148218	Further reading	2.81
9	145087	Track listing	2.75
10	122415	Bibliography	2.32
11	114564	Sources	2.17
12	110000	Early life	2.09
13	109315	Cast	2.07
14	106490	Geography	2.02
15	98892	Personal life	1.87
16	94728	Background	1.8
17	94726	Plot	1.8
18	86593	Reception	1.64
19	80904	Description	1.53
20	79214	Discography	1.5
21	77536	Demographics	1.47
22	76750	Awards	1.45
23	75455	Personnel	1.43
24	63232	Education	1.2
25	58490	Results	1.11
26	56609	Filmography	1.07
27	52229	Life	0.99
28	52182	Honours	0.99
29	48520	Production	0.92
30	47715	Works	0.9
31	44535	Gallery	0.84
32	38929	Overview	0.74
33	38527	Singles	0.73
34	36777	Footnotes	0.7
35	35906	Death	0.68
36	34101	Charts	0.65
37	32771	Distribution	0.62
38	30184	Population	0.57
39	29933	Publications	0.57
40	29678	Legacy	0.56
41	29258	Family	0.55
42	26886	Music	0.51
43	26723	Club career	0.51
44	26523	Early life and education	0.5
45	26453	Television	0.5
46	26328	Species	0.5
47	26249	Events	0.5
48	26221	Final	0.5
49	25783	Books	0.49
50	25289	Critical reception	0.48
51	25235	Albums	0.48
52	25108	Economy	0.48
53	24934	Early years	0.47
54	24572	Location	0.47
55	24463	International career	0.46
56	23888	Schedule	0.45
57	23759	Professional career	0.45
58	22722	Career statistics	0.43
59	22160	Notable people	0.42
60	21110	Etymology	0.4
61	21090	Notes and references	0.4
62	21023	Synopsis	0.4
63	20975	Playing career	0.4
64	20652	Chart performance	0.39
65	19553	Development	0.37
66	19511	Climate	0.37
67	19491	Political career	0.37
68	19444	Transportation	0.37
69	18832	Politics	0.36
70	18714	Other	0.35
71	18473	Release	0.35
72	18408	Film	0.35
73	18214	Life and career	0.35
74	18076	Aftermath	0.34
75	18073	People	0.34
76	17476	Deaths	0.33
77	17385	Soundtrack	0.33
78	17346	Early career	0.33
79	17265	Citations	0.33
80	17092	Transport	0.32
81	16918	Music video	0.32
82	16085	Awards and nominations	0.3
83	15993	Members	0.3
84	15938	Births	0.3
85	15913	Club	0.3
86	15616	Selected filmography	0.3
87	15615	Statistics	0.3
88	15419	In popular culture	0.29
89	15377	Episodes	0.29
90	15269	Literature	0.29
91	15239	Gameplay	0.29
92	15132	Plot summary	0.29
93	15115	Characters	0.29
94	15106	Regular season	0.29
95	14802	Seeds	0.28
96	14574	Places	0.28
97	14315	Notable alumni	0.27
98	14306	Sports	0.27
99	14132	Roster	0.27
100	13856	Release history	0.26

French

	number_of_articles	section_title	article_percentage
1	793236	Notes et références	43.85
2	732207	Liens externes	40.48
3	427548	Voir aussi	23.63
4	336423	Références	18.6
5	328723	Articles connexes	18.17
6	289374	Biographie	16.0
7	231921	Histoire	12.82
8	230700	Bibliographie	12.75
9	153257	Lien externe	8.47
10	119463	Géographie	6.6
11	102788	Notes	5.68
12	102284	Annexes	5.65
13	97813	Palmarès	5.41
14	86004	Description	4.75
15	85473	Historique	4.72
16	81072	Distribution	4.48
17	80643	Démographie	4.46
18	79613	Sources	4.4
19	65770	Synopsis	3.64
20	54188	Fiche technique	3.0
21	46953	Carrière	2.6
22	46324	Filmographie	2.56
23	43431	Article connexe	2.4
24	40367	Liens internes	2.23
25	39969	Économie	2.21
26	37215	Politique et administration	2.06
27	37065	Lieux et monuments	2.05
28	36600	Publication originale	2.02
29	35545	Toponymie	1.96
30	35222	Personnalités liées à la commune	1.95
31	33932	Source	1.88
32	31157	Œuvres	1.72
33	28891	Discographie	1.6
34	25621	Répartition	1.42
35	24583	Administration	1.36
36	22934	Communes limitrophes	1.27
37	22423	Étymologie	1.24
38	21455	Localisation	1.19
39	21247	Statistiques	1.17
40	20097	Compléments	1.11
41	20079	Publications	1.11
42	20064	Distinctions	1.11
43	18609	Cinéma	1.03
44	18173	Récompenses	1.0
45	17552	Galerie	0.97
46	16409	Événements	0.91
47	16214	Télévision	0.9
48	16135	Héraldique	0.89
49	15906	Résultats	0.88
50	15189	Composition	0.84
51	14568	Résumé	0.81
52	14235	Présentation	0.79
53	13338	Caractéristiques	0.74
54	13194	Décès	0.73
55	12824	Culture locale et patrimoine	0.71
56	12022	Liste des titres	0.66
57	12001	Liste des maires	0.66
58	11783	Classement	0.65
59	11713	Source de la traduction	0.65
60	11528	Culture	0.64
61	11462	Naissances	0.63
62	10996	Situation	0.61
63	10717	Accueil	0.59
64	10519	Architecture	0.58
65	10282	Population et société	0.57
66	10245	Patronyme	0.57
67	10223	Liste des espèces	0.57
68	10195	Référence	0.56
69	9945	En club	0.55
70	9925	Œuvre	0.55
71	9703	Parcours	0.54
72	9648	Théâtre	0.53
73	9551	Production	0.53
74	9342	Jumelages	0.52
75	9270	Autres	0.51
76	9242	Autour du film	0.51
77	9107	Contexte	0.5
78	9104	Climat	0.5
79	8892	Pour approfondir	0.49
80	8862	Personnages	0.49
81	8571	Personnalités	0.47
82	8558	Enseignement	0.47
83	8481	Vie privée	0.47
84	8423	Hydrographie	0.47
85	8351	Albums	0.46
86	8240	Références taxinomiques	0.46
87	8153	Évolution démographique	0.45
88	8078	Titres	0.45
89	7963	Population	0.44
90	7919	Musique	0.44
91	7893	Compétition	0.44
92	7837	[[Hameau italien\|Hameaux]]	0.43
93	7520	Records	0.42
94	7434	Organisation	0.41
95	7316	Système de jeu	0.4
96	7278	Nominations	0.4
97	7189	Famille	0.4
98	7163	Ouvrages	0.4
99	7097	Origine	0.39
100	7036	Toponyme	0.39

German

	number_of_articles	section_title	article_percentage
1	1266663	Weblinks	63.55
2	929079	Einzelnachweise	46.61
3	561919	Literatur	28.19
4	297422	Geschichte	14.92
5	261187	Leben	13.1
6	182654	Siehe auch	9.16
7	78065	Quellen	3.92
8	59574	Auszeichnungen	2.99
9	58404	Karriere	2.93
10	57293	Geographie	2.87
11	52344	Werke	2.63
12	42128	Sehenswürdigkeiten	2.11
13	39915	Handlung	2.0
14	37414	Beschreibung	1.88
15	33011	Geografie	1.66
16	32742	Erfolge	1.64
17	32433	Werdegang	1.63
18	31757	Verkehr	1.59
19	30840	Politik	1.55
20	30571	Lage	1.53
21	29715	Anmerkungen	1.49
22	29445	Schriften	1.48
23	26928	Diskografie	1.35
24	25494	Filmografie (Auswahl)	1.28
25	25313	Leben und Wirken	1.27
26	24873	Persönlichkeiten	1.25
27	23077	Bevölkerungsentwicklung	1.16
28	22387	Ehrungen	1.12
29	22166	Merkmale	1.11
30	21014	Hintergrund	1.05
31	19620	Sonstiges	0.98
32	18932	Biografie	0.95
33	18639	Werke (Auswahl)	0.94
34	18339	Kritik	0.92
35	17799	Wappen	0.89
36	17705	Filmografie	0.89
37	17473	Fußnoten	0.88
38	17434	Wirtschaft	0.87
39	17427	Rezeption	0.87
40	16534	Bevölkerung	0.83
41	16185	Familie	0.81
42	16027	Belege	0.8
43	15423	Veröffentlichungen	0.77
44	15353	Wirtschaft und Infrastruktur	0.77
45	14724	Kritiken	0.74
46	14248	Kultur und Sehenswürdigkeiten	0.71
47	13712	Leben und Werk	0.69
48	13469	Systematik	0.68
49	13052	Nationalmannschaft	0.65
50	12961	Inhalt	0.65
51	12530	Werk	0.63
52	12024	Verbreitung	0.6
53	11542	Schriften (Auswahl)	0.58
54	11444	Ergebnisse	0.57
55	10761	Alben	0.54
56	10721	Geographische Lage	0.54
57	10252	Lebensweise	0.51
58	10025	Einwohnerentwicklung	0.5
59	9726	Verlauf	0.49
60	9590	Quelle	0.48
61	9585	Singles	0.48
62	9550	Referenzen	0.48
63	9327	Söhne und Töchter der Stadt	0.47
64	9288	Verwendung	0.47
65	9164	Vorkommen	0.46
66	9094	Weblink	0.46
67	9076	Trivia	0.46
68	8969	Eigenschaften	0.45
69	8869	Architektur	0.44
70	8192	Gemeindegliederung	0.41
71	8003	Bildung	0.4
72	7903	Nachweise	0.4
73	7882	Technische Daten	0.4
74	7724	Publikationen	0.39
75	7705	Sport	0.39
76	7688	Erfolge und Auszeichnungen	0.39
77	7545	Entstehung	0.38
78	7443	Ausstattung	0.37
79	7432	Statistik	0.37
80	7325	Vorgeschichte	0.37
81	7312	Galerie	0.37
82	7284	Privates	0.37
83	7199	Bauwerke	0.36
84	7093	Finale	0.36
85	6896	Titelliste	0.35
86	6862	Verein	0.34
87	6665	Modus	0.33
88	6592	Produktion	0.33
89	6565	International	0.33
90	6524	Allgemeines	0.33
91	6491	Vereine	0.33
92	6479	Herkunft	0.33
93	6470	Bedeutung	0.32
94	6265	Gemeinden	0.31
95	6227	Städtepartnerschaften	0.31
96	6106	Bilder	0.31
97	6087	Einzel	0.31
98	6026	Leben und Karriere	0.3
99	5972	Gemeinderat	0.3
100	5940	Setzliste	0.3

Italian

	number_of_articles	section_title	article_percentage
1	686509	Collegamenti esterni	52.42
2	585769	Note	44.73
3	411609	Altri progetti	31.43
4	246274	Voci correlate	18.8
5	220758	Bibliografia	16.86
6	139173	Biografia	10.63
7	116651	Storia	8.91
8	77859	Trama	5.95
9	67758	Carriera	5.17
10	60746	Palmarès	4.64
11	57354	Società	4.38
12	55784	Evoluzione demografica	4.26
13	55750	Tracce	4.26
14	36585	Produzione	2.79
15	35365	Distribuzione	2.7
16	35188	Descrizione	2.69
17	32684	Club	2.5
18	30414	Formazione	2.32
19	28690	Statistiche	2.19
20	28591	Risultati	2.18
21	25504	Tabellone	1.95
22	24082	Filmografia	1.84
23	23701	Teste di serie	1.81
24	23278	Opere	1.78
25	22063	Discografia	1.68
26	21854	Geografia fisica	1.67
27	21555	Nazionale	1.65
28	20642	Onorificenze	1.58
29	18295	Cinema	1.4
30	17490	Competizioni nazionali	1.34
31	16350	Curiosità	1.25
32	15833	Finale	1.21
33	15829	Televisione	1.21
34	15491	Caratteristiche	1.18
35	15065	Rosa	1.15
36	13722	Fonti	1.05
37	13372	Amministrazione	1.02
38	13283	Galleria d'immagini	1.01
39	13180	Classifiche	1.01
40	12993	Codici	0.99
41	12438	Distribuzione e habitat	0.95
42	11440	Riconoscimenti	0.87
43	11225	Parte bassa	0.86
44	11224	Parte alta	0.86
45	11192	Doppio	0.85
46	10771	Singolare	0.82
47	10731	Tassonomia	0.82
48	10162	Persone	0.78
49	9854	Caratteristiche tecniche	0.75
50	9683	Monumenti e luoghi d'interesse	0.74
51	9681	Edizioni	0.74
52	9593	Musica	0.73
53	9589	Altro	0.73
54	9164	Cenni storici	0.7
55	8719	Classifica finale	0.67
56	8710	Biologia	0.67
57	8418	Squadre partecipanti	0.64
58	8311	Stagione	0.63
59	8292	Personaggi	0.63
60	8214	Il disco	0.63
61	7680	Filmografia parziale	0.59
62	7476	Pagine correlate	0.57
63	7460	Singoli	0.57
64	7457	Sport	0.57
65	7389	Territorio	0.56
66	7323	Economia	0.56
67	7161	Semifinali	0.55
68	6929	Gemellaggi	0.53
69	6799	Classifica	0.52
70	6752	Girone di ritorno	0.52
71	6685	Toponimi	0.51
72	6568	Etimologia	0.5
73	6543	Struttura	0.5
74	6351	Eventi	0.48
75	6327	Girone di andata	0.48
76	6320	Vita privata	0.48
77	6277	Albo d'oro	0.48
78	6277	Giocatore	0.48
79	6144	Episodi	0.47
80	6134	Album	0.47
81	6021	Presenze e reti nei club	0.46
82	5954	Infrastrutture e trasporti	0.45
83	5936	Premi e riconoscimenti	0.45
84	5727	Allenatore	0.44
85	5665	Geografia	0.43
86	5643	Critica	0.43
87	5579	Cultura	0.43
88	5529	Calendario	0.42
89	5489	Individuale	0.42
90	5468	Riproduzione	0.42
91	5414	Sigle	0.41
92	5352	Alimentazione	0.41
93	5228	Finali	0.4
94	5211	Verdetti	0.4
95	5145	Cronologia presenze e reti in Nazionale	0.39
96	5054	Musicisti	0.39
97	5034	Campioni	0.38
98	5003	Statistiche di squadra	0.38
99	4861	Composizione	0.37
100	4704	Sezione 2	0.36

Spanish

	number_of_articles	section_title	article_percentage
1	832356	Referencias	66.58
2	710186	Enlaces externos	56.81
3	273686	Véase también	21.89
4	143671	Bibliografía	11.49
5	133542	Historia	10.68
6	114151	Demografía	9.13
7	95259	Biografía	7.62
8	82735	Geografía	6.62
9	54006	Distribución geográfica	4.32
10	42439	Notas	3.39
11	39687	Descripción	3.17
12	31863	Palmarés	2.55
13	30381	Clubes	2.43
14	29044	Taxonomía	2.32
15	24475	Trayectoria	1.96
16	23951	Discografía	1.92
17	21204	Filmografía	1.7
18	20906	Lista de canciones	1.67
19	19805	Carrera	1.58
20	19619	Argumento	1.57
21	18345	Características	1.47
22	17713	Morfología	1.42
23	16471	Hábitat	1.32
24	15416	Obras	1.23
25	14505	Premios	1.16
26	14063	Notas y referencias	1.12
27	13418	Reparto	1.07
28	13174	Sinopsis	1.05
29	12120	Economía	0.97
30	11845	Especies	0.95
31	11359	Obra	0.91
32	11118	Fuentes	0.89
33	10966	Selección nacional	0.88
34	10549	Clasificación	0.84
35	10508	Resultados	0.84
36	9846	Antecedentes	0.79
37	9634	Palmarés internacional	0.77
38	9612	Distribución y hábitat	0.77
39	8987	Televisión	0.72
40	8667	Campeonatos nacionales	0.69
41	8590	Personajes	0.69
42	8045	Clima	0.64
43	7939	Recepción	0.64
44	7914	Distribución	0.63
45	7893	Producción	0.63
46	7882	Galería	0.63
47	7874	Estadísticas	0.63
48	7576	Población	0.61
49	7550	Primeros años	0.6
50	7202	Evolución demográfica	0.58
51	6918	Ubicación	0.55
52	6498	Vida personal	0.52
53	6464	Toponimia	0.52
54	6425	Personal	0.51
55	6163	Teatro	0.49
56	6135	Curiosidades	0.49
57	6128	Premios y nominaciones	0.49
58	6058	Cine	0.48
59	5926	Libros	0.47
60	5885	Honores	0.47
61	5800	Educación	0.46
62	5701	Algunas publicaciones	0.46
63	5574	Etimología	0.45
64	5404	Sencillos	0.43
65	5398	Composición	0.43
66	5164	Publicaciones	0.41
67	5152	Películas	0.41
68	5142	Cultura	0.41
69	5124	Elenco	0.41
70	5080	Música	0.41
71	5024	Miembros	0.4
72	4986	Créditos	0.4
73	4954	Desarrollo	0.4
74	4954	Localización	0.4
75	4752	Final	0.38
76	4742	Designación y nombre	0.38
77	4669	Jugadores	0.37
78	4665	Equipos	0.37
79	4629	Fiestas	0.37
80	4603	Trayectoria deportiva	0.37
81	4594	Inicios	0.37
82	4291	Álbumes	0.34
83	4231	Vida	0.34
84	4151	Reproducción	0.33
85	4149	Servicios	0.33
86	4110	Transporte	0.33
87	4035	Origen	0.32
88	3962	Turismo	0.32
89	3927	Usos	0.31
90	3918	Administración	0.31
91	3885	Universidad	0.31
92	3856	Equipos participantes	0.31
93	3831	Trama	0.31
94	3822	Alimentación	0.31
95	3739	Profesional	0.3
96	3727	Participaciones en Copas del Mundo	0.3
97	3685	Muerte	0.29
98	3684	Referencia	0.29
99	3621	Temporada regular	0.29
100	3476	Lugares de interés	0.28