pagecounts-raw-sorted
----------------------

This dataset consists of hourly pagecounts for Wikipedia pages sorted by article, ordered by `(project, page, timestamp)`. It has been created by processing Wikimedia's [`pagecounts-raw`](https://dumps.wikimedia.org/other/pagecounts-raw/) dataset.

The original dataset holds the desktop sites' pageview data (separately for every page) for the period from December 2007 to July 2016, with hourly granularity for all Wikipedia editions. More info about the original dataset are available on [Wikitech](https://wikitech.wikimedia.org/wiki/Analytics/Archive/Data/Pagecounts-raw). Note that **these are not unique visits**.

Here's an excerpt of the file [`pagecounts-20071210-000000.gz`](https://dumps.wikimedia.org/other/pagecounts-raw/2007/2007-12/pagecounts-20071210-000000.gz):
```
en Albert_Edward_Bridge 2 2
en Albert_Edward_Kemp 1 1
en Albert_Ehrenstein 1 1
en Albert_Einstein 540 540
en Albert_Einstein#_note-76 1 1
en Albert_Einstein%27s_brain 23 23
en Albert_Einstein's_brain 14 14
```

and in the processed dataset:
```
en Albert_Einstein 20071209-190000 471 471
en Albert_Einstein 20071209-200000 545 545
en Albert_Einstein 20071209-210000 546 546
en Albert_Einstein 20071209-220000 497 497
en Albert_Einstein 20071209-230000 564 564
en Albert_Einstein 20071210-000000 540 540
en Albert_Einstein 20071210-010000 567 567
en Albert_Einstein 20071210-020000 547 547
en Albert_Einstein 20071210-030000 557 557
en Albert_Einstein 20071210-040000 624 624
```

In the new dataset pages are sorted in a way such that the pagecounts for each article are grouped togheter and sorted by time, so that retrieving pagecounts data for specific articles is easier.

## FAQs

* **What is the total size of the dataset, the number of files and the largest file in the dataset?**
The total dataset size is 3.5TB, and it contains ~ 76,000 files. The average size is 45 MB and the largest file is 1.4GB.

* **How are files organized?**
Files are divided in directories, one for each month, like this:
```
.
├── 2007-12
│   ├── index.2007-12.txt
│   ├── md5sums.2007-12.txt
│   ├── part-00000.gz
│   ├── part-00001.gz
│   ├── ...
│   └── part-00533.gz
├── 2008-01
│   ├── index.2008-01.txt
│   ├── md5sums.2008-01.txt
│   ├── part-00000.gz
│   ├── part-00001.gz
│   ├── ...
│   └── part-00742.gz
├── 2008-02
│   ├── index.2008-02.txt
│   ├── md5sums.2008-02.txt
│   ├── part-00000.gz
│   ├── part-00001.gz
│   ├── ...
│   └── part-00695.gz
├── 2008-03
...
└── 2016-08
```
The average directory size is ~34 GB, and on average each contains 731 part files.

* **Who produced this dataset and why?**
The dataset has been produced by Cristian Consonni and Alberto Montresor, from the Department of Information Engineering and Computer Science (DISI), University of Trento, Trento, Italy.

This research has been supported by Microsoft Azure Research Award CRM:0518942 as part of the ["Azure for Research Award: Data Science" program](https://www.microsoft.com/en-us/research/academic-program/data-science-award/).

This dataset has also been utilized in the research related to the ENGINEROOM project, in collaboration with David Laniado of Eurecat - Centre Tecnològic de Catalunya, Barcelona, Spain. EU ENGINEROOM has received funding from the European Union's Horizon 2020 research and innovation programme under the Grant Agreement no 780643.

* **Is this dataset cited in some papers?**

This dataset has been used for multiple papers that are currently in-review or in preparation.

This dataset is published as part of the ENGINEROOM project and it will be cited in the related scientific publications.

## License

This dataset is released in the [Public Domain](http://creativecommons.org/publicdomain/mark/1.0/).