Analyzing PyPI Metadata

PyPI, the Python Package Index, gives a very crappy but simple interface to query metadata about its packages. I scrapped all of the packages metadata. 53,533 packages were scrapped (date: 2015-01-18), because I wanted to see if there is malware on PyPI (related to this question on security.SE).

The database looks like this:

Exploring the data

When I scapped the data from PyPI, I made all database fields "varchar 255" as there seems to be no information about the possible values. Then I explored the data

• name: The longest package name is 80 characters long (Aaaaaaaaaaa...), the shortest packages have only one character.
• author: 911× "UNKOWN", 741× empty, 195× "None", 151× "Zope Foundation and Contributors". There are about 22,000 different authors. There are 595 authors who wrote more than 10 packages.
• author_email: 2059× "UNKOWN", 932× empty, 323× "[email protected]", 216× "None" and 204× "TODO".
• maintainer: 46879× "None", 5507× empty, 17× Paul Boddie.
• requires_python: 53467× "None", 3× "UNKNOWN", 2× ">=2.7,!=3.0,!=3.1", 2× ">=2.5", 1× "2.6", 1× ">= 3.3", 1× ">=2.4", 1× ">= 2.7" - that's it. No other values. Seems as if this is pretty much useless.
• docs_url: Either begins with http://pythonhosted.org or is empty.

Most active authors

SELECT
    `author`, COUNT(`id`) as `created_packages`
FROM
    `packages`
GROUP BY
    `author`
ORDER BY
    COUNT(`id`) DESC, `author` ASC
LIMIT
    50

gives

The total number of authors is 28 183 (06.12.2015):

SELECT COUNT(DISTINCT `author`) AS `total_authors` FROM `packages`

Maximum Length

I guess many values are stored on PyPI as Varchar(255). To check if I might have missed some values, I checked which entry was the longest one for all columns. I did this with

SELECT `name`, LENGTH(`name`) FROM `packages` ORDER BY LENGTH(`name`) DESC

We can see multiple problems here:

• URLs: localhost / 127.0.0.1 is almost certainly not desired
• cheesecake_installability_id, cheesecake_documentation_id, cheesecake_code_kwalitee_id should have NULL and None should be casted to NULL. The data type is likely to be numeric.
• _pypi_ordering should be numeric
• _pypi_hidden should be boolean

So I changed the types in the database. Let's continue.

What people don't use

All metadata is provided by the authors of the packages. Some fields, like the package name or the description, are used very often, but some fields are only rarely used:

Maintainer

maintainer and maintainer_email is interesting for people who want to send bug reports. If this is empty, I would guess the package is dead.

Platform

SELECT
    `platform`, COUNT(`id`)
FROM
    `packages`
GROUP BY
    `platform`
ORDER BY
    COUNT(`id`) DESC

gives 759 different results. The TOP-10 were

You can see that there are alternative ways to express the same thing. Also, the ";" should not be here as the manual states it should be a list.

The manual is also too short for this entry. It only says "a list of platforms" which seems to be pretty much useless.

docstore.mik.ua gives a little bit more information

A list of platforms on which this distribution is known to work. You should provide this information if you have reasons to believe this distribution may not work everywhere. This information should be reasonably concise, so this field may refer for details to a file in the distribution or to a URL.

Bugtracker

This one is very important. Users should have an easy possibility to report bugs. So please help them by added your bug tracker URL wherever it makes sense. Here is how you add it on PyPI:

License

Another important one. Add a license to your software!

SELECT
    `license`, COUNT(`id`)
FROM
    packages
GROUP BY
    `license`
ORDER BY
    COUNT(`id`) DESC

gives

and about 6500 other licenses. Most might be variants in writing, e.g.

• Apache License 2.0
• Apache License, Version 2.0
• Apache 2.0

are all the same license. Some packages have no licence (Unknown and empty). Many are invalid values, like

• LICENSE.txt
• None
• LICENSE
• Python

These indicate that people have no idea what to input there. tldrlegal.com might help in that case. I think the Python community should try to eliminate variants in writing license names as it makes finding, filtering and analyzing packages more difficult.

Classifiers

Python makes use of so called trove classifiers. They are defined in PEP 301 and listed here.

The following table gives the TOP-10 most commonly used classifiers:

When you analyze the licenses with the trove classifiers you get a different image:

SELECT
    `classifier`, COUNT(`id`) / 53533
FROM
    `package_classifiers`
WHERE
    `classifier` LIKE "License%"
GROUP BY
    `classifier`
ORDER BY
    COUNT(`id`) DESC

There are 94 trove classifiers with five or less packages which use this classifier. I guess many of them are not in the official list of classifiers.

Package type

SELECT
    `packagetype`, COUNT(`id`)
FROM
    `urls`
GROUP BY
    packagetype
ORDER BY
    COUNT(`id`) DESC

gives

Can anybody explain what this means?

Downloads

SELECT
    `name`, `url`, `downloads`
FROM
    `urls`
JOIN
    `packages` ON `urls`.`package_id` = `packages`.`id`
ORDER BY
    `urls`.`downloads`  DESC
LIMIT 10

gives

and

SELECT
    `name`, SUM(`downloads`)
FROM
    `releases`
JOIN
    `packages` ON `releases`.`package_id` = `packages`.`id`
GROUP BY
    `name`
ORDER BY
    SUM(`downloads`)  DESC
LIMIT 10

Size

What is the biggest Python package?

SELECT
    `name`, `release_number`, `size`
FROM
    `releases`
JOIN
    `packages` ON `releases`.`package_id` = `packages`.`id`
ORDER BY
    `releases`.`size` DESC
LIMIT 30

Code

See github.com/MartinThoma/algorithms.

Related

• Why are some packages on pypi.python.org/simple, but have no page?
• How can I find out when the last interaction on PyPI happened for a given package?
• What is cheesecake_code_kwalitee_id on PyPI good for?

Further Ideas

• Build a dependency graph: Some of the code was already written. However, one has to download about 25 GB of data, extract it and run over those files. This is quite a bit of work.
• Analyze package quality
  • Missing requirements
  • Missing metadata / description
  • Missing documentation
  • PEP8
  • Code duplication
• Malicious package search:
• Check which package names are prefixes of other package names.
• Find packages which upload data (dropbox?)
• Find pacakges which remove data from your file system