skdata

I really like Machine Learning. I like reading papers, understanding and evaluating new ideas. But one part I always have to spend quite a bit of time on is loading the data. It's always a mess to find the datasets, understand where exactly I can download them and how they've packaged the information. Just a few days ago I found skdata. It is a Python package which aims at helping to load standard datasets. If I can trust the git commit message, then the development was started in August 2011 by James Bergstra! This is before AlexNet!

edit: Although it seemed to be a cool project, it seems to be dead, too. The last commit is from July 2015.

Usage

One way to use skdata is the following:

#!/usr/bin/env python

"""MNIST example with skdata."""

try:
    from skdata.mnist.view import OfficialVectorClassification
except ImportError:
    # Fallback, if you have an old version
    from skdata.mnist.views import OfficialVectorClassification
from sklearn.tree import DecisionTreeClassifier

# Load the data
view = OfficialVectorClassification()
train_idx = view.fit_idxs  # indices of training data
val_idx = view.val_idxs  # incices of validation data
test_idx = view.tst_idxs  # indices of test data

# Fit a simple classifier
print("Start fitting DecisionTreeClassifier.")
clf = DecisionTreeClassifier(max_depth=5)
features = view.all_vectors[train_idx]  # select features of training data
targets = view.all_labels[train_idx]  # select labels of training data
clf.fit(features, targets)

# Evaluate the classifier
predict = clf.predict(view.all_vectors[test_idx])
accuracy = sum(predict == view.all_labels[test_idx]) / float(len(test_idx))
print("Fitted DecisionTreeClassifier has test accuracy of %0.4f." % accuracy)

However, it is inteded to be used like this:

from skdata.mnist.view import OfficialVectorClassification
from sklearn.tree import DecisionTreeClassifier

# Load the data
mnist_view = OfficialVectorClassification()
train_idx = mnist_view.fit_idxs
val_idx = mnist_view.val_idxs
test_idx = mnist_view.tst_idxs

# Fit a simple classifier
from skdata.base import SklearnClassifier
learning_algo = SklearnClassifier(DecisionTreeClassifier)
mnist_view.protocol(learning_algo)
print(learn_algo.results['loss'][0]['task_name'] == 'tst')

... but this doesn't work (for me)

Other Data-Loading Projects

You can access R data with rpy2.

from rpy2.robjects import r
from rpy2.robjects import pandas2ri
def data(name): return pandas2ri.ri2py(r[name])
df = data('iris')
print(df.describe())

You can also load mnist, cifar10,cifar100, imdb, reuters with keras:

from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

See also

Published


by

Category

Machine Learning

Tags

Contact