scikit-learn addon to operate on set/"group"-based features

Hi @dougalsutherland ,

Thanks for sharing your code; it is well documented and well written.

I am working on a problem and comparing different divergences. KL and Hellinger already produce good results but I am interested to compute linear affinity for a different purpose. Unfortunately more than 99% of the computed affinity values are infinity and the rest are very large number. Do you why is that and it can be resolved? l2 distance also produce a lot of inf and nan.

Thanks, Kayhan

Hi Dougal, I wanted to use skl-groups on some accelerometer data I have and was going through your example. I installed everything through conda and everything works until I try to fit the KNNDivergenceEstimator object. Any ideas?

from sklearn.pipeline import Pipeline
from skl_groups.divergences import KNNDivergenceEstimator
from skl_groups.kernels import PairwisePicker, Symmetrize, RBFize, ProjectPSD
model = Pipeline([
    ('divs', KNNDivergenceEstimator(div_funcs=['kl'], Ks=[2])),
    ('pick', PairwisePicker((0, 0))),
    ('symmetrize', Symmetrize()),
    ('rbf', RBFize(gamma=1, scale_by_median=True)),
    ('project', ProjectPSD()),
    ('svm', SVC(C=1, kernel='precomputed')),
])
model.fit(feats[:6], labels[:6]).predict(feats[6:])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-454843114715> in <module>()
     10     ('svm', SVC(C=1, kernel='precomputed')),
     11 ])
---> 12 model.fit(feats[:6], labels[:6]).predict(feats[6:])

/Users/wgmueller/anaconda/lib/python2.7/site-packages/sklearn/pipeline.pyc in fit(self, X, y, **fit_params)
    127         data, then fit the transformed data using the final estimator.
    128         """
--> 129         Xt, fit_params = self._pre_transform(X, y, **fit_params)
    130         self.steps[-1][-1].fit(Xt, y, **fit_params)
    131         return self

/Users/wgmueller/anaconda/lib/python2.7/site-packages/sklearn/pipeline.pyc in _pre_transform(self, X, y, **fit_params)
    117         for name, transform in self.steps[:-1]:
    118             if hasattr(transform, "fit_transform"):
--> 119                 Xt = transform.fit_transform(Xt, y, **fit_params_steps[name])
    120             else:
    121                 Xt = transform.fit(Xt, y, **fit_params_steps[name]) \

/Users/wgmueller/anaconda/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y, **fit_params)
    427         else:
    428             # fit method of arity 2 (supervised transformation)
--> 429             return self.fit(X, y, **fit_params).transform(X)
    430 
    431 

/Users/wgmueller/anaconda/lib/python2.7/site-packages/skl_groups/divergences/knn.pyc in fit(self, X, y, get_rhos)
    308             memory = Memory(cachedir=memory, verbose=0)
    309 
--> 310         self.indices_ = id = memory.cache(_build_indices)(X, self._flann_args())
    311         if get_rhos:
    312             self.rhos_ = _get_rhos(X, id, Ks, max_K, save_all_Ks, self.min_dist)

/Users/wgmueller/anaconda/lib/python2.7/site-packages/skl_groups/divergences/knn.pyc in _flann_args(self, X)
    264         # check that arguments are correct
    265         try:
--> 266             FLANNParameters().update(args)
    267         except AttributeError as e:
    268             msg = "flann_args contains an invalid argument:\n  {}"

TypeError: update() takes exactly 0 positional arguments (1 given)

I've been using KNNDivergenceEstimator to estimate the divergence between two samples. This might be a dumb question, but how can the renyi-alpha divergences estimates be negative?

I understand that if I set clamp=True then it imposes a limit, but I'm not sure how the divergence estimator from B. Poczos, L. Xiong, D. J. Sutherland, & J. Schneider (2012) could return negative values.

Thanks!

Like in Junier's paper: http://jmlr.org/proceedings/papers/v33/oliva14a.pdf (but the RBF/ridge stuff can be handled by pipelines with sklearn.kernel_approximation.RBFSampler and sklearn.linear_model.Ridge)

Probably only support the cosine basis, but maybe design the API to allow other options.

Hi Dougal,

I think, there might be a bug here in this line about the median trick:

https://github.com/dougalsutherland/skl-groups/blob/2584c10a413626c6d5f9078cdbf3dcc84e4e9a5b/skl_groups/kernels/transform.py#L197

I think scale = 1/median_ as you also mentioned in the help of the function. Your gamma is the inverse of the bandwidth (squared).

Besr

Hi Dougal,

I recently updated package and conda and I run into this issue:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-75fc8d6dda07> in <module>()
----> 1 divKernel_PSD, K_Chol = computeKernel(div)

<ipython-input-15-adc14e99cc3e> in computeKernel(divMatrix, gammaVal, projectName)
     18     tic()
     19     print( "Projecting on the PSD cone ...." )
---> 20     K_PSD = fcn().fit_transform( div_symmet_norm_RBF )
     21     print( "Done !" )
     22     toc()

/home/batmanghelich/anaconda2/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y, **fit_params)
    492         if y is None:
    493             # fit method of arity 1 (unsupervised transformation)
--> 494             return self.fit(X, **fit_params).transform(X)
    495         else:
    496             # fit method of arity 2 (supervised transformation)

/home/batmanghelich/anaconda2/lib/python2.7/site-packages/skl_groups/kernels/transform.pyc in fit(self, X, y)
    542 
    543         memory = get_memory(self.memory)
--> 544         lo, = memory.cache(scipy.linalg.eigvalsh)(X, eigvals=(0, 0))
    545         self.shift_ = max(self.min_eig - lo, 0)
    546 

/home/batmanghelich/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/memory.pyc in __call__(self, *args, **kwargs)
    281 
    282     def __call__(self, *args, **kwargs):
--> 283         return self.func(*args, **kwargs)
    284 
    285     def call_and_shelve(self, *args, **kwargs):

/home/batmanghelich/anaconda2/lib/python2.7/site-packages/scipy/linalg/decomp.pyc in eigvalsh(a, b, lower, overwrite_a, overwrite_b, turbo, eigvals, type, check_finite)
    682                 overwrite_a=overwrite_a, overwrite_b=overwrite_b,
    683                 turbo=turbo, eigvals=eigvals, type=type,
--> 684                 check_finite=check_finite)
    685 
    686 

/home/batmanghelich/anaconda2/lib/python2.7/site-packages/scipy/linalg/decomp.pyc in eigh(a, b, lower, eigvals_only, overwrite_a, overwrite_b, turbo, eigvals, type, check_finite)
    347             (lo, hi) = eigvals
    348             w_tot, v, info = evr(a1, uplo=uplo, jobz=_job, range="I",
--> 349                                  il=lo, iu=hi, overwrite_a=overwrite_a)
    350             w = w_tot[0:hi-lo+1]
    351 

ValueError: On entry to SSBRDB parameter number 12 had an illegal value

As you see svd issue from scipy. I couldn't find a good solution for it. Do you have any idea how to resolve it?

Thanks

Started in https://github.com/dougalsutherland/skl-groups/commit/2584c10a413626c6d5f9078cdbf3dcc84e4e9a5b, but:

• should check it doesn't slow things down too much
• should think a little more carefully about the exact things we're doing
• should add a numerical stability check
• should do it for the Cython alpha-divergence estimator too

• [ ] Helper function to compute the two-sample test statistic
  • [ ] Kernel learning
• Null statistic via:
  • [ ] bootstrapping
  • [ ] concentration inequalities
  • [ ] asymptotic approximations
  • ... what other ways are there?