The cdf name was an attempt to make it more similar to the probability distributions in scipy.stats but in retrospect that was confusing since the convoys models do not represent probability distributions – they are more like regression models that output something between 0 and 1.

This deprecates the old method names but keeps them for the time being.

EDIT: also splits the prediction method into predict vs predict_ci along the lines of https://martinfowler.com/bliki/FlagArgument.html

Instead of using bootstrapping to estimate uncertainty of c, just fit a Beta distribution directly

This is probably 10-100x faster although a few more lines of math (lots of gammaln)

Will do the same thing for Weibull and Gamma and then remove the bootstrapping (and the old non-beta models). Will also remove a few other things like sharing parameters etc.

Hey big fan of the project.

A Numpy Call to np.empty in Emcee is unhappy since numpy 1.11 (I think) that n_iterations is a float.

np.ceil() is used to calculate the number of iterations and it returns a float. Changing the result to an int fixes Generalized Gamma as of Numpy version 1.21.2.

Implemented a regression model using pymc3, so fully Bayesian.

The good news is that it was very easy to implement (once I discovered the pymc3.Potential class). The bad news is this is slow as hell. It barely works for more than a few thousand datapoints – whereas the current models happily fit 100k datapoints in a few seconds.

I'm tempted to revert to my original idea of just fitting this using MAP and then computing the Hessian of the MAP to get a normal approximation of the posterior distribution. Seems a bit janky but I think it will be a pretty good approximation in practice, and probably ~100x faster.

This changes a bunch of stuff

• Minibatches for faster convergence (although it makes every epoch a lot slower). This also gets rid of the checks for nan etc
• Dropout rather than regression. Dropout on the inputs.
• Fit an intercept to the regression model as a separate parameter just so that it's not affected by dropout
• Turns out tf.igamma has a bug in the gradient wrt a (I'll open an issue with tensorflow later). Rather than setting it using gradient descent, do a minor perturbation after each step
• Once gamma was fixed, turns out we can have a much simpler learning rate scheme, not save snapshots etc.

I too have struggled with gamma derivatives! This PR is my current best effort, so I can share it with convoys.

1. Use an analytical solution for derivative w.r.t the x variable
2. Use a O(h^4) order finite difference method w.r.t. the a (k) variable. This has much better stability in the second derivative too.

I've also included gammaincc, as gammaincc=1-gammainc and you have 1-gammainc in your code, so this is a possible small convenience.

PR for #93

Check if user is calling GeneralizedGamma within a Jupyter Notebook and suppress sys.stdout.flush() if true. Still need to think about how to replicate the carriage return so we can see the updating as the model fits.

Current output:

PR output:

Found a few issues

• Something with the covariance matrix not being positive semidefinite, causing numpy.random.normal_multivariate to complain and generate bogus values. This is despite eigenvalues being all positive
• A few of the variances were super large, causing enormous values of z that expit can't handle. Clipping fixes that problem
• It seems like a bias is introduced due to maybe floating point rounding. Not sure what's up, but with n=1000 its end up having a fairly substantial upwards bias early in the distribution, whereas this disappears for n=100. Not quite sure what's up.

After these changes, Weibull estimation of synthetic Weibull data lines up fairly well with the nonparametric estimation

Hey there! We, the Buffer data team, recently discovered this awesome package, and we're starting to use it in different analysis.

We're used to doing most of the plotting with R. I've started to work on getting the data back from the Matplotlib figure but seems like a hack and was wondering if you've thought about decoupling the plotting from the modelling.

Prophet, from Facebook, does a great job at that and it'll return a DataFrame with the required data to plot. The same prophet library will also have a default .plot function that uses Matplotlib. That helps users use other plotting frameworks.

I'm happy to help with the coding if I can figure out how to better do the decoupling. Let me know if you have any questions too. :smile:

Thanks for open sourcing such a helpful library!

PS: We've also found that using a large group size will result in a confusing legend in the final plot. This one can be probably fixed using the proper Matplotlib arguments though. This example shows weeks in one of our plots:

Throwing the time diff calculation in get_arrays into an apply function for vectorization -- seemed to make it run 5x faster when I tested it locally. Not too different in terms of code, I'm guessing the T_raw.append that took the longest time

this was some debugging thing i accidentally committed

Also simplified the Gamma stuff by switching to Powell which is a lot slower but seems a lot simpler and doesn't seem to suffer from weird ass nan behavior.

Steps to reproduce:

• clone the repo
• python -m venv venv and source venv/bin/activate
• pip install convoys==0.2.1
• python examples/dob_violations.py

Stacktrace:

File "examples/dob_violations.py", line 50, in run() File "examples/dob_violations.py", line 25, in run convoys.plotting.plot_cohorts(G, B, T, model=model, ci=0.95, File "/Users/jacopotagliabue/Documents/repos/convoys/venv/lib/python3.8/site-packages/convoys/plotting.py", line 62, in plot_cohorts m.fit(G, B, T) File "/Users/jacopotagliabue/Documents/repos/convoys/venv/lib/python3.8/site-packages/convoys/multi.py", line 31, in fit self.base_model.fit(X, B, T) File "/Users/jacopotagliabue/Documents/repos/convoys/venv/lib/python3.8/site-packages/convoys/regression.py", line 269, in fit for i, _ in enumerate(sampler.sample(p0, iterations=n_iterations)): File "/Users/jacopotagliabue/Documents/repos/convoys/venv/lib/python3.8/site-packages/emcee/ensemble.py", line 379, in sample self.backend.grow(iterations, state.blobs) File "/Users/jacopotagliabue/Documents/repos/convoys/venv/lib/python3.8/site-packages/emcee/backends/backend.py", line 175, in grow a = np.empty((i, self.nwalkers, self.ndim), dtype=self.dtype) TypeError: 'numpy.float64' object cannot be interpreted as an integer

I am running into an numpy type error (see below) whenever I set value for the parameters mcmc==True or ci==0.95 in any convoys model. This is consistent across data sources and even occurs when using the example data sets. If I remove these parameters the code runs as expected with no errors. Is this something anyone else has come across? Any help is much appreciated!

TypeError: 'numpy.float64' object cannot be interpreted as an integer
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-160-acdf74229cef> in <module>
----> 1 model_test.fit(X,B,T)

~/opt/anaconda3/lib/python3.8/site-packages/convoys/multi.py in fit(self, G, B, T)
     29         for i, group in enumerate(G):
     30             X[i,group] = 1
---> 31         self.base_model.fit(X, B, T)
     32 
     33     def _get_x(self, group):

~/opt/anaconda3/lib/python3.8/site-packages/convoys/regression.py in fit(self, X, B, T, W)
    267                     ' %d walkers [' % n_walkers,
    268                     progressbar.AdaptiveETA(), ']'])
--> 269             for i, _ in enumerate(sampler.sample(p0, iterations=n_iterations)):
    270                 bar.update(i+1)
    271             result['samples'] = sampler.chain[:, n_burnin:, :] \

~/opt/anaconda3/lib/python3.8/site-packages/emcee/ensemble.py in sample(self, initial_state, log_prob0, rstate0, blobs0, iterations, tune, skip_initial_state_check, thin_by, thin, store, progress, progress_kwargs)
    377             checkpoint_step = thin_by
    378             if store:
--> 379                 self.backend.grow(iterations, state.blobs)
    380 
    381         # Set up a wrapper around the relevant model functions

~/opt/anaconda3/lib/python3.8/site-packages/emcee/backends/backend.py in grow(self, ngrow, blobs)
    173         self._check_blobs(blobs)
    174         i = ngrow - (len(self.chain) - self.iteration)
--> 175         a = np.empty((i, self.nwalkers, self.ndim), dtype=self.dtype)
    176         self.chain = np.concatenate((self.chain, a), axis=0)
    177         a = np.empty((i, self.nwalkers), dtype=self.dtype)

TypeError: 'numpy.float64' object cannot be interpreted as an integer

Calling fit on GeneralizedGamma within a Jupyter Labs notebook results in a lot of whitespace in the output. I think this is because sys.stdout.flush() does not play well Labs.