Hi, I think there is a little bug at numpy-ml/numpy_ml/neural_nets/activations/activations.py Line 64.

your code

fn_x = self.fn_x

but

self.fn_x Never defined

A lot of linear regression operate in static mode which means that we have to retrain to get the parameters after getting a new data point. This limits the functionality of the linear regression and make working in real-time difficult. I will implement this online formulation using ideas from the paper attached to this issue. This is based on recursive least square formulation. I have implemented in one of my past project. However, if I get approvals, then I will clean it up and create a new PR

rls.pdf

• What I did

Implement Gaussian naive Bayes Class and Basic documentation.

• How I did it

Refer to the formula of Gaussian naive Bayes from the notes

• How to verify it

compare the performance with the gaussianNB from sci-kit learn.

Dear Post Dr.Bourgin, I wanna give a perfect method code, however , unfortunately may be it can not work well (I have not test the method).So.....uh~~ may I ask your help to finish it, then get the honor that be a contributor of this great project (5k+ stars).... THANK YOU!

Hello, this is a great project , I am learning how to implement model without sklearn/tensorflow , it really help me a lot .

I have a question on https://github.com/ddbourgin/numpy-ml/blob/4f37707c6c7c390645dec5a503c12a48e624b249/numpy_ml/linear_models/lm.py#L252

Since P-norm is defined as

l1norms(self.beta) means the sum of all absulote value of each element in self.beta . I don't quite understand why the simple gamma * beta stand for `L2 ?

PS: May I ask what IDE and code document plugin you are using ? I see some annotation don't beyond to latex , it would be nice to see beautiful math symbols than raw latex :)

hi, I think neural_nets/utils/utils.py line 797 has a bug!

your code

i0, i1 = i * s, (i * s) + fr * (d + 1) - d j0, j1 = j * s, (j * s) + fc * (d + 1) - d

right code

i0, i1 = i * s, (i * s) + fr j0, j1 = j * s, (j * s) + fc

because fr and fc are already dilated size !

This work is to create an online version of Linear regression as described in issue https://github.com/ddbourgin/numpy-ml/issues/70 . The implement the online version of the Ordinary least square using Matrix Inversion Lemma RLS, version 1 ( https://github.com/ddbourgin/numpy-ml/files/6536069/rls.pdf ).

Beware that there will be trade-off in some performance during the noisy nature of training on a sample at a time, in comparison to a batch. On the average, this should not be too significant.

To Do

Add unit test

• [x] Is the code you are submitting your own work? Yes
• [x] Did you properly attribute the authors of any code you referenced? Yes
• [x] Did you write unit tests for your new model?
• [x] Does your submission pass the unit tests?
• [ ] Did you write documentation for your new model?
• [ ] Have you formatted your code using the black deaults?

A sample for the API call is

olr = LinearRegression()
olr.fit(X, y)
olr.predict(rest-x)
# on new data just call update to modify beta field and update the model in an online manner
olr.update(x_new, y_new)

Note: no modification has happened to existing implementation as it is only active when update method is called.

Taking a quick look, some of the grad and grad2 functions might benefit from some optimizations. Here's on example:

https://github.com/ddbourgin/numpy-ml/blob/fce2acfd7c370f55373bdc6dff1761a8258bfe27/numpy_ml/neural_nets/activations/activations.py#L38-L39

Here the function could be changed such that fn(x) is only computed once:

def grad(self, x):
    fn_x = self.fn(x)
    return fn_x * (1 - fn_x)

The extra mem used to store the calculation should be immediately collected after the function ends so that shouldn't be a problem. Would love a second opinion @ddbourgin before making a PR with the necessary changes.

- What bug I fixed

According to @jjjjohnson in #16 , we can apply multi dimension covariance in gmm.

This pull request fixes #16.

- How I fixed it

1. change dimension from 2 to self.d.

- How you can verify it

The tests did not pass because of the following: And tests did pass before changing. We can ask @jjjjohnson to take a look.

/Users/zhuoran/Documents/git/numpy-ml/gmm/gmm.py:66: RuntimeWarning: invalid value encountered in double_scalars
  if np.isnan(vlb) or np.abs((vlb - prev_vlb) / prev_vlb) <= tol:
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
Singular matrix: components collapsed
Components collapsed; Refitting
/Users/zhuoran/Documents/git/numpy-ml/gmm/gmm.py:116: RuntimeWarning: invalid value encountered in true_divide
  self.mu[ix, :] = num / den
/Users/zhuoran/Documents/git/numpy-ml/gmm/gmm.py:43: RuntimeWarning: divide by zero encountered in log
  log_pi_k = np.log(pi_k)
/usr/local/lib/python3.7/site-packages/numpy/linalg/linalg.py:1817: RuntimeWarning: invalid value encountered in slogdet
  sign, logdet = _umath_linalg.slogdet(a, signature=signature)
Singular matrix: components collapsed
Components collapsed; Refitting
Traceback (most recent call last):
  File "/Users/zhuoran/Documents/git/numpy-ml/gmm/tests.py", line 111, in <module>
    plot()
  File "/Users/zhuoran/Documents/git/numpy-ml/gmm/tests.py", line 100, in plot
    ax = plot_clusters(G, X, ax)
  File "/Users/zhuoran/Documents/git/numpy-ml/gmm/tests.py", line 52, in plot_clusters
    rv = multivariate_normal(model.mu[c], model.sigma[c], allow_singular=True)
  File "/usr/local/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 363, in __call__
    seed=seed)
  File "/usr/local/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 736, in __init__
    self.cov_info = _PSD(self.cov, allow_singular=allow_singular)
  File "/usr/local/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 156, in __init__
    s, u = scipy.linalg.eigh(M, lower=lower, check_finite=check_finite)
  File "/usr/local/lib/python3.7/site-packages/scipy/linalg/decomp.py", line 374, in eigh
    a1 = _asarray_validated(a, check_finite=check_finite)
  File "/usr/local/lib/python3.7/site-packages/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/usr/local/lib/python3.7/site-packages/numpy/lib/function_base.py", line 1233, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

Hi, I am thinking about implementing Naive Bayes Methods and make a pull request. But I am unsure about the Unit testing part, Should I compare the performance with the ScikitLearn library?

This is an awesome library, thanks @ddbourgin!!

Users might not know the best way to install this package and try it out. (I didn't, so I eventually just copied the source files.) Neither the readme nor readthedocs have install instructions.

I couldn't find it on PyPi or Anaconda, and there doesn't appear to be a pyproject.toml, setup.cfg, setup.py, or conda recipe.

Moreover, the tests aren't in a standard path like tests/. This is uncommon and therefore confusion, and it makes it harder to run them. Edit: I wasn't expecting them under the source, so I initially wrote that I couldn't find them.

I think it would be great to document how to install numpy-ml, and run its tests & see them to clarify the behavior of some of the functions.

There are some great build and CI tools for Python available, which I recently learned how to use effectively. I'm happy to make a pull request if it would be helpful.

All Submissions

• [x] Is the code you are submitting your own work?
• [ ] Have you followed the contributing guidelines?
• [ ] Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Model Submissions

• [x] Is the code you are submitting your own work?
• [ ] Did you properly attribute the authors of any code you referenced?
• [ ] Did you write unit tests for your new model?
• [ ] Does your submission pass the unit tests?
• [ ] Did you write documentation for your new model?
• [ ] Have you formatted your code using the black deaults?

Changes to Existing Models

• [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
• [ ] Have you written new tests for your changes, as applicable?
• [ ] Have you successfully ran tests with your changes locally?

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 12.6
• Python version: 3.10.7
• NumPy version: 1.23.3

Describe the current behavior import of numpy_ml fails due to an ImportError with the collections module.

Describe the expected behavior Importing just the module should not generate an ImportError

Code to reproduce the issue

# Python 3.10 or newer
import numpy_ml

Other info / logs In Python 3.10 the deprecated aliases were removed.

Remove deprecated aliases to Collections Abstract Base Classes from the collections module. (Contributed by Victor Stinner in bpo-37324.)

from What’s New In Python 3.10

fix To fix the bug, in /numpy_ml/utils/data_structures.py change

from collections import Hashable

to

from collections.abc import Hashable

@ddbourgin Have an issue where updates to gradients cannot be performed since shapes conflict during backprop... specifically in the optimizer file.

Error reads:

C[param_name]["mean"] = d1 * mean + (1 - d1) * param_grad
ValueError: operands could not be broadcast together with shapes (100,10) (3072,100) 

Model architecture is as follows:

Input -> n_samples, 3072 FC1 -> 3072, 100 FC2 -> 100, 10

The model code is as follows:

def _build_model(self):
    self.model = OrderedDict()
    self.model['fc1'] = FullyConnected(n_out=self.layers[0],
                                       act_fn=ReLU(),
                                       init=self.initializer,
                                       optimizer=self.optimizer)


    self.model['fc2'] = FullyConnected(n_out=self.layers[1],
                                       act_fn=Affine(slope=1, intercept=0),
                                       init=self.initializer,
                                       optimizer=self.optimizer)


    self.model['out'] = Softmax(dim=-1,
                                optimizer=self.optimizer)

@property
def parameters(self):
    return {k: v.parameters for k, v in self.model.items()}

@property
def hyperparameters(self):
    return {k: v.hyperparameters for k, v in self.model.items()}

@property
def derived_variables(self):
    return {k: v.derived_variables for k, v in self.model.items()}

@property
def gradients(self):
    return {k: v.gradients for k, v in self.model.items()}

def forward(self, x):
    out = x
    for k, v in self.model.items():
        out = v.forward(out)
    return out

def backward(self, y, y_pred):
    """Compute dLdy and then backprop through the layers in self.model"""
    dY_pred = self.loss.grad(y, y_pred)
    for k, v in reversed(list(self.model.items())):
        dY_pred = v.backward(dY_pred)
        self._dv['d' + k] = dY_pred
    return dY_pred

def update(self, cur_loss):
    """Perform gradient updates"""
    for k, v in reversed(list(self.model.items())):
        v.update(cur_loss)
    self.flush_gradients()

Hoping we can fix this and also create an example for people to follow. Thanks

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Python version: 3.7.12
• NumPy version: 1.21.5 (environment is Google Colab on 20-Mar, 2022.)

Describe the current behavior I have copy-pasted the code for numpy_ml.linear_model.LinearRegression from github and did .fit() and .predict() on some dummy data. I got ValueError on .predict() like this:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-10-4be896198177>](https://localhost:8080/#) in <module>()
----> 1 npml_lin_reg2_preds = npml_lin_reg2.predict(X_val)
      2 npml_lin_reg2_preds[:10]

[<ipython-input-8-fc521849e158>](https://localhost:8080/#) in predict(self, X)
    206         if self.fit_intercept:
    207             X = np.c_[np.ones(X.shape[0]), X]
--> 208         return X @ self.beta

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1 is different from 11)

Describe the expected behavior Expected behaviour is that .predict() doesn't generate ValueError.

Code to reproduce the issue not code, here is the link to the notebook: https://colab.research.google.com/drive/12q9r2j4-UpUrPnzvMiPC6rxafa73cY5L?usp=sharing

Other info / logs

G'day, how's it going?

I've just started looking into machine learning stuff, and stumbled upon this, looks awesome!

I just want to know what kind of methods I should use for the following:

• Text identification (Spam checker for example)
• Image analysis (Detects whether the image given after training is a male or female)

Kind regards,

Machine-Learning newbie, Mitch!

All Submissions

• [Yes] Is the code you are submitting your own work?
• [Yes] Have you followed the contributing guidelines?
• [Yes] Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Model Submissions

• [Yes] Is the code you are submitting your own work?
• [Yes] Did you properly attribute the authors of any code you referenced?
• [Yes] Did you write unit tests for your new model?
• [Yes] Does your submission pass the unit tests?
• [No] Did you write documentation for your new model?
• [No] Have you formatted your code using the black deaults?

Changes to Existing Models

• [Yes] Have you added an explanation of what your changes do and why you'd like us to include them?
• [Yes] Have you written new tests for your changes, as applicable?
• [Yes] Have you successfully ran tests with your changes locally?