windows 10, cuda10, spacy100, python3.6

I tried to check the folder as https://github.com/explosion/thinc/issues/79#issuecomment-461230144 and did not see the cpython file.

Then I proceed to build gpu_ops by running "pip install --force-reinstall --no-binary :all: thinc-gpu-ops"

But still there is no cpython file being generated.

thinc_gpu_ops.AVAILABLE evaluates to False

Thank you, D

On Win10, thinc 6.12.0, spacy 2.0.16, cupy 4.1.0

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-2432a7701a48> in <module>()
      1 # defining doc
----> 2 doc = nlp("Jill laughed at John Johnson.")
      3 spacy.displacy.render(doc, style='dep', options={'distance' : 140}, jupyter=True)

E:\Anaconda3python\lib\site-packages\spacy\language.py in __call__(self, text, disable)
    344             if not hasattr(proc, '__call__'):
    345                 raise ValueError(Errors.E003.format(component=type(proc), name=name))
--> 346             doc = proc(doc)
    347             if doc is None:
    348                 raise ValueError(Errors.E005.format(name=name))

pipeline.pyx in spacy.pipeline.Tagger.__call__()

pipeline.pyx in spacy.pipeline.Tagger.predict()

E:\Anaconda3python\lib\site-packages\thinc\neural\_classes\model.py in __call__(self, x)
    159             Must match expected shape
    160         '''
--> 161         return self.predict(x)
    162 
    163     def pipe(self, stream, batch_size=128):

E:\Anaconda3python\lib\site-packages\thinc\api.py in predict(self, X)
     53     def predict(self, X):
     54         for layer in self._layers:
---> 55             X = layer(X)
     56         return X
     57 

E:\Anaconda3python\lib\site-packages\thinc\neural\_classes\model.py in __call__(self, x)
    159             Must match expected shape
    160         '''
--> 161         return self.predict(x)
    162 
    163     def pipe(self, stream, batch_size=128):

E:\Anaconda3python\lib\site-packages\thinc\api.py in predict(seqs_in)
    291     def predict(seqs_in):
    292         lengths = layer.ops.asarray([len(seq) for seq in seqs_in])
--> 293         X = layer(layer.ops.flatten(seqs_in, pad=pad))
    294         return layer.ops.unflatten(X, lengths, pad=pad)
    295 

E:\Anaconda3python\lib\site-packages\thinc\neural\_classes\model.py in __call__(self, x)
    159             Must match expected shape
    160         '''
--> 161         return self.predict(x)
    162 
    163     def pipe(self, stream, batch_size=128):

E:\Anaconda3python\lib\site-packages\thinc\api.py in predict(self, X)
     53     def predict(self, X):
     54         for layer in self._layers:
---> 55             X = layer(X)
     56         return X
     57 

E:\Anaconda3python\lib\site-packages\thinc\neural\_classes\model.py in __call__(self, x)
    159             Must match expected shape
    160         '''
--> 161         return self.predict(x)
    162 
    163     def pipe(self, stream, batch_size=128):

E:\Anaconda3python\lib\site-packages\thinc\neural\_classes\model.py in predict(self, X)
    123 
    124     def predict(self, X):
--> 125         y, _ = self.begin_update(X)
    126         return y
    127 

E:\Anaconda3python\lib\site-packages\thinc\api.py in uniqued_fwd(X, drop)
    372                                                     return_counts=True)
    373         X_uniq = layer.ops.xp.ascontiguousarray(X[ind])
--> 374         Y_uniq, bp_Y_uniq = layer.begin_update(X_uniq, drop=drop)
    375         Y = Y_uniq[inv].reshape((X.shape[0],) + Y_uniq.shape[1:])
    376         def uniqued_bwd(dY, sgd=None):

E:\Anaconda3python\lib\site-packages\thinc\api.py in begin_update(self, X, drop)
     59         callbacks = []
     60         for layer in self._layers:
---> 61             X, inc_layer_grad = layer.begin_update(X, drop=drop)
     62             callbacks.append(inc_layer_grad)
     63         def continue_update(gradient, sgd=None):

E:\Anaconda3python\lib\site-packages\thinc\api.py in begin_update(X, *a, **k)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in <listcomp>(.0)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in wrap(*args, **kwargs)
    256     '''
    257     def wrap(*args, **kwargs):
--> 258         output = func(*args, **kwargs)
    259         if splitter is None:
    260             to_keep, to_sink = output

E:\Anaconda3python\lib\site-packages\thinc\api.py in begin_update(X, *a, **k)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in <listcomp>(.0)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in wrap(*args, **kwargs)
    256     '''
    257     def wrap(*args, **kwargs):
--> 258         output = func(*args, **kwargs)
    259         if splitter is None:
    260             to_keep, to_sink = output

E:\Anaconda3python\lib\site-packages\thinc\api.py in begin_update(X, *a, **k)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in <listcomp>(.0)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in wrap(*args, **kwargs)
    256     '''
    257     def wrap(*args, **kwargs):
--> 258         output = func(*args, **kwargs)
    259         if splitter is None:
    260             to_keep, to_sink = output

E:\Anaconda3python\lib\site-packages\thinc\api.py in begin_update(X, *a, **k)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in <listcomp>(.0)
    174     def begin_update(X, *a, **k):
    175         forward, backward = split_backward(layers)
--> 176         values = [fwd(X, *a, **k) for fwd in forward]
    177 
    178         output = ops.xp.hstack(values)

E:\Anaconda3python\lib\site-packages\thinc\api.py in wrap(*args, **kwargs)
    256     '''
    257     def wrap(*args, **kwargs):
--> 258         output = func(*args, **kwargs)
    259         if splitter is None:
    260             to_keep, to_sink = output

E:\Anaconda3python\lib\site-packages\thinc\neural\_classes\hash_embed.py in begin_update(self, ids, drop)
     49         if ids.ndim >= 2:
     50             ids = self.ops.xp.ascontiguousarray(ids[:, self.column], dtype='uint64')
---> 51         keys = self.ops.hash(ids, self.seed) % self.nV
     52         vectors = self.vectors[keys].sum(axis=1)
     53         mask = self.ops.get_dropout_mask((vectors.shape[1],), drop)

ops.pyx in thinc.neural.ops.CupyOps.hash()

AttributeError: module 'thinc_gpu_ops' has no attribute 'hash'

The remap_ids layer in spaCy is never used as far as I can tell, but I am implementing a MultiEmbed version of MultiHashEmbed. This uses the more conventional Embed layer instead of the HashEmbed.

Before running Embed it runs remap_ids to convert token.i into positions in the embedding table: chain(remap_ids(), Embed().

To be able to use it as a replacement for MultiHashEmbed a couple modifications had to be made. HashEmbed takes a column attribute, because in spaCy the FeatureExtractor returns an Ints2d which is number of tokens x number of features. To be able to run the same idea with Embed the remap_ids now also implements column: chain(remap_ids(column=column), Embed(column=0)).

Furthermore in the common case where the mapping_table is a Dict[int, int] there was a bit of a strange error: whereas with numpy this works {3: 'q'}[np.array([3])[0]] with cupy we get:

----> 1 {3: 'q'}[cp.array([3])[0]]

TypeError: unhashable type: 'cupy._core.core.ndarray'

In the new version when we detect a cupy integer array we cast the elements to int.

Finally, the backprop used to return a [], but when the input is an xp array then this can lead to confusing errors e.g.: if the remap_ids is within a with_array block as is the case with Embed. Now it returns an empty array instead.

Introduce SparseLinear_v2 to fix indexing issues SparseLinear does not correctly index the gradient/weight matrix (#752). This change fixes the indexing, so that the full matrix is used.

To retain compatibility with existing models that use SparseLinear, which works relatively well if there are not too many hash collisions, the fixed version is renamed to SparseLinear_v2.

While at it, fix another indexing-related bug:

The output of MurMur hashes were mapped to array indices as follows:

idx = hash & (nr_weight-1)

This works well when nr_weight is a power of two. For instance, if we have 16 buckets:

idx = hash & 15
idx = hash & 0b1111

However, when the user uses a bucket count that is not a power of two, this breaks down. For instance, if we have 15 buckets:

idx = hash & 14
idx = hash & 0b1110

This would mask out all odd indices. We fix this by using the modulus instead. To preserve compatibility with existing models, this change is only added to SparseLinear_v2.

Behavior of this with numpyops seems wrong, as instead of giving an error it produces nans, as though it's accessing out of bounds memory or something. See https://github.com/explosion/spaCy/pull/9669.

Some more general issues / questions around this:

1. Do we want to add this kind of check for all cases where appropriate?
2. Is there a better place (like in numpyops) for this check or an equivalent check?
3. Other asserts in Thinc seem to not use messages, should the message here be removed?

error showed in terminal:
File "thinc/backends/numpy_ops.pyx", line 1, in init thinc.backends.numpy_ops ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

numpy with version 1.19.0 is installed in my server Could you please show me how to solve this problem ? thx!

The test suite for spacy-transformers breaks in my local environment since PR https://github.com/explosion/thinc/pull/686

numpy                             1.23.1
spacy                             3.4.1        (current master)
spacy-transformers                1.1.2        (current master)
thinc                             8.1.0        (current master)
torch                             1.9.0+cu111

e.g. when I run test_model_sequence_classification, it crashes at https://github.com/explosion/thinc/blob/master/thinc/util.py#L365:

>           torch_tensor = torch.utils.dlpack.from_dlpack(xp_tensor)
E           RuntimeError: from_dlpack received an invalid capsule. Note that DLTensor capsules can be consumed only once, so you might have already constructed a tensor from it once.

But, when I look at the code behind from_dlpack and run this directly (having confirmed my system doesn't hit the if statement checking the device):

dlpack = xp_tensor.__dlpack__()
torch_tensor = torch._C._from_dlpack(dlpack)

Then the test flies through.

Can anyone reproduce the failing tests on the spacy-transformers test suite with a relatively new numpy?

This PR makes the CategoricalCrossentropy loss more strict only allowing guesses and truths that represent exclusive classes and it fixes the computation of the loss value. It also changes all the tests that involve the CategoricalCrossentropy and SequenceCategoricalCrossentropy.

It also separates CategoricalCrossentropy from SparseCategoricalCrossentropy and CategoricalCrossentropy takes only Floats2d as truths whereas SparseCategoricalCrossentropy takes Union[Ints1d, Sequence[int], Sequence[str]].

It also adds a way of handling legacy modules in thinc.

I have referred to https://github.com/explosion/thinc/blob/master/examples/03_pos_tagger_basic_cnn.ipynb to get a better understanding of the thinc layers. The model is as follows in the example

model = strings2arrays() >> with_array(
        HashEmbed(nO=width, nV=vector_width, column=0)
        >> expand_window(window_size=1)
        >> ReLu(nO=width, nI=width * 3)
        >> ReLu(nO=width, nI=width)
        >> Softmax(nO=nr_classes, nI=width)
    )

Can someone please explain what a string2arrays does? The documentation says that it takes a sequence of sequence of string and produces a list[int2D].

The input X_train is something like [["this","is","awesome"],["thinc","is","cool"]]. What does the strings2arrays transform this example to? I am unable to wrap my head around what exactly strings2arrays does and how it transforms the input from a List[List] (2D lists) to List[Int2D] technically (3D lists/sequence).

Will leave this in draft mode until explosionbot is updated to support the Buildkite thinc slow test suite.

Requirements

• explosion/explosion-bot#3
• #672

• A bug was introduced in NumpyOps that caused the output pointer not to be incremented for zero-length sequences.
• When using uninitialized arrays, the sum or mean for a zero-length array was not correctly set to zero in Ops.

We should also do something about zero-length sequences for reduce_max. However, it's unclear what the corresponding which should be set to (since 0 is not a valid index for a zero-sized sequence). Maybe we should throw an exception when trying to apply reduce_max to a zero-length sequence?

This PR fixes the required maximum version of typing-extensions.

Currently it is bounded to <4.2.0: typing_extensions>=3.7.4.1,<4.2.0; python_version < "3.8"

This PR sets the upper bound to all compatible versions, until the next major release <5.0.0.

Required:

• [ ] https://github.com/explosion/confection/pull/20

See:

• https://github.com/explosion/spaCy/issues/12034

See issue in pydantic:

• https://github.com/pydantic/pydantic/issues/4885

See fixing PR in pydantic (typing-extensions>=4.2.0), which will be incompatible with your requirement typing_extensions>=3.7.4,<4.2.0; python_version < "3.8":

• https://github.com/pydantic/pydantic/pull/4886

I closed the old class-weights PR #614, because it was way outdated compared to the completely refactored v9 cross-entropy. This new PR adds the same functionality and a bit more flexibility.

All cross-entropy loss functions take a class_weights now, which is is used to compute sample_weights for each sample i.e.: the gradient and loss for each sample is multiplied by the class_weight corresponding to the target class.

The CategoricalCrossentropy can take class_weights as a Dict[int, float] or a Floats1d and if its a Dict it checks the types and converts to Floats1d. The SparseCategoricalCrossentropy in addition also allows for Dict[str, float] in which case it requires names to be provided and checks against them.

It also includes a convenience function to compute class_weights based on a commonly used formula e.g. also used in sklearn. I also went for a dynamic approach here where it takes label_data: Dict[Hashable, int], where the keys are the labels and values are the label counts. It returns Dict[Hashable, float] the weight for each label.

Its not expected for class_weights to always be useful, but can be helpful sometimes when there is label imbalance and the relevant performance metric is balance sensitive. I think it makes spaCy a more complete library if we can engage this functionality.

The smooth_one_hot function in thinc.util is being called in the CategoricalCrossentropy to apply label smoothing. This PR fixes the check for valid label-smoothing parameter, which was fixed before for to_categorical, but not for smooth_one_hot. Also added a test for smooth_one_hot which was also missing.

The idea of the with_flatten layer is that it flattens a nested sequence, passes it to the wrapped layer, and then unflattens the output of the wrapped layer.

However, the layer was asymmetric in that it passes a list to the wrapped layer, but expects back an XP array. This breaks composition with other Thinc layers, such as with_array.

This change makes with_flatten symmetric, in that the inputs/outputs of the with_flatten and the wrapped layer are symmetric.

It seems that this layer is not used in Thinc or spaCy, so maybe it never worked correctly? At any rate, I needed to flatten a nested list in distillation with with_flatten(with_array(...)) in distillation and found that it doesn't actually work.

This PR is an attempted fix of the issue outlined in https://github.com/explosion/spaCy/issues/11968. I don't completely understand how the resizable textcat is supposed to work yet, and am not sure if this is the right approach, but locally it got rid of errors and I was able to train a model with reasonable performance.

There are also no tests yet.