How can we mask/pad tokens for sequences of varied length ? When we use fft along dimension (-2) , sequences , if we just use zero padding the result will be skewed. torch.fft.fft(torch.fft.fft(hidden_states.float(), dim=-1), dim=-2).real

Dear @rishikksh20 ,

Thanks for your implementation! Can I ask what do you think about the effectiveness of Furier features as well as FNet when it's used to replace multi-head attention in TTS encoders which is quite a concrete approach so far! I'm thinking about if it can help to reduce the complexity of the TTS models but still maintain good performance :thinking:

Thanks!

Max

Hi, thanks for a great implementation!

I wanted to clarify one thing that mismatches with the code, proposed in the article itself. In your code, you pre-normalize inputs, so that they are passed through LayerNorm before FFT. In the code, presented in the article, they have:

class FNetEncoderBlock ( nn . Module ) :
30 f o u r i e r _ l a y e r : Fou rie rT ran sfo rmLa ye r
31 f f _ l a y e r : FeedForwardLayer
32
33 @nn. compact
34 def _ _ c a l l _ _ ( s e l f , x , d e t e r m i n i s t i c ) :
35 m i x i n g _ o ut p ut = s e l f . f o u r i e r _ l a y e r ( x )
36 x = nn . LayerNorm (1 e−12 , name=" mixing_laye r_no rm " ) ( x + &
m i x i n g _ o ut p ut )
37 fe ed _fo rw a rd _o utp ut = s e l f . f f _ l a y e r ( x , d e t e r m i n i s t i c )
38 r e t u r n nn . LayerNorm (
39 1e−12 , name=" output_la ye r_no rm " ) ( x + fee d_fo rwa rd _outp ut )

which in my view is done in the opposite order. Am I mistaken or is it indeed a bug?

thanks for your impl, and I found that you may not code emdedding part, I think the part is important too.

I suggest that you could write a demo from input to prediction if you have free time, excespecially how to code embedding part.