Hi,

When getting data from get-data-xnli.sh , I notice that most dataset is not in French. Hence, I wonder how you used that in practice?

I am currently looking for a some NLI-like and STS-like datasets in French. That would be great to fine-tune Flaubert!

As a suggestion, translation the English version of NLI and STS to French could be a good option to fine tune Flaubert on such tasks.

Good afternoon,

I tried to follow your instructions to train my own corpus with Flaubert in order to get a model to use for my classification task but I am having trouble understanding the procedure ?

You said we should use this line to train on our preprocessed data :

/Flaubert$ python train.py --exp_name flaubert_base_lower --dump_path ./dumped/ --data_path ./own_data/data/ --lgs 'fr' --clm_steps '' --mlm_steps 'fr' --emb_dim 768 --n_layers 12 --n_heads 12 --dropout 0.1 --attention_dropout 0.1 --gelu_activation true --batch_size 16 --bptt 512 --optimizer "adam_inverse_sqrt,lr=0.0006,warmup_updates=24000,beta1=0.9,beta2=0.98,weight_decay=0.01,eps=0.000001" --epoch_size 300000 --max_epoch 100000 --validation_metrics _valid_fr_mlm_ppl --stopping_criterion _valid_fr_mlm_ppl,20 --fp16 true --accumulate_gradients 16 --word_mask_keep_rand '0.8,0.1,0.1' --word_pred '0.15'

I tried it after cloning flaubert and all the necessary librairies but I am getting this error :

FAISS library was not found. FAISS not available. Switching to standard nearest neighbors search implementation. ./own_data/data/train.fr.pth not found ./own_data/data/valid.fr.pth not found ./own_data/data/test.fr.pth not found Traceback (most recent call last): File "train.py", line 387, in check_data_params(params) File "/ho/ge/ke/eXP/Flaubert/xlm/data/loader.py", line 302, in check_data_params assert all([all([os.path.isfile(p) for p in paths.values()]) for paths in params.mono_dataset.values()]) AssertionError

Does this mean I have to split my own data in three corpus after preprocessing it ? (train , valid and test ??) Should I use your preprossed script on my own before executing the command ?

Hello!

I am using FlauBERT to generate word embeddings as part of a study on word sense disambiguation (WSD).

The FlauBERT tokenizer does not recognize a significant number of words in my corpus and, as a result, segments them. For example, the FlauBERT tokenizer does not recognize the archaic orthography of some verbs. It also does not recognize plural forms of a number of other words in the corpus. As a result, the tokenizer segments a number of words in the corpus, some of which are significant to my research.

I understand that I could further train FlauBERT on a corpus of eighteenth-century French in order to create a new model and a new tokenizer specifically for eighteenth-century French. However, compiling a corpus of eighteenth-century French that is large and heterogeneous enough to be useful would be challenging (and perhaps not even possible).

As an alternative to training a new model, I thought I might lemmatize my corpus before running it through FlauBERT. Stanford's Stanza NLP package (stanfordnlp.github.io/stanza/) recognizes the archaic orthography of the verbs in my corpus and turns them into the infinitive form, a form FlauBERT recognizes. Similarly, Stanza also changes the plural forms of other words into singular forms, forms FlauBERT also recognizes. Thus, if I were to lemmatize my corpus in Stanza, the FlauBERT tokenizer would then be able to recognize substantially more words in my corpus.

Would lemmatizing my corpus in this way adversely affect my FlauBERT results and a WSD analysis in particular? In general, does lemmatization have a negative effect on BERT results and WSD analyses more particularly?

Given that FlauBERT is not trained on lemmatized text, I imagine that lemmatizing the corpus would indeed negatively effect the results of the analysis. As an alternative to training FlauBERT on a corpus of eighteenth-century French (which may not be possible), could I instead train it on a corpus of lemmatized French and then use this new model for a WSD analysis on my corpus of lemmatized eighteenth-century French? Would that work?

I'm not sure if this is the right place for these sorts of questions!

Thank you in advance for your time.

Hi I got this error when running the run_flue.py from transformers import flue_compute_metrics as compute_metrics Import Error: cannot import name 'flue_compute_metrics'

I already preinstalled the requirements and update the transformer directory.

Hi,

Hope you are all well !

Is it possible with Flaubert to do some tweet sentiment analysis written in french ? If so, how can we do that ?

Vive la France ! :-)

Cheers, X

Transformers version 2.3.0 Pytorch version 1.4

When running readme.md code

import torch
from transformers import XLMModel, XLMTokenizer
modelname="xlm_bert_fra_base_lower" # Or absolute path to where you put the folder

# Load model
flaubert, log = XLMModel.from_pretrained(modelname, output_loading_info=True)
# check import was successful, the dictionary should have empty lists as values
print(log)

# Load tokenizer
flaubert_tokenizer = XLMTokenizer.from_pretrained(modelname, do_lowercase_and_remove_accent=False)

sentence="Le chat mange une pomme."
sentence_lower = sentence.lower()

token_ids = torch.tensor([flaubert_tokenizer.encode(sentence_lower)])
last_layer = flaubert(token_ids)[0]
print(last_layer.shape)

Output

OSError: Model name 'xlm_bert_fra_base_lower' was not found in model name list 
(xlm-mlm-en-2048, xlm-mlm-ende-1024, xlm-mlm-enfr-1024, xlm-mlm-enro-1024, xlm-mlm-tlm-xnli15-1024, xlm-mlm-xnli15-1024, xlm-clm-enfr-1024, xlm-clm-ende-1024,
 xlm-mlm-17-1280, xlm-mlm-100-1280). We assumed 'xlm_bert_fra_base_lower' 
was a path or url to a configuration file named config.json 
or a directory containing such a file but couldn't find any such file at this path or url.

Tried downloading the model as tar file (lower and normal), extracting it and putting its absolute folder path as modelname but keeps getting the same error....

I may be missing something so stupid but I don't see a config.json file in the archive file

What is wrong?

Hi I have tried to download Flaubert as described:

import torch
from transformers import FlaubertModel, FlaubertTokenizer

Unfortunately it returns an ImportError:

  from transformers import FlaubertModel, FlaubertTokenizer
ImportError: cannot import name 'FlaubertModel'

The script failed with : No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK? I managed to fix it with : sudo update-alternatives --config java and selecting /usr/lib/jvm/java-11-openjdk-amd64/bin/java as default

Bonjour bonjour ! Merci d'avoir partagé le modèle !

Dans Camembert il est assez facile de deviner un mot à partir du contexte, y a-t-il un working example dans Flaubert ?

Merci d'avance !

from fairseq.models.roberta import CamembertModel
camembert = CamembertModel.from_pretrained('./camembert-base/')
camembert.eval()
masked_line = 'Le camembert est <mask> :)'
camembert.fill_mask(masked_line, topk=3)
# [('Le camembert est délicieux :)', 0.4909118115901947, ' délicieux'),
]#  ('Le camembert est excellent :)', 0.10556942224502563, ' excellent'),
#  ('Le camembert est succulent :)', 0.03453322499990463, ' succulent')]

Environment info

Environment info

• transformers version: 2.5.1
• Platform: linux
• Python version: 3.7
• PyTorch version (GPU?): 1.4
• Tensorflow version (GPU?):
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help

Model I am using (FlauBert):

The problem arises when trying to produce features with the model, the output which is generated causes the system run out of memory.

• [ ] the official example scripts: (I did not change much , pretty close to the original)

import torch
from transformers import FlaubertModel, FlaubertTokenizer
# Choose among ['flaubert/flaubert_small_cased', 'flaubert/flaubert_base_uncased', 
#               'flaubert/flaubert_base_cased', 'flaubert/flaubert_large_cased']
modelname = 'flaubert/flaubert_base_cased' 

# Load pretrained model and tokenizer
flaubert, log = FlaubertModel.from_pretrained(modelname, output_loading_info=True)
flaubert_tokenizer = FlaubertTokenizer.from_pretrained(modelname, do_lowercase=False)
# do_lowercase=False if using cased models, True if using uncased ones

sentence = "Le chat mange une pomme."
token_ids = torch.tensor([flaubert_tokenizer.encode(sentence)])

last_layer = flaubert(token_ids)[0]
print(last_layer.shape)
# torch.Size([1, 8, 768])  -> (batch size x number of tokens x embedding dimension)

# The BERT [CLS] token correspond to the first hidden state of the last layer
cls_embedding = last_layer[:, 0, :]

• [ ] My own modified scripts: (give details below)

def get_flaubert_layer(texte):

	modelname = "flaubert-base-uncased"
	path = './flau/flaubert-base-unc/'

	flaubert = FlaubertModel.from_pretrained(path)
	flaubert_tokenizer = FlaubertTokenizer.from_pretrained(path)
	tokenized = texte.apply((lambda x: flaubert_tokenizer.encode(x, add_special_tokens=True, max_length=512)))
	max_len = 0
	for i in tokenized.values:
		if len(i) > max_len:
			max_len = len(i)
	padded = np.array([i + [0] * (max_len - len(i)) for i in tokenized.values])
	token_ids = torch.tensor(padded)
	with torch.no_grad():
		last_layer = flaubert(token_ids)[0][:,0,:].numpy()
		
	return last_layer, modelname

The tasks I am working on is:

• [ ] Producing vectors/features from a language model and pass it to others classifiers

To reproduce

Steps to reproduce the behavior:

1. Get transformers library and scikit-learn, pandas and numpy, pytorch
2. Last lines of code

# Reading the file 
filename = "corpus"
sentences = pd.read_excel(os.path.join(root, filename + ".xlsx"), sheet_name= 0)
data_id = sentences.identifiant
print("Total phrases: ", len(data_id))
data = sentences.sent
label = sentences.etiquette
emb, mdlname = get_flaubert_layer(data)  # corpus is dataframe of approximately 40 000 lines

Apperently this line produce something which is huge and which take a lot memory : last_layer = flaubert(token_ids)[0][:,0,:].numpy()

I would have expected it run but I think the fact that I pass the whole dataset to the model is causing the system to break, so I wanted to know if it possible to tell the model to process the data set maybe 500 lines or 1000 lines at at a time so as to not pass the whole dataset. I know that , there is this parameter : batch_size which can be used but since I am not training a model but merely using it to produces embeddings as input for others classifiers , Do you perhaps know how to modify the batch size so the whole dataset is not treated. I am not really familiar with this type of architecture. In the example , they just put one single sentence but in my case I load a whole dataset (dataframe). ?

My expectation is to make the model treat all the sentences and then produced the vectors I need for the task of classification.

Hi,

My goal here is to do clustering on sentences. For this purpose, I chose to use similarities between sentence embedding for all my sentences. Unfortunately, camemBERT seems not great for that task and fine-tuning flauBERT could be a solution.

So thanks to @formiel, I managed to fine tune flauBERT on an NLI dataset. My question is about that fine-tuning. What is the output exactly ? I only got a few files in the dump_path:

• train.log ==> logs of the training
• params.pkl ==> parameters of the training
• test.pred.0 ==> prediction of the test dataset after first epoch
• valid.pred.0 ==> valid classification of the test dataset after first epoch
• test.pred.1 ==> etc

I wonder if after fine-tuning flauBERT, i could use it to make a new embedding of a sentence (like flauBERT before fine-tuning). So where is the new flauBERT model trained on the NLI dataset ? And how use it to make embeddings ?

Thanks in advance

Bonjour, Dans la documentation de Flaubert, il est fait mention de la possibilité de paraphraser. Auriez-vous un exemple de code à utiliser pour, par exemple, d'une phrase en générer une dizaines de similaires ?

Merci

Hello,

I have a problem currently regarding the finetuning of Flaubert on FLUE. I have a model that I re-trained with custom data, so I have new weights for it, and I currently have it as .json file and .bin files. However, when I want to finetune this model on FLUE tasks, they ask me for vocab and codes files from pretraining, that I don't have when using the hugging face library. I see that there is a module to go from XLM to hugging_face, but not the opposite. Is it possible to transform a model under .json and .bin format to get vocab, codes and .pth files ?

Or maybe there is a clever workaround to this problem ?

Many thanks in advance

Hello. :)

I would like to use the "--reload_model" option with your train.py command to further train one of your pretrained FlauBERT models.

Upon trying to run train.py with the "--reload_model" option I got an error message saying that there was a "size mismatch" between the pretrained FlauBERT model and the adapted model I was trying to train.

The error message referred to a "shape torch.Size([67542]) from checkpoint". This was for the flaubert_base_uncased model. I assume that the number 67542 is the vocabulary size of flaubert-base-uncased.

In order to use the "--reload_model" option with your pretrained FlauBERT models, do I need to ensure that the vocabulary of my training data is identical to that of the pretrained model? If so, do you think that I could manage that simply by concatenating the "vocab" file of the pretrained model with my training data?

Thank you in advance for your help!

Hi ! I would like to finetune Flaubert on FLUE task with hugging face library. I downloaded the PAWS datas and use the code you gave on your github repo but I have this error message I can't go through :

any idea on what to do ?

Thanks for this project by the way, him looking forward to use it !

Have a good day, Lisa

I have a query regarding regarding your training corpus.

The News Crawl corpora that you use are both shuffled and de-duplicated. However, the corpora used by other models like BERT, RoBERTa etc. use a non-shuffled corpus where each document within the corpus is also demarcated with an empty line. Now with this un-shuffled form, when you create pre-training instances, you will end up with contiguous sentences in segment A and segment B. But in your case, the segments will contain non-contiguous sentences right?

So my question is what is your opinion on having non-contiguous sentences in the segments? Does it hurt the performance of MLM, or downstream tasks?