cd venv/bin
./pip install -r ../../requirements.txt
./pip install deepspeed==0.3.13
Example with GPT-Neo-2.7B with DeepSpeed
Example with GPT-Neo-1.3B without DeepSpeed
DeepSpeed configuration
Training and testing log
Hardware: Rtx 3090, Ryzen 3600, 64 GB of RAM.
I am trying to train the 1.3B parameter model on a custom dataset. This model training takes more memory due to the longer input(or so it seems, it def takes more memory). Thus I am trying to use DeepSpeed. I have changed nothing other than using the smaller 1.3B model and reducing the batch size to 4.
An issue I am having is that the loss(I think its the loss) is overflowing. I know this is due to using mixed or half-precision in order to reduce memory usage. When training on the provided dataset, this is not an issue. The provided dataset does initially have the overflow issue, but it is quickly resolved through internal adjustments. Is there some configuration change I can make so that this custom dataset will work without overflowing?
Below are some logs, you can see that it still is overflowing even with the loss scale at 1.
python gpt_neo_xl_deepspeed.py
Special tokens have been added in the vocabulary, make sure the associated word embedding are fine-tuned or trained.
Max length: 384
[2021-06-08 12:02:15,302] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
[2021-06-08 12:02:15,601] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+12a53b4, git-hash=12a53b4, git-branch=HEAD
[2021-06-08 12:02:15,622] [INFO] [engine.py:77:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1
Adam Optimizer #0 is created with scalar arithmetic capability.
Config: alpha=0.000050, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1
[2021-06-08 12:02:17,843] [INFO] [engine.py:602:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2021-06-08 12:02:17,843] [INFO] [engine.py:606:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2021-06-08 12:02:17,843] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer
Initializing ZeRO Stage 3
[2021-06-08 12:02:17,844] [WARNING] [stage3.py:35:
I have tried fine tunning this 2.7B parameter model with my RTX 3090 and with 64 GB of RAM. Looking at system resources, I am exhausting all of my RAM before the program is killed. My question is what hardware was used in this repo? Specifically, how much RAM is required to train the 2.7B model?
Hi! Thank you for your great project.
I am following your example. But, I found small bug in tokenizer.
See below.
>>> tokenizer = GPT2Tokenizer.from_pretrained(
... "EleutherAI/gpt-neo-1.3B",
... bos_token="<|startoftext|>",
... eos_token="<|endoftext|>",
... pad_token="<|pad|>",
... )
>>> tokenizer("<|startoftext|>")
{'input_ids': [27, 91, 9688, 1659, 5239, 91, 29], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}
>>> tokenizer("<|pad|>")
{'input_ids': [50257], 'attention_mask': [1]}
>>> tokenizer("<|endoftext|>")
{'input_ids': [50256], 'attention_mask': [1]}
Just a quick question, is anyone aware of how much GPU memory is required to train these models? I'm on Kaggle with a P100(16GB) and I can't see to to call .train() without running out memory on any of the 3 models available.
I've tried with the 2.7B, 1.3B and 125M param models and I get the same result with all 3, surely a P100 can handle the 125M model. 🤔
I might order some more RAM for my home server and try this again on a CPU, I saw from the other post here it looks like I'll need at least 75GB which isn't bad at all, plus it gives me an excuse to upgrade it haha.
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 15.90 GiB total capacity; 14.71 GiB already allocated; 77.75 MiB free; 14.95 GiB reserved in total by PyTorch)
it looks like users try to run this example on torch.distributed with multiple gpus and of course it fails.
So I'm proposing to please at least add a note to remove the torch.distributed emulation hack for when multi-gpu setup is used.
Please feel free to edit the wording to your liking
Thank you!
Prreviously I was able to run the model, but getting loss overflows for any custom data or running out of RAM, but it seemed to work for the Netflix dataset. I made a new conda environment and now I get a point where some Vram is allocated before it just freezes at this point.
python gpt_neo_xl_deepspeed.py Special tokens have been added in the vocabulary, make sure the associated word embedding are fine-tuned or trained. Max length: 384 [2021-06-08 10:46:33,763] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl [2021-06-08 10:46:33,899] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13, git-hash=unknown, git-branch=unknown [2021-06-08 10:46:34,019] [INFO] [engine.py:77:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1 Using /home/user/.cache/torch_extensions as PyTorch extensions root...
I feel like this may be due to environment issues, perhaps cudatoolkit since that is not specified. What cudatoolkit was used? Does anyone have any idea what the issue may be? See below for my env:
name: gpt_neo_train channels:
This is awesome! Thank you for showing the VRAM to show training requirements and speed. Only one thing I didn't find: Do you remember the max_length of the tokenized Netflix titles?
max_length = max([len(tokenizer.encode(description)) for description in descriptions])
Trainer(model=model, args=training_args, train_dataset=train_dataset, eval_dataset=val_dataset, data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]), 'attention_mask': torch.stack([f[1] for f in data]), 'labels': torch.stack([f[0] for f in data])}).train()
CalledProcessError Traceback (most recent call last) File /opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/utils/cpp_extension.py:1900, in _run_ninja_build(build_directory, verbose, error_prefix) 1899 stdout_fileno = 1 -> 1900 subprocess.run( 1901 command, 1902 stdout=stdout_fileno if verbose else subprocess.PIPE, 1903 stderr=subprocess.STDOUT, 1904 cwd=build_directory, 1905 check=True, 1906 env=env) 1907 except subprocess.CalledProcessError as e: 1908 # Python 2 and 3 compatible way of getting the error object.
File /opt/conda/envs/pytorch/lib/python3.9/subprocess.py:528, in run(input, capture_output, timeout, check, *popenargs, **kwargs) 527 if check and retcode: --> 528 raise CalledProcessError(retcode, process.args, 529 output=stdout, stderr=stderr) 530 return CompletedProcess(process.args, retcode, stdout, stderr)
CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last) Cell In [10], line 1 ----> 1 Trainer(model=model, args=training_args, train_dataset=train_dataset, 2 eval_dataset=val_dataset, data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]), 3 'attention_mask': torch.stack([f[1] for f in data]), 4 'labels': torch.stack([f[0] for f in data])}).train()
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/transformers/trainer.py:1527, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs) 1522 self.model_wrapped = self.model 1524 inner_training_loop = find_executable_batch_size( 1525 self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size 1526 ) -> 1527 return inner_training_loop( 1528 args=args, 1529 resume_from_checkpoint=resume_from_checkpoint, 1530 trial=trial, 1531 ignore_keys_for_eval=ignore_keys_for_eval, 1532 )
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/transformers/trainer.py:1596, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval) 1589 delay_optimizer_creation = ( 1590 self.sharded_ddp is not None 1591 and self.sharded_ddp != ShardedDDPOption.SIMPLE 1592 or is_sagemaker_mp_enabled() 1593 or self.fsdp is not None 1594 ) 1595 if args.deepspeed: -> 1596 deepspeed_engine, optimizer, lr_scheduler = deepspeed_init( 1597 self, num_training_steps=max_steps, resume_from_checkpoint=resume_from_checkpoint 1598 ) 1599 self.model = deepspeed_engine.module 1600 self.model_wrapped = deepspeed_engine
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/transformers/deepspeed.py:344, in deepspeed_init(trainer, num_training_steps, resume_from_checkpoint, inference) 333 # keep for quick debug: 334 # from pprint import pprint; pprint(config) 336 kwargs = dict( 337 model=model, 338 model_parameters=model_parameters, (...) 341 lr_scheduler=lr_scheduler, 342 ) --> 344 deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs) 346 if resume_from_checkpoint is not None: 347 348 # it's possible that the user is trying to resume from model_path, which doesn't necessarily 349 # contain a deepspeed checkpoint. e.g. examples just check if the dir exists and assume it's 350 # a resume from a checkpoint and not just a local pretrained weight. So we check here if the 351 # path contains what looks like a deepspeed checkpoint 352 import glob
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/deepspeed/init.py:125, in initialize(args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config, config_params) 122 assert model is not None, "deepspeed.initialize requires a model" 124 if not isinstance(model, PipelineModule): --> 125 engine = DeepSpeedEngine(args=args, 126 model=model, 127 optimizer=optimizer, 128 model_parameters=model_parameters, 129 training_data=training_data, 130 lr_scheduler=lr_scheduler, 131 mpu=mpu, 132 dist_init_required=dist_init_required, 133 collate_fn=collate_fn, 134 config=config, 135 config_params=config_params) 136 else: 137 assert mpu is None, "mpu must be None with pipeline parallelism"
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/deepspeed/runtime/engine.py:330, in DeepSpeedEngine.init(self, args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config, config_params, dont_change_device) 327 model_parameters = self.module.parameters() 329 if has_optimizer: --> 330 self._configure_optimizer(optimizer, model_parameters) 331 self._configure_lr_scheduler(lr_scheduler) 332 self._report_progress(0)
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/deepspeed/runtime/engine.py:1195, in DeepSpeedEngine._configure_optimizer(self, client_optimizer, model_parameters) 1193 log_dist('Using client callable to create basic optimizer', ranks=[0]) 1194 else: -> 1195 basic_optimizer = self._configure_basic_optimizer(model_parameters) 1196 log_dist( 1197 f"Using DeepSpeed Optimizer param name {self.optimizer_name()} as basic optimizer", 1198 ranks=[0]) 1200 self._check_for_duplicates(basic_optimizer)
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/deepspeed/runtime/engine.py:1266, in DeepSpeedEngine._configure_basic_optimizer(self, model_parameters) 1264 else: 1265 from deepspeed.ops.adam import DeepSpeedCPUAdam -> 1266 optimizer = DeepSpeedCPUAdam(model_parameters, 1267 **optimizer_parameters, 1268 adamw_mode=effective_adam_w_mode) 1269 else: 1270 from deepspeed.ops.adam import FusedAdam
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py:94, in DeepSpeedCPUAdam.init(self, model_params, lr, bias_correction, betas, eps, weight_decay, amsgrad, adamw_mode, fp32_optimizer_states) 92 self.adam_w_mode = adamw_mode 93 self.fp32_optimizer_states = fp32_optimizer_states ---> 94 self.ds_opt_adam = CPUAdamBuilder().load() 96 self.ds_opt_adam.create_adam(self.opt_id, 97 lr, 98 betas[0], (...) 102 adamw_mode, 103 should_log_le("info"))
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py:460, in OpBuilder.load(self, verbose) 458 return importlib.import_module(self.absolute_name()) 459 else: --> 460 return self.jit_load(verbose)
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py:495, in OpBuilder.jit_load(self, verbose) 492 torch_arch_list = os.environ.get("TORCH_CUDA_ARCH_LIST") 493 os.environ["TORCH_CUDA_ARCH_LIST"] = "" --> 495 op_module = load( 496 name=self.name, 497 sources=self.strip_empty_entries(sources), 498 extra_include_paths=self.strip_empty_entries(extra_include_paths), 499 extra_cflags=self.strip_empty_entries(self.cxx_args()), 500 extra_cuda_cflags=self.strip_empty_entries(self.nvcc_args()), 501 extra_ldflags=self.strip_empty_entries(self.extra_ldflags()), 502 verbose=verbose) 503 build_duration = time.time() - start_build 504 if verbose:
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/utils/cpp_extension.py:1284, in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates) 1192 def load(name, 1193 sources: Union[str, List[str]], 1194 extra_cflags=None, (...) 1202 is_standalone=False, 1203 keep_intermediates=True): 1204 r''' 1205 Loads a PyTorch C++ extension just-in-time (JIT). 1206 (...) 1282 ... verbose=True) 1283 ''' -> 1284 return _jit_compile( 1285 name, 1286 [sources] if isinstance(sources, str) else sources, 1287 extra_cflags, 1288 extra_cuda_cflags, 1289 extra_ldflags, 1290 extra_include_paths, 1291 build_directory or _get_build_directory(name, verbose), 1292 verbose, 1293 with_cuda, 1294 is_python_module, 1295 is_standalone, 1296 keep_intermediates=keep_intermediates)
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/utils/cpp_extension.py:1508, in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates) 1504 hipified_sources.add(hipify_result[s_abs]["hipified_path"] if s_abs in hipify_result else s_abs) 1506 sources = list(hipified_sources) -> 1508 _write_ninja_file_and_build_library( 1509 name=name, 1510 sources=sources, 1511 extra_cflags=extra_cflags or [], 1512 extra_cuda_cflags=extra_cuda_cflags or [], 1513 extra_ldflags=extra_ldflags or [], 1514 extra_include_paths=extra_include_paths or [], 1515 build_directory=build_directory, 1516 verbose=verbose, 1517 with_cuda=with_cuda, 1518 is_standalone=is_standalone) 1519 finally: 1520 baton.release()
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/utils/cpp_extension.py:1623, in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone) 1621 if verbose: 1622 print(f'Building extension module {name}...', file=sys.stderr) -> 1623 _run_ninja_build( 1624 build_directory, 1625 verbose, 1626 error_prefix=f"Error building extension '{name}'")
File /opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/utils/cpp_extension.py:1916, in _run_ninja_build(build_directory, verbose, error_prefix) 1914 if hasattr(error, 'output') and error.output: # type: ignore[union-attr] 1915 message += f": {error.output.decode(*SUBPROCESS_DECODE_ARGS)}" # type: ignore[union-attr] -> 1916 raise RuntimeError(message) from e
RuntimeError: Error building extension 'cpu_adam'
When replicating the code Deepspeed gets stuck with
[2021-06-29 14:29:44,757] [INFO] [utils.py:13:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1
Any ideas on how to fix this?
If I want to save and run generation on the model later on, I assume I do something like this:
After training:
tokenizer.save_pretrained('./results/')
Later generation:
weights = "./results/"
tokenizer = GPT2Tokenizer.from_pretrained(weights, bos_token='<|startoftext|>', eos_token='<|endoftext|>', pad_token='<|pad|>')
model = GPTNeoForCausalLM.from_pretrained('./results/checkpoint-90/').cuda()
But I get an error about the size not being right. Any idea why?
File "generate.py", line 12, in <module>
model = GPTNeoForCausalLM.from_pretrained('./results/checkpoint-90').cuda()
File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1183, in from_pretrained
raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for GPTNeoForCausalLM:
size mismatch for transformer.wte.weight: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([50258, 768]).
Fine-tune pretrained Convolutional Neural Networks with PyTorch. Features Gives access to the most popular CNN architectures pretrained on ImageNet. A
GPT-Code-Clippy (GPT-CC) is an open source version of GitHub Copilot, a language model -- based on GPT-3, called GPT-Codex -- that is fine-tuned on publicly available code from GitHub.
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
GPT - gMLP This repository will attempt to crack long context autoregressive language modeling (GPT) using variations of gMLPs. Specifically, it will
MLP GPT - Jax (wip) A GPT, made only of MLPs, in Jax. The specific MLP to be used are gMLPs with the Spatial Gating Units. Working Pytorch implementat
"# SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING" i
Differentiable Model Compression via Pseudo Quantization Noise DiffQ performs differentiable quantization using pseudo quantization noise. It can auto
WSDEC This is the official repo for our NeurIPS paper Weakly Supervised Dense Event Captioning in Videos. Description Repo directories ./: global conf
emoji2vec This is my attempt to train, visualize and evaluate emoji embeddings as presented by Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko
TEMOS: TExt to MOtionS Generating diverse human motions from textual descriptions Description Official PyTorch implementation of the paper "TEMOS: Gen
CLIP Guided Diffusion From RiversHaveWings. Generate vibrant and detailed images using only text. See captions and more generations in the Gallery See
SparseML is a toolkit that includes APIs, CLIs, scripts and libraries that apply state-of-the-art sparsification algorithms such as pruning and quantization to any neural network. General, recipe-driven approaches built around these algorithms enable the simplification of creating faster and smaller models for the ML performance community at large.
sequitur sequitur is a library that lets you create and train an autoencoder for sequential data in just two lines of code. It implements three differ
BudgetML is perfect for practitioners who would like to quickly deploy their models to an endpoint, but not waste a lot of time, money, and effort trying to figure out how to do this end-to-end.
Train neural network for semantic segmentation (deep lab V3) with pytorch in 50 lines of code Train net semantic segmentation net using Trans10K datas
Install with: pip install shimoku-api-python Start with: from os import getenv import shimoku_api_python.client as Shimoku
Python wrapper class for OpenVINO Model Server. User can submit inference request to OVMS with just a few lines of code.
Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor
ChatBot-Pytorch A GPT-2 ChatBot implemented using Pytorch and Huggingface-transf
694 Nov 23, 2022
2.3k Jan 9, 2023
8.4k Jan 1, 2023
80 Dec 1, 2022
28 Dec 12, 2022
145 Dec 30, 2022
96 Nov 1, 2022
17 Sep 3, 2022
187 Dec 27, 2022
401 Dec 28, 2022
1.5k Dec 30, 2022
305 Dec 21, 2022
1.3k Dec 25, 2022
17 Dec 19, 2022
5 Nov 7, 2022
7 Jul 27, 2022
27 Dec 20, 2022
42 Dec 9, 2022