Text-to-Image generation

/content/CogView Generate Samples WARNING: No training data specified using world size: 1 and model-parallel size: 1

using dynamic loss scaling initializing model parallel with size 1 initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234 padded vocab (size: 58219) with 21 dummy tokens (new size: 58240) prepare tokenizer done building CogView2 model ... number of parameters on model parallel rank 0: 3928849920 current device: 0 tcmalloc: large alloc 7881007104 bytes == 0x5637e3fb2000 @ 0x7f61e428db6b 0x7f61e42ad379 0x7f6171f1e25e 0x7f6171f1f9d2 0x7f61aff48e7d 0x7f61c0b43120 0x7f61c0781bd9 0x5637152088a8 0x56371527bfd5 0x5637152767ad 0x5637152093ea 0x5637152773b5 0x5637152767ad 0x563715209003 0x563715208b09 0x56371535028d 0x5637152bf1db 0x563715207bb1 0x5637152f8fed 0x56371527b988 0x5637152767ad 0x563715148e2c 0x563715278bb5 0x5637152764ae 0x5637152093ea 0x56371527832a 0x56371520930a 0x5637152773b5 0x56371520930a 0x5637152773b5 0x5637152764ae Load model file pretrained/cogview/cogview-sr/20000/mp_rank_00_model_states.pt Working on No. 0 on 0... Traceback (most recent call last): File "generate_samples.py", line 326, in main() File "generate_samples.py", line 323, in main generate_images_continually(model, args) File "generate_samples.py", line 215, in generate_images_continually for raw_text, seq, output_path in get_context(args, query_template): File "generate_samples.py", line 132, in get_context seq = _parse_and_to_tensor(raw_text, img_size=img_size, query_template=query_template) File "generate_samples.py", line 70, in _parse_and_to_tensor text = query_template.format(*text.split('\t')) IndexError: tuple index out of range /content

Good afternoon,

I tried to register on wudaoai and put in my information so I could download the model, but Wudaoai did not accept my phone number, probably because I am from Brazil. I had to look for a Chinese phone number online to apply, but I doubt they will allow me to download the model because of this. What can I do?

Hi, How do you get "26.0" FID on mscoco using DM-GAN? Because the official result reported in https://github.com/MinfengZhu/DM-GAN is 26.55. I ran DM-GAN myself and managed to get a similar result(26.54), instead of "26.0".

I found a colab file reference in a closed issue. The last cell (inference) shows this error. Does anyone know of a solution or a more recent Colab noteboook? The web-based version of Cogview works but takes a while to process queued requests. I used the "insert code" icon when editing this and for some reason, the output is all connected together making it unreadable.

/content/CogView Traceback (most recent call last): File "generate_samples.py", line 28, in <module> from utils import Timers File "/content/CogView/utils.py", line 25, in <module> from fp16 import FP16_Optimizer File "/content/CogView/fp16/__init__.py", line 15, in <module> from .fp16util import ( File "/content/CogView/fp16/fp16util.py", line 21, in <module> import mpu File "/content/CogView/mpu/__init__.py", line 35, in <module> from .layers import ColumnParallelLinear File "/content/CogView/mpu/layers.py", line 28, in <module> from apex.normalization.fused_layer_norm import FusedLayerNorm as LayerNorm ModuleNotFoundError: No module named 'apex'

[Wed Jun 16 19:21:01 2021] Memory cgroup out of memory: Killed process 15052 (python3) total-vm:19700460kB, anon-rss:11917232kB, file-rss:89696kB, shmem-rss:12288kB, UID:0 pgtables:25896kB oom_score_adj:0

Is what I can get after the process has been killed. Is there a way to optimize this to run on a GPU with lower ram? I'm using a Tesla T4 on Google Colab.

Thanks.

Guys, can you do something to prohibit inputs like this one

Even worse, someone’s already trying to generate underage porn: https://user-images.githubusercontent.com/188197/122180141-d4754600-ce90-11eb-96f6-f8cbacee8da1.mp4

Maybe add some Google-authed signup with the possibility of banning to at least make it harder for creeps to use the model for such things?

想請問一下關於此篇論文數學的部分:

1. 加入文字之後的 ELBO 有更詳細的推導過程嗎? 是不是單純把只有 image 的 ELBO 不等式兩邊各加上一個 NLL loss for text 而已? 因為我看起來 text 並沒有在 VQVAE 訓練過程中起到作用，而這個 ELBO 是給 VQVAE 的，不太懂為什麼會有 text 那項 loss
2. 不太了解 式(2) 如何變成 式(3) 的

謝謝

Hello！Our team plans to load the pre-trained model of cogview-caption to finetune with v100, which is consistent with what you said in the paper about pre-training on the V100. But it turns out that "CUDA out of memory", and the training can't be launched until the model-parallel-size is set to be 4. So, how can we load the pre-trained model and finetune on V100? @neozhangthe1 @Sleepychord @lykeven @cenyk1230 @Somefive

Hello, I wonder how to finetune the CogView model to perform image captioning? Here is my question: what is the format of the input text? I notice that the format of input text in your code is [ROI1], text, [BASE], [BOI1], image, [EOI1]. Therefore, what should I change for finetuning to image captioning? Just change the format into [BASE], [BOI1], image, [EOI1], [ROI1], text, or how?

Looking forward to your reply, thanks!

Is there any possibility to share the Dockerfile?

I was trying to use the docker since it's difficult to build the apex library. Whereas when I used docker pull like the below, "docker pull cogview/cuda111_torch181_deepspeed040"

I got the following error message: "Using default tag: latest Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"

Therefore, an original Dockerfile may be better.

Hi, I'm trying to finetune Cogview pretrained model. However, when I try to load model weights, I get following error: RuntimeError: Error(s) in loading state_dict for GPT2Model:
size mismatch for word_embeddings.weight: copying a param with shape torch.Size([14560, 2560]) from checkpoint, the shape in current model is torch .Size([14592, 2560]).

Here is my script:

`NUM_WORKERS=1 NUM_GPUS_PER_WORKER=4 MP_SIZE=1

script_path=$(realpath $0) echo $script_path script_dir=$(dirname $script_path) main_dir=$(dirname $script_dir)

OPTIONS_NCCL="NCCL_DEBUG=info" HOST_FILE_PATH="hostfile_single"

config_json="$script_dir/ds_config_zero.json" gpt_options="
--experiment-name cogview-test_finetune
--img-tokenizer-num-tokens 8192
--dataset-type TokenizedDataset
--model-parallel-size ${MP_SIZE}
--batch-size 4
--num-layers 48
--hidden-size 2560
--num-attention-heads 40
--save ./
--train-iters 2000
--save-interval 800
--resume-dataloader
--train-data /path/to/my/data
--split 90,5,5
--distributed-backend nccl
--lr-decay-style cosine
--warmup .1
--checkpoint-activations
--deepspeed-activation-checkpointing
--max-position-embeddings 1089
--max-memory-length 0
--fp16
--txt-loss-scale 5
--load /path/to/cogview
--no-load-rng
--model-parallel-size 2
--num-workers 16
--is-sparse 0
--finetune
--shuffle "

gpt_options="${gpt_options} --deepspeed
--deepspeed_config ${config_json}
"

run_cmd="${OPTIONS_NCCL} deepspeed --num_nodes ${NUM_WORKERS} --num_gpus ${NUM_GPUS_PER_WORKER} --hostfile ${HOST_FILE_PATH} pretrain_gpt2.py $@ ${gpt_options}" `

It will be great if you can provide some details for finetuning. Thanks!

Who owns the copyright of the images generated by the engine? Or is the engine generated image assigned to be published under a certain protocol (like CC0)?

After I tried to train with my own data set, the PT file I got was not 7g in size. Use it to execute. / scripts / text2image.sh -- debug error "runtimeerror: error (s) in loading state_dict for gpt2model:". Who can help me?Thank you.

Hi, I found your model checkpoint at https://resource.wudaoai.cn/home?ind=2&name=WuDao%20WenHui&id=1399364355975327744 too large for me to run on my PC, as the hidden size of the checkpoint is 2560. Do you have a checkpoint with a smaller hidden size, such as 1024? or can I easily shift the size of the checkpoint you released smaller? Thank you!

We can read on the readme since March 16th:

News! The demo for a better and faster CogView2 (formal version, March 2022) is available! The lastest model also supports English input, but to translate them into Chinese often could be better.

Is this the same as the just released paper CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers or is it a better model of CogView 1?

Thank you

使用的环境是由作者提供的docker镜像 使用的显卡是 Tesla P100-PCIE 16GB 在运行./scripts/text2image.sh --debug报错 报错代码如下： `Generate Samples WARNING: No training data specified using world size: 1 and model-parallel size: 1

using dynamic loss scaling initializing model parallel with size 1 initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234 padded vocab (size: 58219) with 21 dummy tokens (new size: 58240) prepare tokenizer done building CogView2 model ... number of parameters on model parallel rank 0: 3928849920 current device: 1 Load model file pretrained/cogview/cogview-base/142000/mp_rank_00_model_states.pt Working on No. 0 on 0... show raw text: 一只可爱的小猫。 Traceback (most recent call last): File "generate_samples.py", line 329, in main() File "generate_samples.py", line 326, in main generate_images_continually(model, args) File "generate_samples.py", line 221, in generate_images_continually generate_images_once(model, args, raw_text, seq, num=args.batch_size, output_path=output_path) File "generate_samples.py", line 166, in generate_images_once output_tokens_list.append(filling_sequence(model, seq.clone(), args)) File "/root/cogview/generation/sampling.py", line 128, in filling_sequence logits, *mems = model(tokens, position_ids, attention_mask, txt_indices_bool, img_indices_bool, is_sparse=args.is_sparse, *mems) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, **kwargs) File "/root/cogview/fp16/fp16.py", line 65, in forward return fp16_to_fp32(self.module((fp32_to_fp16(inputs)), **kwargs)) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/cogview/model/gpt2_modeling.py", line 112, in forward transformer_output = self.transformer(embeddings, position_ids, attention_mask, txt_indices_bool, img_indices_bool, is_sparse, *mems) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/cogview/mpu/sparse_transformer.py", line 604, in forward hidden_states = layer(*args, mem=mem_i) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/cogview/mpu/sparse_transformer.py", line 322, in forward attention_output = self.attention(layernorm_output1, ltor_mask, pivot_idx, is_sparse, mem) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/cogview/mpu/sparse_transformer.py", line 166, in forward output = self.dense(context_layer) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/cogview/mpu/layers.py", line 319, in forward output_parallel = F.linear(input_parallel, self.weight) File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear return torch._C._nn.linear(input, weight, bias) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP) ` 希望有人能为我解答这个问题，谢谢