WHAT

• add system and training visualization using H2O Wave
• add services in docker-compose.yml which runs the wave server.
• add Dockerfile for H2O Wave
• add h2o_wave, psutil in dependencies.
• add class for visualization.

The warning bellow is produced when using Differentiable Augmentation(DiffAugment).

[W TensorIterator.cpp:924] Warning: Mixed memory format inputs detected while calling the operator. The operator will output channels_last tensor even if some of the inputs are not in channels_last format. (function operator())

I am guessing this only happens in PyTorch ver1.6.0 or higher. PR#37968@pytorch/pytorch

DiffAugment is an augmentation method, therefore I believe this will not affect the results much.

WHAT

• Add more general status holder
• Add AdaBelief optimizer to gan_utils
• Update code for exponential moving average
• Erase older version of fp16 code (not used in any implementations)

WHAT

• add noise samplers
• add device specification func
• add exponential moving average class
• add kornia to dependences
• add kornia version of diffaugment (not perfectly the same).
• move gan related utils to gan_utils
• adjust import

WHY

• easier coding
• issue#31

WHAT

• add defined loss functions
• add dragan, simplified 0-center gp

+alpha

• ignore .pt files
• make danbooru portraits dataset importable
• fix default transform for danbooru portraits dataset

WHAT

• the implementations are moved to the implementations folder
• the main.py is rewritten to import from the folder

WHY

• the number of implementations are increasing.
• wanted to make the main.py less informative. (will do this later.)

WHAT

• Update PyTorch to version 1.11

TODO

• [x] Update base image of docker container.
• ~~[ ] Implement dataset with torchdata.~~ (Might come back later.)
• [x] Check cpp_extension build (stylegan ops).

Beep boop. Your images are optimized!

Your image file size has been reduced by 5% 🎉

📝 docs | :octocat: repo | 🙋🏾 issues | 🏪 marketplace

~Imgbot - Part of Optimole family

WHAT

• Add minimal implementation of huggineface/accelerate with some changes.

WHY

• Reduce coding for AMP and device placement.

Usage

Original training code.

device = get_device()
dataset = Dataset.asloader(...)
model = Net(...)
optimizer = optim.Adam(model.parameters())
amp = True
scaler = GradScaler() if amp else None

for data in dataset:
    data = data.to(device)
    with autocast(amp):
        output = model(data)
        loss = criterion(output)

    if scaler is not None:
        scaler.scale(loss).backward()
        scaler.step(optimizer)
    else:
        loss.backward()
        optimizer.step()

becomes

# device configuration is optional
# if device is None: it is automatically set by nnutils.get_device()
device = get_device()
dataset = Dataset.asloader(...)
model = Net(...)
optimizer = optim.Adam(model.parameters())
amp = True
# scaler object is held inside accelerator
# scaler = GradScaler() if amp else None

# create accelerator object and prepare for training.
accelerator = MiniAccelerator(amp=amp, device_placement=True, device=device)
dataset, model, optimizer = accelerator.prepare(
    dataset, model, optimizer)

for data in dataset:
    # device placement is done automatically
    # data = data.to(device)
    with accelerator.autocast():
        output = model(data)
        loss = criterion(output)

    # loss scaling is done inside accelerator
    # if scaler is not None:
    #     scaler.scale(loss).backward()
    #     scaler.step(optimizer)
    # else:
    #     loss.backward()
    #     optimizer.step()

    # use accelerator.backward(loss) instead of loss.backward()
    accelerator.backward(loss)
    optimizer.step()

    # update scaler. can be ignored if amp=False
    accelerator.update()

Modifications

Not implemented

• Multi device support
• TPU support
• DeepSpeed support

Changes

• Calling .update() of GradScaler is done in MiniAccelerator().update().

  Originally this is done inside AcceleratedOptimizer().step(), but when using multiple optimizers, like in GAN, this will call GradScaler().update() multiple times per step.

Notes

I will not reimplement existing training codes using MiniAccelerator.

TODOs

• [x] add implementation
• [x] test

Beep boop. Your images are optimized!

Your image file size has been reduced by 27% 🎉

📝 docs | :octocat: repo | 🙋🏾 issues | 🏪 marketplace

~Imgbot - Part of Optimole family

WHAT

• organize repository.
• move general and gan_utils from implementations.
• reimplement dataset, utilities.
• add utilities.
• separate thirdparty codes. (e.g. diffaugment, adabelief)
• add thirdparty codes. (e.g. stylegan2-ada-pytorch ops, ada)
• use renewed dataset, utilities in implementations.
• fix/renew implementation codes.

WHY

• general and gan_utils are weird to be in implementations folder.
• add copies of thirdparty license.

HOW

• [x] add codes out of implementations folder.
• [x] update implementations.
• [x] erase codes not used inside implementations.

Could not get good results in ContraD.

what can be done.

• Train with more simpler model. (vs StyleGAN2)
• Further hyper-parameter tuning.
• Loss function. (taken from original implementation so I believe it's not wrong.)

In the BigGAN paper, they have improved the scores by using bigger batch size and a larger model. I used the batch size of 12, which is the biggest batch size my environment can train...

Also the model collapsed after about 10 epochs.