Official implementation of the MM'21 paper Constrained Graphic Layout Generation via Latent Optimization

Overview

[MM'21] Constrained Graphic Layout Generation via Latent Optimization

This repository provides the official code for the paper "Constrained Graphic Layout Generation via Latent Optimization", especially the code for:

  • LayoutGAN++: generative adversarial networks for layout generation
  • CLG-LO: a framework for generating layouts that satisfy constraints
  • Layout evaluation: measuring the quantitative metrics of Layout FID, Maximum IoU, Alignment, and Overlap for generated layouts

Installation

  1. Clone this repository

    git clone https://github.com/ktrk115/const_layout.git
    cd const_layout
  2. Create a new conda environment (Python 3.8)

    conda create -n const_layout python=3.8
    conda activate const_layout
  3. Install PyTorch 1.8.* and the corresponding versoin of PyTorch Geometric

  4. Install the other dependent libraries

    pip install -r requirements.txt
  5. Prepare data (see this instruction)

  6. Download pre-trained models

    ./download_model.sh

Development environment

  • Ubuntu 18.04, CUDA 11.1

LayoutGAN++

Architecture

Training animation

Generate layouts with LayoutGAN++

python generate.py pretrained/layoutganpp_rico.pth.tar --out_path output/generated_layouts.pkl --num_save 5

Train LayoutGAN++ model

python train.py --dataset rico --batch_size 64 --iteration 200000 --latent_size 4 --lr 1e-05 --G_d_model 256 --G_nhead 4 --G_num_layers 8 --D_d_model 256 --D_nhead 4 --D_num_layers 8

CLG-LO

w/ beautification constraints w/ relational constraints

Generate layouts with beautification constraints

python generate_const.py pretrained/layoutganpp_publaynet.pth.tar --const_type beautify --out_path output/beautify/generated_layouts.pkl --num_save 5

Generate layouts with relational constraints

python generate_const.py pretrained/layoutganpp_publaynet.pth.tar --const_type relation --out_path output/relation/generated_layouts.pkl --num_save 5

Layout evaluation

Evaluate generated layouts

python eval.py rico output/generated_layouts.pkl

A pickle file should be a list of layouts, where each layout is a tuple of bounding boxes and labels. The bounding box is represented by [x, y, width, height] in normalized coordinates, and the label is represented by an index. An example is shown below.

In [x]: layouts
Out[x]:
[(array([[0.47403812, 0.11276676, 0.6250037 , 0.02210438],
         [0.49971417, 0.8550553 , 0.81388366, 0.03492427],
         [0.49919674, 0.47857162, 0.81024694, 0.7070079 ]], dtype=float32),
  array([0, 0, 3]),
  ...

Citation

If this repository helps your research, please consider citing our paper.

@inproceedings{Kikuchi2021,
    title = {Constrained Graphic Layout Generation via Latent Optimization},
    author = {Kotaro Kikuchi and Edgar Simo-Serra and Mayu Otani and Kota Yamaguchi},
    booktitle = {Proceedings of the ACM International Conference on Multimedia},
    series = {MM '21},
    volume = {},
    year = {2021},
    pages = {},
    doi = {10.1145/3474085.3475497}
}

Licence

GNU AGPLv3

Related repositories

Comments
  • about the train loss

    about the train loss

    thank for you work. I train the model with command:

    python train.py --dataset rico --batch_size 64 --iteration 200000 --latent_size 4 --lr 1e-05 --G_d_model 256 --G_nhead 4 --G_num_layers 8 --D_d_model 256 --D_nhead 4 --D_num_layers 8
    

    and it will show me that. does it look normal ? the generator loss become large and Discriminator loss become small : image image

    opened by Usernamezhx 2
  • How to change number of elements?

    How to change number of elements?

    Hey, I've been trying to train a model on Rico with more data In rico.py, I can see that number of elements are capped at 9

    I've been going through your generator and discriminator code and I would greatly appreciate some help in where I need to make changes so the models can accept in data samples with bboxes and labels > 9.

    Thanks so much! Sukriti

    opened by sukritiverma1996 2
  • Key error with constrained generation

    Key error with constrained generation

    Hey, Thanks for releasing your code and environment. I was able to generate samples using pre-trained models. However, when I try to generate samples with constraints, I get the following key error:

      File "generate_const.py", line 199, in <module>
        main()
      File "generate_const.py", line 148, in main
        for z in optimizer.generator(z, data):
      File "/home/sukrverm/const_layout/clg/auglag.py", line 96, in generator
        assert data.attr[0]['has_canvas_element']
    KeyError: 0
    

    I suppose this is some attribute of publaynet data that is False here. Do I need to edit the get_dataset Publaynet class to remove such examples before hand?

    opened by sukritiverma1996 2
  • data link is in 404 status, how can i get the train data?

    data link is in 404 status, how can i get the train data?

    Download rico_dataset_v0.1_semantic_annotations.zip from "UI Screenshots and Hierarchies with Semantic Annotations" and decompress it. but it is image

    opened by Jolinbaby 2
  • Unable to reproduce the CLG-LO

    Unable to reproduce the CLG-LO

    Thank you for source code. I get an error when I run CLG-LO

    If possible, please share the how to fix.

    Traceback (most recent call last):
      File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
        cli.main()
      File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
        run()
      File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
        runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
      File "/usr/lib/python3.8/runpy.py", line 265, in run_path
        return _run_module_code(code, init_globals, run_name,
      File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/home/vscode/const_layout/generate_const.py", line 199, in <module>
        main()
      File "/home/vscode/const_layout/generate_const.py", line 182, in main
        save_gif(out_path, j, netG,
      File "/home/vscode/const_layout/generate_const.py", line 52, in save_gif
        subprocess.run(['convert', '-delay', '50',
      File "/usr/lib/python3.8/subprocess.py", line 493, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'convert'
    
    
    opened by ryhhtn 2
  • Why add

    Why add "to_dense_batch" to process the location information x and category label y?

    What does this mean, please? Why add "to_dense_batch" to process the location information x and category label y?

    for i, data in enumerate(train_dataloader): data = data.to(device) label, mask = to_dense_batch(data.y, data.batch)#??????????????? bbox_real, _ = to_dense_batch(data.x, data.batch)#???????????????

    opened by MtYCNN 1
  • LayoutNet - how to train?

    LayoutNet - how to train?

    Could you share a script to train LayoutNet for FID calculation? I see the model architecture file, could you maybe share the code to train it or the loss functions used?

    opened by sukritiverma1996 1
  • Quantitative comparison of unconstrained layout generation

    Quantitative comparison of unconstrained layout generation

    I tried re-training LayoutGAN++ used magazine dataset. But, I can't reappearance Quantitative comparison, especially overlap. Results of multiple trials, overlap average 40 over.

    ex.

    Dataset: magazine
            FID: 13.88
            Max. IoU: 0.26
            Alignment: 0.82
            Overlap: 40.31
    

    I think using the hyperparameters from the paper. By any chance, cause in the version of os or library?

    NAME="Ubuntu"
    VERSION="20.04.4 LTS (Focal Fossa)"
    
    pytorch-fid==0.2.1
    torch==1.11.0+cu113
    torch-cluster==1.6.0
    torch-geometric==2.0.4
    torch-scatter==2.0.9
    torch-sparse==0.6.13
    torch-spline-conv==1.2.1
    torchaudio==0.11.0+cu113
    torchvision==0.12.0+cu113
    
    opened by ryhhtn 2
Owner
Kotaro Kikuchi
Waseda University
Kotaro Kikuchi
object detection; robust detection; ACM MM21 grand challenge; Security AI Challenger Phase VII

赛题背景 在商品知识产权领域,知识产权体现为在线商品的设计和品牌。不幸的是,在每一天,存在着非法商户通过一些对抗手段干扰商标识别来逃避侵权,这带来了很高的知识产权风险和财务损失。为了促进先进的多媒体人工智能技术的发展,以保护企业来之不易的创作和想法免受恶意使用和剽窃,因此提出了鲁棒性标识检测挑战赛

null 65 Dec 22, 2022
library for nonlinear optimization, wrapping many algorithms for global and local, constrained or unconstrained, optimization

NLopt is a library for nonlinear local and global optimization, for functions with and without gradient information. It is designed as a simple, unifi

Steven G. Johnson 1.4k Dec 25, 2022
I3-master-layout - Simple master and stack layout script

Simple master and stack layout script | ------ | ----- | | | | | Ma

Tobias S 18 Dec 5, 2022
PyTorch implementation of Constrained Policy Optimization

PyTorch implementation of Constrained Policy Optimization (CPO) This repository has a simple to understand and use implementation of CPO in PyTorch. A

Sapana Chaudhary 25 Dec 8, 2022
Official code for "End-to-End Optimization of Scene Layout" -- including VAE, Diff Render, SPADE for colorization (CVPR 2020 Oral)

End-to-End Optimization of Scene Layout Code release for: End-to-End Optimization of Scene Layout CVPR 2020 (Oral) Project site, Bibtex For help conta

Andrew Luo 41 Dec 9, 2022
Harmonious Textual Layout Generation over Natural Images via Deep Aesthetics Learning

Harmonious Textual Layout Generation over Natural Images via Deep Aesthetics Learning Code for the paper Harmonious Textual Layout Generation over Nat

null 7 Aug 9, 2022
Official PyTorch implementation of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution", CVPR 2022.

Deep Constrained Least Squares for Blind Image Super-Resolution [Paper] This is the official implementation of 'Deep Constrained Least Squares for Bli

MEGVII Research 141 Dec 30, 2022
A semismooth Newton method for elliptic PDE-constrained optimization

sNewton4PDEOpt The Python module implements a semismooth Newton method for solving finite-element discretizations of the strongly convex, linear ellip

null 2 Dec 8, 2022
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
PyTorch implementation of "LayoutTransformer: Layout Generation and Completion with Self-attention"

PyTorch implementation of "LayoutTransformer: Layout Generation and Completion with Self-attention" to appear in ICCV 2021

Kamal Gupta 75 Dec 23, 2022
Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

HamasKhan 3 Jul 8, 2022
[CVPR'21] Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation

Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation Weixiang Yang, Qi Li, Wenxi Liu, Yuanlong Yu, Y

null 118 Dec 26, 2022
Official code for paper "Optimization for Oriented Object Detection via Representation Invariance Loss".

Optimization for Oriented Object Detection via Representation Invariance Loss By Qi Ming, Zhiqiang Zhou, Lingjuan Miao, Xue Yang, and Yunpeng Dong. Th

ming71 56 Nov 28, 2022
Prototypical python implementation of the trust-region algorithm presented in Sequential Linearization Method for Bound-Constrained Mathematical Programs with Complementarity Constraints by Larson, Leyffer, Kirches, and Manns.

Prototypical python implementation of the trust-region algorithm presented in Sequential Linearization Method for Bound-Constrained Mathematical Programs with Complementarity Constraints by Larson, Leyffer, Kirches, and Manns.

null 3 Dec 2, 2022
PyTorch implementation of: Michieli U. and Zanuttigh P., "Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations", CVPR 2021.

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations This is the official PyTorch implementation

Multimedia Technology and Telecommunication Lab 42 Nov 9, 2022
Autolfads-tf2 - A TensorFlow 2.0 implementation of Latent Factor Analysis via Dynamical Systems (LFADS) and AutoLFADS

autolfads-tf2 A TensorFlow 2.0 implementation of LFADS and AutoLFADS. Installati

Systems Neural Engineering Lab 11 Oct 29, 2022
Official code of the paper "Expanding Low-Density Latent Regions for Open-Set Object Detection" (CVPR 2022)

OpenDet Expanding Low-Density Latent Regions for Open-Set Object Detection (CVPR2022) Jiaming Han, Yuqiang Ren, Jian Ding, Xingjia Pan, Ke Yan, Gui-So

csuhan 64 Jan 7, 2023
Official Implementation of LARGE: Latent-Based Regression through GAN Semantics

LARGE: Latent-Based Regression through GAN Semantics [Project Website] [Google Colab] [Paper] LARGE: Latent-Based Regression through GAN Semantics Yot

null 83 Dec 6, 2022
This repository contains the code for "SBEVNet: End-to-End Deep Stereo Layout Estimation" paper by Divam Gupta, Wei Pu, Trenton Tabor, Jeff Schneider

SBEVNet: End-to-End Deep Stereo Layout Estimation This repository contains the code for "SBEVNet: End-to-End Deep Stereo Layout Estimation" paper by D

Divam Gupta 19 Dec 17, 2022