FairyTailor: Multimodal Generative Framework for Storytelling

Last update: Dec 30, 2022

Overview

FairyTailor: Multimodal Generative Framework for Storytelling

Human-in-the-loop visual story co-creation.

Users can create a cohesive children's story by weaving generated texts and retrieved images with their input. With co-creation, writers contribute their creative thinking, while generative models contribute to their constant workflow. FairyTailor adds another modality and modifies the text generation process to help producing a coherent and creative story.

Set-up (development)

After cloning the repository:

Client (Vue 2.6)

Install and check that the client compiles:

cd client
npm i
npm run build

Backend (FASTAPI)

Install and activate the environment (conda provided):

conda env create -f environment.yml
conda activate MultiModalStory

Install environment globally in the directory:

pip install -e .
pip install git+https://github.com/openai/CLIP.git

After installation run:

python -m spacy download en_core_web_sm

In python terminal:

nltk.download('wordnet')
nltk.download('sentiwordnet')
nltk.download('averaged_perceptron_tagger')

Large Data Management (dvc)

Our large data files are stored on IBM's Cloud Object Storage, and to pull data files from that platform you will use a special, read-only .dvc/config file.

dvc pull -f

Which will pull:

backend/outputs (five preset stories)
backend/story_generator/downloaded (transformers)
client/public/unsplash25k (styled images)

Running the framework during developemnt

Client:

cd client
npm run devw

Backend (with server auto reload):

uvicorn backend.server:app --reload --reload-dir backend

Open the uvicorn server localhost:8000 in your web browser

Modifications Ideas:

New huggingface transformer

Place the transformer in backend/story_generator/downloaded directory.
Update the current model path by changing the constant FINETUNED_GPT2_PATH in backend/story_generator/constants.py.

New images folder

Replace the folder client/public/unsplash25k/sketch_images1024 with yours.
Update the current path by changing the constant IMAGE_PATH in client/src/components/Constants.js.

API functionalities

Add functions to the backend endpoint at backend/server/main.py.
Update client/src/js/api/mainApi.js to call the backend endpoint from the client.
Update the corresponding user components in client/src/components.

You might also like...

Rethinking the U-Net architecture for multimodal biomedical image segmentation

MultiResUNet Rethinking the U-Net architecture for multimodal biomedical image segmentation This repository contains the original implementation of "M

308 Jan 5, 2023

Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020

XDVioDet Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020. The proj

64 Dec 12, 2022

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos. By adopting a unified pipeline-based API design, PyKale enforces standardization and minimalism, via reusing existing resources, reducing repetitions and redundancy, and recycling learning models across areas.

370 Dec 27, 2022

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction This is the code for the paper Combining E

69 Dec 26, 2022

MERLOT: Multimodal Neural Script Knowledge Models

merlot MERLOT: Multimodal Neural Script Knowledge Models MERLOT is a model for learning what we are calling "neural script knowledge" -- representatio

190 Dec 22, 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

62 Dec 24, 2022

Comments

Error pulling from dvc

When I run dvc pull -f I get the following output:

WARNING: No file hash info found for 'backend/unsplash-dataset'. It won't be created.
WARNING: No file hash info found for 'backend/story_generator/downloaded'. It won't be created.
WARNING: No file hash info found for 'backend/outputs'. It won't be created.
3 files failed
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
backend/unsplash-dataset
backend/story_generator/downloaded
backend/outputs
Is your cache up to date?
<https://error.dvc.org/missing-files>

opened by michael-knight 2

.dvc/config file missing

After installation, the next step is to pull all the files: Our large data files are stored on IBM's Cloud Object Storage, and to pull data files from that platform you will use a special, read-only .dvc/config file.

But after the following command dvc pull -f

I get the following:

Does it mean there is something missing in the repository?

opened by timeeehd 1
fairytailor.org is down?

It appears that (as of ~8/14/2021 1:00pm EST) fairytailor.org is down. No big deal, just wanted to check it out.

According to Down For Everyone or Just Me it's not just me.

opened by sterlinm 1

FairyTailor: Multimodal Generative Framework for Storytelling

Related tags

Overview

FairyTailor: Multimodal Generative Framework for Storytelling

Human-in-the-loop visual story co-creation.

Set-up (development)

Client (Vue 2.6)

Backend (FASTAPI)

Large Data Management (dvc)

Running the framework during developemnt

Modifications Ideas:

New huggingface transformer

New images folder

API functionalities

You might also like...

Rethinking the U-Net architecture for multimodal biomedical image segmentation

Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

MERLOT: Multimodal Neural Script Knowledge Models

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Preprocessed Datasets for our Multimodal NER paper

Code for SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

multimodal transformer

Comments

Error pulling from dvc

.dvc/config file missing

fairytailor.org is down?

Owner

Eden Bens

This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

The code repository for EMNLP 2021 paper "Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization".

A Comparative Framework for Multimodal Recommender Systems

Framework for joint representation learning, evaluation through multimodal registration and comparison with image translation based approaches

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

MediaPipe is a an open-source framework from Google for building multimodal

Minimal PyTorch implementation of Generative Latent Optimization from the paper "Optimizing the Latent Space of Generative Networks"

This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

Deep Multimodal Neural Architecture Search