Scalable Optical Flow-based Image Montaging and Alignment

Overview

SOFIMA

SOFIMA (Scalable Optical Flow-based Image Montaging and Alignment) is a tool for stitching, aligning and warping large 2d, 3d and 4d microscopy datasets.

License

This is not an officially supported Google product.

Installation

SOFIMA is implemented purely in Python, and does not require a build step. To install it directly from the repository, run:

  pip install git+https://github.com/google-research/sofima

Overview

SOFIMA uses optical flow regularized with an elastic mesh to establish maps between data in different coordinate systems. Both the flow estimator as well as the mesh solver are implemented in JAX and will automatically take advantage of GPU acceleration if the hardware if available.

A core data structure used throughout the project is a coordinate map stored as a dense array of relative offsets (see the module docstring in map_utils.py for details). Among other uses, this is the representation of the estimated flow fields and the mesh node positions.

Example usage

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

You might also like...
UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss This repository contains the TensorFlow implementation of the paper UnF

[CVPR 2022] Deep Equilibrium Optical Flow Estimation
[CVPR 2022] Deep Equilibrium Optical Flow Estimation

Deep Equilibrium Optical Flow Estimation This is the official repo for the paper Deep Equilibrium Optical Flow Estimation (CVPR 2022), by Shaojie Bai*

Just Go with the Flow: Self-Supervised Scene Flow Estimation

Just Go with the Flow: Self-Supervised Scene Flow Estimation Code release for the paper Just Go with the Flow: Self-Supervised Scene Flow Estimation,

Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

The official codes of our CVPR2022 paper: A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift
The official codes of our CVPR2022 paper: A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift

TwoStageAlign The official codes of our CVPR2022 paper: A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift Pa

A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization
A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization

University1652-Baseline [Paper] [Slide] [Explore Drone-view Data] [Explore Satellite-view Data] [Explore Street-view Data] [Video Sample] [中文介绍] This

GeneralOCR is open source Optical Character Recognition based on PyTorch.
GeneralOCR is open source Optical Character Recognition based on PyTorch.

Introduction GeneralOCR is open source Optical Character Recognition based on PyTorch. It makes a fidelity and useful tool to implement SOTA models on

Python implementation of O-OFDMNet, a deep learning-based optical OFDM system,

O-OFDMNet This includes Python implementation of O-OFDMNet, a deep learning-based optical OFDM system, which uses neural networks for signal processin

Comments
  • [JAX] Import scipy submodules directly in code that uses them.

    [JAX] Import scipy submodules directly in code that uses them.

    [JAX] Import scipy submodules directly in code that uses them.

    JAX will soon import scipy lazily, so code that relies on JAX having already imported submodules of scipy will fail. Change code to import what it uses.

    opened by copybara-service[bot] 0
  • Aligning consecutive slices

    Aligning consecutive slices

    Hello,

    Use case

    I have an input volume that is a reconstruction of 2D slices into a 3D volume. Therefore, the consecutive slices are jagged and the volume along perpendicular directions are not smooth. My idea is to try to align consecutive slices of that volume.

    Code

    To achieve that, I tried to use this package. Here is the snippet code I am using.

    1. Could you tell me if I am using your repository right?
    2. Do you have some advices concerning the different parameters (patch_size, step, stride, overlap, ...) along this pipeline?
    from sofima.flow_field import JAXMaskedXCorrWithStatsCalculator
    from sofima.flow_utils import clean_flow
    from sofima.warp import ndimage_warp
    
    patch_size = 15
    step = 1
    overlap = 3
    
    maskcorr = JAXMaskedXCorrWithStatsCalculator()
    flow = maskcorr.flow_field(img1, img2, patch_size, step)
    clean_output = clean_flow(
        flow, 
        min_peak_ratio=np.min(flow[2]), 
        min_peak_sharpness=np.min(flow[1]), 
        max_magnitude=0, 
        max_deviation=0,
    )
    warped_img = ndimage_warp(
        img2, 
        clean_output, 
        stride=(step, step), 
        work_size=(patch_size, patch_size), 
        overlap=(overlap, overlap)
    )
    

    Results

    I would like to apply this snippet of code iteratively. Concretely, this means: a. Alignment between slice n and slice n+1 -> warping of slice n+1 b. Alignment between warped slice n+1 and slice n+2 -> warping of slice n+2 c. Alignment between warped slice n+2 and slice n+3 -> ...

    However, currently, applying this iterative procedure is giving strange results: pixels are disappearing, see images below. image 3. Do you have an idea why this is happening?

    Thank you for your help.

    opened by EmilieDel 1
Owner
Google Research
Google Research
A project that uses optical flow and machine learning to detect aimhacking in video clips.

waldo-anticheat A project that aims to use optical flow and machine learning to visually detect cheating or hacking in video clips from fps games. Che

waldo.vision 542 Dec 3, 2022
A project that uses optical flow and machine learning to detect aimhacking in video clips.

waldo-anticheat A project that aims to use optical flow and machine learning to visually detect cheating or hacking in video clips from fps games. Che

waldo.vision 383 Nov 1, 2021
Traditional deepdream with VQGAN+CLIP and optical flow. Ready to use in Google Colab

VQGAN-CLIP-Video cat.mp4 policeman.mp4 schoolboy.mp4 forsenBOG.mp4

null 23 Oct 26, 2022
Demo code for paper "Learning optical flow from still images", CVPR 2021.

Depthstillation Demo code for "Learning optical flow from still images", CVPR 2021. [Project page] - [Paper] - [Supplementary] This code is provided t

null 130 Dec 25, 2022
Learning Optical Flow from a Few Matches (CVPR 2021)

Learning Optical Flow from a Few Matches This repository contains the source code for our paper: Learning Optical Flow from a Few Matches CVPR 2021 Sh

Shihao Jiang (Zac) 159 Dec 16, 2022
Unsupervised Learning of Multi-Frame Optical Flow with Occlusions

This is a Pytorch implementation of Janai, J., Güney, F., Ranjan, A., Black, M. and Geiger, A., Unsupervised Learning of Multi-Frame Optical Flow with

Anurag Ranjan 110 Nov 2, 2022
Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

flownet2-pytorch Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. Multiple GPU training is supported, a

NVIDIA Corporation 2.8k Dec 27, 2022
a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch

pytorch-spynet This is a personal reimplementation of SPyNet [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 269 Jan 2, 2023
A fast model to compute optical flow between two input images.

DCVNet: Dilated Cost Volumes for Fast Optical Flow This repository contains our implementation of the paper: @InProceedings{jiang2021dcvnet, title={

Huaizu Jiang 8 Sep 27, 2021
Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Sensor-Guided Optical Flow Demo code for "Sensor-Guided Optical Flow", ICCV 2021 This code is provided to replicate results with flow hints obtained f

null 10 Mar 16, 2022