🎓Automatically Update CV Papers Daily using Github Actions (Update at 12:00 UTC Every Day)

Overview

2021-10-08

Title Abstract Authors PDF Code
Dense Gaussian Processes For Few-Shot Segmentation Few-shot segmentation is a challenging dense prediction task, which entails segmenting a novel query image given only a small annotated support set. the key problem is thus to design a method that aggregates detailed information from the support set, while being robust to large variations in appearance and context. to this end, we propose a few-shot segmentation method based on dense gaussian process (gp) regression. given the support set, our dense gp learns the mapping from local deep image features to mask values, capable of capturing complex appearance distributions. furthermore, it provides a principled means of capturing uncertainty, which serves as another powerful cue for the final segmentation, obtained by a cnn decoder. instead of a one-dimensional mask output, we further exploit the end-to-end learning capabilities of our approach to learn a high-dimensional output space for the gp. our approach sets a new state-of-the-art for both 1-shot and 5-shot fss on the pascal-5$^i$ and coco-20$^i$ benchmarks, achieving an absolute gain of $+14.9$ miou in the coco-20$^i$ 5-shot setting. furthermore, the segmentation quality of our approach scales gracefully when increasing the support set size, while achieving robust cross-dataset transfer. Joakim Johnander, Johan Edstedt, Michael Felsberg, Fahad Shahbaz Khan, Martin Danelljan 2110.03674 link
One Thing To Fool Them All: Generating Interpretable, Universal, And Physically-Realizable Adversarial Features It is well understood that modern deep networks are vulnerable to adversarial attacks. however, conventional methods fail to produce adversarial perturbations that are intelligible to humans, and they pose limited threats in the physical world. to study feature-class associations in networks and better understand the real-world threats they face, we develop feature-level adversarial perturbations using deep image generators and a novel optimization objective. we term these feature-fool attacks. we show that they are versatile and use them to generate targeted feature-level attacks at the imagenet scale that are simultaneously interpretable, universal to any source image, and physically-realizable. these attacks can also reveal spurious, semantically-describable feature/class associations, and we use them to guide the design of "copy/paste" adversaries in which one natural image is pasted into another to cause a targeted misclassification. Stephen Casper, Max Nadeau, Gabriel Kreiman 2110.03605 link
Towards Accurate Cross-Domain In-Bed Human Pose Estimation Human behavioral monitoring during sleep is essential for various medical applications. majority of the contactless human pose estimation algorithms are based on rgb modality, causing ineffectiveness in in-bed pose estimation due to occlusions by blankets and varying illumination conditions. long-wavelength infrared (lwir) modality based pose estimation algorithms overcome the aforementioned challenges; however, ground truth pose generations by a human annotator under such conditions are not feasible. a feasible solution to address this issue is to transfer the knowledge learned from images with pose labels and no occlusions, and adapt it towards real world conditions (occlusions due to blankets). in this paper, we propose a novel learning strategy comprises of two-fold data augmentation to reduce the cross-domain discrepancy and knowledge distillation to learn the distribution of unlabeled images in real world conditions. our experiments and analysis show the effectiveness of our approach over multiple standard human pose estimation baselines. Mohamed Afham, Udith Haputhanthri, Jathurshan Pradeepkumar, Mithunjha Anandakumar, Ashwin De Silva, Chamira Edussooriya 2110.03578 link
A Few-Shot Learning Graph Multi-Trajectory Evolution Network For Forecasting Multimodal Baby Connectivity Development From A Baseline Timepoint Charting the baby connectome evolution trajectory during the first year after birth plays a vital role in understanding dynamic connectivity development of baby brains. such analysis requires acquisition of longitudinal connectomic datasets. however, both neonatal and postnatal scans are rarely acquired due to various difficulties. a small body of works has focused on predicting baby brain evolution trajectory from a neonatal brain connectome derived from a single modality. although promising, large training datasets are essential to boost model learning and to generalize to a multi-trajectory prediction from different modalities (i.e., functional and morphological connectomes). here, we unprecedentedly explore the question: can we design a few-shot learning-based framework for predicting brain graph trajectories across different modalities? to this aim, we propose a graph multi-trajectory evolution network (gmte-net), which adopts a teacher-student paradigm where the teacher network learns on pure neonatal brain graphs and the student network learns on simulated brain graphs given a set of different timepoints. to the best of our knowledge, this is the first teacher-student architecture tailored for brain graph multi-trajectory growth prediction that is based on few-shot learning and generalized to graph neural networks (gnns). to boost the performance of the student network, we introduce a local topology-aware distillation loss that forces the predicted graph topology of the student network to be consistent with the teacher network. experimental results demonstrate substantial performance gains over benchmark methods. hence, our gmte-net can be leveraged to predict atypical brain connectivity trajectory evolution across various modalities. our code is available at https: //github.com/basiralab/gmte-net. Alaa Bessadok, Ahmed Nebli, Mohamed Ali Mahjoub, Gang Li, Weili Lin, Dinggang Shen, Islem Rekik 2110.03535 link
Unsupervised Image Decomposition With Phase-Correlation Networks The ability to decompose scenes into their object components is a desired property for autonomous agents, allowing them to reason and act in their surroundings. recently, different methods have been proposed to learn object-centric representations from data in an unsupervised manner. these methods often rely on latent representations learned by deep neural networks, hence requiring high computational costs and large amounts of curated data. such models are also difficult to interpret. to address these challenges, we propose the phase-correlation decomposition network (pcdnet), a novel model that decomposes a scene into its object components, which are represented as transformed versions of a set of learned object prototypes. the core building block in pcdnet is the phase-correlation cell (pc cell), which exploits the frequency-domain representation of the images in order to estimate the transformation between an object prototype and its transformed version in the image. in our experiments, we show how pcdnet outperforms state-of-the-art methods for unsupervised object discovery and segmentation on simple benchmark datasets and on more challenging data, while using a small number of learnable parameters and being fully interpretable. Angel Villar-Corrales, Sven Behnke 2110.03473 link
Recurrent Multigraph Integrator Network For Predicting The Evolution Of Population-Driven Brain Connectivity Templates Learning how to estimate a connectional brain template(cbt) from a population of brain multigraphs, where each graph (e.g., functional) quantifies a particular relationship between pairs of brain regions of interest (rois), allows to pin down the unique connectivity patterns shared across individuals. specifically, a cbt is viewed as an integral representation of a set of highly heterogeneous graphs and ideally meeting the centeredness (i.e., minimum distance to all graphs in the population) and discriminativeness (i.e., distinguishes the healthy from the disordered population) criteria. so far, existing works have been limited to only integrating and fusing a population of brain multigraphs acquired at a single timepoint. in this paper, we unprecedentedly tackle the question: given a baseline multigraph population, can we learn how to integrate and forecast its cbt representations at follow-up timepoints? addressing such question is of paramount in predicting common alternations across healthy and disordered populations. to fill this gap, we propose recurrent multigraph integrator network (remi-net), the first graph recurrent neural network which infers the baseline cbt of an input population t1 and predicts its longitudinal evolution over time (ti > t1). our remi-net is composed of recurrent neural blocks with graph convolutional layers using a cross-node message passing to first learn hidden-states embeddings of each cbt node (i.e., brain region of interest) and then predict its evolution at the consecutive timepoint. moreover, we design a novel time-dependent loss to regularize the cbt evolution trajectory over time and further introduce a cyclic recursion and learnable normalization layer to generate well-centered cbts from time-dependent hidden-state embeddings. finally, we derive the cbt adjacency matrix from the learned hidden state graph representation. Oytun Demirbilek, Islem Rekik 2110.03453 link
Inter-Domain Alignment For Predicting High-Resolution Brain Networks Using Teacher-Student Learning Accurate and automated super-resolution image synthesis is highly desired since it has the great potential to circumvent the need for acquiring high-cost medical scans and a time-consuming preprocessing pipeline of neuroimaging data. however, existing deep learning frameworks are solely designed to predict high-resolution (hr) image from a low-resolution (lr) one, which limits their generalization ability to brain graphs (i.e., connectomes). a small body of works has focused on superresolving brain graphs where the goal is to predict a hr graph from a single lr graph. although promising, existing works mainly focus on superresolving graphs belonging to the same domain (e.g., functional), overlooking the domain fracture existing between multimodal brain data distributions (e.g., morphological and structural). to this aim, we propose a novel inter-domain adaptation framework namely, learn to superresolve brain graphs with knowledge distillation network (l2s-kdnet), which adopts a teacher-student paradigm to superresolve brain graphs. our teacher network is a graph encoder-decoder that firstly learns the lr brain graph embeddings, and secondly learns how to align the resulting latent representations to the hr ground truth data distribution using an adversarial regularization. ultimately, it decodes the hr graphs from the aligned embeddings. next, our student network learns the knowledge of the aligned brain graphs as well as the topological structure of the predicted hr graphs transferred from the teacher. we further leverage the decoder of the teacher to optimize the student network. l2s-kdnet presents the first ts architecture tailored for brain graph super-resolution synthesis that is based on inter-domain alignment. our experimental results demonstrate substantial performance gains over benchmark methods. Basar Demir, Alaa Bessadok, Islem Rekik 2110.03452 link
Rhh-Lgp: Receding Horizon And Heuristics-Based Logic-Geometric Programming For Task And Motion Planning Sequential decision-making and motion planning for robotic manipulation induce combinatorial complexity. for long-horizon tasks, especially when the environment comprises many objects that can be interacted with, planning efficiency becomes even more important. to plan such long-horizon tasks, we present the rhh-lgp algorithm for combined task and motion planning (tamp). first, we propose a tamp approach (based on logic-geometric programming) that effectively uses geometry-based heuristics for solving long-horizon manipulation tasks. we further improve the efficiency of this planner by a receding horizon formulation, resulting in rhh-lgp. we demonstrate the effectiveness and generality of our approach on several long-horizon tasks that require reasoning about interactions with a large number of objects. using our framework, we can solve tasks that require multiple robots, including a mobile robot and snake-like walking robots, to form novel heterogeneous kinematic structures autonomously. Cornelius V. Braun, Joaquim Ortiz-Haro, Marc Toussaint, Ozgur S. Oguz 2110.03420 link
Optimized U-Net For Brain Tumor Segmentation We propose an optimized u-net architecture for a brain \mbox{tumor} segmentation task in the brats21 challenge. to find the \mbox{optimal} model architecture and learning schedule we ran an extensive ablation study to test: deep supervision loss, focal loss, decoder attention, drop block, and residual connections. additionally, we have searched for the optimal depth of the u-net and number of convolutional channels. our solution was the winner of the challenge validation phase, with the normalized statistical ranking score of 0.267 and mean dice score of 0.8855 Michał Futrega, Alexandre Milesi, Michal Marcinkiewicz, Pablo Ribalta 2110.03352 link
End-To-End Supermask Pruning: Learning To Prune Image Captioning Models With the advancement of deep models, research work on image captioning has led to a remarkable gain in raw performance over the last decade, along with increasing model complexity and computational cost. however, surprisingly works on compression of deep networks for image captioning task has received little to no attention. for the first time in image captioning research, we provide an extensive comparison of various unstructured weight pruning methods on three different popular image captioning architectures, namely soft-attention, up-down and object relation transformer. following this, we propose a novel end-to-end weight pruning method that performs gradual sparsification based on weight sensitivity to the training loss. the pruning schemes are then extended with encoder pruning, where we show that conducting both decoder pruning and training simultaneously prior to the encoder pruning provides good overall performance. empirically, we show that an 80% to 95% sparse network (up to 75% reduction in model size) can either match or outperform its dense counterpart. the code and pre-trained models for up-down and object relation transformer that are capable of achieving cider scores >120 on the ms-coco dataset but with only 8.7 mb and 14.5 mb in model size (size reduction of 96% and 94% respectively against dense versions) are publicly available at https://github.com/jiahuei/sparse-image-captioning. Jia Huei Tan, Chee Seng Chan, Joon Huang Chuah 2110.03298 link
Propagating State Uncertainty Through Trajectory Forecasting Uncertainty pervades through the modern robotic autonomy stack, with nearly every component (e.g., sensors, detection, classification, tracking, behavior prediction) producing continuous or discrete probabilistic distributions. trajectory forecasting, in particular, is surrounded by uncertainty as its inputs are produced by (noisy) upstream perception and its outputs are predictions that are often probabilistic for use in downstream planning. however, most trajectory forecasting methods do not account for upstream uncertainty, instead taking only the most-likely values. as a result, perceptual uncertainties are not propagated through forecasting and predictions are frequently overconfident. to address this, we present a novel method for incorporating perceptual state uncertainty in trajectory forecasting, a key component of which is a new statistical distance-based loss function which encourages predicting uncertainties that better match upstream perception. we evaluate our approach both in illustrative simulations and on large-scale, real-world data, demonstrating its efficacy in propagating perceptual state uncertainty through prediction and producing more calibrated predictions. Boris Ivanovic, N/A Yifeng, N/A Lin, Shubham Shrivastava, Punarjay Chakravarty, Marco Pavone 2110.03267 link
Gradient Step Denoiser For Convergent Plug-And-Play Plug-and-play methods constitute a class of iterative algorithms for imaging problems where regularization is performed by an off-the-shelf denoiser. although plug-and-play methods can lead to tremendous visual performance for various image problems, the few existing convergence guarantees are based on unrealistic (or suboptimal) hypotheses on the denoiser, or limited to strongly convex data terms. in this work, we propose a new type of plug-and-play methods, based on half-quadratic splitting, for which the denoiser is realized as a gradient descent step on a functional parameterized by a deep neural network. exploiting convergence results for proximal gradient descent algorithms in the non-convex setting, we show that the proposed plug-and-play algorithm is a convergent iterative scheme that targets stationary points of an explicit global functional. besides, experiments show that it is possible to learn such a deep denoiser while not compromising the performance in comparison to other state-of-the-art deep denoisers used in plug-and-play schemes. we apply our proximal gradient algorithm to various ill-posed inverse problems, e.g. deblurring, super-resolution and inpainting. for all these applications, numerical results empirically confirm the convergence results. experiments also show that this new algorithm reaches state-of-the-art performance, both quantitatively and qualitatively. Samuel Hurault, Arthur Leclaire, Nicolas Papadakis 2110.03220 link
Treegcn-Ed: Encoding Point Cloud Using A Tree-Structured Graph Network Point cloud is an efficient way of representing and storing 3d geometric data. deep learning algorithms on point clouds are time and memory efficient. several methods such as pointnet and foldingnet have been proposed for processing point clouds. this work proposes an autoencoder based framework to generate robust embeddings for point clouds by utilizing hierarchical information using graph convolution. we perform multiple experiments to assess the quality of embeddings generated by the proposed encoder architecture and visualize the t-sne map to highlight its ability to distinguish between different object classes. we further demonstrate the applicability of the proposed framework in applications like: 3d point cloud completion and single image based 3d reconstruction. Prajwal Singh, Kaustubh Sadekar, Shanmuganathan Raman 2110.03170 link
Efficient Sharpness-Aware Minimization For Improved Training Of Neural Networks Overparametrized deep neural networks (dnns) often achieve astounding performances, but may potentially result in severe generalization error. recently, the relation between the sharpness of the loss landscape and the generalization error has been established by foret et al. (2020), in which the sharpness aware minimizer (sam) was proposed to mitigate the degradation of the generalization. unfortunately, sam s computational cost is roughly double that of base optimizers, such as stochastic gradient descent (sgd). this paper thus proposes efficient sharpness aware minimizer (esam), which boosts sam s efficiency at no cost to its generalization performance. esam includes two novel and efficient training strategies-stochasticweight perturbation and sharpness-sensitive data selection. in the former, the sharpness measure is approximated by perturbing a stochastically chosen set of weights in each iteration; in the latter, the sam loss is optimized using only a judiciously selected subset of data that is sensitive to the sharpness. we provide theoretical explanations as to why these strategies perform well. we also show, via extensive experiments on the cifar and imagenet datasets, that esam enhances the efficiency over sam from requiring 100% extra computations to 40% vis-a-vis base optimizers, while test accuracies are preserved or even improved. Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, Vincent Y. F. Tan 2110.03141 link
Speed+: Next Generation Dataset For Spacecraft Pose Estimation Across Domain Gap Autonomous vision-based spaceborne navigation is an enabling technology for future on-orbit servicing and space logistics missions. while computer vision in general has benefited from machine learning (ml), training and validating spaceborne ml models are extremely challenging due to the impracticality of acquiring a large-scale labeled dataset of images of the intended target in the space environment. existing datasets, such as spacecraft pose estimation dataset (speed), have so far mostly relied on synthetic images for both training and validation, which are easy to mass-produce but fail to resemble the visual features and illumination variability inherent to the target spaceborne images. in order to bridge the gap between the current practices and the intended applications in future space missions, this paper introduces speed+: the next generation spacecraft pose estimation dataset with specific emphasis on domain gap. in addition to 60,000 synthetic images for training, speed+ includes 9,531 simulated images of a spacecraft mockup model captured from the testbed for rendezvous and optical navigation (tron) facility. tron is a first-of-a-kind robotic testbed capable of capturing an arbitrary number of target images with accurate and maximally diverse pose labels and high-fidelity spaceborne illumination conditions. speed+ will be used in the upcoming international satellite pose estimation challenge co-hosted with the advanced concepts team of the european space agency to evaluate and compare the robustness of spaceborne ml models trained on synthetic images. Tae Ha Park, Marcus Märtens, Gurvan Lecuyer, Dario Izzo, Simone D'Amico 2110.03101 link
Large-Scale Topological Radar Localization Using Learned Descriptors In this work, we propose a method for large-scale topological localization based on radar scan images using learned descriptors. we present a simple yet efficient deep network architecture to compute a rotationally invariant discriminative global descriptor from a radar scan image. the performance and generalization ability of the proposed method is experimentally evaluated on two large scale driving datasets: mulran and oxford radar robotcar. additionally, we present a comparative evaluation of radar-based and lidar-based localization using learned global descriptors. our code and trained models are publicly available on the project website. Jacek Komorowski, Monika Wysoczanska, Tomasz Trzcinski 2110.03081 link
Deepbbs: Deep Best Buddies For Point Cloud Registration Recently, several deep learning approaches have been proposed for point cloud registration. these methods train a network to generate a representation that helps finding matching points in two 3d point clouds. finding good matches allows them to calculate the transformation between the point clouds accurately. two challenges of these techniques are dealing with occlusions and generalizing to objects of classes unseen during training. this work proposes deepbbs, a novel method for learning a representation that takes into account the best buddy distance between points during training. best buddies (i.e., mutual nearest neighbors) are pairs of points nearest to each other. the best buddies criterion is a strong indication for correct matches that, in turn, leads to accurate registration. our experiments show improved performance compared to previous methods. in particular, our learned representation leads to an accurate registration for partial shapes and in unseen categories. Itan Hezroni, Amnon Drory, Raja Giryes, Shai Avidan 2110.03016 link
Adaptive Unfolding Total Variation Network For Low-Light Image Enhancement Real-world low-light images suffer from two main degradations, namely, inevitable noise and poor visibility. since the noise exhibits different levels, its estimation has been implemented in recent works when enhancing low-light images from raw bayer space. when it comes to srgb color space, the noise estimation becomes more complicated due to the effect of the image processing pipeline. nevertheless, most existing enhancing algorithms in srgb space only focus on the low visibility problem or suppress the noise under a hypothetical noise level, leading them impractical due to the lack of robustness. to address this issue,we propose an adaptive unfolding total variation network (utvnet), which approximates the noise level from the real srgb low-light image by learning the balancing parameter in the model-based denoising method with total variation regularization. meanwhile, we learn the noise level map by unrolling the corresponding minimization process for providing the inferences of smoothness and fidelity constraints. guided by the noise level map, our utvnet can recover finer details and is more capable to suppress noise in real captured low-light scenes. extensive experiments on real-world low-light images clearly demonstrate the superior performance of utvnet over state-of-the-art methods. Chuanjun Zheng, Daming Shi, Wentian Shi 2110.00984 link
Student Helping Teacher: Teacher Evolution Via Self-Knowledge Distillation Knowledge distillation usually transfers the knowledge from a pre-trained cumbersome teacher network to a compact student network, which follows the classical teacher-teaching-student paradigm. based on this paradigm, previous methods mostly focus on how to efficiently train a better student network for deployment. different from the existing practices, in this paper, we propose a novel student-helping-teacher formula, teacher evolution via self-knowledge distillation (teskd), where the target teacher (for deployment) is learned with the help of multiple hierarchical students by sharing the structural backbone. the diverse feedback from multiple students allows the teacher to improve itself through the shared feature representations. the effectiveness of our proposed framework is demonstrated by extensive experiments with various network settings on two standard benchmarks including cifar-100 and imagenet. notably, when trained together with our proposed method, resnet-18 achieves 79.15% and 71.14% accuracy on cifar-100 and imagenet, outperforming the baseline results by 4.74% and 1.43%, respectively. the code is available at: https://github.com/zhengli427/teskd. Zheng Li, Xiang Li, Lingfeng Yang, Jian Yang, Zhigeng Pan 2110.00329 link
On Assessing The Usefulness Of Proxy Domains For Developing And Evaluating Embodied Agents In many situations it is either impossible or impractical to develop and evaluate agents entirely on the target domain on which they will be deployed. this is particularly true in robotics, where doing experiments on hardware is much more arduous than in simulation. this has become arguably more so in the case of learning-based agents. to this end, considerable recent effort has been devoted to developing increasingly realistic and higher fidelity simulators. however, we lack any principled way to evaluate how good a "proxy domain" is, specifically in terms of how useful it is in helping us achieve our end objective of building an agent that performs well in the target domain. in this work, we investigate methods to address this need. we begin by clearly separating two uses of proxy domains that are often conflated: 1) their ability to be a faithful predictor of agent performance and 2) their ability to be a useful tool for learning. in this paper, we attempt to clarify the role of proxy domains and establish new proxy usefulness (pu) metrics to compare the usefulness of different proxy domains. we propose the relative predictive pu to assess the predictive ability of a proxy domain and the learning pu to quantify the usefulness of a proxy as a tool to generate learning data. furthermore, we argue that the value of a proxy is conditioned on the task that it is being used to help solve. we demonstrate how these new metrics can be used to optimize parameters of the proxy domain for which obtaining ground truth via system identification is not trivial. Anthony Courchesne, Andrea Censi, Liam Paull 2109.14516 link
A Scaling Law For Synthetic-To-Real Transfer: How Much Is Your Pre-Training Effective? Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. the most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. however, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. how can we know that? in this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. by estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. we empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images. Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-Ichi Maeda, Kohei Hayashi 2108.11018 link
Aprel: A Library For Active Preference-Based Reward Learning Algorithms Reward learning is a fundamental problem in robotics to have robots that operate in alignment with what their human user wants. many preference-based learning algorithms and active querying techniques have been proposed as a solution to this problem. in this paper, we present aprel, a library for active preference-based reward learning algorithms, which enable researchers and practitioners to experiment with the existing techniques and easily develop their own algorithms for various modules of the problem. Erdem Bıyık, Aditi Talati, Dorsa Sadigh 2108.07259 link
Gait-Learning With Morphologically Evolving Robots Generated By L-System When controllers (brains) and morphologies (bodies) of robots simultaneously evolve, this can lead to a problem, namely the brain & body mismatch problem. in this research, we propose a solution of lifetime learning. we set up a system where modular robots can create offspring that inherit the bodies of parents by recombination and mutation. with regards to the brains of the offspring, we use two methods to create them. the first one entails solely evolution which means the brain of a robot child is inherited from its parents. the second approach is evolution plus learning which means the brain of a child is inherited as well, but additionally is developed by a learning algorithm - revdeknn. we compare these two methods by running experiments in a simulator called revolve and use efficiency, efficacy, and the morphology intelligence of the robots for the comparison. the experiments show that the evolution plus learning method does not only lead to a higher fitness level, but also to more morphologically evolving robots. this constitutes a quantitative demonstration that changes in the brain can induce changes in the body, leading to the concept of morphological intelligence, which is quantified by the learning delta, meaning the ability of a morphology to facilitate learning. Jie Luo, Daan Zeeuwe, Agoston E. Eiben 2107.08249 link
Align Before Fuse: Vision And Language Representation Learning With Momentum Distillation Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. in this paper, we introduce a contrastive loss to align the image and text representations before fusing (albef) them through cross-modal attention, which enables more grounded vision and language representation learning. unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. in order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. we provide a theoretical analysis of albef from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. albef achieves state-of-the-art performance on multiple downstream vision-language tasks. on image-text retrieval, albef outperforms methods that are pre-trained on orders of magnitude larger datasets. on vqa and nlvr$^2$, albef achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. code and pre-trained models are available at https://github.com/salesforce/albef/. Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, Steven Hoi 2107.07651 link
Llc: Accurate, Multi-Purpose Learnt Low-Dimensional Binary Codes Learning binary representations of instances and classes is a classical problem with several high potential applications. in modern settings, the compression of high-dimensional neural representations to low-dimensional binary codes is a challenging task and often require large bit-codes to be accurate. in this work, we propose a novel method for learning low-dimensional binary codes (llc) for instances as well as classes. our method does not require any side-information, like annotated attributes or label meta-data, and learns extremely low-dimensional binary codes (~20 bits for imagenet-1k). the learnt codes are super-efficient while still ensuring nearly optimal classification accuracy for resnet50 on imagenet-1k. we demonstrate that the learnt codes capture intrinsically important features in the data, by discovering an intuitive taxonomy over classes. we further quantitatively measure the quality of our codes by applying it to the efficient image retrieval as well as out-of-distribution (ood) detection problems. for imagenet-100 retrieval problem, our learnt binary codes outperform 16 bit hashnet using only 10 bits and also are as accurate as 10 dimensional real representations. finally, our learnt binary codes can perform ood detection, out-of-the-box, as accurately as a baseline that needs ~3000 samples to tune its threshold, while we require none. code is open-sourced at https://github.com/raivnlab/llc. Aditya Kusupati, Matthew Wallingford, Vivek Ramanujan, Raghav Somani, Jae Sung Park, Krishna Pillutla, Prateek Jain, Sham Kakade, Ali Farhadi 2106.01487 link
Agentformer: Agent-Aware Transformers For Socio-Temporal Multi-Agent Forecasting Predicting accurate future trajectories of multiple agents is essential for autonomous systems, but is challenging due to the complex agent interaction and the uncertainty in each agent's future behavior. forecasting multi-agent trajectories requires modeling two key dimensions: (1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others. most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model. this approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information. instead, we would prefer a method that allows an agent's state at one time to directly affect another agent's state at a future time. to this end, we propose a new transformer, agentformer, that jointly models the time and social dimensions. the model leverages a sequence representation of multi-agent trajectories by flattening trajectory features across time and agents. since standard attention operations disregard the agent identity of each element in the sequence, agentformer uses a novel agent-aware attention mechanism that preserves agent identities by attending to elements of the same agent differently than elements of other agents. based on agentformer, we propose a stochastic multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep when inferring an agent's future position. the latent intent of all agents is also jointly modeled, allowing the stochasticity in one agent's behavior to affect other agents. our method substantially improves the state of the art on well-established pedestrian and autonomous driving datasets. Ye Yuan, Xinshuo Weng, Yanglan Ou, Kris Kitani 2103.14023 link
What'S In My Lidar Odometry Toolbox? With the democratization of 3d lidar sensors, precise lidar odometries and slam are in high demand. new methods regularly appear, proposing solutions ranging from small variations in classical algorithms to radically new paradigms based on deep learning. yet it is often difficult to compare these methods, notably due to the few datasets on which the methods can be evaluated and compared. furthermore, their weaknesses are rarely examined, often letting the user discover the hard way whether a method would be appropriate for a use case. in this paper, we review and organize the main 3d lidar odometries into distinct categories. we implemented several approaches (geometric based, deep learning based, and hybrid methods) to conduct an in-depth analysis of their strengths and weaknesses on multiple datasets, guiding the reader through the different lidar odometries available. implementation of the methods has been made publicly available at https://github.com/kitware/pylidar-slam. Pierre Dellenbach, Jean-Emmanuel Deschaud, Bastien Jacquet, François Goulette 2103.09708 link
Efficient Two-Stream Network For Violence Detection Using Separable Convolutional Lstm Automatically detecting violence from surveillance footage is a subset of activity recognition that deserves special attention because of its wide applicability in unmanned security monitoring systems, internet video filtration, etc. in this work, we propose an efficient two-stream deep learning architecture leveraging separable convolutional lstm (sepconvlstm) and pre-trained mobilenet where one stream takes in background suppressed frames as inputs and other stream processes difference of adjacent frames. we employed simple and fast input pre-processing techniques that highlight the moving objects in the frames by suppressing non-moving backgrounds and capture the motion in-between frames. as violent actions are mostly characterized by body movements these inputs help produce discriminative features. sepconvlstm is constructed by replacing convolution operation at each gate of convlstm with a depthwise separable convolution that enables producing robust long-range spatio-temporal features while using substantially fewer parameters. we experimented with three fusion methods to combine the output feature maps of the two streams. evaluation of the proposed methods was done on three standard public datasets. our model outperforms the accuracy on the larger and more challenging rwf-2000 dataset by more than a 2% margin while matching state-of-the-art results on the smaller datasets. our experiments lead us to conclude, the proposed models are superior in terms of both computational efficiency and detection accuracy. Zahidul Islam, Mohammad Rukonuzzaman, Raiyan Ahmed, Md. Hasanul Kabir, Moshiur Farazi 2102.10590 link
Full-Glow: Fully Conditional Glow For More Realistic Image Generation Autonomous agents, such as driverless cars, require large amounts of labeled visual data for their training. a viable approach for acquiring such data is training a generative model with collected real data, and then augmenting the collected real dataset with synthetic images from the model, generated with control of the scene layout and ground truth labeling. in this paper we propose full-glow, a fully conditional glow-based architecture for generating plausible and realistic images of novel street scenes given a semantic segmentation map indicating the scene layout. benchmark comparisons show our model to outperform recent works in terms of the semantic segmentation performance of a pretrained pspnet. this indicates that images from our model are, to a higher degree than from other models, similar to real images of the same kinds of scenes and objects, making them suitable as training data for a visual semantic segmentation or object recognition system. Moein Sorkhei, Gustav Eje Henter, Hedvig Kjellström 2012.05846 link
Multi-Agent Reinforcement Learning For Visibility-Based Persistent Monitoring The visibility-based persistent monitoring (vpm) problem seeks to find a set of trajectories (or controllers) for robots to persistently monitor a changing environment. each robot has a sensor, such as a camera, with a limited field-of-view that is obstructed by obstacles in the environment. the robots may need to coordinate with each other to ensure no point in the environment is left unmonitored for long periods of time. we model the problem such that there is a penalty that accrues every time step if a point is left unmonitored. however, the dynamics of the penalty are unknown to us. we present a multi-agent reinforcement learning (marl) algorithm for the vpm problem. specifically, we present a multi-agent graph attention proximal policy optimization (ma-g-ppo) algorithm that takes as input the local observations of all agents combined with a low resolution global map to learn a policy for each agent. the graph attention allows agents to share their information with others leading to an effective joint policy. our main focus is to understand how effective marl is for the vpm problem. we investigate five research questions with this broader goal. we find that ma-g-ppo is able to learn a better policy than the non-rl baseline in most cases, the effectiveness depends on agents sharing information with each other, and the policy learnt shows emergent behavior for the agents. Jingxi Chen, Amrish Baskaran, Zhongshun Zhang, Pratap Tokekar 2011.01129 link
Light Field Salient Object Detection: A Review And Benchmark Salient object detection (sod) is a long-standing research topic in computer vision and has drawn an increasing amount of research interest in the past decade. this paper provides the first comprehensive review and benchmark for light field sod, which has long been lacking in the saliency community. firstly, we introduce preliminary knowledge on light fields, including theory and data forms, and then review existing studies on light field sod, covering ten traditional models, seven deep learning-based models, one comparative study, and one brief review. existing datasets for light field sod are also summarized with detailed information and statistical analyses. secondly, we benchmark nine representative light field sod models together with several cutting-edge rgb-d sod models on four widely used light field datasets, from which insightful discussions and analyses, including a comparison between light field sod and rgb-d sod models, are achieved. besides, due to the inconsistency of datasets in their current forms, we further generate complete data and supplement focal stacks, depth maps and multi-view images for the inconsistent datasets, making them consistent and unified. our supplemental data makes a universal benchmark possible. lastly, because light field sod is quite a special problem attributed to its diverse data representations and high dependency on acquisition hardware, making it differ greatly from other saliency detection tasks, we provide nine hints into the challenges and future directions, and outline several open issues. we hope our review and benchmarking could help advance research in this field. all the materials including collected models, datasets, benchmarking results, and supplemented light field datasets will be publicly available on our project site https://github.com/kerenfu/lfsod-survey. Keren Fu, Yao Jiang, Ge-Peng Ji, Tao Zhou, Qijun Zhao, Deng-Ping Fan 2010.04968 link
Artificial Fingerprinting For Generative Models: Rooting Deepfake Attribution In Training Data Photorealistic image generation has reached a new level of quality due to the breakthroughs of generative adversarial networks (gans). yet, the dark side of such deepfakes, the malicious use of generated media, raises concerns about visual misinformation. while existing research work on deepfake detection demonstrates high accuracy, it is subject to advances in generation techniques and adversarial iterations on detection countermeasure techniques. thus, we seek a proactive and sustainable solution on deepfake detection, that is agnostic to the evolution of generative models, by introducing artificial fingerprints into the models. our approach is simple and effective. we first embed artificial fingerprints into training data, then validate a surprising discovery on the transferability of such fingerprints from training data to generative models, which in turn appears in the generated deepfakes. experiments show that our fingerprinting solution (1) holds for a variety of cutting-edge generative models, (2) leads to a negligible side effect on generation quality, (3) stays robust against image-level and model-level perturbations, (4) stays hard to be detected by adversaries, and (5) converts deepfake detection and attribution into trivial tasks and outperforms the recent state-of-the-art baselines. our solution closes the responsibility loop between publishing pre-trained generative model inventions and their possible misuses, which makes it independent of the current arms race. Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, Mario Fritz 2007.08457 link
Universal Graph Transformer Self-Attention Networks The transformer has been extensively used in research domains such as computer vision, image processing, and natural language processing. the transformer, however, has not been actively used in graph neural networks. to this end, we introduce a transformer-based advanced gnn model, named ugformer, to learn graph representations. in particular, given an input graph, we present two ugformer variants. the first variant is to leverage the transformer on a set of sampled neighbors for each node, while the second is to leverage the transformer directly on the input graph. experimental results demonstrate that these two ugformer variants achieve state-of-the-art accuracies on well-known benchmark datasets for graph classification and inductive text classification, respectively. the code is available on github: \url{https://github.com/daiquocnguyen/graph-transformer}. Dai Quoc Nguyen, Tu Dinh Nguyen, Dinh Phung 1909.11855 link
You might also like...
A custom DeepStack model for detecting 16 human actions.
A custom DeepStack model for detecting 16 human actions.

DeepStack_ActionNET This repository provides a custom DeepStack model that has been trained and can be used for creating a new object detection API fo

HeatNet is a python package that provides tools to build, train and evaluate neural networks designed to predict extreme heat wave events globally on daily to subseasonal timescales.

HeatNet HeatNet is a python package that provides tools to build, train and evaluate neural networks designed to predict extreme heat wave events glob

An experiment on the performance of homemade Q-learning AIs in Agar.io depending on their state representation and available actions
An experiment on the performance of homemade Q-learning AIs in Agar.io depending on their state representation and available actions

Agar.io_Q-Learning_AI An experiment on the performance of homemade Q-learning AIs in Agar.io depending on their state representation and available act

Wordle Env: A Daily Word Environment for Reinforcement Learning

Wordle Env: A Daily Word Environment for Reinforcement Learning Setup Steps: git pull [email protected]:alex-nooj/wordle_env.git From the wordle_env dire

A decent AI that solves daily Wordle puzzles. Works with different websites with similar wordlists,.

Wordle-AI A decent AI that solves daily "Wordle" puzzles. Works with different websites with similar wordlists. When prompted with "Word:" enter the w

Citation Intent Classification in scientific papers using the Scicite dataset an Pytorch

Citation Intent Classification Table of Contents About the Project Built With Installation Usage Acknowledgments About The Project Citation Intent Cla

ByteTrack(Multi-Object Tracking by Associating Every Detection Box)のPythonでのONNX推論サンプル

ByteTrack-ONNX-Sample ByteTrack(Multi-Object Tracking by Associating Every Detection Box)のPythonでのONNX推論サンプルです。 ONNXに変換したモデルも同梱しています。 変換自体を試したい方はByteT

A python bot to move your mouse every few seconds to appear active on Skype, Teams or Zoom as you go AFK. 🐭 🤖
A python bot to move your mouse every few seconds to appear active on Skype, Teams or Zoom as you go AFK. 🐭 🤖

PyMouseBot If you're from GT and annoyed with SGVPN idle timeouts while working on development laptop, You might find this useful. A python cli bot to

Very simple NCHW and NHWC conversion tool for ONNX. Change to the specified input order for each and every input OP. Also, change the channel order of RGB and BGR. Simple Channel Converter for ONNX.
Very simple NCHW and NHWC conversion tool for ONNX. Change to the specified input order for each and every input OP. Also, change the channel order of RGB and BGR. Simple Channel Converter for ONNX.

scc4onnx Very simple NCHW and NHWC conversion tool for ONNX. Change to the specified input order for each and every input OP. Also, change the channel

Owner
Realcat
⭐️Focusing on SLAM, SfM and Visual Localization.
Realcat
The tl;dr on a few notable transformer/language model papers + other papers (alignment, memorization, etc).

The tl;dr on a few notable transformer/language model papers + other papers (alignment, memorization, etc).

Will Thompson 166 Jan 4, 2023
arxiv-sanity, but very lite, simply providing the core value proposition of the ability to tag arxiv papers of interest and have the program recommend similar papers.

arxiv-sanity, but very lite, simply providing the core value proposition of the ability to tag arxiv papers of interest and have the program recommend similar papers.

Andrej 671 Dec 31, 2022
Code I use to automatically update my videos' metadata on YouTube

mCodingYouTube This repository contains the code I use to automatically update my videos' metadata on YouTube, including: titles, descriptions, tags,

James Murphy 19 Oct 7, 2022
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
Implementation of 'lightweight' GAN, proposed in ICLR 2021, in Pytorch. High resolution image generations that can be trained within a day or two

512x512 flowers after 12 hours of training, 1 gpu 256x256 flowers after 12 hours of training, 1 gpu Pizza 'Lightweight' GAN Implementation of 'lightwe

Phil Wang 1.5k Jan 2, 2023
A community run, 5-day PyTorch Deep Learning Bootcamp

Deep Learning Winter School, November 2107. Tel Aviv Deep Learning Bootcamp : http://deep-ml.com. About Tel-Aviv Deep Learning Bootcamp is an intensiv

Shlomo Kashani. 1.3k Sep 4, 2021
Where2Act: From Pixels to Actions for Articulated 3D Objects

Where2Act: From Pixels to Actions for Articulated 3D Objects The Proposed Where2Act Task. Given as input an articulated 3D object, we learn to propose

Kaichun Mo 69 Nov 28, 2022
Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.

actions-includes Allows including an action inside another action (by preprocessing the Yaml file). Instead of using uses or run in your action step,

Tim Ansell 70 Nov 4, 2022
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (CVPR2021)

NExT-QA We reproduce some SOTA VideoQA methods to provide benchmark results for our NExT-QA dataset accepted to CVPR2021 (with 1 'Strong Accept' and 2

Junbin Xiao 50 Nov 24, 2022
Alex Pashevich 62 Dec 24, 2022