Avoid PyCharm Overloading ========================= This speeds up PyCharm's package index processes and avoids CPU & memory overloading. On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces PyCharm's package scanning from 15 minutes to 45 seconds. Why? ---- When PyCharm has multiple conda environments, it will sometimes launch concurrent package scanning across those environments. To do so, it uses the `conda` cli, and a tool called conda_packaging_tool.py, available from the JetBrains intellij-community github repository (see below). Unfortunately, these tools do not behave well when launched in parallel, as they tend to overload CPU & memory in this case. How? ---- To avoid stalling these processes, we modify their sources as follows: 1. modify the conda cli $ conda activate base $ pip install runfast $ nano `which conda` Modify the main section: if __name__ == '__main__': from conda.cli import main from runfast import cached cached(main) 2. modify the conda_packaging_tool.py if __name__ == '__main__': from runfast import cached cached(main) How does this work? ------------------- Two simple steps: 1. When called in parallel, only one process is allowed to proceed. 2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute, given the same command line parameters That is, when PyCharm launches 5 package scanning (`conda list`) commands, only one of them will run immediately. If some of the scans are for the same environment, only one of them will actually run, while the others simply return the same output. How to clear the cache or avoid caching at all? $ export RUNFAST_NOCACHE=1 Testing ------- a) conda packaging tool # first time $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py real 0m13.271s user 0m9.631s sys 0m1.936s # second time $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py real 0m3.159s user 0m0.139s sys 0m1.298s b) conda cli # first time $ time conda list -p /path/to/env real 0m7.985s user 0m7.883s sys 0m0.087s # second time real 0m0.152s user 0m0.116s sys 0m0.032s References ---------- * https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py * https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
This speeds up PyCharm's package index processes and avoids CPU & memory overloading
Overview
You might also like...
python's memory-saving dictionary data structure
ConstDict python代替的Dict数据结构 若字典不会增加字段,只读/原字段修改 使用ConstDict可节省内存 Dict()内存主要消耗的地方: 1、Dict扩容机制,预留内存空间 2、Dict也是一个对象,内部会动态维护__dict__,增加slot类属性可以节省内容 节省内存大小
Demo of using DataLoader to prevent out of memory
Demo of using DataLoader to prevent out of memory
Library for Memory Trace Statistics in Python
Memory Search Library for Memory Trace Statistics in Python The library uses tracemalloc as a core module, which is why it is only available for Pytho
Holographic Declarative Memory for Python ACT-R
HDM This is the repository for the Holographic Declarative Memory (HDM) module for Python ACT-R. This repository contains: documentation: a paper, con
GWCelery is a simple and reliable package for annotating and orchestrating LIGO/Virgo alerts
GWCelery is a simple and reliable package for annotating and orchestrating LIGO/Virgo alerts, built from widely used open source components.
A Lite Package focuses on making overwrite and mending functions easier and more flexible.
Overwrite Make Overwrite More flexible In Python A Lite Package focuses on making overwrite and mending functions easier and more flexible. Certain Me
A python package to manage the stored receiver-side Strain Green's Tensor (SGT) database of 3D background models and able to generate Green's function and synthetic waveform
A python package to manage the stored receiver-side Strain Green's Tensor (SGT) database of 3D background models and able to generate Green's function and synthetic waveform
A deployer and package manager for OceanBase open-source software.
OceanBase Deploy OceanBase Deploy (简称 OBD)是 OceanBase 开源软件的安装部署工具。OBD 同时也是包管理器,可以用来管理 OceanBase 所有的开源软件。本文介绍如何安装 OBD、使用 OBD 和 OBD 的命令。 安装 OBD 您可以使用以下方
Hydralit package is a wrapping and template project to combine multiple independant Streamlit applications into a multi-page application.
Hydralit The Hydralit package is a wrapping and template project to combine multiple independant (or somewhat dependant) Streamlit applications into a
Run python scripts and pass data between multiple python and node processes using this npm module
Run python scripts and pass data between multiple python and node processes using this npm module. process-communication has a event based architecture for interacting with python data and errors inside nodejs.
Module for remote in-memory Python package/module loading through HTTP/S
httpimport Python's missing feature! The feature has been suggested in Python Mailing List Remote, in-memory Python package/module importing through H
GMHI: Gut Microbiome Health Index
GMHI: Gut Microbiome Health Index Description Gut Microbiome Health Index (GMHI)
BMI-Calculator: Program to Calculate Body Mass Index (BMI)
The Body Mass Index (BMI) or Quetelet index is a value derived from the mass (weight) and height of an individual, male or female.
Socorro is the Mozilla crash ingestion pipeline. It accepts and processes Breakpad-style crash reports. It provides analysis tools.
Socorro Socorro is a Mozilla-centric ingestion pipeline and analysis tools for crash reports using the Breakpad libraries. Support This is a Mozilla-s
A python tool for synchronizing the messages from different threads, processes, or hosts.
Sync-stream This project is designed for providing the synchoronization of the stdout / stderr among different threads, processes, devices or hosts.
Used the pyautogui library to automate some processes on the computer
Pyautogui Utilizei a biblioteca pyautogui para automatizar alguns processos no c
Return-Parity-MDP - Towards Return Parity in Markov Decision Processes
Towards Return Parity in Markov Decision Processes Code for the AISTATS 2022 pap
A test repository to build a python package and publish the package to Artifact Registry using GCB
A test repository to build a python package and publish the package to Artifact Registry using GCB. Then have the package be a dependency in a GCF function.
A python script developed to process Windows memory images based on triage type.
Overview A python script developed to process Windows memory images based on triage type. Requirements Python3 Bulk Extractor Volatility2 with Communi