Conda package for artifact creation that enables offline environments. Ideal for air-gapped deployments.

Related tags

GPU Utilities conda-vendor
Overview

Conda-Vendor

Conda Vendor is a tool to create local conda channels and manifests for vendored deployments

Installation

To install with pip, run:

pip install conda-vendor

Usage

Conda-vendor has two main steps to create a local channel. First, a meta-manifest file is created as an intermediate artifact. With an existing meta-manifest file, a local conda channel can then be created.

The intermediate meta-manifest is generated to allow for the creation of custom software manifests. These manifests can then be used obtain package approval from an organization's cybersecurity team.

Creating a Meta-manifest

Conda-vendor solves an environment with conda from an environment.yaml and determines all the packages that are required. The metadata for these required packages is stored in a file called meta_manifest.yaml. To create this file, run:

conda vendor create-meta-manifest --environment-yaml environment.yaml

The above command will output a meta_manifest.yaml file in the current directory.

Creating a Local Channel

With a meta-manifest file created, conda-vendor can then create local channels.

conda vendor create-channels --meta-manifest-path ./meta_manifest.yaml

This will create a directory called local_channel that will contain the same number of channels as were listed in the original environment.yaml file. These local channels will only contain the packages that are needed to satisfy the solved environment from the create-meta-manifest step.

Using the Local channel

There are several ways to use the local channel. If python was in the input environment.yaml file for example, the following could be used:

conda create -n test_env python -c <path_to_local_channel> --offline

The --offline flag will prevent conda from reaching out to the internet for packages. To verify that the environment created only contains packages contained in the local channel, run the following:

conda activate test_env
conda list --explicit

This should show a list of all the packages in the environment the local paths to their source code (typically tar.bz2 files).

Creating Environment with all Packages from Input Environment.yaml

To generate a conda environment yaml that contains all the packages from the input environment.yaml, run the following:

conda vendor create-local-yaml --meta-manifest-path ./meta_manifest.yaml --channel-root <absolute_path_to_local_channel_dir>

This will create a environment file inside the local_channel directory called local_conda-vendor-env.yaml. An environment can then be created with:

conda env create -f local_channel/local_conda-vendor-env.yaml

The environment will be created with the packages that are contained in the local channel.

Creating a Custom Manifest for Package Security Validation

The following functionality is only applicable if there is an organization that requires a list of packages for security validation. Currently the Iron Bank format is supported, but support for other formats can be added to the source code in custom_manifest.py.

To generate an iron bank manifest from the meta-manifest, run:

conda vendor create-custom-manifest --meta-manifest-path ./meta-manifest.yaml --output-manifest-path ./custom_manifest.yaml

This will output a manifest file in the Iron Bank format.

Comments
  • Conda lock import changes + environment signing/verification

    Conda lock import changes + environment signing/verification

    Upstream conda-lock has some module changes in 1.0.3 that break the current version of conda-vendor.

    • dependencies are now returned as a List(VersionedDependency) instead of List(str)
    • Channels are now returned as Channel instead of str Fixes https://github.com/MetroStar/conda-vendor/issues/33

    EDIT: Adding signing + verification for generated conda environments

    • [ ] add cosign signing and verification to vendored conda environments
    • [ ] add cosign conda environment signature to meta-manifest
    • [ ] add in-toto attestation spec for manifest format?
    opened by rigzba21 5
  • conda-vendor improvements for version 1.0.0

    conda-vendor improvements for version 1.0.0

    Improved CLI user experience:

    Background/Overview

    conda-lock 1.x has significant updates/improvements that require changes to conda-vendor

    micromamba now uses the libmamba and libmambapy solver as default, so the old conda-vendor + old conda-lock combination produced channels that had issues solving in our IronBank containers when using up-to-date versions of micromamba.

    • [x] Remove meta-manifest intermediary step in favor of conda-lock's new 1.0.x API and lockfile format
    • [x] Make vendor the primary command
    • [x] Add flag for different solvers: conda, mamba, micromamba based on conda-lock's new 1.0.x API for solve_specs_for_arch
    • [x] Add a --dry-run option to generate formatted JSON of conda-lock's solved FetchActions object. This replaces the old meta-manifest generation intermediary step and can be piped to other SBOM tools that accept JSON.
    • [x] IronBank hardening_manifest.yaml resources formatted text output, using filtered + merged FETCH actions in conda-lock's FetchActions object
    • [x] click cli integration tests
    • [x] update test fixtures to use conda-lock's FetchActions, LockSpecification, and VersionedDependency objects
    • [x] Update GitHub Actions to include dependencies for integration tests

    New Usage for version 1.0.0:

    # use conda as the solver for linux-64
conda-vendor vendor --file environment.yaml --solver conda --platform linux-64

# use mamba as the solver for osx-64
conda-vendor vendor --file environment.yaml --solver mamba --platform osx-64

# use micromamba as the solver for the host platform
conda-vendor vendor --file environment.yaml --solver micromamba

# dry-run outputs formatted JSON 
conda-vendor vendor --file environment.yaml --solver mamba --platform linux-64 --dry-run True

    Screenshots:

    Conda-vendor Improved UX:

    image

    repodata.json hotfix progress bar and package download progress bar:

    image

    Dry-Run formatted JSON output:

    image

    ironbank-gen subcommand (returns formatted text that can be copied into Ironbank's hardening manifest):

    image

    Example IronBank Workflow:

    1) Generate the vendored channel and output the IronBank Hardening Manifest resources to stdout:

    conda-vendor vendor --file my-environment.yaml --solver micromamba --platform linux-64 --ironbank-gen True

    2) Copy the output resources block to your IronBank hardening_manifest.yaml:

    - url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
  filename: micromamba-0.22.0-0.tar.bz2
  validation:
    type: sha256
    value: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
- url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
  filename: _libgcc_mutex-0.1-conda_forge.tar.bz2
  validation:
    type: sha256
    value: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
- url: https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
  filename: libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
  validation:
    type: sha256
    value: cc84f71bb9dbecde453a25ba8c5aefc9773da5d619633c103eb8bac1ab4afda0

    3) Copy over your patched repodata.json files from your vendored channel to the appropriate directory in your IronBank Project

    documentation enhancement 
    opened by rigzba21 3
  • Enhancement: ability to vendor multiple environment.yaml files into a single local-channel

    Enhancement: ability to vendor multiple environment.yaml files into a single local-channel

    I ran into a situation where I needed to create two separate conda environments, each with different versions of the same package but solvable from a single conda-vendor'ed local channel.

    Example: environment-1.yaml:

    name: environment-one
channels:
  - conda-forge
dependencies:
  - python=3.9.7
  - pip
  - gxx_linux-64=9.4.0

    environment-2.yaml:

    name: environment-two
channels:
  - conda-forge
dependencies:
  - python=3.9.7
  - pip
  - gxx_linux-64=11.2.0

    My current workaround involved creating two separate conda-vendor'ed local-channels for each environment.yaml, then manually combining the .tar.bz2 packages into one local-channel, deleting the repodata.json files, re-indexing that channel with conda index ., and then manually merging the meta-manifest files into one (removing any duplicate entries).

    It would be awesome to have the ability to create a single local channel from multiple environment.yaml files without the workaround!

    opened by rigzba21 2
  • Use pathlib in conda_vendor.py, fix issue 38

    Use pathlib in conda_vendor.py, fix issue 38

    Fixes #38

    I believe this fixes the issue that was causing trouble last week. While working through the fix, I took's @kcpevey suggestion and converted path-like objects from str to pathlib.Path objects. This did also require swapping out the tmpdir fixture for tmp_path_factory, the latter of which returns a pathlib.Path object.

    opened by iameskild 1
  • setting a default value filename for --output-manifest-path

    setting a default value filename for --output-manifest-path

    Setting the default value to for the option --output-manifest-path to ./output_manifest.yaml preventing an error if the --output-manifest-path flag is not given.

    Fixes https://github.com/MetroStar/conda-vendor/issues/21

    opened by rigzba21 1
  • Excluding `--output-manifest-path` results in error when trying to create a custom manifest

    Excluding `--output-manifest-path` results in error when trying to create a custom manifest

    Description

    Given a generated meta_manifest.yaml file that is passed as an option to the create-custom-manifest command, but excluding the --output-manifest-path option, produces an error.

    Help Message for create-custom-manifest sub-command:

    conda vendor create-custom-manifest --help
Usage: conda-vendor create-custom-manifest [OPTIONS]

  custom manifest from meta-manifest file

Options:
  -v, --verbose                verbose logging
  --manifest-type TEXT         type of custom manifest to create
  --meta-manifest-path TEXT    path to meta manifest file
  --output-manifest-path TEXT  output manifest path
  --help                       Show this message and exit.

    Attempted Usage:

    conda vendor create-custom-manifest --meta-manifest-path meta_manifest.yaml

    results in the following error:

    INFO:conda_vendor.custom_manifest:Input Manifest : /Users/rigzba21/meta_manifest.yaml
INFO:conda_vendor.custom_manifest:Output Custom Manifest : /Users/rigzba21
Traceback (most recent call last):
  File "/Users/rigzba21/mambaforge/bin/conda-vendor", line 10, in <module>
    sys.exit(cli())
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/conda_vendor/__main__.py", line 104, in create_custom_manifest
    create_ironbank_from_meta_manifest(
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/conda_vendor/cli.py", line 26, in create_ironbank_from_meta_manifest
    custom_manifest.write_custom_manifest(output_manifest_dir)
  File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/conda_vendor/custom_manifest.py", line 40, in write_custom_manifest
    with open(output_file_path, "w") as f:
IsADirectoryError: [Errno 21] Is a directory: '.'

    conda vendor --version:

    conda-vendor, version 0.1.8

    Discussion Points

    It is not clear which options are required for the create-custom-manifest sub-command. Should we update the --help message output to show required options? Should the create-custom-manifest sub-command generate a custom manifest file with a default name, instead of requiring the --output-manifest-path option?

    opened by rigzba21 1
  • [BUG] `--offline` mode failing due to issues with file location

    [BUG] `--offline` mode failing due to issues with file location

    Due to a potential upstream change, the conda-vendor --offline mode failed to build because of issues handling paths related to repodata.json. Adding a test to ensure --offline mode is working as expected might catch some of these changes.

    Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/conda/exceptions.py", line 1082, in __call__
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/conda_env/cli/main.py", line 80, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/opt/conda/lib/python3.9/site-packages/conda_env/cli/main_create.py", line 142, in execute
    result[installer_type] = installer.install(prefix, pkg_specs, args, env)
  File "/opt/conda/lib/python3.9/site-packages/conda_env/installers/conda.py", line 50, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
  File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 195, in solve_for_diff
    final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
  File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 300, in solve_final_state
    ssc = self._collect_all_metadata(ssc)
  File "/opt/conda/lib/python3.9/site-packages/conda/common/io.py", line 88, in decorated
    return f(*args, **kwds)
  File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 463, in _collect_all_metadata
    index, r = self._prepare(prepared_specs)
  File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 1058, in _prepare
    reduced_index = get_reduced_index(self.prefix, self.channels,
  File "/opt/conda/lib/python3.9/site-packages/conda/core/index.py", line 288, in get_reduced_index
    new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
  File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 140, in query_all
    result = tuple(concat(executor.map(subdir_query, channel_urls)))
  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/opt/conda/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 132, in <lambda>
    subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
  File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 145, in query
    self.load()
  File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 210, in load
    _internal_state = self._load()
  File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 407, in _load
    _internal_state = self._process_raw_repodata_str(raw_repodata_str)
  File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 485, in _process_raw_repodata_str
    assert subdir == self.channel.subdir
AssertionError
    opened by iameskild 0
  • Ruamel dependency

    Ruamel dependency

    • pyyaml silently allows for duplicate keys in yaml files. This was causing an error for some users. Switch dependency to ruamel which throws an error on duplicate keys in yaml files.
    opened by tylerpotts 0
  • Fix bug for windows

    Fix bug for windows

    Fixes bug where windows is referenced as "windows-64" and "windows-32" when it should be "win-64" and "win-32"

    opened by tylerpotts 0
  • Add custom platform option to conda vendor. Black some files, add tests.

    Add custom platform option to conda vendor. Black some files, add tests.

    Fixes https://github.com/MetroStar/conda-vendor/issues/19

    • bump version to 0.1.9
    • add option in create-meta-manifest to create a channel for a platform different than the one currently being used. Supported platforms include: ['linux-64', 'linux-32', 'windows-64', 'windows-32', 'osx-64']
    • black formatting on some additional files
    opened by tylerpotts 0
  • bug fix

    bug fix "packages.conda" not in repodata.json

    We had a bug in conda-vendor where the vendored repodata.json is not correct when the packages should live in the "packages.conda" subdictionary. I've attached a quick fix.

    incidentally - bumped version needed conda-lock >= 1.2 to work on my mac.

    opened by RR-USAF 0
  • no configuration of requests available

    no configuration of requests available

    Hi,

    as far as I can see there is no possibility to run the code with verify=False for requests.get such that no ssl verification is done. Or alternatively configure the ssl certificate path. I would really appretiate such functionality.

    Kind regards Juri

    opened by j-merger 0
  • Support Multiple Environment Files

    Support Multiple Environment Files

    Using conda-lock's compound specification it would be nice to have the ability to pass in a list of environment files to vendor in as a single conda channel. example potential usage:

    conda-vendor vendor --file environment-one.yaml --file environment-two.yaml --file environment-three.yaml --solver mamba --platform linux-64
    enhancement help wanted 
    opened by rigzba21 0
  • IronBank Scripts Integration

    IronBank Scripts Integration

    opened by rigzba21 1
  • Proposed conda-vendor improvements

    Proposed conda-vendor improvements

    Edit: Closed https://github.com/MetroStar/conda-vendor/pull/32 in favor of tracking progress here as this is a much bigger refactor.

    Background

    conda-lock has some awesome improvements in 1.x 🔭 that will allow us to reduce duplicated functionality in conda-vendor's implementation.

    Example conda-lock usage for 1.x:

    Given an environment.yaml:

    name: minimal
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - micromamba

    Generating a lockfile (conda-lock supports multiple solvers such as mamba and micromamba): conda lock --file environment.yaml -p linux-64 --mamba

    Produces the following conda-lock.yml:

    # This lock file was generated by conda-lock (https://github.com/conda-incubator/conda-lock). DO NOT EDIT!
#
# A "lock file" contains a concrete list of package versions (with checksums) to be installed. Unlike
# e.g. `conda env create`, the resulting environment will not change as new package versions become
# available, unless you explicitly update the lock file.
#
# Install this environment as "YOURENV" with:
#     conda-lock install -n YOURENV --file conda-lock.yml
# To update a single package to the latest version compatible with the version constraints in the source:
#     conda-lock lock --lockfile conda-lock.yml --update PACKAGE
# To re-solve the entire environment, e.g. after changing a version constraint in the source file:
#     conda-lock -f environment.yaml --lockfile conda-lock.yml
metadata:
  channels:
  - url: conda-forge
    used_env_vars: []
  content_hash:
    linux-64: 0e3f55cef4b08ecec24c4dda8e5ce0617f9f55f4e457bca0d9c16c9d8fe00bc5
  platforms:
  - linux-64
  sources:
  - environment.yaml
package:
- category: main
  dependencies: {}
  hash:
    md5: d7c89558ba9fa0495403155b64376d81
    sha256: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
  manager: conda
  name: _libgcc_mutex
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
  version: '0.1'
- category: main
  dependencies: {}
  hash:
    md5: 575611b8a84f45960e87722eeb51fa26
    sha256: d13c8774129e0d8d1427f5758fba53cfa915b6a12cd4dbd2bfe612d9eab0506d
  manager: conda
  name: ca-certificates
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2021.10.8-ha878542_0.tar.bz2
  version: 2021.10.8
- category: main
  dependencies: {}
  hash:
    md5: bd4f2e711b39af170e7ff15163fe87ee
    sha256: ad7985a9ff622880cf87c42db1ffe2dfb040d8175c1bb352fc8f3705c7e0962f
  manager: conda
  name: ld_impl_linux-64
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.36.1-hea4e1c9_2.tar.bz2
  version: 2.36.1
- category: main
  dependencies: {}
  hash:
    md5: 24072cb5ef3fa80347bd35f184dfdaed
    sha256: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
  manager: conda
  name: micromamba
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
  version: 0.22.0
- category: main
  dependencies: {}
  hash:
    md5: 84be5301069417a2221187d2f435e0f7
    sha256: 74d8c1fbccae1a78c9bd2b2d1cda73df425cc28717a637198c23bd1c9b53b60e
  manager: conda
  name: tzdata
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/tzdata-2022a-h191b570_0.tar.bz2
  version: 2022a
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
  hash:
    md5: a77fb1a92411cb8d979de1c2d81dd210
    sha256: 1da28d8d10c93e43c78fb5020dd9022fe24687f759acc25de699185bdfa84e9b
  manager: conda
  name: libgomp
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgomp-11.2.0-h1d223b6_14.tar.bz2
  version: 11.2.0
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
    libgomp: '>=7.5.0'
  hash:
    md5: 561e277319a41d4f24f5c05a9ef63c04
    sha256: 81c74d38c80345e195106dc3a5b4063b61f2209402bf9f6c7e2abadef4f544a3
  manager: conda
  name: _openmp_mutex
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-1_gnu.tar.bz2
  version: '4.5'
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
    _openmp_mutex: '>=4.5'
  hash:
    md5: 47e6c01d149b26090748d9d1ac32491b
    sha256: d24e25272239827012441e3376abcd2859a29418da825e6a593fc517b0c20f61
  manager: conda
  name: libgcc-ng
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-11.2.0-h1d223b6_14.tar.bz2
  version: 11.2.0
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
  hash:
    md5: a1fd65c7ccbf10880423d82bca54eb54
    sha256: cb521319804640ff2ad6a9f118d972ed76d86bea44e5626c09a13d38f562e1fa
  manager: conda
  name: bzip2
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h7f98852_4.tar.bz2
  version: 1.0.8
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: d645c6d2ac96843a2bfaccd2d62b3ac3
    sha256: ab6e9856c21709b7b517e940ae7028ae0737546122f83c2aa5d692860c3b149e
  manager: conda
  name: libffi
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2
  version: 3.4.2
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: 39b1328babf85c7c3a61636d9cd50206
    sha256: 32f4fb94d99946b0dabfbbfd442b25852baf909637f2eed1ffe3baea15d02aad
  manager: conda
  name: libnsl
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-h7f98852_0.tar.bz2
  version: 2.0.0
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
  hash:
    md5: 772d69f030955d9646d3d0eaf21d859d
    sha256: 54f118845498353c936826f8da79b5377d23032bcac8c4a02de2019e26c3f6b3
  manager: conda
  name: libuuid
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.32.1-h7f98852_1000.tar.bz2
  version: 2.32.1
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
  hash:
    md5: 757138ba3ddc6777b82e91d9ff62e7b9
    sha256: b46b66d1cb171be2227a275e226195ca9e56c6f5b16250b85645e82a69518378
  manager: conda
  name: libzlib
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.11-h166bdaf_1014.tar.bz2
  version: 1.2.11
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: fb31bcb7af058244479ca635d20f0f4a
    sha256: bcb38449634bfe58e821c28d6814795b5bbad73514f0c7a9af7a710bbffc8243
  manager: conda
  name: ncurses
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.3-h9c3ff4c_0.tar.bz2
  version: '6.3'
- category: main
  dependencies:
    ca-certificates: ''
    libgcc-ng: '>=10.3.0'
  hash:
    md5: 49bf4e64d1e86676b90a8657c1142f01
    sha256: 123f0bd67843220fb27da6b71ba126934edbe714415a630ddec0f8c8a2b88cf0
  manager: conda
  name: openssl
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.0.2-h166bdaf_1.tar.bz2
  version: 3.0.2
- category: main
  dependencies:
    libgcc-ng: '>=7.5.0'
  hash:
    md5: 33f601066901f3e1a85af3522a8113f9
    sha256: 1e2823cb2a526bc3a7031ad5dbfb992891f9ff9740d1c17cb6dbb8ebdfd33b27
  manager: conda
  name: xz
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.5-h516909a_1.tar.bz2
  version: 5.2.5
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
    ncurses: '>=6.2,<7.0.0a0'
  hash:
    md5: 5788de3c8d7a7d64ac56c784c4ef48e6
    sha256: 30464670b3c81ac739e8df6b2c3c57b56d1e1408572540dec63bf4b8713163e4
  manager: conda
  name: readline
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/readline-8.1-h46c0cb4_0.tar.bz2
  version: '8.1'
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
    libzlib: '>=1.2.11,<1.3.0a0'
  hash:
    md5: 5b8c42eb62e9fc961af70bdd6a26e168
    sha256: 032fd769aad9d4cad40ba261ab222675acb7ec951a8832455fce18ef33fa8df0
  manager: conda
  name: tk
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h27826a3_0.tar.bz2
  version: 8.6.12
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
    libzlib: 1.2.11 h166bdaf_1014
  hash:
    md5: def3b82d1a03aa695bb38ac1dd072ff2
    sha256: ccfdb4dcceae8b191ddd4703e7be84eff2ba82b53788d6bb9298e531bae4eaf9
  manager: conda
  name: zlib
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-h166bdaf_1014.tar.bz2
  version: 1.2.11
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
    libzlib: '>=1.2.11,<1.3.0a0'
    ncurses: '>=6.3,<7.0a0'
    readline: '>=8.1,<9.0a0'
    zlib: '>=1.2.11,<1.3.0a0'
  hash:
    md5: 8057ac02d6d10a162d7eb4b0ca7ed291
    sha256: 5b1f7e51e6f6453c295cd911b826327b7eba4785b0366cf63cf6f828ec346076
  manager: conda
  name: sqlite
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.37.1-h4ff8645_0.tar.bz2
  version: 3.37.1
- category: main
  dependencies:
    bzip2: '>=1.0.8,<2.0a0'
    ld_impl_linux-64: '>=2.36.1'
    libffi: '>=3.4.2,<3.5.0a0'
    libgcc-ng: '>=10.3.0'
    libnsl: '>=2.0.0,<2.1.0a0'
    libuuid: '>=2.32.1,<3.0a0'
    libzlib: '>=1.2.11,<1.3.0a0'
    ncurses: '>=6.3,<7.0a0'
    openssl: '>=3.0.2,<4.0a0'
    readline: '>=8.1,<9.0a0'
    sqlite: '>=3.37.1,<4.0a0'
    tk: '>=8.6.12,<8.7.0a0'
    tzdata: ''
    xz: '>=5.2.5,<5.3.0a0'
  hash:
    md5: 0f72b088a5471e97309031e1636e7b3f
    sha256: 70eb462c28c5467c6e4860d5f574d240350b6ac718990b23cb0cc144d1dbea3f
  manager: conda
  name: python
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python-3.10.4-h2660328_0_cpython.tar.bz2
  version: 3.10.4
- category: main
  dependencies:
    python: 3.10.*
  hash:
    md5: 9e7160cd0d865e98f6803f1fe15c8b61
    sha256: e7e52aaec7cba6e17e45d731f9d38ede007aea0d72aee66670ab71016f5783ed
  manager: conda
  name: python_abi
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.10-2_cp310.tar.bz2
  version: '3.10'
- category: main
  dependencies:
    python: '!=3.0,!=3.1,!=3.2,!=3.3,!=3.4'
  hash:
    md5: 1ca02aaf78d9c70d9a81a3bed5752022
    sha256: aede66e6370f3b936164a703e48362f9080d7162234058fb2ee63cc84d528afc
  manager: conda
  name: wheel
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/wheel-0.37.1-pyhd8ed1ab_0.tar.bz2
  version: 0.37.1
- category: main
  dependencies:
    python: '>=3.10,<3.11.0a0'
    python_abi: 3.10.* *_cp310
  hash:
    md5: 2bf50027b62c5e607310c1755c27e482
    sha256: 2d5aba1f98b586b637e66bd1593424f4d5530cbd73b06883b460f2947abc244e
  manager: conda
  name: setuptools
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/setuptools-61.2.0-py310hff52083_3.tar.bz2
  version: 61.2.0
- category: main
  dependencies:
    python: '>=3.7'
    setuptools: ''
    wheel: ''
  hash:
    md5: b1239ce8ef2a1eec485c398a683c5bff
    sha256: d36bb23fa250be2d6a21cafe1760a7ae434318fb397c85223dd6a0c8e6e5562b
  manager: conda
  name: pip
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/pip-22.0.4-pyhd8ed1ab_0.tar.bz2
  version: 22.0.4
version: 1

    Proposed conda-vendor changes + improvements:

    Remove conda-vendor's meta-manifest generation

    I propose that we remove the meta-manifest generation, as conda-lock's new lockfile format now includes all of the necessary information we use to vendor dependencies into a local channel.

    • [x] Remove intermediary step of generating a meta-manifest in favor of conda-lock's 1.0.x new API
    • [x] Add in vendor command as the primary
    • [x] Add a subcommand to generate formatted output for IronBank's hardening_manifest.yaml resources block, using conda-lock's 1.0.x FetchAction object.

    Remove conda-vendor's combined manifest functionality

    conda-lock now has compound specification for lockfile generation, where you can create a conda-lock.yaml from one or more environment files.

    • [ ] Remove intermediary step of generating a meta-manifest in favor of using conda-lock's 1.0.x compound-specification API NOTE: this would be best tracked as it's own issue

    Signing and Verification

    I propose that we introduce signing and verification of the vendored dependencies within a local channel (and/or the local channel itself), and generate a SLSA compliant in-toto spec attestation. NOTE: this would be best tracked as it's own issue

    • [ ] Define attestation + SBOM format (see notes below)
    • [ ] sigstore digital signing
    enhancement question 
    opened by rigzba21 2
  • Upstream conda-lock changes are non-compatible with the IronBank workflow's update_ib_files.py

    Upstream conda-lock changes are non-compatible with the IronBank workflow's update_ib_files.py

    When running the IronBank update_ib_files.py the following error shows up:

    Traceback (most recent call last):
  File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/bin/conda-vendor", line 6, in <module>
    from conda_vendor.__main__ import cli
  File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_vendor/__main__.py", line 4, in <module>
    from conda_vendor.cli import (
  File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_vendor/cli.py", line 3, in <module>
    from conda_vendor.conda_channel import CondaChannel
  File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_vendor/conda_channel.py", line 11, in <module>
    from conda_lock.conda_lock import solve_specs_for_arch
ImportError: cannot import name 'solve_specs_for_arch' from 'conda_lock.conda_lock' (/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_lock/conda_lock.py)

    Looks like this is due to some changes in upstream conda-lock's src_parser module that introduce dependencies as a List(VersionedDependencies) which breaks the current version of conda-vendor

    ~~WIP PR: https://github.com/MetroStar/conda-vendor/pull/32~~

    bug IronBank 
    opened by rigzba21 2
Releases(v1.0.3)

  • v1.0.3(Sep 16, 2022)

  • v1.0.2(Sep 16, 2022)

  • v1.0.1(Jun 21, 2022)

    Previously the --ironbank-gen flag would output the generated manifest to stdout. This was causing issues when capturing standard output for automated workflows, because informational messages would also be captured.

    This change makes the manifest output to a file called ib_manifest.yaml in the current working directory instead.

    Source code(tar.gz)
    Source code(zip)

  • v1.0.0(Apr 12, 2022)

    • Remove meta-manifest intermediary step in favor of conda-lock's new 1.0.x API and lockfile format
    • Make vendor the primary command
    • Add flag for different solvers: conda, mamba, micromamba based on conda-lock's new 1.0.x API for solve_specs_for_arch
    • Add a --dry-run option to generate formatted JSON of conda-lock's solved FetchActions object. This replaces the old meta-manifest generation intermediary step and can be piped to other SBOM tools that accept JSON.
    • IronBank hardening_manifest.yaml resources formatted text output, using filtered + merged FETCH actions in conda-lock's FetchActions object
    • click cli integration tests
    • update test fixtures to use conda-lock's FetchActions, LockSpecification, and VersionedDependency objects
    • Update GitHub Actions to include dependencies for integration tests

    New Usage for version 1.0.0:

    # use conda as the solver for linux-64
conda-vendor vendor --file environment.yaml --solver conda --platform linux-64

# use mamba as the solver for osx-64
conda-vendor vendor --file environment.yaml --solver mamba --platform osx-64

# use micromamba as the solver for the host platform
conda-vendor vendor --file environment.yaml --solver micromamba

# dry-run outputs formatted JSON 
conda-vendor vendor --file environment.yaml --solver mamba --platform linux-64 --dry-run True

    Screenshots:

    Conda-vendor Improved UX:

    image

    repodata.json hotfix progress bar and package download progress bar:

    image

    Dry-Run formatted JSON output:

    image

    ironbank-gen subcommand (returns formatted text that can be copied into Ironbank's hardening manifest):

    image

    Example IronBank Workflow:

    1) Generate the vendored channel and output the IronBank Hardening Manifest resources to stdout:

    conda-vendor vendor --file my-environment.yaml --solver micromamba --platform linux-64 --ironbank-gen True

    2) Copy the output resources block to your IronBank hardening_manifest.yaml:

    - url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
  filename: micromamba-0.22.0-0.tar.bz2
  validation:
    type: sha256
    value: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
- url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
  filename: _libgcc_mutex-0.1-conda_forge.tar.bz2
  validation:
    type: sha256
    value: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
- url: https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
  filename: libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
  validation:
    type: sha256
    value: cc84f71bb9dbecde453a25ba8c5aefc9773da5d619633c103eb8bac1ab4afda0

    3) Copy over your patched repodata.json files from your vendored channel to the appropriate directory in your IronBank Project

    Source code(tar.gz)
    Source code(zip)

  • v0.1.15(Nov 18, 2021)

  • v0.1.14(Nov 17, 2021)

  • v0.1.13(Nov 17, 2021)

  • v0.1.12(Nov 15, 2021)

    • Adds combine-catalogs option to cli enabling users to combine solves from multiple yaml files into a single set of output local environments
    Source code(tar.gz)
    Source code(zip)

  • v0.1.11(Oct 26, 2021)

  • v0.1.10(Sep 23, 2021)

  • v0.1.9(Sep 22, 2021)

    • add option in meta-manifest to create a channel for a platform different than the one currently being used. Supported platforms include: ['linux-64', 'linux-32', 'windows-64', 'windows-32', 'osx-64']
    Source code(tar.gz)
    Source code(zip)

  • v0.1.8(Sep 10, 2021)

  • v0.1.7(Sep 10, 2021)

  • v0.1.6(Sep 10, 2021)

  • v0.1.5(Sep 9, 2021)

  • v0.1.4(Sep 9, 2021)

  • v0.1.3(Sep 9, 2021)

  • v0.1.2-alpha(Sep 8, 2021)

  • v0.1.1-alpha(Sep 8, 2021)

  • v.0.0.1-alpha(Sep 8, 2021)

Owner
MetroStar - Tech
CSG Demos & Tech Challenges
MetroStar - Tech
GitHub
Use an air-gapped Raspberry Pi Zero to sign for Bitcoin transactions! (and do other cool stuff)

Hello World! Build your own offline, airgapped Bitcoin transaction signing device for less than $35! Also generate seed word 24 or generate a seed phr

null 371 Dec 31, 2022
A test repository to build a python package and publish the package to Artifact Registry using GCB

A test repository to build a python package and publish the package to Artifact Registry using GCB. Then have the package be a dependency in a GCF function.

null 1 Feb 9, 2022
Gapmm2: gapped alignment using minimap2 (align transcripts to genome)

gapmm2: gapped alignment using minimap2 This tool is a wrapper for minimap2 to r

Jon Palmer 2 Jan 27, 2022
Blender addon that enables exporting of xmodels from blender. Great for custom asset creation for cod games

Birdman's XModel Tools For Blender Greetings everyone in the custom cod community. This blender addon should finally enable exporting of custom assets

wast 2 Jul 2, 2022
StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html. Questions? https://forum.stackstorm.com/.

StackStorm is a platform for integration and automation across services and tools, taking actions in response to events. Learn more at www.stackstorm.

StackStorm 5.3k Jan 2, 2023
Kubeflow is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on Kubernetes simple, portable, and scalable.

SDK: Overview of the Kubeflow pipelines service Kubeflow is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on

Kubeflow 3.1k Jan 6, 2023
Boilerplate Django Blog for production deployments!

CFE Django Blog THIS IS COMING SOON This is boilerplate code that you can use to learn how to bring Django into production. TLDR; This is definitely c

Coding For Entrepreneurs 26 Dec 9, 2022
tool for creating installers from conda packages

(conda) Constructor Description Constructor is a tool which allows constructing an installer for a collection of conda packages. It solves needed pack

Conda 386 Jan 4, 2023
Demo Python project using Conda and Poetry

Conda Poetry This is a demonstration of how Conda and Poetry can be used in a Python project for dev dependency management and production deployment.

Ryan Allen 2 Apr 26, 2022
A conda-smithy repository for boost-histogram.

The official Boost.Histogram Python bindings. Provides fast, efficient histogramming with a variety of different storages combined with dozens of composable axes. Part of the Scikit-HEP family.

conda-forge 0 Dec 17, 2021
Workon - A simple project manager for conda, windows 10 and vscode

WORK ON A simple project manager for conda, windows 10 and vscode Installation p

Jesus Alan Hernandez Galvan 1 Jan 16, 2022
Text-Based Ideal Points

Text-Based Ideal Points Source code for the paper: Text-Based Ideal Points by Keyon Vafa, Suresh Naidu, and David Blei (ACL 2020). Update (June 29, 20

Keyon Vafa 37 Oct 9, 2022
Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS)

TOPSIS implementation in Python Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) CHING-LAI Hwang and Yoon introduced TOPSIS

Hamed Baziyad 8 Dec 10, 2022
Research Artifact of USENIX Security 2022 Paper: Automated Side Channel Analysis of Media Software with Manifold Learning

Manifold-SCA Research Artifact of USENIX Security 2022 Paper: Automated Side Channel Analysis of Media Software with Manifold Learning The repo is org

Yuanyuan Yuan 172 Dec 29, 2022
osqueryIR is an artifact collection tool for Linux systems.

osqueryIR osqueryIR is an artifact collection tool for Linux systems. It provides the following capabilities: Execute osquery SQL queries Collect file

AbdulRhman Alfaifi 7 Nov 2, 2022
InDuDoNet+: A Model-Driven Interpretable Dual Domain Network for Metal Artifact Reduction in CT Images

InDuDoNet+: A Model-Driven Interpretable Dual Domain Network for Metal Artifact Reduction in CT Images Hong Wang, Yuexiang Li, Haimiao Zhang, Deyu Men

Hong Wang 4 Dec 27, 2022
Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems

WideLinears Pytorch parallel Neural Networks A package of pytorch modules for fast paralellization of separate deep neural networks. Ideal for agent-b

null 1 Dec 17, 2021
A parser of Windows Defender's DetectionHistory forensic artifact, containing substantial info about quarantined files and executables.

A parser of Windows Defender's DetectionHistory forensic artifact, containing substantial info about quarantined files and executables.

Jordan Klepser 101 Oct 30, 2022
A wordlist generator tool, that allows you to supply a set of words, giving you the possibility to craft multiple variations from the given words, creating a unique and ideal wordlist to use regarding a specific target.

A wordlist generator tool, that allows you to supply a set of words, giving you the possibility to craft multiple variations from the given words, creating a unique and ideal wordlist to use regarding a specific target.

Cycurity 39 Dec 10, 2022
NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.

NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.

NetworkX 12k Jan 2, 2023