Run unpatched binaries on Nix/NixOS

Overview

nix-alien

ci

Introduction

You are running nix/NixOS and have ever encountered the following problem?

$ ./bb
bash: ./bb: No such file or directory

Fear not, now there is nix-alien which will download necessary dependencies for you.

$ ./nix-alien bb            --> Run the binary inside a FHS shell with all needed shared dependencies to execute the binary
$ ./nix-alien-ld bb         --> Spawns you inside a shell with NIX_LD_LIBRARY_PATH set to the needed dependencies, to be used with nix-ld
$ ./nix-alien-find-libs bb  --> Lists all libs needed for the binary

Usage (Flakes)

Assuming you have nix already installed:

$ nix-shell -p nixUnstable nix-index
$ nix-index # this will take a long time
$ nix run --experimental-features 'nix-command flakes' "github:thiagokokada/nix-alien" -- ~/myapp

This will run nix-alien on ~/myapp binary with a FHSUserEnv including all shared library dependencies. The resulting default.nix file will be saved to $XDG_CACHE_HOME/nix-alien/ /default.nix , making the next evaluation faster. You can also pass --recreate flag to force the recreation of default.nix file

You can edit your /etc/nix/nix.conf or ~/.config/nix/nix.conf file and add the following line to avoid having to pass --experimental-features flag every time:

experimental-features = nix-command flakes

From here on this guide will assume the above configuration is done for brevity.

In case you're using nix-ld, there is also nix-alien-ld:

$ nix run "github:thiagokokada/nix-alien#nix-alien-ld" -- ~/myapp 

This will spawn a mkShell instead with NIX_LD_LIBRARY_PATH and NIX_LD setup. The resulting shell.nix file will be saved to $XDG_CACHE_HOME/nix-alien/ /shell.nix , making the next evaluation faster. You can also pass --recreate flag to force the recreation of shell.nix file

If you want to use the fzf based menu to find the libraries for scripting purposes, you can run:

$ nix run "github:thiagokokada/nix-alien#nix-alien-find-libs" -- ~/myapp 

This will print the found libraries on the stdout. The informational messages are printed to stderr, so you can easily redirect them to /dev/null if needed.

To avoid the slow startup of nix-index, you can also download a pre-computed index from nix-index-database:

$ nix run "github:thiagokokada/nix-alien#nix-index-update"

Usage (non-Flakes)

$ $(nix-build default.nix --no-out-link)/bin/nix-alien
$ $(nix-build default.nix --no-out-link)/bin/nix-alien-ld
$ $(nix-build default.nix --no-out-link)/bin/nix-alien-find-libs
$ $(nix-build nix-index-update.nix --no-out-link)/bin/nix-index-update

Limitations

Binaries loading shared libraries dynamically (e.g.: with dlopen) will probably not work with this script. However, this script can still be useful to create an initial default.nix or shell.nix, that can be populated later with the runtime dependencies of the program.

Technical Description

This Python program allows you to download ELF binaries and use them right away! This is achieved by enumerating the shared library dependencies from the ELF header and then searching for the equivalent library in nixpkgs. This is done by querying nix-locate locally.

To be able to use nix-locate, first, the index has to be build. This is done by running nix-index and waiting 10-15 minutes. To speed-up this process, this repo also includes nix-index-update script, that downloads the index from nix-index-database.

Credits

Comments
  • Can't build with nixpkgs master?..

    Can't build with nixpkgs master?..

    It seems like nix-alien fails to build with nixpkgs/master:

    error: builder for '/nix/store/afmw0vxjpzc6h66dbv35pfx6d3mkwd6c-python3.9-pyparsing-2.4.7.drv' failed with exit code 2;
           last 10 log lines:
           >   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
           >   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
           >   File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
           >   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
           >   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
           >   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
           >   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
           > ModuleNotFoundError: No module named 'setuptools'
           >
           >
           For full logs, run 'nix log /nix/store/afmw0vxjpzc6h66dbv35pfx6d3mkwd6c-python3.9-pyparsing-2.4.7.drv'.
    error: builder for '/nix/store/cz1rxqq0db8a4z8svmxf0dfyfdsnzxvq-python3.9-typing-extensions-3.10.0.2.drv' failed with exit code 2;
           last 10 log lines:
           >   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
           >   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
           >   File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
           >   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
           >   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
           >   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
           >   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
           > ModuleNotFoundError: No module named 'setuptools'
           >
           >
           For full logs, run 'nix log /nix/store/cz1rxqq0db8a4z8svmxf0dfyfdsnzxvq-python3.9-typing-extensions-3.10.0.2.drv'.
    error: 1 dependencies of derivation '/nix/store/l7ary7ycww3jhx07gn2s4l726k9zyjhi-python3.9-setuptools-scm-7.0.5.drv' failed to build
    error: 1 dependencies of derivation '/nix/store/q03abknx97z2cllzyd3fihk7lmw5dqr1-python3.9-poetry-core-1.0.8.drv' failed to build
    error: 1 dependencies of derivation '/nix/store/dbxkwn151mwgjg1anpi6bf5bpqfy6fgn-python3.9-nix-alien-0.1.0.drv' failed to build
    

    Related to #193708

    opened by pshirshov 11
  • Flake's overlay should use `prev` instead of `inputs.nixpkgs...` ?

    Flake's overlay should use `prev` instead of `inputs.nixpkgs...` ?

    The flake's overlay should use prev instead of inputs.nixpkgs... ?

    Maybe I'm mistaken about what is expected from overlays, but I thought overlays should work this way and not pull pkgs from the outside.

    Anyway, it makes integration with my system a bit harder, the only solution I found so far is to directly use pkgs.callPackage on nix-alien.nix to force the use of my system's pkgs.

    Maybe my needs are weirds also, I can explain more if needed :)

    opened by PaulGrandperrin 8
  • test_create_fhs_env_drv_flake fails on aarch64-linux

    test_create_fhs_env_drv_flake fails on aarch64-linux

    Since the flake generated by the test test_create_fhs_env_drv_flake gives an output with system = "x86_64-linux, which of course fails on an aarch64 machine. Using aarch64-linux in the test works results in nix-alien's tests passing and nix-alien works perfectly fine.

    opened by LegitMagic 6
  • Split CI/non-CI builds without overriding Python package set with non-modules

    Split CI/non-CI builds without overriding Python package set with non-modules

    This fixes issues with current nixpkgs master where all members of the Python package set are now required to be Python modules

    Let me know if this is a reasonable approach or not! I just wanted to get something that worked to demonstrate the root problem

    Related: #45

    opened by lilyinstarlight 4
  • Installation is heavy because of the compilation of `mypy` from source

    Installation is heavy because of the compilation of `mypy` from source

    I don't know very well how python works and how it's dependencies are integrated with nix but it looks like mypy is not really needed for the end user.

    But nix-alien always wants mypy at version 0.942 which isn't in the cache, which makes every updates quite heavy. On my machines, it monopolize my CPU for about 10 to 20 minutes.

    If that's not really needed, would it be possible to avoid this?

    opened by PaulGrandperrin 3
  • No usable options for libSkiaSharp.so

    No usable options for libSkiaSharp.so

    Trying to run nix-alien on talon voice fails because no good options can be found for libSkiaSharp.so.

    Here are the options presented:

      jellyfin.out                                                                    mission-planner.out                                                             alttpr-opentracker.out                                                        > wasabiwallet.out   
    'libSkiaSharp.so'> 
    

    And here is the attempt to run:

    [nix-shell:~/system]$ nix run "github:thiagokokada/nix-alien#nix-alien" --  --recreate /home/nixos/Downloads/talon/talon
    Selected candidate for 'libQt5Widgets.so.5': libsForQt514.full.out
    Selected candidate for 'libpython3.9.so.1.0': python39Full.out
    Selected candidate for 'libusb-1.0.so.0': libusb1.out
    Selected candidate for 'libSkiaSharp.so': wasabiwallet.out
    No candidate found for 'libw2ldecode.so'
    Selected candidate for 'libw2ldecode.so': None
    Selected candidate for 'libQt5DBus.so.5': libsForQt514.full.out
    Selected candidate for 'libQt5X11Extras.so.5': libsForQt514.full.out
    Selected candidate for 'libGL.so.1': xorg_sys_opengl.out
    Selected candidate for 'libQt5Gui.so.5': libsForQt514.full.out
    Selected candidate for 'libQt5Core.so.5': libsForQt514.full.out
    Selected candidate for 'libstdc++.so.6': gcc-unwrapped.lib
    File '/home/nixos/.cache/nix-alien/7d515bd9-aa23-5d9a-a57a-86c6b77690fb/fhs-env/default.nix' created successfuly!
    /home/nixos/Downloads/talon/talon: error while loading shared libraries: libSkiaSharp.so: cannot open shared object file: No such file or directory
    

    NOTE: I tried each available option presented for libSkiaSharp.so, not just the one above.

    I'm unsure of how to proceed here. Given this library is aimed at newer users just trying to get binaries to run, I figured this issue might be appropriate.

    One potential resolution being a section added to the readme of what to do in this situation.

    Thanks!

    opened by ParetoOptimalDev 3
  • Bad example in README

    Bad example in README

    It turns out that overlays in flakes use main nixpkgs input, but not the nix-alien's nixpkgs

    So this example

              nixpkgs.overlays = [
                self.inputs.nix-alien.overlays.default
              ];
    

    Is incorrect and leads to #34 even if without inputs.nixpkgs.follows = "nixpkgs";

    opened by hissssst 2
  • Avoid using nixpkgs in overlay

    Avoid using nixpkgs in overlay

    This is just an idea to remove the dependency on nixpkgs and maybe solve the issues with glibc mentioned in #20 .

    I didn't encounter any issues so far, but I don't have a lot of experience with this so maybe there are other issues.

    opened by PaulGrandperrin 2
  • Error building on nixos-unstable

    Error building on nixos-unstable

    Hi, I'm currently using nix-alien in my system derivation and I've recently updated nixpkgs to 7e52b35fe98481a279d89f9c145f8076d049d2b9, this broke my derivation build with the error:

    error: 'nixFlakes' has been renamed to/replaced by 'nixVersions.stable'
    

    I've traced it back to nix-alien which uses the now deprecated nixFlakes and removing nix-alien from the derivation does fix the build.

    opened by JCapucho 1
  • README: `nixosModules.nix-ld` is not an overlay

    README: `nixosModules.nix-ld` is not an overlay

    At line 201 in the nixos-installation-with-flakes section, the nix-ld module is put in nixpkgs.overlays. This didn't work on my machine, it should probably be an import.

    opened by lourkeur 1
  • How does this compare to steam-run?

    How does this compare to steam-run?

    steam-run is a Nixpkgs tool for running Steam games unpatched, but can be used to run all other Linux programs unpatched. How does nix-alien compare to that?

    This is a question, but the answer would be great to add to documentation.

    opened by danbst 1
  • can't install nix-alien with flak

    can't install nix-alien with flak

    my flack is build success but can't install nix-alien and nix-alien-xx etc... Screenshot_20221114_200853 with cmd : fd "nix-alien" only get : Screenshot_20221114_201703

    my flake.nix

    {
      description = "NixOS Configuration";
    
      inputs = {
        nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    
        nix-alien.url = "github:thiagokokada/nix-alien";
        nix-ld = {
          url = "github:Mic92/nix-ld";
          inputs.nixpkgs.follows = "nixpkgs";
        };
      };
    
      outputs = inputs@ { self, nixpkgs, nix-alien, home-manager,hyprland, dedsec-grub-theme, ... }:
        let
          users = {
            username = {
              name = "username";
            };
          };
        in
        {
          nixosConfigurations = (
            import ./NixOS-Hosts {
              inherit (nixpkgs) lib;
              inherit inputs self nixpkgs nix-alien home-manager hyprland dedsec-grub-theme users; # Also inherit home-manager so it does not need to be defined here.
            }
          );
        };
    }
    

    my NixOS-Hosts/default.nix

    { self, lib, inputs, nixpkgs, nix-alien, home-manager, hyprland, dedsec-grub-theme, users, ... }:
    
    let
      system = "x86_64-linux";                                  # System architecture
      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = true;                              # Allow proprietary software
      };
      lib = nixpkgs.lib;
    in
    {
        laptop = lib.nixosSystem {                                # Laptop profile
        inherit system pkgs;
        specialArgs = { inherit inputs users self system; };
        modules = [
          hyprland.nixosModules.default
          dedsec-grub-theme.nixosModule
          ({ self, system, ... }: {
                imports = [
                  # Optional, but this is needed for `nix-alien-ld` command
                  self.inputs.nix-ld.nixosModules.nix-ld
                ];
                environment.systemPackages = with pkgs; with self.inputs.nix-alien.packages.${system}; [
                  nix-alien
                  nix-index # not necessary, but recommended
                  nix-index-update
                ];
          })
          ./Laptop
        ];
      };
    }
    
    opened by xBLACKICEx 5
  • Integrate with nixGl

    Integrate with nixGl

    First, thanks for this great program.

    When running a program using openGl (e.g. blender) you get an error:

    $ nix run github:thiagokokada/nix-alien -- blender
    /tmp/blender-3.2.2-linux-x64/blender: error while loading shared libraries: libGL.so.1: cannot open shared object file: No such file or directory
    

    As far as I see the best solution is to use nixGl:

    $ nix run --impure github:guibou/nixGL --override-input nixpkgs nixpkgs/nixos-21.11 -- nix run github:thiagokokada/nix-alien -- blender
    

    This does work but it is definitely not super easy to run... It would be great if nix-alien could directly handle openGl correctly.

    opened by tobiasBora 1
Owner
Thiago Kenji Okada
Software Engineer at @nubank, Linux enthusiast, Anime fan and geek.
Thiago Kenji Okada
Utility functions for working with data from Nix in Python

Pynixutil - Utility functions for working with data from Nix in Python Examples Base32 encoding/decoding import pynixutil input = "v5sv61sszx301i0x6x

Tweag 11 Dec 16, 2022
A multi-platform fuzzer for poking at userland binaries and servers

litefuzz A multi-platform fuzzer for poking at userland binaries and servers litefuzz intro why how it works what it does what it doesn't do support p

null 52 Nov 18, 2022
Automatic and platform-independent unpacker for Windows binaries based on emulation

_ _ __ _ __ _ | | | | / / (_) \ \ | | | | | |_ __ | | _ | | _ __ __ _ ___| | _____ _ __

null 514 Dec 21, 2022
Binjago - Set of tools aiding in analysis of stripped Golang binaries with Binary Ninja

Binjago ?? Set of tools aiding in analysis of stripped Golang binaries with Bina

W3ndige 2 Jul 23, 2022
run-js Goal: The Easiest Way to Run JavaScript in Python

run-js Goal: The Easiest Way to Run JavaScript in Python features Stateless Async JS Functions No Intermediary Files Functional Programming CommonJS a

Daniel J. Dufour 9 Aug 16, 2022
MiniJVM is simple java virtual machine written by python language, it can load class file from file system and run it.

MiniJVM MiniJVM是一款使用python编写的简易JVM,能够从本地加载class文件并且执行绝大多数指令。 支持的功能 1.从本地磁盘加载class并解析 2.支持绝大多数指令集的执行 3.支持虚拟机内存分区以及对象的创建 4.支持方法的调用和参数传递 5.支持静态代码块的初始化 不支

keguoyu 60 Apr 1, 2022
Run-Your-Own Firefox Sync Server

Run-Your-Own Firefox Sync Server This is an all-in-one package for running a self-hosted Firefox Sync server. It bundles the "tokenserver" project for

Mozilla Services 1.7k Dec 30, 2022
A simple service that allows you to run commands on the server using text

Server Text A simple flask service that allows you to run commands on the server/computer over sms. Think of it as a shell where you run commands over

MT Devs 49 Nov 9, 2021
This is a multi-app executor that it used when we have some different task in a our applications and want to run them at the same time

This is a multi-app executor that it used when we have some different task in a our applications and want to run them at the same time. It uses SQLAlchemy for ORM and Alembic for database migrations.

Majid Iranpour 5 Apr 16, 2022
BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset.

BloodCheck BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset. Installation

Mr B0b 16 Nov 5, 2021
Exercise to teach a newcomer to the CLSP grid to set up their environment and run jobs

Exercise to teach a newcomer to the CLSP grid to set up their environment and run jobs

Alexandra 2 May 18, 2022
Run Python code right in your Telegram messages

Run Python code right in your Telegram messages Made with Telethon library, TGPy is a tool for evaluating expressions and Telegram API scripts. Instal

null 29 Nov 22, 2022
A python script to run any executable and pass test cases to it's stdin and compare stdout with correct output.

quera_testcase_checker A python script to run any executable and pass test cases to it's stdin and compare stdout with correct output. proper way to u

k3y1 1 Nov 15, 2021
A simple program to run through inputs for a 3n+1 problem

Author Tyler Windemuth Collatz_Conjecture A simple program to run through inputs for a 3n+1 problem Purpose: doesn't really have a purpose, did this t

null 0 Apr 22, 2022
This repo is for scripts to run various clients at the merge f2f

merge-f2f This repo is for scripts to run various clients at the merge f2f. Tested with Lighthouse! Tested with Geth! General dependecies sudo apt-get

Parithosh Jayanthi 2 Apr 3, 2022
SpaCy3Urdu: run command to setup assets(dataset from UD)

Project setup run command to setup assets(dataset from UD) spacy project assets It uses project.yml file and download the data from UD GitHub reposito

Muhammad Irfan 1 Dec 14, 2021
Meera 2 May 12, 2022
Run python scripts and pass data between multiple python and node processes using this npm module

Run python scripts and pass data between multiple python and node processes using this npm module. process-communication has a event based architecture for interacting with python data and errors inside nodejs.

Tyler Laceby 2 Aug 6, 2021
Run CodeServer on Google Colab using Inlets in less than 60 secs using your own domain.

Inlets Colab Run CodeServer on Colab using Inlets in less than 60 secs using your own domain. Features Optimized for Inlets/InletsPro Use your own Cus

null 2 Dec 30, 2021