Static Token And Credential Scanner

Last update: Dec 27, 2022

Related tags

Overview

Static Token And Credential Scanner

What is it?

STACS is a YARA powered static credential scanner which suports binary file formats, analysis of nested archives, composable rulesets and ignore lists, and SARIF reporting.

What does STACS support?

Currently, STACS supports recursive unpacking of tarballs, gzips, bzips, zips, and xz files. As STACS works on detected file types, rather than the filename, propriatary file formats based on these types are automatically supported (such as Docker images, Android APKs, and Java JAR fles).

Who should use STACS?

STACS is designed for use by any teams who release binary artifacts. STACS provides developers the ability to automatically check for accidental inclusion of static credentials and key material in their releases.

However, this doesn't mean STACS can't help with SaaS applications, enterprise software, or even source code!

As an example, STACS can be used to find static credentials in Docker images uploaded to public and private container registries. It can also be used to find credentials accidentally compiled in to executables, packages for mobile devices, and "enterprise archives" - such as those used by Java application servers.

How does it work?

STACS detects static credentials using "rule packs" provided to STACS when run. These rule packs define a set of YARA rules to run against files provided to STACS. When a match against a rule is found, a "finding" is generated. These findings represent potential credentials inside of a file, and are reported on for a developer to remediate or "ignore".

If the finding is found to be a false positive - that is, a match on something other than a real credential - the developer can generate a set of "ignore lists" to ensure that these matches don't appear in future reports.

The real power from STACS comes from the automatic detection and unpacking of nested archives, and composable ignore lists and rule packs.

Ignore lists?

In order to allow flexible and collaborative usage, STACS supports composable ignore lists. This allows for an ignore list to include other ignore lists which enable composition of a "tree of ignores" based on organisational guidelines. These ignore lists are especially useful in organisations where many of the same frameworks or products are used. If a team has already marked a finding as a false positive, other teams get the benefit of not having to triage the same finding.

Rule packs?

In the same manner as ignore lists, rule packs are also composable. This enables an organisation to define a baseline set of rules for use by all teams, while still allowing teams to maintain rulesets specific to their products.

How do I use it?

The easiest way to use STACS is using the Docker images published to Docker Hub. However, STACS can also be installed directly from Python's PyPI, or by cloning this repository. See the relevant sections below to get started!

A cloud based service is coming soon which allows integration directly in build and release pipelines to enable detection of static credentials before release!

Docker

Using the published images, STACS can be used to scan artifacts right away! The STACS Docker images provides a number of volume mounts for files wanted to be scanned to be mounted directly into the scan container.

As an example, to scan everything in the current folder, the following command can be run (Docker must be installed).

docker run \
    --rm \
    --mount type=bind,source=$(pwd),target=/mnt/stacs/input \
    stacscan/stacs:latest

By default, STACS will output any findings in SARIF format directly to STDOUT and in order to keep things orderly, all log messages will be sent to STDERR. For more advanced use cases, a number of other volume mounts are provided. These allow the user to control the rule packs, ignore lists, and a cache directories to use.

PyPi

STACS can also be installed directly from Python's PyPi. This provides a stacs command which can then be used by developers to scan projects directly in their local development environments.

STACS can be installed directly from PyPi using:

pip install stacs

Please Note: The PyPi release of STACS does not come with any rules. These will also need to be cloned from the community rules repository for STACS to work!

FAQ

Is there a hosted version of STACS?

Not yet. However, there are plans for a hosted version of STACS which can be easily integrated into existing build systems, and which contains additional prebuilt rule packs and ignore lists.

What do I do about false positives?

Unfortunately, false positives are an inevitable side effect during the detection of static credentials. If rules are too granular then rule maintenance becomes a burden and STACS may miss credentials. If rules are too coarse then STACS may generate too many false positives!

In order to assist, STACS provides a number of tools to assist with reducing the number of false positives which make it into final reports.

Primarily, STACS provides a mechanism which allows users to define composable ignore lists which allow a set of findings to be "ignored". These rules can be as coarse as ignoring all files based on a pattern, or as granular as a specific finding on a particular line of a file.

This information is automatically propagated through into reports, so "ignored" findings will be marked as "suppressed" in SARIF output while also including the reason for the ignore in the output for tracking.

How do I view the results?

Currently, the only output format is SARIF v2.1.0. There are a number of viewers available which make this data easier to read, such as this great web based viewer from Microsoft. An example of the findings from a Docker container image has been included below:

The performance is really, really bad when running in Docker on macOS!

Unfortunately, this appears to be due to a limitation of Docker Desktop for Mac. I/O for bind mounts is really, really slow.

Comments

Proposal: non-0 exit-status when matches are found

I propose stacs to return non 0 exit status if at least 1 match (findings) with a rule is found.

The motivation for this is that in order to include stacs in automated pipeline, we need a switch (bool), which tells if stacs passes or fails. As such switch exit code is widely used in other python tools (flake8, mypy, black). In addition, tools like tox provide pass/fail checking by the exit code.

If for some reason having this by default in stacs is not acceptable, I'd ask to have at least an option flag (for example, --fail-on-match), because otherwise I see no other way of including stacs into automated pipeline, but writing another piece of code, which parses stacs's logs, which would be an overhead.
enhancement

opened by ghost 6
Feature request: Multivalue path
Would it be possible to implement a multiple path feature, which might look like this: stacs --rule-pack rules.json my_file_1 my_file_2 ...

This would reduce the boilerplate code and necessity to write:

stacs --rule-pack rules.json my_file_1 stacs --rule-pack rules.json my_file_2 ...

Maybe this could be a possible implementation: @click.argument('path', type=str, nargs=-1) Another implementation idea can be a path of the form "my_file_1,my_file_2,..." and then parsed like path.split(',') to have a list of paths. Then loop.
enhancement
opened by ghost 6

install error

$ pip --version    
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
$ pip install stacs
ERROR: Could not find a version that satisfies the requirement stacs (from versions: none)
ERROR: No matching distribution found for stacs

enhancement

opened by firmianay 5

Unable to generate file list on Netgear WAX620

Hi, currently I do some evaluations of STACS - you can also see our integration into the Firmware Scanner EMBA here: https://github.com/e-m-b-a/emba/pull/207

During these tasks I have done some tests on the Netgear firmware WAX620 from here: https://www.netgear.com/support/product/wax630.aspx#Firmware%20Version%209.5.4.6

During these tests STACS is running into the following error:

└─$ stacs --debug --rule-pack ~/git-repos/stacs-rules/credential.json ~/firmware/emba_logs_manual/netgear-WAX620/firmware                                     254 ⨯
2022-01-07 12:40:17,738 - 1934072 - [INFO] Attempting to load rule pack from /home/m1k3/git-repos/stacs-rules/credential.json
2022-01-07 12:40:17,739 - 1934072 - [INFO] Attempting to get a list of files to scan from /home/m1k3/firmware/emba_logs_manual/netgear-WAX620/firmware
2022-01-07 12:40:22,741 - 1934072 - [ERROR] Unable to generate file list: Unable to extract archive /home/m1k3/firmware/emba_logs_manual/netgear-WAX620/firmware/firmware_binwalk_emba/_firmware.extracted/_nand-ipq807x-apps.img.extracted/195C.ubi_ubi_extracted/195C.ubi/_img-1966538225_vol-kernel.ubifs-0.extracted/_Image.extracted/805788.xz to 805788: Corrupt input data

Before running STACS the extractor of EMBA has extracted the firmware and then I started STACS the shown way.

Is there something like a "ignore errors" option?

enhancement

opened by m-1-k-3 5

Invalid start byte

Hi @darkarnium,

we got a report on EMBA about a failed scan with a stacs error trace here. I can reproduce it with the attached file which is from the deep extractor and somehow corrupted.

┌──(m1k3㉿emba)-[~/github-repos/emba_forked]
└─$ stacs --rule-pack /home/m1k3/github-repos/emba_forked/external/stacs-rules/credential.json --skip-unprocessable /home/m1k3/firmware-stuff/emba_logs_test/firmware/patool_extraction/470ABBI4C0.bin_binwalk_extracted/_470ABBI4C0.bin.extracted/189830_binwalk_extracted/_189830.extracted/1393A94 
2022-11-21 14:00:16,209 - 1507053 - [INFO] STACS running with 10 threads
2022-11-21 14:00:16,209 - 1507053 - [INFO] STACS uses libarchive (licenses may be found at https://github.com/libarchive/libarchive/blob/master/COPYING)
2022-11-21 14:00:16,209 - 1507053 - [INFO] STACS uses yara (licenses may be found at https://github.com/VirusTotal/yara-python/blob/master/LICENSE)
2022-11-21 14:00:16,209 - 1507053 - [INFO] Attempting to load rule pack from /home/m1k3/github-repos/emba_forked/external/stacs-rules/credential.json
2022-11-21 14:00:16,210 - 1507053 - [INFO] Using cache directory at /tmp/1669035616210497
2022-11-21 14:00:16,210 - 1507053 - [INFO] Attempting to get a list of files to scan from /home/m1k3/firmware-stuff/emba_logs_test/firmware/patool_extraction/470ABBI4C0.bin_binwalk_extracted/_470ABBI4C0.bin.extracted/189830_binwalk_extracted/_189830.extracted/1393A94
2022-11-21 14:00:16,279 - 1507053 - [INFO] Found 1 files for analysis
Traceback (most recent call last):
  File "/usr/local/bin/stacs", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/stacs/scan/entrypoint/cli.py", line 143, in main
    getattr(stacs.scan.scanner, scanner).run(targets, pack, workers=threads)
  File "/usr/local/lib/python3.10/dist-packages/stacs/scan/scanner/rules.py", line 222, in run
    findings.extend(future.result())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/stacs/scan/scanner/rules.py", line 176, in matcher
    findings.extend(generate_findings(target, match))
  File "/usr/local/lib/python3.10/dist-packages/stacs/scan/scanner/rules.py", line 147, in generate_findings
    location = generate_location(target, offset)
  File "/usr/local/lib/python3.10/dist-packages/stacs/scan/scanner/rules.py", line 132, in generate_location
    line_number += fin.read(CHUNK_SIZE).count("\n")
  File "/usr/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 9752: invalid start byte

1393A94.zip

bug

opened by m-1-k-3 4

Colorama dependency
Hi Peter,

during checking the pip dependencies of EMBA on the new Kali Linux I found a conflicting dependency on colorama:

Warning!!! Possibly conflicting dependencies found: * stacs==0.4.14 - colorama [required: ==0.4.5, installed: 0.4.6]

Would it be possible to update the colorama dependency to 0.4.6 or is there a reason to stick on 0.4.5?

Best Mike
enhancement
opened by m-1-k-3 3

cant install stacs from github releases and PyPi

My OS and Python version:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04 LTS
Release:	22.04
Codename:	jammy
$ python3 --version
Python 3.10.4

Steps for reproduse:

download latest github release (cli not used)
unpack and go to folder with tool

tar -xzf stacs-0.4.6.tar.gz 
rm stacs-0.4.6.tar.gz 
cd stacs-0.4.6/

create virtual environment

$ virtualenv venv
created virtual environment CPython3.10.4.final.0-64 in 434ms
  creator CPython3Posix(dest=stacs-0.4.6/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

try install using setup.py in release files

python setup.py install
stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
  warnings.warn(
running install
stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing stacs.egg-info/PKG-INFO
writing dependency_links to stacs.egg-info/dependency_links.txt
writing entry points to stacs.egg-info/entry_points.txt
writing requirements to stacs.egg-info/requires.txt
writing top-level names to stacs.egg-info/top_level.txt
reading manifest file 'stacs.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'stacs.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/stacs
creating build/bdist.linux-x86_64/egg/stacs/scan
creating build/bdist.linux-x86_64/egg/stacs/scan/loader
copying build/lib/stacs/scan/loader/archive.py -> build/bdist.linux-x86_64/egg/stacs/scan/loader
copying build/lib/stacs/scan/loader/manifest.py -> build/bdist.linux-x86_64/egg/stacs/scan/loader
copying build/lib/stacs/scan/loader/filepath.py -> build/bdist.linux-x86_64/egg/stacs/scan/loader
copying build/lib/stacs/scan/loader/__init__.py -> build/bdist.linux-x86_64/egg/stacs/scan/loader
copying build/lib/stacs/scan/constants.py -> build/bdist.linux-x86_64/egg/stacs/scan
copying build/lib/stacs/scan/exceptions.py -> build/bdist.linux-x86_64/egg/stacs/scan
copying build/lib/stacs/scan/__about__.py -> build/bdist.linux-x86_64/egg/stacs/scan
copying build/lib/stacs/scan/__init__.py -> build/bdist.linux-x86_64/egg/stacs/scan
creating build/bdist.linux-x86_64/egg/stacs/scan/output
copying build/lib/stacs/scan/output/markdown.py -> build/bdist.linux-x86_64/egg/stacs/scan/output
copying build/lib/stacs/scan/output/__init__.py -> build/bdist.linux-x86_64/egg/stacs/scan/output
copying build/lib/stacs/scan/output/sarif.py -> build/bdist.linux-x86_64/egg/stacs/scan/output
creating build/bdist.linux-x86_64/egg/stacs/scan/model
copying build/lib/stacs/scan/model/pack.py -> build/bdist.linux-x86_64/egg/stacs/scan/model
copying build/lib/stacs/scan/model/manifest.py -> build/bdist.linux-x86_64/egg/stacs/scan/model
copying build/lib/stacs/scan/model/ignore_list.py -> build/bdist.linux-x86_64/egg/stacs/scan/model
copying build/lib/stacs/scan/model/finding.py -> build/bdist.linux-x86_64/egg/stacs/scan/model
copying build/lib/stacs/scan/model/__init__.py -> build/bdist.linux-x86_64/egg/stacs/scan/model
creating build/bdist.linux-x86_64/egg/stacs/scan/scanner
copying build/lib/stacs/scan/scanner/rules.py -> build/bdist.linux-x86_64/egg/stacs/scan/scanner
copying build/lib/stacs/scan/scanner/__init__.py -> build/bdist.linux-x86_64/egg/stacs/scan/scanner
creating build/bdist.linux-x86_64/egg/stacs/scan/filter
copying build/lib/stacs/scan/filter/ignore_list.py -> build/bdist.linux-x86_64/egg/stacs/scan/filter
copying build/lib/stacs/scan/filter/__init__.py -> build/bdist.linux-x86_64/egg/stacs/scan/filter
creating build/bdist.linux-x86_64/egg/stacs/scan/entrypoint
copying build/lib/stacs/scan/entrypoint/__init__.py -> build/bdist.linux-x86_64/egg/stacs/scan/entrypoint
copying build/lib/stacs/scan/entrypoint/cli.py -> build/bdist.linux-x86_64/egg/stacs/scan/entrypoint
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/loader/archive.py to archive.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/loader/manifest.py to manifest.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/loader/filepath.py to filepath.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/loader/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/constants.py to constants.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/exceptions.py to exceptions.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/__about__.py to __about__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/output/markdown.py to markdown.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/output/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/output/sarif.py to sarif.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/model/pack.py to pack.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/model/manifest.py to manifest.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/model/ignore_list.py to ignore_list.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/model/finding.py to finding.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/model/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/scanner/rules.py to rules.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/scanner/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/filter/ignore_list.py to ignore_list.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/filter/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/entrypoint/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/stacs/scan/entrypoint/cli.py to cli.cpython-310.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying stacs.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying stacs.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying stacs.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying stacs.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying stacs.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying stacs.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/stacs-0.4.6-py3.10.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing stacs-0.4.6-py3.10.egg
Copying stacs-0.4.6-py3.10.egg to stacs-0.4.6/venv/lib/python3.10/site-packages
Adding stacs 0.4.6 to easy-install.pth file
Installing stacs script to stacs-0.4.6/venv/bin

Installed stacs-0.4.6/venv/lib/python3.10/site-packages/stacs-0.4.6-py3.10.egg
Processing dependencies for stacs==0.4.6
Searching for python-libarchive
Reading https://pypi.org/simple/python-libarchive/
stacs-0.4.6/venv/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: libarchive-3.1.2-1 is an invalid version and will not be supported in a future release
  warnings.warn(
stacs-0.4.6/venv/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release
  warnings.warn(
stacs-0.4.6/venv/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: libarchive-4.0.0.post1 is an invalid version and will not be supported in a future release
  warnings.warn(
stacs-0.4.6/venv/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: libarchive-4.0.1.post1 is an invalid version and will not be supported in a future release
  warnings.warn(
Downloading https://files.pythonhosted.org/packages/eb/da/ed182b7b738c598d6bcae9534bd6c701f87d73404f164dba92709bca9149/python-libarchive-4.0.1.post1.tar.gz#sha256=488b56a6f42927a60a2d8c5d908be286435139ec211c9770f77b5bd133351160
Best match: python-libarchive 4.0.1.post1
Processing python-libarchive-4.0.1.post1.tar.gz
Writing /tmp/easy_install-9pdfmx65/python-libarchive-4.0.1.post1/setup.cfg
Running python-libarchive-4.0.1.post1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-9pdfmx65/python-libarchive-4.0.1.post1/egg-dist-tmp-n19kwoet
stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/dist.py:516: UserWarning: Normalizing '4.0.1-1' to '4.0.1.post1'
  warnings.warn(tmpl.format(**locals()))
stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
libarchive/_libarchive_wrap.c:2963:10: fatal error: archive.h: No such file or directory
 2963 | #include <archive.h>
      |          ^~~~~~~~~~~
compilation terminated.
error: Setup script exited with error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

try setup with pip

$ pip install stacs
Collecting stacs
  Using cached stacs-0.4.6-py3-none-any.whl
Collecting click
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
[collecting other packages...]
Collecting pyparsing!=3.0.5,>=2.0.2
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 KB 1.8 MB/s eta 0:00:00
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.4-py2.py3-none-any.whl (461 kB)
Building wheels for collected packages: python-libarchive
  Building wheel for python-libarchive (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
      stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/dist.py:516: UserWarning: Normalizing '4.0.1-1' to '4.0.1.post1'
        warnings.warn(tmpl.format(**locals()))
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/__init__.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/_libarchive.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/zip.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/tar.py -> build/lib.linux-x86_64-cpython-310/libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/libarchive
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ilibarchive -I stacs-0.4.6/venv/include -I/usr/include/python3.10 -c libarchive/_libarchive_wrap.c -o build/temp.linux-x86_64-cpython-310/libarchive/_libarchive_wrap.o
      libarchive/_libarchive_wrap.c:2963:10: fatal error: archive.h: No such file or directory
       2963 | #include <archive.h>
            |          ^~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-libarchive
  Running setup.py clean for python-libarchive
Failed to build python-libarchive
Installing collected packages: yara-python, python-libarchive, mypy-extensions, mccabe, distlib, typing-extensions, tomli, toml, six, pyparsing, pyflakes, pycodestyle, py, pluggy, platformdirs, pathspec, isort, filelock, click, virtualenv, pydantic, packaging, flake8, black, tox, stacs
  Running setup.py install for python-libarchive ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for python-libarchive did not run successfully.
  │ exit code: 1
  ╰─> [24 lines of output]
      stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/dist.py:516: UserWarning: Normalizing '4.0.1-1' to '4.0.1.post1'
        warnings.warn(tmpl.format(**locals()))
      running install
      stacs-0.4.6/venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/__init__.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/_libarchive.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/zip.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/tar.py -> build/lib.linux-x86_64-cpython-310/libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/libarchive
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ilibarchive -I stacs-0.4.6/venv/include -I/usr/include/python3.10 -c libarchive/_libarchive_wrap.c -o build/temp.linux-x86_64-cpython-310/libarchive/_libarchive_wrap.o
      libarchive/_libarchive_wrap.c:2963:10: fatal error: archive.h: No such file or directory
       2963 | #include <archive.h>
            |          ^~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> python-libarchive

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Please, fix it.

opened by sv-atoslav 2

Remove libarchive requirement

Overview

The use of libarchive provides support for 7-zip, Debian packages, and ISO files. However, it comes at the cost of needing libarchive. On macOS this causes a lot of confusion in installation due an older version of libarchive reportedly shipping with macOS which is apparently not compatible with the libarchive wrapper in use.

In order to try and remove more native dependencies, this task is to investigate the use of Kaitai structs and extraction implementations added to STACS directly in the archive loader.
enhancement

opened by darkarnium 1
[0.2.0] Namespace STACS scan under `stacs.scan`
Overview

In order to enable better interoperability between other STACS components, this PR is to move the STACS scanner under stacs.scan.

Please Note: This updates represent a breaking change for any users who may be importing STACS as Python module and calling STACS internals directly. Users of the the CLI and the STACS Docker container image are unaffected. Additionally, no changes are required to existing rule packs and ignore lists.

However, the version number has been updated to reflect this potentially breaking change.

🛠️ New Features

N/A

🍩 Improvements

Moving stacs to stacs.scan for improved interoperability with other STACS components.

🐛 Bug Fixes

A few non-impacting updates to docstrings

enhancement
opened by darkarnium 1
Some cleanups and improvements in code
Resolving user paths (user-expanding) in included files

Fixed included ignore lists using os.path.join twice, which technically doesn't matter but doesn't look logical at all

Cleaned up includes code in packs and ignore lists to

enhancement
opened by renbou 1
[0.4.15] Relax pins to allow updates within major versions.
Overview

🛠️ New Features

N/A

🍩 Improvements

Allow updates within the same major version for all dependencies.

Allows installation into environments with newer versions of dependencies. Closes #32.

🐛 Bug Fixes

N/A
opened by darkarnium 0
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0

Releases(0.4.15)

0.4.15(Dec 19, 2022)
Overview

🛠️ New Features

N/A

🍩 Improvements

Allow updates within the same major version for all dependencies.

Allows installation into environments with newer versions of dependencies. Closes #32.

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.14(Nov 24, 2022)
Overview

🛠️ New Features

N/A

🍩 Improvements

Better handling of unexpected / malformed data.

Allows graceful handling of binary data inside of files reported as text (#30)

🐛 Bug Fixes

Unhandled UnicodeDecodeError in some circumstances.

Source code(tar.gz)
Source code(zip)
0.4.13(Nov 6, 2022)
Overview

Additions to internal data-structures to provide additional file metadata for downstream processing.

🛠️ New Features

Add filesize onto internal entries model.

🍩 Improvements

N/A

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.12(Oct 13, 2022)
Overview

This release resolves an issue where if an input archive filename contained no dots (.), the subsequent os.path.join would replace the cache path - due to missing basename call.

🛠️ New Features

N/A

🍩 Improvements

N/A

🐛 Bug Fixes

Input binary overwrite when filename contains no dots (.).

Source code(tar.gz)
Source code(zip)
0.4.11(Oct 11, 2022)
Overview

This release adds support for zlib unpacking, weighted archive detection, decompression of lzma, bz2, and zlib DMG chunks, and resolves a trivial bug in the DMG implementation which lead to incomplete extraction of blobs from DMGs.

🛠️ New Features

Zlib support

🍩 Improvements

Handle lzma, zlib, and bz2 compressed chunks in DMG reader.

Weighted archive detection.

Allows preferencing "container" formats when blobs contain multiple detected formats.

🐛 Bug Fixes

Chunking issue in DMG reader.

Source code(tar.gz)
Source code(zip)
0.4.10(Oct 9, 2022)
Overview

This pull-request adds support for Apple Disk Images (DMGs).

🛠️ New Features

Support for Apple Disk Images (DMGs).

🍩 Improvements

Tags from rule matches are propagated on internal Source model.

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.9(Oct 3, 2022)
Overview

This pull-request adds the parent MD5 sum to the internal manifest.Entry model, and swaps the zstd decompression library to improve compatibility.

🛠️ New Features

Add parent MD5 of nested archive members to internal entries model.

🍩 Improvements

Dependency updates.

Change zstd implementation for better support.

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.8(Aug 25, 2022)
Overview

This release resolves a bug which lead to incorrect unpacking of certain archive types.

🛠️ New Features

N/A

🍩 Improvements

N/A

🐛 Bug Fixes

Fix chunking bug in libarchive unpacker.

Source code(tar.gz)
Source code(zip)
0.4.7(Jul 12, 2022)
Overview

This release provides a human readable output option (--pretty), binary wheels to simplify installation on common platforms, and adds support for additional archive types.

🛠️ New Features

Human readable output.

Via --pretty command-line option.

Designed for humans, where SARIF is designed for machines.

New archives types supported.

eXtensible ARchive format (xar).

Also provides support for macOS pkg files.

CPIO (cpio).

Enables processing of several new archive types, such as rpm.

Microsoft Cabinet (cab).

archiver (ar).

Enables processing of several new archive types, such as dpkg.

Roshal Archive (rar).

🍩 Improvements

Binary wheels are now provided for popular platforms.

This should remove the need to fight with out-of-date libarchive on macOS.

🐛 Bug Fixes

Fixed several issues when processing malformed archives.

Source code(tar.gz)
Source code(zip)
0.4.6(Apr 18, 2022)
Overview

This release introduces a number of convenience and performance improvements to STACS.

🛠️ New Features

Exit status is now non-zero if there are unsuppressed findings.

This was based on the suggestion as part of #19.

Exit status is now 100 when there are unsuppressed findings.

STACS supports multiple paths to scan.

This was based on the suggestion as part of #18.

Mutiple paths can now be passed directly to STACS at runtime, and they will all be scanned.

When running a container, specification of thread count, whether to enable debug, and whether to skip unprocessable archives can be controlled by environment variables

STACS_SKIP_UNPROCESSABLE

STACS_THREADS

STACS_DEBUG

🍩 Improvements

SIGNIFICANT performance improvements due to optimisation of rules.

In testing this has seen scan runtime improvements of up to 8x in some cases.

Dependencies have been updated.

🐛 Bug Fixes

Resolved a minor edge-case where non UTF-8 characters would break sample generation.

Unprocessable samples will now be base64 encoded - the same way that binary data is handled.

Source code(tar.gz)
Source code(zip)
0.4.5(Mar 4, 2022)
Overview

This release adds a few convenience features to STACS. It also relaxes the version constraint to allow STACS to be installed under Python 3.8.

🛠️ New Features

Specification of the number of threads is now possible via --threads (defaults to: 10)

🍩 Improvements

Version constraint has been relaxed to allow Python 3.8 (per request #16).

Cache directories are attempted to be cleaned-up after run.

Cache directories now have a timestamp appended to avoid collisions if previous run failed.

Minor changes to release management process to allow easier rebuilds of containers with new rules.

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.4(Jan 10, 2022)
Overview

This release makes a small change to the way the STACS container runs. The container entrypoint now checks whether the scan directory contains an ignore list. If so, this will be loaded automatically.

🛠️ New Features

The STACS container now attempts to automatically load a stacs.ignore.json from the scan directory.

Latest STACS rules, including new rules for:

JSON Web Tokens (JWTs)

Stripe API tokens (secret keys).

🍩 Improvements

Clean-up of Pydantic validator for the ignore list model.

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.3(Jan 7, 2022)
Overview

🛠️ New Features

Added the ability to skip corrupt archives rather than failing the STACS run.

This must be explicitly enabled using the --skip-unprocessable flag.

A warning message will be produced for each 'skipped' archive.

🍩 Improvements

Added additional debug level logging related to file enumeration.

This output is only provided when run with the --debug flag.

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.2(Jan 4, 2022)
Overview

This releases uses a different libarchive wrapper in order to improve compatibility, due to an unresolved issue with the currently used wrapper.

🛠️ New Features

N/A

🍩 Improvements

Move to python-libarchive from libarchive for better cross distribution compatibility.

This is mostly a work around, so it may be swapped in future, but this should be a transparent change.

🐛 Bug Fixes

N/A

Source code(tar.gz)
Source code(zip)
0.4.1(Dec 31, 2021)
Overview

🛠️ New Features

N/A

🍩 Improvements

N/A

🐛 Bug Fixes

Adjust samples to ensure that the sample after the finding starts from the end of the finding, rather than the start.

Source code(tar.gz)
Source code(zip)
0.4.0(Dec 29, 2021)
Overview

🛠️ New Features

N/A

🍩 Improvements

Added development extras to simplify development environment setup.

Identification of rules now uses the rule identifier, rather than name.

🐛 Bug Fixes

Fix off-by-one in finding line number counting.

Source code(tar.gz)
Source code(zip)
0.3.2(Dec 27, 2021)
Overview

🛠️ New Features

N/A

🍩 Improvements

N/A

🐛 Bug Fixes

Ensure ignore-lists without a module specified default to a fully qualified module (stacs.scan.scanner.rules).

Source code(tar.gz)
Source code(zip)
0.3.1(Dec 11, 2021)
Overview

🛠️ New Features

N/A

🍩 Improvements

N/A

🐛 Bug Fixes

Minor fixes to use the correct offset for ISO file magic.

Source code(tar.gz)
Source code(zip)
0.3.0(Dec 11, 2021)
Overview

Minor fixes to handle edge-cases in ZIP handling, as well as adding support for more archive types 🎉

🛠️ New Features

Support for iso, rpm, and 7z archive types - via libarchive.

🍩 Improvements

N/A

🐛 Bug Fixes

Gracefully handle encrypted ZIPs and ZIPs with invalid members.

Source code(tar.gz)
Source code(zip)
0.2.0(Aug 29, 2021)
🛠️ New Features

N/A

🍩 Improvements

Moving stacs to stacs.scan for improved interoperability with other STACS components.

🐛 Bug Fixes

A few non-impacting updates to docstrings

Source code(tar.gz)
Source code(zip)
0.1.4(Aug 27, 2021)
🛠️ New Features

Pulled in latest STACS rules, which includes new rules for the following crypt hashes:

md5

sha1

sha256

sha512

bcrypt

nthash

🍩 Improvements

Better path expansion (with thanks to @renbou!)

🐛 Bug Fixes

Spelling fixes and code clean-up (with thanks to @renbou!)

Source code(tar.gz)
Source code(zip)
0.1.3(Aug 18, 2021)
🛠️ New Features

Recursive archive scanning (tar, gz, xz, bz2, zip formats).

Based on identified file formats, rather than file suffix.

Able to support formats based on these types - such as apk, jar, etc.

Composable ignore lists.

Composable from local files.

Composable rule packs.

Composable from local files.

SARIF format output.

20-byte "Context" for findings.

Provides information about lines / bytes before and after the finding.

CLI support.

Docker container support.

Scan results are sent to STDOUT and log messages to STDERR.

🍩 Improvements

N/A

🐛 Bug Fixes

Fixes binary context encoding issue during SARIF output.

Source code(tar.gz)
Source code(zip)