An easy-to-use library for emulating code in minidump files.

Overview

dumpulator

Note: This is a work-in-progress prototype, please treat it as such.

An easy-to-use library for emulating code in minidump files.

Example

The example below opens test.dmp (download a copy here), allocates some memory and calls the decryption function at 0x140001000 to decrypt the string at 0x140003000:

from dumpulator import Dumpulator

dp = Dumpulator("test.dmp", trace=True)
temp_addr = dp.allocate(256)
dp.call(0x140001000, [temp_addr, 0x140003000])
decrypted = dp.read_str(temp_addr)
print(f"decrypted: '{decrypted}'")

The test.dmp is collected at the entry point of the tests/StringEncryptionFun example.

Collecting the dump

There is a simple plugin for x64dbg available in the MiniDumpPlugin folder (you can also download a precompiled binary in the releases). To use it you pause execution and execute the command MiniDump my.dmp.

Installation

From PyPI (latest release):

python -m pip install dumpulator

To install from source:

python setup.py install

Install for a development environment:

python setup.py develop

Credits

Comments
  • Implement a handle manager

    Implement a handle manager

    Handles are currently hacked in. Add a handle manager that allows something like this:

    def ZwCreateFile_syscall(...):
        handle_data = (...)
        hFile = dp.handles.new(data)
        return hFile
    
    def ZwReadFile_syscall(hFile: HANDLE, ...):
        handle_data = dp.handles.get(hFile, None)
        return
    
    def ZwCloseHandle(hFile):
        dp.handles.close(hFile)
    

    It should also support duplication (every handle points to refcounted data).

    feature 
    opened by mrexodia 5
  • Implementing a standalone memory manager

    Implementing a standalone memory manager

    This is the first pass of the memory manager, I have not implemented it into dumpulator code. On its own, it works, it is not optimized at all.

    Please let me know if you think anything should be done another way and how I should start replacing mem_map calls.

    opened by Calastrophe 4
  • x64 dump of x86 process fail to emulate

    x64 dump of x86 process fail to emulate

    I have noticed if you do a dump on x64 OS of a x86 process (from task manager or procdump64 for example) dumpulator would try to emulate everything in x64 even though most of the dump is actually x86 causing some unexpected behavior, it doesn't happen if you do a x86 dump from x32dbg or from procdump(32).

    feature 
    opened by thewhitegoatcb 4
  • Memory manager refactor

    Memory manager refactor

    I rewrote the memory manager to use the paging, although an issue that arose was that the checking function is a bit limited at the moment.

    I have had my hand at trying to improve it - it will detect if something was allocating in already allocated space, or "enveloping" allocated space, but if it were to have one 'foot' in at any side of the region - it won't catch the allocation error.

    This one is way faster ( mainly because the checker function had to be limited ), hopefully you guys can point me in the right direction if there is something glaringly wrong. Thanks.

    opened by Calastrophe 3
  • File Handle Fixes + New Functionality

    File Handle Fixes + New Functionality

    Latest update to the handle manager broke how normal files were loaded and used. This is intended to fix this issue as well as adding in new functionality.

    Inspired by the mapped files idea, I have implemented create_file within handles, this will depending on creation flags either open an existing file and load it in as a mapped file with it's data or create a new empty mapped file.

    It's not perfect yet but it's a good start in the right direction, this way when emulating something that relies on touching files we should in theory be able to follow and capture the changes without them effecting the system, I'm sure there will be bugs behind this, something like \\.\PhysicalDrive0 might be an issue but I hope it would just raise an exception rather than try to load the entire disk into memory...

    Added handles.create_file to add new files to mapped files Refactored FileObject and added write Added CreateDisposition flags Updated HandleTest exe

    opened by oopsmishap 3
  • Implement exception handling

    Implement exception handling

    When unicorn gets an exception it needs to build a stack frame and set EIP/RIP on KiUserExceptionDispatcher

    http://www.nynaeve.net/?p=201

    It also needs a test application and a prepared dump in Windows Sandbox for CI.

    feature 
    opened by mrexodia 3
  • Module Manager

    Module Manager

    First pass of a Module Manager for #17 Probably good for a PR as is but I would like to finish it and have good unit test coverage before then. Just creating the PR so you can have a look 👍

    opened by oopsmishap 2
  • Basic Export Forwaring

    Basic Export Forwaring

    Resolves export forwards.

    1. In Module's _parse_pe, if export points to .rdata store as an export forwarder (only caveat is x64 ntdll.dll's RtlNtdllName which is located in .rdata)
    2. After all modules have been loaded in iterate over modules to then parse the export forwarders, this is done by overwriting the current export address with the resolved forward export's address
    3. If we could not resolve the forwarder leave address to zero and assert we could not resolve the function within find_export

    Currently does not resolve API sets, but we can cross that bridge later.

    opened by oopsmishap 1
  • Incorrect width for Enums in 64bit

    Incorrect width for Enums in 64bit

    When a syscall is processes an enum with a 64bit dump the value is treated as a 64bit integer. Within Windows syscalls most enums are treated as 32bit values. I can't find the documentation for this but if you start going through syscalls that use enums you will notice the trend.

    When a syscall with an enum is called Dumpulator processes it as the following resulting in a ValueError

    Error: ValueError: 549755813892 is not a valid FSINFOCLASS
    
    549755813892 = 0x8000000004
    
    Previous argument in the call had the value: 0x8
    

    I have tried to implement a CTypes style of python enum's to no avail. The following hotfix is enough to fix the issue, but could run into issues if an enum size is anything other than 32bit wide, however I have yet to find one that isn't 32bit wide.

    elif issubclass(argtype, Enum):
        try:
            argvalue = argtype(dp.args[i] & 0xFFFFFFFF)
        except KeyError as x:
            raise Exception(f"Unknown enum value {dp.args[i]} for {type(argtype)}")
    
    opened by oopsmishap 1
  • Handle manager

    Handle manager

    PR for #16 for the implantation of a handle manager.

    Also added needed syscall functionality to get simple file reads working. Source I tested is included with the PR. This currently is working for 32bit.

    Unicorn failed on 64bit due to the extended instruction set usage within ntdll!memset

    opened by oopsmishap 1
  • Fix loading of kernel32 on modern systems

    Fix loading of kernel32 on modern systems

    SizeOfImage is not consecutive in memory. So unicorn fails to read the memory. Fix would be to read the memory in chunks of 0x1000 pages, or only pass the first page to pefile and load in pages when necessary.

    bug good first issue 
    opened by mrexodia 1
  • Dumpulator takes a long time to load when handling large dumps

    Dumpulator takes a long time to load when handling large dumps

    Feature Request: Please look into addressing the issue where large dump files aren't easily processed by dumpulator.

    Description: Dumpulator is taking an excessively long time to load a dump file of 40MB. I was testing the decryption of an RC4 encrypted configuration block from gh0stRAT. I dumped out the process and it was 40MB. I fed this dump into dumpulator and it was still trying to load after 20 minutes.

    Sample: https://www.virustotal.com/gui/file/0a9c881b857d4044cb6dfeba682190d7d9dc6ef94bc149cac77f3e0f65e9c69a

    feature 
    opened by gakar06 3
  • Proper (extensible) testing

    Proper (extensible) testing

    Currently the "tests" are just running the "getting-started" examples. This doesn't scale and makes it difficult to find regressions.

    The test framework:

    • [x] A single unified Visual Studio solution with test executables
    • [ ] Build the solution with GitHub Actions so new feature PRs can be tested in the same PR
    • [x] A dump of a minimal Windows usermode environment (ntdll, kernel32, kernelbase, x64+x86)
    • [x] A dump of a more "complete" environment (winsock, cryptapi, advapi32, shell32, user32, etc.)
    • [x] Load the test executables with dp.map_module
    • [x] Execute the (exported) test functions on a fresh Dumpulator instance each time
    • [ ] Calculate code coverage per test

    Necessary tests:

    All of these should use /NODEFAULTLIB to not have to care about the MSVC runtime initialization (which uses a lot of unimplemented syscalls)

    • [ ] ExceptionTest (for testing different exception scenarios)
    • [ ] LoadLibrary (to test map_module and the whole PEB loading chain)
    • [ ] StringEncryptionFun
    • [ ] More depending on coverage...
    feature 
    opened by mrexodia 0
  • Trace points

    Trace points

    The idea would be to generate a unique number for each syscall invocation and print it in the log. The sequence should be deterministic for a given dumpulator version (although preferably across versions as well). Uses cases:

    • "Breakpoint" (stop emulation or literally a debugger breakpoint) at a certain ID you saw previously in the log for closer inspection.
    • Enable single step tracing (slow) only after/between trace points.
    • Dump the state after a certain trace point to avoid re-running the same code over and over again (needs #13)
    feature 
    opened by mrexodia 0
  • Saving/restoring state

    Saving/restoring state

    Right now the state after a .call or .start persists. It would be nice to have a feature to roll back memory changes without having to reload the dump.

    I think unicorn has a feature for this.

    feature 
    opened by mrexodia 1
  • Design ctypes equivalent for syscall implementation

    Design ctypes equivalent for syscall implementation

    Currently the type system for syscalls is very rough and you need to do a lot of manual work. A type system similar to ctypes needs to be implemented where you can set struct members, work with enums etc.

    Once the type system is complete a pdb/header parser can be implemented to support all the native types.

    feature 
    opened by mrexodia 1
Releases(v0.1.2)
  • v0.1.2(Jan 6, 2023)

  • v0.1.1(Nov 23, 2022)

    What's Changed

    • Fix for argument width for Enums by @oopsmishap in https://github.com/mrexodia/dumpulator/pull/31
    • Basic Export Forwaring by @oopsmishap in https://github.com/mrexodia/dumpulator/pull/32
    • Start implementing a test framework https://github.com/mrexodia/dumpulator/pull/33
    • Fix a crash when fetching instructions from unmapped memory
    • Support WRITECOMBINE, NOCACHE and GUARD pages
    • Fix regression where 32-bit pointers were not masked correctly
    • Emulate unsupported instructions (RDRAND)
    • Improve performance of the memory map on startup

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.1.0...v0.1.1

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Oct 11, 2022)

    What's Changed

    • Initial version of exception support, VEH, VCH, SEH, UnhandledExceptionFilter are all working!
    • 8 implement write byte write word by @oopsmishap in https://github.com/mrexodia/dumpulator/pull/18
    • Handle manager by @oopsmishap in https://github.com/mrexodia/dumpulator/pull/19
    • Added support for setting/getting flags in Registers() by @Calastrophe in https://github.com/mrexodia/dumpulator/pull/20
    • File Handle Fixes + New Functionality by @oopsmishap in https://github.com/mrexodia/dumpulator/pull/21
    • Implementing a standalone memory manager by @Calastrophe in https://github.com/mrexodia/dumpulator/pull/22
    • New memory manager by @mrexodia in https://github.com/mrexodia/dumpulator/pull/25
    • Module manager by @oopsmishap and @mrexodia in https://github.com/mrexodia/dumpulator/pull/26

    New Contributors

    • @Calastrophe made their first contribution in https://github.com/mrexodia/dumpulator/pull/20

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.11...v0.1.0

    Source code(tar.gz)
    Source code(zip)
  • v0.0.11(Jun 20, 2022)

    What's Changed

    • Documentation improvements
    • Minor bugfixes

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.10...v0.0.11

    Source code(tar.gz)
    Source code(zip)
  • v0.0.10(Apr 4, 2022)

    What's Changed

    • Use the thread that caused the exception in the dump instead of the first thread

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.9...v0.0.10

    Source code(tar.gz)
    Source code(zip)
  • v0.0.9(Mar 20, 2022)

    What's Changed

    • Add support for Dumpulator(quiet=True)
    • Allow dp.regs[name]
    • Stores handles in Dumpulator class
    • Implement a few more syscalls

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.8...v0.0.9 @mrexodia

    Source code(tar.gz)
    Source code(zip)
  • v0.0.8(Mar 11, 2022)

    What's Changed

    • Add empty bodies for every single syscall by @mrexodia in https://github.com/mrexodia/dumpulator/pull/11

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.7...v0.0.8

    Source code(tar.gz)
    Source code(zip)
  • v0.0.7(Dec 23, 2021)

    What's Changed

    • Support unicorn2 by @jtorreno in https://github.com/mrexodia/dumpulator/pull/7
    • Add support for dp.call(0x140001000, regs={'rcx': temp_addr, 'rdx': 0x140017000})

    New Contributors

    • @jtorreno made their first contribution in https://github.com/mrexodia/dumpulator/pull/7

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.6...v0.0.7

    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Nov 29, 2021)

    • Deterministic order for operand registers in trace
    • Remove trap flag at the beginning of a trace and set gs_base
    • Decode utf-16 strings properly
    • Fix a bug with memory mapping addresses >= 0x80000000 on 32-bit
    • Add a helper script to convert x64dbg traces to dumpulator traces
    • Ignore some files and stub ZwQueryInformationToken

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.4...v0.0.5

    Source code(tar.gz)
    Source code(zip)
  • v0.0.4(Nov 24, 2021)

    What's Changed

    • Improve the trace format to make it more human-readable (include module transitions and export labels)

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.3...v0.0.4

    Source code(tar.gz)
    Source code(zip)
  • v0.0.3(Nov 21, 2021)

    What's Changed

    • Polish README
    • Initial implementation of 32-bit processes
    • Move the MiniDumpPlugin to a separate repository
    • Add GitHub Actions to publish to PyPI

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.2...v0.0.3

    Source code(tar.gz)
    Source code(zip)
  • v0.0.2(Nov 21, 2021)

    What's Changed

    • Vendor the locally patched minidump library, see: https://github.com/skelsec/minidump/pull/28

    Full Changelog: https://github.com/mrexodia/dumpulator/compare/v0.0.1...v0.0.2

    Source code(tar.gz)
    Source code(zip)
Owner
Duncan Ogilvie
Passionate C++ developer and reverse engineer. Main developer of @x64dbg. Also familiar with C#, Haskell, Assembly, Python and a bunch of web-related languages.
Duncan Ogilvie
Python function to stream unzip all the files in a ZIP archive: without loading the entire ZIP file or any of its files into memory at once

Python function to stream unzip all the files in a ZIP archive: without loading the entire ZIP file or any of its files into memory at once

Department for International Trade 206 Jan 2, 2023
csv2ir is a script to convert ir .csv files to .ir files for the flipper.

csv2ir csv2ir is a script to convert ir .csv files to .ir files for the flipper. For a repo of .ir files, please see https://github.com/logickworkshop

Alex 38 Dec 31, 2022
shred - A cross-platform library for securely deleting files beyond recovery.

shred Help the project financially: Donate: https://smartlegion.github.io/donate/ Yandex Money: https://yoomoney.ru/to/4100115206129186 PayPal: https:

null 4 Sep 4, 2021
ValveVMF - A python library to parse Valve's VMF files

ValveVMF ValveVMF is a Python library for parsing .vmf files for the Source Engi

pySourceSDK 2 Jan 2, 2022
Uproot is a library for reading and writing ROOT files in pure Python and NumPy.

Uproot is a library for reading and writing ROOT files in pure Python and NumPy. Unlike the standard C++ ROOT implementation, Uproot is only an I/O li

Scikit-HEP Project 164 Dec 31, 2022
A Python library that provides basic functions to read / write Aseprite format files

A Python library that provides basic functions to read / write Aseprite format files

Joe Trewin 1 Jan 13, 2022
A simple library for temporary storage of small files

TemporaryStorage An simple library for temporary storage of small files. Navigation Install Usage In Python console As a standalone application List o

null 2 Apr 17, 2022
pydicom - Read, modify and write DICOM files with python code

pydicom is a pure Python package for working with DICOM files. It lets you read, modify and write DICOM data in an easy "pythonic" way.

DICOM in Python 1.5k Jan 4, 2023
Python script for converting figma produced SVG files into C++ JUCE framework source code

AutoJucer Python script for converting figma produced SVG files into C++ JUCE framework source code Watch the tutorial here! Getting Started Make some

SuperConductor 1 Nov 26, 2021
Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series. The Fourier series can be animated and visualized, the function can be output as a two dimensional vector for Desmos and there is a method to output the coefficients as LaTeX code.

Alexander 12 Jan 1, 2023
Python code snippets for extracting PDB codes from .fasta files

Python_snippets_for_bioinformatics Python code snippets for extracting PDB codes from .fasta files If you have a single .fasta file for all protein se

Sofi-Mukhtar 3 Feb 9, 2022
RMfuse provides access to your reMarkable Cloud files in the form of a FUSE filesystem

RMfuse provides access to your reMarkable Cloud files in the form of a FUSE filesystem. These files are exposed either in their original format, or as PDF files that contain your annotations. This lets you manage files in the reMarkable Cloud using the same tools you use on your local system.

Robert Schroll 82 Nov 24, 2022
A JupyterLab extension that allows opening files and directories with external desktop applications.

A JupyterLab extension that allows opening files and directories with external desktop applications.

martinRenou 0 Oct 14, 2021
This is a junk file creator tool which creates junk files in Internal Storage

This is a junk file creator tool which creates junk files in Internal Storage

KiLL3R_xRO 3 Jun 20, 2021
Maltego transforms to pivot between PE files based on their VirusTotal codeblocks

VirusTotal Codeblocks Maltego Transforms Introduction These Maltego transforms allow you to pivot between different PE files based on codeblocks they

Ariel Jungheit 18 Feb 3, 2022
MHS2 Save file editing tools. Transfers save files between players, switch and pc version, encrypts and decrypts.

SaveTools MHS2 Save file editing tools. Transfers save files between players, switch and pc version, encrypts and decrypts. Credits Written by Asteris

null 31 Nov 17, 2022
Creates folders into a directory to categorize files in that directory by file extensions and move all things from sub-directories to current directory.

Categorize and Uncategorize Your Folders Table of Content TL;DR just take me to how to install. What are Extension Categorizer and Folder Dumper Insta

Furkan Baytekin 1 Oct 17, 2021
A tool written in python to generate basic repo files from github

A tool written in python to generate basic repo files from github

Riley 7 Dec 2, 2021
dotsend is a web application which helps you to upload your large files and share file via link

dotsend is a web application which helps you to upload your large files and share file via link

Devocoe 0 Dec 3, 2022