Python codecs extension featuring CLI tools for encoding/decoding anything

Alex

Last update: Dec 30, 2022

CodExt

Encode/decode anything.

This library extends the native codecs library (namely for adding new custom encodings and character mappings) and provides a myriad of new encodings (static or parametrized, like rot or xor), hence its named combining CODecs EXTension.

$ pip install codext

Want to contribute a new codec ?	Want to contribute a new macro ?
Check the documentation first Then PR your new codec	PR your updated version of `macros.json`

🔍 Demonstrations

💻 Usage (main CLI tool)

$ codext -i test.txt encode dna-1
GTGAGCGGGTATGTGA

$ echo -en "test" | codext encode morse
- . ... -

$ echo -en "test" | codext encode braille
⠞⠑⠎⠞

$ echo -en "test" | codext encode base100
👫👜👪👫

Chaining codecs

$ echo -en "Test string" | codext encode reverse
gnirts tseT

$ echo -en "Test string" | codext encode reverse morse
--. -. .. .-. - ... / - ... . -

$ echo -en "Test string" | codext encode reverse morse dna-2
AGTCAGTCAGTGAGAAAGTCAGTGAGAAAGTGAGTGAGAAAGTGAGTCAGTGAGAAAGTCAGAAAGTGAGTGAGTGAGAAAGTTAGAAAGTCAGAAAGTGAGTGAGTGAGAAAGTGAGAAAGTC

$ echo -en "Test string" | codext encode reverse morse dna-2 octal
101107124103101107124103101107124107101107101101101107124103101107124107101107101101101107124107101107124107101107101101101107124107101107124103101107124107101107101101101107124103101107101101101107124107101107124107101107124107101107101101101107124124101107101101101107124103101107101101101107124107101107124107101107124107101107101101101107124107101107101101101107124103

$ echo -en "AGTCAGTCAGTGAGAAAGTCAGTGAGAAAGTGAGTGAGAAAGTGAGTCAGTGAGAAAGTCAGAAAGTGAGTGAGTGAGAAAGTTAGAAAGTCAGAAAGTGAGTGAGTGAGAAAGTGAGAAAGTC" | codext -d dna-2 morse reverse
test string

Using macros

$ codext add-macro my-encoding-chain gzip base63 lzma base64

$ codext list macros
example-macro, my-encoding-chain

$ echo -en "Test string" | codext encode my-encoding-chain
CQQFAF0AAIAAABuTgySPa7WaZC5Sunt6FS0ko71BdrYE8zHqg91qaqadZIR2LafUzpeYDBalvE///ug4AA==

$ codext remove-macro my-encoding-chain

$ codext list macros
example-macro

💻 Usage (base CLI tool)

$ echo "Test string !" | base122
*.7!ft9�-f9Â

$ echo "Test string !" | base91 
"ONK;WDZM%Z%xE7L

$ echo "Test string !" | base91 | base85
B2P|BJ6A+nO(j|-cttl%

$ echo "Test string !" | base91 | base85 | base36 | base58-flickr
QVx5tvgjvCAkXaMSuKoQmCnjeCV1YyyR3WErUUErFf

$ echo "Test string !" | base91 | base85 | base36 | base58-flickr | base58-flickr -d | base36 -d | base85 -d | base91 -d
Test string !

$ echo "Test string !" | base91 | base85 | base36 | base58-flickr | debase -m 3
Test string !

$ echo "Test string !" | base91 | base85 | base36 | base58-flickr | debase -f Test
Test string !

💻 Usage (Python)

Getting the list of available codecs:

>> codext.encode("this is a test", "base58-ripple") 'jo9rA2LQwr44eBmZK7E' >>> codext.encode("this is a test", "base58-url") 'JN91Wzkpa1nnDbLyjtf' >>> codecs.encode("this is a test", "base100") '👫👟👠👪🐗👠👪🐗👘🐗👫👜👪👫' >>> codecs.decode("👫👟👠👪🐗👠👪🐗👘🐗👫👜👪👫", "base100") 'this is a test' >>> for i in range(8): print(codext.encode("this is a test", "dna-%d" % (i + 1))) GTGAGCCAGCCGGTATACAAGCCGGTATACAAGCAGACAAGTGAGCGGGTATGTGA CTCACGGACGGCCTATAGAACGGCCTATAGAACGACAGAACTCACGCCCTATCTCA ACAGATTGATTAACGCGTGGATTAACGCGTGGATGAGTGGACAGATAAACGCACAG AGACATTCATTAAGCGCTCCATTAAGCGCTCCATCACTCCAGACATAAAGCGAGAC TCTGTAAGTAATTCGCGAGGTAATTCGCGAGGTAGTGAGGTCTGTATTTCGCTCTG TGTCTAACTAATTGCGCACCTAATTGCGCACCTACTCACCTGTCTATTTGCGTGTC GAGTGCCTGCCGGATATCTTGCCGGATATCTTGCTGTCTTGAGTGCGGGATAGAGT CACTCGGTCGGCCATATGTTCGGCCATATGTTCGTCTGTTCACTCGCCCATACACT >>> codext.decode("GTGAGCCAGCCGGTATACAAGCCGGTATACAAGCAGACAAGTGAGCGGGTATGTGA", "dna-1") 'this is a test' >>> codecs.encode("this is a test", "morse") '- .... .. ... / .. ... / .- / - . ... -' >>> codecs.decode("- .... .. ... / .. ... / .- / - . ... -", "morse") 'this is a test' >>> with open("morse.txt", 'w', encoding="morse") as f: f.write("this is a test") 14 >>> with open("morse.txt",encoding="morse") as f: f.read() 'this is a test' >>> codext.decode(""" = X : x n r y Y y p a ` n | a o h ` g o z """, "whitespace-after+before") 'CSC{not_so_invisible}' >>> print(codext.encode("An example test string", "baudot-tape")) ***.** . * ***.* * . .* * .* . * ** .* ***.** ** .** .* * . * *. * .* * *. * *. * * . * *. * *. * ***. *.* ***.* * .* ">

>>> import codext

>>> codext.list()
['ascii85', 'base85', 'base100', 'base122', ..., 'tomtom', 'dna', 'html', 'markdown', 'url', 'resistor', 'sms', 'whitespace', 'whitespace-after-before']

>>> codext.encode("this is a test", "base58-bitcoin")
'jo91waLQA1NNeBmZKUF'

>>> codext.encode("this is a test", "base58-ripple")
'jo9rA2LQwr44eBmZK7E'

>>> codext.encode("this is a test", "base58-url")
'JN91Wzkpa1nnDbLyjtf'

>>> codecs.encode("this is a test", "base100")
'👫👟👠👪🐗👠👪🐗👘🐗👫👜👪👫'

>>> codecs.decode("👫👟👠👪🐗👠👪🐗👘🐗👫👜👪👫", "base100")
'this is a test'

>>> for i in range(8):
        print(codext.encode("this is a test", "dna-%d" % (i + 1)))
GTGAGCCAGCCGGTATACAAGCCGGTATACAAGCAGACAAGTGAGCGGGTATGTGA
CTCACGGACGGCCTATAGAACGGCCTATAGAACGACAGAACTCACGCCCTATCTCA
ACAGATTGATTAACGCGTGGATTAACGCGTGGATGAGTGGACAGATAAACGCACAG
AGACATTCATTAAGCGCTCCATTAAGCGCTCCATCACTCCAGACATAAAGCGAGAC
TCTGTAAGTAATTCGCGAGGTAATTCGCGAGGTAGTGAGGTCTGTATTTCGCTCTG
TGTCTAACTAATTGCGCACCTAATTGCGCACCTACTCACCTGTCTATTTGCGTGTC
GAGTGCCTGCCGGATATCTTGCCGGATATCTTGCTGTCTTGAGTGCGGGATAGAGT
CACTCGGTCGGCCATATGTTCGGCCATATGTTCGTCTGTTCACTCGCCCATACACT
>>> codext.decode("GTGAGCCAGCCGGTATACAAGCCGGTATACAAGCAGACAAGTGAGCGGGTATGTGA", "dna-1")
'this is a test'

>>> codecs.encode("this is a test", "morse")
'- .... .. ... / .. ... / .- / - . ... -'

>>> codecs.decode("- .... .. ... / .. ... / .- / - . ... -", "morse")
'this is a test'

>>> with open("morse.txt", 'w', encoding="morse") as f:
	f.write("this is a test")
14

>>> with open("morse.txt",encoding="morse") as f:
	f.read()
'this is a test'

>>> codext.decode("""
      =            
              X         
   :            
      x         
  n  
    r 
        y   
      Y            
              y        
     p    
         a       
 `          
            n            
          |    
  a          
o    
       h        
          `            
          g               
           o 
   z      """, "whitespace-after+before")
'CSC{not_so_invisible}'

>>> print(codext.encode("An example test string", "baudot-tape"))
***.**
   . *
***.* 
*  .  
   .* 
*  .* 
   . *
** .* 
***.**
** .**
   .* 
*  .  
* *. *
   .* 
* *.  
* *. *
*  .  
* *.  
* *. *
***.  
  *.* 
***.* 
 * .*

📃 List of codecs

BaseXX

ascii85: classical ASCII85 (Python3 only)
baseN: see base encodings (incl base32, 36, 45, 58, 62, 63, 64, 91, 100, 122)
base-genericN: see base encodings ; supports any possible base

Binary

Common

a1z26: keeps words whitespace-separated and uses a custom character separator
cases: set of case-related encodings (including camel-, kebab-, lower-, pascal-, upper-, snake- and swap-case, slugify, capitalize, title)
dummy: set of simple encodings (including reverse and word-reverse)
octal: dummy octal conversion (converts to 3-digits groups)
octal-spaced: variant of octal ; dummy octal conversion, handling whitespace separators
ordinal: dummy character ordinals conversion (converts to 3-digits groups)
ordinal-spaced: variant of ordinal ; dummy character ordinals conversion, handling whitespace separators

Compression

gzip: standard Gzip compression/decompression
lz77: compresses the given data with the algorithm of Lempel and Ziv of 1977
lz78: compresses the given data with the algorithm of Lempel and Ziv of 1978
pkzip_deflate: standard Zip-deflate compression/decompression
pkzip_bzip2: standard BZip2 compression/decompression
pkzip_lzma: standard LZMA compression/decompression

Cryptography

affine: aka Affine Cipher
atbash: aka Atbash Cipher
bacon: aka Baconian Cipher
barbie-N: aka Barbie Typewriter (N belongs to [1, 4])
citrix: aka Citrix CTX1 passord encoding
rotN: aka Caesar cipher (N belongs to [1,25])
scytaleN: encrypts using the number of letters on the rod (N belongs to [1,[)
shiftN: shift ordinals (N belongs to [1,255])
xorN: XOR with a single byte (N belongs to [1,255])

Languages

braille: well-known braille language (Python 3 only)
ipsum: aka lorem ipsum
leetspeak: based on minimalistic elite speaking rules
morse: uses whitespace as a separator
navajo: only handles letters (not full words from the Navajo dictionary)
radio: aka NATO or radio phonetic alphabet
southpark: converts letters to Kenny's language from Southpark (whitespace is also handled)
southpark-icase: case insensitive variant of southpark
tomtom: similar to morse, using slashes and backslashes

Others

dna: implements the 8 rules of DNA sequences (N belongs to [1,8])
html: implements entities according to this reference
letter-indices: encodes consonants and/or vowels with their corresponding indices
markdown: unidirectional encoding from Markdown to HTML
url: aka URL encoding

Steganography

klopf: aka Klopf code ; Polybius square with trivial alphabetical distribution
resistor: aka resistor color codes
sms: also called T9 code ; uses "-" as a separator for encoding, "-" or "_" or whitespace for decoding
whitespace: replaces bits with whitespaces and tabs
whitespace_after_before: variant of whitespace ; encodes characters as new characters with whitespaces before and after according to an equation described in the codec name (e.g. "whitespace+2*after-3*before")

👏 Supporters

Comments

using Codext guess / Codext rank gives an error

When I run it on linux and try to use "codext guess" or "codext rank" I get an error message saying:

# echo "3yZe7d" | codext rank
Traceback (most recent call last):
  File "/usr/local/bin/codext", line 8, in &lt;module&gt;
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/codext/__init__.py", line 184, in main
    args.include, args.exclude = __format_list(args.include), __format_list(args.exclude, False)
AttributeError: 'Namespace' object has no attribute 'include'

opened by GitSunburn 1

UU decoding raises an exception in some cases

# cat raw.txt 
1Oh6axLwfecHErbVpRbtNj8t5JACsSQrofdnnMdQtBmoU8cQj6EyLcVRsQLz1MyWfXbqQDIc9wGyyBuH7uV95lBVpFGn3syGIw0IVLx8wJr4ABsIH9Q71hBH4AvIgljx7XnfjfmafahBNrPMDkK3dsJF0r41nzyMnOf7l36NcllAOgRLoB6Qh0APotZu6plYpkSiRCAkDKowOFm0mybKp336TAJe4JiDecD9hNbl5fBDLkGNYhmSkzOQzLBH1aPumW4o

# codext -i raw.txt decode base62
begin 666 <data>
M'XL( /H^)V("__-)SBB-<O7*+#;,K8PL\/ H\C#UR2LP= H)C0@-RZ\*"P@M
M\"T-\*@(#\]V\<A,*R^W\"@I,W9Q-,LU\8XR=$XM*TYR3PG,2G)+RO P#_'.
?"LRJS#)US0X(KPJH<$U.J@P(<TGR  #3A(K<90      
 
end

# codext -i raw.txt decode base62 | codext decode UU
Traceback (most recent call last):
  File "/usr/lib/python3.8/encodings/uu_codec.py", line 58, in uu_decode
    data = binascii.a2b_uu(s)
binascii.Error: Illegal char

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jeane/.local/bin/codext", line 8, in <module>
    sys.exit(main())
  File "/home/jeane/.local/lib/python3.8/site-packages/codext/__init__.py", line 124, in main
    c = getattr(codecs, ["encode", "decode"][args.command == "decode"])(c, encoding, args.errors)
  File "/home/jeane/.local/lib/python3.8/site-packages/codext/__common__.py", line 691, in decode
    return lookup(encoding).decode(obj, errors)[0]
  File "/usr/lib/python3.8/encodings/uu_codec.py", line 62, in uu_decode
    data = binascii.a2b_uu(s[:nbytes])
binascii.Error: Illegal char

opened by RomainJennes 1

Implement Rail Fence Cipher

Tests pass.

RegEx might need a fix : rail-5-3- is recognized as a valid codec (ending dash is too much). Only rail-5-3 or rail-5-3-up should be recognized. Couldn't manage to get that right.

When encoding, there might be trailing whitespaces (which is normal), but users might forget one when copy/pasting. I left them invisible to ease the integration with other tools.

opened by smarbal 0
Add tap/knock language

Tap encoding and decoding is implemented and added to the documentation. Couldn't make it work with add_map() due to problems with letter and word separations (single space/double space). Did my own encoding/decoding functions instead.

opened by smarbal 0
[Snyk] Security upgrade markdown2 from 2.3.10 to 2.4.0
Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

requirements.txt

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 768/1000
Why? Proof of Concept exploit, Recently disclosed, Has a fix available, CVSS 7.5 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-MARKDOWN2-1063233 | markdown2:
2.3.10 -> 2.4.0
| No | Proof of Concept

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic
opened by dhondta 0

pip 22.3 / python 3.10 warning on install

Looks like there's a warning on the installation method via pip:

pip install codext         
Collecting codext
  Downloading codext-1.14.0.tar.gz (116 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.2/116.2 kB 1.7 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from codext) (1.16.0)
Collecting markdown2>=2.4.0
  Downloading markdown2-2.4.6-py2.py3-none-any.whl (37 kB)
Installing collected packages: markdown2, codext
  DEPRECATION: codext is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for codext ... done
Successfully installed codext-1.14.0 markdown2-2.4.6

opened by mikeatlas 0