Generic ASN.1 library for Python

Overview

ASN.1 library for Python

PyPI Python Versions Build status Coverage Status GitHub license

This is a free and open source implementation of ASN.1 types and codecs as a Python package. It has been first written to support particular protocol (SNMP) but then generalized to be suitable for a wide range of protocols based on ASN.1 specification.

Features

  • Generic implementation of ASN.1 types (X.208)
  • Standards compliant BER/CER/DER codecs
  • Can operate on streams of serialized data
  • Dumps/loads ASN.1 structures from Python types
  • 100% Python, works with Python 2.7 and 3.5+
  • MT-safe
  • Contributed ASN.1 compiler Asn1ate

Why using pyasn1

ASN.1 solves the data serialisation problem. This solution was designed long ago by the wise Ancients. Back then, they did not have the luxury of wasting bits. That is why ASN.1 is designed to serialise data structures of unbounded complexity into something compact and efficient when it comes to processing the data.

That probably explains why many network protocols and file formats still rely on the 30+ years old technology. Including a number of high-profile Internet protocols and file formats.

Quite a number of books cover the topic of ASN.1. Communication between heterogeneous systems by Olivier Dubuisson is one of those high quality books freely available on the Internet.

The pyasn1 package is designed to help Python programmers tackling network protocols and file formats at the comfort of their Python prompt. The tool struggles to capture all aspects of a rather complicated ASN.1 system and to represent it on the Python terms.

How to use pyasn1

With pyasn1 you can build Python objects from ASN.1 data structures. For example, the following ASN.1 data structure:

Record ::= SEQUENCE {
  id        INTEGER,
  room  [0] INTEGER OPTIONAL,
  house [1] INTEGER DEFAULT 0
}

Could be expressed in pyasn1 like this:

class Record(Sequence):
    componentType = NamedTypes(
        NamedType('id', Integer()),
        OptionalNamedType(
            'room', Integer().subtype(
                implicitTag=Tag(tagClassContext, tagFormatSimple, 0)
            )
        ),
        DefaultedNamedType(
            'house', Integer(0).subtype(
                implicitTag=Tag(tagClassContext, tagFormatSimple, 1)
            )
        )
    )

It is in the spirit of ASN.1 to take abstract data description and turn it into a programming language specific form. Once you have your ASN.1 data structure expressed in Python, you can use it along the lines of similar Python type (e.g. ASN.1 SET is similar to Python dict, SET OF to list):

>>> record = Record()
>>> record['id'] = 123
>>> record['room'] = 321
>>> str(record)
Record:
 id=123
 room=321
>>>

Part of the power of ASN.1 comes from its serialisation features. You can serialise your data structure and send it over the network.

>>> from pyasn1.codec.der.encoder import encode
>>> substrate = encode(record)
>>> hexdump(substrate)
00000: 30 07 02 01 7B 80 02 01 41

Conversely, you can turn serialised ASN.1 content, as received from network or read from a file, into a Python object which you can introspect, modify, encode and send back.

>>> from pyasn1.codec.der.decoder import decode
>>> received_record, rest_of_substrate = decode(substrate, asn1Spec=Record())
>>>
>>> for field in received_record:
>>>    print('{} is {}'.format(field, received_record[field]))
id is 123
room is 321
house is 0
>>>
>>> record == received_record
True
>>> received_record.update(room=123)
>>> substrate = encode(received_record)
>>> hexdump(substrate)
00000: 30 06 02 01 7B 80 01 7B

The pyasn1 classes struggle to emulate their Python prototypes (e.g. int, list, dict etc.). But ASN.1 types exhibit more complicated behaviour. To make life easier for a Pythonista, they can turn their pyasn1 classes into Python built-ins:

>>> from pyasn1.codec.native.encoder import encode
>>> encode(record)
{'id': 123, 'room': 321, 'house': 0}

Or vice-versa -- you can initialize an ASN.1 structure from a tree of Python objects:

>>> from pyasn1.codec.native.decoder import decode
>>> record = decode({'id': 123, 'room': 321, 'house': 0}, asn1Spec=Record())
>>> str(record)
Record:
 id=123
 room=321
>>>

With ASN.1 design, serialisation codecs are decoupled from data objects, so you could turn every single ASN.1 object into many different serialised forms. As of this moment, pyasn1 supports BER, DER, CER and Python built-ins codecs. The extremely compact PER encoding is expected to be introduced in the upcoming pyasn1 release.

More information on pyasn1 APIs can be found in the documentation, compiled ASN.1 modules for different protocols and file formats could be found in the pyasn1-modules repo.

How to get pyasn1

The pyasn1 package is distributed under terms and conditions of 2-clause BSD license. Source code is freely available as a GitHub repo.

You could pip install pyasn1 or download it from PyPI.

If something does not work as expected, open an issue at GitHub or post your question on Stack Overflow or try browsing pyasn1 mailing list archives.

Copyright (c) 2005-2020, Ilya Etingof. All rights reserved.

Comments
  • IndexError: tuple index out of range when encoding X509 Cert

    IndexError: tuple index out of range when encoding X509 Cert

    I think this one was not reported yet :)

    I used pyasn1 to extract the X509 certificate part of an PKCS#7 Message, here is an example:

    import binascii
    from pyasn1.codec.der.decoder import decode
    from pyasn1.codec.der.encoder import encode
    
    certhex  = b"3082039506092a864886f70d010702a082038630820382020101310b300906052b0e03021a0500300b06092a864886f70d010701a08202d0308202cc30820289a003020102020466eda548300b06072a8648ce380403050030363110300e06035504061307736961766173683110300e060355040a1307736961766173683110300e06035504031307736961766173683020170d3136303132353037323533305a180f32303534303532353037323533305a30363110300e06035504061307736961766173683110300e060355040a1307736961766173683110300e0603550403130773696176617368308201b83082012c06072a8648ce3804013082011f02818100fd7f53811d75122952df4a9c2eece4e7f611b7523cef4400c31e3f80b6512669455d402251fb593d8d58fabfc5f5ba30f6cb9b556cd7813b801d346ff26660b76b9950a5a49f9fe8047b1022c24fbba9d7feb7c61bf83b57e7c6a8a6150f04fb83f6d3c51ec3023554135a169132f675f3ae2b61d72aeff22203199dd14801c70215009760508f15230bccb292b982a2eb840bf0581cf502818100f7e1a085d69b3ddecbbcab5c36b857b97994afbbfa3aea82f9574c0b3d0782675159578ebad4594fe67107108180b449167123e84c281613b7cf09328cc8a6e13c167a8b547c8d28e0a3ae1e2bb3a675916ea37f0bfa213562f1fb627a01243bcca4f1bea8519089a883dfe15ae59f06928b665e807b552564014c3bfecf492a0381850002818100e9b46a500aabcbb7b42e93477db9f8ac7c819eadd392f4fa2dd7c10973c399d00d45796052bfb376567a518cfe1432a0f845e4be75aa76462335bf7f84b6c634bb6c884b6ae2e7b6f1cc78c3be6f98b65980832c40d9da3d2b047cba98b24c877b2ace7908b0af56fd62367df355e011a0482c9b8751c373a8d131076cd01421a321301f301d0603551d0e04160414eeac32e5f08781bb6f2f392814951b328868ae4c300b06072a8648ce3804030500033000302d0215009209eb12e966aafae281620e7da9743774d7846302144bf36a9634967ca47639bbc10712571b036e13d531818e30818b020101303e30363110300e06035504061307736961766173683110300e060355040a1307736961766173683110300e0603550403130773696176617368020466eda548300906052b0e03021a0500300b06072a8648ce3804010500042e302c02145de23a86f0c2e45dcad93ef8cab675ec56a6ff4b021404c1a9f7f791631ae00d61fe8ae23f502c33fe84"
    
    pkcs7message = binascii.unhexlify(certhex)
    
    message, _ = decode(pkcs7message)
    cert = encode(message[1][3])
    print(cert)
    

    With version 0.3.3 this fails due to the following error, it worked fine with the versions before 0.3.1:

    Traceback (most recent call last):
      File "bla.py", line 12, in <module>
        cert = encode(message[1][3])
      File "C:\Program Files\Anaconda3\lib\site-packages\pyasn1\codec\der\encoder.py", line 56, in __call__
        return encoder.Encoder.__call__(self, value, defMode, maxChunkSize, ifNotEmpty=ifNotEmpty)
      File "C:\Program Files\Anaconda3\lib\site-packages\pyasn1\codec\cer\encoder.py", line 208, in __call__
        return encoder.Encoder.__call__(self, value, defMode, maxChunkSize, ifNotEmpty)
      File "C:\Program Files\Anaconda3\lib\site-packages\pyasn1\codec\ber\encoder.py", line 483, in __call__
        self, value, defMode, maxChunkSize, ifNotEmpty=ifNotEmpty
      File "C:\Program Files\Anaconda3\lib\site-packages\pyasn1\codec\ber\encoder.py", line 61, in encode
        encodeFun, value, defMode, maxChunkSize, ifNotEmpty=ifNotEmpty
      File "C:\Program Files\Anaconda3\lib\site-packages\pyasn1\codec\cer\encoder.py", line 156, in encodeValue
        substrate = encodeFun(value[idx], defMode, maxChunkSize, namedTypes[idx].isOptional) + substrate
      File "C:\Program Files\Anaconda3\lib\site-packages\pyasn1\type\namedtype.py", line 155, in __getitem__
        return self.__namedTypes[idx]
    IndexError: tuple index out of range
    

    Maybe, it has to do how I use it, maybe it is a real bug.

    By the way, this is how I use it: https://github.com/androguard/androguard/blob/master/androguard/core/bytecodes/apk.py#L836 I also need to remove the first identifier in the DER Coded sequence, which does not look right, but I did not found another way. (If you know a better way to extract the cert from the message please tell me :))

    bug 
    opened by reox 17
  • Streamed version of ber.decoder.decode

    Streamed version of ber.decoder.decode

    (An updated version of #175):

    I implemented the functionality so that instead of copying strings, it passes a ~~BytesIO object~~ stream whenever possible. It should solve the #174 issue.

    Changes in API:

    • decodeStream in the modules accepts a stream directly
    • decode behaves as it did before, constructing a new BytesIO object and passing it to decodeStream

    Thus, no API should break unless the author decides so (which I suggest as another issue).

    I tested heavily with Python 2.7 and 3.7, CI should care of the rest.

    • test for open() results added. If any other kind of stream is supposed to be used, it can (and should be added).

    Performance:

    • for the tiniest objects (like "false"), the new decode is ~5 % slower.
    • for example objects of 10-50 bytes, it's ~0-2 % faster.
    • for a 2MB file containing many objects, it's ~5 times faster.

    Note: As file object in Python 2.7 does not allow direct setting of attributes, it's first read into memory in this version. A trivial patch, that slows the overall performance by 2-3 %, but avoids pre-reading the file into memory, can be provided.

    What was considered as well (but did not work / was not performing enough):

    • using memoryview (this could potentially work with bytes if the code were refactored a lot)
    • wrapping input in a custom stream-like object that keeps the content and position - the cost of creating objects and accessing attributes was too high and still all streams were not covered.

    Please, let me know what changes are there still to be made. Thanks Jan

    opened by janpipek 15
  • ImportError: cannot import name 'opentype'

    ImportError: cannot import name 'opentype'

    Since the last update of 4 hours ago my CI broke because a module is using your pyasn1 module. I received the following in my stack trace:

      File "/usr/local/lib/python3.5/dist-packages/oauth2client/_pure_python_crypt.py", line 24, in <module>
        from pyasn1_modules.rfc2459 import Certificate
      File "/usr/local/lib/python3.5/dist-packages/pyasn1_modules/rfc2459.py", line 20, in <module>
        from pyasn1.type import opentype
    ImportError: cannot import name 'opentype'
    

    Is this anything related to the last changes made in this module and a way this is fixable on my side or with a new release of this module?

    question 
    opened by DanielMartinus 14
  • Bugfix/emv

    Bugfix/emv

    This is a small pile of changes required to make EMV tags be capable of being uniquely identified and parsed more correctly. As far as I can tell, these don't interfere with existing tag encoding/decoding and probably should be applied to DER and CER if applicable.

    opened by mmattice 13
  • Add method equivalent of x[component]=None

    Add method equivalent of x[component]=None

    Add a new method which allows setting a default value to a component. The following lines are equivalent:

    x.setComponentToDefault(comp) x[comp] = None

    but the first one is more explicit and raises fewer eyebrows during code review.

    opened by viraptor 12
  • PyAsn1Error when decoding explicitly tagged subtypes

    PyAsn1Error when decoding explicitly tagged subtypes

    I've got a little wrapper library at https://github.com/jfinkhaeuser/bran that goes a little beyond the ASN.1 specs to providing a canonical presentation.

    Since upgrading to pyasn1 0.3.x my builds fail, and I'm having a hard time figuring out what if anything I'm doing wrong.

    Summarized, I do the following:

    • Encode more complex types as sequences:
        self.MAPPING = univ.Sequence(
          tagSet = univ.Sequence.tagSet.tagExplicitly(
            tag.Tag(tag.tagClassContext, tag.tagFormatConstructed, 0x04)
          )
        )
    
    • In the mapping case, it's actually a sequence of two item sequences, representing the key and value.
    • When decoding, there's now a PyAsn1Error when the sequence end is reached. Previously iteration would just stop.

    I can circumvent that by catching the error, but that might hide other issues. I can explicitly add the length of the sequence, but my understanding of ASN.1 is that this shouldn't be necessary.

    The error is "Compontent type not defined".

    Do you have any pointers for me?

    bug 
    opened by jfinkhaeuser 11
  • Drop support for EOL Pythons

    Drop support for EOL Pythons

    Python 2.4, 2.5, 2.6, 3.2 and 3.3 are end of life. They are no longer receiving bug fixes, including for security issues. These Pythons went EOL on:

    Python 2.6: 2013-10-29 Python 3.2: 2016-02-20 Python 3.3: 2017-09-29

    For additional details on supported Python versions, see:

    https://devguide.python.org/#status-of-python-branches

    Removing support for EOL Pythons will reduce testing and maintenance resources.

    Using pypinfo, here are the download statistics of the project for the last 30 days, showing limited downloads from EOL Pythons:

    | python_version | percent | download_count | | -------------- | ------: | -------------: | | 2.7 | 90.33% | 7,975,885 | | 3.6 | 4.95% | 437,261 | | 3.5 | 2.99% | 263,637 | | 3.4 | 1.02% | 89,723 | | 2.6 | 0.69% | 60,511 | | 3.7 | 0.02% | 2,024 | | 3.3 | 0.00% | 441 | | 3.2 | 0.00% | 38 | | 3.8 | 0.00% | 9 | | None | 0.00% | 2 |

    By removing support for older Pythons, can remove or update the following workarounds and code:

    • Built-in function bin() is always available.
    • Built-in function callable() is always available.
    • Built-in function all() is always available.
    • The collections.OrderedDict class is always available.
    • The datetime.strptime() method is always available.
    • The platform module is always available.
    • The string.partition() method is always available.
    • The NullHandler class is always available.
    • The method __trunc__ is always available.
    • Can use set literals.
    • Updated unittest is always available. The unittest2 module is unnecessary.

    Users on older versions of Python can continue to download, install, and use older versions of the library that continue to have support for older Pythons.

    opened by jdufresne 9
  • AttributeError: 'property' object has no attribute 'presentTypes'

    AttributeError: 'property' object has no attribute 'presentTypes'

    I'm not sure about this I don't know much of python myself, coming from another script (freeipa) (ipa-server-install)

        from pyasn1_modules import rfc2315, rfc2459
      File "/usr/lib/python2.7/site-packages/pyasn1_modules/rfc2315.py", line 67, in <module>
        class DigestedData(univ.Sequence):
      File "/usr/lib/python2.7/site-packages/pyasn1_modules/rfc2315.py", line 72, in DigestedData
        namedtype.NamedType('digest', Digest)
      File "/root/.local/lib/python2.7/site-packages/pyasn1/type/namedtype.py", line 159, in __init__
        self.__ambiguousTypes = 'terminal' not in kwargs and self.__computeAmbiguousTypes() or {}
      File "/root/.local/lib/python2.7/site-packages/pyasn1/type/namedtype.py", line 278, in __computeAmbiguousTypes
        ambigiousTypes[idx] = NamedTypes(*partialAmbigiousTypes, **dict(terminal=True))
      File "/root/.local/lib/python2.7/site-packages/pyasn1/type/namedtype.py", line 158, in __init__
        self.__tagToPosMap = self.__computeTagToPosMap()
      File "/root/.local/lib/python2.7/site-packages/pyasn1/type/namedtype.py", line 251, in __computeTagToPosMap
        for _tagSet in tagMap.presentTypes:
    AttributeError: 'property' object has no attribute 'presentTypes'```
    opened by davidsielert 9
  • Default encoding of OctetString

    Default encoding of OctetString

    Hi!

    The default defined encoding of a univ.OctetString seems to be ISO-8859-1 (see the code here). I was wondering why this choice was made, instead of something more universal, like UTF8?

    This default encoding is causing some weird behavior in some programs relying on this library (see here or here).

    Is it possible to change the default encoding to UTF8? If not, why not, and what do you suggest we can do to ensure proper encoding in our programs?

    Thanks!

    Y

    opened by the-useless-one 7
  • With 0.3.1 all ldap3 tests fail

    With 0.3.1 all ldap3 tests fail

    Hi Ilya, with version 0.3.1 I get the following error on every test:

    ...
      File "C:\Python\OpenSource\ldap3\ldap3\operation\bind.py", line 124, in bind_response_to_dict
        'saslCreds': bytes(response['serverSaslCreds']) if response['serverSaslCreds'] is not None else None}
      File "C:\Python\Python36\lib\site-packages\pyasn1\type\univ.py", line 965, in __bytes__
        return bytes(self._value)
    TypeError: cannot convert 'NoValue' object to bytes
      File "C:\Python\Python36\lib\site-packages\pyasn1\type\univ.py", line 965, in __bytes__
        return bytes(self._value)
    TypeError: cannot convert 'NoValue' object to bytes
    
    

    All tests were ok with version 0.2.3.

    opened by cannatag 7
  • Wrong constraint verification

    Wrong constraint verification

    Getting error:

    pyasn1.type.error.ValueConstraintError: <ValueSizeConstraint object at 0x7ff642e28e48 consts 128, 128> failed at: ValueConstraintError(b'51D389643EF12BB201912511824') at IMSCHARGE
    

    But "51D389643EF12BB201912511824" is 28 byte long, not 128+

    opened by kaxap 6
  • Sad news and transfer of ownership - pyasn1 and pyasn1-modules

    Sad news and transfer of ownership - pyasn1 and pyasn1-modules

    Unfortunately, this repo won't be maintained anymore.

    With great sorrow, we have to announce that @etingof has passed away. Please, find more information here: https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030062.html

    The pyasn1 and pyasn1-modules ownership was transferred to https://github.com/pyasn1/pyasn1 Please, report your issues and file pull requests there.

    Thank you! And take care.

    opened by droideck 0
  • parsing LDAP search message openldap

    parsing LDAP search message openldap

    I've tried to parse an LDAP SearchQuery Message from ldapsearch from openldap. It fails with specific search queries when given an ASN.1 spec for LDAP messages, but also fails when trying to decode the ldap message without any ASN.1 specification. As this is used in an ldap proxy that should just pass-through LDAP search messages, I am mostly interested in pyasn1 not failing at special messages.

    The message it fails at is given in hex here: 30560201026351041064633d6578616d6c652c64633d6f72670a01020a0100020100020100010100a02c870b6f626a656374436c617373a31d040e73616d4163636f756e744e616d65040b6578616d706c65557365723000

    The message can be parsed by asn1js here: https://lapo.it/asn1js/#MFYCAQJjUQQQZGM9ZXhhbWxlLGRjPW9yZwoBAgoBAAIBAAIBAAEBAKAshwtvYmplY3RDbGFzc6MdBA5zYW1BY2NvdW50TmFtZQQLZXhhbXBsZVVzZXIwAA

    Testcode: from binascii import unhexlify from pyasn1.codec.ber.decoder import decode as ber_decoder from pyasn1 import debug

    debug.setLogger(debug.Debug('all'))

    data = unhexlify('30560201026351041064633d6578616d6c652c64633d6f72670a01020a0100020100020100010100a02c870b6f626a656374436c617373a31d040e73616d4163636f756e744e616d65040b6578616d706c65557365723000')

    obj, rest_of_input = ber_decoder(data)

    print(data) print(obj) print(rest_of_input)

    GIT Master shows: 2022-12-12 16:06:12,821 pyasn1: running pyasn1 0.4.9, debug flags all 2022-12-12 16:06:12,821 pyasn1: debug category 'all' enabled 2022-12-12 16:06:12,821 pyasn1: decoder called at scope with state 0, working with up to None octets of substrate: <_io.BytesIO object at 0x7fa36eff9ee0> 2022-12-12 16:06:12,821 pyasn1: tag decoded into <TagSet object, tags 0:32:16>, decoding length 2022-12-12 16:06:12,821 pyasn1: value length decoded into 86 2022-12-12 16:06:12,821 pyasn1: codec SequenceOrSequenceOfPayloadDecoder chosen by a built-in type, decoding value 2022-12-12 16:06:12,821 pyasn1: state is stDecodeValue 2022-12-12 16:06:12,821 pyasn1: decoder called at scope NoneType with state 0, working with up to None octets of substrate: <_io.BytesIO object at 0x7fa36eff9ee0> 2022-12-12 16:06:12,821 pyasn1: tag decoded into <TagSet object, tags 0:0:2>, decoding length 2022-12-12 16:06:12,821 pyasn1: value length decoded into 1 2022-12-12 16:06:12,821 pyasn1: codec IntegerPayloadDecoder chosen by a built-in type, decoding value 2022-12-12 16:06:12,821 pyasn1: state is stDecodeValue 2022-12-12 16:06:12,821 pyasn1: codec IntegerPayloadDecoder yields type Integer, value: 2 ... 2022-12-12 16:06:12,821 pyasn1: decoder left scope NoneType, call completed 2022-12-12 16:06:12,821 pyasn1: decoder called at scope NoneType with state 0, working with up to None octets of substrate: <_io.BytesIO object at 0x7fa36eff9ee0> 2022-12-12 16:06:12,821 pyasn1: tag decoded into <TagSet object, tags 64:32:3>, decoding length 2022-12-12 16:06:12,821 pyasn1: value length decoded into 81 2022-12-12 16:06:12,821 pyasn1: codec chosen by a built-in type, decoding as explicit tag 2022-12-12 16:06:12,821 pyasn1: A: codec RawPayloadDecoder chosen, decoding value 2022-12-12 16:06:12,821 pyasn1: state is stDecodeValue 2022-12-12 16:06:12,821 pyasn1: valueDecoder 2022-12-12 16:06:12,821 pyasn1: valueDecoder B decodeFun=<pyasn1.codec.ber.decoder.SingleItemDecoder object at 0x7fa36efdbd90> len=81 2022-12-12 16:06:12,821 pyasn1: decoder called at scope NoneType.? with state 0, working with up to 81 octets of substrate: <_io.BytesIO object at 0x7fa36eff9ee0> 2022-12-12 16:06:12,821 pyasn1: tag decoded into <TagSet object, tags 0:0:4-64:32:3>, decoding length 2022-12-12 16:06:12,821 pyasn1: value length decoded into 16 2022-12-12 16:06:12,822 pyasn1: codec OctetStringPayloadDecoder chosen by a built-in type, decoding value 2022-12-12 16:06:12,822 pyasn1: state is stDecodeValue 2022-12-12 16:06:12,822 pyasn1: codec OctetStringPayloadDecoder yields type OctetString, value: dc=examle,dc=org ... 2022-12-12 16:06:12,822 pyasn1: decoder left scope NoneType.?, call completed Traceback (most recent call last): File "/home/ron/test-pyasn1/test.py", line 10, in obj, rest_of_input = ber_decoder(data) File "/home/ron/test-pyasn1/pyasn1/codec/ber/decoder.py", line 2013, in call for asn1Object in streamingDecoder: File "/home/ron/test-pyasn1/pyasn1/codec/ber/decoder.py", line 1928, in iter for asn1Object in self._singleItemDecoder( File "/home/ron/test-pyasn1/pyasn1/codec/ber/decoder.py", line 1788, in call for value in concreteDecoder.valueDecoder( File "/home/ron/test-pyasn1/pyasn1/codec/ber/decoder.py", line 673, in valueDecoder for asn1Object in self._decodeComponentsSchemaless( File "/home/ron/test-pyasn1/pyasn1/codec/ber/decoder.py", line 609, in _decodeComponentsSchemaless for component in decodeFun(substrate, **options): File "/home/ron/test-pyasn1/pyasn1/codec/ber/decoder.py", line 1797, in call raise PyAsn1Error( pyasn1.error.PyAsn1Error: Read 18 bytes instead of expected 81.

    Making RawPayloadDecoder.valueDecoder loop like RawPayloadDecoder.indefLenValueDecoder changes the error messages as it fails at decoding the query message part.

    opened by michael-dev 1
  • Fix. Decode Unsigned Values as Unsigned

    Fix. Decode Unsigned Values as Unsigned

    Fix some issues in pysnmp

    https://github.com/etingof/pysnmp/issues/190 https://github.com/etingof/pysnmp/issues/357 https://github.com/etingof/pysnmp/issues/384

    Example for fast test:

    from pyasn1.codec.ber import decoder
    from pysnmp.hlapi import *
    
    print(decoder.decode(bytes.fromhex('4104985D4B44'), Counter32()))
    

    On the current master, it will fall into a ValueConstraintError:

    pyasn1.type.error.ValueConstraintError: <ConstraintsIntersection object, consts <ValueRangeConstraint object, consts 0, 4294967295>> failed at: ValueConstraintError('<ValueRangeConstraint object, consts 0, 4294967295> failed at: ValueConstraintError(-1738716348)') at Counter32
    

    And after correction it is correctly decoded:

    (<Counter32 value object, tagSet <TagSet object, tags 64:0:1>, subtypeSpec <ConstraintsIntersection object, consts <ValueRangeConstraint object, consts 0, 4294967295>>, payload [2556250948]>, b'')
    
    opened by nigredon1991 0
  • [PSA] Transfer of ownership

    [PSA] Transfer of ownership

    Public Service Announcement

    Ilya Etingof is currently unable to maintain pyasn1 and pyasn1-modules. PyPI package ownership for pyasn1 and pyasn1-module has been transfered to me and Simon Pichugin in PyPI support ticket #2090. The upstream repositories for pyasn1 and pyasn1-modules are now in the GitHub organization https://github.com/pyasn1/.

    Please join us at https://github.com/pyasn1 if you like to help. We are looking for more contributors and maintainers to spread the workload.

    I'm also looking for users to test pyasn1-0.5.0rc1 and pyasn1_modules-0.3.0rc1. You can install the pre-release from PyPI:

    pip install --pre pyasn1 pyasn1-modules
    
    opened by tiran 1
  • Unexpected error with tagging

    Unexpected error with tagging

    I'm having issue decoding a certain file. I've tested with

    I've seen this was an issue with earlier versions as well. Could this be a problem with the certain file I'm trying to decode or is there another bug?

    image

    opened by AntonyPapadakis 0
Releases(v0.4.8)
  • v0.4.8(Nov 16, 2019)

    This is a minor feature release:

    • Added ability to combine SingleValueConstraint and PermittedAlphabetConstraint objects into one for proper modeling FROM ... EXCEPT ... ASN.1 clause.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.7(Sep 1, 2019)

    The major feature of this release is SET ... WITH COMPONENTS constraint support.

    • Added WithComponentsConstraint along with related ComponentPresentConstraint and ComponentAbsentConstraint classes to be used with Sequence/Set types representing SET ... WITH COMPONENTS ... like ASN.1 constructs.

    The other important change is that sizeSpec attribute of ASN.1 objects is now deprecated in favor of uniform subtypeSpec, which is now used for all constraints.

    • Deprecate subtypeSpec attributes and keyword argument. It is now recommended to pass ValueSizeConstraint, as well as all other constraints, to subtypeSpec.
    • Added isInconsistent property to all constructed types. This property conceptually replaces verifySizeSpec method to serve a more general purpose e.g. ensuring all required fields are in a good shape. By default this check invokes subtype constraints verification and is run by codecs on value de/serialisation.

    In the bug fixes department we have just one fix:

    • Fixed a design bug in a way of how the items assigned to constructed types are verified. Now if Asn1Type-based object is assigned, its compatibility is verified based on having all tags and constraint objects as the type in field definition. When a bare Python value is assigned, then field type object is cloned and initialized with the bare value (constraints verificaton would run at this moment).

    Complete list of changes can be found in CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.6(Jul 31, 2019)

    This release brings support for one overlooked ASN.1 feature:

    • Added previously missing SET OF ANY construct encoding/decoding support.

    As a consequence, SequenceOf/SetOf objects behavior has been made closer to Python list and more consistent with the rest of pyasn1 types (at the expense of potential backward compatibility issues):

    • New elements to SequenceOf/SetOf objects can now be added at any position - the requirement for the new elements to reside at the end of the existing ones (i.e. s[len(s)] = 123) is removed.
    • List-like slicing support added to SequenceOf/SetOf objects.
    • Removed default initializer from SequenceOf/SetOf types to ensure consistent behaviour with the rest of ASN.1 types. Before this change, SequenceOf/SetOf instances immediately become value objects behaving like an empty list. With this change, SequenceOf/SetOf objects remain schema objects unless a component is added or .clear() is called. This change can potentially cause incompatibilities with existing pyasn1 objects which assume SequenceOf/SetOf instances are value objects right upon instantiation. The behaviour of Sequence/Set types depends on the componentType initializer: if no componentType is given, the behavior is the same as SequenceOf/SetOf have. If componentType is given, but neither optional nor defaulted components are present, created instance remains being the schema object. If, however, either optional or defaulted component is present, created instance immediately becomes a value object.
    • Added .reset() method to all constructed types to turn value object into a schema object.

    Also, a couple of minor usability improvements:

    • Added omitEmptyOptionals option which is respected by Sequence and Set encoders. When omitEmptyOptionals is set to True, empty initialized optional components are not encoded. Default is False.
    • Added PyAsn1UnicodeDecodeError/PyAsn1UnicodeDecodeError exceptions to help the caller treating unicode errors happening internally to pyasn1 at the upper layers.
    • Added support for subseconds CER/DER encoding edge cases in GeneralizedTime codec.

    And, of course, bug fixes \o/

    • Fixed 3-digit fractional seconds value CER/DER encoding of GeneralizedTime.
    • Fixed AnyDecoder to accept possible TagMap as asn1Spec to make dumping raw value operational

    Complete list of changes can be found in CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.5(Dec 29, 2018)

  • v0.4.4(Jul 26, 2018)

    This is a maintenance release fixing a pair of annoying bugs:

    • Fixed native encoder type map to include all ASN.1 types rather than just ambiguous ones
    • Fixed crash in .prettyPrint of Sequence and Set occurring at OPTIONAL components

    Changes are noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.3(May 23, 2018)

  • v0.4.2(Nov 23, 2017)

    This is an emergency release fixing a regression.

    It fixes a single bug in OctetString/BitString encoders which blows up when you BER-encode ASN.1 structure produced by BER decoder.

    Changes are noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.1(Nov 23, 2017)

    This release introduces ASN.1 open type support and brings a handful of other fixes and improvements. Most important changes are:

    • ANY DEFINED BY clause support implemented
    • Encoders refactored to take either a value (as ASN.1 object) or a Python value plus ASN.1 schema
    • The ASN.1 types' __repr__() and __str__() implementations reworked for better usability
    • Improved backward-compatibility with older pyasn1 versions
    • Sphinx documentation rearranged, simplified and reworded

    All changes are noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.7(Oct 4, 2017)

  • v0.3.6(Sep 20, 2017)

    This is mostly a bug fix release. Most important changes are:

    • The __getitem__/__setitem__ behavior of Set/Sequence and SetOf/SequenceOf objects aligned with the canonical Mapping and Sequence protocols in part
    • Fixed crash in ASN.1 encoder when encoding an explicitly tagged component of a Set/Sequence object

    More changes noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.5(Sep 16, 2017)

    This is mostly a bug fix release. Most important fixes include:

    • Explicit tag encoding optimized to avoid unnecessary copying
    • End-of-octets sentinel encoding optimized
    • Refactored ASN.1 codecs properties to silently enforce proper length and chunk size encoding modes
    • Fixed DER encoder to always produce primitive encoding
    • Fixed crash at SequenceOf native decoder
    • Fixed a couple of crashes when debug mode is enabled

    More changes noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.4(Sep 7, 2017)

    This is mostly a bug fix release. Most important fixes include:

    • Added missing component-less SEQUENCE/SET objects dict duck-typing support
    • Fixed crash at SEQUENCE and SEQUENCE OF CER encoder when running in schemaless mode
    • Fixed Character types instantiation from OctetString type -- double unicode decoding may have scrambled the data

    More changes noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.3(Aug 26, 2017)

    This is a bugfix release addressing a handful of regressions introduced in 0.3.2.

    Most importantly:

    • Fixed exponential index size growth bug when building ambiguous NamedTypes tree
    • Fixed constructed types decoding failure at BER codec if running in unguided mode
    • Fixed SetOf ordering at CER/DER encoder

    More changes noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.2(Aug 4, 2017)

    This is a bugfix release addressing a handful of regressions introduced in 0.3.1.

    Most importantly:

    • Fixed SequenceOf/SetOf types initialization syntax to remain backward compatible with pyasn1 0.2.*
    • Rectified thread safety issues by moving lazy, run-time computation into object initializer.
    • Fixed GeneralizedTime/UTCTime CER/DER codecs to actually get invoked
    • Fixed DER/CER encoders handling optional SEQUENCE/SET fields containing nested SEQUENCE/SET with optional fields.

    More changes noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.1(Jul 26, 2017)

    The main theme for this release is API and performance improvements.

    Be aware that by way of usability improvements, a some backward compatibilities were introduced. For the details please refer to the CHANGELOG.

    The highlights:

    • ASN.1 types __init__(), .clone() and .subtype() signatures refactored to expect keyword arguments
    • Many "getters" of ASN.1 types refactored into Python properties
    • SetOf/SequenceOf ASN.1 types better mimic Python list
    • The NamedValues implementation refactored to mimic Python dict
    • The GeneralizedTime and UTCTime types now support to/from Python datetime object conversion
    • Many minor performance improvements and bug fixes

    More changes noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • 0.2.3(Feb 25, 2017)

    This is a performance improvement release:

    • BitString type implementation rebased onto Python int to behave like an integer and improve serialization performance (100x times!).
    • Integer codecs reimplemented to base on Python's built-in integer serialization codecs to improve serialization performance (~40%).
    • Decoding performance of Set and Choice types improved by caching once computed tags maps.
    • ASN.1 character types refactored to duck-type unicode strings thus making them more convenient in text processing context.
    • Documentation updated at many places.

    More changes noted in the CHANGELOG.

    Source code(tar.gz)
    Source code(zip)
  • v0.2.2(Feb 7, 2017)

    This release fixes a regression in Enumerated ASN.1 type subtyping procedure.

    Besides that, canonical decoders were hardened to reject non-canonical serialization on input.

    More details can be learned from the change log

    Source code(tar.gz)
    Source code(zip)
  • v0.2.1(Feb 4, 2017)

    This is a bug fix and usability improvement release:

    • Conversion between Python built-in types and pyasn1 objects implemented
    • Python types emulation improved to make pyasn1 types look more like their Python counterparts
    • BitString type usability improved in many ways
    • Sphinx documentation added

    More changes noted in the CHANGELOG.

    The upcoming release is to be focused on codecs performance improvement.

    Source code(tar.gz)
    Source code(zip)
Owner
Ilya Etingof
Sr. Software Engineer at Red Hat; OpenStack hardware provisioning team
Ilya Etingof
A lightweight library for converting complex objects to and from simple Python datatypes.

marshmallow: simplified object serialization marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, t

marshmallow-code 6.4k Jan 2, 2023
Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy

orjson orjson is a fast, correct JSON library for Python. It benchmarks as the fastest Python library for JSON and is more correct than the standard j

null 4.1k Dec 30, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 19.6k Jan 1, 2023
MessagePack serializer implementation for Python msgpack.org[Python]

MessagePack for Python What's this MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JS

MessagePack 1.7k Dec 29, 2022
Ultra fast JSON decoder and encoder written in C with Python bindings

UltraJSON UltraJSON is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 3.6+. Install with pip: $ python -m pip insta

null 3.9k Jan 2, 2023
simplejson is a simple, fast, extensible JSON encoder/decoder for Python

simplejson simplejson is a simple, fast, complete, correct and extensible JSON <http://json.org> encoder and decoder for Python 3.3+ with legacy suppo

null 1.5k Dec 31, 2022
Extended pickling support for Python objects

cloudpickle cloudpickle makes it possible to serialize Python constructs not supported by the default pickle module from the Python standard library.

null 1.3k Jan 5, 2023
serialize all of python

dill serialize all of python About Dill dill extends python's pickle module for serializing and de-serializing python objects to the majority of the b

The UQ Foundation 1.8k Jan 7, 2023
🦉 Modern high-performance serialization utilities for Python (JSON, MessagePack, Pickle)

srsly: Modern high-performance serialization utilities for Python This package bundles some of the best Python serialization libraries into one standa

Explosion 329 Dec 28, 2022
Python wrapper around rapidjson

python-rapidjson Python wrapper around RapidJSON Authors: Ken Robbins <[email protected]> Lele Gaifax <[email protected]> License: MIT License Sta

null 469 Jan 4, 2023
Python bindings for the simdjson project.

pysimdjson Python bindings for the simdjson project, a SIMD-accelerated JSON parser. If SIMD instructions are unavailable a fallback parser is used, m

Tyler Kennedy 562 Jan 8, 2023
Python ASN.1 library with a focus on performance and a pythonic API

asn1crypto A fast, pure Python library for parsing and serializing ASN.1 structures. Features Why Another Python ASN.1 Library? Related Crypto Librari

Will Bond 282 Dec 11, 2022
a generic C++ library for image analysis

VIGRA Computer Vision Library Copyright 1998-2013 by Ullrich Koethe This file is part of the VIGRA computer vision library. You may use,

Ullrich Koethe 378 Dec 30, 2022
Pygments is a generic syntax highlighter written in Python

Welcome to Pygments This is the source of Pygments. It is a generic syntax highlighter written in Python that supports over 500 languages and text for

null 1.2k Jan 6, 2023
Generic template for python service

Cookie cutter template example Technology stack Flask Gevent UWSGI Poetry Docker Docker-compose Installation pip install cookiecutter cookiecutter git

Churkin Oleg 11 Oct 22, 2022
PyGMT - A Python interface for the Generic Mapping Tools

PyGMT A Python interface for the Generic Mapping Tools Documentation (development version) | Contact | Try Online Why PyGMT? A beautiful map is worth

The Generic Mapping Tools (GMT) 564 Dec 28, 2022
Generic python project template

generic-python-project-template generic-python-project-template STEPS - STEP 01- Create a repository by using template repository STEP 02- Clone the n

SUNNY BHAVEEN CHANDRA 3 Oct 3, 2022
A Python implementation of GRAIL, a generic framework to learn compact time series representations.

GRAIL A Python implementation of GRAIL, a generic framework to learn compact time series representations. Requirements Python 3.6+ numpy scipy tslearn

null 3 Nov 24, 2021
The Generic Manipulation Driver Package - Implements a ROS Interface over the robotics toolbox for Python

Armer Driver Armer aims to provide an interface layer between the hardware drivers of a robotic arm giving the user control in several ways: Joint vel

QUT Centre for Robotics (QCR) 13 Nov 26, 2022
Python scripts for a generic performance testing infrastructure using Locust.

TODOs Reference to published paper or online version of it loadtest_plotter.py: Cleanup and reading data from files ARS_simulation.py: Cleanup, docume

Juri Tomak 3 Dec 15, 2022