Protocol Buffers for the Rest of Us

Related tags

Miscellaneous python protobuf
Overview

Protocol Buffers for the Rest of Us

Motivation

protoletariat has one goal: fixing the broken imports for the Python code generated by protoc.

Usage

Here's an example of how to use the tool, called protol:

  1. Create a few protobuf files
// thing1.proto
syntax = "proto3";

import "thing2.proto";

package things;

message Thing1 {
  Thing2 thing2 = 1;
}
// thing2.proto
syntax = "proto3";

package things;

message Thing2 {
  string data = 1;
}
  1. Run protoc on those files
$ mkdir out
$ protoc --python_out=out --proto_path=directory/containing/protos thing1.proto thing2.proto
  1. Run protol on the generated code
$ protol --create-init --overwrite -g out --proto-path=directory/containing/protos thing1.proto thing2.proto

The out/thing1_pb2.py file should show a diff containing at least these lines:

-import thing2_pb2 as thing2__pb2
-
+from . import thing2_pb2 as thing2__pb2

How it works

At a high level, protoletariat converts absolute imports to relative imports.

However, it doesn't just convert any absolute import to a relative import.

The protol tool will only convert imports that were generated from .proto files. It does this by inspecting FileDescriptorProtos from the protobuf files.

The core mechanism is implemented using a simplified form of pattern matching, that looks at the Python AST, and if an import pattern is matched and corresponding rewrite rule is fired.

Help

$ protol --help
Usage: protol [OPTIONS] PROTO_FILES...

  Rewrite protoc-generated imports for use by the proletariat.

Options:
  -g, --generated-python-dir DIRECTORY
                                  Directory containing generated Python code
                                  [required]
  -p, --proto-path DIRECTORY      Protobuf file search path(s). Accepts
                                  multiple values.  [required]
  --overwrite / --no-overwrite    Overwrite all generated Python files with
                                  modified imports
  --create-init / --dont-create-init
                                  Create an __init__.py file under the
                                  `generated-python-dir` directory
  --help                          Show this message and exit.
Comments
  • Usage on Windows

    Usage on Windows

    I currently am not able to make it work on Windows. However it works in the WSL shell with the same syntax. I keep getting a FileNotFoundError: [WinError 2] on Windows. Am I doing something wrong with the syntax?

    This one is not working:

    protol --create-package --in-place --python-out=cosmospy_protobuf protoc --proto-path=C:\Users\felix\Documents\dev\cosmospy-protobuf\cosmospy_protobuf  C:\Users\felix\Documents\dev\cosmospy-protobuf\cosmospy_protobuf\gogoproto\gogo.proto

    This one is working:

    protol --create-package --in-place --python-out=cosmospy_protobuf protoc --proto-path=/mnt/c/Users/felix/Documents/dev/cosmospy-protobuf/cosmospy_protobuf  /mnt/c/Users/felix/Documents/dev/cosmospy-protobuf/cosmospy_protobuf/gogoproto/gogo.proto

    Same behaviour when using subprocess.run() in a .py file:

    absolute_path = os.path.abspath('cosmospy_protobuf')

#Genrated in a loop but this is how the path is generated
os.path.abspath(os.path.join(root, filename)

cmd = [sys.executable, '-m', 'protoletariat',
           '--create-package', '--in-place',
           '--python-out', 'cosmospy_protobuf',
           'protoc', f'--proto-path={absolute_path}',
           filepath]
subprocess.run(cmd)

    This works on Linux but not on windows

    opened by ctrl-Felix 15
  • fix: ensure every path is decoded using os.fsdecode

    fix: ensure every path is decoded using os.fsdecode

    I'm using Click 7.1.2. For some reason, the input file paths were bytes instead of str, causing lots of errors when I tried to run protol.

    This PR fixed the problem for me. It's probably a safe thing to merge, and it guards against other strange Click incompatibilities.

    In case it matters, here's how I was invoking protol:

    https://github.com/shawwn/tensorflow-checkpoint-reader/blob/c8a77dc82c88f52971f11a96e59d5ace7b595611/build.sh#L15-L16

    bug 
    opened by shawwn 11
  • Generated .pyi files include __pycache__, are not idempotent

    Generated .pyi files include __pycache__, are not idempotent

    I've noticed a few things about generated .pyi files:

    • __pycache__ gets picked up and imported,
    • Entries are not sorted (though this isn't really a big deal, it kinda helps with reviewing the diff)
    • Entries get duplicated if run multiple times/generation is not idempotent

    The last one is probably harder to solve since it appears Protoletariat is not assuming it has full control of the pyi (understandable), or else it could overwrite instead of append.

    opened by lidavidm 4
  • chore(deps): update dependency flake8 to v6

    chore(deps): update dependency flake8 to v6

    Mend Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | flake8 (changelog) | >=5.0.0,<6 -> >=6,<7 | age | adoption | passing | confidence |

    Release Notes

    pycqa/flake8

    v6.0.0

    Compare Source

    Configuration

    ๐Ÿ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Enabled.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    dependencies 
    opened by renovate[bot] 3
  • bug: including the google.protobuf types breaks the code

    bug: including the google.protobuf types breaks the code

    Hello, Whenever I add gRPC well-known types like Empty, the protoletariat breaks the python code changing:

    from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2

    to

    from .google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
    bug 
    opened by bartek1912 3
  • chore(deps): update dependency flake8 to v6

    chore(deps): update dependency flake8 to v6

    Mend Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | flake8 (changelog) | >=5.0.0,<6 -> >=6,<7 | age | adoption | passing | confidence |

    Release Notes

    pycqa/flake8

    v6.0.0

    Compare Source

    Configuration

    ๐Ÿ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Enabled.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    dependencies 
    opened by renovate[bot] 2
  • chore(deps): update dependency mypy to ^0.930

    chore(deps): update dependency mypy to ^0.930

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | mypy (source, changelog) | ^0.920 -> ^0.930 | age | adoption | passing | confidence |

    Release Notes

    python/mypy

    v0.930

    Compare Source

    v0.921

    Compare Source

    Configuration

    ๐Ÿ“… Schedule: At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Disabled due to failing status checks.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

    dependencies 
    opened by renovate[bot] 2
  • chore(deps): update dependency platformdirs to <2.6.3

    chore(deps): update dependency platformdirs to <2.6.3

    Mend Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | platformdirs | <2.6.1 -> <2.6.3 | age | adoption | passing | confidence |

    Release Notes

    platformdirs/platformdirs

    v2.6.2

    Compare Source

    • Fix missing typing-extensions dependency.

    v2.6.1

    Compare Source

    • Add detection of $PREFIX for android.

    Configuration

    ๐Ÿ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Enabled.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    dependencies 
    opened by renovate[bot] 1
  • fix(deps): update dependency protobuf to v4

    fix(deps): update dependency protobuf to v4

    Mend Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | protobuf | ^3.19.1 -> ^4.0.0 | age | adoption | passing | confidence |

    Configuration

    ๐Ÿ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Enabled.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by Mend Renovate. View repository job log here.

    dependencies 
    opened by renovate[bot] 1
  • chore(flake/nixpkgs): `19e187fb` -> `836c52e6`

    chore(flake/nixpkgs): `19e187fb` -> `836c52e6`

    | SHA256 | Commit Message | | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | fbccf6ea | python39Packages.ipywidgets: use pytestCheckHook and general cleanup | | 162d2d3a | emacs.pkgs.ement: unstable-2022-04-22 -> unstable-2022-05-05 | | c58ed2fc | python3Packages.awkward: update meta.homepage | | f02a08d4 | headset-charge-indicator: init at 2021-08-15 | | 881a8cbe | nixos/kanidm: fix release notes | | c126babb | nixos/kanidm: init | | 3b5fc1fd | kanidm: init at 1.1.0-alpha.8 | | 9af7f009 | python3Packages.azure-mgmt-msi: update propagatedBuildInputs | | 974603c9 | ecdsautils: 0.4.0 -> 0.4.1 | | 6bffe188 | python310Packages.databricks-connect: 9.1.14 -> 9.1.15 | | 1d33da66 | python310Packages.aiolimiter: init at 1.0.0 | | 38c70972 | python310Packages.azure-mgmt-resource: 21.0.0 -> 21.1.0 | | bb117f73 | python310Packages.globus-sdk: 3.7.0 -> 3.8.0 | | 321d31a4 | iwd: 1.26 -> 1.27 | | ab27f31b | home-assistant: update component-packages | | 38c5c8c0 | python310Packages.raincloudy: init at 1.1.1 | | 815f83a0 | home-assistant: update component-packages | | d2905c6f | python310Packages.pyrainbird: init at 0.4.3 | | d487bcce | bolt: 0.9.1 โ†’ 0.9.2 | | 23c4e4aa | sqlite-utils: 3.26 -> 3.26.1 | | e9f479ec | librewolf: 100.0-1 -> 100.0-2 | | 0d03ca8d | hopper: 4.5.29 -> 5.5.3 | | 0558cb24 | libffi_3_3: init | | cb7c4401 | maintainers: add armeenm | | f0255a6f | python310Packages.nbclient: run tests | | a38b583a | python310Packages.setupmeta: 3.3.0 -> 3.3.1 | | 7ec5bd9e | sift: add self to maintainers | | 763a2d7b | sift: add bash completion | | 3b26a349 | python310Packages.impacket: 0.9.24 -> 0.10.0 | | e33dbfd2 | opentabletdriver: 0.6.0.3 -> 0.6.0.4 | | 6f960f08 | home-assistant: update component-packages | | a2c236d7 | python310Packages.meater-python: init at 0.0.8 | | 58ec9c11 | python310Packages.scancode-toolkit: relax intbitset constraint | | 0f5df81d | python310Packages.intbitset: fix license | | 01853e27 | bsp-layout: fix postInstall | | ca8c8a1a | python39Packages.wandb: use the correct nbclient dependency | | 41fbaa22 | python3Packages.bond-api: 0.1.16 -> 0.1.17 | | 1dcc140d | sweethome3d: fix impurity | | 6036f050 | melody: 0.13.10 -> 0.18.0 | | f6c4cf25 | flip-link: 0.1.4 -> 0.1.6 | | f989e139 | zfs: Support zfs_force=y on the command line as well. | | 3ed4d50f | xfsprogs: 5.15.0 -> 5.16.0 | | bf59489f | liferea: 1.12.9 -> 1.13.8 | | d0b632a1 | proxysql: fix build | | abc98356 | maintainers: add zebreus | | b8b17d9b | power-profiles-daemon: 0.10.1 โ†’ 0.11.1 | | d9d51eec | libwebsockets: disable -Werror | | e0b5ba54 | nixos: Don't use grep to request ZFS credentials, and consider keystatus | | 3a71b113 | nixos: Include zfsroot in installer-systemd-stage-1 tests | | 0a161580 | zfs: Update comment for https://github.com/zfsonlinux/zfs/pull/4943 | | 8555a7fd | zfs: Allow three tries to decrypt datasets | | 44a6882f | nixos/stage-1-systemd: ZFS support | | 56ab4f61 | nixos/lxd: improve tests | | eb50502c | lucenepp: install header files of lucene-contrib | | 023a12f0 | inspircd: 3.12.0 -> 3.13.0 | | 14d54882 | libwebsockets: drop unmaintained older versions | | c517610e | swaynotificationcenter: 0.3 -> 0.5 | | 014b59a4 | umockdev: Make library path references absolute | | 35b85a12 | umockdev: 0.17.8 โ†’ 0.17.9 | | 47f0136d | neomutt: 20220415 -> 20220429 | | e6acae67 | wiki-js: 2.5.277 -> 2.5.279 | | f7ad1883 | mediainfo-gui: 21.09 -> 22.03 | | 7532dcd6 | libmediainfo: 21.09 -> 22.03 | | 3d76f7ec | input-remapper: unstable-2022-02-09 -> 1.4.2 | | a3c0afb1 | got: 0.68.1 -> 0.69 | | 4986504f | python38Packages.backports-zoneinfo: test data for zoneinfo 2022a | | 568cb2d6 | nixos/systemd/nspawn: Add missing nspawn unit options | | 0d8a3244 | AusweisApp2: 1.22.4 -> 1.22.5 | | 5a47886f | microsoft-edge: 98.0.1108.56 -> 100.0.1185.44 | | 82060bee | portfolio: 0.57.1 -> 0.57.2 | | 2f99b713 | timeular: 3.9.1 -> 4.7.1 | | 22419c93 | weechat-otr: Fix build and knownVulnerabilities | | a2b02ea6 | telegraf: 1.22.0 -> 1.22.1 | | 76591b5b | psi-plus: 1.5.1615 -> 1.5.1618 | | dcbe74f3 | protoc-gen-twirp_php: set version | | 5f63e522 | protoc-gen-twirp_php: 0.8.0 -> 0.8.1 | | 16131300 | p4v: 2020.1.1966006 -> 2021.3.2186916 |

    dependencies autorebase:opt-in released 
    opened by phillip-ground[bot] 1
  • chore(flake/nixpkgs): `ab0bd3af` -> `19e187fb`

    chore(flake/nixpkgs): `ab0bd3af` -> `19e187fb`

    | SHA256 | Commit Message | | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | 5a51d04e | onedrive: 2.4.16 -> 2.4.17 | | b8dd32df | python310Packages.afsapi: 0.2.3 -> 0.2.4 | | 57ce8ad6 | skytemple-rust: add Foundation library on darwin | | c777cdf5 | easycrypt-runtest: init at 2022.04 | | b20934a6 | easycrypt: init at 2022.04 | | 04297436 | python310Packages.sqlmap: 1.6.4 -> 1.6.5 | | cda4885f | python310Packages.peaqevcore: 0.0.21 -> 0.0.22 | | ae75c14f | awesome-4-0: remove | | 0ab6bceb | libdeltachat: 1.78.0 -> 1.79.0 | | 55352488 | rust-analyzer: 2022-04-11 -> 2022-05-02 | | 9dd25bbe | chezmoi: 2.15.1 -> 2.15.2 | | 02900c45 | tut: 0.0.42 -> 0.0.46 | | e02a0ba3 | python310Packages.nbclient: remove comment | | c83d28b8 | lookatme: remove | | 11ca7ae7 | home-assistant: disable more dependency tests | | 5aad7233 | home-assistant: 2022.4.7 -> 2022.5.0 | | b4049c2b | python3Packages.ansible: add meta.changelog | | 3581eb92 | python3Packages.ansible: 5.7.0 -> 5.7.1 | | f886c041 | python3Packages.insteon-frontend-home-assistant: init at 0.1.0 | | 9170fcf7 | python3Packages.gcal-sync: init at 0.7.1 | | 063015ee | poetry2nix: 1.27.1 -> 1.28.0 | | 5860a9e5 | senpai: unstable-2021-12-14 -> unstable-2022-04-29 | | ec0ae174 | Revert "networkmanager-applet: rename from networkmanagerapplet" | | f7269f24 | vulkan-loader: fix cross-compilation | | 440f2e1d | python3Packages.pydeconz: 90 -> 91 | | 2401e18f | python3Packages.total-connect-client: 2022.2.1 -> 2022.3 | | db3a1b15 | python310Packages.PyChromecast: 11.0.0 -> 12.0.0 | | f58df705 | python3Packages.voluptuous: 0.13.0 -> 0.13.1 | | 8006deeb | python3Packages.zwave-js-server-python: 0.35.3 -> 0.36.1 | | fc3ed4e8 | python3Packages.zigpy: 0.44.2 -> 0.45.1 | | b1a2be80 | python3Packages.simplisafe-python: 2022.03.3 -> 2022.5.0 | | f4c6bea0 | python3Packages.pytomorrowio: 0.2.1 -> 0.3.3 | | ed5b29c1 | python310Packages.pynws: 1.3.2 -> 1.4.1 | | f5b90ecc | python310Packages.pyinsteon: 1.0.16 -> 1.1.0 | | f12a40ff | python3Packages.pyevilgenius: 1.0.0 -> 2.0.0 | | 24971206 | python310Packages.pydeconz: 87 -> 90 | | c38b06a3 | python3Packages.async-upnp-client: 0.28.0 -> 0.29.0 | | 44f8ea01 | python3Packages.androidtv: 0.0.66 -> 0.0.67 | | 9a99c652 | python3Packages.aioslimproto: 2.0.0 -> 2.0.1 | | e9476f79 | python3Packages.aioairzone: 0.3.4 -> 0.4.2 | | 9146215b | python3Packages.nbclient: 0.5.13 -> 0.6.0 | | 5f8ac4de | python3Packages.nbconvert: 6.4.5 -> 6.5.0 | | 60b4f7d8 | zerobin: raise version bounds for bleach to <6 | | 0c90ed17 | python3Packages.bleach: 4.1.0 -> 5.0.0 | | 2d5d1d4c | pssh: 2.3.1 -> 2.3.4 | | de4a99db | brave: fix GPU acceleration on Wayland | | 443e8f64 | caprine-bin: 2.55.2 -> 2.55.4 | | 166b1378 | snakemake: 7.5.0 -> 7.6.1 | | b139ab79 | qownnotes: 22.4.1 -> 22.5.0 | | ef459179 | python310Packages.pykulersky: 0.5.3 -> 0.5.4 | | eb8ce992 | ssm-session-manager-plugin: 1.2.54.0 -> 1.2.312.0 | | 2efd1090 | rich-cli: 1.5.1 -> 1.7.0 | | 3f87b172 | python3Packages.textual: 0.1.15 -> 0.1.18 | | 716b60f6 | wolfssl: 5.2.0 -> 5.3.0 | | 3c95672b | syft: 0.44.1 -> 0.45.1 | | e310b2cd | grype: 0.36.0 -> 0.36.1 | | a5151720 | python39Packages.howdoi: disable failing tests | | ab093932 | modem-manager-gui: fix build with meson >= 0.61 | | 3ea13a6f | hare: run test suite | | c3a7edf7 | purescript: drop ncurses dep | | 7d5e8244 | just: 1.1.2 -> 1.1.3 | | cc67617d | udiskie: 2.4.0 -> 2.4.2 | | ecf564b2 | cargo-sync-readme: 1.0 -> 1.1 | | c0bb20e0 | nodejs-18_x: fix completion generation | | 7b1a7987 | platformsh: 3.79.1 -> 3.79.2 | | 70f212bb | nodejs-14_x: 14.19.1 -> 14.19.2 | | 596c5e7e | Static bwa | | 47588733 | Static build for samtools | | 967a5d78 | Static builds for HTSLIB | | 0b1c28d5 | Static build for megahit | | 342e99a5 | seafile-client: build v8.0.7 from proper commit | | be8f036f | gopass: 1.14.0 โ†’ 1.14.1 | | bb7bb243 | cloudflared: 2022.4.1 -> 2022.5.0 | | 6a4134d8 | fulcio: 0.4.0 -> 0.4.1 | | b060edea | upwork: 5.6.10.7 -> 5.6.10.13 | | 37cada5b | vlc: 3.0.17 -> 3.0.17.3 | | 5aec68a7 | prometheus-aws-s3-exporter: 0.4.1 -> 0.5.0 | | c25b7497 | runitor: init at 0.9.2 | | b918d3c7 | platformsh: 3.79.0 -> 3.79.1 | | 26a47abf | python310Packages.snowflake-connector-python: 2.7.6 -> 2.7.7 | | d64b2cc1 | twister: remove | | 67b88714 | nmap: remove graphical support | | 1be5ae11 | xpf: remove | | 8ddb45c1 | caffe2: remove | | 0fd723f3 | xpdf: 4.03 -> 4.04 | | 0f1bb1f4 | jadx: 1.3.4 -> 1.3.5 | | 153b2faf | goocanvas2: switch to python3 | | f0c470f5 | oath-toolkit: Rename from oathToolkit to oath-toolkit |

    dependencies autorebase:opt-in released 
    opened by phillip-ground[bot] 1
  • chore(deps): lock file maintenance

    chore(deps): lock file maintenance

    Mend Renovate

    This PR contains the following updates:

    | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed |

    ๐Ÿ”ง This Pull Request updates lock files to use the latest dependency versions.

    Configuration

    ๐Ÿ“… Schedule: Branch creation - "before 5am on monday" (UTC), Automerge - At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Enabled.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ‘ป Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    dependencies 
    opened by renovate[bot] 0
  • mypy not happy with `.pyi` rewriting

    mypy not happy with `.pyi` rewriting

    This project is a great, thank you very much for making and maintaining it.

    I have tried using it with typing however and I notice that mypy is not entirely happy with the rewriting of .pyi files.

    I have setup an example here: https://github.com/aucampia/issue-20221116-python_protobuf_cloudevents/tree/a245d29c2b1600b6580c6db861b4b715e72b9b51

    The specific .pyi file that upsets mypy is generated/protol/_plgen/example/v1/messages_pb2.pyi and the untransformed file is generated/proto/example/v1/messages_pb2.pyi

    With the mypy config from the example, running mypy yields:

    $ python -m poetry run mypy --show-error-codes --show-error-context 
tests/test_conflicting_ns.py:80: note: In module imported here:
generated/protol/_plgen/example/v1/messages_pb2.pyi: note: In member "field_b" of class "Something":
generated/protol/_plgen/example/v1/messages_pb2.pyi:24: error: Name "io.cloudevents.v1.cloudevents_pb2.CloudEvent" is not defined  [name-defined]
generated/protol/_plgen/example/v1/messages_pb2.pyi: note: In member "__init__" of class "Something":
generated/protol/_plgen/example/v1/messages_pb2.pyi:27: error: Name "io.cloudevents.v1.cloudevents_pb2.CloudEvent" is not defined  [name-defined]
Found 2 errors in 1 file (checked 5 source files)
make: *** [Makefile:78: python-validate-static] Error 1

    The specific part of the transformation that is causing the problem is this I believe:

    $ diff -u -r generated/proto/example/v1/messages_pb2.pyi generated/protol/_plgen/example/v1/messages_pb2.pyi 
--- generated/proto/example/v1/messages_pb2.pyi	2022-11-22 09:08:54.417836214 +0100
+++ generated/protol/_plgen/example/v1/messages_pb2.pyi	2022-11-22 09:08:55.062838530 +0100
@@ -5,32 +5,31 @@
 import builtins
 import google.protobuf.descriptor
 import google.protobuf.message
-import io.cloudevents.v1.cloudevents_pb2
+from ... import io
 import sys

    If I change this by hand to instead do from ...io.cloudevents.v1 import cloudevents_pb2 as io_dot_cloudevents_dot_v1_dot_cloudevents__pb2 and make corresponding changes in the rest of the file to use io_dot_cloudevents_dot_v1_dot_cloudevents__pb2 instead of io.cloudevents.v1.cloudevents_pb2 then mypy is entirely happy.

    opened by aucampia 0
  • No such option `--descriptor_set_in`

    No such option `--descriptor_set_in`

    It would be great if protoletariat would pass the --descriptor_set_in parameter to protoc compiler. Now it just returns an error:

    Usage: protol protoc [OPTIONS] PROTO_FILES...
Try 'protol protoc --help' for help.

Error: No such option: --descriptor_set_in
    opened by bartek1912 1
  • Error while using `grpc-tools.protoc` for `protoc` subcommand

    Error while using `grpc-tools.protoc` for `protoc` subcommand

    Console error:

      File "/Users/seanchok/Documents/rpc/.venv/lib/python3.10/site-packages/protoletariat/fdsetgen.py", line 137, in generate_file_descriptor_set_bytes
    subprocess.check_output(
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'python -m grpc_tools.protoc'

    Steps to reproduce:

    1. install grpc-tools through pip to use protoc

    2. generate python bindings

    python -m grpc-tools.protoc \
  --proto_path=. \
  --python_out=out \
  --grpc_python_out=out \
  *.proto
    1. Use protol protoc
    protol \
  --create-package \
  --in-place \
  --python_out out
  protoc \
  --protoc-path "python -m grpc_tools.protoc" \
  --proto-path .
  *.proto

    My own research:

    1. I read through the proloteriat source code, and suspected that the subprocess.check_output call may not be able to call python to run executable modules just like that. But I'm not entirely sure because the error thrown appears to be FileNotFound.

    2. I created a workaround by first generating the FileDescriptorSet file, and then running protol raw instead of protol protoc.

    touch descriptor.temp

# generate FileDescriptorSet
python -m grpc_tools.protoc \
  --include_imports \
  --descriptor_set_out=descriptor.temp \
  --proto_path=. \
  *.proto

# run post-processing to change absolute imports generated by protoc to relative imports
protol \
  --create-package \
  --in-place \
  --python-out out \
  raw \
  descriptor.temp

    I got the inspiration for the workaround after reading this section of code.

    I'm not sure if protol needs to be updated to make running python -m grpc_tools.protoc possible. Or if this workaround should be made known to other people who are using the grpc-tools package as I am.

    opened by 98sean98 3
  • Dependency Dashboard

    Dependency Dashboard

    This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

    Open

    These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

    Detected dependencies

    github-actions
    .github/workflows/auto-rebase.yml
    • tibdex/github-app-token v1
    • Label305/AutoRebase v0.1
    .github/workflows/ci.yml
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    • actions/checkout v3
    • bufbuild/buf-setup-action v1.11.0
    • actions/setup-python v4
    • conda-incubator/setup-miniconda v2
    • actions/checkout v3
    • bufbuild/buf-setup-action v1.11.0
    • actions/setup-python v4
    • conda-incubator/setup-miniconda v2
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    • actions/checkout v3
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    • tibdex/github-app-token v1
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    .github/workflows/codeql-analysis.yml
    • actions/checkout v3
    • github/codeql-action v2
    • github/codeql-action v2
    • github/codeql-action v2
    .github/workflows/docker.yml
    • actions/checkout v3
    • cachix/install-nix-action v18
    • cachix/cachix-action v12
    • docker/metadata-action v4
    • docker/login-action v2
    • docker/build-push-action v3
    .github/workflows/update-deps.yml
    • actions/checkout v3
    • cachix/install-nix-action v18
    • actions/checkout v3
    • cachix/install-nix-action v18
    • tibdex/github-app-token v1
    • cpcloud/flake-update-action v1.0.4
    poetry
    pyproject.toml
    • astunparse >=1.6.3,<2
    • click >=8,<9
    • protobuf >=3.19.1,<5
    • black >=22,<24
    • grpc-stubs >=1.24.7,<2
    • mypy >=0.991,<1
    • mypy-protobuf >=3,<4
    • types-protobuf >=4.21,<5
    • grpcio >=1.42,<2
    • pytest >=7,<8
    • pytest-randomly >=3.10.1,<4
    • ruff >=0.0.198,<1
    • tomli >=1,<3
    • [ ] Check this box to trigger a request for Renovate to run again on this repository
    no-op 
    opened by renovate[bot] 0
Releases(2.0.0)
Owner
Phillip Cloud
I like puns.
Phillip Cloud
GitHub
Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.

MetaCall Polyglot REPL Description This repository implements a Polyglot REPL which shares the state of the meta-object protocol between the REPLs. Us

MetaCall 10 Dec 28, 2022
๐Ÿ’ป Algo-Phantoms-Backend is an Application that provides pathways and quizzes along with a code editor to help you towards your DSA journey.๐Ÿ“ฐ๐Ÿ”ฅ This repository contains the REST APIs of the application.โœจ

Algo-Phantom-Backend ?? Algo-Phantoms-Backend is an Application that provides pathways and quizzes along with a code editor to help you towards your D

Algo Phantoms 44 Nov 15, 2022
Enhanced version of blender's bvh add-on with more settings supported. The bvh's rest pose should have the same handedness as the armature while could use a different up/forward definiton.

Enhanced bvh add-on (importer/exporter) for blender Enhanced bvh add-on (importer/exporter) for blender Enhanced bvh importer Enhanced bvh exporter Ho

James Zhao 16 Dec 20, 2022
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Protocol Buffers 57.6k Jan 3, 2023
A very simple tool for situations where optimization with onnx-simplifier would exceed the Protocol Buffers upper file size limit of 2GB, or simply to separate onnx files to any size you want.

sne4onnx A very simple tool for situations where optimization with onnx-simplifier would exceed the Protocol Buffers upper file size limit of 2GB, or

Katsuya Hyodo 10 Aug 30, 2022
Run your jupyter notebooks as a REST API endpoint. This isn't a jupyter server but rather just a way to run your notebooks as a REST API Endpoint.

Jupter Notebook REST API Run your jupyter notebooks as a REST API endpoint. This isn't a jupyter server but rather just a way to run your notebooks as

Invictify 54 Nov 4, 2022
RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

Microsoft 1.8k Jan 4, 2023
User-related REST API based on the awesome Django REST Framework

Django REST Registration User registration REST API, based on Django REST Framework. Documentation Full documentation for the project is available at

Andrzej Pragacz 399 Jan 3, 2023
Django-rest-auth provides a set of REST API endpoints for Authentication and Registration

This app makes it extremely easy to build Django powered SPA's (Single Page App) or Mobile apps exposing all registration and authentication related functionality as CBV's (Class Base View) and REST (JSON)

Tivix 2.4k Dec 29, 2022
Atualizando o projeto APIs REST Django REST 2.0

APIs REST Django REST 3.0-KevinSoffa Atualizaรงรฃo do projeto APIs REST Django REST 2.0-Kevin Soffa Melhorando e adicionando funcionalidades O que jรก fo

Kevin Soffa 2 Dec 13, 2022
A music recommendation REST API which makes a machine learning algorithm work with the Django REST Framework

music-recommender-rest-api A music recommendation REST API which makes a machine learning algorithm work with the Django REST Framework How it works T

The Reaper 1 Sep 28, 2021
Pf-flask-rest-com - Flask REST API Common Implementation by Problem Fighter Library

In the name of God, the Most Gracious, the Most Merciful. PF-Flask-Rest-Com Docu

Problem Fighter 3 Jan 15, 2022
Mlflow-rest-client - Python client for MLflow REST API

Python Client for MLflow Python client for MLflow REST API. Features: Minimal de

MTS 35 Dec 23, 2022
The leading native Python SSHv2 protocol library.

Paramiko Paramiko: Python SSH module Copyright: Copyright (c) 2009 Robey Pointer <[email protected]> Copyright: Copyright (c) 2020 Jeff Forcier <

null 8.1k Jan 8, 2023
FLEX (Federated Learning EXchange๏ผŒFLEX) protocol is a set of standardized federal learning agreements designed by Tongdun AI Research Groupใ€‚

Click to view Chinese version FLEX (Federated Learning Exchange) protocol is a set of standardized federal learning agreements designed by Tongdun AI

ๅŒ็›พ็ง‘ๆŠ€ 50 Nov 29, 2022
A python module to parse the Open Graph Protocol

OpenGraph is a module of python for parsing the Open Graph Protocol, you can read more about the specification at http://ogp.me/ Installation $ pip in

Erik Rivera 213 Nov 12, 2022
A Django app for managing robots.txt files following the robots exclusion protocol

Django Robots This is a basic Django application to manage robots.txt files following the robots exclusion protocol, complementing the Django Sitemap

Jazzband 406 Dec 26, 2022
The leading native Python SSHv2 protocol library.

Paramiko Paramiko: Python SSH module Copyright: Copyright (c) 2009 Robey Pointer <[email protected]> Copyright: Copyright (c) 2020 Jeff Forcier <

null 8.1k Jan 4, 2023
fastapi-mqtt is extension for MQTT protocol

fastapi-mqtt MQTT is a lightweight publish/subscribe messaging protocol designed for M2M (machine to machine) telemetry in low bandwidth environments.

Sabuhi 144 Dec 28, 2022
fastapi-mqtt is extension for MQTT protocol

fastapi-mqtt MQTT is a lightweight publish/subscribe messaging protocol designed for M2M (machine to machine) telemetry in low bandwidth environments.

Sabuhi 23 Feb 11, 2021