3.1. partypy: simulate attendance at your party!

• [x] Love the package name partpy 😂
• [x] Can we give some more details at a high level after we write "We can repeat this process as many times as we like to generate multiple estimates of how many guests might attend our real-life event." to explain how we will use these multiple estimates to get a final, single estimate of how many people might attend your party. Many of our readers may not be data scientists, or data scientists that have expertise with simulations. So we should clearly and explicitly close the loop here.
• [x] print(f"Average guests: {results.mean()}") does not print anything out that I can see when the book is rendered.
• [x] in the "partypy: simulate attendance at your party!" section, I think we should call it "Simulate attendance at your party!" and not use the packaged code, that's a bit confusing I think. I think the raw/unpackaged code should be here to show and explain the example. Perhaps then you can refactor it into functions, and then say, the remainder of this chapter will package up these functions?
• [x] There is also a big expert leap going on in this section with regards to the raw data > histogram. How does this magic happen? I think my feedback above, will help this. I am slightly worried that this package example is too complex, and so it distracts from the point of the chapter (packaging your code), but let me read on and see...
• [x] How do we interpret the histogram? This section seems a bit unfinished. This interpretation is needed to close the motivational loop.

Hi @TomasBeuzen and @ttimbers -- thank you again for this great guide.

I'm wondering if it would be worth adding some sort of callout somewhere, like a note or warning, about putting upper caps on dependencies?

By "upper caps" I mean what you get when you accept the default "caret" operator that poetry uses, to mean "This version and any version that is less than a 'major' version change under semantic versioning".

As you show here for your "adding dependencies example": https://github.com/py-pkgs/py-pkgs/blob/eb68c4fcb7b48cf736d064e023aabdea7070dd91/py-pkgs/bookdown/03-how-to-package-a-python.Rmd#L644

and also shown here for Python itself: https://github.com/py-pkgs/py-pkgs/blob/eb68c4fcb7b48cf736d064e023aabdea7070dd91/py-pkgs/bookdown/03-how-to-package-a-python.Rmd#L492

Guessing you know what I mean but just making sure I'm clear.

The basic issue with this default is that it's "infectious" -- it forces everyone else that depends on your package to meet the same constraints. This means that they can't, e.g., use a newer version of some dependency that fixes a security issue.

By making their caret operator a default, poetry promotes this, as does language in their documentation. I think for the poetry devs this recommendation makes sense, because they are thinking mainly in terms of deployed applications, and because they think every package should adhere to the strictest interpretation of Semantic Versioning.

But it can have really painful knock-on effects for downstream users, especially relative beginners like me who have no idea why everything is suddenly broken. I have run into issues resulting from accepting that default for my own packages.

The knock-on effects can get pretty nuanced (and I'm not sure I understand them all). See these detailed blog posts from @henryiii for more background: https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/poetry-versions/

Like you all (and @henryiii ) I really like poetry because it provides an all-in-one solution, but I'm a bit worried about a lot of people reading your wonderful guide, just accepting the defaults, and then we're all forced to specify python=>3.8.1 <4.0 and jedi^0.19 all the time. I really don't want to constantly have to be tracking when all my dependencies release new versions.

Not sure where the best place to insert this kind of language would be, but just thought I'd suggest it so maybe others can avoid my pain 🙂

@ttimbers I've been thinking a lot about a potentially different package we could develop through the book that will make it easier to showcase various important elements of the packaging process:

• multiple functions/sub-packages
• functions that create plots/writing tests for viz
• shipping data with the package
• something with multiple dependencies (e.g., numpy, matplotlib, etc)
• etc

One idea I had was to build a package similar to my simple pyguest package. A package for estimating attendance at an event (like a party or wedding) using simulations of Bernoulli RVs and probabilities stored in a dataframe. It's simple, easy to describe, easy to expand (e.g., add functionality to also simulate if +1's will come), we could ship example data with it, we could have plotting function (e.g., histogram of simulation outcomes). Also the name partypy is currently available on pypi 😆 Let me know what you think. I can always whip up a quick draft of what Chapter 3 How to package a Python might look like with this revised package.

This PR includes:

• General structural changes to book and repo (closes: #28)
• Added a preface and introduction (closes: #31, closes: #32)
• Move some chapters to appendices

I've been developing some packages lately and have found myself going over this book searching for particular commands I know I want to use. As someone who understands what's going on and only needs the commands to execute certain tasks, I think a "cheatsheet" would be really helpful (maybe as an appendix or preface?). I envision a very simply document showing the step-by-step commands for creating a package, but without much (if any) documentation, e.g.:

Package Set Up

1. Create directory structure: cookiecutter https://github.com/UBC-MDS/cookiecutter-ubc-mds.git
2. Initialize poetry: cd pkg-name & poetry init etc

Package Buidling/Testing

1. Add dependencies: poetry add numpy
2. Add dev dependencies: poetry add --dev sphinx
3. Lock dependency list/install dependencies/install package locally: poetry install etc

Documentation

1. Modify contents of docs
2. Download napoleon: poetry add --dev sphinxcontrib-napoleon
3. Render code documentation with poetry run sphinx-apidoc -f -o docs/source foocat
4. Render package documentation: cd docs, then poetry run make html etc

I've really enjoyed the book and plan to implement your suggestions to two packages that I currently have. I have these currently on GitHub that I install via pip install git+XXX, so they show up in my environment using their existing names (when I do pip list). I also already have GitHub project names for these two projects.

But now, I kind of want to start over to get these packages going, yet I already have folders and GitHub projects for these packages. I also already have conda environments set up using these package names as well. So, how do you best recommend starting up new repos for these two new packages, when I don't really want to change the names of these pypi versions? For example, I have a package called jakomics that I would like to put into pypi with that name, but I already have all of this history and "code baggage" using that name?

Hopefully this isn't too vague of a question, but I'm also sure I'm not the first to convert a local package to a pypi package. Thanks for your help!

How come (in section 7.2.2) the CI/CD pipeline is not (re-)triggered when the python-semantic-release package commits a new version to the 'main' branch? Shouldn't a new commit count as a push and therefore start a new pipeline execution (which would start an infinite loop)?

name: ci-cd

on: [push, pull_request]

jobs:

We use the hypothetical foocat package throughout the book at the moment. But in all chapters after "The Whole Game" it would be great to have this package available on pypi so that readers can actually pip install it and run through the code. Even better, we could pip install it so that later chapters can actually execute the code they are showing, rather than having it embedded in markdown.

Hi there,

I'm having trouble getting the example to run. I cannot seem to get the module to import.

I'm at section 3.5. I've installed poetry (poetry --version returns 1.1.12), and set up a conda environment (pycounts).

The guide is a little unclear about if the poetry calls need to be within the conda env or not, but neither works. From the pycounts folder, if I run poetry install I get the following

The virtual environment found in C:\Users\taren\miniconda3\envs\pycounts seems to be broken.
Recreating virtualenv pycounts-tZwN6m1u-py3.9 in C:\Users\taren\AppData\Local\pypoetry\Cache\virtualenvs\pycounts-tZwN6m1u-py3.9
Installing dependencies from lock file

Installing the current project: pycounts (0.1.0)

However, opening a python shell and trying to import the module gives:

>>> from pycounts.pycounts import count_words
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pycounts

Repeating these steps but not in the conda env does not give the warning about a broken virtual env

Installing dependencies from lock file

Installing the current project: pycounts (0.1.0)

But the import error is the same:

>>> from pycounts.pycounts import count_words
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pycounts'

Is there any chance you could help diagnose the issue? I'm using Windows. I've tried:

• A complete wipe of miniconda, including all envs
• Deleting the package and starting again
• Removing and reinstalling poetry

Really appreciate any help.

Looks like the command to install Poetry has been changed. The current command in the book is causing issues... This PR updates it. If this changes a lot, perhaps we just want to point at the installation page long-term so this is not so brittle?

Thanks for the great work - really very helpful to get up to speed in Python with some background in R.

Two minor points:

1. we can now build our documentation with sphinx using the following command from our root package directory This did not work for me (on Windows) from the root directory, but from directory docs.

2. At some points in the html-document, the string "\newpage" appears.

The mention of namespaces on this page is fine if rudimentary. However, there is also the concept of a "namespace package", something that my organization is making use of. This involves a somewhat different project structure than is shown in the book's examples; and it presupposes that there will be multiple projects sharing the same namespace, which may or may not reside in the same repository.

In the attention block of https://py-pkgs.org/03-how-to-package-a-python#testing-your-package:

If you’re not developing your package in a conda virtual environment, poetry will automatically create a virtual environment for you using a tool called venv (read more in the documentation). You’ll need to tell poetry to use this environment by prepending any command you run with poetry run, like: poetry run pytest tests/.

If you want the code to be a bit more tool agnostic (i.e., not always rely on poetry), it seems that after you install pytest into whatever environment you are in, you can reference pytest by using python -m. The book already uses python -c earlier to create the text file for the example, so this can also be mentioned?

Using poetry run:

(py-pkgs-poetry) danielchen@heartsbane pycounts_poetry % poetry run pytest tests/
============================================ test session starts =============================================
platform darwin -- Python 3.10.3, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/danielchen/temp/py-pkgs-poetry/pycounts
collected 1 item                                                                                             

tests/test_pycounts.py .                                                                               [100%]

============================================= 1 passed in 0.00s ==============================================

Using python -m:

(py-pkgs-poetry) danielchen@heartsbane pycounts_poetry % python -m pytest tests/
============================================ test session starts =============================================
platform darwin -- Python 3.10.3, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/danielchen/temp/py-pkgs-poetry/pycounts
collected 1 item                                                                                             

tests/test_pycounts.py .                                                                               [100%]

============================================= 1 passed in 0.00s ==============================================

The tip in 3.5 (page /03-how-to-package-a-python.html) refers to the "Angular style" for Git commit messages:

In this book, we use the Angular style for Git commit messages.

Just learned that style follows what's known as "conventional commits": https://www.conventionalcommits.org/ (it also references the angular conventions for other types)

edited by Tomas Beuzen

This is a scratchpad for topics we'd like to add or cover more comprehensively in the book:

• Dependency updating with Dependabot. This would fit nicely in the cookiecutter and book.
• Advanced fixture usage:
  • Mocking
  • Setting up environments for test to run in, e.g., directory structures, environment variables, etc.
• Style checking and formatting:
  • Using flake8 and/or black, and integrating into CI
• Packages with extensions in other languages (relevant comment)
• Releasing packages to Anaconda/conda-forge repositories
• Mention PyScaffold as an alternative templating tool
• Consider changing poetry to pdm (see discussions in #95)
• Consider providing more guidance on file/directory/repository/distribution/package naming conventions (see https://github.com/py-pkgs/py-pkgs-cookiecutter/issues/48 and this post)
• Consider adding more information/examples about namespace packages - currently we only briefly mention them at the bottom of [Section 4.2.1](https://py-pkgs.org/04-package-structure#package-contents (see #119)