|
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291 |
- # Managing dependencies with Poetry
-
- This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/).
-
- # Installing
-
- See the [contributing guide](contributing_guide.md#4-install-the-dependencies).
-
- Developers should use Poetry 1.3.2 or higher. If you encounter problems related
- to poetry, please [double-check your poetry version](#check-the-version-of-poetry-with-poetry---version).
-
- # Background
-
- Synapse uses a variety of third-party Python packages to function as a homeserver.
- Some of these are direct dependencies, listed in `pyproject.toml` under the
- `[tool.poetry.dependencies]` section. The rest are transitive dependencies (the
- things that our direct dependencies themselves depend on, and so on recursively.)
-
- We maintain a locked list of all our dependencies (transitive included) so that
- we can track exactly which version of each dependency appears in a given release.
- See [here](https://github.com/matrix-org/synapse/issues/11537#issue-1074469665)
- for discussion of why we wanted this for Synapse. We chose to use
- [`poetry`](https://python-poetry.org/) to manage this locked list; see
- [this comment](https://github.com/matrix-org/synapse/issues/11537#issuecomment-1015975819)
- for the reasoning.
-
- The locked dependencies get included in our "self-contained" releases: namely,
- our docker images and our debian packages. We also use the locked dependencies
- in development and our continuous integration.
-
- Separately, our "broad" dependencies—the version ranges specified in
- `pyproject.toml`—are included as metadata in our "sdists" and "wheels" [uploaded
- to PyPI](https://pypi.org/project/matrix-synapse). Installing from PyPI or from
- the Synapse source tree directly will _not_ use the locked dependencies; instead,
- they'll pull in the latest version of each package available at install time.
-
- ## Example dependency
-
- An example may help. We have a broad dependency on
- [`phonenumbers`](https://pypi.org/project/phonenumbers/), as declared in
- this snippet from pyproject.toml [as of Synapse 1.57](
- https://github.com/matrix-org/synapse/blob/release-v1.57/pyproject.toml#L133
- ):
-
- ```toml
- [tool.poetry.dependencies]
- # ...
- phonenumbers = ">=8.2.0"
- ```
-
- In our lockfile this is
- [pinned]( https://github.com/matrix-org/synapse/blob/dfc7646504cef3e4ff396c36089e1c6f1b1634de/poetry.lock#L679-L685)
- to version 8.12.44, even though
- [newer versions are available](https://pypi.org/project/phonenumbers/#history).
-
- ```toml
- [[package]]
- name = "phonenumbers"
- version = "8.12.44"
- description = "Python version of Google's common library for parsing, formatting, storing and validating international phone numbers."
- category = "main"
- optional = false
- python-versions = "*"
- ```
-
- The lockfile also includes a
- [cryptographic checksum](https://github.com/matrix-org/synapse/blob/release-v1.57/poetry.lock#L2178-L2181)
- of the sdists and wheels provided for this version of `phonenumbers`.
-
- ```toml
- [metadata.files]
- # ...
- phonenumbers = [
- {file = "phonenumbers-8.12.44-py2.py3-none-any.whl", hash = "sha256:cc1299cf37b309ecab6214297663ab86cb3d64ae37fd5b88e904fe7983a874a6"},
- {file = "phonenumbers-8.12.44.tar.gz", hash = "sha256:26cfd0257d1704fe2f88caff2caabb70d16a877b1e65b6aae51f9fbbe10aa8ce"},
- ]
- ```
-
- We can see this pinned version inside the docker image for that release:
-
- ```
- $ docker pull matrixdotorg/synapse:v1.57.0
- ...
- $ docker run --entrypoint pip matrixdotorg/synapse:v1.57.0 show phonenumbers
- Name: phonenumbers
- Version: 8.12.44
- Summary: Python version of Google's common library for parsing, formatting, storing and validating international phone numbers.
- Home-page: https://github.com/daviddrysdale/python-phonenumbers
- Author: David Drysdale
- Author-email: dmd@lurklurk.org
- License: Apache License 2.0
- Location: /usr/local/lib/python3.9/site-packages
- Requires:
- Required-by: matrix-synapse
- ```
-
- Whereas the wheel metadata just contains the broad dependencies:
-
- ```
- $ cd /tmp
- $ wget https://files.pythonhosted.org/packages/ca/5e/d722d572cc5b3092402b783d6b7185901b444427633bd8a6b00ea0dd41b7/matrix_synapse-1.57.0rc1-py3-none-any.whl
- ...
- $ unzip -c matrix_synapse-1.57.0rc1-py3-none-any.whl matrix_synapse-1.57.0rc1.dist-info/METADATA | grep phonenumbers
- Requires-Dist: phonenumbers (>=8.2.0)
- ```
-
- # Tooling recommendation: direnv
-
- [`direnv`](https://direnv.net/) is a tool for activating environments in your
- shell inside a given directory. Its support for poetry is unofficial (a
- community wiki recipe only), but works solidly in our experience. We thoroughly
- recommend it for daily use. To use it:
-
- 1. [Install `direnv`](https://direnv.net/docs/installation.html) - it's likely
- packaged for your system already.
- 2. Teach direnv about poetry. The [shell config here](https://github.com/direnv/direnv/wiki/Python#poetry)
- needs to be added to `~/.config/direnv/direnvrc` (or more generally `$XDG_CONFIG_HOME/direnv/direnvrc`).
- 3. Mark the synapse checkout as a poetry project: `echo layout poetry > .envrc`.
- 4. Convince yourself that you trust this `.envrc` configuration and project.
- Then formally confirm this to `direnv` by running `direnv allow`.
-
- Then whenever you navigate to the synapse checkout, you should be able to run
- e.g. `mypy` instead of `poetry run mypy`; `python` instead of
- `poetry run python`; and your shell commands will automatically run in the
- context of poetry's venv, without having to run `poetry shell` beforehand.
-
-
- # How do I...
-
- ## ...reset my venv to the locked environment?
-
- ```shell
- poetry install --all-extras --sync
- ```
-
- ## ...delete everything and start over from scratch?
-
- ```shell
- # Stop the current virtualenv if active
- $ deactivate
-
- # Remove all of the files from the current environment.
- # Don't worry, even though it says "all", this will only
- # remove the Poetry virtualenvs for the current project.
- $ poetry env remove --all
-
- # Reactivate Poetry shell to create the virtualenv again
- $ poetry shell
- # Install everything again
- $ poetry install --extras all
- ```
-
- ## ...run a command in the `poetry` virtualenv?
-
- Use `poetry run cmd args` when you need the python virtualenv context.
- To avoid typing `poetry run` all the time, you can run `poetry shell`
- to start a new shell in the poetry virtualenv context. Within `poetry shell`,
- `python`, `pip`, `mypy`, `trial`, etc. are all run inside the project virtualenv
- and isolated from the rest o the system.
-
- Roughly speaking, the translation from a traditional virtualenv is:
- - `env/bin/activate` -> `poetry shell`, and
- - `deactivate` -> close the terminal (Ctrl-D, `exit`, etc.)
-
- See also the direnv recommendation above, which makes `poetry run` and
- `poetry shell` unnecessary.
-
-
- ## ...inspect the `poetry` virtualenv?
-
- Some suggestions:
-
- ```shell
- # Current env only
- poetry env info
- # All envs: this allows you to have e.g. a poetry managed venv for Python 3.7,
- # and another for Python 3.10.
- poetry env list --full-path
- poetry run pip list
- ```
-
- Note that `poetry show` describes the abstract *lock file* rather than your
- on-disk environment. With that said, `poetry show --tree` can sometimes be
- useful.
-
-
- ## ...add a new dependency?
-
- Either:
- - manually update `pyproject.toml`; then `poetry lock --no-update`; or else
- - `poetry add packagename`. See `poetry add --help`; note the `--dev`,
- `--extras` and `--optional` flags in particular.
-
- Include the updated `pyproject.toml` and `poetry.lock` files in your commit.
-
- ## ...remove a dependency?
-
- This is not done often and is untested, but
-
- ```shell
- poetry remove packagename
- ```
-
- ought to do the trick. Alternatively, manually update `pyproject.toml` and
- `poetry lock --no-update`. Include the updated `pyproject.toml` and `poetry.lock`
- files in your commit.
-
- ## ...update the version range for an existing dependency?
-
- Best done by manually editing `pyproject.toml`, then `poetry lock --no-update`.
- Include the updated `pyproject.toml` and `poetry.lock` in your commit.
-
- ## ...update a dependency in the locked environment?
-
- Use
-
- ```shell
- poetry update packagename
- ```
-
- to use the latest version of `packagename` in the locked environment, without
- affecting the broad dependencies listed in the wheel.
-
- There doesn't seem to be a way to do this whilst locking a _specific_ version of
- `packagename`. We can workaround this (crudely) as follows:
-
- ```shell
- poetry add packagename==1.2.3
- # This should update pyproject.lock.
-
- # Now undo the changes to pyproject.toml. For example
- # git restore pyproject.toml
-
- # Get poetry to recompute the content-hash of pyproject.toml without changing
- # the locked package versions.
- poetry lock --no-update
- ```
-
- Either way, include the updated `poetry.lock` file in your commit.
-
- ## ...export a `requirements.txt` file?
-
- ```shell
- poetry export --extras all
- ```
-
- Be wary of bugs in `poetry export` and `pip install -r requirements.txt`.
-
- ## ...build a test wheel?
-
- I usually use
-
- ```shell
- poetry run pip install build && poetry run python -m build
- ```
-
- because [`build`](https://github.com/pypa/build) is a standardish tool which
- doesn't require poetry. (It's what we use in CI too). However, you could try
- `poetry build` too.
-
-
- # Troubleshooting
-
- ## Check the version of poetry with `poetry --version`.
-
- The minimum version of poetry supported by Synapse is 1.3.2.
-
- It can also be useful to check the version of `poetry-core` in use. If you've
- installed `poetry` with `pipx`, try `pipx runpip poetry list | grep
- poetry-core`.
-
- ## Clear caches: `poetry cache clear --all pypi`.
-
- Poetry caches a bunch of information about packages that isn't readily available
- from PyPI. (This is what makes poetry seem slow when doing the first
- `poetry install`.) Try `poetry cache list` and `poetry cache clear --all
- <name of cache>` to see if that fixes things.
-
- ## Remove outdated egg-info
-
- Delete the `matrix_synapse.egg-info/` directory from the root of your Synapse
- install.
-
- This stores some cached information about dependencies and often conflicts with
- letting Poetry do the right thing.
-
-
-
- ## Try `--verbose` or `--dry-run` arguments.
-
- Sometimes useful to see what poetry's internal logic is.
|