* Upgrade to new lockfile format Now requires poetry >= 1.2.2 to read and poetry >= 1.3.0 to write. Cheat sheet: ``` poetry --version poetry show > scratch/before pipx upgrade poetry poetry --version poetry show > scratch/after diff scratch{before,after} && echo "no change!" ``` * Use Poetry 1.3.2 when reading or writing lockfile * Remove unneeded(?) poetry dep for cibuildwheel * Update docs * Remove redundant call to setup-python * Remove outdated comments related to Poetry 1.x * Remove outdated docs line was fixed in #13082 * Minor improvements to poetry cheat sheet * Invoke setup-python-poetry with explicit version Not sure about this. It's hardcoding versions everywhere. * Changelog * Check the lockfile is version 2.0 Might one day incorporate other checks like #14742 * Typo fixes, thanks Sean Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>tags/v1.76.0rc1
@@ -0,0 +1,23 @@ | |||||
#! /usr/bin/env python | |||||
import sys | |||||
if sys.version_info < (3, 11): | |||||
raise RuntimeError("Requires at least Python 3.11, to import tomllib") | |||||
import tomllib | |||||
with open("poetry.lock", "rb") as f: | |||||
lockfile = tomllib.load(f) | |||||
try: | |||||
lock_version = lockfile["metadata"]["lock-version"] | |||||
assert lock_version == "2.0" | |||||
except Exception: | |||||
print( | |||||
"""\ | |||||
Lockfile is not version 2.0. You probably need to upgrade poetry on your local box | |||||
and re-run `poetry lock --no-update`. See the Poetry cheat sheet at | |||||
https://matrix-org.github.io/synapse/develop/development/dependencies.html | |||||
""" | |||||
) | |||||
raise |
@@ -53,7 +53,7 @@ with open('pyproject.toml', 'w') as f: | |||||
" | " | ||||
python3 -c "$REMOVE_DEV_DEPENDENCIES" | python3 -c "$REMOVE_DEV_DEPENDENCIES" | ||||
pip install poetry==1.2.0 | |||||
pip install poetry==1.3.2 | |||||
poetry lock | poetry lock | ||||
echo "::group::Patched pyproject.toml" | echo "::group::Patched pyproject.toml" | ||||
@@ -37,7 +37,7 @@ jobs: | |||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
python-version: "3.x" | python-version: "3.x" | ||||
poetry-version: "1.2.0" | |||||
poetry-version: "1.3.2" | |||||
extras: "all" | extras: "all" | ||||
# Dump installed versions for debugging. | # Dump installed versions for debugging. | ||||
- run: poetry run pip list > before.txt | - run: poetry run pip list > before.txt | ||||
@@ -127,7 +127,7 @@ jobs: | |||||
python-version: "3.x" | python-version: "3.x" | ||||
- name: Install cibuildwheel | - name: Install cibuildwheel | ||||
run: python -m pip install cibuildwheel==2.9.0 poetry==1.2.0 | |||||
run: python -m pip install cibuildwheel==2.9.0 | |||||
- name: Set up QEMU to emulate aarch64 | - name: Set up QEMU to emulate aarch64 | ||||
if: matrix.arch == 'aarch64' | if: matrix.arch == 'aarch64' | ||||
@@ -33,11 +33,10 @@ jobs: | |||||
runs-on: ubuntu-latest | runs-on: ubuntu-latest | ||||
steps: | steps: | ||||
- uses: actions/checkout@v3 | - uses: actions/checkout@v3 | ||||
- uses: actions/setup-python@v4 | |||||
with: | |||||
python-version: "3.x" | |||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
python-version: "3.x" | |||||
poetry-version: "1.3.2" | |||||
extras: "all" | extras: "all" | ||||
- run: poetry run scripts-dev/generate_sample_config.sh --check | - run: poetry run scripts-dev/generate_sample_config.sh --check | ||||
- run: poetry run scripts-dev/config-lint.sh | - run: poetry run scripts-dev/config-lint.sh | ||||
@@ -52,6 +51,15 @@ jobs: | |||||
- run: "pip install 'click==8.1.1' 'GitPython>=3.1.20'" | - run: "pip install 'click==8.1.1' 'GitPython>=3.1.20'" | ||||
- run: scripts-dev/check_schema_delta.py --force-colors | - run: scripts-dev/check_schema_delta.py --force-colors | ||||
check-lockfile: | |||||
runs-on: ubuntu-latest | |||||
steps: | |||||
- uses: actions/checkout@v3 | |||||
- uses: actions/setup-python@v4 | |||||
with: | |||||
python-version: "3.x" | |||||
- run: .ci/scripts/check_lockfile.py | |||||
lint: | lint: | ||||
uses: "matrix-org/backend-meta/.github/workflows/python-poetry-ci.yml@v2" | uses: "matrix-org/backend-meta/.github/workflows/python-poetry-ci.yml@v2" | ||||
with: | with: | ||||
@@ -88,6 +96,7 @@ jobs: | |||||
ref: ${{ github.event.pull_request.head.sha }} | ref: ${{ github.event.pull_request.head.sha }} | ||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
poetry-version: "1.3.2" | |||||
extras: "all" | extras: "all" | ||||
- run: poetry run scripts-dev/check_pydantic_models.py | - run: poetry run scripts-dev/check_pydantic_models.py | ||||
@@ -163,6 +172,7 @@ jobs: | |||||
- lint-pydantic | - lint-pydantic | ||||
- check-sampleconfig | - check-sampleconfig | ||||
- check-schema-delta | - check-schema-delta | ||||
- check-lockfile | |||||
- lint-clippy | - lint-clippy | ||||
- lint-rustfmt | - lint-rustfmt | ||||
runs-on: ubuntu-latest | runs-on: ubuntu-latest | ||||
@@ -219,6 +229,7 @@ jobs: | |||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
python-version: ${{ matrix.job.python-version }} | python-version: ${{ matrix.job.python-version }} | ||||
poetry-version: "1.3.2" | |||||
extras: ${{ matrix.job.extras }} | extras: ${{ matrix.job.extras }} | ||||
- name: Await PostgreSQL | - name: Await PostgreSQL | ||||
if: ${{ matrix.job.postgres-version }} | if: ${{ matrix.job.postgres-version }} | ||||
@@ -294,6 +305,7 @@ jobs: | |||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
python-version: '3.7' | python-version: '3.7' | ||||
poetry-version: "1.3.2" | |||||
extras: "all test" | extras: "all test" | ||||
- run: poetry run trial -j6 tests | - run: poetry run trial -j6 tests | ||||
@@ -328,6 +340,7 @@ jobs: | |||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
python-version: ${{ matrix.python-version }} | python-version: ${{ matrix.python-version }} | ||||
poetry-version: "1.3.2" | |||||
extras: ${{ matrix.extras }} | extras: ${{ matrix.extras }} | ||||
- run: poetry run trial --jobs=2 tests | - run: poetry run trial --jobs=2 tests | ||||
- name: Dump logs | - name: Dump logs | ||||
@@ -419,6 +432,7 @@ jobs: | |||||
- run: sudo apt-get -qq install xmlsec1 postgresql-client | - run: sudo apt-get -qq install xmlsec1 postgresql-client | ||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
poetry-version: "1.3.2" | |||||
extras: "postgres" | extras: "postgres" | ||||
- run: .ci/scripts/test_export_data_command.sh | - run: .ci/scripts/test_export_data_command.sh | ||||
env: | env: | ||||
@@ -470,6 +484,7 @@ jobs: | |||||
- uses: matrix-org/setup-python-poetry@v1 | - uses: matrix-org/setup-python-poetry@v1 | ||||
with: | with: | ||||
python-version: ${{ matrix.python-version }} | python-version: ${{ matrix.python-version }} | ||||
poetry-version: "1.3.2" | |||||
extras: "postgres" | extras: "postgres" | ||||
- run: .ci/scripts/test_synapse_port_db.sh | - run: .ci/scripts/test_synapse_port_db.sh | ||||
id: run_tester_script | id: run_tester_script | ||||
@@ -148,7 +148,7 @@ jobs: | |||||
run: | | run: | | ||||
set -x | set -x | ||||
DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx | DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx | ||||
pipx install poetry==1.2.0 | |||||
pipx install poetry==1.3.2 | |||||
poetry remove -n twisted | poetry remove -n twisted | ||||
poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk | poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk | ||||
@@ -0,0 +1 @@ | |||||
Poetry 1.3.2 or higher is now required when `poetry install`ing from source. |
@@ -31,12 +31,11 @@ case $(dpkg-architecture -q DEB_HOST_ARCH) in | |||||
esac | esac | ||||
# Manually install Poetry and export a pip-compatible `requirements.txt` | # Manually install Poetry and export a pip-compatible `requirements.txt` | ||||
# We need a Poetry pre-release as the export command is buggy in < 1.2 | |||||
TEMP_VENV="$(mktemp -d)" | TEMP_VENV="$(mktemp -d)" | ||||
python3 -m venv "$TEMP_VENV" | python3 -m venv "$TEMP_VENV" | ||||
source "$TEMP_VENV/bin/activate" | source "$TEMP_VENV/bin/activate" | ||||
pip install -U pip | pip install -U pip | ||||
pip install poetry==1.2.0 | |||||
pip install poetry==1.3.2 | |||||
poetry export \ | poetry export \ | ||||
--extras all \ | --extras all \ | ||||
--extras test \ | --extras test \ | ||||
@@ -1,3 +1,9 @@ | |||||
matrix-synapse-py3 (1.75.1) UNRELEASED; urgency=medium | |||||
* Use Poetry 1.3.2 to manage the bundled virtualenv included with this package. | |||||
-- Synapse Packaging team <packages@matrix.org> Tue, 17 Jan 2023 15:08:00 +0000 | |||||
matrix-synapse-py3 (1.75.0) stable; urgency=medium | matrix-synapse-py3 (1.75.0) stable; urgency=medium | ||||
* New Synapse release 1.75.0. | * New Synapse release 1.75.0. | ||||
@@ -17,14 +17,8 @@ | |||||
# Irritatingly, there is no blessed guide on how to distribute an application with its | # Irritatingly, there is no blessed guide on how to distribute an application with its | ||||
# poetry-managed environment in a docker image. We have opted for | # poetry-managed environment in a docker image. We have opted for | ||||
# `poetry export | pip install -r /dev/stdin`, but there are known bugs in | |||||
# in `poetry export` whose fixes (scheduled for poetry 1.2) have yet to be released. | |||||
# In case we get bitten by those bugs in the future, the recommendations here might | |||||
# be useful: | |||||
# https://github.com/python-poetry/poetry/discussions/1879#discussioncomment-216865 | |||||
# https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker?answertab=scoredesc | |||||
# `poetry export | pip install -r /dev/stdin`, but beware: we have experienced bugs in | |||||
# in `poetry export` in the past. | |||||
ARG PYTHON_VERSION=3.9 | ARG PYTHON_VERSION=3.9 | ||||
@@ -49,7 +43,7 @@ RUN \ | |||||
# We install poetry in its own build stage to avoid its dependencies conflicting with | # We install poetry in its own build stage to avoid its dependencies conflicting with | ||||
# synapse's dependencies. | # synapse's dependencies. | ||||
RUN --mount=type=cache,target=/root/.cache/pip \ | RUN --mount=type=cache,target=/root/.cache/pip \ | ||||
pip install --user "poetry==1.2.0" | |||||
pip install --user "poetry==1.3.2" | |||||
WORKDIR /synapse | WORKDIR /synapse | ||||
@@ -67,7 +67,7 @@ pipx install poetry | |||||
but see poetry's [installation instructions](https://python-poetry.org/docs/#installation) | but see poetry's [installation instructions](https://python-poetry.org/docs/#installation) | ||||
for other installation methods. | for other installation methods. | ||||
Synapse requires Poetry version 1.2.0 or later. | |||||
Developing Synapse requires Poetry version 1.3.2 or later. | |||||
Next, open a terminal and install dependencies as follows: | Next, open a terminal and install dependencies as follows: | ||||
@@ -2,6 +2,13 @@ | |||||
This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/). | This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/). | ||||
# Installing | |||||
See the [contributing guide](contributing_guide.md#4-install-the-dependencies). | |||||
Developers should use Poetry 1.3.2 or higher. If you encounter problems related | |||||
to poetry, please [double-check your poetry version](#check-the-version-of-poetry-with-poetry---version). | |||||
# Background | # Background | ||||
Synapse uses a variety of third-party Python packages to function as a homeserver. | Synapse uses a variety of third-party Python packages to function as a homeserver. | ||||
@@ -123,7 +130,7 @@ context of poetry's venv, without having to run `poetry shell` beforehand. | |||||
## ...reset my venv to the locked environment? | ## ...reset my venv to the locked environment? | ||||
```shell | ```shell | ||||
poetry install --extras all --remove-untracked | |||||
poetry install --all-extras --sync | |||||
``` | ``` | ||||
## ...delete everything and start over from scratch? | ## ...delete everything and start over from scratch? | ||||
@@ -183,7 +190,6 @@ Either: | |||||
- manually update `pyproject.toml`; then `poetry lock --no-update`; or else | - manually update `pyproject.toml`; then `poetry lock --no-update`; or else | ||||
- `poetry add packagename`. See `poetry add --help`; note the `--dev`, | - `poetry add packagename`. See `poetry add --help`; note the `--dev`, | ||||
`--extras` and `--optional` flags in particular. | `--extras` and `--optional` flags in particular. | ||||
- **NB**: this specifies the new package with a version given by a "caret bound". This won't get forced to its lowest version in the old deps CI job: see [this TODO](https://github.com/matrix-org/synapse/blob/4e1374373857f2f7a911a31c50476342d9070681/.ci/scripts/test_old_deps.sh#L35-L39). | |||||
Include the updated `pyproject.toml` and `poetry.lock` files in your commit. | Include the updated `pyproject.toml` and `poetry.lock` files in your commit. | ||||
@@ -196,7 +202,7 @@ poetry remove packagename | |||||
``` | ``` | ||||
ought to do the trick. Alternatively, manually update `pyproject.toml` and | ought to do the trick. Alternatively, manually update `pyproject.toml` and | ||||
`poetry lock --no-update`. Include the updated `pyproject.toml` and poetry.lock` | |||||
`poetry lock --no-update`. Include the updated `pyproject.toml` and `poetry.lock` | |||||
files in your commit. | files in your commit. | ||||
## ...update the version range for an existing dependency? | ## ...update the version range for an existing dependency? | ||||
@@ -240,9 +246,6 @@ poetry export --extras all | |||||
Be wary of bugs in `poetry export` and `pip install -r requirements.txt`. | Be wary of bugs in `poetry export` and `pip install -r requirements.txt`. | ||||
Note: `poetry export` will be made a plugin in Poetry 1.2. Additional config may | |||||
be required. | |||||
## ...build a test wheel? | ## ...build a test wheel? | ||||
I usually use | I usually use | ||||
@@ -260,7 +263,7 @@ doesn't require poetry. (It's what we use in CI too). However, you could try | |||||
## Check the version of poetry with `poetry --version`. | ## Check the version of poetry with `poetry --version`. | ||||
The minimum version of poetry supported by Synapse is 1.2. | |||||
The minimum version of poetry supported by Synapse is 1.3.2. | |||||
It can also be useful to check the version of `poetry-core` in use. If you've | It can also be useful to check the version of `poetry-core` in use. If you've | ||||
installed `poetry` with `pipx`, try `pipx runpip poetry list | grep | installed `poetry` with `pipx`, try `pipx runpip poetry list | grep | ||||
@@ -100,6 +100,13 @@ and vice versa. | |||||
Once all workers are upgraded to v1.76 (or downgraded to v1.75), account data | Once all workers are upgraded to v1.76 (or downgraded to v1.75), account data | ||||
and device replication will resume as normal. | and device replication will resume as normal. | ||||
## Minimum version of Poetry is now 1.3.2 | |||||
The minimum supported version of Poetry is now 1.3.2 (previously 1.2.0, [since | |||||
Synapse 1.67](#upgrading-to-v1670)). If you have used `poetry install` to | |||||
install Synapse from a source checkout, you should upgrade poetry: see its | |||||
[installation instructions](https://python-poetry.org/docs/#installation). | |||||
For all other installation methods, no acction is required. | |||||
# Upgrading to v1.74.0 | # Upgrading to v1.74.0 | ||||