Merge branch 'release-v1.57' into develop

vor 2 Jahren · a468768104
--- a/CHANGES.md
+++ b/CHANGES.md
--- a/changelog.d/11881.feature
+++ b/changelog.d/11881.feature
@@ -1 +0,0 @@
 Send device list changes to application services as specified by [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202), using unstable prefixes. The `msc3202_transaction_extensions` experimental homeserver config option must be enabled and `org.matrix.msc3202: true` must be present in the application service registration file for device list changes to be sent. The "left" field is currently always empty.
--- a/changelog.d/12040.feature
+++ b/changelog.d/12040.feature
@@ -1 +0,0 @@
 Optimise fetching large quantities of missing room state over federation.
--- a/changelog.d/12115.bugfix
+++ b/changelog.d/12115.bugfix
@@ -1 +0,0 @@
 Fix a long-standing bug that updating the server notices user profile (display name/avatar URL) in the configuration would not be applied to pre-existing rooms. Contributed by Jorge Florian.
--- a/changelog.d/12165.misc
+++ b/changelog.d/12165.misc
@@ -1 +0,0 @@
 Remove lingering unstable references to MSC2403 (knocking).
--- a/changelog.d/12191.misc
+++ b/changelog.d/12191.misc
@@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12193.misc
+++ b/changelog.d/12193.misc
@@ -1 +0,0 @@
 Omit sending "offline" presence updates to application services after they are initially configured.
--- a/changelog.d/12209.misc
+++ b/changelog.d/12209.misc
@@ -1 +0,0 @@
 Switch to using a sequence to generate AS transaction IDs. Contributed by Nick Beeper. If running synapse with a dedicated appservice worker, this MUST be stopped before upgrading the main process and database.
--- a/changelog.d/12235.bugfix
+++ b/changelog.d/12235.bugfix
@@ -1 +0,0 @@
 Fix a long-standing bug where events from ignored users were still considered for bundled aggregations.
--- a/changelog.d/12251.feature
+++ b/changelog.d/12251.feature
@@ -1 +0,0 @@
 Offload the `update_client_ip` background job from the main process to the background worker, when using Redis-based replication.
--- a/changelog.d/12252.feature
+++ b/changelog.d/12252.feature
@@ -1 +0,0 @@
 Move `update_client_ip` background job from the main process to the background worker.
--- a/changelog.d/12267.misc
+++ b/changelog.d/12267.misc
@@ -1 +0,0 @@
 Add missing type hints for storage.
--- a/changelog.d/12271.doc
+++ b/changelog.d/12271.doc
@@ -1 +0,0 @@
 Clarify documentation for running SyTest against Synapse, including use of Postgres and worker mode.
--- a/changelog.d/12280.misc
+++ b/changelog.d/12280.misc
@@ -1 +0,0 @@
 Add missing type definitions for scripts in docker folder. Contributed by Jorge Florian.
--- a/changelog.d/12293.removal
+++ b/changelog.d/12293.removal
@@ -1 +0,0 @@
 Remove the unused and unstable `/aggregations` endpoint which was removed from [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675).
--- a/changelog.d/12295.misc
+++ b/changelog.d/12295.misc
@@ -1 +0,0 @@
 Move [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654) support behind an experimental configuration flag.
--- a/changelog.d/12302.feature
+++ b/changelog.d/12302.feature
@@ -1 +0,0 @@
 Add a module callback to react to new 3PID (email address, phone number) associations.
--- a/changelog.d/12310.feature
+++ b/changelog.d/12310.feature
@@ -1 +0,0 @@
 Add a configuration option to remove a specific set of rooms from sync responses.
--- a/changelog.d/12315.doc
+++ b/changelog.d/12315.doc
@@ -1 +0,0 @@
 Document the behaviour of `LoggingTransaction.call_after` and `LoggingTransaction.call_on_exception` methods when transactions are retried.
--- a/changelog.d/12316.misc
+++ b/changelog.d/12316.misc
@@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12317.misc
+++ b/changelog.d/12317.misc
@@ -1 +0,0 @@
 Update docstrings to explain how to decipher live and historic pagination tokens.
--- a/changelog.d/12321.misc
+++ b/changelog.d/12321.misc
@@ -1 +0,0 @@
 Add ground work for speeding up device list updates for users in large numbers of rooms.
--- a/changelog.d/12326.misc
+++ b/changelog.d/12326.misc
@@ -1 +0,0 @@
 Fix typechecker problems exposed by signedjson 1.1.2.
--- a/changelog.d/12327.feature
+++ b/changelog.d/12327.feature
@@ -1 +0,0 @@
 Add a module callback to react to account data changes.
--- a/changelog.d/12329.bugfix
+++ b/changelog.d/12329.bugfix
@@ -1 +0,0 @@
 Fix non-member state events not resolving for historical events when used in [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716) `/batch_send` `state_events_at_start`.
--- a/changelog.d/12330.misc
+++ b/changelog.d/12330.misc
@@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12331.doc
+++ b/changelog.d/12331.doc
@@ -1 +0,0 @@
 Update dead links in `check-newsfragment.sh` to point to the correct documentation URL.
--- a/changelog.d/12332.misc
+++ b/changelog.d/12332.misc
@@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12333.bugfix
+++ b/changelog.d/12333.bugfix
@@ -1 +0,0 @@
 Fix a long-standing bug affecting URL previews that would generate a 500 response instead of a 403 if the previewed URL includes a port that isn't allowed by the relevant blacklist.
--- a/changelog.d/12334.misc
+++ b/changelog.d/12334.misc
@@ -1 +0,0 @@
 Remove the `tox` packaging job: it will be redundant once #11537 lands.
--- a/changelog.d/12335.misc
+++ b/changelog.d/12335.misc
@@ -1 +0,0 @@
 Ignore `.envrc` for `direnv` users.
--- a/changelog.d/12336.misc
+++ b/changelog.d/12336.misc
@@ -1 +0,0 @@
 Remove the (broadly unused, dev-only) dockerfile for pg tests.
--- a/changelog.d/12338.bugfix
+++ b/changelog.d/12338.bugfix
@@ -1 +0,0 @@
 Fix a long-standing bug where events from ignored users were still considered for bundled aggregations.
--- a/changelog.d/12339.doc
+++ b/changelog.d/12339.doc
@@ -1 +0,0 @@
 Upgrade the version of `mdbook` in CI to 0.4.17.
--- a/changelog.d/12341.feature
+++ b/changelog.d/12341.feature
@@ -1 +0,0 @@
 Allow setting user admin status using the module API. Contributed by Famedly.
--- a/changelog.d/12345.doc
+++ b/changelog.d/12345.doc
@@ -1 +0,0 @@
 Updates to the Room DAG concepts development document to clarify that we mark events as outliers because we don't have any state for them.
--- a/changelog.d/12346.misc
+++ b/changelog.d/12346.misc
@@ -1 +0,0 @@
 Remove redundant `get_success` calls in test code.
--- a/changelog.d/12347.misc
+++ b/changelog.d/12347.misc
@@ -1 +0,0 @@
 Add type annotations for `tests/unittest.py`.
--- a/changelog.d/12348.misc
+++ b/changelog.d/12348.misc
@@ -1 +0,0 @@
 Move single-use methods out of `TestCase`.
--- a/changelog.d/12349.misc
+++ b/changelog.d/12349.misc
@@ -1 +0,0 @@
 Remove broken and unused development scripts.
--- a/changelog.d/12350.bugfix
+++ b/changelog.d/12350.bugfix
@@ -1 +0,0 @@
 Default to `private` room visibility rather than `public` when a client does not specify one, according to spec.
--- a/changelog.d/12351.misc
+++ b/changelog.d/12351.misc
@@ -1 +0,0 @@
 Remove broken and unused development scripts.
--- a/changelog.d/12353.misc
+++ b/changelog.d/12353.misc
@@ -1 +0,0 @@
 Convert `Linearizer` tests from `inlineCallbacks` to async.
--- a/changelog.d/12354.misc
+++ b/changelog.d/12354.misc
@@ -1 +0,0 @@
 Update docstrings for `ReadWriteLock` tests.
--- a/changelog.d/12355.misc
+++ b/changelog.d/12355.misc
@@ -1 +0,0 @@
 Remove broken and unused development scripts.
--- a/changelog.d/12357.misc
+++ b/changelog.d/12357.misc
@@ -1 +0,0 @@
 Refactor `Linearizer`, convert methods to async and use an async context manager.
--- a/changelog.d/12358.misc
+++ b/changelog.d/12358.misc
@@ -1 +0,0 @@
 Fix a long-standing bug where `Linearizer`s could get stuck if a cancellation were to happen at the wrong time.
--- a/changelog.d/12364.bugfix
+++ b/changelog.d/12364.bugfix
@@ -1 +0,0 @@
 Fix a spec compliance issue where requests to the `/publicRooms` federation API would specify `limit` as a string.
--- a/changelog.d/12366.misc
+++ b/changelog.d/12366.misc
@@ -1 +0,0 @@
 Make `StreamToken.from_string` and `RoomStreamToken.parse` propagate cancellations instead of replacing them with `SynapseError`s.
--- a/changelog.d/12367.feature
+++ b/changelog.d/12367.feature
@@ -1 +0,0 @@
 Reduce overhead of restarting synchrotrons.
--- a/changelog.d/12369.doc
+++ b/changelog.d/12369.doc
@@ -1 +0,0 @@
 Update the link to Redis pub/sub documentation in the workers documentation..
--- a/changelog.d/12370.feature
+++ b/changelog.d/12370.feature
@@ -1 +0,0 @@
 Update `/messages` to use historic pagination tokens if no `from` query parameter is given.
--- a/changelog.d/12371.misc
+++ b/changelog.d/12371.misc
@@ -1 +0,0 @@
 Add type hints to tests files.
--- a/changelog.d/12372.feature
+++ b/changelog.d/12372.feature
@@ -1 +0,0 @@
 Reduce overhead of restarting synchrotrons.
--- a/changelog.d/12376.misc
+++ b/changelog.d/12376.misc
@@ -1 +0,0 @@
 Allow specifying the Postgres database's port when running unit tests with Postgres.
--- a/changelog.d/12379.misc
+++ b/changelog.d/12379.misc
@@ -1 +0,0 @@
 Remove temporary pin of signedjson<=1.1.1 that was added in Synapse 1.56.0.
--- a/changelog.d/12380.misc
+++ b/changelog.d/12380.misc
@@ -1 +0,0 @@
 Add opentracing spans to calls to external cache.
--- a/changelog.d/12381.misc
+++ b/changelog.d/12381.misc
@@ -1 +0,0 @@
 Lay groundwork for using `poetry` to manage Synapse's dependencies.
--- a/changelog.d/12384.misc
+++ b/changelog.d/12384.misc
@@ -1 +0,0 @@
 Make missing `importlib_metadata` dependency explicit.
--- a/changelog.d/12385.docker
+++ b/changelog.d/12385.docker
@@ -1 +0,0 @@
 Bundle locked versions of dependencies into the Docker image.
--- a/changelog.d/12389.misc
+++ b/changelog.d/12389.misc
@@ -1 +0,0 @@
 Update type annotations for compatiblity with prometheus_client 0.14.
--- a/changelog.d/12390.bugfix
+++ b/changelog.d/12390.bugfix
@@ -1 +0,0 @@
 Fix a bug introduced in Synapse 1.49.0 which caused the `synapse_event_persisted_position` metric to have invalid values.
--- a/changelog.d/12391.feature
+++ b/changelog.d/12391.feature
@@ -1 +0,0 @@
 Add a module API for reading and writing global account data.
--- a/changelog.d/12392.doc
+++ b/changelog.d/12392.doc
@@ -1 +0,0 @@
 Remove documentation for converting a legacy structured logging configuration to the new format.
--- a/changelog.d/12398.misc
+++ b/changelog.d/12398.misc
@@ -1 +0,0 @@
 Remove support for the unstable identifiers specified in [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).
--- a/changelog.d/12400.misc
+++ b/changelog.d/12400.misc
@@ -1 +0,0 @@
 Make missing `importlib_metadata` dependency explicit.
--- a/changelog.d/12402.misc
+++ b/changelog.d/12402.misc
@@ -1 +0,0 @@
 Add missing type hints to configuration classes.
--- a/changelog.d/12403.feature
+++ b/changelog.d/12403.feature
@@ -1 +0,0 @@
 Support the stable `v1` endpoint for `/relations`, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675).
--- a/changelog.d/12404.misc
+++ b/changelog.d/12404.misc
@@ -1 +0,0 @@
 Add files used to build the Docker image used for complement testing into the Synapse repository.
--- a/changelog.d/12405.misc
+++ b/changelog.d/12405.misc
@@ -1 +0,0 @@
 Fix up healthcheck generation for workers docker image.
--- a/changelog.d/12407.misc
+++ b/changelog.d/12407.misc
@@ -1 +0,0 @@
 Run the olddeps CI job using Poetry.
--- a/changelog.d/12408.misc
+++ b/changelog.d/12408.misc
@@ -1 +0,0 @@
 Do not include groups in the sync response when disabled.
--- a/changelog.d/12409.misc
+++ b/changelog.d/12409.misc
@@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12410.bugfix
+++ b/changelog.d/12410.bugfix
@@ -1 +0,0 @@
 Fix a spec compliance issue where requests to the `/publicRooms` federation API would specify `limit` as a string.
--- a/changelog.d/12412.misc
+++ b/changelog.d/12412.misc
@@ -1 +0,0 @@
 Run the CI portdb script in the locked poetry environment.
--- a/changelog.d/12415.misc
+++ b/changelog.d/12415.misc
@@ -1 +0,0 @@
 Improve type hints related to HTTP query parameters.
--- a/changelog.d/12418.misc
+++ b/changelog.d/12418.misc
@@ -1 +0,0 @@
 Run the CI export-data script in the locked poetry environment.
--- a/changelog.d/12420.misc
+++ b/changelog.d/12420.misc
@@ -1 +0,0 @@
 Stop maintaining a list of lint targets.
--- a/changelog.d/12421.misc
+++ b/changelog.d/12421.misc
@@ -1 +0,0 @@
 Make `synapse._scripts` pass type checks.
--- a/changelog.d/12422.misc
+++ b/changelog.d/12422.misc
@@ -1 +0,0 @@
 Make `synapse._scripts` pass type checks.
--- a/changelog.d/12423.misc
+++ b/changelog.d/12423.misc
@@ -1 +0,0 @@
 Add some type hints to datastore.
--- a/changelog.d/12435.misc
+++ b/changelog.d/12435.misc
@@ -1 +0,0 @@
 Enable certificate checking during complement tests.
--- a/changelog.d/12436.feature
+++ b/changelog.d/12436.feature
@@ -1,2 +0,0 @@
 Include bundled aggregations in search results 
 ([MSC3666](https://github.com/matrix-org/matrix-spec-proposals/pull/3666)).
--- a/changelog.d/12439.misc
+++ b/changelog.d/12439.misc
@@ -1 +0,0 @@
 Bundle locked versions of dependencies into the Docker image.
--- a/changelog.d/12444.misc
+++ b/changelog.d/12444.misc
@@ -1 +0,0 @@
 Explicitly specify the `tls` extra for Twisted dependency.
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
 matrix-synapse-py3 (1.57.0~rc1) stable; urgency=medium

  * New synapse release 1.57.0~rc1.

 -- Synapse Packaging team <packages@matrix.org>  Tue, 12 Apr 2022 13:36:25 +0100

 matrix-synapse-py3 (1.56.0) stable; urgency=medium

  * New synapse release 1.56.0.
--- a/docs/changelogs/CHANGES-2019.md
+++ b/docs/changelogs/CHANGES-2019.md
--- a/docs/changelogs/CHANGES-2020.md
+++ b/docs/changelogs/CHANGES-2020.md
--- a/docs/changelogs/CHANGES-2021.md
+++ b/docs/changelogs/CHANGES-2021.md
--- a/docs/changelogs/CHANGES-pre-1.0.md
+++ b/docs/changelogs/CHANGES-pre-1.0.md
--- a/docs/changelogs/README.md
+++ b/docs/changelogs/README.md
@@ -0,0 +1 @@
 This directory contains changelogs for previous years.
--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -105,6 +105,45 @@ without any risk of reusing transaction IDs.
 Deployments which do not use separate worker processes can be upgraded as normal. Similarly,
 deployments where no applciation services are in use can be upgraded as normal.

 <details>
 <summary><b>Recovering from an incorrect upgrade</b></summary>

 If the database schema is upgraded *without* stopping the worker responsible
 for AS traffic, then the following error may be given when attempting to start
 a Synapse worker or master process:

 ```
 **********************************************************************************
 Error during initialisation:

 Postgres sequence 'application_services_txn_id_seq' is inconsistent with associated
 table 'application_services_txns'. This can happen if Synapse has been downgraded and
 then upgraded again, or due to a bad migration.

 To fix this error, shut down Synapse (including any and all workers)
 and run the following SQL:

     SELECT setval('application_services_txn_id_seq', (
         SELECT GREATEST(MAX(txn_id), 0) FROM application_services_txns
     ));

 See docs/postgres.md for more information.

 There may be more information in the logs.
 **********************************************************************************
 ```

 This error may also be seen if Synapse is *downgraded* to an earlier version,
 and then upgraded again to v1.57.0 or later.

 In either case:

 1. Ensure that the worker responsible for AS traffic is stopped.
 2. Run the SQL command given in the error message via `psql`.

 Synapse should then start correctly.
 </details>

 # Upgrading to v1.56.0

 ## Open registration without verification is now disabled by default
@@ -139,15 +178,15 @@ The `synctl` script
 [has been made](https://github.com/matrix-org/synapse/pull/12140) an
 [entry point](https://packaging.python.org/en/latest/specifications/entry-points/)
 and no longer exists at the root of Synapse's source tree. If you wish to use
 `synctl` to manage your homeserver, you should invoke `synctl` directly, e.g. 
 `synctl start` instead of `./synctl start` or `/path/to/synctl start`. 
 `synctl` to manage your homeserver, you should invoke `synctl` directly, e.g.
 `synctl start` instead of `./synctl start` or `/path/to/synctl start`.

 You will need to ensure `synctl` is on your `PATH`.
  - This is automatically the case when using
    [Debian packages](https://packages.matrix.org/debian/) or
    [docker images](https://hub.docker.com/r/matrixdotorg/synapse)
    provided by Matrix.org.
  - When installing from a wheel, sdist, or PyPI, a `synctl` executable is added 
  - When installing from a wheel, sdist, or PyPI, a `synctl` executable is added
    to your Python installation's `bin`. This should be on your `PATH`
    automatically, though you might need to activate a virtual environment
    depending on how you installed Synapse.
@@ -183,8 +222,8 @@ the `/_matrix/client/` path.

 ## Stablisation of MSC3231

 The unstable validity-check endpoint for the 
 [Registration Tokens](https://spec.matrix.org/v1.2/client-server-api/#get_matrixclientv1registermloginregistration_tokenvalidity) 
 The unstable validity-check endpoint for the
 [Registration Tokens](https://spec.matrix.org/v1.2/client-server-api/#get_matrixclientv1registermloginregistration_tokenvalidity)
 feature has been stabilised and moved from:

 `/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity`
@@ -198,9 +237,9 @@ Please update any relevant reverse proxy or firewall configurations appropriatel
 ## Time-based cache expiry is now enabled by default

 Formerly, entries in the cache were not evicted regardless of whether they were accessed after storing.
 This behavior has now changed. By default entries in the cache are now evicted after 30m of not being accessed. 
 To change the default behavior, go to the `caches` section of the config and change the `expire_caches` and 
 `cache_entry_ttl` flags as necessary. Please note that these flags replace the `expiry_time` flag in the config.  
 This behavior has now changed. By default entries in the cache are now evicted after 30m of not being accessed.
 To change the default behavior, go to the `caches` section of the config and change the `expire_caches` and
 `cache_entry_ttl` flags as necessary. Please note that these flags replace the `expiry_time` flag in the config.
 The `expiry_time` flag will still continue to work, but it has been deprecated and will be removed in the future.

 ## Deprecation of `capability` `org.matrix.msc3283.*`
--- a/synapse/init.py
+++ b/synapse/init.py
@@ -68,7 +68,7 @@ try:
 except ImportError:
    pass

 __version__ = "1.56.0"
 __version__ = "1.57.0rc1"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when