[feature] Handle retractions of domain permission subscription entries (#4261)

# Description

> If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.
>
> If this is a documentation change, please briefly describe what you've changed and why.

This pull request adds logic for nicely handling retractions of entries from domain permission subscriptions.

See docs for how this works but basically retracted entries will either be removed (and possibly picked up by a lower-prio subscription), or orphaned (and then possibly adopted), depending on the config of the domain permission subscription.

closes https://codeberg.org/superseriousbusiness/gotosocial/issues/4101

## Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]`

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [x] I/we have made any necessary changes to documentation.
- [x] I/we have added tests that cover new code.
- [ ] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.

Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4261
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>

docs/admin/domain_permission_subscriptions.md

@@ -21,9 +21,28 @@ The subscription with the higher priority is the one that now creates and manage
 
 If the subscription with the higher priority is removed, then the next time all the subscriptions are fetched, "Less Important List" will create (or take ownership of) the domain allow instead.
 
+## Retractions
+
+Sometimes, an entry that was present on a subscribed block or allow list will be removed later by the curator(s) of that list. When this happens, the removed domain permission entry can be said to have been "retracted".
+
+For example, say your instance subscribes to one block list, and that block list contains an entry for `baddies.example.org`. A corresponding domain block for `baddies.example.org` has therefore been created in your database, with the subscription ID of your block list. In other words, the domain block is in force, and is managed by your block list subscription.
+
+At some point, your instance fetches the list again, and this time it sees that the entry for `baddies.example.org` is no longer present in the list, because it has been removed by the list curator(s) (perhaps the admins turned their policies around, or the instance was shut down, etc). Thus, according to your instance, the block for `baddies.example.org` is now a "retracted" domain permission entry.
+
+If the domain permission subscription is set to "Remove retracted permissions," then the now-retracted domain block will be removed from the database, and will no longer be enforced. In this example, that means your instance will start federating (again) with `baddies.example.org`.
+
+If the domain permission subscription is *not* set to "Remove retracted permissions," then instead of the retracted block being removed from the database, it will be kept in the database but "orphaned" -- ie., it will still be in force, but it will be marked as no longer being managed by the subscription. In this example, that means your instance will keep blocking `baddies.example.org`.
+
+!!!! Note "Retracted permissions and other subscriptions"
+    When a permission is retracted and removed from the database, but an entry for it exists on the list of *another* subscription of a lower priority than the one it was retracted from, then the permission will be recreated as an entry managed by the lower priority list.
+
+    For example, say you subscribe to List1 at priority 255, and List2 at priority 128, and `baddies.example.org` is present on both lists. That means the domain block entry will be managed by List1. If List1 later *retracts* the entry, it will be removed from your database (assuming you have "Remove retracted permissions" set). However, as soon as List2 is checked (usually seconds after List1), then an entry for `baddies.example.org` will be created again, but managed by List2 this time.
+
+    In other words, it is only when an entry is retracted from *every list you subscribe to* that it will truly be removed.
+
 ## Orphan Permissions
 
-Domain permissions (blocks or allows) that are not currently managed by a domain permission subscription are considered "orphan" permissions. This includes permissions that an admin created in the settings panel by hand, or which were imported manually via the import/export page.
+Domain permissions (blocks or allows) that are not currently managed by a domain permission subscription are considered "orphan" permissions. This includes permissions that an admin created in the settings panel by hand, entries which were imported manually via the import/export page, or entries that belonged to a subscription but have since been retracted but not removed.
 
 If you wish, when creating a domain permission subscription, you can set ["adopt orphans"](./settings.md#adopt-orphan-permissions) to true for that subscription. If a domain permission subscription that is set to adopt orphans encounters an orphan permission which is *also present on the list at the subscription's URI*, then it will "adopt" the orphan by setting the orphan's subscription ID to its own ID.
 

docs/admin/settings.md

@@ -123,7 +123,13 @@ If you check this box, then any existing domain permissions will become managed
 1. They don't already have a subscription ID (ie., they're not managed by any domain permission subscription).
 2. They match a domain permission included in the list at the URL of this subscription.
 
-For more information on orphan permissions, please see the separate [domain permission subscriptions](./domain_permission_subscriptions.md) document.
+For more information on orphan permissions, please see the separate [domain permission subscriptions](./domain_permission_subscriptions.md#orphan-permissions) document.
+
+##### Remove Retracted Permissions
+
+This setting controls how retractions are handled by this domain permission subscription: if "Remove retracted permissions" is checked, retracted entries will be removed from the database; if "Remove retracted permissions" is not checked, retracted entries will just be orphaned instead.
+
+For more detail on how retractions work, with examples, please see the separate [domain permission subscriptions](./domain_permission_subscriptions.md#retractions) document.
 
 ##### Create Permissions as Drafts
 

docs/api/swagger.yaml

@@ -1289,6 +1289,11 @@ definitions:
                 format: uint8
                 type: integer
                 x-go-name: Priority
+            remove_retracted:
+                description: If true, then when a list is processed, if the list does *not* contain entries that it *did* contain previously, ie., retracted entries, then domain permissions corresponding to those entries will be removed. If false, they will just be orphaned instead.
+                example: true
+                type: boolean
+                x-go-name: RemoveRetracted
             successfully_fetched_at:
                 description: Time of the most recent successful fetch (ISO 8601 Datetime).
                 example: "2021-07-30T09:20:25+00:00"
@@ -6751,6 +6756,11 @@ paths:
                   in: formData
                   name: adopt_orphans
                   type: boolean
+                - default: true
+                  description: If true, then when a list is processed, if the list does *not* contain entries that it *did* contain previously, ie., retracted entries, then domain permissions corresponding to those entries will be removed. If false, they will just be orphaned instead.
+                  in: formData
+                  name: remove_retracted
+                  type: boolean
                 - description: URI to call in order to fetch the permissions list.
                   in: formData
                   name: uri
@@ -6830,6 +6840,11 @@ paths:
                   in: formData
                   name: adopt_orphans
                   type: boolean
+                - default: true
+                  description: If true, then when a list is processed, if the list does *not* contain entries that it *did* contain previously, ie., retracted entries, then domain permissions corresponding to those entries will be removed. If false, they will just be orphaned instead.
+                  in: formData
+                  name: remove_retracted
+                  type: boolean
                 - description: MIME content type to use when parsing the permissions list. One of "text/plain", "text/csv", and "application/json".
                   in: formData
                   name: content_type