mirrors/gotosocial
1
0
Fork 0
mirror of https://github.com/superseriousbusiness/gotosocial.git synced 2025-10-28 06:32:25 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

[feature] Handle retractions of domain permission subscription entries (#4261)

Browse source 
# Description

> If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.
>
> If this is a documentation change, please briefly describe what you've changed and why.

This pull request adds logic for nicely handling retractions of entries from domain permission subscriptions.

See docs for how this works but basically retracted entries will either be removed (and possibly picked up by a lower-prio subscription), or orphaned (and then possibly adopted), depending on the config of the domain permission subscription.

closes https://codeberg.org/superseriousbusiness/gotosocial/issues/4101

## Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]`

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [x] I/we have made any necessary changes to documentation.
- [x] I/we have added tests that cover new code.
- [ ] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.

Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4261
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Co-committed-by: tobi <tobi.smethurst@protonmail.com>
This commit is contained in:
tobi 2025-06-15 12:36:51 +02:00 • committed by tobi
commit a9b2d4ee35
19 changed files with 662 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

21
docs/admin/domain_permission_subscriptions.md
View file

@ -21,9 +21,28 @@ The subscription with the higher priority is the one that now creates and manage
If the subscription with the higher priority is removed, then the next time all the subscriptions are fetched, "Less Important List" will create (or take ownership of) the domain allow instead.
## Retractions
Sometimes, an entry that was present on a subscribed block or allow list will be removed later by the curator(s) of that list. When this happens, the removed domain permission entry can be said to have been "retracted".
For example, say your instance subscribes to one block list, and that block list contains an entry for `baddies.example.org`. A corresponding domain block for `baddies.example.org` has therefore been created in your database, with the subscription ID of your block list. In other words, the domain block is in force, and is managed by your block list subscription.
At some point, your instance fetches the list again, and this time it sees that the entry for `baddies.example.org` is no longer present in the list, because it has been removed by the list curator(s) (perhaps the admins turned their policies around, or the instance was shut down, etc). Thus, according to your instance, the block for `baddies.example.org` is now a "retracted" domain permission entry.
If the domain permission subscription is set to "Remove retracted permissions," then the now-retracted domain block will be removed from the database, and will no longer be enforced. In this example, that means your instance will start federating (again) with `baddies.example.org`.
If the domain permission subscription is *not* set to "Remove retracted permissions," then instead of the retracted block being removed from the database, it will be kept in the database but "orphaned" -- ie., it will still be in force, but it will be marked as no longer being managed by the subscription. In this example, that means your instance will keep blocking `baddies.example.org`.
!!!! Note "Retracted permissions and other subscriptions"
When a permission is retracted and removed from the database, but an entry for it exists on the list of *another* subscription of a lower priority than the one it was retracted from, then the permission will be recreated as an entry managed by the lower priority list.
For example, say you subscribe to List1 at priority 255, and List2 at priority 128, and `baddies.example.org` is present on both lists. That means the domain block entry will be managed by List1. If List1 later *retracts* the entry, it will be removed from your database (assuming you have "Remove retracted permissions" set). However, as soon as List2 is checked (usually seconds after List1), then an entry for `baddies.example.org` will be created again, but managed by List2 this time.
In other words, it is only when an entry is retracted from *every list you subscribe to* that it will truly be removed.
## Orphan Permissions
Domain permissions (blocks or allows) that are not currently managed by a domain permission subscription are considered "orphan" permissions. This includes permissions that an admin created in the settings panel by hand, or which were imported manually via the import/export page.
Domain permissions (blocks or allows) that are not currently managed by a domain permission subscription are considered "orphan" permissions. This includes permissions that an admin created in the settings panel by hand, entries which were imported manually via the import/export page, or entries that belonged to a subscription but have since been retracted but not removed.
If you wish, when creating a domain permission subscription, you can set ["adopt orphans"](./settings.md#adopt-orphan-permissions) to true for that subscription. If a domain permission subscription that is set to adopt orphans encounters an orphan permission which is *also present on the list at the subscription's URI*, then it will "adopt" the orphan by setting the orphan's subscription ID to its own ID.

8
docs/admin/settings.md
View file

@ -123,7 +123,13 @@ If you check this box, then any existing domain permissions will become managed
1. They don't already have a subscription ID (ie., they're not managed by any domain permission subscription).
2. They match a domain permission included in the list at the URL of this subscription.
For more information on orphan permissions, please see the separate [domain permission subscriptions](./domain_permission_subscriptions.md) document.
For more information on orphan permissions, please see the separate [domain permission subscriptions](./domain_permission_subscriptions.md#orphan-permissions) document.
##### Remove Retracted Permissions
This setting controls how retractions are handled by this domain permission subscription: if "Remove retracted permissions" is checked, retracted entries will be removed from the database; if "Remove retracted permissions" is not checked, retracted entries will just be orphaned instead.
For more detail on how retractions work, with examples, please see the separate [domain permission subscriptions](./domain_permission_subscriptions.md#retractions) document.
##### Create Permissions as Drafts

15
docs/api/swagger.yaml
View file

@ -1289,6 +1289,11 @@ definitions:
format: uint8
type: integer
x-go-name: Priority
remove_retracted:
description: If true, then when a list is processed, if the list does *not* contain entries that it *did* contain previously, ie., retracted entries, then domain permissions corresponding to those entries will be removed. If false, they will just be orphaned instead.
example: true
type: boolean
x-go-name: RemoveRetracted
successfully_fetched_at:
description: Time of the most recent successful fetch (ISO 8601 Datetime).
example: "2021-07-30T09:20:25+00:00"
@ -6751,6 +6756,11 @@ paths:
in: formData
name: adopt_orphans
type: boolean
- default: true
description: If true, then when a list is processed, if the list does *not* contain entries that it *did* contain previously, ie., retracted entries, then domain permissions corresponding to those entries will be removed. If false, they will just be orphaned instead.
in: formData
name: remove_retracted
type: boolean
- description: URI to call in order to fetch the permissions list.
in: formData
name: uri
@ -6830,6 +6840,11 @@ paths:
in: formData
name: adopt_orphans
type: boolean
- default: true
description: If true, then when a list is processed, if the list does *not* contain entries that it *did* contain previously, ie., retracted entries, then domain permissions corresponding to those entries will be removed. If false, they will just be orphaned instead.
in: formData
name: remove_retracted
type: boolean
- description: MIME content type to use when parsing the permissions list. One of "text/plain", "text/csv", and "application/json".
in: formData
name: content_type

11
internal/api/client/admin/domainpermissionsubscriptioncreate.go
View file

@ -94,6 +94,15 @@ import (
// type: boolean
// default: false
// -
// name: remove_retracted
// in: formData
// description: >-
// If true, then when a list is processed, if the list does *not* contain entries that
// it *did* contain previously, ie., retracted entries, then domain permissions
// corresponding to those entries will be removed. If false, they will just be orphaned instead.
// type: boolean
// default: true
// -
// name: uri
// required: true
// in: formData
@ -234,6 +243,8 @@ func (m *Module) DomainPermissionSubscriptionPOSTHandler(c *gin.Context) {
contentType,
permType,
asDraft,
form.AdoptOrphans,
form.RemoveRetracted,
util.PtrOrZero(form.FetchUsername), // Optional.
util.PtrOrZero(form.FetchPassword), // Optional.
)

11
internal/api/client/admin/domainpermissionsubscriptionupdate.go
View file

@ -97,6 +97,15 @@ import (
// type: boolean
// default: false
// -
// name: remove_retracted
// in: formData
// description: >-
// If true, then when a list is processed, if the list does *not* contain entries that
// it *did* contain previously, ie., retracted entries, then domain permissions
// corresponding to those entries will be removed. If false, they will just be orphaned instead.
// type: boolean
// default: true
// -
// name: content_type
// in: formData
// description: >-
@ -227,6 +236,7 @@ func (m *Module) DomainPermissionSubscriptionPATCHHandler(c *gin.Context) {
contentType == nil &&
form.AsDraft == nil &&
form.AdoptOrphans == nil &&
form.RemoveRetracted == nil &&
form.FetchUsername == nil &&
form.FetchPassword == nil {
const errText = "no updateable fields set on request"
@ -244,6 +254,7 @@ func (m *Module) DomainPermissionSubscriptionPATCHHandler(c *gin.Context) {
contentType,
form.AsDraft,
form.AdoptOrphans,
form.RemoveRetracted,
form.FetchUsername,
form.FetchPassword,
)

7
internal/api/model/domain.go
View file

@ -131,6 +131,9 @@ type DomainPermissionSubscription struct {
// If true, this domain permission subscription will "adopt" domain permissions which already exist on the instance, and which meet the following conditions: 1) they have no subscription ID (ie., they're "orphaned") and 2) they are present in the subscribed list. Such orphaned domain permissions will be given this subscription's subscription ID value.
// example: false
AdoptOrphans bool `json:"adopt_orphans"`
// If true, then when a list is processed, if the list does *not* contain entries that it *did* contain previously, ie., retracted entries, then domain permissions corresponding to those entries will be removed. If false, they will just be orphaned instead.
// example: true
RemoveRetracted bool `json:"remove_retracted"`
// Time at which the subscription was created (ISO 8601 Datetime).
// example: 2021-07-30T09:20:25+00:00
CreatedAt string `json:"created_at"`
@ -198,6 +201,10 @@ type DomainPermissionSubscriptionRequest struct {
// in the subscribed list. Such orphaned domain permissions will be given this
// subscription's subscription ID value and be managed by this subscription.
AdoptOrphans *bool `form:"adopt_orphans" json:"adopt_orphans"`
// If true, then when a list is processed, if the list does *not* contain entries that
// it *did* contain previously, ie., retracted entries, then domain permissions
// corresponding to those entries will be removed. If false, they will just be orphaned instead.
RemoveRetracted *bool `form:"remove_retracted" json:"remove_retracted"`
// (Optional) username to set for basic auth when doing a fetch of URI.
// example: admin123
FetchUsername *string `form:"fetch_username" json:"fetch_username"`

28
internal/db/bundb/domain.go
View file

@ -97,6 +97,20 @@ func (d *domainDB) GetDomainAllows(ctx context.Context) ([]*gtsmodel.DomainAllow
return allows, nil
}
func (d *domainDB) GetDomainAllowsBySubscriptionID(ctx context.Context, subscriptionID string) ([]*gtsmodel.DomainAllow, error) {
allows := []*gtsmodel.DomainAllow{}
if err := d.db.
NewSelect().
Model(&allows).
Where("? = ?", bun.Ident("subscription_id"), subscriptionID).
Scan(ctx); err != nil {
return nil, err
}
return allows, nil
}
func (d *domainDB) GetDomainAllowByID(ctx context.Context, id string) (*gtsmodel.DomainAllow, error) {
var allow gtsmodel.DomainAllow
@ -225,6 +239,20 @@ func (d *domainDB) GetDomainBlocks(ctx context.Context) ([]*gtsmodel.DomainBlock
return blocks, nil
}
func (d *domainDB) GetDomainBlocksBySubscriptionID(ctx context.Context, subscriptionID string) ([]*gtsmodel.DomainBlock, error) {
blocks := []*gtsmodel.DomainBlock{}
if err := d.db.
NewSelect().
Model(&blocks).
Where("? = ?", bun.Ident("subscription_id"), subscriptionID).
Scan(ctx); err != nil {
return nil, err
}
return blocks, nil
}
func (d *domainDB) GetDomainBlockByID(ctx context.Context, id string) (*gtsmodel.DomainBlock, error) {
var block gtsmodel.DomainBlock

76
internal/db/bundb/migrations/20250603125942_domain_perm_sub_retractions.go Normal file
View file

@ -0,0 +1,76 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package migrations
import (
"context"
"fmt"
"reflect"
"code.superseriousbusiness.org/gotosocial/internal/db/bundb/migrations/20250603125942_domain_perm_sub_retractions"
"code.superseriousbusiness.org/gotosocial/internal/log"
"github.com/uptrace/bun"
)
func init() {
up := func(ctx context.Context, db *bun.DB) error {
return db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
// Bail if "remove_retracted"
// column already created.
if exists, err := doesColumnExist(
ctx,
tx,
"domain_permission_subscriptions",
"remove_retracted",
); err != nil {
return err
} else if exists {
return nil
}
// Derive column definition.
var permSub *gtsmodel.DomainPermissionSubscription
permSubType := reflect.TypeOf(permSub)
colDef, err := getBunColumnDef(tx, permSubType, "RemoveRetracted")
if err != nil {
return fmt.Errorf("error making column def: %w", err)
}
log.Info(ctx, "adding domain_permission_subscriptions.remove_retracted column...")
if _, err := tx.
NewAddColumn().
Model(permSub).
ColumnExpr(colDef).
Exec(ctx); err != nil {
return fmt.Errorf("error adding column: %w", err)
}
return nil
})
}
down := func(ctx context.Context, db *bun.DB) error {
return db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
return nil
})
}
if err := Migrations.Register(up, down); err != nil {
panic(err)
}
}

42
internal/db/bundb/migrations/20250603125942_domain_perm_sub_retractions/domainpermissionsubscription.go Normal file
View file

@ -0,0 +1,42 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package gtsmodel
import "time"
type DomainPermissionSubscription struct {
ID string `bun:"type:CHAR(26),pk,nullzero,notnull,unique"`
Priority uint8 `bun:""`
Title string `bun:",nullzero,unique"`
PermissionType uint8 `bun:",nullzero,notnull"`
AsDraft *bool `bun:",nullzero,notnull,default:true"`
AdoptOrphans *bool `bun:",nullzero,notnull,default:false"`
CreatedByAccountID string `bun:"type:CHAR(26),nullzero,notnull"`
URI string `bun:",nullzero,notnull,unique"`
ContentType int16 `bun:",nullzero,notnull"`
FetchUsername string `bun:",nullzero"`
FetchPassword string `bun:",nullzero"`
FetchedAt time.Time `bun:"type:timestamptz,nullzero"`
SuccessfullyFetchedAt time.Time `bun:"type:timestamptz,nullzero"`
LastModified time.Time `bun:"type:timestamptz,nullzero"`
ETag string `bun:"etag,nullzero"`
Error string `bun:",nullzero"`
// This is the field added by this migration.
RemoveRetracted *bool `bun:",nullzero,notnull,default:true"`
}

6
internal/db/domain.go
View file

@ -43,6 +43,9 @@ type Domain interface {
// GetDomainAllows returns all instance-level domain allows currently enforced by this instance.
GetDomainAllows(ctx context.Context) ([]*gtsmodel.DomainAllow, error)
// GetDomainAllowsBySubscriptionID gets all domain allows that have the given subscription ID.
GetDomainAllowsBySubscriptionID(ctx context.Context, subscriptionID string) ([]*gtsmodel.DomainAllow, error)
// UpdateDomainAllow updates the given domain allow, setting the provided columns (empty for all).
UpdateDomainAllow(ctx context.Context, allow *gtsmodel.DomainAllow, columns ...string) error
@ -58,6 +61,9 @@ type Domain interface {
// GetDomainBlockByID returns one instance-level domain block with the given id, if it exists.
GetDomainBlockByID(ctx context.Context, id string) (*gtsmodel.DomainBlock, error)
// GetDomainBlocksBySubscriptionID gets all domain blocks that have the given subscription ID.
GetDomainBlocksBySubscriptionID(ctx context.Context, subscriptionID string) ([]*gtsmodel.DomainBlock, error)
// GetDomainBlocks returns all instance-level domain blocks currently enforced by this instance.
GetDomainBlocks(ctx context.Context) ([]*gtsmodel.DomainBlock, error)

5
internal/gtsmodel/adminaction.go
View file

@ -74,6 +74,7 @@ const (
AdminActionSuspend
AdminActionUnsuspend
AdminActionExpireKeys
AdminActionUnallow
)
func (t AdminActionType) String() string {
@ -92,6 +93,8 @@ func (t AdminActionType) String() string {
return "unsuspend"
case AdminActionExpireKeys:
return "expire-keys"
case AdminActionUnallow:
return "unallow"
default:
return "unknown"
}
@ -113,6 +116,8 @@ func ParseAdminActionType(in string) AdminActionType {
return AdminActionUnsuspend
case "expire-keys":
return AdminActionExpireKeys
case "unallow":
return AdminActionUnallow
default:
return AdminActionUnknown
}

90
internal/gtsmodel/domainpermissionsubscription.go
View file

@ -20,23 +20,79 @@ package gtsmodel
import "time"
type DomainPermissionSubscription struct {
ID string `bun:"type:CHAR(26),pk,nullzero,notnull,unique"` // ID of this item in the database.
Priority uint8 `bun:""` // Priority of this subscription compared to others of the same permission type. 0-255 (higher = higher priority).
Title string `bun:",nullzero,unique"` // Moderator-set title for this list.
PermissionType DomainPermissionType `bun:",nullzero,notnull"` // Permission type of the subscription.
AsDraft *bool `bun:",nullzero,notnull,default:true"` // Create domain permission entries resulting from this subscription as drafts.
AdoptOrphans *bool `bun:",nullzero,notnull,default:false"` // Adopt orphaned domain permissions present in this subscription's entries.
CreatedByAccountID string `bun:"type:CHAR(26),nullzero,notnull"` // Account ID of the creator of this subscription.
CreatedByAccount *Account `bun:"-"` // Account corresponding to createdByAccountID.
URI string `bun:",nullzero,notnull,unique"` // URI of the domain permission list.
ContentType DomainPermSubContentType `bun:",nullzero,notnull"` // Content type to expect from the URI.
FetchUsername string `bun:",nullzero"` // Username to send when doing a GET of URI using basic auth.
FetchPassword string `bun:",nullzero"` // Password to send when doing a GET of URI using basic auth.
FetchedAt time.Time `bun:"type:timestamptz,nullzero"` // Time when fetch of URI was last attempted.
SuccessfullyFetchedAt time.Time `bun:"type:timestamptz,nullzero"` // Time when the domain permission list was last *successfuly* fetched, to be transmitted as If-Modified-Since header.
LastModified time.Time `bun:"type:timestamptz,nullzero"` // "Last-Modified" time received from the server (if any) on last successful fetch. Used for HTTP request caching.
ETag string `bun:"etag,nullzero"` // "ETag" header last received from the server (if any) on last successful fetch. Used for HTTP request caching.
Error string `bun:",nullzero"` // If latest fetch attempt errored, this field stores the error message. Cleared on latest successful fetch.
// ID of this item in the database.
ID string `bun:"type:CHAR(26),pk,nullzero,notnull,unique"`
// Priority of this subscription compared
// to others of the same permission type.
// 0-255 (higher = higher priority).
Priority uint8 `bun:""`
// Moderator-set title for this list.
Title string `bun:",nullzero,unique"`
// Permission type of the subscription.
PermissionType DomainPermissionType `bun:",nullzero,notnull"`
// Create domain permission entries
// resulting from this subscription as drafts.
AsDraft *bool `bun:",nullzero,notnull,default:true"`
// Adopt orphaned domain permissions
// present in this subscription's entries.
AdoptOrphans *bool `bun:",nullzero,notnull,default:false"`
// Account ID of the creator of this subscription.
CreatedByAccountID string `bun:"type:CHAR(26),nullzero,notnull"`
// Account corresponding to createdByAccountID.
CreatedByAccount *Account `bun:"-"`
// URI of the domain permission list.
URI string `bun:",nullzero,notnull,unique"`
// Content type to expect from the URI.
ContentType DomainPermSubContentType `bun:",nullzero,notnull"`
// Username to send when doing
// a GET of URI using basic auth.
FetchUsername string `bun:",nullzero"`
// Password to send when doing
// a GET of URI using basic auth.
FetchPassword string `bun:",nullzero"`
// Time when fetch of URI was last attempted.
FetchedAt time.Time `bun:"type:timestamptz,nullzero"`
// Time when the domain permission list
// was last *successfuly* fetched, to be
// transmitted as If-Modified-Since header.
SuccessfullyFetchedAt time.Time `bun:"type:timestamptz,nullzero"`
// "Last-Modified" time received from the
// server (if any) on last successful fetch.
// Used for HTTP request caching.
LastModified time.Time `bun:"type:timestamptz,nullzero"`
// "ETag" header last received from the
// server (if any) on last successful fetch.
// Used for HTTP request caching.
ETag string `bun:"etag,nullzero"`
// If latest fetch attempt errored,
// this field stores the error message.
// Cleared on latest successful fetch.
Error string `bun:",nullzero"`
// If true, then when a list is processed, if the
// list does *not* contain entries that it *did*
// contain previously, ie., retracted entries,
// then domain permissions corresponding to those
// entries will be removed.
//
// If false, they will just be orphaned instead.
RemoveRetracted *bool `bun:",nullzero,notnull,default:true"`
}
type DomainPermSubContentType enumType

2
internal/processing/admin/domainallow.go
View file

@ -176,7 +176,7 @@ func (p *Processor) deleteDomainAllow(
ID: id.NewULID(),
TargetCategory: gtsmodel.AdminActionCategoryDomain,
TargetID: domainAllow.Domain,
Type: gtsmodel.AdminActionUnsuspend,
Type: gtsmodel.AdminActionUnallow,
AccountID: adminAcct.ID,
}

13
internal/processing/admin/domainpermissionsubscription.go
View file

@ -142,6 +142,8 @@ func (p *Processor) DomainPermissionSubscriptionCreate(
contentType gtsmodel.DomainPermSubContentType,
permType gtsmodel.DomainPermissionType,
asDraft bool,
adoptOrphans *bool,
removeRetracted *bool,
fetchUsername string,
fetchPassword string,
) (*apimodel.DomainPermissionSubscription, gtserror.WithCode) {
@ -151,12 +153,14 @@ func (p *Processor) DomainPermissionSubscriptionCreate(
Title: title,
PermissionType: permType,
AsDraft: &asDraft,
AdoptOrphans: adoptOrphans,
CreatedByAccountID: acct.ID,
CreatedByAccount: acct,
URI: uri,
ContentType: contentType,
FetchUsername: fetchUsername,
FetchPassword: fetchPassword,
RemoveRetracted: removeRetracted,
}
err := p.state.DB.PutDomainPermissionSubscription(ctx, permSub)
@ -184,6 +188,7 @@ func (p *Processor) DomainPermissionSubscriptionUpdate(
contentType *gtsmodel.DomainPermSubContentType,
asDraft *bool,
adoptOrphans *bool,
removeRetracted *bool,
fetchUsername *string,
fetchPassword *string,
) (*apimodel.DomainPermissionSubscription, gtserror.WithCode) {
@ -230,6 +235,11 @@ func (p *Processor) DomainPermissionSubscriptionUpdate(
columns = append(columns, "adopt_orphans")
}
if removeRetracted != nil {
permSub.RemoveRetracted = removeRetracted
columns = append(columns, "remove_retracted")
}
if fetchPassword != nil {
permSub.FetchPassword = *fetchPassword
columns = append(columns, "fetch_password")
@ -342,12 +352,13 @@ func (p *Processor) DomainPermissionSubscriptionTest(
// Call the permSub.URI and parse a list of perms from it.
// Any error returned here is a "real" one, not an error
// from fetching / parsing the list.
createdPerms, err := p.subscriptions.ProcessDomainPermissionSubscription(
createdPerms, _, err := p.subscriptions.ProcessDomainPermissionSubscription(
ctx,
permSub,
tsport,
higherPrios,
true, // Dry run.
true, // Skip caching.
)
if err != nil {
err := gtserror.Newf("error doing dry-run: %w", err)

216
internal/subscriptions/domainperms.go
View file

@ -166,17 +166,19 @@ func (s *Subscriptions) ProcessDomainPermissionSubscriptions(
return
}
var skipCache bool
for i, permSub := range permSubs {
// Higher priority permission subs = everything
// above this permission sub in the slice.
higherPrios := permSubs[:i]
_, err := s.ProcessDomainPermissionSubscription(
_, retracted, err := s.ProcessDomainPermissionSubscription(
ctx,
permSub,
tsport,
higherPrios,
false, // Not dry. Wet, if you will.
false, // Not dry. Wet, if you will.
skipCache, // Skip cache if necessary.
)
if err != nil {
// Real db error.
@ -187,6 +189,16 @@ func (s *Subscriptions) ProcessDomainPermissionSubscriptions(
return
}
// If any retractions have been done, skip caching
// when doing subsequent fetches. This makes it so
// that if an entry was present in a higher-priority
// list and a lower-priority list, but was retracted
// from the higher-priority list, it will be created
// and managed by the lower-priority list instead.
if retracted && !skipCache {
skipCache = true
}
// Update this perm sub.
err = s.state.DB.UpdateDomainPermissionSubscription(ctx, permSub)
if err != nil {
@ -203,7 +215,10 @@ func (s *Subscriptions) ProcessDomainPermissionSubscriptions(
// entry in the database, or ignoring it if it's excluded or already
// covered by a higher-priority subscription.
//
// On success, the slice of discovered DomainPermissions will be returned.
// On success, the slice of discovered DomainPermissions will be returned,
// including a boolean to indicate whether or not any retractions have been
// performed since the list was last checked (if ever).
//
// In case of parsing error, or error on the remote side, permSub.Error
// will be updated with the calling/parsing error, and `nil, nil` will be
// returned. In case of an actual db error, `nil, err` will be returned and
@ -215,6 +230,10 @@ func (s *Subscriptions) ProcessDomainPermissionSubscriptions(
// If dry == true, then the URI will still be called, and permissions
// will be parsed, but they will not actually be created.
//
// If skipCache == true, then conditional HTTP request headers will not be
// sent, and so cached values for the domain permission subscription list
// will not be used (ie., the list will always be fetched "fresh").
//
// Note that while this function modifies fields on the given permSub,
// it's up to the caller to update it in the database (if desired).
func (s *Subscriptions) ProcessDomainPermissionSubscription(
@ -223,7 +242,8 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
tsport transport.Transport,
higherPrios []*gtsmodel.DomainPermissionSubscription,
dry bool,
) ([]gtsmodel.DomainPermission, error) {
skipCache bool,
) ([]gtsmodel.DomainPermission, bool, error) {
l := log.
WithContext(ctx).
WithFields(kv.Fields{
@ -235,10 +255,10 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
// going to attempt this now.
permSub.FetchedAt = time.Now()
// Call the URI, and only skip
// cache if we're doing a dry run.
// Call the URI, skipping conditional requests
// (caching) if we've been told to do so.
resp, err := tsport.DereferenceDomainPermissions(
ctx, permSub, dry,
ctx, permSub, skipCache,
)
if err != nil {
// Couldn't get this one,
@ -246,7 +266,7 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
errStr := err.Error()
l.Warnf("couldn't dereference permSubURI: %+v", err)
permSub.Error = errStr
return nil, nil
return nil, false, nil
}
// If the permissions at URI weren't modified
@ -257,7 +277,7 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
permSub.ETag = resp.ETag
permSub.LastModified = resp.LastModified
permSub.SuccessfullyFetchedAt = permSub.FetchedAt
return nil, nil
return nil, false, nil
}
// At this point we know we got a 200 OK
@ -289,7 +309,7 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
errStr := err.Error()
l.Warnf("couldn't parse results: %+v", err)
permSub.Error = errStr
return nil, nil
return nil, false, nil
}
if len(wantedPerms) == 0 {
@ -299,7 +319,7 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
const errStr = "fetch successful but parsed zero usable results"
l.Warn(errStr)
permSub.Error = errStr
return nil, nil
return nil, false, nil
}
// This can now be considered a successful fetch.
@ -325,7 +345,7 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
)
if err != nil {
// Proper db error.
return nil, err
return nil, false, err
}
if !created {
@ -335,7 +355,30 @@ func (s *Subscriptions) ProcessDomainPermissionSubscription(
createdPerms = append(createdPerms, wantedPerm)
}
return createdPerms, nil
if dry {
// Don't do any further
// processing with a dry run.
return createdPerms, false, nil
}
// Process any retractions since
// the last time list was checked.
//
// Being unable to do retractions
// isn't the end of the world as it
// can be tried again next time the
// list is updated, so if there was
// an error just warn it.
retracted, err := s.processRetractions(
ctx, l,
permSub,
wantedPerms,
)
if err != nil {
l.Warnf("error doing retractions: %+v", err)
}
return createdPerms, retracted, nil
}
// processDomainPermission processes one wanted domain
@ -895,3 +938,150 @@ func (s *Subscriptions) adoptPerm(
return err
}
func (s *Subscriptions) processRetractions(
ctx context.Context,
l log.Entry,
permSub *gtsmodel.DomainPermissionSubscription,
wantedPerms []gtsmodel.DomainPermission,
) (bool, error) {
var (
isBlocks = permSub.PermissionType == gtsmodel.DomainPermissionBlock
removeRetracted = *permSub.RemoveRetracted
// True if at least one
// retraction has occurred.
retracted bool
)
// Gather existing perms into an interface type.
existingPerms := []gtsmodel.DomainPermission{}
if isBlocks {
existingBlocks, err := s.state.DB.GetDomainBlocksBySubscriptionID(ctx, permSub.ID)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
// Proper db error.
err := gtserror.Newf("db error getting existing blocks owned by perm sub: %w", err)
return retracted, err
}
for _, existingBlock := range existingBlocks {
existingPerms = append(existingPerms, existingBlock)
}
} else {
existingAllows, err := s.state.DB.GetDomainAllowsBySubscriptionID(ctx, permSub.ID)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
// Proper db error.
err := gtserror.Newf("db error getting existing allows owned by perm sub: %w", err)
return retracted, err
}
for _, existingAllow := range existingAllows {
existingPerms = append(existingPerms, existingAllow)
}
}
// For each existing permission, check if
// it's included in the list of wanted perms
// that's just been freshly fetched + created.
//
// If it's not, we should consider it retracted
// and handle retraction effects appropriately.
for _, existingPerm := range existingPerms {
if slices.ContainsFunc(
wantedPerms,
func(wantedPerm gtsmodel.DomainPermission) bool {
return existingPerm.GetDomain() == wantedPerm.GetDomain()
},
) {
// This permission from the
// database exists in wanted
// perms, so it's not been
// retracted, leave it alone.
continue
}
// This perm exists in the database but
// not in wanted perms, so it has been
// retracted, check what we need to do.
domain := existingPerm.GetDomain()
l.WithField("domain", domain).Info("handling retraction")
var (
dbF func() error
action *gtsmodel.AdminAction
actionF admin.ActionF
)
switch {
// Remove this block.
case isBlocks && removeRetracted:
dbF = func() error { return s.state.DB.DeleteDomainBlock(ctx, domain) }
action = &gtsmodel.AdminAction{
ID: id.NewULID(),
TargetCategory: gtsmodel.AdminActionCategoryDomain,
TargetID: domain,
Type: gtsmodel.AdminActionUnsuspend,
AccountID: permSub.CreatedByAccountID,
}
actionF = s.state.AdminActions.DomainUnblockF(
action.ID,
existingPerm.(*gtsmodel.DomainBlock),
)
// Remove this allow.
case !isBlocks && removeRetracted:
dbF = func() error { return s.state.DB.DeleteDomainAllow(ctx, domain) }
action = &gtsmodel.AdminAction{
ID: id.NewULID(),
TargetCategory: gtsmodel.AdminActionCategoryDomain,
TargetID: domain,
Type: gtsmodel.AdminActionUnallow,
AccountID: permSub.CreatedByAccountID,
}
actionF = s.state.AdminActions.DomainUnallowF(
action.ID,
existingPerm.(*gtsmodel.DomainAllow),
)
// Orphan this block.
case isBlocks:
block := existingPerm.(*gtsmodel.DomainBlock)
block.SubscriptionID = ""
dbF = func() error { return s.state.DB.UpdateDomainBlock(ctx, block, "subscription_id") }
// Orphan this allow.
case !isBlocks:
allow := existingPerm.(*gtsmodel.DomainAllow)
allow.SubscriptionID = ""
dbF = func() error { return s.state.DB.UpdateDomainAllow(ctx, allow, "subscription_id") }
}
// Run the retraction db
// func to either delete
// or update the perm.
if err := dbF(); err != nil {
return retracted, err
}
// Mark that at least one
// retraction has been done.
if !retracted {
retracted = true
}
if action == nil {
// No side effects;
// nothing else to do.
continue
}
// Run the side effects.
if err := s.state.AdminActions.Run(ctx, action, actionF); err != nil {
err := gtserror.Newf("error running side effects: %w", err)
return retracted, err
}
// TODO: Remove draft(s) as well?
}
return retracted, nil
}

132
internal/subscriptions/subscriptions_test.go
View file

@ -949,6 +949,138 @@ func (suite *SubscriptionsTestSuite) TestDomainAllowsAndBlocks() {
suite.Equal(existingAllow.SubscriptionID, testAllowSubscription.ID)
}
func (suite *SubscriptionsTestSuite) TestRemoveRetraction() {
var (
ctx = suite.T().Context()
testStructs = testrig.SetupTestStructs(rMediaPath, rTemplatePath)
testAccount = suite.testAccounts["admin_account"]
subscriptions = subscriptions.New(
testStructs.State,
testStructs.TransportController,
testStructs.TypeConverter,
)
// A subscription for a plain list of
// baddies, which removes retracted entries.
testSubscription = &gtsmodel.DomainPermissionSubscription{
ID: "01JGE681TQSBPAV59GZXPKE62H",
Priority: 255,
Title: "whatever!",
PermissionType: gtsmodel.DomainPermissionBlock,
AsDraft: util.Ptr(false),
AdoptOrphans: util.Ptr(false),
CreatedByAccountID: testAccount.ID,
CreatedByAccount: testAccount,
URI: "https://lists.example.org/baddies.txt",
ContentType: gtsmodel.DomainPermSubContentTypePlain,
RemoveRetracted: util.Ptr(true),
}
// Block owned by testSubscription
// that no longer exists on the remote
// list, ie., it's been retracted.
retractedBlock = &gtsmodel.DomainBlock{
ID: "01JHX2V5WN250TKB6FQ1M3QE1H",
Domain: "retracted.example.org",
CreatedByAccount: testAccount,
CreatedByAccountID: testAccount.ID,
SubscriptionID: "01JGE681TQSBPAV59GZXPKE62H",
}
)
defer testrig.TearDownTestStructs(testStructs)
// Store test subscription.
if err := testStructs.State.DB.PutDomainPermissionSubscription(
ctx, testSubscription,
); err != nil {
suite.FailNow(err.Error())
}
// Store the retracted block.
if err := testStructs.State.DB.PutDomainBlock(
ctx, retractedBlock,
); err != nil {
suite.FailNow(err.Error())
}
// Process subscriptions.
subscriptions.ProcessDomainPermissionSubscriptions(ctx, testSubscription.PermissionType)
// Retracted block should be removed.
if !testrig.WaitFor(func() bool {
_, err := testStructs.State.DB.GetDomainBlock(ctx, retractedBlock.Domain)
return errors.Is(err, db.ErrNoEntries)
}) {
suite.FailNow("timed out waiting for block to be removed")
}
}
func (suite *SubscriptionsTestSuite) TestOrphanRetraction() {
var (
ctx = suite.T().Context()
testStructs = testrig.SetupTestStructs(rMediaPath, rTemplatePath)
testAccount = suite.testAccounts["admin_account"]
subscriptions = subscriptions.New(
testStructs.State,
testStructs.TransportController,
testStructs.TypeConverter,
)
// A subscription for a plain list of
// baddies, which orphans retracted entries.
testSubscription = &gtsmodel.DomainPermissionSubscription{
ID: "01JGE681TQSBPAV59GZXPKE62H",
Priority: 255,
Title: "whatever!",
PermissionType: gtsmodel.DomainPermissionBlock,
AsDraft: util.Ptr(false),
AdoptOrphans: util.Ptr(false),
CreatedByAccountID: testAccount.ID,
CreatedByAccount: testAccount,
URI: "https://lists.example.org/baddies.txt",
ContentType: gtsmodel.DomainPermSubContentTypePlain,
RemoveRetracted: util.Ptr(false),
}
// Block owned by testSubscription
// that no longer exists on the remote
// list, ie., it's been retracted.
retractedBlock = &gtsmodel.DomainBlock{
ID: "01JHX2V5WN250TKB6FQ1M3QE1H",
Domain: "retracted.example.org",
CreatedByAccount: testAccount,
CreatedByAccountID: testAccount.ID,
SubscriptionID: "01JGE681TQSBPAV59GZXPKE62H",
}
)
defer testrig.TearDownTestStructs(testStructs)
// Store test subscription.
if err := testStructs.State.DB.PutDomainPermissionSubscription(
ctx, testSubscription,
); err != nil {
suite.FailNow(err.Error())
}
// Store the retracted block.
if err := testStructs.State.DB.PutDomainBlock(
ctx, retractedBlock,
); err != nil {
suite.FailNow(err.Error())
}
// Process subscriptions.
subscriptions.ProcessDomainPermissionSubscriptions(ctx, testSubscription.PermissionType)
// Retracted block should be orphaned.
if !testrig.WaitFor(func() bool {
block, err := testStructs.State.DB.GetDomainBlock(ctx, retractedBlock.Domain)
return err == nil && block.SubscriptionID == ""
}) {
suite.FailNow("timed out waiting for block to be orphaned")
}
}
func TestSubscriptionTestSuite(t *testing.T) {
suite.Run(t, new(SubscriptionsTestSuite))
}

1
internal/typeutils/internaltofrontend.go
View file

@ -2212,6 +2212,7 @@ func (c *Converter) DomainPermSubToAPIDomainPermSub(
PermissionType: d.PermissionType.String(),
AsDraft: *d.AsDraft,
AdoptOrphans: *d.AdoptOrphans,
RemoveRetracted: *d.RemoveRetracted,
CreatedBy: d.CreatedByAccountID,
CreatedAt: util.FormatISO8601(createdAt),
URI: uri,

6
web/source/settings/views/moderation/domain-permissions/subscriptions/detail.tsx
View file

@ -139,6 +139,7 @@ function UpdateDomainPermSub({ permSub }: { permSub: DomainPermSub }) {
}),
content_type: useTextInput("content_type", { source: permSub }),
title: useTextInput("title", { source: permSub }),
remove_retracted: useBoolInput("remove_retracted", { source: permSub }),
as_draft: useBoolInput("as_draft", { source: permSub }),
adopt_orphans: useBoolInput("adopt_orphans", { source: permSub }),
useBasicAuth: useBoolInput("useBasicAuth", {
@ -311,6 +312,11 @@ function UpdateDomainPermSub({ permSub }: { permSub: DomainPermSub }) {
field={form.adopt_orphans}
/>
<Checkbox
label="Remove retracted permissions"
field={form.remove_retracted}
/>
<Checkbox
label="Create permissions as drafts"
field={form.as_draft}

6
web/source/settings/views/moderation/domain-permissions/subscriptions/new.tsx
View file

@ -39,6 +39,7 @@ export default function DomainPermissionSubscriptionNew() {
content_type: useTextInput("content_type", { defaultValue: "text/csv" }),
permission_type: useTextInput("permission_type", { defaultValue: "block" }),
title: useTextInput("title"),
remove_retracted: useBoolInput("remove_retracted", { defaultValue: true }),
as_draft: useBoolInput("as_draft", { defaultValue: true }),
adopt_orphans: useBoolInput("adopt_orphans", { defaultValue: false }),
fetch_username: useTextInput("fetch_username", {
@ -198,6 +199,11 @@ export default function DomainPermissionSubscriptionNew() {
field={form.adopt_orphans}
/>
<Checkbox
label="Remove retracted permissions"
field={form.remove_retracted}
/>
<Checkbox
label="Create permissions as drafts"
field={form.as_draft}