mirrors/gotosocial
1
0
Fork 0
mirror of https://github.com/superseriousbusiness/gotosocial.git synced 2025-10-29 02:32:25 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

[bugfix] Store and expose status content type (#3870)

Browse source 
* Add ContentType to internal models

* Add ContentType to API models StatusSource and StatusEdit

* Add helpers to convert between API/internal StatusContentType

* Write status content type on create/edit

* Add migration

* Update API docs

go run github.com/go-swagger/go-swagger/cmd/swagger generate spec --scan-models --exclude-deps --output docs/api/swagger.yaml

* ensure ContentType is updated anywhere Text is

* Update docs, take care of TODOs

* Set ContentType in more places where Text is set

* We don't actually use ContentType on the API status model

* Update StatusSource test

* Remove unused helper function I copied

* Revert change to StatusContentType swagger annotation

I'm going to include this in a follow-on PR instead.

* Add test for updating content type in edits

* Return a value from processContentType instead of modifying the existing status

Fixes an issue that was caught by the test I just added - the recorded edit would be marked with the *new* content type instead of the old one, which is obviously bad

* Add test for handling of statuses with no stored content type

* repurpose an existing test status instead of adding a new one to avoid breaking other tests

* Add test to ensure newly created statuses always have content type saved

* Do include content type on status API model actually

This is mostly important when deleting and redrafting.

The comment on `apimodel.Status.Text` implies that it's not sent except in response to status deletion, but actually this doesn't seem to be the case; it also appears to be present in responses to creations and normal fetches and stuff. So I'm treating `ContentType` the same here.

* Update new tests to check content type on API statuses

* Check content type of API statuses in all tests where text is checked

* update other api tests with status content type field

* Add test ensuring text and content type are returned when deleting a status

* Convert processContentType to free function and remove unused parameter

* check for the correct value in the deletion test

* Be explicit about this test status having an empty content type

* Use omitempty consistently on API models

* clean up the final diff a bit

* one more swagger regen for the road

* Handle nil statuses in processContentType

* Don't pass processContentType the entire edit form, it doesn't need it

* Move processContentType to common.go and use for creation as well

* Remove unused parameters to ContentTypeToAPIContentType
This commit is contained in:
ewwwin 2025-03-06 11:31:52 -05:00 • committed by GitHub
commit 424f62dd70
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
26 changed files with 560 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

3
internal/api/client/statuses/statusboost_test.go
View file

@ -145,6 +145,7 @@ func (suite *StatusBoostTestSuite) TestPostBoost() {
"bookmarked": true,
"card": null,
"content": "hello world! #welcome ! first post on the instance :rainbow: !",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [
@ -331,6 +332,7 @@ func (suite *StatusBoostTestSuite) TestPostBoostOwnFollowersOnly() {
"bookmarked": false,
"card": null,
"content": "hi!",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -543,6 +545,7 @@ func (suite *StatusBoostTestSuite) TestPostBoostImplicitAccept() {
"bookmarked": false,
"card": null,
"content": "<p>Hi <span class=\"h-card\"><a href=\"http://localhost:8080/@1happyturtle\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>1happyturtle</span></a></span>, can I reply?</p>",
"content_type": "text/markdown",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],

12
internal/api/client/statuses/statuscreate_test.go
View file

@ -139,6 +139,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatus() {
"bookmarked": false,
"card": null,
"content": "<p>this is a brand new status! <a href=\"http://localhost:8080/tags/helloworld\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>helloworld</span></a></p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -225,6 +226,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatusIntPolicy() {
"bookmarked": false,
"card": null,
"content": "<p>this is a brand new status! <a href=\"http://localhost:8080/tags/helloworld\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>helloworld</span></a></p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -321,6 +323,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatusIntPolicyJSON() {
"bookmarked": false,
"card": null,
"content": "<p>this is a brand new status! <a href=\"http://localhost:8080/tags/helloworld\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>helloworld</span></a></p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -535,6 +538,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatusMarkdown() {
"bookmarked": false,
"card": null,
"content": "<h1>Title</h1><h2>Smaller title</h2><p>This is a post written in <a href=\"https://www.markdownguide.org/\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">markdown</a></p>",
"content_type": "text/markdown",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -619,6 +623,7 @@ func (suite *StatusCreateTestSuite) TestMentionUnknownAccount() {
"bookmarked": false,
"card": null,
"content": "<p>hello <span class=\"h-card\"><a href=\"https://unknown-instance.com/@brand_new_person\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>brand_new_person</span></a></span></p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -697,6 +702,7 @@ func (suite *StatusCreateTestSuite) TestPostStatusWithLinksAndTags() {
"bookmarked": false,
"card": null,
"content": "<p><a href=\"http://localhost:8080/tags/test\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>test</span></a> alright, should be able to post <a href=\"http://localhost:8080/tags/links\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>links</span></a> with fragments in them now, let's see........<br><br><a href=\"https://docs.gotosocial.org/en/latest/user_guide/posts/#links\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">https://docs.gotosocial.org/en/latest/user_guide/posts/#links</a><br><br><a href=\"http://localhost:8080/tags/gotosocial\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>gotosocial</span></a><br><br>(tobi remember to pull the docker image challenge)</p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -781,6 +787,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatusWithEmoji() {
"bookmarked": false,
"card": null,
"content": "<p>here is a rainbow emoji a few times! :rainbow: :rainbow: :rainbow:<br>here's an emoji that isn't in the db: :test_emoji:</p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [
@ -879,6 +886,7 @@ func (suite *StatusCreateTestSuite) TestReplyToLocalStatus() {
"bookmarked": false,
"card": null,
"content": "<p>hello <span class=\"h-card\"><a href=\"http://localhost:8080/@1happyturtle\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>1happyturtle</span></a></span> this reply should work!</p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -962,6 +970,7 @@ func (suite *StatusCreateTestSuite) TestAttachNewMediaSuccess() {
"bookmarked": false,
"card": null,
"content": "<p>here's an image attachment</p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -1067,6 +1076,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatusWithNoncanonicalLanguageTag
"bookmarked": false,
"card": null,
"content": "<p>English? what's English? i speak American</p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -1142,6 +1152,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatusWithPollForm() {
"bookmarked": false,
"card": null,
"content": "<p>this is a status with a poll!</p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -1239,6 +1250,7 @@ func (suite *StatusCreateTestSuite) TestPostNewStatusWithPollJSON() {
"bookmarked": false,
"card": null,
"content": "<p>this is a status with a poll!</p>",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],

4
internal/api/client/statuses/statusdelete_test.go
View file

@ -77,6 +77,10 @@ func (suite *StatusDeleteTestSuite) TestPostDelete() {
suite.NoError(err)
suite.NotNil(statusReply)
// Check that text and content type are returned for delete and redraft
suite.Equal("hello everyone!", statusReply.Text)
suite.Equal(apimodel.StatusContentTypePlain, statusReply.ContentType)
if !testrig.WaitFor(func() bool {
_, err := suite.db.GetStatusByID(ctx, targetStatus.ID)
return errors.Is(err, db.ErrNoEntries)

2
internal/api/client/statuses/statusfave_test.go
View file

@ -104,6 +104,7 @@ func (suite *StatusFaveTestSuite) TestPostFave() {
"bookmarked": false,
"card": null,
"content": "🐕🐕🐕🐕🐕",
"content_type": "text/plain",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],
@ -228,6 +229,7 @@ func (suite *StatusFaveTestSuite) TestPostFaveImplicitAccept() {
"bookmarked": false,
"card": null,
"content": "<p>Hi <span class=\"h-card\"><a href=\"http://localhost:8080/@1happyturtle\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>1happyturtle</span></a></span>, can I reply?</p>",
"content_type": "text/markdown",
"created_at": "right the hell just now babyee",
"edited_at": null,
"emojis": [],

2
internal/api/client/statuses/statusmute_test.go
View file

@ -149,6 +149,7 @@ func (suite *StatusMuteTestSuite) TestMuteUnmuteStatus() {
"card": null,
"poll": null,
"text": "hello everyone!",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [
@ -238,6 +239,7 @@ func (suite *StatusMuteTestSuite) TestMuteUnmuteStatus() {
"card": null,
"poll": null,
"text": "hello everyone!",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [

3
internal/api/client/statuses/statussource_test.go
View file

@ -92,7 +92,8 @@ func (suite *StatusSourceTestSuite) TestGetSource() {
suite.Equal(`{
"id": "01F8MHAMCHF6Y650WCRSCP4WMY",
"text": "hello everyone!",
"spoiler_text": "introduction post"
"spoiler_text": "introduction post",
"content_type": "text/plain"
}`, dst.String())
}

7
internal/api/model/status.go
View file

@ -110,6 +110,10 @@ type Status struct {
// so the user may redraft from the source text without the client having to reverse-engineer
// the original text from the HTML content.
Text string `json:"text,omitempty"`
// Content type that was used to parse the status's text. Returned when
// status is deleted, so if the user is redrafting the message the client
// can default to the same content type.
ContentType StatusContentType `json:"content_type,omitempty"`
// A list of filters that matched this status and why they matched, if there are any such filters.
Filtered []FilterResult `json:"filtered,omitempty"`
// The interaction policy for this status, as set by the status author.
@ -320,6 +324,9 @@ type StatusSource struct {
// Plain-text version of spoiler text.
SpoilerText string `json:"spoiler_text"`
// Content type that was used to parse the text.
ContentType StatusContentType `json:"content_type,omitempty"`
}
// StatusEdit represents one historical revision of a status, containing

2
internal/cache/size.go vendored
View file

@ -649,6 +649,7 @@ func sizeofStatus() uintptr {
URL: exampleURI,
Content: exampleText,
Text: exampleText,
ContentType: gtsmodel.StatusContentTypePlain,
AttachmentIDs: []string{exampleID, exampleID, exampleID},
TagIDs: []string{exampleID, exampleID, exampleID},
MentionIDs: []string{},
@ -694,6 +695,7 @@ func sizeofStatusEdit() uintptr {
Content: exampleText,
ContentWarning: exampleUsername, // similar length
Text: exampleText,
ContentType: gtsmodel.StatusContentTypePlain,
Language: "en",
Sensitive: func() *bool { ok := false; return &ok }(),
AttachmentIDs: []string{exampleID, exampleID, exampleID},

78
internal/db/bundb/migrations/20250226013442_add_status_content_type.go Normal file
View file

@ -0,0 +1,78 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package migrations
import (
"context"
"reflect"
gtsmodel "github.com/superseriousbusiness/gotosocial/internal/db/bundb/migrations/20250226013442_add_status_content_type"
"github.com/uptrace/bun"
)
func init() {
up := func(ctx context.Context, db *bun.DB) error {
return db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
// Generate new Status.ContentType column definition from bun.
statusType := reflect.TypeOf((*gtsmodel.Status)(nil))
colDef, err := getBunColumnDef(tx, statusType, "ContentType")
if err != nil {
return err
}
// Add column to Status table.
_, err = tx.NewAddColumn().
Model((*gtsmodel.Status)(nil)).
ColumnExpr(colDef).
Exec(ctx)
if err != nil {
return err
}
// same for StatusEdit
statusEditType := reflect.TypeOf((*gtsmodel.StatusEdit)(nil))
colDef, err = getBunColumnDef(tx, statusEditType, "ContentType")
if err != nil {
return err
}
_, err = tx.NewAddColumn().
Model((*gtsmodel.StatusEdit)(nil)).
ColumnExpr(colDef).
Exec(ctx)
if err != nil {
return err
}
return nil
})
}
down := func(ctx context.Context, db *bun.DB) error {
return db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
return nil
})
}
if err := Migrations.Register(up, down); err != nil {
panic(err)
}
}

89
internal/db/bundb/migrations/20250226013442_add_status_content_type/status.go Normal file
View file

@ -0,0 +1,89 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package gtsmodel
import (
"time"
"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
)
// Status represents a user-created 'post' or 'status' in the database, either remote or local
type Status struct {
ID string `bun:"type:CHAR(26),pk,nullzero,notnull,unique"` // id of this item in the database
CreatedAt time.Time `bun:"type:timestamptz,nullzero,notnull,default:current_timestamp"` // when was item created
EditedAt time.Time `bun:"type:timestamptz,nullzero"` // when this status was last edited (if set)
FetchedAt time.Time `bun:"type:timestamptz,nullzero"` // when was item (remote) last fetched.
PinnedAt time.Time `bun:"type:timestamptz,nullzero"` // Status was pinned by owning account at this time.
URI string `bun:",unique,nullzero,notnull"` // activitypub URI of this status
URL string `bun:",nullzero"` // web url for viewing this status
Content string `bun:""` // content of this status; likely html-formatted but not guaranteed
AttachmentIDs []string `bun:"attachments,array"` // Database IDs of any media attachments associated with this status
Attachments []*gtsmodel.MediaAttachment `bun:"attached_media,rel:has-many"` // Attachments corresponding to attachmentIDs
TagIDs []string `bun:"tags,array"` // Database IDs of any tags used in this status
Tags []*gtsmodel.Tag `bun:"attached_tags,m2m:status_to_tags"` // Tags corresponding to tagIDs. https://bun.uptrace.dev/guide/relations.html#many-to-many-relation
MentionIDs []string `bun:"mentions,array"` // Database IDs of any mentions in this status
Mentions []*gtsmodel.Mention `bun:"attached_mentions,rel:has-many"` // Mentions corresponding to mentionIDs
EmojiIDs []string `bun:"emojis,array"` // Database IDs of any emojis used in this status
Emojis []*gtsmodel.Emoji `bun:"attached_emojis,m2m:status_to_emojis"` // Emojis corresponding to emojiIDs. https://bun.uptrace.dev/guide/relations.html#many-to-many-relation
Local *bool `bun:",nullzero,notnull,default:false"` // is this status from a local account?
AccountID string `bun:"type:CHAR(26),nullzero,notnull"` // which account posted this status?
Account *gtsmodel.Account `bun:"rel:belongs-to"` // account corresponding to accountID
AccountURI string `bun:",nullzero,notnull"` // activitypub uri of the owner of this status
InReplyToID string `bun:"type:CHAR(26),nullzero"` // id of the status this status replies to
InReplyToURI string `bun:",nullzero"` // activitypub uri of the status this status is a reply to
InReplyToAccountID string `bun:"type:CHAR(26),nullzero"` // id of the account that this status replies to
InReplyTo *Status `bun:"-"` // status corresponding to inReplyToID
InReplyToAccount *gtsmodel.Account `bun:"rel:belongs-to"` // account corresponding to inReplyToAccountID
BoostOfID string `bun:"type:CHAR(26),nullzero"` // id of the status this status is a boost of
BoostOfURI string `bun:"-"` // URI of the status this status is a boost of; field not inserted in the db, just for dereferencing purposes.
BoostOfAccountID string `bun:"type:CHAR(26),nullzero"` // id of the account that owns the boosted status
BoostOf *Status `bun:"-"` // status that corresponds to boostOfID
BoostOfAccount *gtsmodel.Account `bun:"rel:belongs-to"` // account that corresponds to boostOfAccountID
ThreadID string `bun:"type:CHAR(26),nullzero"` // id of the thread to which this status belongs; only set for remote statuses if a local account is involved at some point in the thread, otherwise null
EditIDs []string `bun:"edits,array"` //
Edits []*gtsmodel.StatusEdit `bun:"-"` //
PollID string `bun:"type:CHAR(26),nullzero"` //
Poll *gtsmodel.Poll `bun:"-"` //
ContentWarning string `bun:",nullzero"` // cw string for this status
Visibility gtsmodel.Visibility `bun:",nullzero,notnull"` // visibility entry for this status
Sensitive *bool `bun:",nullzero,notnull,default:false"` // mark the status as sensitive?
Language string `bun:",nullzero"` // what language is this status written in?
CreatedWithApplicationID string `bun:"type:CHAR(26),nullzero"` // Which application was used to create this status?
CreatedWithApplication *gtsmodel.Application `bun:"rel:belongs-to"` // application corresponding to createdWithApplicationID
ActivityStreamsType string `bun:",nullzero,notnull"` // What is the activitystreams type of this status? See: https://www.w3.org/TR/activitystreams-vocabulary/#object-types. Will probably almost always be Note but who knows!.
Text string `bun:""` // Original text of the status without formatting
ContentType StatusContentType `bun:",nullzero"` // Content type used to process the original text of the status
Federated *bool `bun:",notnull"` // This status will be federated beyond the local timeline(s)
InteractionPolicy *gtsmodel.InteractionPolicy `bun:""` // InteractionPolicy for this status. If null then the default InteractionPolicy should be assumed for this status's Visibility. Always null for boost wrappers.
PendingApproval *bool `bun:",nullzero,notnull,default:false"` // If true then status is a reply or boost wrapper that must be Approved by the reply-ee or boost-ee before being fully distributed.
PreApproved bool `bun:"-"` // If true, then status is a reply to or boost wrapper of a status on our instance, has permission to do the interaction, and an Accept should be sent out for it immediately. Field not stored in the DB.
ApprovedByURI string `bun:",nullzero"` // URI of an Accept Activity that approves the Announce or Create Activity that this status was/will be attached to.
}
type enumType int16
// StatusContentType is the content type with which a status's text is
// parsed. Can be either plain or markdown. Empty will default to plain.
type StatusContentType enumType
const (
StatusContentTypePlain StatusContentType = 1
StatusContentTypeMarkdown StatusContentType = 2
StatusContentTypeDefault = StatusContentTypePlain
)

51
internal/db/bundb/migrations/20250226013442_add_status_content_type/statusedit.go Normal file
View file

@ -0,0 +1,51 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package gtsmodel
import (
"time"
"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
)
// StatusEdit represents a **historical** view of a Status
// after a received edit. The Status itself will always
// contain the latest up-to-date information.
//
// Note that stored status edits may not exactly match that
// of the origin server, they are a best-effort by receiver
// to store version history. There is no AP history endpoint.
type StatusEdit struct {
ID string `bun:"type:CHAR(26),pk,nullzero,notnull,unique"` // ID of this item in the database.
Content string `bun:""` // Content of status at time of edit; likely html-formatted but not guaranteed.
ContentWarning string `bun:",nullzero"` // Content warning of status at time of edit.
Text string `bun:""` // Original status text, without formatting, at time of edit.
ContentType StatusContentType `bun:",nullzero"` // Content type used to process the original text of the status.
Language string `bun:",nullzero"` // Status language at time of edit.
Sensitive *bool `bun:",nullzero,notnull,default:false"` // Status sensitive flag at time of edit.
AttachmentIDs []string `bun:"attachments,array"` // Database IDs of media attachments associated with status at time of edit.
AttachmentDescriptions []string `bun:",array"` // Previous media descriptions of media attachments associated with status at time of edit.
Attachments []*gtsmodel.MediaAttachment `bun:"-"` // Media attachments relating to .AttachmentIDs field (not always populated).
PollOptions []string `bun:",array"` // Poll options of status at time of edit, only set if status contains a poll.
PollVotes []int `bun:",array"` // Poll vote count at time of status edit, only set if poll votes were reset.
StatusID string `bun:"type:CHAR(26),nullzero,notnull"` // The originating status ID this is a historical edit of.
CreatedAt time.Time `bun:"type:timestamptz,nullzero,notnull,default:current_timestamp"` // The creation time of this version of the status content (according to receiving server).
// We don't bother having a *gtsmodel.Status model here
// as the StatusEdit is always just attached to a Status,
// so it doesn't need a self-reference back to it.
}

1
internal/db/bundb/timeline_test.go
View file

@ -45,6 +45,7 @@ func getFutureStatus() *gtsmodel.Status {
URL: "http://localhost:8080/@admin/statuses/" + id,
Content: "it's the future, wooooooooooooooooooooooooooooooooo",
Text: "it's the future, wooooooooooooooooooooooooooooooooo",
ContentType: gtsmodel.StatusContentTypePlain,
AttachmentIDs: []string{},
TagIDs: []string{},
MentionIDs: []string{},

1
internal/federation/dereferencing/status.go
View file

@ -1239,6 +1239,7 @@ func (d *Dereferencer) handleStatusEdit(
edit.Content = existing.Content
edit.ContentWarning = existing.ContentWarning
edit.Text = existing.Text
edit.ContentType = existing.ContentType
edit.Language = existing.Language
edit.Sensitive = existing.Sensitive
edit.StatusID = status.ID

11
internal/gtsmodel/status.go
View file

@ -69,6 +69,7 @@ type Status struct {
CreatedWithApplication *Application `bun:"rel:belongs-to"` // application corresponding to createdWithApplicationID
ActivityStreamsType string `bun:",nullzero,notnull"` // What is the activitystreams type of this status? See: https://www.w3.org/TR/activitystreams-vocabulary/#object-types. Will probably almost always be Note but who knows!.
Text string `bun:""` // Original text of the status without formatting
ContentType StatusContentType `bun:",nullzero"` // Content type used to process the original text of the status
Federated *bool `bun:",notnull"` // This status will be federated beyond the local timeline(s)
InteractionPolicy *InteractionPolicy `bun:""` // InteractionPolicy for this status. If null then the default InteractionPolicy should be assumed for this status's Visibility. Always null for boost wrappers.
PendingApproval *bool `bun:",nullzero,notnull,default:false"` // If true then status is a reply or boost wrapper that must be Approved by the reply-ee or boost-ee before being fully distributed.
@ -376,6 +377,16 @@ func (v Visibility) String() string {
}
}
// StatusContentType is the content type with which a status's text is
// parsed. Can be either plain or markdown. Empty will default to plain.
type StatusContentType enumType
const (
StatusContentTypePlain StatusContentType = 1
StatusContentTypeMarkdown StatusContentType = 2
StatusContentTypeDefault = StatusContentTypePlain
)
// Content models the simple string content
// of a status along with its ContentMap,
// which contains content entries keyed by

1
internal/gtsmodel/statusedit.go
View file

@ -31,6 +31,7 @@ type StatusEdit struct {
Content string `bun:""` // Content of status at time of edit; likely html-formatted but not guaranteed.
ContentWarning string `bun:",nullzero"` // Content warning of status at time of edit.
Text string `bun:""` // Original status text, without formatting, at time of edit.
ContentType StatusContentType `bun:",nullzero"` // Content type used to process the original text of the status.
Language string `bun:",nullzero"` // Status language at time of edit.
Sensitive *bool `bun:",nullzero,notnull,default:false"` // Status sensitive flag at time of edit.
AttachmentIDs []string `bun:"attachments,array"` // Database IDs of media attachments associated with status at time of edit.

41
internal/processing/status/common.go
View file

@ -30,6 +30,7 @@ import (
"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
"github.com/superseriousbusiness/gotosocial/internal/id"
"github.com/superseriousbusiness/gotosocial/internal/text"
"github.com/superseriousbusiness/gotosocial/internal/typeutils"
"github.com/superseriousbusiness/gotosocial/internal/util/xslices"
"github.com/superseriousbusiness/gotosocial/internal/validate"
)
@ -106,11 +107,39 @@ type statusContent struct {
Tags []*gtsmodel.Tag
}
// Returns the final content type to use when creating or editing a status.
func processContentType(
requestContentType apimodel.StatusContentType,
existingStatus *gtsmodel.Status,
accountDefaultContentType string,
) gtsmodel.StatusContentType {
switch {
// Content type set in the request, return the new value.
case requestContentType != "":
return typeutils.APIContentTypeToContentType(requestContentType)
// No content type in the request, return the existing
// status's current content type if we know of one.
case existingStatus != nil && existingStatus.ContentType != 0:
return existingStatus.ContentType
// We aren't editing an existing status, or if we are
// it's an old one that doesn't have a saved content
// type. Use the user's default content type setting.
case accountDefaultContentType != "":
return typeutils.APIContentTypeToContentType(apimodel.StatusContentType(accountDefaultContentType))
// uhh.. Fall back to global default.
default:
return gtsmodel.StatusContentTypeDefault
}
}
func (p *Processor) processContent(
ctx context.Context,
author *gtsmodel.Account,
statusID string,
contentType string,
contentType gtsmodel.StatusContentType,
content string,
contentWarning string,
language string,
@ -146,20 +175,14 @@ func (p *Processor) processContent(
// function, according to the provided content-type.
var format text.FormatFunc
if contentType == "" {
// If content type wasn't specified, use
// the author's preferred content-type.
contentType = author.Settings.StatusContentType
}
switch contentType {
// Format status according to text/plain.
case "", string(apimodel.StatusContentTypePlain):
case gtsmodel.StatusContentTypePlain:
format = p.formatter.FromPlain
// Format status according to text/markdown.
case string(apimodel.StatusContentTypeMarkdown):
case gtsmodel.StatusContentTypeMarkdown:
format = p.formatter.FromMarkdown
// Unknown.

6
internal/processing/status/create.go
View file

@ -66,11 +66,14 @@ func (p *Processor) Create(
// Generate new ID for status.
statusID := id.NewULID()
// Process incoming content type.
contentType := processContentType(form.ContentType, nil, requester.Settings.StatusContentType)
// Process incoming status content fields.
content, errWithCode := p.processContent(ctx,
requester,
statusID,
string(form.ContentType),
contentType,
form.Status,
form.SpoilerText,
form.Language,
@ -163,6 +166,7 @@ func (p *Processor) Create(
Content: content.Content,
ContentWarning: content.ContentWarning,
Text: form.Status, // raw
ContentType: contentType,
// Set gathered mentions.
MentionIDs: content.MentionIDs,

30
internal/processing/status/create_test.go
View file

@ -238,6 +238,36 @@ func (suite *StatusCreateTestSuite) TestProcessReplyToUnthreadedRemoteStatus() {
suite.NotEmpty(dbStatus.ThreadID)
}
func (suite *StatusCreateTestSuite) TestProcessNoContentTypeUsesDefault() {
ctx := context.Background()
creatingAccount := suite.testAccounts["local_account_1"]
creatingApplication := suite.testApplications["application_1"]
statusCreateForm := &apimodel.StatusCreateRequest{
Status: "poopoo peepee",
SpoilerText: "",
MediaIDs: []string{},
Poll: nil,
InReplyToID: "",
Sensitive: false,
Visibility: apimodel.VisibilityPublic,
LocalOnly: util.Ptr(false),
ScheduledAt: nil,
Language: "en",
ContentType: "",
}
apiStatus, errWithCode := suite.status.Create(ctx, creatingAccount, creatingApplication, statusCreateForm)
suite.NoError(errWithCode)
suite.NotNil(apiStatus)
suite.Equal("<p>poopoo peepee</p>", apiStatus.Content)
// the test accounts don't have settings, so we're comparing to
// the global default value instead of the requester's default
suite.Equal(apimodel.StatusContentTypeDefault, apiStatus.ContentType)
}
func TestStatusCreateTestSuite(t *testing.T) {
suite.Run(t, new(StatusCreateTestSuite))
}

7
internal/processing/status/edit.go
View file

@ -84,11 +84,14 @@ func (p *Processor) Edit(
return nil, errWithCode
}
// Process incoming content type.
contentType := processContentType(form.ContentType, status, requester.Settings.StatusContentType)
// Process incoming status edit content fields.
content, errWithCode := p.processContent(ctx,
requester,
statusID,
string(form.ContentType),
contentType,
form.Status,
form.SpoilerText,
form.Language,
@ -256,6 +259,7 @@ func (p *Processor) Edit(
edit.Content = status.Content
edit.ContentWarning = status.ContentWarning
edit.Text = status.Text
edit.ContentType = status.ContentType
edit.Language = status.Language
edit.Sensitive = status.Sensitive
edit.StatusID = status.ID
@ -298,6 +302,7 @@ func (p *Processor) Edit(
status.Content = content.Content
status.ContentWarning = content.ContentWarning
status.Text = form.Status
status.ContentType = contentType
status.Language = content.Language
status.Sensitive = &form.Sensitive
status.AttachmentIDs = form.MediaIDs

137
internal/processing/status/edit_test.go
View file

@ -26,6 +26,7 @@ import (
"github.com/stretchr/testify/suite"
apimodel "github.com/superseriousbusiness/gotosocial/internal/api/model"
"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
"github.com/superseriousbusiness/gotosocial/internal/typeutils"
"github.com/superseriousbusiness/gotosocial/internal/util"
"github.com/superseriousbusiness/gotosocial/internal/util/xslices"
)
@ -90,6 +91,142 @@ func (suite *StatusEditTestSuite) TestSimpleEdit() {
previousEdit := latestStatus.Edits[len(latestStatus.Edits)-1]
suite.Equal(status.Content, previousEdit.Content)
suite.Equal(status.Text, previousEdit.Text)
suite.Equal(status.ContentType, previousEdit.ContentType)
suite.Equal(status.ContentWarning, previousEdit.ContentWarning)
suite.Equal(*status.Sensitive, *previousEdit.Sensitive)
suite.Equal(status.Language, previousEdit.Language)
suite.Equal(status.UpdatedAt(), previousEdit.CreatedAt)
}
func (suite *StatusEditTestSuite) TestEditChangeContentType() {
// Create cancellable context to use for test.
ctx, cncl := context.WithCancel(context.Background())
defer cncl()
// Get a local account to use as test requester.
requester := suite.testAccounts["local_account_1"]
requester, _ = suite.state.DB.GetAccountByID(ctx, requester.ID)
// Get requester's existing plain text status to perform an edit on.
status := suite.testStatuses["local_account_1_status_6"]
status, _ = suite.state.DB.GetStatusByID(ctx, status.ID)
// Prepare edit with a Markdown body.
form := &apimodel.StatusEditRequest{
Status: "ooh the status is *fancy* now!",
ContentType: apimodel.StatusContentTypeMarkdown,
SpoilerText: "shhhhh",
Sensitive: true,
Language: "fr", // hoh hoh hoh
MediaIDs: nil,
MediaAttributes: nil,
Poll: nil,
}
// Pass the prepared form to the status processor to perform the edit.
apiStatus, errWithCode := suite.status.Edit(ctx, requester, status.ID, form)
suite.NotNil(apiStatus)
suite.NoError(errWithCode)
// Check response against input form data.
suite.Equal(form.Status, apiStatus.Text)
suite.Equal(form.ContentType, apiStatus.ContentType)
suite.Equal(form.SpoilerText, apiStatus.SpoilerText)
suite.Equal(form.Sensitive, apiStatus.Sensitive)
suite.Equal(form.Language, *apiStatus.Language)
suite.NotEqual(util.FormatISO8601(status.EditedAt), *apiStatus.EditedAt)
// Fetched the latest version of edited status from the database.
latestStatus, err := suite.state.DB.GetStatusByID(ctx, status.ID)
suite.NoError(err)
// Check latest status against input form data.
suite.Equal(form.Status, latestStatus.Text)
suite.Equal(typeutils.APIContentTypeToContentType(form.ContentType), latestStatus.ContentType)
suite.Equal(form.SpoilerText, latestStatus.ContentWarning)
suite.Equal(form.Sensitive, *latestStatus.Sensitive)
suite.Equal(form.Language, latestStatus.Language)
suite.Equal(len(status.EditIDs)+1, len(latestStatus.EditIDs))
suite.NotEqual(status.UpdatedAt(), latestStatus.UpdatedAt())
// Populate all historical edits for this status.
err = suite.state.DB.PopulateStatusEdits(ctx, latestStatus)
suite.NoError(err)
// Check previous status edit matches original status content.
previousEdit := latestStatus.Edits[len(latestStatus.Edits)-1]
suite.Equal(status.Content, previousEdit.Content)
suite.Equal(status.Text, previousEdit.Text)
suite.Equal(status.ContentType, previousEdit.ContentType)
suite.Equal(status.ContentWarning, previousEdit.ContentWarning)
suite.Equal(*status.Sensitive, *previousEdit.Sensitive)
suite.Equal(status.Language, previousEdit.Language)
suite.Equal(status.UpdatedAt(), previousEdit.CreatedAt)
}
func (suite *StatusEditTestSuite) TestEditOnStatusWithNoContentType() {
// Create cancellable context to use for test.
ctx, cncl := context.WithCancel(context.Background())
defer cncl()
// Get a local account to use as test requester.
requester := suite.testAccounts["local_account_1"]
requester, _ = suite.state.DB.GetAccountByID(ctx, requester.ID)
// Get requester's existing status, which has no
// stored content type, to perform an edit on.
status := suite.testStatuses["local_account_1_status_2"]
status, _ = suite.state.DB.GetStatusByID(ctx, status.ID)
// Prepare edit without setting a new content type.
form := &apimodel.StatusEditRequest{
Status: "how will this text be parsed? it is a mystery",
SpoilerText: "shhhhh",
Sensitive: true,
Language: "fr", // hoh hoh hoh
MediaIDs: nil,
MediaAttributes: nil,
Poll: nil,
}
// Pass the prepared form to the status processor to perform the edit.
apiStatus, errWithCode := suite.status.Edit(ctx, requester, status.ID, form)
suite.NotNil(apiStatus)
suite.NoError(errWithCode)
// Check response against input form data.
suite.Equal(form.Status, apiStatus.Text)
suite.NotEqual(util.FormatISO8601(status.EditedAt), *apiStatus.EditedAt)
// Check response against requester's default content type setting
// (the test accounts don't actually have settings on them, so
// instead we check that the global default content type is used)
suite.Equal(apimodel.StatusContentTypeDefault, apiStatus.ContentType)
// Fetched the latest version of edited status from the database.
latestStatus, err := suite.state.DB.GetStatusByID(ctx, status.ID)
suite.NoError(err)
// Check latest status against input form data
suite.Equal(form.Status, latestStatus.Text)
suite.Equal(form.Sensitive, *latestStatus.Sensitive)
suite.Equal(form.Language, latestStatus.Language)
suite.Equal(len(status.EditIDs)+1, len(latestStatus.EditIDs))
suite.NotEqual(status.UpdatedAt(), latestStatus.UpdatedAt())
// Check latest status against requester's default content
// type (again, actually just checking for the global default)
suite.Equal(gtsmodel.StatusContentTypeDefault, latestStatus.ContentType)
// Populate all historical edits for this status.
err = suite.state.DB.PopulateStatusEdits(ctx, latestStatus)
suite.NoError(err)
// Check previous status edit matches original status content.
previousEdit := latestStatus.Edits[len(latestStatus.Edits)-1]
suite.Equal(status.Content, previousEdit.Content)
suite.Equal(status.Text, previousEdit.Text)
suite.Equal(status.ContentType, previousEdit.ContentType)
suite.Equal(status.ContentWarning, previousEdit.ContentWarning)
suite.Equal(*status.Sensitive, *previousEdit.Sensitive)
suite.Equal(status.Language, previousEdit.Language)

2
internal/processing/status/get.go
View file

@ -24,6 +24,7 @@ import (
apimodel "github.com/superseriousbusiness/gotosocial/internal/api/model"
"github.com/superseriousbusiness/gotosocial/internal/gtserror"
"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
"github.com/superseriousbusiness/gotosocial/internal/typeutils"
)
// Get gets the given status, taking account of privacy settings and blocks etc.
@ -56,5 +57,6 @@ func (p *Processor) SourceGet(ctx context.Context, requester *gtsmodel.Account,
ID: status.ID,
Text: status.Text,
SpoilerText: status.ContentWarning,
ContentType: typeutils.ContentTypeToAPIContentType(status.ContentType),
}, nil
}

10
internal/typeutils/frontendtointernal.go
View file

@ -44,6 +44,16 @@ func APIVisToVis(m apimodel.Visibility) gtsmodel.Visibility {
return 0
}
func APIContentTypeToContentType(m apimodel.StatusContentType) gtsmodel.StatusContentType {
switch m {
case apimodel.StatusContentTypePlain:
return gtsmodel.StatusContentTypePlain
case apimodel.StatusContentTypeMarkdown:
return gtsmodel.StatusContentTypeMarkdown
}
return 0
}
func APIMarkerNameToMarkerName(m apimodel.MarkerName) gtsmodel.MarkerName {
switch m {
case apimodel.MarkerNameHome:

12
internal/typeutils/internaltofrontend.go
View file

@ -1391,6 +1391,7 @@ func (c *Converter) baseStatusToFrontend(
Emojis: apiEmojis,
Card: nil, // TODO: implement cards
Text: s.Text,
ContentType: ContentTypeToAPIContentType(s.ContentType),
InteractionPolicy: *apiInteractionPolicy,
}
@ -1626,6 +1627,17 @@ func (c *Converter) VisToAPIVis(ctx context.Context, m gtsmodel.Visibility) apim
return ""
}
// Converts a gts status content type into its api equivalent
func ContentTypeToAPIContentType(m gtsmodel.StatusContentType) apimodel.StatusContentType {
switch m {
case gtsmodel.StatusContentTypePlain:
return apimodel.StatusContentTypePlain
case gtsmodel.StatusContentTypeMarkdown:
return apimodel.StatusContentTypeMarkdown
}
return ""
}
// InstanceRuleToAdminAPIRule converts a local instance rule into its api equivalent for serving at /api/v1/admin/instance/rules/:id
func InstanceRuleToAPIRule(r gtsmodel.Rule) apimodel.InstanceRule {
return apimodel.InstanceRule{

10
internal/typeutils/internaltofrontend_test.go
View file

@ -578,6 +578,7 @@ func (suite *InternalToFrontendTestSuite) TestStatusToFrontend() {
"card": null,
"poll": null,
"text": "hello world! #welcome ! first post on the instance :rainbow: !",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [
@ -758,6 +759,7 @@ func (suite *InternalToFrontendTestSuite) TestWarnFilteredStatusToFrontend() {
"card": null,
"poll": null,
"text": "hello world! #welcome ! first post on the instance :rainbow: ! fnord",
"content_type": "text/plain",
"filtered": [
{
"filter": {
@ -943,6 +945,7 @@ func (suite *InternalToFrontendTestSuite) TestWarnFilteredBoostToFrontend() {
"card": null,
"poll": null,
"text": "hello world! #welcome ! first post on the instance :rainbow: ! fnord",
"content_type": "text/plain",
"filtered": [
{
"filter": {
@ -1676,6 +1679,7 @@ func (suite *InternalToFrontendTestSuite) TestStatusToFrontendUnknownLanguage()
"card": null,
"poll": null,
"text": "hello world! #welcome ! first post on the instance :rainbow: !",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [
@ -1774,6 +1778,7 @@ func (suite *InternalToFrontendTestSuite) TestStatusToFrontendPartialInteraction
"card": null,
"poll": null,
"text": "this is a very personal post that I don't want anyone to interact with at all, and i only want mutuals to see it",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [
@ -1897,6 +1902,7 @@ func (suite *InternalToFrontendTestSuite) TestStatusToAPIStatusPendingApproval()
"card": null,
"poll": null,
"text": "Hi @1happyturtle, can I reply?",
"content_type": "text/markdown",
"interaction_policy": {
"can_favourite": {
"always": [
@ -3375,6 +3381,7 @@ func (suite *InternalToFrontendTestSuite) TestIntReqToAPI() {
"card": null,
"poll": null,
"text": "🐢 i don't mind people sharing and liking this one but I want to moderate replies to it 🐢",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [
@ -3473,6 +3480,7 @@ func (suite *InternalToFrontendTestSuite) TestIntReqToAPI() {
"card": null,
"poll": null,
"text": "Hi @1happyturtle, can I reply?",
"content_type": "text/markdown",
"interaction_policy": {
"can_favourite": {
"always": [
@ -3632,6 +3640,7 @@ func (suite *InternalToFrontendTestSuite) TestConversationToAPISelfConvo() {
"card": null,
"poll": null,
"text": "hello everyone!",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [
@ -3801,6 +3810,7 @@ func (suite *InternalToFrontendTestSuite) TestConversationToAPI() {
"card": null,
"poll": null,
"text": "hello everyone!",
"content_type": "text/plain",
"interaction_policy": {
"can_favourite": {
"always": [