gotosocial/internal/util/xslices/slices.go
kim 6a6a499333
[performance] rewrite timelines to rely on new timeline cache type (#3941)
* start work rewriting timeline cache type

* further work rewriting timeline caching

* more work integration new timeline code

* remove old code

* add local timeline, fix up merge conflicts

* remove old use of go-bytes

* implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr

* remove old timeline package, add local timeline cache

* remove references to old timeline types that needed starting up in tests

* start adding page validation

* fix test-identified timeline cache package issues

* fix up more tests, fix missing required changes, etc

* add exclusion for test.out in gitignore

* clarify some things better in code comments

* tweak cache size limits

* fix list timeline cache fetching

* further list timeline fixes

* linter, ssssssssshhhhhhhhhhhh please

* fix linter hints

* reslice the output if it's beyond length of 'lim'

* remove old timeline initialization code, bump go-structr to v0.9.4

* continued from previous commit

* improved code comments

* don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts

* finish writing more code comments

* some variable renaming, for ease of following

* change the way we update lo,hi paging values during timeline load

* improved code comments for updated / returned lo , hi paging values

* finish writing code comments for the StatusTimeline{} type itself

* fill in more code comments

* update go-structr version to latest with changed timeline unique indexing logic

* have a local and public timeline *per user*

* rewrite calls to public / local timeline calls

* remove the zero length check, as lo, hi values might still be set

* simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions

* swap the lo, hi values 🤦

* add (now) missing slice reverse of tag timeline statuses when paging ASC

* remove local / public caches (is out of scope for this work), share more timeline code

* remove unnecessary change

* again, remove more unused code

* remove unused function to appease the linter

* move boost checking to prepare function

* fix use of timeline.lastOrder, fix incorrect range functions used

* remove comments for repeat code

* remove the boost logic from prepare function

* do a maximum of 5 loads, not 10

* add repeat boost filtering logic, update go-structr, general improvements

* more code comments

* add important note

* fix timeline tests now that timelines are returned in page order

* remove unused field

* add StatusTimeline{} tests

* add more status timeline tests

* start adding preloading support

* ensure repeat boosts are marked in preloaded entries

* share a bunch of the database load code in timeline cache, don't clear timelines on relationship change

* add logic to allow dynamic clear / preloading of timelines

* comment-out unused functions, but leave in place as we might end-up using them

* fix timeline preload state check

* much improved status timeline code comments

* more code comments, don't bother inserting statuses if timeline not preloaded

* shift around some logic to make sure things aren't accidentally left set

* finish writing code comments

* remove trim-after-insert behaviour

* fix-up some comments referring to old logic

* remove unsetting of lo, hi

* fix preload repeatBoost checking logic

* don't return on status filter errors, these are usually transient

* better concurrency safety in Clear() and Done()

* fix test broken due to addition of preloader

* fix repeatBoost logic that doesn't account for already-hidden repeatBoosts

* ensure edit submodels are dropped on cache insertion

* update code-comment to expand CAS accronym

* use a plus1hULID() instead of 24h

* remove unused functions

* add note that public / local timeline requester can be nil

* fix incorrect visibility filtering of tag timeline statuses

* ensure we filter home timeline statuses on local only

* some small re-orderings to confirm query params in correct places

* fix the local only home timeline filter func
2025-04-26 09:56:15 +00:00

233 lines
5.3 KiB
Go

// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package xslices
import (
"slices"
)
// ToAny converts a slice of any input type
// to the abstrace empty interface slice type.
func ToAny[T any](in []T) []any {
out := make([]any, len(in))
for i, v := range in {
out[i] = v
}
return out
}
// GrowJust increases slice capacity to guarantee
// extra room 'size', where in the case that it does
// need to allocate more it ONLY allocates 'size' extra.
// This is different to typical slices.Grow behaviour,
// which simply guarantees extra through append() which
// may allocate more than necessary extra size.
func GrowJust[T any](in []T, size int) []T {
if cap(in)-len(in) < size {
// Reallocate enough for in + size.
in2 := make([]T, len(in), len(in)+size)
_ = copy(in2, in)
in = in2
}
return in
}
// AppendJust appends extra elements to slice,
// ONLY allocating at most len(extra) elements. This
// is different to the typical append behaviour which
// will append extra, in a manner to reduce the need
// for new allocations on every call to append.
func AppendJust[T any](in []T, extra ...T) []T {
l := len(in)
if cap(in)-l < len(extra) {
// Reallocate enough for + extra.
in2 := make([]T, l+len(extra))
_ = copy(in2, in)
in = in2
} else {
// Reslice for + extra.
in = in[:l+len(extra)]
}
// Copy extra into slice.
_ = copy(in[l:], extra)
return in
}
// Deduplicate deduplicates entries in the given slice.
func Deduplicate[T comparable](in []T) []T {
var (
inL = len(in)
unique = make(map[T]struct{}, inL)
deduped = make([]T, 0, inL)
)
for _, v := range in {
if _, ok := unique[v]; ok {
// Already have this.
continue
}
unique[v] = struct{}{}
deduped = append(deduped, v)
}
return deduped
}
// DeduplicateFunc deduplicates entries in the given
// slice, using the result of key() to gauge uniqueness.
func DeduplicateFunc[T any, C comparable](in []T, key func(v T) C) []T {
var (
inL = len(in)
unique = make(map[C]struct{}, inL)
deduped = make([]T, 0, inL)
)
if key == nil {
panic("nil func")
}
for _, v := range in {
k := key(v)
if _, ok := unique[k]; ok {
// Already have this.
continue
}
unique[k] = struct{}{}
deduped = append(deduped, v)
}
return deduped
}
// Gather will collect the values of type V from input type []T,
// passing each item to 'get' and appending V to the return slice.
func Gather[T, V any](out []V, in []T, get func(T) V) []V {
if get == nil {
panic("nil func")
}
// Starting write index
// in the resliced / re
// alloc'd output slice.
start := len(out)
// Total required slice len.
total := start + len(in)
if total > cap(out) {
// Reallocate output with
// capacity for total len.
out2 := make([]V, len(out), total)
copy(out2, out)
out = out2
}
// Reslice with capacity
// up to total required.
out = out[:total]
// Gather vs from 'in'.
for i, v := range in {
j := start + i
out[j] = get(v)
}
return out
}
// GatherIf is functionally similar to Gather(), but only when return bool is true.
// If you don't need to check the boolean, Gather() will be very slightly faster.
func GatherIf[T, V any](out []V, in []T, get func(T) (V, bool)) []V {
if get == nil {
panic("nil func")
}
if cap(out)-len(out) < len(in) {
// Reallocate output with capacity for 'in'.
out2 := make([]V, len(out), cap(out)+len(in))
copy(out2, out)
out = out2
}
// Gather vs from 'in'.
for _, v := range in {
if v, ok := get(v); ok {
out = append(out, v)
}
}
return out
}
// Collate will collect the values of type K from input type []T,
// passing each item to 'get' and deduplicating the end result.
// This is equivalent to calling Gather() followed by Deduplicate().
func Collate[T any, K comparable](in []T, get func(T) K) []K {
if get == nil {
panic("nil func")
}
ks := make([]K, 0, len(in))
km := make(map[K]struct{}, len(in))
for i := 0; i < len(in); i++ {
// Get next k.
k := get(in[i])
if _, ok := km[k]; !ok {
// New value, add
// to map + slice.
ks = append(ks, k)
km[k] = struct{}{}
}
}
return ks
}
// OrderBy orders a slice of given type by the provided alternative slice of comparable type.
func OrderBy[T any, K comparable](in []T, keys []K, key func(T) K) {
if key == nil {
panic("nil func")
}
// Create lookup of keys->idx.
m := make(map[K]int, len(in))
for i, k := range keys {
m[k] = i
}
// Sort according to the reverse lookup.
slices.SortFunc(in, func(a, b T) int {
ai := m[key(a)]
bi := m[key(b)]
if ai < bi {
return -1
} else if bi < ai {
return +1
}
return 0
})
}