[feature] Clean up/uncache remote media (#407)

* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
This commit is contained in:
tobi 2022-03-07 11:08:26 +01:00 committed by GitHub
commit 07727753b9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
424 changed files with 637100 additions and 176498 deletions

View file

@ -2,4 +2,6 @@
package uuid
var CAPI = map[string]struct{}{}
var CAPI = map[string]struct{}{
"__darwin_check_fd_set_overflow": {},
}

View file

@ -2,4 +2,6 @@
package uuid
var CAPI = map[string]struct{}{}
var CAPI = map[string]struct{}{
"__darwin_check_fd_set_overflow": {},
}

View file

@ -51,6 +51,8 @@ type X__uint128_t = struct {
type X__builtin_va_list = uintptr /* <builtin>:46:14 */
type X__float128 = float64 /* <builtin>:47:21 */
var X__darwin_check_fd_set_overflow uintptr /* <builtin>:146:5: */
// Public include file for the UUID library
//
// Copyright (C) 1996, 1997, 1998 Theodore Ts'o.
@ -183,6 +185,13 @@ type X__float128 = float64 /* <builtin>:47:21 */
// in between its arguments. __CONCAT can also concatenate double-quoted
// strings produced by the __STRING macro, but this only works with ANSI C.
// In non-ANSI C environments, new programs will want ANSI-only C keywords
// deleted from the program and old programs will want them left alone.
// When using a compiler other than gcc, programs using the ANSI C keywords
// const, inline etc. as normal identifiers should define -DNO_ANSI_KEYWORDS.
// When using "gcc -traditional", we assume that this is the intent; if
// __GNUC__ is defined but __STDC__ is not, we leave the new keywords alone.
// __unused denotes variables and functions that may not be used, preventing
// the compiler from warning about it if not used.

View file

@ -53,6 +53,8 @@ type X__uint128_t = struct {
type X__builtin_va_list = uintptr /* <builtin>:46:14 */
type X__float128 = float64 /* <builtin>:47:21 */
var X__darwin_check_fd_set_overflow uintptr /* <builtin>:146:5: */
// Public include file for the UUID library
//
// Copyright (C) 1996, 1997, 1998 Theodore Ts'o.
@ -185,6 +187,13 @@ type X__float128 = float64 /* <builtin>:47:21 */
// in between its arguments. __CONCAT can also concatenate double-quoted
// strings produced by the __STRING macro, but this only works with ANSI C.
// In non-ANSI C environments, new programs will want ANSI-only C keywords
// deleted from the program and old programs will want them left alone.
// When using a compiler other than gcc, programs using the ANSI C keywords
// const, inline etc. as normal identifiers should define -DNO_ANSI_KEYWORDS.
// When using "gcc -traditional", we assume that this is the intent; if
// __GNUC__ is defined but __STDC__ is not, we leave the new keywords alone.
// __pure2 can be used for functions that are only a function of their scalar
// arguments (meaning they can't dereference pointers).
//