[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
											 
										 
										
											2023-03-28 14:03:14 +01:00 
										
									 
								 
							 
							
								
							 
							
								 
							
							
								// GoToSocial  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// Copyright (C) GoToSocial Authors admin@gotosocial.org  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// SPDX-License-Identifier: AGPL-3.0-or-later  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								//  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// This program is free software: you can redistribute it and/or modify  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// it under the terms of the GNU Affero General Public License as published by  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// the Free Software Foundation, either version 3 of the License, or  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// (at your option) any later version.  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								//  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// This program is distributed in the hope that it will be useful,  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// but WITHOUT ANY WARRANTY; without even the implied warranty of  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// GNU Affero General Public License for more details.  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								//  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// You should have received a copy of the GNU Affero General Public License  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// along with this program.  If not, see <http://www.gnu.org/licenses/>.  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								package  visibility  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								import  (  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									"context" 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									"fmt" 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									"github.com/superseriousbusiness/gotosocial/internal/cache" 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									"github.com/superseriousbusiness/gotosocial/internal/gtsmodel" 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									"github.com/superseriousbusiness/gotosocial/internal/log" 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								)  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// StatusesVisible calls StatusVisible for each status in the statuses slice, and returns a slice of only statuses which are visible to the requester.  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								func  ( f  * Filter )  StatusesVisible ( ctx  context . Context ,  requester  * gtsmodel . Account ,  statuses  [ ] * gtsmodel . Status )  ( [ ] * gtsmodel . Status ,  error )  {  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									// Preallocate slice of maximum possible length. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									filtered  :=  make ( [ ] * gtsmodel . Status ,  0 ,  len ( statuses ) ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									for  _ ,  status  :=  range  statuses  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Check whether status is visible to requester. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										visible ,  err  :=  f . StatusVisible ( ctx ,  requester ,  status ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  err  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  nil ,  err 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  visible  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											// Add filtered status to ret slice. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											filtered  =  append ( filtered ,  status ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									return  filtered ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								}  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// StatusVisible will check if given status is visible to requester, accounting for requester with no auth (i.e is nil), suspensions, disabled local users, account blocks and status privacy.  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								func  ( f  * Filter )  StatusVisible ( ctx  context . Context ,  requester  * gtsmodel . Account ,  status  * gtsmodel . Status )  ( bool ,  error )  {  
						 
					
						
							
								
									
										
										
										
											2023-04-06 16:11:25 +01:00 
										
									 
								 
							 
							
								
									
										 
								
							 
							
								 
							
							
									const  vtype  =  cache . VisibilityTypeStatus 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
									
										
											 
										
											
												[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
											 
										 
										
											2023-03-28 14:03:14 +01:00 
										
									 
								 
							 
							
								
							 
							
								 
							
							
									// By default we assume no auth. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									requesterID  :=  noauth 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  requester  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Use provided account ID. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										requesterID  =  requester . ID 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									visibility ,  err  :=  f . state . Caches . Visibility . Load ( "Type.RequesterID.ItemID" ,  func ( )  ( * cache . CachedVisibility ,  error )  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Visibility not yet cached, perform visibility lookup. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										visible ,  err  :=  f . isStatusVisible ( ctx ,  requester ,  status ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  err  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  nil ,  err 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Return visibility value. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  & cache . CachedVisibility { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											ItemID :       status . ID , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											RequesterID :  requesterID , 
							 
						 
					
						
							
								
									
										
										
										
											2023-04-06 16:11:25 +01:00 
										
									 
								 
							 
							
								
									
										 
								
							 
							
								 
							
							
											Type :         vtype , 
							 
						 
					
						
							
								
									
										
											 
										
											
												[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
											 
										 
										
											2023-03-28 14:03:14 +01:00 
										
									 
								 
							 
							
								
							 
							
								 
							
							
											Value :        visible , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} ,  nil 
							 
						 
					
						
							
								
									
										
										
										
											2023-04-06 16:11:25 +01:00 
										
									 
								 
							 
							
								
									
										 
								
							 
							
								 
							
							
									} ,  vtype ,  requesterID ,  status . ID ) 
							 
						 
					
						
							
								
									
										
											 
										
											
												[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
											 
										 
										
											2023-03-28 14:03:14 +01:00 
										
									 
								 
							 
							
								
							 
							
								 
							
							
									if  err  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  false ,  err 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									return  visibility . Value ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								}  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// isStatusVisible will check if status is visible to requester. It is the "meat" of the logic to Filter{}.StatusVisible() which is called within cache loader callback.  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								func  ( f  * Filter )  isStatusVisible ( ctx  context . Context ,  requester  * gtsmodel . Account ,  status  * gtsmodel . Status )  ( bool ,  error )  {  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									// Ensure that status is fully populated for further processing. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  err  :=  f . state . DB . PopulateStatus ( ctx ,  status ) ;  err  !=  nil  { 
							 
						 
					
						
							
								
									
										
										
										
											2023-04-06 16:11:25 +01:00 
										
									 
								 
							 
							
								
									
										 
								
							 
							
								 
							
							
										return  false ,  fmt . Errorf ( "isStatusVisible: error populating status %s: %w" ,  status . ID ,  err ) 
							 
						 
					
						
							
								
									
										
											 
										
											
												[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
											 
										 
										
											2023-03-28 14:03:14 +01:00 
										
									 
								 
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									// Check whether status accounts are visible to the requester. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									visible ,  err  :=  f . areStatusAccountsVisible ( ctx ,  requester ,  status ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  err  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  false ,  fmt . Errorf ( "isStatusVisible: error checking status %s account visibility: %w" ,  status . ID ,  err ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									}  else  if  ! visible  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  status . Visibility  ==  gtsmodel . VisibilityPublic  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// This status will be visible to all. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  requester  ==  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// This request is WITHOUT auth, and status is NOT public. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										log . Trace ( ctx ,  "unauthorized request to non-public status" ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  status . Visibility  ==  gtsmodel . VisibilityUnlocked  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// This status is visible to all auth'd accounts. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  requester . ID  ==  status . AccountID  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Author can always see their own status. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  status . MentionsAccount ( requester . ID )  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Status mentions the requesting account. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  status . BoostOf  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  ! status . BoostOf . MentionsPopulated ( )  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											// Boosted status needs its mentions populating, fetch these from database. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											status . BoostOf . Mentions ,  err  =  f . state . DB . GetMentions ( ctx ,  status . BoostOf . MentionIDs ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											if  err  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
												return  false ,  fmt . Errorf ( "isStatusVisible: error populating boosted status %s mentions: %w" ,  status . BoostOfID ,  err ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  status . BoostOf . MentionsAccount ( requester . ID )  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											// Boosted status mentions the requesting account. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									switch  status . Visibility  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									case  gtsmodel . VisibilityFollowersOnly : 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Check requester follows status author. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										follows ,  err  :=  f . state . DB . IsFollowing ( ctx , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											requester . ID , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											status . AccountID , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  err  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  false ,  fmt . Errorf ( "isStatusVisible: error checking follow %s->%s: %w" ,  requester . ID ,  status . AccountID ,  err ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  ! follows  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											log . Trace ( ctx ,  "follow-only status not visible to requester" ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									case  gtsmodel . VisibilityMutualsOnly : 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Check mutual following between requester and author. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										mutuals ,  err  :=  f . state . DB . IsMutualFollowing ( ctx , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											requester . ID , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											status . AccountID , 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  err  !=  nil  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  false ,  fmt . Errorf ( "isStatusVisible: error checking mutual follow %s<->%s: %w" ,  requester . ID ,  status . AccountID ,  err ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  ! mutuals  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											log . Trace ( ctx ,  "mutual-only status not visible to requester" ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									case  gtsmodel . VisibilityDirect : 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										log . Trace ( ctx ,  "direct status not visible to requester" ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									default : 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										log . Warnf ( ctx ,  "unexpected status visibility %s for %s" ,  status . Visibility ,  status . URI ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								}  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								// areStatusAccountsVisible calls Filter{}.AccountVisible() on status author and the status boost-of (if set) author, returning visibility of status (and boost-of) to requester.  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								func  ( f  * Filter )  areStatusAccountsVisible ( ctx  context . Context ,  requester  * gtsmodel . Account ,  status  * gtsmodel . Status )  ( bool ,  error )  {  
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									// Check whether status author's account is visible to requester. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									visible ,  err  :=  f . AccountVisible ( ctx ,  requester ,  status . Account ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  err  !=  nil  { 
							 
						 
					
						
							
								
									
										
										
										
											2023-04-06 16:11:25 +01:00 
										
									 
								 
							 
							
								
									
										 
								
							 
							
								 
							
							
										return  false ,  fmt . Errorf ( "error checking status author visibility: %w" ,  err ) 
							 
						 
					
						
							
								
									
										
											 
										
											
												[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
											 
										 
										
											2023-03-28 14:03:14 +01:00 
										
									 
								 
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  ! visible  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										log . Trace ( ctx ,  "status author not visible to requester" ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									if  status . BoostOfID  !=  ""  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// This is a boosted status. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  status . AccountID  ==  status . BoostOfAccountID  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											// Some clout-chaser boosted their own status, tch. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										// Check whether boosted status author's account is visible to requester. 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										visible ,  err  :=  f . AccountVisible ( ctx ,  requester ,  status . BoostOfAccount ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  err  !=  nil  { 
							 
						 
					
						
							
								
									
										
										
										
											2023-04-06 16:11:25 +01:00 
										
									 
								 
							 
							
								
									
										 
								
							 
							
								 
							
							
											return  false ,  fmt . Errorf ( "error checking boosted author visibility: %w" ,  err ) 
							 
						 
					
						
							
								
									
										
											 
										
											
												[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
											 
										 
										
											2023-03-28 14:03:14 +01:00 
										
									 
								 
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										if  ! visible  { 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											log . Trace ( ctx ,  "boosted status author not visible to requester" ) 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
											return  false ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
										} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									} 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
									return  true ,  nil 
							 
						 
					
						
							
								
							 
							
								
							 
							
								 
							
							
								}