[performance] improved request batching (removes need for queueing) (#1687)

* revamp http client to not limit requests, instead use sender worker

Signed-off-by: kim <grufwub@gmail.com>

* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping

Signed-off-by: kim <grufwub@gmail.com>

* improve batch senders to keep popping recipients until remote URL found

Signed-off-by: kim <grufwub@gmail.com>

* fix recipient looping issue

Signed-off-by: kim <grufwub@gmail.com>

* fix missing mutex unlock

Signed-off-by: kim <grufwub@gmail.com>

* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP

Signed-off-by: kim <grufwub@gmail.com>

* slight code reformatting

Signed-off-by: kim <grufwub@gmail.com>

* a whitespace

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code

Signed-off-by: kim <grufwub@gmail.com>

* add missing license headers

Signed-off-by: kim <grufwub@gmail.com>

* fix request backoff calculation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>

internal/workers/workers.go

@@ -31,8 +31,12 @@ type Workers struct {
 	// Main task scheduler instance.
 	Scheduler sched.Scheduler
 
-	// ClientAPI / federator worker pools.
+	// ClientAPI provides a worker pool that handles both
+	// incoming client actions, and our own side-effects.
 	ClientAPI runners.WorkerPool
+
+	// Federator provides a worker pool that handles both
+	// incoming federated actions, and our own side-effects.
 	Federator runners.WorkerPool
 
 	// Enqueue functions for clientAPI / federator worker pools,