I feel like there's a lot of FUD around this subject, because people bring it up as if it's purely a negative without talking about the reasons why it's done the way it is. The whole point of F-Droid is that it's a repository (not a store) of free software applications. They have an inclusion policy forbidding proprietary code and dependencies, and in order to enforce this policy they have to build from publicly available source code, and in order to do so they need to sign the builds themselves. This means, yes, you are trusting F-Droid instead of the upstream developer - but given F-Droid has higher standards than upstream developers this is a tradeoff I am willing to make.
Reproducible builds solves this in a way that preserves the standards of F-Droid, however, "security peoples'" favored "alternatives" (such as Accrescent, Obtainium, and Google Play Store/Aurora Store) forego this entirely, showing they don't either have a viable solution to offer or that they don't really care about the problem that F-Droid is addressing to begin with.