Is our universe decaying at an astronomical rate?

Unless our universe is decaying at an astronomical rate (i.e., on the present cosmological timescale of Gigayears, rather than on the quantum recurrence timescale of googolplexes), it would apparently produce an infinite number of observers per comoving volume by thermal or vacuum fluctuations (Boltzmann brains). If the number of ordinary observers per comoving volume is finite, this scenario seems to imply zero likelihood for us to be ordinary observers and minuscule likelihoods for our actual observations. Hence, our observations suggest that this scenario is incorrect and that perhaps our universe is decaying at an astronomical rate.