That’s like thinking the soul is in the body. In fact, the body is in the soul. So says St. Thomas Aquinas. And since time measures the movements of material bodies, while prayer measures the movements of the soul, time is really in prayer rather than prayer in time. Prayer determines and changes and miraculously multiplies time (the loaves and fishes).
But prayer multiplies time only if and when we sacrifice our time, offer it up. There’s the rub. We fear sacrifice. It’s a kind of death.