Tantra is called the worldly path because it’s meant for the common householder, rather than the renunciate or the monk in particular. While there are many different schools in Tantra and different Tantric texts with more ascetic practices, there are many more that are truly meant for everyone as an equal opportunity on the path to enlightenment.
You don’t have to leave the material world and in fact sometimes this causes suffering for many people who feel that they have to choose and therefore create fragmentation within themselves.
If you are totally living in your higher centers and in the astral worlds then you lose touch with this reality. This happens very often with people who abuse sacred plant medicines and don’t integrate back into this reality and with people who fully isolate themselves from this reality (avoiding people) without proper knowledge of how to reconcile their inner world with their outer experience.
All paths leads to the same destination. Spiritual lessons will come about whether you take one road vs another, but more often than not neglecting this physical reality and checking out delays the constructive use of the time element and leads to problems when you inevitably check back in. All aspects of your experience are part of the Self.
Take care of the practical and you can be even more forthright in the spiritual. Wisdom is often in the central point, the middle path beyond duality. Too much spirit and you become a non living entity in the Earth plane, you forget about the human experience. You end up avoiding your challenges rather than winning them. Too much material and you are a robot that has forgotten their connection to the bigger picture. Stay in the center.