As a note, Python 2.7 has been deprecated (meaning getting neither bug nor security fixes) for almost 4 years and is no longer supported by Astropy (i.e. wherever you installed it from you must have got a very old version).
Python 3.0 to my knowledge has never been in production use, so in case you might ever want to give Astropy another try, you should try to setup an installation with at least Python 3.9 or whatever is the latest version you can get an installation for on your Pi.
As for how to (safely) remove the installed folder, that is not really an Astropy-specific or even very Python-related question. It seems if your machine only has a Python 3.x installation, there should exist no python2.7/site-packages at all, but to find out more about the Unix shell on your system and how to use its commands, you’d best ask in a Raspberry Pi forum. That said, if conftest.py is the only file or subdirectory you see in that astropy folder, there might not be much to worry about, as it takes up insignificant space.