I’m having an issue with doing a simple fits.open on some CCD taken fits files. A few months ago I was able to open these files and do analysis on them but when I tried to run the code again last week I got the above error. I was worried something may have happened to the old fits fits so I used some newly taken ones from the day before and got the same error. Worried all my files were somehow corrupted I downloaded a third party fits file viewer (AvisFV) to see if a different software could open them and luckily it did! The images look good and I see the HDU on the bottom so I don’t believe anything is wrong with my actual files. I’m worried that the astropy upgrade is somehow causing the issue when trying to run it. Since I do have to do analysis on these files it’s not enough for me to view them, which is why I need astropy but it keeps throwing me the corruption error. I looked through the astropy 6.0 upgrades and it didn’t mention anything about syntax differences so I’m still just using the command fits.open(filename, ignore_missing_simple=True). I have to include that second condition because otherwise I get an error stating that the file doesn’t seem to be a valid FITS file. If anybody knows how to work around this I would greatly appreciate it!
Are you getting a warning message like “The HDU will be treated as corrupted”?
Then there should be a warning preceding that giving more details on the exception occurring when trying to parse the file; if you could post that, we may be able to narrow the issue down.
When trying to run an older FITS file (only a couple months old) there’s no errors at all just the message at the bottom saying “OSError:Empty or Corrupt FITS files” and they ran totally fine a couple months ago, I didn’t change any of the code either. When using the new files, I get these warning messages: “WARNING: VerifyWarning: Error validating header for HDU #0 (note: Astropy uses zero-based indexing).
Header size is not multiple of 2880: 4736
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]”. The one difference between “old” and “new” file sets is that the old ones are a simple one frame, one exposure in each file but in the new ones, each individual FITS file is an average of 100 exposures which I’m assuming is the reason for the header warnings. I looked to see what the header looks like in AvisFV for my most recent files (that the warnings in astropy show up for) and this is the table that shows up:
SIMPLE = T / file does conform to FITS standard
BITPIX = -32 / number of bits per data pixel
NAXIS = 3 / number of data axes
NAXIS1 = 1024 / length of data axis 1
NAXIS2 = 1024 / length of data axis 2
NAXIS3 = 1 / length of data axis 3
EXTEND = T / FITS dataset may contain extensions
COMMENT FITS (Flexible Image Transport System) format is defined in ‘Astronomy
COMMENT and Astrophysics’, volume 376, page 359; bibcode: 2001A&A…376…359H
DATE = ‘2023-12-19T16:23:25’ / file creation date (YYYY-MM-DDThh:mm:ss UT)
DATE-OBS= ‘2023-12-19T16:23:23’ / Date of data acquisition
EXPTIME = '10 ’ / Exposure time
END
This is becoming confusing – you are getting the error message above (it is an error, not a warning, so normally one would expect the read operation to fail after that), yet you have “no errors at all” with those older files?
That error message itself is not part of any Astropy code though, so you will need to provide an example of the full list of commands you are using to open those files, ideally also post a link to an example file, if you can share them.
That is a warning indeed, and as it indicates, from the astropy.io.fits module, meaning that (some part of) the file is not conforming to the FITS standard. However in such cases Astropy will make any attempt to read the file as far as possible, and if there have been no further error messages after that, it should be read.
Which software has created these files, and was there a version change between “old” and “new” ones?
Averaging as such should not cause a problem; from what I can read from the header info, this simply means the file contains one image that has already been averaged from the 100 exposures (it indicates there is a single image frame of 1024*1024 pixels). Or does it also include the full 100 individual exposures in additional extensions? That should still not cause the read to fail, but most functions in io.fits will only read the first HDU unless you explicitly indicate another one.
But what AvisFV has extracted then cannot be the complete header, as it only contains 13 lines of 13*80 bytes, way less than a single header block of 36*80=2880 bytes.
Again, seeing an example of your script and ideally the file would be necessary to track those problems down.