Fido.search and the JSOC Series tag - why the differences?

SunPy
,
1

I have some python scripts for routinely generating various full-cadence movies from SDO using SunPy, preferably ASAP after the event for use in various Helio stories for OCOMM. In the process of doing some improvements to the scripts, I encountered some confusing behavior.

If I run Fido.search with attrs.jsoc.Series(‘aia.lev1_euv_12s’) I can select on the wavelength and time sampling, but the Level 1 data does not appear to be available until 6-7 days after the event.

However if I run Fido.search with attrs.jsoc.Series(‘aia.lev1’), I can’t select on the wavelength and time sampling, but the Level 1 data is available just 3-4 days after the event, and will a bunch of wavelengths available.

Why the difference? What detail am I missing on how this works?

Is there a way to select a subset of the records returned by ‘aia_lev1’ to download by Fido.fetch()? I’ve not found a clear example of doing this. Does Fido return a pandas dataframe?

Alternatively, is there a better way to select and retrieve Level 1 SDO data via python?

Thanks,
Tom
Scientific Visualization Studio

2

Hello @wtbsvs,

Why the difference? What detail am I missing on how this works?

So the JSOC define their series as the following:

AIA Level 1:aia.lev1

This includes all the Level 1 data corresponding to all ten wavelength channels on the AIA. Any combination of FITS header keywords FSN and T_REC is unique for the DRMS series aia.lev1.

AIA Level 1, 12 second cadence: aia.lev1_euv_12s

This series includes data from the soft xray - ultraviolet wavelength band taken at a cadence of 12s.So this series includes data corresponding to wavelengths 94,131,171,193, 211,304 and 335 Å. Any combination of FITS header keywords T_REC and WAVELNTH are unique for the aia.lev1

So you are in fact not missing anything, this is just how those series are defined.

Is there a way to select a subset of the records returned by ‘aia_lev1’ to download by Fido.fetch()? I’ve not found a clear example of doing this. Does Fido return a pandas dataframe?

Unfortunately, the JSOC does not support post search filtering, you get only what the original query is, nothing more, nothing less.

Alternatively, is there a better way to select and retrieve Level 1 SDO data via python?

If you need the quickest data, I think you will need to ask the JSOC directly for access to any of the NRT series like, aia.lev1_nrt2.

What you can do is use the VSO (I only learned this today from
alasdairwilson), if you search for AIA data for today, it will return a 1k by 1k level 1.5 NRT data that is eventually replaced by the fully processed data.

Fido.search(a.Time('2024-4-4', '2024-4-4'), a.Level("1.5"), a.Instrument.aia)

You do not need the Level attr but it showcases that its labeled as such.

Hopefully that helps.

3

Unfortunately NRT and 1Kx1K is not adequate. I really do need to get the full 4K with all the updated tags for co-registration since these products are often cropped and combined in different ways by myself or the video producers so we want the best resolution possible.

Our real-quick-turnaround needs usually rely on the QuickLook product. For the high-quality product, the 3-4 day lag of aia_lev1 is okay, but the 6-7 day lag of aia_lev1_euv_12s gets a bit tricky to manage when processing multiple datasets per week.

IIRC SolarSoft used the VSO but the downloaded file name convention is often different so converting to Level 1.5 gets trickly.

So if I understand you correctly, there is no way to fetch a subset of products returned by aia_lev1 series?

Thanks. That at least clarifies some of my difficulties.

Tom
Scientific Visualization Studio

4

So if I understand you correctly, there is no way to fetch a subset of products returned by aia_lev1 series?

As far as I am aware, no, you need the query to be exact for the subset of data you are after. I do not think this is supported by their website either.

However, maybe the team behind JSOC will know of something?