Best practices for SDO/HMI imagery cleanup in SunPy?

SunPy
, ,
1

The IDL SolarSoft package had AIA_PREP and HMI_PREP routines for cleanup of their respective image products to a Level 1.5 data quality suitable for image and movie generation.

aiapy has some recommended counterparts for AIA data.

Are there recommended steps in SunPy for HMI data?

Thanks,
Tom

2

For aiapy, the documentation about the prep code is here: Preparing AIA data from level 1 to level 1.5 — aiapy 0.8.1.dev1+gc3bd709 documentation

You can also run HMI files through most of those functions but I don’t think it will do much other than the rotation correction and metadata update.

These prep steps are not needed unless you need to ensure co-alignment between each channel or need the updated pointing information.

3

I can see update_pointing and register as probably being compatible between AIA and HMI.

But I suspect despike, deconvolve, and correct_degradation might not be as applicable?

Tom

4

But I suspect despike, deconvolve, and correct_degradation might not be as applicable?

The AIA files should already be despiked, I assume that is the same for HMI files.

But yes those functions will only work for AIA.

5

Thanks for the feedback.

Tom

6

Okay, this does not seem to be valid. The T_OBS keyword seems to use a different time format between AIA and HMI.

dataViewPupdate = update_pointing(dataViewLevel1) # update pointing information
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/aiapy/calibrate/meta.py", line 117, in update_pointing
    t_obs = astropy.time.Time(t_obs)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/astropy/time/core.py", line 1977, in __init__
    self._init_from_vals(
  File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/astropy/time/core.py", line 548, in _init_from_vals
    self._time = self._get_time_fmt(
                 ^^^^^^^^^^^^^^^^^^^
  File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/astropy/time/core.py", line 644, in _get_time_fmt
    raise ValueError(message)
ValueError: Input values did not match any of the formats where the format keyword is optional:
- 'datetime': Input values for datetime class must be datetime objects
- 'ymdhms': input must be dict or table-like
- 'iso': could not convert string to float: '.898_TAI'
- 'isot': could not convert string to float: '.898_TAI'
- 'yday': could not convert string to float: '.898_TAI'
- 'datetime64': Input values for datetime64 class must be datetime64 objects
- 'fits': Time 2019.09.15_00:00:02.898_TAI does not match fits format
- 'byear_str': Time 2019.09.15_00:00:02.898_TAI does not match byear_str format
- 'jyear_str': Time 2019.09.15_00:00:02.898_TAI does not match jyear_str format
- 'astropy_time': Input values for astropy_time class must all be the same astropy Time type.
7

Ah my unfamiliarity with HMI data strikes again.

It feels like a bug somewhere (either in sunpy or in aiapy) for those times not to be parsed.
I will need to investigate.

For now, I would bypass that for the HMI images and just rotate them.

8

I opened a PR: Added extra HMI TAI JSOC format with micro(or milli???) seconds by nabobalis · Pull Request #7656 · sunpy/sunpy · GitHub

It adds support for that time format in sunpy but it won’t come out until 6.0 (hopefully later this month).

9

Thanks.
The SolarSoft package had fairly symmetric support functions between AIA and HMI so it was reasonably straightforward to do equivalent processing on them for media use.

Open Source gets written by those who need it and have the background to do it, so functionally ‘accretes’ more that developed around an overall philosophy.

10

Do you have a link to documentation or the names of the IDL functions which are run on the HMI data?

I would like to work out what we are missing in aiapy.

11

Seems difficult to find those old docs online.
SolarSoft generally had functions like AIA_PREP, HMI_PREP, STEREO_PREP, etc. that processed Level 1 data to Level 1.5 with various processing options.

I found AIA_PREP at GSFC: https://hesperia.gsfc.nasa.gov/ssw/sdo/aia/idl/calibration/aia_prep.pro

Here’s an old tutorial: https://hesperia.gsfc.nasa.gov/~bdennis/Folders/Missions/SDO/SDOD0060_N_Guide_to_SDO_Data_Analysis_sdoguide.pdf

A few weeks ago I tried to update my local SolarSoft copy but could not get it to run sufficiently to update itself. I suspect this is a local configuration issue as I run software for so many different missions, some do not play nice on the same machine/account so I have to try funky ways to keep them from stepping on each other. ;^)

I probably still have some of the old IDL codes that ran these processes if that might be any help.

Thanks,
Tom

12

As far as I understand, aiapy has the aiaprep already re-implemented in discrete steps.

It is the HMI side that while in theory we support in some manner, it wasn’t part of the original goal of the library to support fully. I would like to improve on that in some manner in the future.

Thanks for the links, I will need to find hmi_prep and see what it does and see how involved it gets.

13

@nabobalis Don’t forget about drms! It also has support for T_REC.

1 Like
14

Ah yes, I forgot! Thanks for the reminder.