# Best practices for SDO/HMI imagery cleanup in SunPy? **URL:** \ **Category:** SunPy\ **Tags:** sunpy, fits, question\ **Created:** [May 22, 2024, 5:05pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983 "2024-05-22T17:05:18Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![wtbsvs](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/wtbsvs/32/532_2.png) [@wtbsvs](https://community.openastronomy.org/u/wtbsvs)\ **Post date:** [May 22, 2024, 5:05pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/1 "2024-05-22T17:05:18Z")
The IDL SolarSoft package had AIA\_PREP and HMI\_PREP routines for cleanup of their respective image products to a Level 1.5 data quality suitable for image and movie generation. aiapy has some recommended counterparts for AIA data. Are there recommended steps in SunPy for HMI data? Thanks, Tom ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [May 22, 2024, 5:23pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/2 "2024-05-22T17:23:22Z")
For aiapy, the documentation about the prep code is here: [Preparing AIA data from level 1 to level 1.5 — aiapy 0.8.1.dev1+gc3bd709 documentation](https://aiapy.readthedocs.io/en/latest/preparing_data.html) You can also run HMI files through most of those functions but I don’t think it will do much other than the rotation correction and metadata update. These prep steps are not needed unless you need to ensure co-alignment between each channel or need the updated pointing information. ---
**Author:** ![wtbsvs](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/wtbsvs/32/532_2.png) [@wtbsvs](https://community.openastronomy.org/u/wtbsvs)\ **Post date:** [May 22, 2024, 5:28pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/3 "2024-05-22T17:28:08Z")
I can see update\_pointing and register as probably being compatible between AIA and HMI. But I suspect despike, deconvolve, and correct\_degradation might not be as applicable? Tom ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [May 22, 2024, 5:42pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/4 "2024-05-22T17:42:18Z")
> But I suspect despike, deconvolve, and correct\_degradation might not be as applicable? The AIA files should already be despiked, I assume that is the same for HMI files. But yes those functions will only work for AIA. ---
**Author:** ![wtbsvs](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/wtbsvs/32/532_2.png) [@wtbsvs](https://community.openastronomy.org/u/wtbsvs)\ **Post date:** [May 22, 2024, 5:52pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/5 "2024-05-22T17:52:03Z")
Thanks for the feedback. Tom ---
**Author:** ![wtbsvs](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/wtbsvs/32/532_2.png) [@wtbsvs](https://community.openastronomy.org/u/wtbsvs)\ **Post date:** [May 31, 2024, 12:44pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/6 "2024-05-31T12:44:13Z")
Okay, this does not seem to be valid. The T\_OBS keyword seems to use a different time format between AIA and HMI. ```auto dataViewPupdate = update_pointing(dataViewLevel1) # update pointing information ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/aiapy/calibrate/meta.py", line 117, in update_pointing t_obs = astropy.time.Time(t_obs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/astropy/time/core.py", line 1977, in __init__ self._init_from_vals( File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/astropy/time/core.py", line 548, in _init_from_vals self._time = self._get_time_fmt( ^^^^^^^^^^^^^^^^^^^ File "/Users/wbridgma/.conda/envs/python311/lib/python3.11/site-packages/astropy/time/core.py", line 644, in _get_time_fmt raise ValueError(message) ValueError: Input values did not match any of the formats where the format keyword is optional: - 'datetime': Input values for datetime class must be datetime objects - 'ymdhms': input must be dict or table-like - 'iso': could not convert string to float: '.898_TAI' - 'isot': could not convert string to float: '.898_TAI' - 'yday': could not convert string to float: '.898_TAI' - 'datetime64': Input values for datetime64 class must be datetime64 objects - 'fits': Time 2019.09.15_00:00:02.898_TAI does not match fits format - 'byear_str': Time 2019.09.15_00:00:02.898_TAI does not match byear_str format - 'jyear_str': Time 2019.09.15_00:00:02.898_TAI does not match jyear_str format - 'astropy_time': Input values for astropy_time class must all be the same astropy Time type. ``` ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [May 31, 2024, 7:24pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/7 "2024-05-31T19:24:32Z")
Ah my unfamiliarity with HMI data strikes again. It feels like a bug somewhere (either in sunpy or in aiapy) for those times not to be parsed. I will need to investigate. For now, I would bypass that for the HMI images and just rotate them. ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [June 3, 2024, 5:40am UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/8 "2024-06-03T05:40:21Z")
I opened a PR: [Added extra HMI TAI JSOC format with micro(or milli???) seconds by nabobalis · Pull Request #7656 · sunpy/sunpy · GitHub](https://github.com/sunpy/sunpy/pull/7656) It adds support for that time format in sunpy but it won’t come out until 6.0 (hopefully later this month). ---
**Author:** ![wtbsvs](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/wtbsvs/32/532_2.png) [@wtbsvs](https://community.openastronomy.org/u/wtbsvs)\ **Post date:** [June 3, 2024, 10:42am UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/9 "2024-06-03T10:42:36Z")
Thanks. The SolarSoft package had fairly symmetric support functions between AIA and HMI so it was reasonably straightforward to do equivalent processing on them for media use. Open Source gets written by those who need it and have the background to do it, so functionally ‘accretes’ more that developed around an overall philosophy. ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [June 3, 2024, 3:08pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/10 "2024-06-03T15:08:02Z")
Do you have a link to documentation or the names of the IDL functions which are run on the HMI data? I would like to work out what we are missing in aiapy. ---
**Author:** ![wtbsvs](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/wtbsvs/32/532_2.png) [@wtbsvs](https://community.openastronomy.org/u/wtbsvs)\ **Post date:** [June 3, 2024, 3:48pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/11 "2024-06-03T15:48:17Z")
Seems difficult to find those old docs online. SolarSoft generally had functions like AIA\_PREP, HMI\_PREP, STEREO\_PREP, etc. that processed Level 1 data to Level 1.5 with various processing options. I found AIA\_PREP at GSFC: [https://hesperia.gsfc.nasa.gov/ssw/sdo/aia/idl/calibration/aia\_prep.pro](https://hesperia.gsfc.nasa.gov/ssw/sdo/aia/idl/calibration/aia_prep.pro) Here’s an _old_ tutorial: [https://hesperia.gsfc.nasa.gov/~bdennis/Folders/Missions/SDO/SDOD0060\_N\_Guide\_to\_SDO\_Data\_Analysis\_sdoguide.pdf](https://hesperia.gsfc.nasa.gov/~bdennis/Folders/Missions/SDO/SDOD0060_N_Guide_to_SDO_Data_Analysis_sdoguide.pdf) A few weeks ago I tried to update my local SolarSoft copy but could not get it to run sufficiently to update itself. I suspect this is a local configuration issue as I run software for so many different missions, some do not play nice on the same machine/account so I have to try funky ways to keep them from stepping on each other. ;^) I probably still have some of the old IDL codes that ran these processes if that might be any help. Thanks, Tom ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [June 3, 2024, 5:38pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/12 "2024-06-03T17:38:21Z")
As far as I understand, aiapy has the aiaprep already re-implemented in discrete steps. It is the HMI side that while in theory we support in some manner, it wasn’t part of the original goal of the library to support fully. I would like to improve on that in some manner in the future. Thanks for the links, I will need to find hmi\_prep and see what it does and see how involved it gets. ---
**Author:** ![mbobra](https://avatars.discourse-cdn.com/v4/letter/m/ee59a6/32.png) [@mbobra](https://community.openastronomy.org/u/mbobra)\ **Post date:** [June 3, 2024, 10:29pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/13 "2024-06-03T22:29:34Z")
@nabobalis Don’t forget about [drms](https://docs.sunpy.org/projects/drms/en/stable/tutorial.html#basic-usage)! It also has support for T\_REC. ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [June 3, 2024, 10:41pm UTC](https://community.openastronomy.org/t/best-practices-for-sdo-hmi-imagery-cleanup-in-sunpy/983/14 "2024-06-03T22:41:13Z")
Ah yes, I forgot! Thanks for the reminder.