# Downloading HMI intensity grams to match a set of HARPS-N data timestamps **URL:** \ **Category:** SunPy\ **Tags:** sunpy\ **Created:** [March 23, 2025, 5:07am UTC](https://community.openastronomy.org/t/downloading-hmi-intensity-grams-to-match-a-set-of-harps-n-data-timestamps/1219 "2025-03-23T05:07:43Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Petra63](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/petra63/32/757_2.png) [@Petra63](https://community.openastronomy.org/u/Petra63)\ **Post date:** [March 23, 2025, 5:07am UTC](https://community.openastronomy.org/t/downloading-hmi-intensity-grams-to-match-a-set-of-harps-n-data-timestamps/1219/1 "2025-03-23T05:07:43Z")
I am attempting to download a suite of hmi\_Ic\_45s files to match HARPS-N solar observation timestamps. Is there a simple python code that has been created to do this. I have started to write a code that will do this, but I keep getting 70+ hmi files and I only have 64 HARPS-N files. Am I pushing a cart sideways up a hill. Cheers ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [March 23, 2025, 11:57am UTC](https://community.openastronomy.org/t/downloading-hmi-intensity-grams-to-match-a-set-of-harps-n-data-timestamps/1219/2 "2025-03-23T11:57:14Z")
Hello @Petra63, I am not sure if what I will say will help but if you just need to download HMI files for that specific series, you should be able to use sunpy’s Fido to get that. I forgot if the Virtual Solar Obseractory directly carries that series but you can use [Finding and Downloading Data from JSOC — sunpy 6.1.1 documentation](https://docs.sunpy.org/en/stable/tutorial/acquiring_data/jsoc.html) to get that dataset for sure. Cheers ---
**Author:** ![Petra63](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/petra63/32/757_2.png) [@Petra63](https://community.openastronomy.org/u/Petra63)\ **Post date:** [March 24, 2025, 1:24am UTC](https://community.openastronomy.org/t/downloading-hmi-intensity-grams-to-match-a-set-of-harps-n-data-timestamps/1219/3 "2025-03-24T01:24:19Z")
Hi there and thank you for the response I am basically following all the best practices from the SunPy docs: specifying the series, restricting to “continuum,” picking a single camera, filtering by “QUALITY=0,” and retrieving only the closest record in time. The link does not reveal anything new for my situation beyond that: I think I am covering everything. Why I am getting more than 64 test files?? maybe it’s typically one of: 1. **Multiple files** in the same row (two continuum versions). 2. **Some leftover** from a previous run. 3. **Reprocessed** vs. near‐realtime versions both being offered by the JSOC. ill try printing the file URLs that I fetch for each HARPS time, to see exactly where the “extra 12” come from—and then I can decide how to filter them out or ignore them. BUT having said that if anyone has any other suggestions that will allow me to just download the hmi for each HARPSN observation exactly - I would be thrilled. : -) Cheers ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [March 24, 2025, 5:12am UTC](https://community.openastronomy.org/t/downloading-hmi-intensity-grams-to-match-a-set-of-harps-n-data-timestamps/1219/4 "2025-03-24T05:12:26Z")
Can you provide the code you are running? I will see if I can work out what is going on. ---
**Author:** ![Petra63](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/petra63/32/757_2.png) [@Petra63](https://community.openastronomy.org/u/Petra63)\ **Post date:** [March 24, 2025, 10:42am UTC](https://community.openastronomy.org/t/downloading-hmi-intensity-grams-to-match-a-set-of-harps-n-data-timestamps/1219/5 "2025-03-24T10:42:13Z")
#!/usr/bin/env python3 “”" This script matches HMI Ic\_45s data (camera=2, continuum only) to HARPS-N observations for 2015-09-01, ensuring QUALITY == 0. It then writes a CSV of matched times and files. It also disables most logging and progress bars: - sets drms, sunpy, parfive loggers to ERROR - uses a parfive Downloader with progress=False for silent downloads “”" import os import glob import csv import numpy as np import sunpy.map from astropy.time import Time import astropy.units as u from [sunpy.net](http://sunpy.net) import Fido, attrs as a import logging from parfive import Downloader # ----------------------------------------------------------------------------- # this will suppress some of the logging info I am getting # ----------------------------------------------------------------------------- logging.getLogger(“drms”).setLevel(logging.ERROR) logging.getLogger(“sunpy”).setLevel(logging.ERROR) logging.getLogger(“parfive”).setLevel(logging.ERROR) # ----------------------------------------------------------------------------- # Sentinel Check: Exit if Already Processed # ----------------------------------------------------------------------------- base\_folder = os.path.expanduser(“~/Desktop/Solar\_Comparison\_Test/”) results\_folder = os.path.join(base\_folder, “Results”) sentinel\_path = os.path.join(results\_folder, “matching\_done.txt”) if os.path.exists(sentinel\_path): print(“Matching has already been performed. Exiting.”) exit(0) # ----------------------------------------------------------------------------- # 1) Folder Setup # ----------------------------------------------------------------------------- harps\_folder = os.path.join(base\_folder, “HARPSN/”) hmi\_folder = os.path.join(base\_folder, “HMI/”) os.makedirs(results\_folder, exist\_ok=True) # ----------------------------------------------------------------------------- # 2) Loading HARPS-N Timestamps in the test folder # ----------------------------------------------------------------------------- harps\_files = sorted(glob.glob(os.path.join(harps\_folder, “r.HARPN.2015-09-01T\*.fits”))) harps\_times = for file in harps\_files: stamp\_str = os.path.basename(file).split(“r.HARPN.”)[-1].split(“\_S1D\_A\_filtered.fits”)[0] date\_part, time\_part = stamp\_str.split(“T”) time\_part = time\_part.replace(“-”, “:”) iso\_str = f"{date\_part}T{time\_part}" harps\_times.append(iso\_str) harps\_times = Time(harps\_times, format=“isot”, scale=“utc”) print(f" Loaded {len(harps\_times)} HARPS-N timestamps.") # ----------------------------------------------------------------------------- # 3) Just a convenience step built in to know how many files I already have before fetching new data from JSOC. # ----------------------------------------------------------------------------- hmi\_files = sorted(glob.glob(os.path.join(hmi\_folder, “\*.fits”))) print(f"Found {len(hmi\_files)} HMI FITS files in {hmi\_folder}.") if len(hmi\_files) == 0: print(“No local HMI files found. We’ll fetch from JSOC as needed.”) hmi\_times = for hf in hmi\_files: try: hmi\_map = sunpy.map.Map(hf) hmi\_times.append(Time(hmi\_map.date)) # Already UTC except Exception: pass print(f" Loaded {len(hmi\_times)} existing local HMI intensitygrams (if any).") # ----------------------------------------------------------------------------- # 4) Prepares a Downloader That Doesn’t Show Progress # ----------------------------------------------------------------------------- dl = Downloader(max\_conn=1, progress=False) # ----------------------------------------------------------------------------- # 5) Query & Download HMI for Each HARPS-N Timestamp # ----------------------------------------------------------------------------- jsoc\_email = “my emai address” # Your JSOC-registered email time\_window\_half = 5 \* u.second matched\_info = # Will store (HARPS-N time, downloaded HMI filename) def parse\_jsoc\_time(tstr): “”" Convert something like ‘2015.09.01\_09:39:45\_TAI’ into ‘2015-09-01T09:39:45’. “”" tstr = tstr.replace(“_TAI", “”) parts = tstr.split("_”) date\_part = parts[0].replace(“.”, “-”) time\_part = parts[1] return f"{date\_part}T{time\_part}" for t in harps\_times: # ±5 second window window = a.Time(t - time\_window\_half, t + time\_window\_half) ``` # Only specify recognized prime keys: result = Fido.search( window, a.jsoc.Series("hmi.Ic_45s"), a.jsoc.Notify(jsoc_email), a.jsoc.Segment("continuum"), a.jsoc.PrimeKey("CAMERA", 2) ) if len(result) == 0: # No data found in that 10 s window continue table = result[0] if "QUALITY" in table.colnames: # Filter out rows with nonzero QUALITY table = table[table["QUALITY"] == 0] if len(table) == 0: continue # Convert T_REC or Start Time to Astropy Time try: times_iso = [parse_jsoc_time(x) for x in table["T_REC"]] except KeyError: times_iso = [parse_jsoc_time(x) for x in table["Start Time"]] query_times = Time(times_iso, format="isot", scale="utc") # Pick the single closest row idx = np.argmin(np.abs(query_times - t)) single_result = table[idx:idx+1] # Download that single row's data (could be 1 or 2 files) files = Fido.fetch( single_result, path=os.path.join(hmi_folder, "{file}"), downloader=dl ) # -- **** Here's the key part: if multiple files come down, keep only the first -- if len(files) > 1: files = [files[0]] if len(files) > 0: matched_info.append((t.isot, os.path.basename(files[0]))) ``` print(f" Completed downloads. Found matches for {len(matched\_info)} HARPS times.") # ----------------------------------------------------------------------------- # 6) Save Matched Timestamps to CSV # ----------------------------------------------------------------------------- csv\_path = os.path.join(results\_folder, “matched\_times.csv”) with open(csv\_path, “w”, newline=“”) as f: writer = csv.writer(f) writer.writerow([“HARPS-N Time (UTC)”, “Downloaded HMI File”]) for harps\_time, hmi\_file in matched\_info: writer.writerow([harps\_time, hmi\_file]) print(f" Saved matched timestamps to {csv\_path}") # ----------------------------------------------------------------------------- # 7) Create Sentinel File # ----------------------------------------------------------------------------- with open(sentinel\_path, “w”) as f: f.write(“Matching completed successfully.\n”) print(f" Sentinel file created at {sentinel\_path}") ---
**Author:** ![nabobalis](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/nabobalis/32/14_2.png) [@nabobalis](https://community.openastronomy.org/u/nabobalis)\ **Post date:** [April 1, 2025, 11:00pm UTC](https://community.openastronomy.org/t/downloading-hmi-intensity-grams-to-match-a-set-of-harps-n-data-timestamps/1219/6 "2025-04-01T23:00:18Z")
Hello @Petra63, So I don’t have any SHARPS data to run the full code. When you say SHARPs-N, you mean the NRT data? One issue with the downloading with FIDO and the JSOCClient is that slicing the table and passing that into Fido will result in the original table being downloaded. We do not issue any warning about this, so its very common to be caught out our even forget about this behaviour. You will have to match the query exactly before you download if you just want a subset of files. So if you do need to filter by quality, you should pass that into the query: ```python result = Fido.search( a.Time( Time("2015-09-01T00:00:00"), Time("2015-09-01T00:01:00"), ), a.jsoc.Series("hmi.Ic_45s"), a.jsoc.Notify(jsoc_email), a.jsoc.Segment("continuum"), a.jsoc.PrimeKey("CAMERA", 2), a.jsoc.Keyword("QUALITY") == 0, ) ``` I am not sure if this will fix your original issue.