# Pyvo TAP result to csv (via astropy.Table?)

**URL:** <https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198>\
**Category:** Astropy\
**Tags:** astropy, table, question\
**Created:** [October 20, 2021, 4:16pm UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198 "2021-10-20T16:16:50Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ygrange](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/ygrange/32/133_2.png) [@ygrange](https://community.openastronomy.org/u/ygrange)\
**Post date:** [October 20, 2021, 4:16pm UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/1 "2021-10-20T16:16:50Z")

</div>

I have a TAP table in a VO service that I want to dump to csv. So for this I basically was thinking of  
using the following code snippet (replaced the VO service by a public one for demonstration purposes):  
=========8\<========  
import pyvo

tapurl = ‘[Information on Service 'The VO @ ASTRON TAP service'](https://vo.astron.nl/tap)’  
qry= “select \* from tgssadr.img\_main”  
tap\_service = pyvo.dal.TAPService(tapurl)  
tap\_result = tap\_service.run\_sync(qry)

tbl = tap\_result.to\_table()  
tbl.write(“my.csv”)

=========8\<========

However the last command returns:

Traceback (most recent call last):  
File “”, line 1, in   
File “/usr/local/lib/python3.9/site-packages/astropy/table/connect.py”, line 127, in **call**  
registry.write(instance, \*args, \*\*kwargs)  
File “/usr/local/lib/python3.9/site-packages/astropy/io/registry.py”, line 570, in write  
writer(data, \*args, \*\*kwargs)  
File “/usr/local/lib/python3.9/site-packages/astropy/io/ascii/connect.py”, line 26, in io\_write  
return write(table, filename, \*\*kwargs)  
File “/usr/local/lib/python3.9/site-packages/astropy/io/ascii/ui.py”, line 842, in write  
writer.write(table, output)  
File “/usr/local/lib/python3.9/site-packages/astropy/io/ascii/fastbasic.py”, line 207, in write  
self.\_write(table, output, {‘fill\_values’: [(core.masked, ‘’)]})  
File “/usr/local/lib/python3.9/site-packages/astropy/io/ascii/fastbasic.py”, line 184, in \_write  
writer.write(output, header\_output, output\_types)  
File “astropy/io/ascii/cparser.pyx”, line 1125, in astropy.io.ascii.cparser.FastWriter.write  
TypeError: unhashable type: ‘MaskedArray’

This confuses me because first of all, the Table is not masked (tbl.masked is set to False). I guess this may be a bug but I don’t really know if this is an issue with pyvo or astropy.Table . Hopefully the crowd has seen this issue before, or can indicate me another way to do this.

---

<div class="post-metadata">

**Author:** ![dhomeier](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/dhomeier/32/65_2.png) [@dhomeier](https://community.openastronomy.org/u/dhomeier)\
**Post date:** [October 21, 2021, 11:07pm UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/2 "2021-10-21T23:07:32Z")

</div>

That’s odd, but I’d guess the format returned by pyvo does play a role here.  
What do `tbl.dtype` and `tbl.mask` return (it looks like the table was created from `np.ma.MaskedArray` rather than by setting `mask=True`, in which case it can apparently still have a mask even though `tbl.masked` says `False`)?  
Still, simple masked arrays of both types should be supported even by the fast writer (on Astropy 4.3.1); but does `tbl.write("my.csv", fast_reader=False)` work?  
I suspect this could rather be a problem with a more complex column dtype, which might not be supported by csv.

---

<div class="post-metadata">

**Author:** ![tomdonaldson](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@tomdonaldson](https://community.openastronomy.org/u/tomdonaldson)\
**Post date:** [October 22, 2021, 1:57am UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/3 "2021-10-22T01:57:38Z")

</div>

That is strange. I do see that `tbl.has_masked_columns` is `True`, and that there are several masked columns in the table. That said, I don’t know why a masked column wouldn’t be written to csv. I would expect the masked (null) values to be written as empty strings.

Seeing `FastWriter` in the stack trace made me curious, so I tried writing without it. This seemed to work for me, so might be a good workaround:  
`tbl.write('my.csv', fast_writer=False)`

I’m not an expert on masked columns, but having inconsistent behavior between the fast and slow writers is surely a bug, and again, I don’t see any reason why masked columns shouldn’t be written out.

---

<div class="post-metadata">

**Author:** ![dhomeier](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/dhomeier/32/65_2.png) [@dhomeier](https://community.openastronomy.org/u/dhomeier)\
**Post date:** [October 22, 2021, 1:44pm UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/4 "2021-10-22T13:44:36Z")

</div>

> [@tomdonaldson](#):
>
> I’m not an expert on masked columns, but having inconsistent behavior between the fast and slow writers is surely a bug, and again, I don’t see any reason why masked columns shouldn’t be written out.

As per my tests mentioned above, simple cases like

```python
tbl = Table([np.ma.MaskedArray(range(4), mask=[0, 1, 0, 0]), np.arange(3, 7)])`
tbl.write('my.csv', fast_writer='force')

```

are working just fine (writing indeed empty fields for the masked elements), and generally the writer should automatically switch to the non-fast version where necessary, as long as it is not explicitly forced with `fast_writer='force'`.  
At least that is the behaviour for the table _reader_.

The `TypeError: unhashable type: ‘MaskedArray’` is in fact an ancient issue –

> <https://github.com/numpy/numpy/issues/4660>
>
> It seems that in python 2.7 I can do \`a = {np.ma.masked: 'AAA'}\` while in python…3 this code throws \`TypeError: unhashable type: 'MaskedConstant'\` (numpy 1.8. in both cases).

and precisely the reason why `self._write()` above uses an `astropy.io.ascii.masked` instance instead – no idea why that would not be working in this case…

---

<div class="post-metadata">

**Author:** ![dhomeier](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/dhomeier/32/65_2.png) [@dhomeier](https://community.openastronomy.org/u/dhomeier)\
**Post date:** [October 22, 2021, 7:03pm UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/5 "2021-10-22T19:03:34Z")

</div>

As far as I can see the error results from the multi-dimensional columns like

```python
>>> tap_result['pixelsize']
masked_array(data=[masked_array(data=[2903, 2903, 1, 1],
                                mask=[False, False, False, False],
                          fill_value=999999,
                               dtype=int32) ,
                   masked_array(data=[2903, 2903, 1, 1],
                                mask=[False, False, False, False],
                          fill_value=999999,
                               dtype=int32) ,
                   masked_array(data=[2903, 2903, 1, 1],
                                mask=[False, False, False, False],
                          fill_value=999999,
                               dtype=int32) , ...

```

– since all those columns are already converted to dtype “Object” on `.to_table()` handling them becomes rather inconvenient (the data in each column are a `np.ma.MaskedArray` _each element_ of which in turn is a `np.ma.MaskedArray` in itself!) Still not sure what happens internally in the writer, but I would recommend saving it as an ECSV instead, which simply works as

```python
tbl.write("my.ecsv")

```

and mostly preserves the internal structure of those columns. In CSV they could only be written as strings `"[2903, 2903, 1, 1]"` etc. – in fact without all that weird wrapping trying to writ as CSV should already raise an error like  
` column(s) with dimension > 1 cannot be be written with this format, try using 'ecsv' (Enhanced CSV) format`.

---

<div class="post-metadata">

**Author:** ![tomdonaldson](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@tomdonaldson](https://community.openastronomy.org/u/tomdonaldson)\
**Post date:** [October 22, 2021, 10:19pm UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/6 "2021-10-22T22:19:16Z")

</div>

The csv output is interesting for those fields with  
`tbl.write('my.csv', fast_writer=False)`  
yielding strings like  
`[2903 2903 1 1]`  
for `pixelsize` (space delimited values inside square brackets).

This is semi-reasonable I suppose, but it definitely doesn’t round trip well. I.e., reading that csv into a table not only doesn’t give the same structure (the new column is a string type), but it looks like the rows are messed up due to newlines present in the output of the `coverage` column objects. Aside from round-tripping, the csv might have been usable by another script, but those newlines (another bug I think) make the csv less than usable.

---

<div class="post-metadata">

**Author:** ![dhomeier](https://dub1.discourse-cdn.com/flex005/user_avatar/community.openastronomy.org/dhomeier/32/65_2.png) [@dhomeier](https://community.openastronomy.org/u/dhomeier)\
**Post date:** [October 22, 2021, 11:03pm UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/7 "2021-10-22T23:03:42Z")

</div>

> [@tomdonaldson](#):
>
> This is semi-reasonable I suppose, but it definitely doesn’t round trip well. I.e., reading that csv into a table not only doesn’t give the same structure (the new column is a string type),

Yes, I think that’s the reason why `write(..., format='csv')` normally doesn’t accept `ndim>1` columns but tries to guide people to more sensible formats like ECSV instead.  
Without knowing much about the background of the `pyvo` implementation it feels a bit like it tries very hard to bypass that check by packing the higher-dimensional fields deep inside object arrays.

---

<div class="post-metadata">

**Author:** ![tomdonaldson](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@tomdonaldson](https://community.openastronomy.org/u/tomdonaldson)\
**Post date:** [October 23, 2021, 12:20am UTC](https://community.openastronomy.org/t/pyvo-tap-result-to-csv-via-astropy-table/198/8 "2021-10-23T00:20:56Z")

</div>

Ah OK, that makes sense to me. `ndim>1` seems like a good thing for the csv writer to check.

`pyvo` uses the `astropy.io.votable` reader to parse the results coming from VO services like this. VOTables are allowed to have arrays of primitives in individual table cells, and the parser puts them into the object arrays. From what you’re saying, it seems like that might not be the best way to store those values in the astropy Table.

At least the `astropy.io.votable` reader and writer are consistent with respect to the VOTable format. So `tbl.write('out.xml', format='votable')` results in a valid VOTable with those array values represented correctly.