RuntimeError: Table DataManager error: StandardStMan::addColumn bucketsize too small for adding column COPY

[botteon]

Hi,

The following code

ts = pt.table(ms, readonly=False)
desc = ts.getcoldesc('DATA')
desc['name'] = 'COPY'
ts.addcols(desc)

generates a Segmentation fault (core dumped), raising the error reported in the title of the issue.
I used this command a number of times in the past with different instruments, but now with MeerKAT data it does not work.
For completeness, the description of the column (the desc variable) is

{'valueType': 'complex',
 'dataManagerType': 'DyscoStMan',
 'dataManagerGroup': 'DyscoData',
 'option': 5,
 'maxlen': 0,
 'comment': 'The data column',
 'ndim': 2,
 'shape': array([873,   4]),
 '_c_order': True,
 'keywords': {'UNIT': 'Jy'},
 'name': 'COPY'}

What is the problem and how to solve it? Thanks.

[tammojan]

It's confusing that the raised error mentions StandardStMan, while the DATA column apparently is a DyscoStMan column.

Maybe you can use a new taql feature to achieve the same thing:

pt.taql("ALTER TABLE $ms COPY COLUMN COPY=DATA")

This feature is documented here.

[botteon]

Hi Tammo, thanks.
Indeed, the RuntimeError appeared only while I was doing some tests, I cannot reproduce it again. Please ignore it and sorry for the confusion.

Now I get only the Segmentation fault (core dumped), full output:

In [1]: import casacore.tables as pt

In [2]: ts = pt.table('pippo.ms', readonly=False)
Successful read/write open of default-locked table pippo.ms: 26 columns, 1874430 rows

In [3]: colnames = ts.colnames()

In [4]: colnames
Out[4]: 
['UVW',
 'FLAG_CATEGORY',
 'WEIGHT',
 'SIGMA',
 'ANTENNA1',
 'ANTENNA2',
 'ARRAY_ID',
 'DATA_DESC_ID',
 'EXPOSURE',
 'FEED1',
 'FEED2',
 'FIELD_ID',
 'FLAG_ROW',
 'INTERVAL',
 'OBSERVATION_ID',
 'PROCESSOR_ID',
 'SCAN_NUMBER',
 'STATE_ID',
 'TIME',
 'TIME_CENTROID',
 'DATA',
 'FLAG',
 'WEIGHT_SPECTRUM',
 'MODEL_DATA',
 'CORRECTED_PREAPPLY0',
 'CORRECTED_DATA']

In [5]: desc = ts.getcoldesc('DATA')

In [6]: desc
Out[6]: 
{'valueType': 'complex',
 'dataManagerType': 'DyscoStMan',
 'dataManagerGroup': 'DyscoData',
 'option': 5,
 'maxlen': 0,
 'comment': 'The data column',
 'ndim': 2,
 'shape': array([873,   4]),
 '_c_order': True,
 'keywords': {'UNIT': 'Jy'}}

In [7]: desc['name'] = 'COPY'

In [8]: ts.addcols(desc)
Segmentation fault (core dumped)

I also tried the taql command you suggested, but:

In [1]: import casacore.tables as pt

In [2]: pt.taql("ALTER TABLE pippo.ms COPY COLUMN COPY=DATA")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 1
----> 1 pt.taql("ALTER TABLE pippo.ms COPY COLUMN COPY=DATA")

File /opt/lofar/pyenv-py3/lib64/python3.10/site-packages/casacore/tables/table.py:154, in taql(command, style, tables, globals, locals)
    152 if style:
    153     cmd = 'using style ' + style + ' ' + cmd
--> 154 tab = table(cmd, tabs, _oper=2)
    155 result = tab._getcalcresult()
    156 # If result is empty, it was a normal TaQL command resulting in a table.
    157 # Otherwise it is a record containing calc values.

File /opt/lofar/pyenv-py3/lib64/python3.10/site-packages/casacore/tables/table.py:329, in table.__init__(self, tablename, tabledesc, nrow, readonly, lockoptions, ack, dminfo, endian, memorytable, concatsubtables, _columnnames, _datatypes, _oper, _delete)
    324     Table.__init__(self, tabname, tabledesc, nrow, readonly,
    325                    lockoptions, ack, dminfo, endian, memorytable,
    326                    _columnnames, _datatypes)
    327 elif _oper == 2:
    328     # This is the query or calc constructor.
--> 329     Table.__init__(self, tablename, tabledesc)
    330     if len(self._getcalcresult()) != 0:
    331         # Do not make row object for a calc result
    332         return

RuntimeError: Error in TaQL command: 'using style Python ALTER TABLE pippo.ms COPY COLUMN COPY=DATA'
  Invalid Table operation: Data manager name DyscoData is already used in table /data2/botteon/test/pippo.ms

I tested the taql command on a LOFAR dataset (zorro.ms) but it crashes with the same error, while the ts.addcols works:

In [1]: import casacore.tables as pt

In [2]: ts = pt.table('zorro.ms', readonly=False)
Successful read/write open of default-locked table zorro.ms: 23 columns, 1645170 rows

In [3]: colnames = ts.colnames()

In [4]: colnames
Out[4]: 
['UVW',
 'FLAG_CATEGORY',
 'WEIGHT',
 'SIGMA',
 'ANTENNA1',
 'ANTENNA2',
 'ARRAY_ID',
 'DATA_DESC_ID',
 'EXPOSURE',
 'FEED1',
 'FEED2',
 'FIELD_ID',
 'FLAG_ROW',
 'INTERVAL',
 'OBSERVATION_ID',
 'PROCESSOR_ID',
 'SCAN_NUMBER',
 'STATE_ID',
 'TIME',
 'TIME_CENTROID',
 'DATA',
 'FLAG',
 'WEIGHT_SPECTRUM']

In [5]: desc = ts.getcoldesc('DATA')

In [6]: desc
Out[6]: 
{'valueType': 'complex',
 'dataManagerType': 'DyscoStMan',
 'dataManagerGroup': 'DyscoData',
 'option': 5,
 'maxlen': 0,
 'comment': 'The data column',
 'ndim': 2,
 'shape': array([120,   4]),
 '_c_order': True,
 'keywords': {'LOFAR_APPLIED_BEAM_MODE': 'Full',
  'LOFAR_APPLIED_BEAM_DIR': {'type': 'direction',
   'refer': 'J2000',
   'm1': {'value': 0.14481374958468218, 'unit': 'rad'},
   'm0': {'unit': 'rad', 'value': -0.07448868897971053}}}}

In [7]: pt.taql("ALTER TABLE zorro.ms COPY COLUMN COPY=DATA")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[7], line 1
----> 1 pt.taql("ALTER TABLE zorro.ms COPY COLUMN COPY=DATA")

File /opt/lofar/pyenv-py3/lib64/python3.10/site-packages/casacore/tables/table.py:154, in taql(command, style, tables, globals, locals)
    152 if style:
    153     cmd = 'using style ' + style + ' ' + cmd
--> 154 tab = table(cmd, tabs, _oper=2)
    155 result = tab._getcalcresult()
    156 # If result is empty, it was a normal TaQL command resulting in a table.
    157 # Otherwise it is a record containing calc values.

File /opt/lofar/pyenv-py3/lib64/python3.10/site-packages/casacore/tables/table.py:329, in table.__init__(self, tablename, tabledesc, nrow, readonly, lockoptions, ack, dminfo, endian, memorytable, concatsubtables, _columnnames, _datatypes, _oper, _delete)
    324     Table.__init__(self, tabname, tabledesc, nrow, readonly,
    325                    lockoptions, ack, dminfo, endian, memorytable,
    326                    _columnnames, _datatypes)
    327 elif _oper == 2:
    328     # This is the query or calc constructor.
--> 329     Table.__init__(self, tablename, tabledesc)
    330     if len(self._getcalcresult()) != 0:
    331         # Do not make row object for a calc result
    332         return

RuntimeError: Error in TaQL command: 'using style Python ALTER TABLE zorro.ms COPY COLUMN COPY=DATA'
  Invalid Table operation: Data manager name DyscoData is already used in table /data2/botteon/test/zorro.ms

In [8]: desc['name'] = 'COPY'

In [9]: ts.addcols(desc)

In [10]: 

I don't if it is relevant, but the LOFAR dataset is 2.3 GB while the MeerKAT dataset is 98 GB.