Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fixed-type tuple error for level_3 tso data #7666

Closed
stscijgbot-jp opened this issue Jun 29, 2023 · 17 comments
Closed

fixed-type tuple error for level_3 tso data #7666

stscijgbot-jp opened this issue Jun 29, 2023 · 17 comments
Labels
level_3_failure Software Affected: white_light team-coffee tso

Comments

@stscijgbot-jp
Copy link
Collaborator

Issue JP-3277 was created on JIRA by Michael Ridgaway:

During build reprocessing following the install of JWST DMS Build 9.2.1 the following error was seen during Level 3 processing of program 1224 and 2149 data. "ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int."

Logs can be found here:

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01224-o002_20230528t111952_tso3_00001_1685490152.276564

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02149-o002_20230528t221201_tso3_00001_1685336491.324613

 

This is the same error seen in JP-2900 that was fixed for nrc_wfss data.
@stscijgbot-jp
Copy link
Collaborator Author

Comment by Hien Tran on JIRA:

also seeing this for jw02158-o001_20230528t222142_tso3_00001

Directory Path: /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02158-o001_20230528t222142_tso3_00001_1685438395.977274

ALOG_1685438945_repro_level_3_jw02158-o001_20230528t222142_tso3_00001.out

2023-05-30 09:29:18,264 - CRDS - DEBUG - Computing best references locally.
2023-05-30 09:29:18,266 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_BRIGHTOBJ',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1',
'META.INSTRUMENT.FILTER [FILTER]': 'CLEAR',
'META.INSTRUMENT.GRATING [GRATING]': 'PRISM',
'META.INSTRUMENT.LAMP_MODE [OPMODE]': 'BRIGHTOBJ',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-03-11',
'META.OBSERVATION.TIME [TIME-OBS]': '03:17:05.936',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUB512',
'META.SUBARRAY.XSIZE [SUBSIZE1]': '512.0',
'META.SUBARRAY.XSTART [SUBSTRT1]': '1025.0',
'META.SUBARRAY.YSIZE [SUBSIZE2]': '32.0',
'META.SUBARRAY.YSTART [SUBSTRT2]': '959.0',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
....
<snip>

2023-05-31 03:25:36,867 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-05-31 03:32:35,517 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-05-31 03:32:54,735 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw02158001001_04101_00001-seg003_nrs1_calints.fits>,).
2023-05-31 03:32:54,736 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': None, 'max_wavelength': None}
2023-05-31 03:32:55,193 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
Traceback (most recent call last):
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module>
    step = Step.from_cmdline(sys.argv[1:])
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
    return cmdline.step_from_cmdline(args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
    step.run(*positional)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
    step_result = self.process(*args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 193, in process
    self.save_model(x1d_result, suffix='x1dints')
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 978, in save_model
    output_path = model.save(
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 530, in save
    self.to_fits(output_path, *args, **kwargs)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 642, in to_fits
    hdulist = fits_support.to_fits(self._instance, self._schema)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 494, in to_fits
    hdulist.append(_create_asdf_hdu(tree))
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 506, in _create_asdf_hdu
    column = fits.Column(array=data, format=fmt, name="ASDF_METADATA")
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 726, in __init__
    array = self._convert_to_valid_data_type(array)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 1451, in _convert_to_valid_data_type
    dtype = np.dtype(numpy_format).base
ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int.
2023155030646 INFO src=ssb_calibration_wrapper._strun_analyze_and_log_failure fsn=jw02158-o001_20230528t222142_tso3_00001_asn msg="strun /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.cfg FAILED (exit=1) on jw02158-o001_20230528t222142_tso3_00001_asn.json."
2023155030646 ERROR src=ssb_calibration_wrapper.calibrate._strun._strun_analyze_and_log_failure fsn=jw02158-o001_20230528t222142_tso3_00001_asn msg="strun exit status=1" ```

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Hien Tran on JIRA:

same problem seen for jw02508-o001_20230527t034736_tso3_00001, which is a MIR_LRS-SLITLESS tso3. so the problem seems to be connected to time series for all different inst/modes 

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Hien Tran on JIRA:

another case, jw02488-o001_20230528t230213_tso3_00001, which is a NRS_BRIGHTOBJ tso

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Hien Tran on JIRA:

many failure cases (exited = 1) for 2512, but some also succeeded (exit code 0). these are all nrs_brightobj tso. don't see any obvious differences between them, except the failure seems occur for those asns with more than 6 science input members

jw02512-c1002_20230603t201124_tso3_00001           REPRO_LEVEL_3        2023-06-04 01:31:14  2023-06-05 00:39:58  Exited          1          owlmgr             
jw02512-c1004_20230603t201124_tso3_00001           REPRO_LEVEL_3        2023-06-06 15:23:35  2023-06-08 06:44:35  Exited          1          owlmgr             
jw02512-o008_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:12:59  2023-06-04 22:17:47  Exited          1          owlmgr             
jw02512-o009_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:11:14  2023-06-04 17:01:07  Exited          1          owlmgr             
jw02512-o010_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:01:50  2023-06-04 15:23:49  Exited          1          owlmgr             
jw02512-o003_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:31:09  2023-06-04 11:29:52  Exited          0          owlmgr             
jw02512-o004_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:12:59  2023-06-04 10:25:02  Exited          0          owlmgr             
jw02512-o006_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:00:45  2023-06-04 09:06:53  Exited          0          owlmgr             
jw02512-o013_20230603t201124_tso3_00001            LEVEL_3              2023-06-05 23:11:15  2023-06-06 07:25:31  Exited          0          owlmgr       ```

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Howard Bushouse on JIRA:

Logs show that failure is occurring during the white_light step, after having successfully completed all steps through extract_1d.

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Michael Ridgaway on JIRA:

We are also seeing this error occur for program 1201 data. Logs can be found here:

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01201-o002_20230612t154303_tso3_00001_1686632977.502441

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Hien Tran on JIRA:

latest failure for jw02512-c1005_20230603t201124_tso3_00001, a 18-science member nrs_brightobj tso3 with sub2048 array

023-06-28 09:31:01,955 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_BRIGHTOBJ',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1',
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP',
'META.INSTRUMENT.GRATING [GRATING]': 'G395H',
'META.INSTRUMENT.LAMP_MODE [OPMODE]': 'BRIGHTOBJ',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-06-27',
'META.OBSERVATION.TIME [TIME-OBS]': '04:55:57.198',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUB2048',
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0',
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0',
'META.SUBARRAY.YSIZE [SUBSIZE2]': '32.0',
'META.SUBARRAY.YSTART [SUBSTRT2]': '965.0',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
....


2023-06-28 10:32:28,321 - stpipe.Tso3Pipeline.extract_1d - INFO - All 1525 integrations done
2023-06-28 10:32:38,739 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-06-28 10:33:16,877 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-06-28 10:33:24,340 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw02512012001_04102_00001-seg002_nrs1_calints.fits>,).
2023-06-28 10:33:24,345 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': None, 'max_wavelength': None}
2023-06-28 10:33:24,590 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module>
step = Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 193, in process
self.save_model(x1d_result, suffix='x1dints')
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 978, in save_model
output_path = model.save(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 530, in save
self.to_fits(output_path, *args, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 642, in to_fits
hdulist = fits_support.to_fits(self._instance, self._schema)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 494, in to_fits
hdulist.append(_create_asdf_hdu(tree))
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 506, in _create_asdf_hdu
column = fits.Column(array=data, format=fmt, name="ASDF_METADATA")
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 726, in __init__
array = self._convert_to_valid_data_type(array)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 1451, in _convert_to_valid_data_type
dtype = np.dtype(numpy_format).base
ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int. ```

@stscijgbot-jp
Copy link
Collaborator Author

stscijgbot-jp commented Jun 29, 2023 •
edited
Loading

Comment by John Scott on JIRA:

a list of affected datasets (cleaning these out of the pipeline for now)

jw01274-o004_20230531t014455_tso3_00002

jw01224-o002_20230528t111952_tso3_00001

jw02512-o009_20230603t201124_tso3_00001

jw02512-o008_20230603t201124_tso3_00001

jw02488-o001_20230528t230213_tso3_00001

jw02508-o001_20230527t034736_tso3_00001

jw02512-c1003_20230603t201124_tso3_00001

jw01633-o005_20230601t145410_tso3_00001

jw02149-o002_20230528t221201_tso3_00001

jw01952-o001_20230602t204759_tso3_00001

jw01633-o004_20230601t145410_tso3_00001

jw02512-c1004_20230603t201124_tso3_00001

jw02512-c1002_20230603t201124_tso3_00001

jw02512-c1006_20230603t201124_tso3_00001

jw02158-o001_20230528t222142_tso3_00001

jw01274-o003_20230531t014455_tso3_00002

jw01274-o002_20230531t014455_tso3_00002

jw01274-o001_20230531t014455_tso3_00002

jw01201-o002_20230612t154303_tso3_00001

jw02512-c1005_20230603t201124_tso3_00001

jw02512-o010_20230603t201124_tso3_00001

jw02512-o017_20230603t201124_tso3_00001

jw02159-o001_20230706t061547_tso3_00001

jw02512-c1003_20230603t201124_tso3_00001

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Tyler Pauly on JIRA:

Quick investigation with a successful tso3 run, using a truncated input association, indicates that the ASDF extension takes up more memory than the other ~6098 extract1d extensions. Sending to ASDF experts - Brett Graham  maybe?

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Brett Graham on JIRA:

Tyler Pauly do you have an easily sharable link to the hopefully similar and successful tso3 run? I can take a look at the file. My first concern would be that the extensions are being duplicated again for some reason (similar to asdf-format/asdf#1232

@hbushouse
Copy link
Collaborator

Fixed by spacetelescope/stdatamodels#178

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Howard Bushouse on JIRA:

Fixed by spacetelescope/stdatamodels#178

Will be included in final release candidate for B9.3 (jwst 1.11.2, B9.3rc3).

@stscijgbot-jp
Copy link
Collaborator Author

Comment by John Scott on JIRA:

another case

 

jw01201-o002_20230612t154303_tso3_00001

@stscijgbot-jp
Copy link
Collaborator Author

Comment by John Scott on JIRA:

jw02965-o003_20230709t121734_tso3_00001

is another case.

@stscijgbot-jp
Copy link
Collaborator Author

stscijgbot-jp commented Jul 19, 2023 •
edited
Loading

Comment by John Scott on JIRA:

another

jw01185-o010_20230715t193546_tso3_00002
jw01185-o012_20230715t193546_tso3_00001

@stscijgbot-jp
Copy link
Collaborator Author

Comment by John Scott on JIRA:

More (maybe some duplicates)

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o005_20230724t181148_tso3_00001_1690242142.899445/ALOG_1690554631_repro_level_3_jw02084-o005_20230724t181148_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o002_20230724t181148_tso3_00002_1690239819.034980/ALOG_1690559959_repro_level_3_jw02084-o002_20230724t181148_tso3_00002.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o001_20230724t181148_tso3_00001_1690239815.591388/ALOG_1690560200_repro_level_3_jw02084-o001_20230724t181148_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01633-o005_20230722t185115_tso3_00002_1690070138.815810/ALOG_1690070637_repro_level_3_jw01633-o005_20230722t185115_tso3_00002.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01633-o004_20230722t185115_tso3_00001_1690070141.316165/ALOG_1690070641_repro_level_3_jw01633-o004_20230722t185115_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02512-c1002_20230603t201124_tso3_00001_1690409345.284946/ALOG_1690409539_repro_level_3_jw02512-c1002_20230603t201124_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o003_20230724t181148_tso3_00001_1690244754.860707/ALOG_1690554631_repro_level_3_jw02084-o003_20230724t181148_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o003_20230724t181148_tso3_00001_1690244754.860707/ALOG_1690752134_repro_level_3_jw02084-o003_20230724t181148_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01952-o001_20230724t114228_tso3_00001_1690213624.254566/ALOG_1690554631_repro_level_3_jw01952-o001_20230724t114228_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01952-o001_20230724t114228_tso3_00001_1690213624.254566/ALOG_1690752143_repro_level_3_jw01952-o001_20230724t114228_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01274-o003_20230719t112441_tso3_00001_1689831012.394827/ALOG_1689960087_repro_level_3_jw01274-o003_20230719t112441_tso3_00001.err
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02512-o018_20230603t201124_tso3_00001_1690223125.068266/ALOG_1690251734_level_3_jw02512-o018_202

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Hien Tran on JIRA:

problem no longer seen with recent repro runs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
level_3_failure Software Affected: white_light team-coffee tso
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hbushouse @stscijgbot-jp