spack-stack/2.0.0 gsi-ncdiag/1.1.2 seg faults

[RussTreadon-NOAA]

Description

When GSI is built and run on Ursa using spack-stack/2.0.0, gsi.x seg faults when calling ncdw_metadata.F90 from gsi-ncdiag/1.1.2. Building the same GSI source code with spack-stack/1.9.2 gsi-ncdiag/1.1.2 yields a gsi.x which successfully runs to completion.

To Reproduce

1. git clone --recursive https://github.com/ADCollard/GSI.git .
2. git checkout GSI_CRTMv3_dev_cdf
3. git submodule update --init --recursive
4. cd modulefiles
5. comment out the setenv("CRTM_FIX" in gsi_ursa.intel.lua
6. cd ../regression`
7. open global_4denvar.sh in an editor
8. replace .bin with .nc for fixcrtm files. Use .nc4 for AerosolCoeff and CloudCoeff
9. cd ../ush
10. ./build.sh > log.build 2>&1 &
11. cd ../build
12. ctest -VV -R global_4denvar
13. cntrl-c after you see the slurm job number, $jobid, printed to the screen
14. scontrol show job $jobid to get the path/name of the job log output file
15. tail job log file after job starts running
16. cd to job run directory after job fails
17. open stdout in an editor
18. search for forrtl: severe (174): SIGSEGV, segmentation fault occurred to find the traceback below

GLBSOI: jiter,jiterstart,jiterlast,jiterend=    1    1    2    1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
libc.so.6          000014B8C88CABF0  Unknown               Unknown  Unknown
gsi.x              0000000001AA7EA2  nc_diag_metadata_         502  ncdw_metadata.F90
gsi.x              0000000001A8B22B  nc_diag_write             536  nc_diag_write_mod.F90
gsi.x              000000000134A732  setupps                   675  setupps.f90

Expected behavior

The spack-stack/2.0.0 gsi.x should successfully run the GSI global_4denvar ctest ... just like the spack-stack/1.9.2 gsi.x

System, compiler, code, ...

gsi.x built and run on Ursa with the following modules loaded

  1) intel-oneapi-compilers/2025.2.1         8) curl/8.11.1     15) netcdf-fortran/4.6.1     22) python/3.11.11   29) sigio/2.3.3       36) crtm/3.1.2
  2) stack-intel-oneapi-compilers/2025.2.1   9) cmake/3.31.8    16) intel-oneapi-mkl/2024.2  23) python-venv/1.0  30) sfcio/1.4.2       37) gsi-ncdiag/1.1.2
  3) intel-oneapi-mpi/2021.13.1             10) snappy/1.2.1    17) pigz/2.8                 24) py-numpy/1.26.4  31) nemsio/2.5.5      38) prod_util/2.1.2
  4) stack-intel-oneapi-mpi/2021.13         11) zstd/1.5.7      18) tar/1.35                 25) bufr/12.1.0      32) wrf-io/1.3.0      39) gsi_common
  5) glibc/2.34                             12) c-blosc/1.21.6  19) gettext/0.23.1           26) bacio/2.6.0      33) ncio/1.1.2        40) gsi_ursa.intel
  6) nghttp2/1.65.0                         13) hdf5/1.14.5     20) sqlite/3.46.0            27) w3emc/2.11.0     34) crtm-fix/3.1.2.0
  7) zlib/1.2.13                            14) netcdf-c/4.9.2  21) util-linux-uuid/2.41     28) ip/5.4.0         35) git-lfs/3.5.1

gsi.x built with

-- The C compiler identification is IntelLLVM 2025.2.1
-- The Fortran compiler identification is IntelLLVM 2025.2.1

Additional context

Add any other context about the problem here.

[RussTreadon-NOAA]

FYI @ADCollard

[rickgrubin-noaa]

@RussTreadon-NOAA

Perhaps try the test stacks from #1900 ? I'm not confident of success, but doing so would allow us to understand if the problem persists.

Please also note that for spack-stack@2.0 (and forward) built with oneAPI compilers, the Fortran compiler is ifx, and for spack-stack-1.9.x, the Fortran compiler is ifort. Unsure if this is relevant, however it may be a notable difference.

[RussTreadon-NOAA]

Thank you @rickgrubin-noaa for your reply. I tried your intel spack-stack build from #1900. The same failure still occurs.

The change from ifort to ifx may have exposed issues in either how GSI uses gsi-ncdiag or in the gsi-ncdiag code itself. Unfortunately, I am not familiar with the gs-ncdiag code.

[ADCollard]

I checked out the v1.1.2 from GSI-ncdiag. I loaded the GSI modules for Ursa and built and installed the package.

I then recompiled GSI ensuring that I was pointing to my GSI-ncdiag build:

build.log:-- Found ncdiag: /scratch3/NCEPDEV/da/Andrew.Collard/git/GSI-ncdiag/install/lib64/libncdiag.a (found version "1.1.2")

A standalone job then completes successfully.

So maybe this is a compiler mismatch?

[ADCollard]

Is it possible to get guidance on how to build GSI-ncdiag to mimic the the spack-stack install?

[RussTreadon-NOAA]

Great progress @ADCollard . A successful end-to-end run is excellent! You resolved not only the ncdiag issue but also the crtm/3.1.2 issue.

I have never installed spack-stack. Does the following spack-stack wiki help answer your installation question?
New and chained environments for existing sites

Can you share how you built gsi-ncdiag/1.1.2? Maybe @rickgrubin-noaa can compare how you build gsi-ncdiag/1.1.2 with how spack-stack/2.0.0 builds gsi-ncdiag/1.1.2.

[ADCollard]

Thanks @RussTreadon-NOAA. I tried following the instructions you linked to. I get the following warning when I run

> spack stack create env --site=hera --template=unified-dev --name=unified-dev.hera.intel --compiler=intel
.
.
.
WARNING: /scratch3/NCEPDEV/da/Andrew.Collard/git/spack-stack_2.0.0/configs/common/packages_intel.yaml not found, please check if this is correct

In `/scratch3/NCEPDEV/da/Andrew.Collard/git/spack-stack_2.0.0/configs/common/' I find:

packages_intel.yaml.NOTYETUPDATED

This may or may not be related to the final error I get:

spack concretize --force --fresh
${SPACK_STACK_DIR}/util/show_duplicate_packages.py
==> Warning: The packages:all:compiler preference has been deprecated in Spack v1.0, and is currently ignored. It will be removed from config in Spack v1.2.
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v1/blobs/sha256:29fda6eb0e8149d3fb662e9e3047fc49584c940c6450c56b36cadac0f59f0300
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v1/blobs/sha256:a572ab6db954f4a850d1292bb1ef6d6055916784a894d149d657996fa98d0367
==> Installing "clingo-bootstrap@=spack~docs+ipo+optimized+python+static_libstdcpp build_system=cmake build_type=Release generator=make patches:=bebb819,ec99431 arch=linux-centos7-x86_64" from a buildcache
==> Compilers have been configured automatically from PATH inspection
==> Warning: while setting up external spec intel-oneapi-mpi@=2021.5.1 %intel@2021.5.0+classic-names: No such variant 'classic-names' in package intel-oneapi-compilers-classic
==> Error: failed to concretize `ewok-env`, `esmf@=8.6.1`, `mapl@2.53.0 ^esmf@8.6.1`, `crtm@v2.4.1-jedi.2`, ..., `esmf@=8.9.1` for the following reasons:
     1. Cannot satisfy 'crtm@v2.4.1-jedi.2'
        required because crtm@v2.4.1-jedi.2 requested explicitly 
     2. Cannot satisfy 'crtm@v2.4.1-jedi.2'
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/da/Andrew.Collard/git/spack-stack_2.0.0/util/show_duplicate_packages.py", line 43, in <module>
    with open(os.path.join(basedir, "spack.lock"), "r") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/scratch3/NCEPDEV/da/Andrew.Collard/git/spack-stack_2.0.0/envs/unified-dev.hera.intel/spack.lock'

[climbfuji]

@ADCollard @RussTreadon-NOAA If you are on a fully supported platform like Ursa (I can't see which system you are using), then it's relatively easy as long as you have the correct code. I'll put together basic instructions and post them here shortly.

[ADCollard]

Here are my steps for building and installing GSI-ncdiag

git clone -b v1.1.2 --recursive https://github.com/NOAA-EMC/GSI-ncdiag.git
cd GSI-ncdiag
module use /scratch3/NCEPDEV/da/Andrew.Collard/git/GSI_develop/modulefiles
module load gsi_ursa.intel 
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=../install ..
make -j8
make install

Then in GSI (this is in /scratch3/NCEPDEV/da/Andrew.Collard/git/GSI_spack2) add the following (indicated with a "+"):

 load(pathJoin("cmake", cmake_ver))
 
 load("gsi_common")
 
+local gsi_ncdiags_root = "/scratch3/NCEPDEV/da/Andrew.Collard/git/GSI-ncdiag/install"
+setenv("GSI_NCDIAGS_ROOT", gsi_ncdiags_root)
+if gsi_ncdiags_root and gsi_ncdiags_root ~= "" then
+  -- For CMake find_package()/package config discovery
+  prepend_path("CMAKE_PREFIX_PATH", gsi_ncdiags_root)
+
+  -- If something uses pkg-config (maybe not here, but safe if you have pc files)
+  prepend_path("PKG_CONFIG_PATH", pathJoin(gsi_ncdiags_root, "lib64", "pkgconfig"))
+  prepend_path("PKG_CONFIG_PATH", pathJoin(gsi_ncdiags_root, "lib", "pkgconfig"))
+
+  -- For compile/link lines that use -I/-L derived from env
+  prepend_path("LIBRARY_PATH", pathJoin(gsi_ncdiags_root, "lib64"))
+  prepend_path("LIBRARY_PATH", pathJoin(gsi_ncdiags_root, "lib"))
+  prepend_path("CPATH", pathJoin(gsi_ncdiags_root, "include"))
+
+  -- Runtime loader path (mostly relevant for .so; harmless for static-only)
+  prepend_path("LD_LIBRARY_PATH", pathJoin(gsi_ncdiags_root, "lib64"))
+  prepend_path("LD_LIBRARY_PATH", pathJoin(gsi_ncdiags_root, "lib"))
+
+  -- Optional: some builds look for ncdiag_DIR explicitly; your tree suggests this path
+  setenv("ncdiag_DIR", pathJoin(gsi_ncdiags_root, "lib64", "cmake", "ncdiag"))
+end
+
 pushenv("GSI_BINARY_SOURCE_DIR", "/scratch3/NCEPDEV/global/role.glopara/fix/gsi/20251105")
-setenv("CRTM_FIX", pathJoin("/scratch3/NCEPDEV/global/role.glopara/fix/crtm", "v" .. crtm_fix_ver))
 
 whatis("Description: GSI environment on Ursa with Intel Compilers")

and run ush/build.sh

[ADCollard]

@ADCollard @RussTreadon-NOAA If you are on a fully supported platform like Ursa (I can't see which system you are using), then it's relatively easy as long as you have the correct code. I'll put together basic instructions and post them here shortly.

Yes I am using Ursa. Thanks!

[climbfuji]

Here are the commands that I used, and it is building now. Note that I started from the release/2.1 branches, which we created today for the release end of next week. It makes sense to do so, because any bugs we identify will be fixed for the 2.1.* releases (we won't go back and fix the 2.0.* releases).

git clone -b release/2.1 --recurse-submodules https://github.com/jcsda/spack-stack spst-rel21
cd spst-rel21
module purge
source setup.sh
spack stack create env --name=ue-oneapi-2025.2.1 --site=ursa --template=unified-dev --compiler=oneapi-2025.2.1 2>&1 | tee log.create.ue-oneapi-2025.2.1.001
spack env activate -p envs/ue-oneapi-2025.2.1/
spack concretize --force --fresh 2>&1 | tee log.concretize.ue-oneapi-2025.2.1.001

# Check build options and install
cat log.concretize.ue-oneapi-2025.2.1.001 | grep gsi-ncdiag
spack install --verbose --no-cache gsi-ncdiag 2>&1 | tee log.install.ue-oneapi-2025.2.1.gsi-ncdiag

[climbfuji]

@ADCollard If you want to split the install logs to just look at the gsi-diag install, then you can also replace the last step (spack install ...) with

spack install --verbose --no-cache --only=dependencies gsi-ncdiag 2>&1 | tee log.install.ue-oneapi-2025.2.1.gsi-ncdiag-deps
spack install --verbose --no-cache gsi-ncdiag 2>&1 | tee log.install.ue-oneapi-2025.2.1.gsi-ncdiag

[ADCollard]

Thankyou for that @climbfuji. I built the spack-stack following your instructions and the segmentation fault returned! I can now at least put in some print statements!

[climbfuji]

Thankyou for that @climbfuji. I built the spack-stack following your instructions and the segmentation fault returned! I can now at least put in some print statements!

You may need some additional help with modifying the code and then building it with spack. We are working on a much better solution for that, but for now the following steps should work:

After installing the gsi-ncdiag dependencies:

spack stage gsi-ncdiag
spack cd gsi-ncdiag
pwd
# Edit the code
spack install --overwrite --verbose --dont-restage --keep-stage gsi-ncdiag 2>&1 | tee log.install.gsi-ncdiag

If you do this in one shell and run your modified code in another, then you should get a reasonable turnaround.