Skip to content

32-bit adjoint test main branch failures #298

@DrTVockerodtMO

Description

@DrTVockerodtMO

Version

main

Are there any linked Issues or Pull Requests?

No response

What happened?

Since the TLM boundary layer code was committed, we're experiencing adjoint test failures in the 32-bit runs. These should pass consistently.

A routine that consistently fails is atlt_bl_inc_alg, but other routines in the transport layer also sporadically fail their tests.

Relevant log output

20260225163814.622+0000:DEBUG:  atlt_bl_inc_alg: ad test for atl_bl_inc_kernel_type: ip1=   1.00000000       1.00000000
20260225163814.622+0000:DEBUG:  atlt_bl_inc_alg: ad test for atl_bl_inc_kernel_type: ip2=   2.03747026E-04   1.99751616
20260225163814.622+0000:DEBUG:  atlt_bl_inc_alg: ad test for atl_bl_inc_kernel_type: s(ip1),s(ip2)=   2.00000000       1.99771988
0260225163814.622+0000:ERROR:  FAILED tl_bl_inc:   2.00000000       1.99771988       9563.50000
20260226083005.134+0000:WARN : Field Min/max ref_mass, step        2 =       0.00000000 ****************
20260226083005.136+0000:WARN :    excess count =                1                0
20260226083005.137+0000:WARN :    degeneracy =                1                1
20260226083005.137+0000:WARN :      at level =               53               69
20260226083005.137+0000:WARN :     at height =   2.15054375E+04   7.60666875E+04
20260226083005.137+0000:WARN :   at latitude =      18.41423607     -84.70424652
20260226083005.137+0000:WARN :  at longitude =    -101.25000000      45.00000381
20260226083005.137+0000:ERROR: The transport scheme has a detected negative or tiny value   0.00000000E+00in the transport reference field ref_mass, step        2. This makes it impossible to find sensible departure distances for (Flux Form) Semi-Lagrangian transport. Stopping now to fail gracefully.

This is a common way for the model to fail. Most likely the transporting wind passed to the transport scheme has some unphysical values, and the corresponding maximum Lipschitz number exceeds 1 somewhere, so check these values in the log above, which may also indicate the geographical location of the issue.

If this failure has occurred in the first timestep, it can signify issues with the initial conditions or ancillaries.
#0  0x50e433 in __log_mod_MOD_log_event
	at utilities/log_mod.F90:466
#1  0x8271b2 in __flux_precomputations_alg_mod_MOD_check_negative_field
	at algorithm/transport/common/flux_precomputations_mod.f90:1209
#2  0x82ac20 in __flux_precomputations_alg_mod_MOD_get_ref_mass
	at algorithm/transport/common/flux_precomputations_mod.f90:667
#3  0x9295f7 in __init_from_controller_alg_mod_MOD_init_flux_pc_fieldvals
	at algorithm/transport/init_from_controller_alg_mod.f90:85
#4  0x88794b in __atlt_transport_field_alg_mod_MOD_atlt_transport_field_alg
	at algorithm/transport/control/atlt_transport_field_alg_mod.f90:87

Metadata

Metadata

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions