lfric_atm: Improve OpenMP thread scaling for SW and LW radiation kernels

[jirudaya]

Are there any linked Issues or Pull Requests?

No response

Brief description

The sw_kernel_mod.F90 and lw_kernel_mod.F90 files have manually added OpenMP pragmas to parallelize the loops that calculate the fluxes. These files are paired with the sw_kernel_mod.py and lw_kernel_mod.py transmute scripts, which are intended to process the .F90 files through PSyclone and re-add the OpenMP pragmas.

Currently, due to TransformationErrors, the transmute scripts remove the OpenMP pragmas but fail to re-add them. As a result, scaling performance worsens as the number of OpenMP threads increases.

Further details of the issue.

Message from sw_kernel_mod.py transmute script execution:

WARNING:root:Could not transform because:
 Transformation Error: Loop cannot be parallelised because:
Error: The write access to 'aer_mix_ratio' in 'aer_mix_ratio(mode_1:mode_last)' causes a write-write race condition. Variable: 'aer_mix_ratio'.
Error: The write access to 'aer_sw_absorption' in 'aer_sw_absorption(rmode_1:rmode_last)' causes a write-write race condition. Variable: 'aer_sw_absorption'.
Error: The write access to 'aer_sw_asymmetry' in 'aer_sw_asymmetry(rmode_1:rmode_last)' causes a write-write race condition. Variable: 'aer_sw_asymmetry'.
Error: The write access to 'aer_sw_scattering' in 'aer_sw_scattering(rmode_1:rmode_last)' causes a write-write race condition. Variable: 'aer_sw_scattering'.
Error: The write access to 'ch4' in 'ch4(wth_1:wth_last)' causes a write-write race condition. Variable: 'ch4'.
Error: The write access to 'cloud_drop_no_conc' in 'cloud_drop_no_conc(wth_1:wth_last)' causes a write-write race condition. Variable: 'cloud_drop_no_conc'.
Error: The write access to 'co' in 'co(wth_1:wth_last)' causes a write-write race condition. Variable: 'co'.
Error: The write access to 'co2' in 'co2(wth_1:wth_last)' causes a write-write race condition. Variable: 'co2'.
Error: The write access to 'conv_frozen_fraction' in 'conv_frozen_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_frozen_fraction'.
Error: The write access to 'conv_frozen_mmr' in 'conv_frozen_mmr(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_frozen_mmr'.
Error: The write access to 'conv_frozen_number' in 'conv_frozen_number(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_frozen_number'.
Error: The write access to 'conv_liquid_fraction' in 'conv_liquid_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_liquid_fraction'.
Error: The write access to 'conv_liquid_mmr' in 'conv_liquid_mmr(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_liquid_mmr'.
Error: The write access to 'cos_zenith_angle_rts' in 'cos_zenith_angle_rts(twod_1:twod_last)' causes a write-write race condition. Variable: 'cos_zenith_angle_rts'.
Error: The write access to 'cs' in 'cs(wth_1:wth_last)' causes a write-write race condition. Variable: 'cs'.
Error: The write access to 'd_mass' in 'd_mass(wth_1:wth_last)' causes a write-write race condition. Variable: 'd_mass'.
Error: The write access to 'frozen_fraction' in 'frozen_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'frozen_fraction'.
Error: The write access to 'h2' in 'h2(wth_1:wth_last)' causes a write-write race condition. Variable: 'h2'.
Error: The write access to 'h2o' in 'h2o(wth_1:wth_last)' causes a write-write race condition. Variable: 'h2o'.
Error: The write access to 'hcn' in 'hcn(wth_1:wth_last)' causes a write-write race condition. Variable: 'hcn'.
Error: The write access to 'he' in 'he(wth_1:wth_last)' causes a write-write race condition. Variable: 'he'.
Error: The write access to 'layer_heat_capacity' in 'layer_heat_capacity(wth_1:wth_last)' causes a write-write race condition. Variable: 'layer_heat_capacity'.
Error: The write access to 'li' in 'li(wth_1:wth_last)' causes a write-write race condition. Variable: 'li'.
Error: The write access to 'liquid_fraction' in 'liquid_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'liquid_fraction'.
Error: The write access to 'mcf' in 'mcf(wth_1:wth_last)' causes a write-write race condition. Variable: 'mcf'.
Error: The write access to 'mcl' in 'mcl(wth_1:wth_last)' causes a write-write race condition. Variable: 'mcl'.
Error: The write access to 'n2' in 'n2(wth_1:wth_last)' causes a write-write race condition. Variable: 'n2'.
Error: The write access to 'n2o' in 'n2o(wth_1:wth_last)' causes a write-write race condition. Variable: 'n2o'.
Error: The write access to 'n_ice' in 'n_ice(wth_1:wth_last)' causes a write-write race condition. Variable: 'n_ice'.
Error: The write access to 'na' in 'na(wth_1:wth_last)' causes a write-write race condition. Variable: 'na'.
Error: The write access to 'nh3' in 'nh3(wth_1:wth_last)' causes a write-write race condition. Variable: 'nh3'.
Error: The write access to 'o2' in 'o2(wth_1:wth_last)' causes a write-write race condition. Variable: 'o2'.
Error: The write access to 'o3' in 'o3(wth_1:wth_last)' causes a write-write race condition. Variable: 'o3'.
Error: The write access to 'orographic_correction_rts' in 'orographic_correction_rts(twod_1:twod_last)' causes a write-write race condition. Variable: 'orographic_correction_rts'.
Error: The write access to 'potassium' in 'potassium(wth_1:wth_last)' causes a write-write race condition. Variable: 'potassium'.
Error: The write access to 'pressure_in_wth' in 'pressure_in_wth(wth_1:wth_last)' causes a write-write race condition. Variable: 'pressure_in_wth'.
Error: The write access to 'profile_list' in 'profile_list(seg_start:seg_end)' causes a write-write race condition. Variable: 'profile_list'.
Error: The write access to 'radiative_cloud_fraction' in 'radiative_cloud_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'radiative_cloud_fraction'.
Error: The write access to 'radiative_conv_fraction' in 'radiative_conv_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'radiative_conv_fraction'.
Error: The write access to 'rand_seed' in 'rand_seed(twod_1:twod_last)' causes a write-write race condition. Variable: 'rand_seed'.
Error: The write access to 'rb' in 'rb(wth_1:wth_last)' causes a write-write race condition. Variable: 'rb'.
Error: The write access to 'rho_in_wth' in 'rho_in_wth(wth_1:wth_last)' causes a write-write race condition. Variable: 'rho_in_wth'.
Error: The write access to 'sigma_mi' in 'sigma_mi(wth_1:wth_last)' causes a write-write race condition. Variable: 'sigma_mi'.
Error: The write access to 'sigma_ml' in 'sigma_ml(wth_1:wth_last)' causes a write-write race condition. Variable: 'sigma_ml'.
Error: The write access to 'so2' in 'so2(wth_1:wth_last)' causes a write-write race condition. Variable: 'so2'.
Error: The write access to 'stellar_irradiance_rts' in 'stellar_irradiance_rts(twod_1:twod_last)' causes a write-write race condition. Variable: 'stellar_irradiance_rts'.
Error: The write access to 'sulphuric' in 'sulphuric(wth_1:wth_last)' causes a write-write race condition. Variable: 'sulphuric'.
Error: The write access to 'temperature_in_wth' in 'temperature_in_wth(wth_1:wth_last)' causes a write-write race condition. Variable: 'temperature_in_wth'.
Error: The write access to 'tile_fraction' in 'tile_fraction(tile_1:tile_last)' causes a write-write race condition. Variable: 'tile_fraction'.
Error: The write access to 'tile_sw_diffuse_albedo' in 'tile_sw_diffuse_albedo(rtile_1:rtile_last)' causes a write-write race condition. Variable: 'tile_sw_diffuse_albedo'.
Error: The write access to 'tile_sw_direct_albedo' in 'tile_sw_direct_albedo(rtile_1:rtile_last)' causes a write-write race condition. Variable: 'tile_sw_direct_albedo'.
Error: The write access to 'tio' in 'tio(wth_1:wth_last)' causes a write-write race condition. Variable: 'tio'.
Error: The write access to 'vo' in 'vo(wth_1:wth_last)' causes a write-write race condition. Variable: 'vo'.
Consider using the "ignore_dependencies_for" transformation option if this is a false dependency
Consider using the "array_privatisation" transformation option if this is a write-write dependency

Message from lw_kernel_mod.py transmute script execution:

WARNING:root:Could not transform because:
 Transformation Error: Loop cannot be parallelised because:
Error: The write access to 'aer_lw_absorption' in 'aer_lw_absorption(rmode_1:rmode_last)' causes a write-write race condition. Variable: 'aer_lw_absorption'.
Error: The write access to 'aer_lw_asymmetry' in 'aer_lw_asymmetry(rmode_1:rmode_last)' causes a write-write race condition. Variable: 'aer_lw_asymmetry'.
Error: The write access to 'aer_lw_scattering' in 'aer_lw_scattering(rmode_1:rmode_last)' causes a write-write race condition. Variable: 'aer_lw_scattering'.
Error: The write access to 'aer_mix_ratio' in 'aer_mix_ratio(mode_1:mode_last)' causes a write-write race condition. Variable: 'aer_mix_ratio'.
Error: The write access to 'ch4' in 'ch4(wth_1:wth_last)' causes a write-write race condition. Variable: 'ch4'.
Error: The write access to 'cloud_drop_no_conc' in 'cloud_drop_no_conc(wth_1:wth_last)' causes a write-write race condition. Variable: 'cloud_drop_no_conc'.
Error: The write access to 'co' in 'co(wth_1:wth_last)' causes a write-write race condition. Variable: 'co'.
Error: The write access to 'co2' in 'co2(wth_1:wth_last)' causes a write-write race condition. Variable: 'co2'.
Error: The write access to 'conv_frozen_fraction' in 'conv_frozen_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_frozen_fraction'.
Error: The write access to 'conv_frozen_mmr' in 'conv_frozen_mmr(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_frozen_mmr'.
Error: The write access to 'conv_frozen_number' in 'conv_frozen_number(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_frozen_number'.
Error: The write access to 'conv_liquid_fraction' in 'conv_liquid_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_liquid_fraction'.
Error: The write access to 'conv_liquid_mmr' in 'conv_liquid_mmr(wth_1:wth_last)' causes a write-write race condition. Variable: 'conv_liquid_mmr'.
Error: The write access to 'cs' in 'cs(wth_1:wth_last)' causes a write-write race condition. Variable: 'cs'.
Error: The write access to 'd_mass' in 'd_mass(wth_1:wth_last)' causes a write-write race condition. Variable: 'd_mass'.
Error: The write access to 'frozen_fraction' in 'frozen_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'frozen_fraction'.
Error: The write access to 'h2' in 'h2(wth_1:wth_last)' causes a write-write race condition. Variable: 'h2'.
Error: The write access to 'h2o' in 'h2o(wth_1:wth_last)' causes a write-write race condition. Variable: 'h2o'.
Error: The write access to 'hcn' in 'hcn(wth_1:wth_last)' causes a write-write race condition. Variable: 'hcn'.
Error: The write access to 'he' in 'he(wth_1:wth_last)' causes a write-write race condition. Variable: 'he'.
Error: The write access to 'layer_heat_capacity' in 'layer_heat_capacity(wth_1:wth_last)' causes a write-write race condition. Variable: 'layer_heat_capacity'.
Error: The write access to 'li' in 'li(wth_1:wth_last)' causes a write-write race condition. Variable: 'li'.
Error: The write access to 'liquid_fraction' in 'liquid_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'liquid_fraction'.
Error: The write access to 'mcf' in 'mcf(wth_1:wth_last)' causes a write-write race condition. Variable: 'mcf'.
Error: The write access to 'mcl' in 'mcl(wth_1:wth_last)' causes a write-write race condition. Variable: 'mcl'.
Error: The write access to 'n2' in 'n2(wth_1:wth_last)' causes a write-write race condition. Variable: 'n2'.
Error: The write access to 'n2o' in 'n2o(wth_1:wth_last)' causes a write-write race condition. Variable: 'n2o'.
Error: The write access to 'n_ice' in 'n_ice(wth_1:wth_last)' causes a write-write race condition. Variable: 'n_ice'.
Error: The write access to 'na' in 'na(wth_1:wth_last)' causes a write-write race condition. Variable: 'na'.
Error: The write access to 'nh3' in 'nh3(wth_1:wth_last)' causes a write-write race condition. Variable: 'nh3'.
Error: The write access to 'o2' in 'o2(wth_1:wth_last)' causes a write-write race condition. Variable: 'o2'.
Error: The write access to 'o3' in 'o3(wth_1:wth_last)' causes a write-write race condition. Variable: 'o3'.
Error: The write access to 'potassium' in 'potassium(wth_1:wth_last)' causes a write-write race condition. Variable: 'potassium'.
Error: The write access to 'pressure_in_wth' in 'pressure_in_wth(wth_1:wth_last)' causes a write-write race condition. Variable: 'pressure_in_wth'.
Error: The write access to 'profile_list' in 'profile_list(seg_start:seg_end)' causes a write-write race condition. Variable: 'profile_list'.
Error: The write access to 'radiative_cloud_fraction' in 'radiative_cloud_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'radiative_cloud_fraction'.
Error: The write access to 'radiative_conv_fraction' in 'radiative_conv_fraction(wth_1:wth_last)' causes a write-write race condition. Variable: 'radiative_conv_fraction'.
Error: The write access to 'rand_seed' in 'rand_seed(twod_1:twod_last)' causes a write-write race condition. Variable: 'rand_seed'.
Error: The write access to 'rb' in 'rb(wth_1:wth_last)' causes a write-write race condition. Variable: 'rb'.
Error: The write access to 'rho_in_wth' in 'rho_in_wth(wth_1:wth_last)' causes a write-write race condition. Variable: 'rho_in_wth'.
Error: The write access to 'sigma_mi' in 'sigma_mi(wth_1:wth_last)' causes a write-write race condition. Variable: 'sigma_mi'.
Error: The write access to 'sigma_ml' in 'sigma_ml(wth_1:wth_last)' causes a write-write race condition. Variable: 'sigma_ml'.
Error: The write access to 'so2' in 'so2(wth_1:wth_last)' causes a write-write race condition. Variable: 'so2'.
Error: The write access to 'sulphuric' in 'sulphuric(wth_1:wth_last)' causes a write-write race condition. Variable: 'sulphuric'.
Error: The write access to 't_layer_boundaries' in 't_layer_boundaries(flux_0:flux_last)' causes a write-write race condition. Variable: 't_layer_boundaries'.
Error: The write access to 'temperature_in_wth' in 'temperature_in_wth(wth_1:wth_last)' causes a write-write race condition. Variable: 'temperature_in_wth'.
Error: The write access to 'tile_fraction' in 'tile_fraction(tile_1:tile_last)' causes a write-write race condition. Variable: 'tile_fraction'.
Error: The write access to 'tile_lw_albedo' in 'tile_lw_albedo(rtile_1:rtile_last)' causes a write-write race condition. Variable: 'tile_lw_albedo'.
Error: The write access to 'tile_temperature' in 'tile_temperature(tile_1:tile_last)' causes a write-write race condition. Variable: 'tile_temperature'.
Error: The write access to 'tio' in 'tio(wth_1:wth_last)' causes a write-write race condition. Variable: 'tio'.
Error: The write access to 'vo' in 'vo(wth_1:wth_last)' causes a write-write race condition. Variable: 'vo'.
Consider using the "ignore_dependencies_for" transformation option if this is a false dependency
Consider using the "array_privatisation" transformation option if this is a write-write dependency