The combination of threading and MPI regions that cross cubed-sphere panel edges is not bit-reproducible #294
Description
Are there any linked Issues or Pull Requests?
What happened?
It was found that after #138 was merged, that the gungho_model_baroclinic-C48_MG-3panel_azspice_gnu_fast-debug-64bit would sporadically produce different KGOs.
This will be avoided for now on main by setting the test to use a single thread (#293) but this suggests that tests which use both threading and have MPI regions that cross cubed-sphere panel edges are not bit-reproducible. My best hypothesis for now is that it relates to MetOffice/lfric_core#252