lfric2lfric: do not partition multigrid meshes

[ukmo-juan-castillo]

Are there any linked Issues or Pull Requests?

MetOffice/lfric_core#288

Brief description

When reading a multigrid mesh file, by default all meshes this file contains are partitioned. This behaviour is suitable for lfric jobs, as all meshes need to be partitioned. In lfric2lfric this is not true, and only the source and destination meshes need to be partitioned. In any case, partitioning all meshes in the multigrid mesh file imposes a series of conditions regarding the number of CPUs that can be used in a job. This can be a limitation in lfric2lfric, given that already the source and destination meshes can be very different.

One simple solution is for lfric2lfric to just read mesh files containing one single mesh. For flexibility though, it would be desirable that lfric2lfric just partitioned, and made checks on mesh dimensions against CPU number, on the source and destination meshes. This functionality is already present in lfric_core via the any_maps parameter of the get_partition subroutine of the panel_decomposition_type object. On the other hand, this object subroutine is not called directly in most applications, so this functionality should be made more accessible.

Further details of the issue.

No response

[ukmo-juan-castillo]

Comments from @mo-rickywong :

"When reading a multigrid mesh file, by default all meshes this file contains are partitioned. It would be desirable for lfric2lfric to just partition, and make checks on mesh dimensions against CPU number, on the source and destination meshes."
I don't see your point, you are already reading different meshes and applying different partitioning strategies to each mesh.

True, but the source and destination meshes are read from mesh files, each of which can be multigrid mesh files. Having a multigrid mesh file imposes constraints in the number of CPUs that can be used, as each mesh dimension needs to be divisible by the number of CPUs. It is you who requested eliminating this constraint, and this PR addresses exactly that.

"This functionality is already present in lfric_core via the any_maps parameter of the get_partition subroutine of the panel_decomposition_type object. On the other hand, this object subroutine is not called directly in most applications, so this functionality should be made more accessible."
any_maps should never have been part of partitioning infrastructure. It was added without due oversight. I wouldn't want it to be expanded on, if anything it should be removed or changed.

I do not argue this point. This ticket is adding the new requested functionality based in the present status of the code. If the existing code needs to be redesigned, a dedicated issue should be opened to address it.

The solution proposed here is the creation of a logical variable in the global_mesh object, with set/get subroutines, indicating if the multiple meshes have to be partitioned.
This doesn't make sense, lfric2lfric wants to partition the source and destination meshes. It's not whether lfric2lfric wants to partition or not, it's whether it is concerned with the fact that the mesh input file contains intergrid maps.

Again, the purpose of the ticket is not stopping partitioning the meshes, but to read only the mesh that is actually used in the multigrid mesh file, and this PR achieves that. Partitioning the meshes is code deep within lfric_core, and if you suggest that this partitioning should be done in lfric2lfric, this would require large pieces of lfric_core duplicated in lfric2lfric. This is impractical: if other application would require this in the future, the code would have to be duplicated again, and it would be more difficult to maintain and keep synchronized.

This is similar to what is presently done, where the global_mesh variable ntarget_meshes is used to find if there are multiple meshes.
Is it? ntarget_meshes is used to inform as to how many intergrid maps are in the file where this mesh is the source. It has nothing to do with how many meshes should be read or tested for partitioning.

It is also used to decide if more than one mesh needs to be partitioned, which is the point concerning this PR. I should have said 'if there are multiple meshes to be partitioned'.

Extracting the any_maps parameter of the subroutine get_partition contained in the object panel_decomposition_type in successive subroutine calls until it can be used in the calling subroutine create_local_mesh. This would imply large changes in both lfric_core and lfric_apps
any_maps is set from the call to the mesh global_mesh%get_nmaps() > 0 so you don't actually need to do anything panel_decomposition_type. I'm not sure if what you and I consider as large is the same, though I'd have to look in more detail. In any case, lfric2lfric doesn't care whether if the mesh has any_maps=.true., it just doesn't want the partitioning check to be applied. Remember, nmaps has nothing to do with maps on the object, it just says how many maps for that mesh as source are available in the file, the application doesn't have to read them in.

any_maps is used in the panel_decomposition_type object when deciding if more than one mesh needs to be partitioned. Changing this behaviour outside the panel_decomposition_type object would imply, as a minimum, duplicating the code for panel_decomposition_type, as lfric2lfric still needs to know the partition method for each mesh.

Doing the same as before, but using optional subroutine parameters. Although this would minimise the number of changes in lfric_core and lfric_apps, the addition of new optional subroutine parameters is discouraged.
'same as before' ??? what are you referring to? I'd discourage optional arguments, if you pass in a logical, it's .true. or .false. so you don't need to test on it's presence.

I am referring at passing the any_maps parameter successively as a parameter, but this time as an optional parameter.

Make changes in lfric2lfric to use the existing any_maps parameter of the subroutine get_partition. Lfric_core would not change, but this would imply duplicating large pieces of lfric_core code in lfric2lfric.
See earlier comments referring to the use of any_maps, I'm not sure which bits of code you are talking about duplicating.

Again, as a minimum the panel_decomposition_type object would need to be duplicated.

Unfortunately, the core code has had code added to it which it should know nothing about (in my opinion). It's all connected with multigrid which is a Gungho implementation and the restrictions should all be in Gungho science and not in infrastructure. All of your suggestions appear to be building on this dubious code that would appear to be in the wrong place. This will only make it more difficult to extract this code out of infrastructure and put it where it belongs in Gungho science.

If there is dubious code at the moment, then a dedicated issue should be opened to fix it. This PR is just building on top of the existing code and its intention is not redesigning a large piece of code, just adding a functionality that was requested, and does exactly that by adding as few new lines of code as I can think of. Again, I would be happy to implement and test any other solution that is proposed.