Get build of drivers and postproc to work with Git

[marcstring]

Now the MOCI code has moved to Git is there a way to build both the drivers and postproc using the Git code?

I've tried to do this for my UKESM2_proto job, u-dw561, in https://code.metoffice.gov.uk/trac/UKESM/wiki/ticket/1087/drivers and https://code.metoffice.gov.uk/trac/UKESM/wiki/ticket/1087/postproc, but I didn't really know what I was doing. Hence, I e-mailed the SSD team to see if they had ideas on how this should be done. They replied after a few days with some general tips on how this might be done.

I had hoped the SSD team might take this on. After the CRUM team, who use to own and manage the MOCI software was disbanded, I think the SSD team were lumped with managing the MOCI software, but I appreciate that they've also been given very little additional resource, if any, to do so.

Hence, in https://code.metoffice.gov.uk/trac/UKESM/wiki/ticket/1087/drivers_postproc_follow_up I've had another go at building the drivers and postproc code. In here I've created a new job, https://code.metoffice.gov.uk/trac/roses-u/log/d/x/0/3/3, where I extract the MOCI code using their merge scripts and adding the MOCI repository to the dependencies.yaml file.

I think the drivers only require the code extracting. Hence, in u-dx033 I've scrapped the fcm_make_drivers and fcm_make2_drivers tasks. However, the run_model script references the directory with the code in it in several places, so I've had to edit this. This I've done in my branch https://github.com/marcstring/moci/tree/build_drivers_and_postproc.

I think postproc is more complicated than the drivers, because its code is spread across many directories and yet I think when postproc runs the code is expected to be in one directory. Hence, I think we can scrap the fcm_make*_pp task which extracts the code (this is now done through the merge scripts and by adding MOCI to dependencies.yaml), but we still need one fcm_make*pp task to pull all the required code into directory. I think this requires editing Postproc/fcm_make/postproc.cfg, which I've done in my build_drivers_and_postproc branch.

I still think there need to be conversations with people like those at the coupled model meetings and Pierre and Erica from the SSD team to see if this best way of doing this.

[Pierre-siddall]

Hi Marc,

Having taken a quick read of this my gut feeling (with the still somewhat shaky knowledge of MOCI I have ) is that this seems correct to me , as @ericaneininger and I intend to keep MOCI as software which can build using on the standard python libraries. When it comes to how postproc and drivers would actually build I think your approach to change old svn links to point at github is a good start, however I think Erica would need to confirm this as I'm not fully certain on the MOCI builds.

[ericaneininger]

I am currently looking at possibilities for this with some of the MOCI test suites.

You are perfectly correct in that we can get rid of the fcm_make_drivers entirely, since it is only ever extracted these days anyway. The only fcm_make functionality used by fcm_make_pp outside of the no longer required mirror is, as you say, to get everything into a nice flat directory structure. It may be better just to write out own "build" script which does this. This is particularly relevant since we are working on a mocilib library which currently is accessed only by modifying PYTHONPATH for the application run tasks - not ideal, though this is how access to the drivers located in the extract directory is done.

One potential pitfall to your current approach is that there is every possibility that drivers and postproc will want different revisions - certainly there are keywords on the repository for both applications and metadata to match those. There might need to be two extractions (moci_drivers & moci_pp). Currently the SSD clone and merge scripts will only extract the entire repo - which is probably overkill here.

It's perfectly possibly to test the changes you've put into postproc.cfg without actually changing the configuration in the repo - just with appropriate settings in the rose-app.conf.

[marcstring]

Thanks a lot Pierre and Erica. It's great that you're taking a look at this. I'm aware that with CRUM disbanded there is unfortunately very little people resource to support this.

I think you're aware that some of us have developments to both the drivers and postproc. We're told we have to move to git if we ever want them on the trunk, any yet it's still unclear how to use this software under git. Hence, great if you guys are remedying this.

Erica, I agree that in the past we've often used different revisions of MOCI for the drivers and postproc, and my suggested method above would fix you to one revision for both - a definite downside. Slightly mitigating this is that is seems to be easier to upgrade branches to later revisions of the trunk (aka main) with git than svn, so easier to get driver and postproc branches to the same revision/tag, but great if you've found a better way of building both drivers and postproc.

I'll leave this with you if my help is not required. Thanks!

[ericaneininger]

I've been making proposed changes to rose-stem which retires the fcm_make_[pp|drivers] tasks . This appears to do a sensible "build" of postproc, which happily runs on both spice and HPC.
I'm having trouble testing the coupled model, but at the moment these appear to be intermittent Git and HPC issues rather than anything real. I'm pretty sure the changes to Coupled_Drivers/run_model are sound.

One concern I have about this set up is that anything which is put into the CYLC_WORKFLOW_RUN_DIR/bin directory after install (see SOURCE_EXTRACTION task) is removed if we do a re-install. Not really an issue for MOCI rose-stem, but potentially a problem for long running suites.

https://github.com/ericaneininger/moci/tree/fcm_make_redundancy

[ericaneininger]

I've opened PR #35 for my rose-stem branch. It doesn't directly close this issue because it's doesn't directly change UKESM.

Both postproc and coupled tasks run smoothly in rose-stem - with one new app added - pp_build. Coupled tasks require the addition/modification of the environment variable $DRIVERS_EXTRACT_DIR=<dependencies.yaml sources extraction directory>/moci

I DO NOT recommend copying the current rose-stem SOURCE_EXTRACTION task copy to $CYLC_WORKFLOW_RUN_DIR in a normal suite - anything put in here outside of the install process will be deleted upon reinstall of the suite. A possible alternative location is $CYLC_WORKFLOW_SHARE_DIR/bin, which is also added to the path automatically but is not affected by reinstall. This is where I have put the postproc source code by default.