Repository navigation
Using ExtDataDriver.x
ExtDataDriver.x is a program maintained as part of the MAPL library. Despite the name the program is designed to allow for testing of BOTH the ExtData and History components for CI testing, development, and troubleshooting these two components. The program basically implements something like the MAPL Cap but for several reasons does not use the MAPL cap. This "CAP" object instantiates ExtData/History/and a root MAPL component just like the regular MAPL Cap and can tilmestep these components in the same order but allows for a little finer grain control. In addition the the program allows to user to run essentially multiple executions of this "Cap" within a single execution of the overall ExtDataDriver.x and modify the behavior of ExtData, History, and root component in each execution. Finally as part of the application a custom MAPL component has been implemented. This component allows runtime fields to be added to the component in either the import or export state on any supported layout and grid that MAPL supports and in the case of export fields fill those with synthetic data. In this way, pretty much any field or group of fields can be generated to feed to either ExtData or History for testing/development/debugging purposes. An obvious use case if a user for example says, I tried this combination of History options for a collection and it failed. The the root component can be setup to produce any number/dimensionality of fields that can be written by History where it can be debugged to mimic the fields the user may be trying to use for example.
Like the a regular MAPL application that uses the MAPL Cap ExtDataDriver.x is driven by a CAP.rc which in turn specifies the History.rc, ExtData.rc, and rc file for the root of the MAPL hierarchy so this should be familiar to anyone who has used the GEOS model.
The main differences are:
- The "CAP.rc" actually specifies a list of other CAP.rc files (I'll call those the sub_CAP.rc files that are similar to the regular CAP.rc and allows one execution of the program to run multiple instantiations of the History/ExtData/MAPL hierarchy in one execution. The reasons for why this is powerful will hopefully become clear in the examples.
- The sub_CAP.rc files are similar to the regular CAP.rc in that they specify the HEARTBEAT and the name of the Root/History/ExtData rc files but allows for more flexibility. For example these allow the user specify times the system should run for example.
- The root component rc file specifies the component grid just like the AGCM.rc in the GEOS model and has options to at runtime specify the fields that should to in the import and export state of the component, as well as the dimensionality and how to fill with synthetic data. It also doing some operations with these fields that will be described later.
ExtDataDriver.x consists of the following pieces that can be found under the Test directory in MAPL
ExtDataDriver.F90: This is the top level program, not much here, looks like GEOSgcm.x
ExtDataDriverMod.F90: This is basically a reimplementation of MAPL_Cap with some extra bells and whistles. It initializes MAPL, starts the output server and runs the "cap-like" grid comp which is the grid comp defined in ExtDataDriverGridComp.F90. The crucial difference is that it can run multiple instanciations of the GridComp defined in ExtDataDriverGridComp.F90 sequentially if desired.
ExtDataDriverGridComp.F90: This is basically a reimplementation of MAPL_CapGridComp with a few more bells and whistles but at the end of the day runs ExtData, a root component, and History at one or more time steps and ticks a clock. It just allows more fine-grained control over when it's 3 child components run rather than just ticking a clock and running them at every step.
ExtDataRoot_GridComp.F90: This is a MAPL component that is the root component (and only component) underneath the ExtDataDriverGridComp. Like any good gridded component it creates a grid, some fields in its import/export/internal state on that grid and may do something with them.
As part of the MAPL library, inputs for multiple test cases that are used to run ExtDataDriver.x for CI testing are maintained. These can be found under the main MAPL directory at Tests/ExtData_Testing_Framework/test_cases.
To illustrate how ExtDataDriver.x works I will go over the various input files for case1 under the test cases
CASES::
CAP1.rc
CAP2.rc
::
The main CAP.rc is just a list of the sub CAP.rc files. It instantiates a ExtDataDriverGridComp for each one, runs said instance using the sub CAP.rc sequentially. If there is only one item in this list, it is very must like just the standard MAPL_Cap. In the example above, it will will instantiate a ExtDataDriverGridComop using CAP1.rc, run it (this means run it's init/run/finalize methods), then instantiate a new gridcomp using CAP2.rc and run that.
Let's look at CAP1.rc:
ROOT_NAME: Root
ROOT_CF: AGCM1.rc
HIST_CF: HISTORY1.rc
BEG_DATE: 20040101 210000
JOB_SGMT: 00001200 000000
HEARTBEAT_DT: 3600
#RUN_EXTDATA: .false.
RUN_TIMES::
20040115 210000
20040215 210000
20040315 210000
20040415 210000
20040515 210000
20040615 210000
20040715 210000
20040815 210000
20040915 210000
20041015 210000
20041115 210000
20041215 210000
::
This looks similar to the regular CAP.rc used when running GEOS with a few exceptions. At the top it defines the ROOT configuration files and the ExtData and History configuration files. In this case it says, my root component will used a file named AGCM1.rc and my History will use a file name HISTORY1.rc and since I didn't specify an ExtData file it uses the default name. It defines the BEG_DATE of the clock it will create, the length of time to tick that clock (JOB_SGMT), and the HEARTBEAT_DT. The difference the RUN_TIMES table. This is optional but says, only execute the child (History/Root/ExtData) at these specify times. If omitted it just ticks the clock for the desired length and runs at every step.
At the steps when ExtDataDriverGridComp does execute the full run method it will execute ExtData, the root component, then History
There is really nothing to say about these. They are just the input to History and ExtData so you can look elsewhere for the syntax but in this case HISTORY1.rc is saying output VAR2D and VAR3D from a component named Root into a single file per year and HISTORY2.rc defines no collections to a History instantiation using this will do nothing!
So by now you are probably asking what's the point of of this. Why not just run ExtDataDriver.x twice and not bother with this CAP.rc file that itself specifies the individual RC files. You are right, you could do that but this way I can get away with one execution. But you will say, ok fine, but what is the point of this case?
In this case and indeed all the test cases I have in MAPL I can do self consistent testing of both History AND ExtData. In one execution of ExtDataDriver.x I can have it generate output files via History, then read those files back in via ExtData, then test that what I read in were read in properly (since I know what I put out in the first place!). That's the point of these test cases.
So CAP1.rc defines a instantiation of ExtDataDriverGridComp that outputs some stuff in History from the exports of the root component which are filled with data by the root component. CAP2.rc defines an instantiation of ExtDataDriverGridComp that outputs nothing, but now has some imports which get filled with ExtData, it has some exports that are filled by root component and it compares the states, field by field. If they don't match, something either in History or ExtData did not do something right!