Offline Mode: Initializing the parallelization for PDAF

Overview

The PDAF release provides example code for the offline mode in tutorial/offline_2D_parallel. We refer to this code to use it as a basis.

If the the tutorial implementation tutorial/offline_2D_parallel, or the template code in templates/offline is used as the basis for the implementation, it one can simply keep this part of the program, i.e. the call to init_parallel_pdaf unchanged and directly proceed to the initialization of PDAF for the offline mode.

Like many numerical models, PDAF uses the MPI standard for the parallelization. PDAF requires for the compilation that an MPI library is available. In any case, it is necessary to execute the routine init_parallel_pdaf as described below.

For completeness, we describe here the communicators required for PDAF as well as the template initialization routine.

Three communicators

MPI uses so-called 'communicators' to define sets of parallel processes. In the offline mode, the communicators of the PDAF parallelization are only used internally to PDAF. However, as the communicators need to be initialized and provided to PDAF_init to initialized PDAF, we describe them here.

The initialization routine PDAF_init is used for both the offline and online modes. Due to this, three communicators need to be initialized that define the groups of processes that are involved in different tasks of the data assimilation system. The required communicators are initialized in the routine init_parallel_pdaf and are named

• COMM_model - defines the processes that are involved in the model integrations
• COMM_filter - defines the processes that perform the filter analysis step
• COMM_couple - defines the processes that are involved when data are transferred between the model and the filter

For the offline mode, only COMM_filter is relevant. COMM_model is identical to COMM_filter and COMM_couple is not actively used.

The parallel region of an MPI parallel program is initialized by calling MPI_init. By calling MPI_init, the communicator MPI_COMM_WORLD is initialized. This communicator is pre-defined by MPI to contain all processes of the MPI-parallel program. In the offline mode, it would be sufficient to conduct all parallel communication using only MPI_COMM_WORLD. However, as PDAF uses internally the three communicators listed above, they have to be initialized. In general they will be identical to MPI_COMM_WORLD with the exception of COMM_couple, which is not used in the offline mode.

Figure 1: Example of a typical configuration of the communicators for the offline coupled assimilation with parallelization. We have 4 processes in this example. The communicators COMM_model, COMM_filter, and COMM_couple need to be initialized. COMM_model and COMM_filter use all processes (as MPI_COMM_WORLD) while COMM_couple is a group of communicators each holding only one process. Thus, the initialization of the communicators is simpler than in case of the parallelization for the online coupling.

Initializing the parallelization

The routine init_parallel_pdaf, which is supplied in templates/offline and tutorial/offline_2D_parallel, initializes the necessary communicators for the assimilation program and PDAF. The routine is called at the beginning of the assimilation program directly after the initialization of the parallelization described above. The provided routine init_parallel_pdaf is a template implementation. For the offline mode, it should not be necessary to modify it!

init_parallel_pdaf is identical for the online and offline modes. The routine uses the variable n_modeltasks. For the offline mode, the provided code sets this variable to 1, because no integrations are performed in the assimilation program. The routine the variables npes_world, mype_world as well as npes_filter and mype_filter are defined. Here npes_world and npes_filter provide the number of processes involved in the program, while mype_world and mype_filter identify the index of a process (its 'rank'). These variables can be used in the user-supplied routines to control, for example, which process write information to the screen. In the example codes we use 'mype_filter' in the calll-back routines, while we use 'mype_world' in the main part of the code.

init_parallel_pdaf defines several more variables that are declared and held in the module mod_parallel_pdaf. We recommend to use this module from the provided code as some of these variables are required when the initialization routine of PDAF (PDAF_init) is called.

Arguments of init_parallel_pdaf

The routine init_parallel_pdaf has two arguments, which are the following:

SUBROUTINE init_parallel_pdaf(dim_ens, screen)

• dim_ens: An integer defining the ensemble size. This allows to check the consistency of the ensemble size with the number of processes of the program. For the offline mode, one should set this variable to 0. In this case no consistency check for the ensemble size with regard to parallelization is performed.
• screen: An integer defining whether information output is displayed. The following choices are available:
  • 0: quite mode - no information is displayed.
  • 1: Display standard information about the configuration of the processes (recommended)
  • 2: Display detailed information for debugging

Testing the assimilation program

One can compile the template code and run it, e.g., with mpirun -np 4 PDAF_offline. Close to the beginning of the output, one will see the lines

 Initialize communicators for assimilation with PDAF

                  PE configuration:
   world   filter     model        couple     filterPE
   rank     rank   task   rank   task   rank    T/F
  ----------------------------------------------------------
     0       0      1      0      1      0       T
     1       1      1      1      2      0       T
     2       2      1      2      3      0       T
     3       3      1      3      4      0       T

These lines show the configuration of the communicators. Here, 'world rank' is the value of 'mype_world', and 'filter rank' is the value of 'mype_filter'. These ranks always start with 0.