Changes between Version 31 and Version 32 of AdaptParallelization


Ignore:
Timestamp:
Sep 20, 2024, 11:42:58 AM (25 hours ago)
Author:
lnerger
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • AdaptParallelization

    v31 v32  
    2121Like many numerical models, PDAF uses the MPI standard for the parallelization. In the description below, we assume that the model is parallelized using MPI.
    2222
    23 PDAF supports a 2-level parallelization: First, the numerical model can be parallelized and can be executed using several processors. Second, several model tasks can be computed in parallel, i.e. a parallel ensemble integration can be performed. This 2-level parallelization has to be initialized before it can be used. The templates-directory  `templates/` contains the file `init_parallel_pdaf.F90` that can be used as a template for the initialization. The required variables are defined in `mod_parallel.F90`, which is stored int he same directory and can also be used as a template. If the numerical model itself is parallelized, this parallelization has to be adapted and modified for the 2-level parallelization of the data assimilation system generated by adding PDAF to the model. The necessary steps are described below.
     23PDAF supports a 2-level parallelization: First, the numerical model can be parallelized and can be executed using several processors. Second, several model tasks can be computed in parallel, i.e. a parallel ensemble integration can be performed. This 2-level parallelization has to be initialized before it can be used. The templates-directory  `templates/` contains the file `init_parallel_pdaf.F90` that can be used as a template for the initialization. The required variables are defined in `mod_parallel.F90`, which is stored in the same directory and can also be used as a template. If the numerical model itself is parallelized, this parallelization has to be adapted and modified for the 2-level parallelization of the data assimilation system generated by adding PDAF to the model. The necessary steps are described below.
    2424
    2525
    2626== Three communicators ==
    2727
    28 MPI uses so-called 'communicators' to define sets of parallel processes. In order to provide the 2-level parallelism, three communicators need to be initialized that define the processes that are involved in different tasks of the data assimilation system.
    29 The required communicators are initialized in the routine `init_parallel_pdaf` and called
    30  * `COMM_model` - defines the processes that are involved in the model integrations
    31  * `COMM_filter` - defines the processes that perform the filter analysis step
    32  * `COMM_couple` - defines the processes that are involved when data are transferred between the model and the filter
     28MPI uses so-called 'communicators' to define groups of parallel processes. These groups can then conveniently exchange information. In order to provide the 2-level parallelism for PDAF, three communicators need to be initialized that define the processes that are involved in different tasks of the data assimilation system.
     29The required communicators are initialized in the routine `init_parallel_pdaf`. There are called
     30 * `COMM_model` - defines the groups of processes that are involved in the model integrations (one group for each model task)
     31 * `COMM_filter` - defines the group of processes that perform the filter analysis step
     32 * `COMM_couple` - defines the groups of processes that are involved when data are transferred between the model and the filter
    3333
    3434The parallel region of an MPI parallel program is initialized by calling `MPI_init`.  By calling `MPI_init`, the communicator `MPI_COMM_WORLD` is initialized. This communicator is pre-defined by MPI to contain all processes of the MPI-parallel program. Often it is sufficient to conduct all parallel communication using only `MPI_COMM_WORLD`. Thus, numerical models often use only this communicator to control all communication. However, as `MPI_COMM_WORLD` contains all processes of the program, this approach will not allow for parallel model tasks. In order to allow parallel model tasks, it is required to replace `MPI_COMM_WORLD` by an alternative communicator that is split for the model tasks. We will denote this communicator `COMM_model`. If a model code already uses a communicator distinct from `MPI_COMM_WORLD`, it should be possible to use that communicator.
    3535
    3636[[Image(//pics/communicators_PDAFonline.png)]]
    37 [[BR]]'''Figure 1:''' Example of a typical configuration of the communicators using a parallelized model. In this example we have 12 processes over all, which are distributed over 3 model tasks (COMM_model) so that 3 model states can be integrated at the same time. COMM_couple combines each set of 3 communicators of the different model tasks. The filter is executed using COMM_filter which uses the same processes of the first model tasks, i.e. COMM_model 1 (Fig. credits: A. Corbin)
     37[[BR]]'''Figure 1:''' Example of a typical configuration of the communicators using a parallelized model. In this example we have 12 processes over all, which are distributed over 3 model tasks (COMM_model) so that 3 model states can be integrated at the same time. COMM_couple combines each set of 3 communicators of the different model tasks. The filter is executed using COMM_filter which uses the same processes of the first model tasks, i.e. COMM_model 1 (Figure credits: A. Corbin)
    3838
    3939== Using COMM_model ==
    … …  
    4545      CALL MPI_Comm_Size(MPI_COMM_WORLD, size, ierr)
    4646}}}
    47 (The call to `MPI_init` is mandatory, while the second an third line are optional) If the model itself is not parallelized, the MPI-initialization will not be present. Please see the section '[#Non-parallelmodels Non-parallel models]' below for this case.
     47(The call to `MPI_init` is mandatory, while the second an third lines are optional) If the model itself is not parallelized, the MPI-initialization will not be present. Please see the section '[#Non-parallelmodels Non-parallel models]' below for this case.
    4848
    4949Subsequently, one can define `COMM_model` by adding
    … …  
    5151      COMM_model = MPI_COMM_WORLD
    5252}}}
    53 In addition, the variable `COMM_model` has to be declared in a way such that all routines using the communicator can access it. The parallelization variables of the model are frequently hold in a module. In this case, it is easiest to add `COMM_model` as an integer variable here.  (The example declares `COMM_model` and other parallelization-related variables in `mod_parallel.F90`)
     53In addition, the variable `COMM_model` has to be declared in a way such that all routines using the communicator can access it. The parallelization variables of the model are frequently held in a Fortran module. In this case, it is easiest to add `COMM_model` as an integer variable here.  (The example declares `COMM_model` and other parallelization-related variables in `mod_parallel.F90`)
    5454
    5555Having defined the communicator `COMM_model`, the communicator `MPI_COMM_WORLD` has to be replaced by `COMM_model` in all routines that perform MPI communication, except in calls to `MPI_init`, `MPI_finalize`, and `MPI_abort`.