Changes between Version 3 and Version 4 of OnlineAdaptParallelization_PDAF3


Ignore:
Timestamp:
Apr 16, 2026, 6:47:08 PM (9 hours ago)
Author:
lnerger
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • OnlineAdaptParallelization_PDAF3

    v3 v4  
    1717[[PageOutline(2-3,Contents of this page)]]
    1818
     19|| This page described the adaption of the model parallelization for PDAF V3.0 and later. For PDAF V2.x, see the  [wiki:AdaptParallelization_PDAF23 Page on adapting the parallelization in PDAF 2]
     20
    1921== Overview ==
    2022
    21 The PDAF release provides example code for the online mode in `tutorial/online_2D_serialmodel` and `tutorial/online_2D_parallelmodel`. We refer to this code to use it as a basis.
     23The PDAF release provides example code for the online mode in `tutorial/online_2D_serialmodel` (for a model without parallelization) and `tutorial/online_2D_parallelmodel` (for a parallelized model). We refer to this code to use it as a basis.
    2224
    23 In the tutorial code and the templates in `templates/online`, the parallelization is initialized in the routine `init_parallel_pdaf` (file `init_parallel_pdaf.F90`). The required variables are defined in `mod_parallel.F90`. These files can be used as templates.
     25In the tutorial code and the templates in `templates/online`, the parallelization is initialized in the routine `init_parallel_pdaf` (file `init_parallel_pdaf.F90`). The required variables are defined in `mod_parallel_pdaf.F90`. These files can be used as templates.
    2426
    25 In implementations done with PDAF V3.0 and later, `init_parallel_pdaf` provides the parallelization variables to PDAF by a call to `PDAF3_set_parallel`. Implementations done before do usually not include this call, but provide the variables to PDAF in the initialization call to `PDAF_init`.
     27In implementations done with PDAF V3.1 and later `init_parallel_pdaf` calls the routine `PDAF3_init_parallel` to configure the parallelization for PDAF. In PDAF V3.0 and before the `init_parallel_pdaf` contains the actual configuration of the parallelization. In PDAF V3.0 the parallelization variables are then provided to PDAF by a call to `PDAF3_set_parallel`. Implementations done before do usually not include this call, but provide the variables to PDAF in the initialization call to `PDAF_init`.
    2628
    2729Like many numerical models, PDAF uses the MPI standard for the parallelization. For the case of a parallelized model, we assume in the description below that the model is parallelized using MPI. If the model is parallelized using OpenMP, one can follow the explanations for a non-parallel model below.
    … …  
    3133There are two possible cases regarding the parallelization for enabling the 2-level parallelization:
    32341. The model itself is parallelized using MPI
    33  * In this case we need to adapt the parallelization of the model
    34 2. The model is not parallelized or uses only shared-memory parallelization using OpenMP
    35  * In this case we need to add parallelization
     35 * In this case we need to adapt the parallelization of the model for the data assimilation with PDAF
     362. The model is not parallelized, i.e. a ''serial'' model, or uses only shared-memory parallelization using OpenMP
     37 * In this case we use PDAF to add parallelization
    3638
    3739== Adaptions for a parallelized model ==
    … …  
    4244
    4345If you are experienced with MPI, the steps are the following:
    44 1. Find the call to `MPI_init`
    45 1. Insert the call `init_parallel_pdaf` directly after `MPI_init`.
    46461. Check whether the model uses `MPI_COMM_WORLD`.
    4747 * If yes, then replace `MPI_COMM_WORLD` in all places, except `MPI_abort` and `MPI_finalize` by a user-defined communicator (we call it `COMM_mymodel` here), which can be initialized as `COMM_mymodel=MPI_COMM_WORLD`.
    48  * If no, then take note of the name of the communicator variable (we assume here it's `COMM_mymodel`). The next steps are valid if the model uses `COMM_mymodel` in a way that all processes of the program are included (thus analogous to `MPI_COMM_WORLD`). If the model is using less processes, this is a [#SpecialCase special case, which we discuss further below].
    49 1. Adapt `init_parallel_pdaf` so that at the end of this routine you set `COMM_mymodel=COMM_model`. Potentially, also set the rank and size variables of the model, respectively, by `mype_model` and `npes_model`.
    50 1. The number of model tasks in variable `n_modeltasks` is required by `init_parallel_pdaf` to perform commucator splitting. In the tutorial code we added a command-line parsing to set the variable (it is parsing for `dim_ens`). One could also read the value from a configuration file.
     48 * If no, then take note of the name of the communicator variable (we assume here it's `COMM_mymodel`).
     491. Find the call to `MPI_init`
     501. Insert the call `init_parallel_pdaf` directly after `MPI_init` and possible calls to `MPI_Comm_size` and `MPI_Comm_rank` providing `COMM_mymodel` and the related rank and size variables.
     511. The number of model tasks in variable `n_modeltasks` is required by `PDAF3_init_parallel` to perform communicator splitting. In the tutorial code we include a command-line parsing to set the variable (parsed with the keyword `dim_ens`, thus specifying `-dim_ens N_MODELTASKS` on the command line, where N_MODELTASKS should be the ensemble size). One could also read the value from a configuration file.
    5152
    5253
    5354=== Details on adapting a parallelized model ===
    5455
    55 Any program parallelized with MPI will need to call `MPI_Init` for the initialziation of MPI. Frequently, the parallelization of a model is initialized in the model by the lines:
     56Any program parallelized with MPI will need to call `MPI_Init` for the initialziation of MPI.
     57Frequently, the parallelization of a model is initialized in the model by the lines:
    5658{{{
    5759      CALL MPI_Init(ierr)
    … …  
    6365In the model code, we have to find the place where `MPI_init` is called to check how the parallelization is set up. In particular, we have to check if the parallelization is ready to be split into model tasks.
    6466For this one has to check if `MPI_COMM_WORLD`, e.g. checking the calls to `MPI_Comm_rank` and `MPI_Comm_size`, or MPI communication calls in the code (e.g. MPI_Send, MPI_Recv, MPI_Barrier).
    65 * If the model uses `MPI_COMM_WORLD` we have to replace this by a user-defined communicator, e.g `COMM_mymodel`. One has to declare this variable in a module and initialize it as `COMM_mymodel=MPI_COMM_WORLD`. This change must not influence the execution of the model. It can be useful to do a test run to check for this.
    66 * If the model uses a different communicator, one should take note of its name (below we refer to it as `COMM_mymodel`). We will then overwrite it to be able to represent the ensemble of model tasks. Please check if COMM_mymodel is identical to MPI_COMM_WORLD, thus including all processes of the program. If not, this is a [#SpecialCase special case, which we discuss further below].
     67* If the model uses `MPI_COMM_WORLD` we have to replace this by a user-defined communicator, e.g `COMM_mymodel`. One has to declare this as an integer variable in a module and initialize it as `COMM_mymodel=MPI_COMM_WORLD`. Then all occurences of `MPI_COMM_WORLD` in the code are replaced by `COMM_mymodel`. This change must not influence the execution of the model, as on e can check with a test run.
     68* If the model uses a different communicator, one should take note of its name (below we refer to it as `COMM_mymodel`). PDAF will reconfigure it to be able to represent the ensemble of model tasks.
    6769
    68 Now, `COMM_mymodel` will be replaced by the communicator that represents the ensemble of model tasks. For this we
    69 * Adapt `init_parallel_pdaf` so that at the end of this routine we set `COMM_mymodel=COMM_model`. Potentially, also set the rank and size variables of the model, respectively, by `mype_model` and `npes_model`.
     70Now, we add the call to `init_parallel_pdaf` which reconfigures `COMM_mymodel` and the related `rank` and `size` variables. These variables are arguments to the template variant of `init_parallel_pdaf` (i.e `template/online/init_parallel_pdaf.F90)
    7071
    71 Finally, we have a ensure that the number of model tasks is correctly set. In the template and tutorial codes, the number of model tasks is specified by the variable `n_modeltasks`. This has to be set before the operations on communicators are done in `init_parallel_pdaf`. In the tutorial code we added a command-line parsing to set the variable (it is parsing for `dim_ens`). One could also read the value from a configuration file.
     72Finally, we have a ensure that the number of model tasks is correctly set. In the template and tutorial codes, the number of model tasks is specified by the variable `n_modeltasks`. This has to be set before the operations on communicators are done in `init_parallel_pdaf`. In the tutorial code we added a command-line parsing to set the variable (parsed with the keyword `dim_ens`, thus specifying `-dim_ens N_MODELTASKS` on the command line, where N_MODELTASKS should be the ensemble size). One could also read the value from a configuration file.
    7273
    7374If the program is executed with these extensions using multiple model tasks, the issues discussed in '[#Compilingtheextendedprogram Compiling the extended program]' can occur. This one has to take care about which processes will perform output to the screen or to files.
    … …  
    7879=== The adaptions in short ===
    7980
    80 If you are experienced with MPI, the steps are the following using the files `mod_parallel_pdaf.F90`, `init_parallel_pdaf.F90`, `finalize_pdaf.F90` and `parser_mpi.F90` from `tutorial/online_2D_serialmodel`.
    81 1. Insert `CALL init_parallel_pdaf(0, 1)` at the beginning of the main program. This routine will also call `MPI_Init`.
     81If you are experienced with MPI, the steps are the following using the files `mod_parallel_pdaf.F90`, `init_parallel_pdaf.F90.serialmodel`, `finalize_pdaf.F90` and `parser_mpi.F90` from `tutorial/online_2D_serialmodel`.
     821. rename the file  `init_parallel_pdaf.F90.serialmodel` to `init_parallel_pdaf.F90` when copying it.
     831. Insert `CALL init_parallel_pdaf(1)` at the beginning of the main program. This routine performs the initialization of MPI and the configuration of communicators for the data assimilation
    82841. Insert `CALL finalize_pdaf()` at the end of the main program. This routine will also finalize MPI.
    83 1. The number of model tasks in variable `n_modeltasks` is determined in `init_parallel_pdaf` by command line parsing (it is parsing for `dim_ens`, thus setting `-dim_ens N_MODELTASKS`, where N_MODELTASKS should be the ensemble size). One could also read the value from a configuration file.
     851. The number of model tasks in variable `n_modeltasks` is determined in `init_parallel_pdaf` by command line parsing (it is parsing for `dim_ens`, thus setting `-dim_ens N_MODELTASKS` on the command line, where N_MODELTASKS should be the ensemble size). One could also read the value from a configuration file.
    8486
    8587
    8688=== Details on adapting a serial model ===
    8789
    88 If the numerical model is not parallelized (i.e. serial), we need to add the parallelization for the ensemble. We follow here the approach used in the tutorial code `tutorial/online_2D_serialmodel`. The files `mod_parallel_pdaf.F90`, `init_parallel_pdaf.F90`, `finalize_pdaf.F90` (and `parser_mpi.F90`) can be directly used. Note, the `init_parallel_pdaf` uses command-line parsing (from `parser_mpi.F90`) to read the number of model tasks (`n_modeltasks`) from the command line (specifying `-dim_ens N_MODELTASKS`, where N_MODELTASKS should be the ensemble size). One can replace this, e.g., by reading from a configuration file.
     90If the numerical model is not parallelized (i.e. serial), we need to add the parallelization for the ensemble. We follow here the approach used in the tutorial code `tutorial/online_2D_serialmodel`. The files `mod_parallel_pdaf.F90`, `init_parallel_pdaf.F90`, `finalize_pdaf.F90` can be directly used. Note, that `init_parallel_pdaf` uses command-line parsing to read the number of model tasks (`n_modeltasks`) from the command line (specifying `-dim_ens N_MODELTASKS`, where N_MODELTASKS should be the ensemble size). One can replace this, e.g., by reading from a configuration file.
    8991
    9092In the tutorial code, the parallelization is simply initialized by adding the line
    9193{{{
    92   CALL init_parallel_pdaf(0, 1)
     94  CALL init_parallel_pdaf(1)
    9395}}}
    94 into the source code of the main program. This is done at the very beginning of the functional code part.
    95 The initialization of MPI itself (by a call to `MPI_Init`) is included in `init_parallel_pdaf`.
     96into the source code of the main program. This is done at the very beginning of the functional code part. This routine will call `PDAF3_init_parallel` to intialize and configure the parallelization.
    9697
    97 The finalization of MPI is included in `finalize_pdaf.F90` by a call to `finalize_parallel()`. The line
     98The finalization of MPI is included in `finalize_pdaf.F90` by a call to `PDAF_finalize`. The line
    9899{{{
    99100  CALL finalize_pdaf()
    … …  
    108109== Further Information ==
    109110
    110 === Communicators created by `init_parallel_pdaf` ===
     111=== Communicators created for the data assimilation ===
    111112
    112113MPI uses so-called 'communicators' to define groups of parallel processes. These groups can then conveniently exchange information. In order to provide the 2-level parallelism for PDAF, three communicators are initialized that define the processes that are involved in different tasks of the data assimilation system.
    113 The required communicators are initialized in the routine `init_parallel_pdaf`. They are called
     114The required communicators are initialized in the call to `PDAF3_init_parallel` in the routine `init_parallel_pdaf`. They are called
    114115 * `COMM_model` - defines the groups of processes that are involved in the model integrations (one group for each model task)
    115116 * `COMM_filter` - defines the group of processes that perform the filter analysis step
    116  * `COMM_couple` - defines the groups of processes that are involved when data are transferred between the model and the filter. This is used to distribute and collect ensemble states.
     117 * `COMM_couple` - defines the groups of processes that are involved when data are transferred between the model and the filter. This is used to distribute and collect ensemble states (it is only used inside PDAF and not provided to the user code)
    117118
    118119[[Image(//pics/communicators_PDAFonline.png)]]
    119120[[BR]]'''Figure 1:''' Example of a typical configuration of the communicators using a parallelized model. In this example we have 12 processes over all, which are distributed over 3 model tasks (COMM_model) so that 3 model states can be integrated at the same time. COMM_couple combines each set of 3 communicators of the different model tasks. The filter is executed using COMM_filter which uses the same processes of the first model tasks, i.e. COMM_model 1 (Figure credits: A. Corbin)
    120121
    121 An important aspect is that the template version of `init_parallel_pdaf` itself uses `MPI_COMM_WORLD` and splits this to create the three communicators.
    122 
    123122
    124123=== Arguments of `init_parallel_pdaf` ===
    125124
    126 The routine `init_parallel_pdaf` has two arguments, which are the following:
     125There are two variants of `init_parallel_pdaf`, one for a parallelized moden and one for a serial (or pure OpenMP) model.
     126
     127For a **parallelized model**, the routine has the arguments
    127128{{{
    128 SUBROUTINE init_parallel_pdaf(dim_ens, screen)
     129  SUBROUTINE init_parallel_pdaf(screen, COMM_mymodel, rank_mymodel, size_mymodel)
    129130}}}
    130  * `dim_ens`: An integer defining the ensemble size. This allows to check the consistency of the ensemble size with the number of processes of the program. If the ensemble size is specified after the call to `init_parallel_pdaf` (as in the example) it is recommended to set this argument to 0. In this case no consistency check is performed.
     131with
    131132 * `screen`: An integer defining whether information output is written to the screen (i.e. standard output). The following choices are available:
    132133  * 0: quite mode - no information is displayed.
    133134  * 1: Display standard information about the configuration of the processes (recommended)
    134135  * 2: Display detailed information for debugging
     136 * `COMM_mymodel`: The model communicator
     137 * `rank_mymodel`: In integer giving the `rank` of the process in `COMM_mymodel` (usually initilized by a call to `MPI_Comm_rank)
     138 * `size_mymodel`: In integer giving the number of processes in `COMM_mymodel` (usually initilized by a call to `MPI_Comm_size)
    135139
     140
     141For a **serial model**, the routine has one argument
     142{{{
     143  SUBROUTINE init_parallel_pdaf(screen)
     144}}}
     145with
     146 * `screen`: An integer defining whether information output is written to the screen (i.e. standard output). The following choices are available:
     147  * 0: quite mode - no information is displayed.
     148  * 1: Display standard information about the configuration of the processes (recommended)
     149  * 2: Display detailed information for debugging
     150 
     151The arguments of `PDAF3_init_parallel` are described on the [wiki:PDAF3_init_parallel documentation page on PDAF3_init_parallel].
    136152
    137153=== Compiling and testing the extended program ===
    138154
    139 To compile the model with the adaption for modified parallelization, one needs to ensure that the additional files (`init_parallel_pdaf.F90`, `mod_parallel_pdaf.F90` and potentially `parser_mpi.F90`) are included in the compilation.
     155To compile the model with the adaption for modified parallelization, one needs to ensure that the additional files (`init_parallel_pdaf.F90`, `mod_parallel_pdaf.F90`, `finalize_pdaf.F90`) are included in the compilation.
    140156
    141157If a serial model is used, one needs to adapt the compilation to compile for MPI-parallelization. Usually, this is done with a compiler wrapper like 'mpif90', which should be part of the OpenMP installation. If you compiled a PDAF tutorial case before, you can check the compile settings for the tutorial.
    142158
    143 One can test the extension by running the compiled model. It should run as without these changes, because `mod_parallel` defines by default that a single model task is executed (`n_modeltasks=1`). If `screen` is set to 1 in the call to init_parallel_pdaf, the standard output should include lines like
     159One can test the extension by running the compiled model (for the template in `templates/online` with `mpirun -np 4 ./PDAF_online -dim_ens 4`). With the default setting `screen=1`  in the call to `init_parallel_pdaf`, the standard output should include lines like
    144160{{{
    145  Initialize communicators for assimilation with PDAF
     161PDAF  *** Initialize MPI communicators for assimilation with PDAF ***
     162PDAF    Pconf  Process configuration:
     163PDAF    Pconf   world    assim       model           couple        assimPE
     164PDAF    Pconf   rank     rank    task     rank    task     rank      T/F
     165PDAF    Pconf  ------------------------------------------------------------
     166PDAF    Pconf      0        0       1        0       1        0       T
     167PDAF    Pconf      2                3        0       1        2       F
     168PDAF    Pconf      3                4        0       1        3       F
     169PDAF    Pconf      1                2        0       1        1       F
     170}}}
     171In this example only a single process will compute the filter analysis. There are 4 processes (with ranks 0 to 3) for 'world'. We have 4 model tasks (i.e. `COMM_mymodel` exists now 4 times), each using a single process, and one 'couple' comunicator with ranks 0 to 3. Thus, `size_mycomm` will be 1, and `rank_mycomm` will always be 0. (If one runs the exampe with `mpirun -np 8 ./PDAF_online -dim_ens 4` one will get 8 lines instead of 4, `size_mycomm` will be 2 and `rank_mycomm` will show values 0 and 1.) Note that the couple communicator is only used internally by PDAF and is not accessible to the user code.
    146172
    147                   PE configuration:
    148    world   filter     model        couple     filterPE
    149    rank     rank   task   rank   task   rank    T/F
    150   ----------------------------------------------------------
    151      0       0      1      0      1      0       T
    152      1       1      1      1      2      0       T
    153      2       2      1      2      3      0       T
    154      3       3      1      3      4      0       T
     173Using multiple model tasks can result in the following effects:
     174 * The standard screen output of the model can by shown multiple times. For a serial model this is the case, since all processes can now write output. For a parallel model, this is due to the fact that often the process with `rank=0` performs screen output. By splitting the communicator `COMM_mymodel`, each model task will have a process with rank 0. Here one can adapt the code to use `mype_world==0` from `mod_parallel_pdaf`.
     175 * Each model task might write file output. This can lead to the case that several processes try to generate the same file or try to write into the same file. In the extreme case this can result in a program crash. For this reason, it might be useful to restrict the file output to a single model task. This can be implemented using the variable `task_id`, which is initialized by `init_parallel_pdaf` and holds the index of the model task ranging from 1 to `n_modeltasks`.
     176  * A convenient approach for the ensemble assimilation can be to run each model task in a separate directory. This needs an adapted setup. For example, for the tutorial in `tutorial/online_2D_parallelmodel` we can to the following:
     177   * in `tutorial/` create sub-directories `ens1`, `ens2`, `ens3`, `ens4`
     178   * copy `online_2D_parallelmodel/model_pdaf` into each of these directories
     179   * for ensemble size 4 and each model task using 2 processes, we can now run
     180{{{
     181mpirun -np 2 -wdir ens1 model_pdaf -dim_ens 4 : \
     182-np 2 -wdir ens2 model_pdaf -dim_ens 4 : \
     183-np 2 -wdir model_pdaf -dim_ens 4 : \
     184-np 2 -wdir model_pdaf -dim_ens 4
    155185}}}
    156 These lines show the configuration of the communicators. This example was executed using 4 processes and `n_modeltasks=1`, i.e. `mpirun -np 4 ./model_pdaf -dim_ens 4` in `tutorial/online_2D_parallelmodel`. (In this case, the variables `npes_filter` and `npes_model` will have a value of 4.)
     186   This approach has the advantage that each model writes files into a separate directory. This also allows to use the model's restart files, if it writes such.
     187  * One can also switch off the regular file output of the model completely. As each model task holds only a single member of the ensemble, this output might not be useful and might just slow down the program and lead to overly large use a disk space. In this case, the file output for the state estimate and perhaps all ensemble members should be done in the pre/poststep routine (`prepoststep_pdaf`) of the assimilation system. This approach allows to run the full data assimilation in a single directory, but it can be combined with the use of separate directories.
     188 
    157189
    158190
    159 To test parallel model tasks one has to set the variable `n_modeltasks` to a value larger than one. Now, the model will execute parallel model tasks. For `n_modeltasks=4` and running on a total of 4 processes the output from init_parallel_pdaf will look like the following:
    160 {{{
    161  Initialize communicators for assimilation with PDAF
    162 
    163                   PE configuration:
    164    world   filter     model        couple     filterPE
    165    rank     rank   task   rank   task   rank    T/F
    166   ----------------------------------------------------------
    167      0       0      1      0      1      0       T
    168      1              2      0      1      1       F
    169      2              3      0      1      2       F
    170      3              4      0      1      3       F
    171 
    172 }}}
    173 In this example only a single process will compute the filter analysis (`filterPE=.true.`). There are now 4 model tasks, each using a single process. Thus, both `npes_filter` and `npes_model` will be one.
    174 
    175 Using multiple model tasks can result in the following effects:
    176  * The standard screen output of the model can by shown multiple times. For a serial model this is the case, since all processes can now write output. For a parallel model, this is due to the fact that often the process with `rank=0` performs screen output. By splitting the communicator `COMM_model`, there will be as many processes with rank 0 as there are model tasks. Here one can adapt the code to use `mype_world==0` from `mod_parallel_pdaf`.
    177  * Each model task might write file output. This can lead to the case that several processes try to generate the same file or try to write into the same file. In the extreme case this can result in a program crash. For this reason, it might be useful to restrict the file output to a single model task. This can be implemented using the variable `task_id`, which is initialized by `init_parallel_pdaf` and holds the index of the model task ranging from 1 to `n_modeltasks`.
    178   * For the ensemble assimilation, it can be useful to switch off the regular file output of the model completely. As each model tasks holds only a single member of the ensemble, this output might not be useful. In this case, the file output for the state estimate and perhaps all ensemble members should be done in the pre/poststep routine (`prepoststep_pdaf`) of the assimilation system.
    179   * An alternative approach can be use run each model task in a separate directory. This needs an adapted setup. For example, for the tutorial in `tutorial/online_2D_parallelmodel`:
    180    * create sub-directories `x1`, `x2`, `x3`, `x4`
    181    * copy `model_pdaf` into each of these directories
    182    * for ensemble size 4 and each model task using 2 processes, we can now run
    183 {{{
    184 mpirun -np 2 ./x1/model_pdaf -dim_ens 4 : \
    185 -np 2 ./x2/model_pdaf -dim_ens 4 : \
    186 -np 2 ./x3/model_pdaf -dim_ens 4 : \
    187 -np 2 ./x4/model_pdaf -dim_ens 4
    188 }}}
    189    
    190 
    191 == Special Case ==
    192 
    193 === COMM_mymodel does not include all processes ===
     191=== If `COMM_mymodel` does not include all processes ===
    194192
    195193If the parallelized model uses a communicator that is named different from `MPI_COMM_WORLD` (we use `COMM_mymodel` here), there can be two cases:
    196  1. The easy case, described above is that `COMM_mymodel` includes all processes of the program. Often this is set as `COMM_mymodel = MPI_COMM_WORLD` and `COMM_mymodel` is introduced to have the option to define it differently.
     194 1. The more typical case, described above is that `COMM_mymodel` includes all processes of the program. Often this is set as `COMM_mymodel = MPI_COMM_WORLD` and `COMM_mymodel` is introduced to have the option to define it differently.
    197195 1. In some models `COMM_mymodel` is introduced to only represent a sub-set of the processes in `MPI_COMM_WORLD`. A use cases for this are coupled models, like atmosphere-ocean models. Here, the ocean might use a sub-set of processes and the program defines separate communicators for the ocean and atmosphere. There are also models that use a few of the processes to perform file operations with a so-called IO-server. In this case, the model runs only on nearly all processes and `COMM_mymodel` would be defined accordingly.
    198196
    199 For case 2, one needs to edit `init_parallel_pdaf.F90':
    200     1. Include the PDAF module with `USE PDAF`
    201     1. Replace `MPI_COMM_WORLD` in all cases by `COMM_mymodel`.
     197Case 2 is handled in PDAF V3.0 and later by PDAF because `PDAF_init_parallel` (or, for PDAF V3.0 the routine `PDAF_set_parallel`) is provided with the communicator `COMM_mymodel` and only reconfigures this communicator (thus, PDAF never uses `MPI_COMM_WORLD`.
    202198
    203 If `init_parallel_pdaf` includes a call to `PDAF3_set_parallel` from the PDAF3 interface, no further changes are required. However, when one uses the previous interface of PDAF 2.3.1 and before, one needs the following:
    204 
    205     1. At the end of `init_parallel_pdaf` set the main communicator for PDAF by
    206 {{{
    207   CALL PDAF_set_comm_pdaf(COMM_ensemble)
    208 }}}
     199Older implementations for PDAF 2.3.1 and before use a different approach. See the [wiki:AdaptParallelization_PDAF23 Page on adapting the parallelization for PDAF V2.x] for more information.