Changes between Version 5 and Version 6 of OfflineAdaptParallelization


Ignore:
Timestamp:
May 18, 2025, 4:13:20 PM (5 hours ago)
Author:
lnerger
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • OfflineAdaptParallelization

    v5 v6  
    3030== Three communicators ==
    3131
    32 MPI uses so-called 'communicators' to define sets of parallel processes. As the ensemble integrations are performed externally to the assimilation program in the case of the offline mode, the communicators of the PDAF parallelization are only used internally to PDAF. However, as the communicators need to be initialized, we still describe them here.
     32MPI uses so-called 'communicators' to define sets of parallel processes. In the offline mode, the communicators of the PDAF parallelization are only used internally to PDAF. However, as the communicators need to be initialized and provided to `PDAF_init` to initialized PDAF, we describe them here.
    3333
    34 In order to provide the 2-level parallelism, three communicators need to be initialized that define the processes that are involved in different tasks of the data assimilation system.
    35 The required communicators are initialized in the routine `init_parallel_pdaf` and called
     34The initialization routine `PDAF_init` is used for both the offline and online modes. Due to this, three communicators need to be initialized that define the groups of processes that are involved in different tasks of the data assimilation system.
     35The required communicators are initialized in the routine `init_parallel_pdaf` and are named
    3636 * `COMM_model` - defines the processes that are involved in the model integrations
    3737 * `COMM_filter` - defines the processes that perform the filter analysis step
    3838 * `COMM_couple` - defines the processes that are involved when data are transferred between the model and the filter
     39For the offline mode, only `COMM_filter` is relevant. `COMM_model` is identical to `COMM_filter` and `COMM_couple` is not actively used.
    3940
    40 The parallel region of an MPI parallel program is initialized by calling `MPI_init`.  By calling `MPI_init`, the communicator `MPI_COMM_WORLD` is initialized. This communicator is pre-defined by MPI to contain all processes of the MPI-parallel program. In the offline mode, it would be sufficient to conduct all parallel communication using only `MPI_COMM_WORLD`. However, as PDAF uses the three communicators listed above internally, they have to be initialized. In general they will be identical to `MPI_COMM_WORLD` with the exception of `COMM_couple`, which is not used in the offline mode.
     41The parallel region of an MPI parallel program is initialized by calling `MPI_init`.  By calling `MPI_init`, the communicator `MPI_COMM_WORLD` is initialized. This communicator is pre-defined by MPI to contain all processes of the MPI-parallel program. In the offline mode, it would be sufficient to conduct all parallel communication using only `MPI_COMM_WORLD`. However, as PDAF uses internally the three communicators listed above, they have to be initialized. In general they will be identical to `MPI_COMM_WORLD` with the exception of `COMM_couple`, which is not used in the offline mode.
    4142
    4243[[Image(//pics/communicators_PDAFoffline.png)]]
    43 [[BR]]'''Figure 1:''' Example of a typical configuration of the communicators for the offline coupled assimilation with parallelization. In this example we have 4 processes. The communicators COMM_model, COMM_filter, and COMM_couple need to be initialized. However, COMM_model and COMM_filter use all processes (as MPI_COMM_WORLD) and COMM_couple is a group of communicators each holding only one process. Thus the initialization of the communicators is simpler than in case of the [wiki:AdaptParallelization parallelization for the online coupling].
     44[[BR]]'''Figure 1:''' Example of a typical configuration of the communicators for the offline coupled assimilation with parallelization. We have 4 processes in this example. The communicators COMM_model, COMM_filter, and COMM_couple need to be initialized. COMM_model and COMM_filter use all processes (as MPI_COMM_WORLD) while COMM_couple is a group of communicators each holding only one process. Thus, the initialization of the communicators is simpler than in case of the [wiki:AdaptParallelization parallelization for the online coupling].
    4445
    4546
    46 == Initializing the communicators ==
     47== Initializing the parallelization ==
    4748
    48 The routine `init_parallel_init`, which is supplied with the PDAF package, initializes the necessary communicators for the assimilation program and PDAF. The routine should be added to the model usually directly after the initialization of the parallelization described above.
    49 The routine `init_parallel_pdaf` also defines the communicators `COMM_filter` and `COMM_couple` that were described above. The provided routine `init_parallel_init` is a template implementation. For the offline mode, it should not be necessary to modify it!
     49The routine `init_parallel_pdaf`, which is supplied in `templates/offline` and `tutorial/offline_2D_parallel`, initializes the necessary communicators for the assimilation program and PDAF. The routine is called at the beginning of the assimilation program directly after the initialization of the parallelization described above. The provided routine `init_parallel_pdaf` is a template implementation. For the offline mode, it should not be necessary to modify it!
    5050
    51 For the offline mode, the variable `n_modeltasks` should always be set to one, because no integrations are performed in the assimilation program. The routine `init_parallel_pdaf` splits the communicator `MPI_COMM_WORLD` and defines the three communicators described above. In addition, the variables `npes_world` and `mype_world` are defined. These can be used in the user-supplied routines to control, for example, which process write information to the screen. The routine defines several more variables that are declared and held in the module `mod_parallel`. It can be useful to use this module with the model code as some of these variables are required when the initialization routine of PDAF (`PDAF_init`) is called.
     51`init_parallel_pdaf` is identical for the online and offline modes. The routine uses the variable `n_modeltasks`. For the offline mode, the provided code sets this variable to 1, because no integrations are performed in the assimilation program. The routine the variables `npes_world`, `mype_world` as well as `npes_filter` and `mype_filter` are defined. Here `npes_world` and `npes_filter` provide the number of processes involved in the program, while `mype_world` and `mype_filter` identify the index of a process (its 'rank'). These variables can be used in the user-supplied routines to control, for example, which process write information to the screen. In the example codes we use 'mype_filter' in the calll-back routines, while we use 'mype_world' in the main part of the code.
    5252
    53 The initialization of the communicators is different for the offline and online modes. You can find a template in the tutorial implementations for the offline case, e.g. in `tutorial/offline_2D_parallel/`.
     53`init_parallel_pdaf` defines several more variables that are declared and held in the module `mod_parallel_pdaf`. We recommend to use this module from the provided code as some of these variables are required when the initialization routine of PDAF (`PDAF_init`) is called.
     54
    5455
    5556== Arguments of `init_parallel_pdaf` ==
    … …  
    6061}}}
    6162 * `dim_ens`: An integer defining the ensemble size. This allows to check the consistency of the ensemble size with the number of processes of the program. For the offline mode, one should set this variable to 0. In this case no consistency check for the ensemble size with regard to parallelization is performed.
    62  * `screen`: An integer defining whether information output is written to the screen (i.e. standard output). The following choices are available:
     63 * `screen`: An integer defining whether information output is displayed. The following choices are available:
    6364  * 0: quite mode - no information is displayed.
    6465  * 1: Display standard information about the configuration of the processes (recommended)
    … …  
    6869
    6970
    70 == Compiling and testing the assimilation program ==
     71== Testing the assimilation program ==
    7172
    72 IF one want to check the initialization of the parallelization, one can compile the assimilation program with just the call to `init_parallel_pdaf` as well as calls to `MPI_init` and `MPI_finalize`. One can test the extension by running the compiled assimilation program.  If `screen` in the call to `init_parallel_pdaf` is set to 1 in the call to init_parallel_pdaf, the standard output should include lines like
     73One can compile the template code and run it, e.g., with `mpirun -np 4 PDAF_offline`. Close to the beginning of the output, one will see the lines
    7374{{{
    74  PDAF: Initializing communicators
     75 Initialize communicators for assimilation with PDAF
    7576
    7677                  PE configuration:
    … …  
    8384     3       3      1      3      4      0       T
    8485}}}
    85 These lines show the configuration of the communicators. This example was executed using 4 processes and `n_modeltasks=1`.
     86These lines show the configuration of the communicators. Here, 'world rank' is the value of 'mype_world', and 'filter rank' is the value of 'mype_filter'. These ranks always start with 0.
    8687