| | 1 | = Offline Mode: Initializing the parallelization for PDAF = |
| | 2 | |
| | 3 | {{{ |
| | 4 | #!html |
| | 5 | <div class="wiki-toc"> |
| | 6 | <h4>Offline Mode: Implementation Guide for PDAF2</h4> |
| | 7 | <ol><li><a href="OfflineImplementationGuide_PDAF23">Main page</a></li> |
| | 8 | <li>Initializing the parallelization</li> |
| | 9 | <li><a href="OfflineInitPdaf_PDAF23">Initialization of PDAF</a></li> |
| | 10 | <li><a href="OfflineImplementationofAnalysisStep_PDAF23">Implementation of the analysis step</a></li> |
| | 11 | <li><a href="OfflineAddingMemoryandTimingInformation_PDAF23">Memory and timing information</a></li> |
| | 12 | </ol> |
| | 13 | </div> |
| | 14 | }}} |
| | 15 | |
| | 16 | [[PageOutline(2-3,Contents of this page)]] |
| | 17 | |
| | 18 | == Overview == |
| | 19 | |
| | 20 | The PDAF release provides template code for the offline mode in `templates/offline`. We refer to this code to use it as a basis. |
| | 21 | |
| | 22 | If the template code in `templates/offline` or the tutorial implementation `tutorial/offline_2D_parallel` is used as the base for the implementation, it should be possible to simply keep this part of the program, i.e. the call to `init_parallel_pdaf` unchanged and directly proceed to the [OfflineInitPdaf initialization of PDAF] for the offline mode. |
| | 23 | |
| | 24 | Like many numerical models, PDAF uses the MPI standard for the parallelization. PDAF requires for the compilation that an MPI library is available. In any case, it is necessary to execute the routine `init_parallel_pdaf` as described below. |
| | 25 | |
| | 26 | For completeness, we describe here the communicators required for PDAF as well as the template initialization routine. |
| | 27 | |
| | 28 | |
| | 29 | |
| | 30 | == Three communicators == |
| | 31 | |
| | 32 | MPI uses so-called 'communicators' to define sets of parallel processes. In the offline mode, the communicators of the PDAF parallelization are only used internally to PDAF. However, as the communicators need to be initialized and provided to `PDAF_init` to initialized PDAF, we describe them here. |
| | 33 | |
| | 34 | The initialization routine `PDAF_init` is used for both the offline and online modes. Due to this, three communicators need to be initialized that define the groups of processes that are involved in different tasks of the data assimilation system. |
| | 35 | The required communicators are initialized in the routine `init_parallel_pdaf` and are named |
| | 36 | * `COMM_model` - defines the processes that are involved in the model integrations |
| | 37 | * `COMM_filter` - defines the processes that perform the filter analysis step |
| | 38 | * `COMM_couple` - defines the processes that are involved when data are transferred between the model and the filter |
| | 39 | For the offline mode, only `COMM_filter` is relevant. `COMM_model` is identical to `COMM_filter` and `COMM_couple` is not actively used. |
| | 40 | |
| | 41 | The parallel region of an MPI parallel program is initialized by calling `MPI_init`. By calling `MPI_init`, the communicator `MPI_COMM_WORLD` is initialized. This communicator is pre-defined by MPI to contain all processes of the MPI-parallel program. In the offline mode, it would be sufficient to conduct all parallel communication using only `MPI_COMM_WORLD`. However, as PDAF uses internally the three communicators listed above, they have to be initialized. In general they will be identical to `MPI_COMM_WORLD` with the exception of `COMM_couple`, which is not used in the offline mode. |
| | 42 | |
| | 43 | [[Image(//pics/communicators_PDAFoffline.png)]] |
| | 44 | [[BR]]'''Figure 1:''' Example of a typical configuration of the communicators for the offline coupled assimilation with parallelization. We have 4 processes in this example. The communicators COMM_model, COMM_filter, and COMM_couple need to be initialized. COMM_model and COMM_filter use all processes (as MPI_COMM_WORLD) while COMM_couple is a group of communicators each holding only one process. |
| | 45 | |
| | 46 | |
| | 47 | == Initializing the parallelization == |
| | 48 | |
| | 49 | The routine `init_parallel_pdaf`, which is supplied in `templates/offline` and `tutorial/offline_2D_parallel`, initializes the necessary communicators for the assimilation program and PDAF. The routine is called at the beginning of the assimilation program directly after the initialization of the parallelization described above. The provided routine `init_parallel_pdaf` is a template implementation. For the offline mode, it should not be necessary to modify it! |
| | 50 | |
| | 51 | `init_parallel_pdaf` is identical for the online and offline modes. The routine uses the variable `n_modeltasks`. For the offline mode, the provided code sets this variable to 1, because no integrations are performed in the assimilation program. The routine the variables `npes_world`, `mype_world` as well as `npes_filter` and `mype_filter` are defined. Here `npes_world` and `npes_filter` provide the number of processes involved in the program, while `mype_world` and `mype_filter` identify the index of a process (its 'rank'). These variables can be used in the user-supplied routines to control, for example, which process write information to the screen. In the example codes we use 'mype_filter' in the calll-back routines, while we use 'mype_world' in the main part of the code. |
| | 52 | |
| | 53 | `init_parallel_pdaf` defines several more variables that are declared and held in the module `mod_parallel_pdaf`. We recommend to use this module from the provided code as some of these variables are required when the initialization routine of PDAF (`PDAF_init`) is called. |
| | 54 | |
| | 55 | |
| | 56 | == Arguments of `init_parallel_pdaf` == |
| | 57 | |
| | 58 | The routine `init_parallel_pdaf` has two arguments, which are the following: |
| | 59 | {{{ |
| | 60 | SUBROUTINE init_parallel_pdaf(dim_ens, screen) |
| | 61 | }}} |
| | 62 | * `dim_ens`: An integer defining the ensemble size. This allows to check the consistency of the ensemble size with the number of processes of the program. For the offline mode, one should set this variable to 0. In this case no consistency check for the ensemble size with regard to parallelization is performed. |
| | 63 | * `screen`: An integer defining whether information output is displayed. The following choices are available: |
| | 64 | * 0: quite mode - no information is displayed. |
| | 65 | * 1: Display standard information about the configuration of the processes (recommended) |
| | 66 | * 2: Display detailed information for debugging |
| | 67 | |
| | 68 | |
| | 69 | |
| | 70 | |
| | 71 | == Testing the assimilation program == |
| | 72 | |
| | 73 | One can compile the template code and run it, e.g., with `mpirun -np 4 PDAF_offline`. Close to the beginning of the output, one will see the lines |
| | 74 | {{{ |
| | 75 | Initialize communicators for assimilation with PDAF |
| | 76 | |
| | 77 | PE configuration: |
| | 78 | world filter model couple filterPE |
| | 79 | rank rank task rank task rank T/F |
| | 80 | ---------------------------------------------------------- |
| | 81 | 0 0 1 0 1 0 T |
| | 82 | 1 1 1 1 2 0 T |
| | 83 | 2 2 1 2 3 0 T |
| | 84 | 3 3 1 3 4 0 T |
| | 85 | }}} |
| | 86 | These lines show the configuration of the communicators. Here, 'world rank' is the value of 'mype_world', and 'filter rank' is the value of 'mype_filter'. These ranks always start with 0. |
| | 87 | |
