Changes between Version 38 and Version 39 of AdaptParallelization

Timestamp:

May 20, 2025, 7:29:23 PM (12 days ago)

Author:

lnerger

Comment:

--

AdaptParallelization

@@ -43 +43 @@
 2. Check whether the model uses `MPI_COMM_WORLD`. 
  * If yes, then replace `MPI_COMM_WORLD` in all places except MPI_abort and MPI_finalize by a user-defined communicator, e.g `COMM_mymodel`, which can be initialized as `COMM_mymodel=MPI_COMM_WORLD`.
- * If no, then take note of the name of the communicator variable (we assume it's `COMM_mymodel`)
+ * If no, then take note of the name of the communicator variable (we assume it's `COMM_mymodel`). The next steps are valid if the model uses COMM_mymodel, but setting it to MPI_COMM_WORLD. If the model is using less processes, this is a [#SpecialCase special case, which we discuss further below].
 3. Insert the call `init_parallel_pdaf` directly after `MPI_init`. 
 4. Adapt `init_parallel_pdaf` so that at the end of this routine you set `COMM_mymodel=COMM_model`. Potentially, also set the rank and size variables of the model, respectively, by `mype_model` and `npes_model`.
@@ -62 +62 @@
 For this one has to check if `MPI_COMM_WORLD`, e.g. checking the calls to `MPI_Comm_rank` and `MPI_Comm_size`, or MPI communication calls in the code (e.g. MPI_Send, MPI_Recv, MPI_Barrier).
 * If the model uses `MPI_COMM_WORLD` we have to replace this by a user-defined communicator, e.g `COMM_mymodel`. One has to declare this variable in a module and initialize it as `COMM_mymodel=MPI_COMM_WORLD`. This change must not influence the execution of the model. It can be useful to do a test run to check for this.
-* If the model uses a different communicator, one should take note of its name (below we refer to it as `COMM_mymodel`). We will then overwrite it to be able to represent the ensemble of model tasks.
-
-Now, `COMM_mymodel` will be replaced by the communicator that represent the ensemble of model tasks. For this we
+* If the model uses a different communicator, one should take note of its name (below we refer to it as `COMM_mymodel`). We will then overwrite it to be able to represent the ensemble of model tasks. Please check if COMM_mymodel is identical to MPI_COMM_WORLD, thus including all processes of the program. If not, this is a [#SpecialCase special case, which we discuss further below].
+
+Now, `COMM_mymodel` will be replaced by the communicator that represents the ensemble of model tasks. For this we
 * Adapt `init_parallel_pdaf` so that at the end of this routine we set `COMM_mymodel=COMM_model`. Potentially, also set the rank and size variables of the model, respectively, by `mype_model` and `npes_model`.
 
@@ -117 +117 @@
 [[BR]]'''Figure 1:''' Example of a typical configuration of the communicators using a parallelized model. In this example we have 12 processes over all, which are distributed over 3 model tasks (COMM_model) so that 3 model states can be integrated at the same time. COMM_couple combines each set of 3 communicators of the different model tasks. The filter is executed using COMM_filter which uses the same processes of the first model tasks, i.e. COMM_model 1 (Figure credits: A. Corbin)
 
+An important aspect is that the template version of `init_parallel_pdaf` itself uses `MPI_COMM_WORLD` and splits this to create the three communicators. 
 
 
@@ -185 +186 @@
 }}}
    
+
+== Special Case ==
+
+=== COMM_mymodel does not include all processes ===
+
+If the parallelized model uses a communicator that is named different from `MPI_COMM_WORLD` (we use `COMM_mymodel` here), there can be two cases: 
+ 1. The easy case, described above is that `COMM_mymodel` includes all processes of the program. Often this is set as `COMM_mymodel = MPI_COMM_WORLD` and `COMM_mymodel` is introduced to have the option to define it differently.
+ 1. In some models `COMM_mymodel` is introduced to only represent a sub-set of the processes in `MPI_COMM_WORLD`. A use cases for this are coupled models, like atmosphere-ocean models. Here, the ocean might use a sub-set of processes and the program defines separate communicators for the ocean and atmosphere. There are also models that use a few of the processes to perform file operations with a so-called IO-server. In this case, the model runs only on nearly all processes and `COMM_mymodel` would be defined accordingly.
+
+For case 2, one needs to edit `init_parallel_pdaf.F90': 
+    1. Include the PDAF module with `USE PDAF`
+    1. Replace `MPI_COMM_WORLD` in all cases by `COMM_mymodel`. 
+    1. At the end of `init_parallel_pdaf` set the main communicator for PDAF by 
+{{{
+  CALL PDAF_set_comm_pdaf(COMM_ensemble)
+}}}