| 25 | | The general concept is depicted in figure 1. The left hand side shows a typical abstract structure of a model. When the program is executed, the typical steps don ein the program are the following: |
| 26 | | 1. The model is initialized. Thus arrays for the model fields are allocated and filled with initial fields. Thus, the model grid is build up virtually in the program. |
| 27 | | 2. After the initialization the time stepping loop is performed. Here the model fields are propagated through time. |
| 28 | | 3. When the integration of the model fields is completed after a defined number of time steps, various post-processing operations can be performed. This usually includes file output, e.g. writing restart files. Afterwards, the program stops. |
| | 25 | == ''Fully-parallel'' implementation == |
| | 26 | |
| | 27 | The assimilation system is built by adding subroutine calls to the general part of the model code. In these routines one can define variables for PDAF, use-include the PDAF module and call the PDAF core subroutines. Usually only single lines with a subroutine call are inserted into the model code. Because only minimal changes to the model code are required, we refer to this as "attaching" PDAF to the model. |
| | 28 | |
| | 29 | The general concept is depicted in figure 1. The left hand side shows a typical abstract structure of a model. When the program is executed, the typical steps done in the program are the following: |
| | 30 | 1. The model might intialize parallelization |
| | 31 | 1. The model is initialized. Thus, model fields arrays are allocated and filled with initial fields. Thus, the model grid is build up virtually in the program. |
| | 32 | 2. The time stepping loop is performed over ''isteps'' time steps. Here the model fields are propagated through time. |
| | 33 | 3. After time stepping, post-processing operations can be performed. This usually includes file output, e.g. writing restart files. Afterwards, the program stops. |
| 36 | | * **init_parallel_pdaf**: This subroutine is inserted close to the start of the model code. If the model itself is parallelized the correct location is directly after the initialization of the parallelization in the model code. `init_parallel_pdaf` creates the parallel environment that allows to compute the time stepping for several models ("model tasks") at the same time. |
| 37 | | * **init_pdaf**: This subroutine is added after the initialization part of the model, just before the time stepping loop. This this subroutine one defines parameters for PDAF and then one calls the core initialization routine `PDAF_init`. This core routine also initializes the array of ensemble states using a user-provided call-back routine. Subsequently, the PDAF-core routine `PDAF_init_forecast` is called (in implementations of PDAF before version 3.0, this routine was called `PDAF_get_state`). This routine initializes model fields from the array of ensemble states using a call-back routine. In addition it returns the the number of time steps that have to be computed in the following forecast phase. |
| 38 | | * **assimilate_pdaf**: This routine is added to the model code at the end of the time stepping loop (usually just before the ''END DO'' in a Fortran program). The routine declares the names of user-supplied subroutines and calls a PDAF-core routine `PDAF3_assimilate`. (In implementations of PDAF before version 3.0, different routine named `PDAFomi_assimilate_X` with, e.g., X=`local`, for local filters are used). This routine has to be called at the end of each time step. It counts whether all time steps of the current forecast phase have been computed. If this is not the case, the program continues integrating the model. If the forecast phase is completed, the analysis step (i.e. the actual assimilation of the observations) is computed. Subsequently, the next forecast phase is initialized by writing the analysis state vector into the model fields and setting the number of time steps in the next forecast phase. |
| 39 | | * **finalize_pdaf**: This routine is optional. It is used to let PDAF display timing and memory information and to call PDAF to deallocate its internal arrays. |
| | 41 | * **init_parallel_PDAF**: This subroutine is inserted close to the start of the model code. If the model itself is parallelized the correct location is directly after the initialization of the parallelization in the model code. `init_parallel_pdaf` creates the parallel environment that allows to compute the time stepping for an ensemble of models ("model tasks") at the same time. |
| | 42 | * **init_PDAF**: This subroutine is added after the initialization part of the model, just before the time stepping loop. In this subroutine one defines parameters for PDAF and then one calls the core initialization routine `PDAF_init`. This core routine initializes internal parameters and afterwards the array of ensemble states using a user-provided call-back routine. Subsequently, one calls the PDAF core routine `PDAF_init_forecast` (in implementations of PDAF before version 3.0, this routine was called `PDAF_get_state`). This routine initializes model fields from the array of ensemble states using a call-back routine. In addition, it returns the number of time steps that have to be computed in the following forecast phase. |
| | 43 | * **assimilate_PDAF**: This routine is added to the model code before the end of the time stepping loop (usually just before the ''END DO'' in a Fortran program). The routine declares the names of user-supplied subroutines and calls a PDAF-core routine `PDAF3_assimilate`. (In implementations of PDAF before version 3.0, different routines named `PDAFomi_assimilate_X` with, e.g., X=`local`, for local filters are used). This routine has to be called at the end of each time step. It counts whether all time steps of the current forecast phase have been computed. If this is not the case, the program continues integrating the model. If the forecast phase is completed, the analysis step, i.e. the actual assimilation of the observations, is computed. Subsequently, the next forecast phase is initialized by writing the analysis state vector into the model fields and setting the number of time steps in the next forecast phase. |
| | 44 | * **finalize_PDAF**: This routine is used to let PDAF display timing and memory information and to call PDAF to deallocate its internal arrays. |
| 47 | | * The implementation concept of PDAF attaches the data assimilation functionality to the model. With this approach, the |
| 48 | | * The implementation with PDAF does not require that the time stepping part of the model is implemented as a subroutine. Instead calls to subroutines that control of the ensemble integration are added to the model code before and after the code parts performing the time stepping. This minimizes the changes in the model code. |
| | 52 | * The implementation concept of PDAF attaches the data assimilation functionality to the model. With this approach, the model with attached data assimilation is finally run analogous to a normal model run, but with additional processors and additional options for the data assimilation. |
| | 53 | * The implementation with PDAF does not require that the time stepping part of the model is implemented as a subroutine. Instead calls to subroutines that control the the ensemble integration are added to the model code. This minimizes the changes in the model code. |
| 61 | | PDAF adds the possibility to perform parallel ensemble forecasts, even for models that by themselves do not use parallelization. The structure of the parallelized data assimilation program is displayed in Figure 3. In the forecast phase of the data assimilation application, several model state integrations can be performed at the same time by several model tasks. If the numerical model it parallelized by itself, the parallel ensemble forecast adds a second level of parallelization. For the analysis step, in which the DA method combines the ensemble of model states with the observations, PDAF provides several parallelized methods. If the model uses domain decomposition for the parallelization, the same decomposition is typically used in the filter. Before the analysis step, all ensemble members are gathered by the processes that compute the filter analysis. Subsequently to the analysis step, the ensemble members are distributed to all model tasks to enable the next parallel ensemble forecast. These operations are performed within PDAF, so that a user can directly benefit from the second level of parallelization. For the required extension of the parallelization configuration, a fully implemented template routine is provided with PDAF. The adaptation of the parallelization is described in the [ImplementationGuide Implementation Guide]. |
| | 66 | PDAF adds the possibility to perform parallel ensemble forecasts, even for models that by themselves do not use parallelization. The structure of the parallelized data assimilation process is displayed in Figure 3. In the forecast phase of the data assimilation application, several model tasks can perform model state integrations at the same time. If the numerical model it parallelized by itself, the parallel ensemble forecast adds a second level of parallelization. For the analysis step, in which the DA method combines the ensemble of model states with the observations, PDAF provides several parallelized DA methods. If the model uses domain decomposition for the parallelization, the same decomposition is typically used in the DA method. Before the analysis step, all ensemble members are gathered by the processes that compute the filter analysis. Subsequently to the analysis step, the ensemble members are distributed to all model tasks to enable the next parallel ensemble forecast. These operations are performed within PDAF, so that a user can directly benefit from the second level of parallelization. For the required extension of the parallelization configuration, a fully implemented template routine is provided with PDAF. The adaptation of the parallelization is described in the [ImplementationGuide_PDAF3 Implementation Guide]. |
| 83 | | * **init_parallel_pdaf**: This routine is inserted close to the start of the model code in the same way as for the `fully parallel` mode. |
| 84 | | * **init_pdaf**: This routine is added after the initialization part of the model in the same way as for the `fully parallel` mode. The routine also determines the number of time steps for the initial forecast phase. |
| 85 | | * **Ensemble loop**: In order to allow for the integration of the state ensemble, an unconditional loop is added around the time stepping loop of the model. This loop will allow to compute the time stepping loop multiple times to integrate all ensemble states. PDAF provides an exit flag for this loop, which is checked to control when to exit this loop. The number of time steps in the forecast (`nsteps`) is initially provided by `init_pdaf` and subsequently by `PDAF_get_fcst_info`. |
| 86 | | * **assimilate_pdaf**: Inside the ensemble loop, a call to this interface routine is added to the code in the same way as for the `fully parallel` mode. |
| 87 | | * ** PDAF_get_fcst_info**: The analysis step is computed when the request number of time steps `steps` are completed. At this point the program steps out of the model time stepping loop. Here, a call to `PDAF_get_fcst_info` is used to obtain the number of time steps for the next forecast phase and the value of the exit flag. Subsequently, one checks the exit flag and when this indcates to end the assimilation process one steps out of the outer unconditional loop. Otherwise, the program continues with the next forecast phase. |
| 88 | | * **finalize_pdaf**: This routine is optional. It is used to let PDAF display timing and memory information and to call PDAF to deallocate its internal arrays. |
| | 88 | * **init_parallel_pdaf**: This routine is inserted close to the start of the model code in the same way as for the ''fully parallel'' mode. |
| | 89 | * **init_pdaf**: This routine is added after the initialization part of the model in the same way as for the ''fully parallel'' mode. The routine also determines the number of time steps for the initial forecast phase. |
| | 90 | * **Ensemble loop**: In order to allow for the integration of the state ensemble, an unconditional loop is added around the time stepping loop of the model. This loop will allow to compute the time stepping loop multiple times to integrate all ensemble states. `PDAF_get_fcst_info` provides an exit flag for this loop, which is checked to control when to exit this loop. |
| | 91 | * **PDAF_get_fcst_info**: The call to `PDAF_get_fcst_info` is used to obtain the number of time steps (`nsteps`) for the next forecast phase and the value of the exit flag. Subsequently, one checks the exit flag and when this indicates to end the assimilation process, one steps out of the outer unconditional loop. Otherwise, the program continues with the next forecast phase. |
| | 92 | * **assimilate_pdaf**: Inside the ensemble loop, a call to this interface routine is added to the code in the same way as for the ''fully parallel'' mode. |
| | 93 | * **finalize_pdaf**: This routine is added to the code in the same way as for the ''fully parallel'' mode. |
