Changes between Version 14 and Version 15 of ImplementationConceptOnline


Ignore:
Timestamp:
May 17, 2025, 1:11:39 PM (19 hours ago)
Author:
lnerger
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ImplementationConceptOnline

    v14 v15  
    3636 * **init_parallel_pdaf**: This subroutine is inserted close to the start of the model code. If the model itself is parallelized the correct location is directly after the initialization of the parallelization in the model code. `init_parallel_pdaf` creates the parallel environment that allows to compute the time stepping for several models ("model tasks") at the same time.
    3737 * **init_pdaf**: This subroutine is added after the initialization part of the model, just before the time stepping loop. This this subroutine one defines parameters for PDAF  and then one calls the core initialization routine `PDAF_init`. This core routine also initializes the array of ensemble states using a user-provided call-back routine. Subsequently, the PDAF-core routine `PDAF_init_forecast` is called (in implementations of PDAF before version 3.0, this routine was called `PDAF_get_state`). This routine initializes model fields from the array of ensemble states using a call-back routine. In addition it returns the the number of time steps that have to be computed in the following forecast phase.
    38  * **assimilate_pdaf**: This routine is added to the model code at the end of the time stepping loop (usually just before the ''END DO'' in a Fortran program). The routine declares the names of user-supplied subroutines and calls a PDAF-core routine `PDAF3_assimilate`. (in implementations of PDAF before version 3.0, this routine was called `PDAFomi_get_state_X` with, e.g., X=`local`, for local filters). This routine has to be called at the end of each time step. It counts whether all time steps of the current forecast phase have been computed. If this is not the case, the program continues integrating the model. If the forecast phase is completed, the analysis step (i.e. the actual assimilation of the observations) is computed. Subsequently, the next forecast phase is initialized by writing the analysis state vector into the model fields and setting the number of time steps in the next forecast phase.
     38 * **assimilate_pdaf**: This routine is added to the model code at the end of the time stepping loop (usually just before the ''END DO'' in a Fortran program). The routine declares the names of user-supplied subroutines and calls a PDAF-core routine `PDAF3_assimilate`. (In implementations of PDAF before version 3.0, different routine named `PDAFomi_assimilate_X` with, e.g., X=`local`, for local filters are used). This routine has to be called at the end of each time step. It counts whether all time steps of the current forecast phase have been computed. If this is not the case, the program continues integrating the model. If the forecast phase is completed, the analysis step (i.e. the actual assimilation of the observations) is computed. Subsequently, the next forecast phase is initialized by writing the analysis state vector into the model fields and setting the number of time steps in the next forecast phase.
    3939
    40 
    41 With the implementation strategy of PDAF, calls to three to four routines have to be added to the model code. In case of the flexible implementation variant also the external loop has to be added. While this looks like a large change in figure 1, this change does actually only affect the general part of the model code. In addition, the amount of source code of the numerical model will be much longer than the addition for the data assimilation system. Please note that the calls to the routines `PDAF_get_state`, `PDAF_put_state_X`, `PDAF_assimilate_X` (X chosen according to the filter) could also be added directly in the model code. However, using the interface routines (`get_state_pdaf` and `put_state_pdaf`) reduces the additions to the model code.
     40With the implementation strategy of PDAF, calls to four routines are added to the model code. These are usually only single lines of code and the changes only affect the general part of the model code. 
    4241
    4342== Important aspects of the implementation concept ==
    4443
    45  * The implementation concept of PDAF does not require that the time stepping part of the model is implemented as a subroutine. Instead calls to subroutines that control of the ensemble integration are added to the model code before and after the code parts performing the time stepping. If the time stepping part is implemented as a subroutine, this subroutine can be called in between the additional routines.
    46  * Depending on the parallelization, there can be cases in which the model has to jump back in time and cases in which the time always moves forward:
    47   * Jumping back in time will be required if the number of model tasks used to evolve the ensemble states is smaller than the number of ensemble members. In this case a model task has integrate more than one model state and will have to jump back in time after the integration of each ensemble member. This case requires to implement the ''flexible'' implementation variant.
    48   * If there are as many model tasks as ensemble members, the model time always moves forward. In this case, one can implement PDAF with the ''fully-parallel'' variant without the external ensemble loop. That is, one calls `PDAF_assimilate_X` at the end of each time step. This strategy might also be simpler if a model uses nested loops (like a loop over minutes inside a loop over hours).
    49  * Model-specific operations like the initialization of the array of ensemble states in `PDAF_init` are actually performed by user-supplied routines. These routines are called through the standard interface of PDAF. Details on the interface and the required routines are given on the pages describing the implementation steps.
    50  * The assimilation system is controlled by the user-supplied routines that are called through PDAF as call-back routines.  With this strategy, the assimilation program is essentially driven by the model part of the program. Thus, logically the model is not a sub-component of the assimilation system, but the implementation with PDAF results in a model extended for data assimilation.
     44* The implementation concept of PDAF attaches the data assimilation functionality to the model. With this approach, the data assimilation is finally run analogous to a normal model run, but with additional processors and additional options for the data assimilation.
     45* The implementation with PDAF does not require that the time stepping part of the model is implemented as a subroutine. Instead calls to subroutines that control of the ensemble integration are added to the model code before and after the code parts performing the time stepping. This minimizes the changes in the model code.
     46* In the ''fully parallel'' mode described here, we use as many model tasks as ensemble members. Thus, the model always moves forward in time.
     47* PDAF uses the concept to 'pull' information at the time when it is needed. All model-specific operations, like the initialization of the array of ensemble states in `PDAF_init`, are performed by user-supplied routines which are called through the standard interface of PDAF as call-back routines. Details on the interface and the required routines are given on the pages describing the implementation steps. The concept of the call-back routines is depicted in Fig. 2.
     48* The assimilation system is controlled by the user-supplied routines that are called through PDAF as call-back routines.  With this strategy, the assimilative model program is essentially driven by the model part of the program. Thus, the model is not a sub-component of the assimilation system, but the implementation with PDAF results in a model extended for data assimilation. 
     49* The user-supplied call-back routines can be implemented in the context of the model analogously to the model code. For example, if the model is implemented using Fortran modules (or even common blocks), these can be used to implement the user-supplied routines, too. This simplifies the implementation of the user-supplied routines knowing about the particularities of their model.
     50
    5151[[Image(//pics/PDAF_callback_online_v3.png)]]
    52 [[BR]]'''Figure 2:''' Use of user-supplied call-back functions in PDAF
     52[[BR]]'''Figure 2:''' Use of user-supplied call-back functions in PDAF. The call-back routines are called by PDAF and can use model information that is provided by Fortran modules.
    5353
    54  * The user-supplied call-back routines can be implemented in the context of the model analogously to the model code. For example, if the model is implemented using Fortran common blocks or modules of the model code, these can be used to implement the user-supplied routines, too. This simplifies the implementation of the user-supplied routines knowing about the particularities of their model.
     54
    5555
    5656
     
    6363
    6464
    65 == The flexible parallelization variant ==
     65== The flexible parallelization mode ==
    6666
     67The ''flexible'' parallelization mode allows to run the assimilation program in a way so that a model task (set of processors running one model integration) can propagate several ensemble states successively. This approach allows to use a smaller number of processes compared to the ''fully parallel'.
     68
     69Implementing the ''flexible'' mode requires additional changes to the model code. These are shown on the right side of figure 3. In particular an external loop has to be added. This change does actually only affect the general part of the model code.
    6770
    6871[[Image(//pics/da_extension2x.png)]]
    69 [[BR]]'''Figure 1:''' (left) Generic structure of a model code, (center) extension for ''fully-parallel'' data assimilation system with PDAF, (right) extension for ''flexible'' data assimilation system with PDAF.
     72[[BR]]'''Figure 3:''' (left) Generic structure of a model code, (center) extension for ''fully-parallel'' data assimilation system with PDAF, (right) extension for ''flexible'' data assimilation system with PDAF.
    7073
    7174'''Extensions for the flexible assimilation system'''[[BR]]
     
    7679 * `get_state_pdaf`: Inside the ensemble loop, a call to the interface routine is added to the code. In this routine the names of user-supplied routines are declared and the PDAF-core routine `PDAF_get_state` is called. This routine initializes model fields from the array of ensemble states and initializes the number of time steps that have to be computed and ensures that the ensemble integration is performed correctly.
    7780 * `put_state_pdaf`: At the end of the external loop, the call to the interface routine `put_state_pdaf` is added to the model code. The routine declares the names of user-supplied routines and calls a PDAF_core routine that is specific for each filter. E.g. for the ESTKF, the routine `PDAF_put_state_estkf` is called. This routine writes the propagated model fields back into a state vector of the ensemble array. Also it checks whether the ensemble integration is complete. If not, the next ensemble member will be integrated. If the ensemble integration is complete, the analysis step (i.e. the actual assimilation of the observations) is computed.
     81 * The ''flexible'' parallelization requires that the model can jump back in time. Jumping back in time will be required if the number of model tasks used to evolve the ensemble states is smaller than the number of ensemble members. In this case a model task has integrate more than one model state and will have to jump back in time after the integration of each ensemble member.