Changes between Version 11 and Version 12 of ImplementationConceptOnline


Ignore:
Timestamp:
Oct 1, 2013, 3:24:36 PM (7 years ago)
Author:
lnerger
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ImplementationConceptOnline

    v11 v12  
    11= Implementation Concept of the Online Mode =
    2 
    3 The online mode offers two implementation variants. The first one, called 'fully-parallel', assumes that you have a sufficient number of processes when running the data assimilation program so that all ensemble states can be propagated concurrently. The parallelism allows for a simplified implementation. The second implementation variant, called 'flexible', allows to run the assimilation program in a way so that a model task (set of processors running one model integration) can propagate several ensemble states successively. This implementation variant is a bit more complicated, because one has to ensure that the model can jump back in time.
    4 
    5 If the data assimilation can be run with a sufficient number of processors to use the 'fully-parallel' variant, we recommend to use it. With PDAF Version 1.10 particular interface routines '''PDAF_assimilate_X''' have been introduced to support this implementation style.
    6 
    7 == Online mode: Attaching PDAF to a model ==
    82
    93{{{
     
    2115[[PageOutline(2-3,Contents of this page)]]
    2216
    23 Here we describe the extensions of the model code for the online mode of PDAF.
     17== Online mode: Attaching PDAF to a model ==
    2418
    25 The assimilation system is built by adding call to PDAF-routines to the general part of the model code. As only minimal changes to the model code are required, we refer to this as "attaching" PDAF to the model.
     19Here we describe the extensions of the model code for the online mode of PDAF. The online mode offers two implementation variants. The first one, called ''fully-parallel'', assumes that you have a sufficient number of processes when running the data assimilation program so that all ensemble states can be propagated concurrently. The parallelism allows for a simplified implementation. The second implementation variant, called ''flexible'', allows to run the assimilation program in a way so that a model task (set of processors running one model integration) can propagate several ensemble states successively. This implementation variant is a bit more complicated, because one has to ensure that the model can jump back in time.
     20
     21If the data assimilation can be run with a sufficient number of processors to use the 'fully-parallel' variant, we recommend to use it. With PDAF Version 1.10 particular interface routines `PDAF_assimilate_X` have been introduced to support this implementation style.
     22
     23The assimilation system is built by adding calls to PDAF-routines to the general part of the model code. As only minimal changes to the model code are required, we refer to this as "attaching" PDAF to the model.
    2624
    2725The general concept is depicted in figure 1. The left hand side shows a typical abstract structure of a numerical model. When the program is executed, the following steps are performed:
     
    3028 3. When the integration of the model fields is completed after a defined number of time steps, various post-processing operations are performed. Then the program stops.
    3129
    32 [[Image(//pics/da_extension.png)]]
    33 [[BR]]'''Figure 1:''' (left) Generic structure of a model code, (right) extension for data assimilation with PDAF
     30[[Image(//pics/da_extension2x.png)]]
     31[[BR]]'''Figure 1:''' (left) Generic structure of a model code, (center) extension for ''fully-parallel'' data assimilation system with PDAF, (right) extension for ''flexible'' data assimilation system with PDAF.
    3432
    35 The right hand side of Figure 1 shows the extensions required for the assimilation system (marked yellow):
     33
     34'''Extensions for the fully-parallel assimilation system'''[[BR]]
     35The center of Figure 1 shows the extensions required for the ''fully-parallel'' assimilation system (marked yellow):
     36 * `init_parallel_pdaf`: This routine is inserted close to the start of the model code. If the model itself is parallelized the correct location is directly after the initialization of the parallelization in the model code. `init_parallel_pdaf` creates the parallel environment that allows to perform several time stepping loops ("model tasks") at the same time.
     37 * `init_pdaf`: This routine is added after the initialization part of the model. In `init_pdaf`, parameters for PDAF can be defined and then the core initialization routine `PDAF_init` is called. This core routine also initializes the array of ensemble states. Subsequently, the PDAF-core routines `PDAF_get_state` is called. This routine initializes model fields from the array of ensemble states and initializes the number of time steps that have to be computed and ensures that the ensemble integration is performed correctly.
     38 * `assimilate_pdaf`: This routine is added to the model code at the end of the time stepping loop (just before the ''END DO'' in a Fortran program). The routine declares the names of user-supplied routines and calls a filter-specific PDAF-core routine `PDAF_assimilate_X` (with, e.g., X=estkf, for the ESTKF). This routine has to be called at the end of each time step. It counts whether all time steps of the current forecast phase have been computed. IF this is not the case, the program continues integrating the model. IF the forecast phase is completed, the analysis step (i.e. the actual assimilation of the observations) is computed. Subsequently, the next forecast phase is initialized by writing the analysis state vector into the model fields and setting the number of time steps in the next forecast phase. (Please note: The routines `PDAF_assimilate_X` have been introduced with Version 1.10 of PDAF.)
     39
     40
     41'''Extensions for the flexible assimilation system'''[[BR]]
     42The right hand side of Figure 1 shows the extensions required for the ''flexible'' assimilation system (marked yellow):
    3643 * `init_parallel_pdaf`: This routine is inserted close to the start of the model code. If the model itself is parallelized the correct location is directly after the initialization of the parallelization in the model code. `init_parallel_pdaf` creates the parallel environment that allows to perform several time stepping loops ("model tasks") at the same time.
    3744 * `init_pdaf`: This routine is added after the initialization part of the model. In `init_pdaf`, parameters for PDAF can be defined and then the core initialization routine `PDAF_init` is called. This core routine also initializes the array of ensemble states.
    38  * Ensemble loop: In order to allow for the integration of the state ensemble an unconditional loop is added around the time stepping loop of the model. This loop will allow to compute the time stepping loop multiple time during the model integration. PDAF provides an exit-flag for this loop. (There are some conditions, under which this external loop is not required. Some notes on this are given further below.)
    39  * `PDAF_get_state`: Inside the ensemble loop, the PDAF core routine `PDAF_get_state` is added to the code. This routine initializes model fields from the array of ensemble states and initializes the number of time steps that have to be computed and ensures that the ensemble integration is performed correctly.
    40  * `PDAF_put_state`: At the end of the external loop, the PDAF core routine `PDAF_put_state` is added to the model code. This routine writes the propagated model fields back into a state vector of the ensemble array. Also it checks whether the ensemble integration is complete. If not, the next ensemble member will be integrated. If the ensemble integration is complete, the analysis step (i.e. the actual assimilation of the observations) is computed.
     45 * Ensemble loop: In order to allow for the integration of the state ensemble an unconditional loop is added around the time stepping loop of the model. This loop will allow to compute the time stepping loop multiple time during the model integration. PDAF provides an exit-flag for this loop. (This external loop can be avoided with the ''fully-parallel'' implementation variant.)
     46 * `get_state_pdaf`: Inside the ensemble loop, a call to the interface routine is added to the code. In this routine the names of user-supplied routines are declared and the PDAF-core routine `PDAF_get_state` is called. This routine initializes model fields from the array of ensemble states and initializes the number of time steps that have to be computed and ensures that the ensemble integration is performed correctly.
     47 * `put_state_pdaf`: At the end of the external loop, the call to the interface routine `put_state_pdaf` is added to the model code. The routine declares the names of user-supplied routines and calls a PDAF_core routine that is specific for each filter. E.g. for the ESTKF, the routine `PDAF_put_state_estkf` is called. This routine writes the propagated model fields back into a state vector of the ensemble array. Also it checks whether the ensemble integration is complete. If not, the next ensemble member will be integrated. If the ensemble integration is complete, the analysis step (i.e. the actual assimilation of the observations) is computed.
    4148
    42 With the implementation strategy of PDAF, four routines and the external loop have to be added to the model code. While this looks like a large change in figure 1, this change does actually only affect the general part of the model code. In addition, the amount of source code of the numerical model will be much longer than the addition for the data assimilation system.
     49With the implementation strategy of PDAF, calls to three to four routines have to be added to the model code. In case of the flexible implementation variant also the external loop has to be added. While this looks like a large change in figure 1, this change does actually only affect the general part of the model code. In addition, the amount of source code of the numerical model will be much longer than the addition for the data assimilation system. Please note that the calls to the routines `PDAF_get_state`, `PDAF_put_state_X`, `PDAF_assimilate_X` (X chosen according to the filter) could also be added directly in the model code. However, using the interface routines (`get_state_pdaf` and `put_state_pdaf`) reduces the additions to the model code.
    4350
    4451== Important aspects of the implementation concept ==
     
    4653 * The implementation concept of PDAF does not require that the time stepping part of the model is implemented as a subroutine. Instead calls to subroutines that control of the ensemble integration are added to the model code before and after the code parts performing the time stepping. If the time stepping part is implemented as a subroutine, this subroutine can be called in between the additional routines.
    4754 * Depending on the parallelization, there can be cases in which the model has to jump back in time and cases in which the time always moves forward:
    48   * Jumping back in time will be required if the number of model tasks used to evolve the ensemble states is smaller than the number of ensemble members. In this case a model task has integrate more than one model state and will have to jump back in time after the integration of each ensemble member.
    49   * If there are as many model tasks as ensemble members, the model time always moves forward. In this case, one can implement PDAF also without the external ensemble loop. That is, one can add calls to `PDAF_get_state` and `PDAF_put_state` directly into the code of the model's time stepping loop. This strategy might also be called for, if a model uses nested loops (like a loop over minutes inside a loop over hours).
    50  * Model-specific operations like the initialization of the array of ensemble states in `PDAF_init` are actually performed by user-supplied routines. These routines are called through the standard interface of `PDAF`. Details on the interface and the required routines are given on the pages describing the implementation steps.
     55  * Jumping back in time will be required if the number of model tasks used to evolve the ensemble states is smaller than the number of ensemble members. In this case a model task has integrate more than one model state and will have to jump back in time after the integration of each ensemble member. This case requires to implement the ''flexible'' implementation variant.
     56  * If there are as many model tasks as ensemble members, the model time always moves forward. In this case, one can implement PDAF with the ''fully-parallel'' variant without the external ensemble loop. That is, one calls `PDAF_assimilate_X` at the end of each time step. This strategy might also be simpler if a model uses nested loops (like a loop over minutes inside a loop over hours).
     57 * Model-specific operations like the initialization of the array of ensemble states in `PDAF_init` are actually performed by user-supplied routines. These routines are called through the standard interface of PDAF. Details on the interface and the required routines are given on the pages describing the implementation steps.
    5158 * The assimilation system is controlled by the user-supplied routines that are called through PDAF as call-back routines.  With this strategy, the assimilation program is essentially driven by the model part of the program. Thus, logically the model is not a sub-component of the assimilation system, but the implementation with PDAF results in a model extended for data assimilation.
    5259[[Image(//pics/PDAF_callback.png)]]