| 52 | | The general espects of the filter-specific routines `PDAF_assimilate_*` have been described on the page [wiki:OnlineModifyModelforEnsembleIntegration_PDAF3 Modification of the model code for the ensemble integration]. |
| 53 | | The interface for the routine `PDAF_assimilate_letkf` contains several routine names for routines that operate on the local analysis domains (marked by `_l` at the end of the routine name). In addition, there are names for routines that consider all available observations required to perform local analyses with LETKF within some sub-domain of a domain-decomposed model (marked by `_f` at the end of the routine name). In case of a serial execution of the assimilation program, these will be all globally available observations. However, if the program is executed with parallelization, this might be a smaller set of observations. |
| 54 | | |
| 55 | | To explain the difference, it is assumed, for simplicity, that a local analysis domain consists of a single vertical column of the model grid. In addition, we assume that the domain decomposition splits the global model domain by vertical boundaries as is typical for ocean models and that the observations are spatially distributed observations of model fields that are part of the state vector. Under these assumptions, the situation is the following: When a model uses domain decomposition, the LETKF algorithm is executed such that for each model sub-domain a loop over all local analysis domains (e.g. vertical columns) that belong to the model sub-domain is performed. As each model sub-domain is treated by a different process, all loops are executed parallel to each other. |
| 56 | | |
| 57 | | For the update of each single vertical column, observations from some larger domain surrounding the vertical column are required. If the influence radius for the observations is sufficiently small there will be vertical columns for which the relevant observations reside completely inside the model sub-domain of the process. However, if a vertical column is considered that is located close to the boundary of the model sub-domain, there will be some observations that don't belong spatially to the local model sub-domain, but to a neighboring model sub-domain. Nonetheless, these observations are required on the local model sub-domain. A simple way to handle this situation is to initialize for each process all globally available observations, together with their coordinates. While this method is simple, it can also become inefficient if the assimilation program is executed using a large number of processes. As for an initial implementation it is usually not the concern to obtain the highest parallel efficiency, the description below assumes that 'full' refers to the global model domain. |
| 58 | | |
| 59 | | The interface when using the LETKF algorithm is the following: |
| 60 | | {{{ |
| 61 | | SUBROUTINE PDAF_assimilate_letkf(U_collect_state, U_distribute_state, U_init_dim_obs_f, U_obs_op_f, & |
| 62 | | U_init_obs_f, U_init_obs_l, U_prepoststep, U_prodRinvA_l, & |
| | 52 | This routine is used both in the ''fully-parallel'' and the ''flexible'' implementation variants of the data assimilation system. (See the page [wiki:OnlineModifyModelforEnsembleIntegration_PDAF3 Modification of the model code for the ensemble integration] for these variants) |
| | 53 | |
| | 54 | The interface for the routine `PDAF_assimilate_letkf` contains several routine names for routines that operate on the local analysis domains (marked by `_l` at the end of the routine name). In addition, there are names for routines that consider all available observations required to perform local analyses with LESTKF within some sub-domain of a domain-decomposed model (we refer to these as 'full' observations, marked by `_f` at the end of the routine name). In case of a serial execution of the assimilation program, these will be all globally available observations. However, if the program is executed with parallelization, one might choose a smaller set of observations. We will explain this is some detail below. |
| | 55 | |
| | 56 | Here, we list the full interface of the routine. Subsequently, the user-supplied routines specified in the call are explained. |
| | 57 | |
| | 58 | The interface is: |
| | 59 | {{{ |
| | 60 | SUBROUTINE PDAF_assimilate_letkf(U_collect_state, U_distribute_state, & |
| | 61 | U_init_dim_obs_f, U_obs_op_f, U_init_obs_f, & |
| | 62 | U_init_obs_l, U_prepoststep, U_prodRinvA_l, & |
| 90 | | |
| 91 | | == `PDAF_put_state_letkf` == |
| 92 | | |
| 93 | | When the 'flexible' implementation variant is chosen for the assimilation system, the routine `PDAF_put_state_letkf` has to be used instead of `PDAF_assimilate_letkf`. The general aspects of the filter specific routines `PDAF_put_state_*` have been described on the page [wiki:OnlineModifyModelforEnsembleIntegration_PDAF3 Modification of the model code for the ensemble integration]. The interface of the routine is identical with that of `PDAF_assimilate_letkf` with the exception the specification of the user-supplied routines `U_distribute_state` and `U_next_observation` are missing. |
| 94 | | |
| 95 | | The interface when using the LETKF algorithm is the following: |
| 96 | | {{{ |
| 97 | | SUBROUTINE PDAF_put_state_letkf(U_collect_state, U_init_dim_obs_f, U_obs_op_f, U_init_obs_f, & |
| 98 | | U_init_obs_l, U_prepoststep, U_prodRinvA_l, U_init_n_domains, & |
| 99 | | U_init_dim_l, U_init_dim_obs_l, & |
| | 90 | == `PDAF_assim_offline_letkf ` == |
| | 91 | |
| | 92 | This routine is used to perform the analysis step for the offline mode of PDAF. |
| | 93 | The interface of the routine is identical with that of the 'assimilate'-routine, except that the user-supplied routines `U_distribute_state`, `U_collect_state` and `U_next_observation` are missing. |
| | 94 | |
| | 95 | The 'assim_offline' routines were introduced with PDAF V3.0 to simplify the [wiki:OfflineImplementationGuide_PDAF3 implementation of the offline mode]. |
| | 96 | |
| | 97 | The interface is: |
| | 98 | {{{ |
| | 99 | SUBROUTINE PDAF_assim_offline_letkf(& |
| | 100 | U_init_dim_obs_f, U_obs_op_f, U_init_obs_f, & |
| | 101 | U_init_obs_l, U_prepoststep, U_prodRinvA_l, & |
| | 102 | U_init_n_domains, U_init_dim_l, U_init_dim_obs_l, & |
| | 108 | |
| | 109 | == `PDAF_put_state_letkf` == |
| | 110 | |
| | 111 | |
| | 112 | This routine exists for backward-compatibility. In implementations that were done for PDAF V2.3.1 and before, a 'put_state' routine was used for the [wiki:OnlineFlexible_PDAF3 'flexible' parallelization variant] and for the [wiki:OfflineImplementationGuide_PDAF3 offline mode]. This routine allows to continue using the previous implementation structure. |
| | 113 | The interface of the routine is identical with that of the 'assimilate'-routine, except that the user-supplied routines `U_distribute_state` and `U_next_observation` are missing. |
| | 114 | |
| | 115 | The interface when using the LETKF algorithm is the following: |
| | 116 | {{{ |
| | 117 | SUBROUTINE PDAF_put_state_letkf(U_collect_state, & |
| | 118 | U_init_dim_obs_f, U_obs_op_f, U_init_obs_f, & |
| | 119 | U_init_obs_l, U_prepoststep, U_prodRinvA_l, & |
| | 120 | U_init_n_domains, U_init_dim_l, U_init_dim_obs_l, & |
| | 121 | U_g2l_state, U_l2g_state, U_g2l_obs, & |
| | 122 | U_init_obsvar, U_init_obsvar_l, status) |
| | 123 | }}} |
| | 124 | |
| | 125 | == Explanation of 'full observations' == |
| | 126 | |
| | 127 | Above we mention the concept of 'full' observations. We distinguish them from the globally available observations for efficiency. |
| | 128 | |
| | 129 | Note: For an initial implementation, one might not needd to worry about high efficiency, so that 'full' can refer to all available observations. |
| | 130 | |
| | 131 | To explain why 'full' observations can be different from globally available observations, we assume, for simplicity, that we have a 2-dimensional domain and that a local analysis domain consists of a single grid point of the model grid. In addition, we assume that the domain decomposition splits the global model domain in compact sub-domains and that the observations are spatially distributed observations of model fields that are part of the state vector. |
| | 132 | |
| | 133 | The LESTKF performs a loop over all local analysis domains, i.e. grid points. When a model uses domain decomposition, the loop is over all grid points that belong to a process sub-domain. As each model sub-domain is treated by a different process, all loops are executed parallel to each other. |
| | 134 | |
| | 135 | For the update of each local analysis domain (grid points), observations within the localization radius around its location are required. If the influence radius for the observations is sufficiently small, there will be grid points a for which the relevant observations reside completely inside the model sub-domain of the process. However, if a grid point is located close to the boundary of the model sub-domain, there will be some observations that reside on a neighboring process sub-domain, but are within the localization radius. One needs to assimilate these observations as otherwise, there could be unrealistic steps in the analysis field. However, there will also be observations that reside far away from the process sub-domain and will never influence the analysis result in this domain. |
| | 136 | |
| | 137 | A simple way to handle this situation is to initialize for each process all globally available observations, together with their coordinates. The observation operator would be applied on each sub-domain and then the observed ensemble would be collect using parallel communication with MPI. While this method is simple, it can also become inefficient if the assimilation program is executed using a large number of processes. In particular, all observation would need to be checked even if they are far away from the process sub-domain. |
| | 138 | |
| | 139 | More efficient is hence to select as 'full' observations only those observations that can have an effect on the local analyses of a process sub-domain. These are the observations that reside within the sub-domain, plus observations in neighboring sub-domains that reside within the localization radius. Setting up 'full' observations in this way leads to a smaller number of observations whose distance need to be checked for each local analysis domain. Howeever, one would need to find an implementation that provides the 'full' observations. |
| | 140 | |
| | 141 | || Note: The handling of 'full' observations is one of the aspects that motivated the development of PDAF-OMI and the relaed avanced interface (now the PDAF3 interface). Here, PDAF-OMI does take case of the 'full' observations. See the [wiki:ImplementationofAnalysisStep_PDAF3 Implementation Guide for the Analysis Step for the advanced interface using PDAF-OMI]. || |
| | 142 | |
| 207 | | This routine can generally be identical to that used for the global SEIK filter, which has already been described on the [wiki:OnlineModifyModelforEnsembleIntegration_PDAF3#U_prepoststepprepoststep_ens_pdaf.F90 page on modifying the model code for the ensemble integration]. For completeness, the description is repeated: |
| 208 | | |
| 209 | | The interface of the routine is identical for all filters. However, the particular operations that are performed in the routine can be specific for each filter algorithm. Here, we exemplify the interface on the example of the ETKF. |
| 210 | | |
| 211 | | The interface for this routine is |
| 212 | | {{{ |
| 213 | | SUBROUTINE prepoststep(step, dim_p, dim_ens, dim_ens_p, dim_obs_p, & |
| 214 | | state_p, Uinv, ens_p, flag) |
| 215 | | |
| 216 | | INTEGER, INTENT(in) :: step ! Current time step |
| 217 | | ! (When the routine is called before the analysis -step is provided.) |
| 218 | | INTEGER, INTENT(in) :: dim_p ! PE-local state dimension |
| 219 | | INTEGER, INTENT(in) :: dim_ens ! Size of state ensemble |
| 220 | | INTEGER, INTENT(in) :: dim_ens_p ! PE-local size of ensemble |
| 221 | | INTEGER, INTENT(in) :: dim_obs_p ! PE-local dimension of observation vector |
| 222 | | REAL, INTENT(inout) :: state_p(dim_p) ! PE-local forecast/analysis state |
| 223 | | ! The array 'state_p' is not generally not initialized in the case of SEIK/EnKF/ETKF. |
| 224 | | ! It can be used freely in this routine. |
| 225 | | REAL, INTENT(inout) :: Uinv(dim_ens, dim_ens) ! Inverse of matrix U |
| 226 | | REAL, INTENT(inout) :: ens_p(dim_p, dim_ens) ! PE-local state ensemble |
| 227 | | INTEGER, INTENT(in) :: flag ! PDAF status flag |
| 228 | | }}} |
| 229 | | |
| 230 | | The routine `U_prepoststep` is called once at the beginning of the assimilation process. In addition, it is called during the assimilation cycles before the analysis step and after the ensemble transformation. The routine is called by all filter processes (that is `filterpe=1`). |
| 231 | | |
| 232 | | The routine provides for the user the full access to the ensemble of model states. Thus, user-controlled pre- and post-step operations can be performed. For example the forecast and the analysis states and ensemble covariance matrix can be analyzed, e.g. by computing the estimated variances. In addition, the estimates can be written to disk. |
| 233 | | |
| 234 | | Hint: |
| 235 | | * If a user considers to perform adjustments to the estimates (e.g. for balances), this routine is the right place for it. |
| 236 | | * Only for the SEEK filter the state vector (`state_p`) is initialized. For all other filters, the array is allocated, but it can be used freely during the execution of `U_prepoststep`. |
| 237 | | * The interface has a difference for ETKF and SEIK: For the ETKF, the array `Uinv` has size `dim_ens` x `dim_ens`. In contrast it has size `dim_ens-1` x `dim_ens-1` for the SEIK filter. |
| 238 | | * The interface through which `U_prepoststep` is called does not include the array of smoothed ensembles. In order to access the smoother ensemble array one has to set a pointer to it using a call to the routine `PDAF_get_smootherens` (see page on [AuxiliaryRoutines auxiliary routines]) |
| | 246 | The routine has already been described for modifying the model for the ensemble integration and for inserting the analysis step. |
| | 247 | |
| | 248 | See the page on [wiki:OnlineModifyModelforEnsembleIntegration_PDAF3#distribute_state_pdafdistribute_state_pdaf.F90 modifying the model code for the ensemble integration] for the description of this routine. |
