Changes between Initial Version and Version 1 of ImplementGenerateObs_PDAF3

Timestamp:

Jun 1, 2025, 7:56:21 PM (4 days ago)

Author:

lnerger

Comment:

--

ImplementGenerateObs_PDAF3

@@ +1 @@
+= Implementation of Observation Generation with PDAF =
+
+[[PageOutline(2-3,Contents of this page)]]
+
+||This page describes the implementation for PDAF2. It will be updated for PDAF3. ||
+
+The observation generation functionality was added with Version 1.14 of PDAF. Here we describe the implementation using PDAF-OMI that was introduced with PDAF version 1.16. (The older implementation variant is documented on the page on [wiki:ImplementGenerateObs_noOMI Implementation of Observation Generation without OMI].)
+
+== Overview ==
+
+Twin data assimilation experiments are a common approach to assess data assimilation methods. In twin experiments one uses the model to generate a ''true'' model state. Further one generates synthetic observations by adding random perturbations to the true state. The, in the actual twin experiment one starts the data assimilation with a state estimate that is different from the true state and assimilates the synthetic observations. One can analyze the assimilation result by comparing the state estimate from the twin experiment with the previously generated true state.
+
+Starting with version 1.14, PDAF provides functionality to generate synthetic observations. The functionality bases on the normal implementation of the assimilation used with PDAF. However, one can run the observation generation with an ensemble of just one member, which should be initialized with the initial true state. PDAF provides the routines `PDAFomi_generate_obs` and `PDAFomi_put_state_generate_obs` to perform the observation generation. These routines use the observation operator routines which the user implements e.g. for assimilating real observations. 
+
+Here we describes the steps need to generate synthetic obsrvations. 
+
+== Initialization ==
+
+The implementation of the initialization of PDAF is explained on the [wikiInitPdaf page on `init_pdaf` and `PDAF_init`].
+
+For the observation generation one just has to set `filtertype = 11`. 
+
+There are no particular options for the observation generation functionality. So for `filter_param_i` one just has to specify the mandatory values of the state dimension and the ensemble size. For `filter_param_r` one has to specify the mandatory values of the forgetting factor (even though, this value is ignored for the observation generation)
+
+== Observation Generation Step ==
+
+This step replaces the analysis step. The implementation is analogous to implementing the analysis step as described on the [wiki:ImplementationofAnalysisStep page on implementing the analysis step]. 
+
+== `PDAFomi_generate_obs` ==
+
+This routine is used in the same way as the filter specific routines `PDAFomi_assimilate_*`. Thus the general aspect have been described on the page [ModifyModelforEnsembleIntegration Modification of the model code for the ensemble integration] and its sub-page on [InsertAnalysisStep inserting the analysis step]. The routine `PDAFomi_generate_obs` is used in the fully-parallel implementation variant of the data assimilation system. When the 'flexible' implementation variant is used, the routines `PDAFomi_put_state_generate_obs' is used as described further below. Here, we list once more the full interface. Subsequently, the full set of user-supplied routines specified in the call to `PDAFomi_generate_obs` is described. Apart from two call-back routines, the routines are identical to e.g. those used for the local filters.
+
+{{{
+  SUBROUTINE PDAFomi_generate_obs(U_collect_state, U_distribute_state, &
+                                  U_init_dim_obs, U_obs_op, U_get_obs_f, &
+                                  U_prepoststep, U_next_observation, status_pdaf)
+}}}
+with the following arguments:
+ * `U_collect_state`: The name of the user-supplied routine that initializes a state vector from the array holding the ensemble of model states from the model fields. This is basically the inverse operation to `U_distribute_state` used in [ModifyModelforEnsembleIntegration#PDAF_get_state PDAF_get_state]
+ * `U_distribute_state`:  The name of a user supplied routine that initializes the model fields from the array holding the ensemble of model state vectors.
+ * `U_init_dim_obs`: The name of the user-supplied routine that provides the size of the full observation vector
+ * `U_obs_op`: The name of the user-supplied routine that acts as the full observation operator on some state vector
+ * `U_get_obs_f`: The name of the user-supplied routine that receives the full vector of generated synthetic observations from PDAF
+ * `U_prepoststep`: The name of the pre/poststep routine as in `PDAF_get_state`
+ * `U_next_observation`: The name of a user supplied routine that initializes the variables `nsteps`, `timenow`, and `doexit`. The same routine is also used in `PDAF_get_state`.
+ * `status_pdaf`: The integer status flag. It is zero, if `PDAF_assimilate_lestkf` is exited without errors.
+
+
+== `PDAFomi_put_state_generate_obs` ==
+
+When the 'flexible' implementation variant is chosen for the assimilation system, the routine `PDAFomi_put_state_generate_obs` has to be used instead of `PDAFomi_generate_obs`. The general aspects of the filter specific routines `PDAFomi_put_state_*` have been described on the page [ModifyModelforEnsembleIntegration Modification of the model code for the ensemble integration]. The interface of the routine is identical with that of `PDAFomi_generate_obs` with the exception the specification of the user-supplied routines `U_distribute_state` and `U_next_observation` are missing. 
+
+The interface is the following:
+{{{
+  SUBROUTINE PDAFomi_put_state_generate_obs(U_collect_state, U_init_dim_obs, U_obs_op, &
+                                  U_get_obs_f, U_prepoststep, status_pdaf)
+}}}
+
+
+
+== User-supplied routines ==
+
+Here, all user-supplied routines are described that are required in the call to `PDAF_generate_obs`. For some of the generic routines, we link to the page on [ModifyModelforEnsembleIntegration modifying the model code for the ensemble integration].
+
+To indicate user-supplied routines we use the prefix `U_`. In the template directory `templates/` as well as in the example implementation in `models/lorenz_96/` these routines exist without the prefix, but with the extension `_pdaf.F90`. The two routines `init_dim_obs_pdafomi` and `obs_op_pdafomi` are part of PDAF-OMI and are contained in the file `callback_obs_pdafomi.F90`. In the section titles below we provide the name of the template file in parentheses.
+
+In the subroutine interfaces some variables appear with the suffix `_p` (short for 'process'). This suffix indicates that the variable is particular to a model sub-domain, if a domain decomposed model is used. Thus, the value(s) in the variable will be different for different model sub-domains. In addition, there will be variables with the suffix `_f` (for 'full').
+
+=== `U_collect_state` (collect_state_pdaf.F90) ===
+
+This routine is independent from the filter algorithm used.
+See the page on [InsertAnalysisStep#U_collect_statecollect_state_pdaf.F90 inserting the analysis step] for the description of this routine.
+
+=== `U_distribute_state` (distribute_state_pdaf.F90) ===
+
+This routine is independent of the filter algorithm used.
+See the page on [InsertAnalysisStep#U_distribute_statedistribute_state_pdaf.F90 inserting the analysis step] for the description of this routine.
+
+
+=== `U_init_dim_obs` (callback_obs_pdafomi.F90) ===
+
+The routine is called at the beginning of each analysis step.  For PDAF, it has to initialize the size `dim_obs_p` of the observation vector according to the current time step. Apart from this routine will initialize overall observation information.  In this routine one just calls `init_dim_obs_TYPE` for each observation type. The routine is described in detail on [wiki:OMI_Callback_obs_pdafomi callback_obs_pdafomi.F90].
+
+
+=== `U_obs_op` (callback_obs_pdafomi.F90) ===
+
+The routine is called during the analysis step. It has to perform the operation of the observation operator acting on a state vector that is provided as `state_p`. The observed state has to be returned in `m_state_p`. In this routine one just calls `obs_op_TYPE` for each observation type. The routine is described in detail on [wiki:OMI_Callback_obs_pdafomi callback_obs_pdafomi.F90].
+
+
+=== `U_get_obs_f` (get_obs_f_pdaf.F90) ===
+
+This routine is specific for the observation generation. In this routine PDAF provides the user with the vector of synthetic observations generated by PDAF. One can then e.g. write the observation vector into a file so that one can use it later in a twin experiment (The template file `readwrite_obs.F90` provides functionality for reading and writing as described on the [wiki:readwrite_obs page on readwrite_obs].
+
+The interface is the following:
+{{{
+SUBROUTINE get_obs_f_pdaf(step, dim_obs_f, observation_f)
+}}}
+with
+ * `step` : `integer, intent(in)`[[BR]] Current time step
+ * `dim_obs_f` : `integer, intent(in)`[[BR]] Size of the full observation vector
+ * `observation_f` : `real, intent(out), dimension(dim_obs_f)`[[BR]] Full vector of synthetic observations (process-local)
+
+Hints:
+ * For the generation of synthetic observations, PDAF does not distinguish between local and global filters. Without parallelization, the full observation vector would be the same for both types of filters. With parallelization the implementation of the observation operator used for generating the observations will define whether different process-domain have the same or distinct observation vectors (i.e. covering the global domain or different process-specific domains). 
+ * In case of the global filters, one uses the functionality of the observation operator for this filter type. With parallelization, the observation operator will initialize an observation vector specifically for each process-domain.
+ * The usual operation performed in this routine is to write the generated synthetic observation into a file. The PDAF package provides the template routine [wiki:readwrite_obs readwrite_obs] for this. Depending on the parallelization, discussed above, one either writes a single file (of the full observation vector is the same for all processes. In this case one a single process calls the writing routine) or a different file for each process (in this case, each process call the routine with a different file name; usually indicating the process-rank number).
+
+
+=== `U_prepoststep` (prepoststep_ens_pdaf.F90) ===
+
+This routine can be identical to that used for the global ESTKF algorithm, which has already been described on the [ModifyModelforEnsembleIntegration#U_prepoststepprepoststep_ens.F90 page on modifying the model code for the ensemble integration]. 
+
+=== `U_next_observation` (next_observation_pdaf.F90) ===
+
+This routine is independent of the filter algorithm used.
+See the page on [InsertAnalysisStep#U_next_observationnext_observation_pdaf.F90 inserting the analysis step] for the description of this routine.
+
+
+== Recommendations for using `PDAFomi_generate_obs` ==
+
+The observation-generation with `PDAFomi_generate_obs` or `PDAFomi_put_state_generate_obs` works analogously to the observation handling in the localized filters like LESTKF and LETKF. However, the observation generation does not modify the ensemble states and `prepoststep_pdaf` is only called once before the each observation generation, but not afterwards. The usual observation functionality of `init_dim_obs_pdafomi` and `obs_op_pdafomi` is used to obtain the observed model state.
+
+One can run the ensemble generation with a single ensemble member (dim_ens=1) or a larger ensemble. If dim_ens>1, the observation operator is applied to the ensemble mean state. The observation error information initialized in `init_dim_obs_pdafomi` is used in combination with Gaussian random noise to compute the perturbations that are added to the true state to generate the observations. Finally `get_obs_f_pdaf` gives the user access to the generated synthetic observation vector so that one can write it to a file for later use (See the [wiki:readwrite_obs page on the template file readwrite_obs.F90] for a description how the observations can be written to a file and used later on).
+
+If one has access to real observations, one can use the implementation of `init_dim_obs_pdafomi` and `obs_ob_pdafomi` for these observations to generate synthetic observations simulating these real observations. Thus, one runs the observation generation using these routines without any modifications. 
+
+'''Note:''' The observation generation should always be performed for a single observation type at a time. Thus one generates separate observation files for each observation type.
+
+
+== Using the synthetic observations in twin experiments ==
+
+To perform a twin experiment using the synthetic observations generated by PDAF, one runs the data assimilation as one would with real observations. If one already initializes the vector of actual observations in the routines `init_dim_obs_TYPE` in the observation modules one only needs a small modification of this routine. Namely, the only required modification is that at the end of `init_dim_obs_TYPE` one overwrites the vector of real observations with the values from the synthetic observations. If one uses the template file `readwrite_obs.F90` for this, one can use `read_syn_obs` from this file at the end of `init_dim_obs_TYPE` to overwrite the observation vector. To allow for a flexible switching between the case using real observations and the twin experiment, one can for example introduce a flag `twin_experiment` that controls whether the real observation values are overwritten. This reading is already included, but out-commented, in the templates. 
+
+Example implementations using `PDAFomi_put_state_generate_obs` and `readwrite_obs.F90` are provided by the implementation of PDAF with the Lorenz-96 model in `models/lorenz96/`. These also use the flag `twin_experiment` to activate the twin experiment (Note: The Lorenz-96 model case always use simulated observations. Nonetheless, it allows to see how the synthetic observations are generated with PDAF and how they are used in a twin experiment).
+