| | 1 | = Offline Mode: Initialization of PDAF and the ensemble by PDAF_init = |
| | 2 | |
| | 3 | {{{ |
| | 4 | #!html |
| | 5 | <div class="wiki-toc"> |
| | 6 | <h4>Offline Mode: Implementation Guide</h4> |
| | 7 | <ol><li><a href="OfflineImplementationGuide">Main page</a></li> |
| | 8 | <li><a href="OfflineAdaptParallelization">Adaptation of the parallelization</a></li> |
| | 9 | <li>Initialization of PDAF</li> |
| | 10 | <li><a href="OfflineImplementationofAnalysisStep">Implementation of the analysis step</a></li> |
| | 11 | <li><a href="OfflineAddingMemoryandTimingInformation">Memory and timing information</a></li> |
| | 12 | </ol> |
| | 13 | </div> |
| | 14 | }}} |
| | 15 | |
| | 16 | [[PageOutline(2-3,Contents of this page)]] |
| | 17 | |
| | 18 | == Overview == |
| | 19 | |
| | 20 | After the initialization of the parallelization for the assimilation program, the initialization of PDAF has to be implemented. Internally to PDAF, the initialization is performed by the routine `PDAF_init`. Typically, the initializations of all variables required for the call to `PDAF_init` can be collected into a single subroutine. This implementation strategy is useful, as only a single addition subroutine call has to be added to the model source code. In the example in `testsuite/src/offline_1D` the routine in the file `init_pdaf_offline_D.F90` shows this strategy. The file `init_pdaf.F90` in `templates/` provides a commented template for this routine, which can be used as the basis of the implementation. |
| | 21 | |
| | 22 | `PDAF_init` itself calls a user-supplied routine to initialize the ensemble of model states through its interface. In the example there are separate routines for the different filters. They are in the files `init_seik_offline_D.F90` (for SEIK, LSEIK, ETKF, LETKF), `init_enkf_offline_D.F90` (for EnKF), and `init_seek_offline_D.F90` (for SEEK). |
| | 23 | |
| | 24 | == Using `init_pdaf` == |
| | 25 | |
| | 26 | In the offline mode, the routine `init_pdaf` is executed after the initialization of the parallelization. (Note: In the main program of example implementation (`main/main_offine.F90`) we added a call to a routine `initialize` in between for clarity of the implementation. As no real model initialization is conducted, this routine simply initializes the size of the model state. This initialization could also be performed in `init_pdaf`.) |
| | 27 | |
| | 28 | In the routine `init_pdaf` a number of variables are defined that are used in the call to `PDAF_init` as described in '[#RequiredargumentsforPDAF_init Required arguments for `PDAF_init`]'. (Please note: All names of subroutines that start with `PDAF_` are core routines of PDAF, while subroutines whose name end with `_pdaf` are generally user-supplied interface routines) There are also a few variables that are initialized in `init_pdaf` but not used in the call to `PDAF_init`. These are variables that are specific for the data assimilation system, but only shared in between the user-supplied routines. For the example case, these variables are described in '[#Othervariablesfortheassimilation Other variables for the assimilation]' |
| | 29 | |
| | 30 | The example implementation and the template version allow to parse all variables through a command line parser. This method provides a convenient way to define an experiment and could also be used for other models. The parser module is provided as `templates/parser_mpi.F90` |
| | 31 | |
| | 32 | == Required arguments for `PDAF_init` == |
| | 33 | |
| | 34 | The call to `PDAF_init` has the following structure: |
| | 35 | {{{ |
| | 36 | CALL PDAF_init(filtertype, subtype, step_null, & |
| | 37 | filter_param_i, length_filter_param_i, & |
| | 38 | filter_param_r, length_filter_param_r, & |
| | 39 | COMM_model, COMM_filter, COMM_couple, & |
| | 40 | task_id, n_modeltasks, filterpe, & |
| | 41 | U_ensemble_init, screen, status_pdaf) |
| | 42 | }}} |
| | 43 | |
| | 44 | The required variables are the following: |
| | 45 | |
| | 46 | * `filtertype`: An integer defining the type of filter algorithm. Available are |
| | 47 | * 0: SEEK |
| | 48 | * 1: SEIK |
| | 49 | * 2: EnKF |
| | 50 | * 3: LSEIK |
| | 51 | * 4: ETKF |
| | 52 | * 5: LETKF |
| | 53 | * `subtype`: An integer defining the sub-type of the filter algorithm (always 5 for the offline mode) |
| | 54 | * `step_null`: Always 0 for the offline mode. |
| | 55 | * `filter_param_i`: Integer array collecting several variables for PDAF. The first two variables are mandatory and equal for all filters. Further variables are optional (see example code). The mandatory variables are in the following order: |
| | 56 | * The size of the local state vector for the current process. |
| | 57 | * The ensemble size for all ensemble-based filters (or the rank of the state covariance matrix for mode-based filters like SEEK) |
| | 58 | * `length_filter_param_i`: An Integer defining the length of the array `filter_param_i`. The entries in the array are parsed up to this index. |
| | 59 | * `filter_param_r`: Array of reals collecting floating point variables for PDAF. The first variable is mandatory and equal for all filters. Further variables are optional (see example code). The mandatory variable is: |
| | 60 | * The value of the forgetting factor (required to be larger than zero) |
| | 61 | * `length_filter_param_r`: An Integer defining the length of the array `filter_param_r`. The entries in the array are parsed up to this index. |
| | 62 | * `COMM_model`: The communicator variable `COMM_model` as initialized by `init_parallel_pdaf`. If the model-communicator is named differently in the actual program, the name has to be adapted |
| | 63 | * `COMM_filter`: The communicator variable `COMM_filter` as initialized by `init_parallel_pdaf`. |
| | 64 | * `COMM_couple`: The communicator variable `COMM_couple` as initialized by `init_parallel_pdaf`. |
| | 65 | * `task_id`: The index of the model tasks as initialized by `init_parallel_pdaf`. |
| | 66 | * `n_modeltasks`: The number of model tasks as defined before the call to `init_parallel_pdaf`. |
| | 67 | * `filterpe`: The flag showing if a process belongs to `COMM_filter` as initialized by `init_parallel_pdaf`. |
| | 68 | * `U_ensemble_init`: The name of the user-supplied routine that is called by `PDAF_init` to initialize the ensemble of model states. (See '[#User-suppliedroutineU_ensemble_init User-supplied routine U_ensemble_init]' |
| | 69 | * `screen`: An integer defining whether information output is written to the screen (i.e. standard output). The following choices are available: |
| | 70 | * 0: quite mode - no information is displayed. |
| | 71 | * 1: Display standard information (recommended) |
| | 72 | * 2: as 1 plus display of timing information during the assimilation process |
| | 73 | * 3: Display detailed information for debugging |
| | 74 | * `status_pdaf`: An integer used as status flag of PDAF. If `status_pdaf` is zero upon exit from `PDAF_init` the initialization was successful. An error occurred for non-zero values. (The error codes are documented in the routine PDAF_init.) |
| | 75 | |
| | 76 | It is recommended that the value of `status_pdaf` is checked in the program after PDAF_init is executed. Only if its value is 0 the initialization was successful. |
| | 77 | |
| | 78 | == Other variables for the assimilation == |
| | 79 | |
| | 80 | The routine `init_pdaf` in the example also initializes several variables that are not used to call `PDAF_init`. These variables control some functionality of the user-supplied routines for the data assimilation system and are shared with these routines through `mod_assimilation`. These variables are for example: |
| | 81 | * `rms_obs`: Assumed observation error |
| | 82 | * `local_range`: Localization radius in grid points for the observation domain in LSEIK and LETKF |
| | 83 | * `srange`: support radius, if a 5th order polynomial is used to weigh the observation errors |
| | 84 | * `locweight`: Type of localizing weight |
| | 85 | It is useful to define variables lke these at this central position. Of course, this definition has to be adapted to the particular model used. |
| | 86 | |
| | 87 | == User-supplied routine `U_ensemble_init` == |
| | 88 | |
| | 89 | The user-supplied routine the we named `U_ensemble_init` here, is called by PDAF through the defined interface described below. The routine is called by all MPI processes that compute the filter analysis step (i.e. those for which 'filterpe' is set to true. In the standard configuration of init_parallel_pdaf these are all processes of the first model task, i.e. task_id=1.) `U_ensemble_init` is only called by `PDAF_init` if no error occurred before; thus the status flag is zero. |
| | 90 | |
| | 91 | The interface is the following: |
| | 92 | {{{ |
| | 93 | SUBROUTINE U_ensemble_init(filtertype, dim_p, dim_ens, & |
| | 94 | state_p, Uinv, ens_p, flag) |
| | 95 | }}} |
| | 96 | with |
| | 97 | * `filtertype`: The integer defining the type of filter algorithm as given in the call to `PDAF_init` |
| | 98 | * `dim_p`: An integer holding the size of the state dimension for the calling process as specified in the call to `PDAF_init` |
| | 99 | * `dim_ens`: An integer holding the size of the ensemble (or the rank of the state covariance matrix for SEEK) |
| | 100 | * `state_p`: A real array of size (`dim_p`) for the local model state of the calling process (Only relevant for mode-based filters) |
| | 101 | * `Uinv`: A real array of size (`dim_ens-1`, `dim_ens-1`) for the inverse of matrix '''U''' from the decomposition of the state error covariance matrix '''P''' = '''VUV^T^''' (Only relevant for mode-based filters like SEEK.) |
| | 102 | * `ens_p`: A real array of size (`dim_p`, `dim_ens`) the has to hold upon exit the ensemble of model states. |
| | 103 | * `flag`: Status flag for PDAF. It is 0 upon entry and can be set by in the user-supplied routine, depending on the success of the ensemble initialization. Preferably, values above 102 should be used for failures to avoid conflicts with the error codes defined within PDAF_init. |
| | 104 | |
| | 105 | In the initialization routine `U_ensemble_init` one has to distinguish between ensemble-based and mode-based filters. The only mode based filter supplied with PDAF is SEEK, while all other methods are ensemble-based. |
| | 106 | |
| | 107 | === Initialization for ensemble-based filters === |
| | 108 | |
| | 109 | Generally, the filters SEIK, LSEIK, EnKF, ETKF, and LETKF are ensemble-based filters. For these filters only the array `ens_p` needs to be initialized by the ensemble of model states. If a model with domain decomposition is used, the full ensemble for the local sub-domain of the MPI process has to be initialized. |
| | 110 | |
| | 111 | The arrays `state_p` and `Uinv` are allocated to their correct sizes because they are used during the assimilation cycles. They are not yet initialized and it is allowed to use these arrays in the initialization. An exception from this is EnKF for which `Uinv` is allocated only with size (`1`,`1`), because `Uinv` is not using for EnKF. |
| | 112 | |
| | 113 | For the offline mode, one will usually read the ensemble states from output files of the model used to the ensemble integrations. Thus, one has to implement a reading routine for the model files. |
| | 114 | |
| | 115 | |
| | 116 | === Initialization for mode-based filters === |
| | 117 | |
| | 118 | The only mode-based filter supplied with PDAF is currenly the SEEK filter. For this filter the initialization bases on the decomposition of the state error covariance matrix in the form '''P''' = '''VUV^T^'''. According to this decomposition, the array `ens_p` has to be initialized to hold the modes from matrix '''V''' and `Uinv` holds the inverse of matrix '''U'''. In addition `state_p` has to be initialized with the initial state estimate. If a model with domain decomposition is used, the part of all modes for the local sub-domain of the MPI process and the corresponding part of the state vector has to be initialized. |
| | 119 | |
| | 120 | == Testing the PDAF initialization == |
| | 121 | |
| | 122 | The PDAF initialization can be tested by compiling the assimilation program (without the later call to `PDAF_put_state_*` and executing it. The Makefile of the model has to be extended to include the additional files. The core part of PDAF can be compiled separately as a library and can then simply be linked to the model code. This is the strategy followed in the PDAF-package. |
| | 123 | One can test if the initialization in PDAF_init is sucessful and if the ensemble array is correctly initialized. |
| | 124 | |
| | 125 | Standard output from PDAF_init should look like the following: |
| | 126 | {{{ |
| | 127 | PDAF: Initialize SEIK Filter |
| | 128 | |
| | 129 | SEIK configuration |
| | 130 | filter sub-type = 0 |
| | 131 | --> Standard SEIK |
| | 132 | --> Transform ensemble with deterministic Omega |
| | 133 | --> Use fixed forgetting factor: 1.00 |
| | 134 | |
| | 135 | Parallelization - Filter on model PEs: |
| | 136 | Total number of PEs: 4 |
| | 137 | Number of parallel model tasks: 1 |
| | 138 | PEs for Filter: 4 |
| | 139 | # PEs per ensemble task and local ensemble sizes: |
| | 140 | Task 1 |
| | 141 | #PEs 4 |
| | 142 | N 300 |
| | 143 | }}} |
| | 144 | |
| | 145 | The correctness of the ensemble initialization in `U_ensemble_init` should be checked by the user. |
