| | 1 | = PDAF Debugging Information (PDAF2) = |
| | 2 | |
| | 3 | [[PageOutline(2-3,Contents of this page)]] |
| | 4 | |
| | 5 | == Overview == |
| | 6 | |
| | 7 | The PDAF debugging output functionality was added with PDAF V2.1. |
| | 8 | |
| | 9 | When implementing PDAF with your model or adding functionality like extending the state vector or adding support for new observations, it can useful to check whether the inputs to the PDAF routines are correctly used inside PDAF. For this purpose, PDAF provides a debugging functionality. It allows you to activate debug output, for example, for a single local analysis domain on a single process of a complex application of a local filter like LESTKF. |
| | 10 | |
| | 11 | Usually, issues arise inside the user-supplied call-back routines or inside PDAF caused by the values that are provided by the user-supplied call-back routines. Thus, a particular aim of the PDAF debug output is to show values of arrays inside PDAF that depend on operations performed inside the user-supplied call-back routines. |
| | 12 | |
| | 13 | || **Note on PDAF-OMI**: When you use PDAF-OMI for the observation handling, there is a separate debug functionality for the observation part. For details see the [wiki:OMI_debugging_PDAF23 documentation on PDAF-OMI debuggging]. One can activate the debugging for PDAF and PDAF-OMI separately or can used both in combination. || |
| | 14 | |
| | 15 | |
| | 16 | == Activating Debugging Output == |
| | 17 | |
| | 18 | Debugging output is activated by a call |
| | 19 | {{{ |
| | 20 | USE PDAFomi, ONLY: PDAF_set_debug_flag ! Include PDAF |
| | 21 | |
| | 22 | INTEGER :: dbg_id ! Debugging flag: >0 for debug output; =0 for no debug output |
| | 23 | |
| | 24 | CALL PDAF_set_debug_flag(dbg_id) |
| | 25 | }}} |
| | 26 | Setting the single argument of `PDAF_set_debug_flag` to a value larger 0 will active the output, while =0 will deactivate it. |
| | 27 | |
| | 28 | The call to `PDAF_set_debug_flag` call can be inserted in different routines. Some recommended places are |
| | 29 | - `init_pdaf`: If the call is inserted before `PDAF_init` is called, one obtains debug output from `PDAF_init`. This allows one to check if the control parameters are handed over correctly and whether the values of the initial ensemble are correct inside PDAF |
| | 30 | - `assimilate_pdaf`: If the call is inserted before `PDAF_assimilate_*` or `PDAF_put_state_*` are called, one obtains debug output for the full analysis step. This variant is particularly useful for the global filters or the 3D-Var schemes. |
| | 31 | - `init_dim_l_pdaf`: If the call is used in this routine one can activate the debug output for a single local analysis domain. For example, to activate debug output only for the local analysis domain `domain_p=4` and filter process 0 (`mype_filter=0`), one uses |
| | 32 | {{{ |
| | 33 | SUBROUTINE init_dim_l_pdaf(step, domain_p, dim_l)) |
| | 34 | |
| | 35 | USE mod_parallel_pdaf, ONLY: mype_filter |
| | 36 | USE PDAFomi, ONLY: PDAF_set_debug_flag |
| | 37 | |
| | 38 | ... |
| | 39 | |
| | 40 | IF (domain_p==4 .AND. mype_filter==0) THEN |
| | 41 | CALL PDAF_set_debug_flag(domain_p) |
| | 42 | ELSE |
| | 43 | CALL PDAF_set_debug_flag(0) |
| | 44 | ENDIF |
| | 45 | |
| | 46 | ... |
| | 47 | |
| | 48 | }}} |
| | 49 | |
| | 50 | For the debugging, it is useful to keep the amount of output low. In particular for a large number of observations, a large ensemble or a large local state vector, the output will be very lengthy. To this end, it can be useful to intentionally reduce the size of the local state vector, number of observations, or the ensemble size for the debugging. For the localized filters it is recommended to only activate the debugging for a single local analysis domain as shown above. The particular domain index can be chosen e.g. based on the coordinates of the domain, which are usually determined in `init_dim_l_pdaf`. Thus, if one encounters an issue in the analysis step for some coordinates one can find the corresponding coordinates in `init_dim_l_pdaf` and subsequently use `PDAF_set_debug_flag` to activate the debug output for this location. |
| | 51 | |
| | 52 | '''For the domain-local filters''', vectors like the local state vector are shown with their full length. Likewise, vector or matrices whose dimension is the ensemble size are shown with their full length. |
| | 53 | |
| | 54 | '''For the global filters''', the state vector or the innovation can have a high dimension. To keep the debug output limited, the outputs of the global filters and the 3D-Var methods are generally limited to the first 6 elements of vectors that have the size of the state vector (like single ensemble states) or the size of the observation vector (like the innovation). Only vectors or matrices whose dimension is the ensemble size are shown with their full length. |
| | 55 | |
| | 56 | |
| | 57 | == Understanding the Debugging Output == |
| | 58 | |
| | 59 | The debug output prints information on configuration parameter values inside PDAF, dimensions and the values of actual arrays inside PDAF. Here, it depends for which routine or part of a data assimilation method the debugging is activated. We discuss a few options here for ensemble filters based on output obtain when activating debug output in the case `/tutorial/offline_2D_serial`. In addition, we discuss the debug output obtained for 3D-Var based on output obtain when activating debug output in the case `/tutorial/3dvar/offline_2D_serial`. WE discuss the debug output for an implementation using PDAF-OMI. However, its use is analogous for the case of the implementation not using PDAF-OMI. |
| | 60 | |
| | 61 | Generally, all debug output lines start with `PDAF-debug`, thus one can use `grep PDAF-debug` when running the data assimilation or on a text file containing the output to see only the debug output. Each line also shows the name of the PDAF routine from which the debug output is written. |
| | 62 | |
| | 63 | Here we discuss some aspects of the debug output from different typical cases: |
| | 64 | - [#DebugoutputfromPDAF_init Debug output from PDAF_init] |
| | 65 | - [#Debugoutputforglobalensemblefilters Debug output from for global ensemble filters] |
| | 66 | - [#Debugoutputforlocalensemblefilters Debug output from for local ensemble filters] |
| | 67 | - [#Debugoutputfor3D-Var Debug output for 3D-Var] |
| | 68 | |
| | 69 | === Debug output from `PDAF_init` === |
| | 70 | |
| | 71 | When we activate the PDAF debug output just before the call the `PDAF_init` and deactivate it directly afterwards (both in init_pdaf_offline.F90) and then run `./PDAF_offline -dim_ens 3|grep debug` we obtain the following output (with line numbers added to support the description): |
| | 72 | |
| | 73 | {{{ |
| | 74 | 1: ++ PDAF-debug set_debug_flag: mype_world 0 activate 1 |
| | 75 | 2: ++ PDAF-debug: 1 PDAF_init -- START |
| | 76 | 3: ++ PDAF-debug PDAF_init: 1 Use MPI_COMM_WORLD for COMM_PDAF |
| | 77 | 4: ++ PDAF-debug PDAF_init: 1 param_int of size 7 values: 648 3 0 0 0 0 0 |
| | 78 | 5: ++ PDAF-debug PDAF_init: 1 param_real of size 1 values: 1.0000000000000000 |
| | 79 | 6: ++ PDAF-debug PDAF_init: 1 Note: If REAL values appear incorrect, please check if you provide them with the correct precision |
| | 80 | 7: ++ PDAF-debug PDAF_init: 1 ensemble member 1 values (1:min(dim_p,6)): 0.96592582999999999 0.99619469999999999 0.99619469999999999 0.96592582999999999 0.90630778999999995 0.81915203999999997 |
| | 81 | 8: ++ PDAF-debug PDAF_init: 1 ensemble member 2 values (1:min(dim_p,6)): 0.99619469999999999 0.99619469999999999 0.96592582999999999 0.90630778999999995 0.81915203999999997 0.70710678000000005 |
| | 82 | 9: ++ PDAF-debug PDAF_init: 1 ensemble member 3 values (1:min(dim_p,6)): 0.99619469999999999 0.96592582999999999 0.90630778999999995 0.81915203999999997 0.70710678000000005 0.57357643999999997 |
| | 83 | 10: ++ PDAF-debug: 1 PDAF_init -- END |
| | 84 | 11: ++ PDAF-debug set_debug_flag: mype_filter 0 deactivate |
| | 85 | }}} |
| | 86 | |
| | 87 | In the debug output we see the following: |
| | 88 | - Lines 1 and 11 show that the debug output is activated, respectively deactivated |
| | 89 | - Lines 2 and 10 mark the beginning and end of the debug output from the routine `PDAF_init` |
| | 90 | - Line 3 shows that `MPI_COMM_WORLD` is used for PDAF. This is the used in most cases (it can be change using a call to `PDAF_set_comm_pdaf`) |
| | 91 | - Lines 4 and 5 show the configuration parameter values (`param_int` and `param_real`. Here, in particular it can happen tha the value of the forgetting factor (`param_real(1)`) is displayed incorrectly. In this case it is likely that this REAL value is not specified with the correct precision or that the user routines and the PDAF library are not compiled for the same precision of REALs. (see the reminder in line 6) |
| | 92 | - Lines 7 - 9 show the first 6 elements of the 3 ensemble states. As for `param_real` it might be the values would be incorrect if the precision of reals in the user code and the PDAF library are inconsistent. |
| | 93 | |
| | 94 | |
| | 95 | === Debug output for global ensemble filters === |
| | 96 | |
| | 97 | To demonstrate the typical output from a global ensemble Kalman filter we use the debug output from the ESTKF. |
| | 98 | When we activate the PDAF debug output just before the call the `PDAF_put_state_global` in assimilation_pdaf_offline.F90 and then run `./PDAF_offline -dim_ens 3|grep debug` we obtain a longer output. The first lines are |
| | 99 | |
| | 100 | {{{ |
| | 101 | 1: ++ PDAF-debug set_debug_flag: mype_filter 0 activate 1 |
| | 102 | 2: ++ PDAF-debug: 1 PDAF_estkf_update -- START |
| | 103 | 3: ++ PDAF-debug PDAF_estkf_update: 1 ensemble member 1 forecast values (1:min(dim_p,6)): 0.96592582999999999 0.99619469999999999 0.99619469999999999 0.96592582999999999 0.90630778999999995 0.81915203999999997 |
| | 104 | 4: ++ PDAF-debug PDAF_estkf_update: 1 ensemble member 2 forecast values (1:min(dim_p,6)): 0.99619469999999999 0.99619469999999999 0.96592582999999999 0.90630778999999995 0.81915203999999997 0.70710678000000005 |
| | 105 | 5: ++ PDAF-debug PDAF_estkf_update: 1 ensemble member 3 forecast values (1:min(dim_p,6)): 0.99619469999999999 0.96592582999999999 0.90630778999999995 0.81915203999999997 0.70710678000000005 0.57357643999999997 |
| | 106 | 6: ++ PDAF-debug PDAF_estkf_update 1 Configuration: param_int(3) dim_lag 0 |
| | 107 | 7: ++ PDAF-debug PDAF_estkf_update 1 Configuration: param_int(4) -not used- |
| | 108 | 8: ++ PDAF-debug PDAF_estkf_update 1 Configuration: param_int(5) type_forget 0 |
| | 109 | 9: ++ PDAF-debug PDAF_estkf_update 1 Configuration: param_int(6) type_trans 0 |
| | 110 | 10: ++ PDAF-debug PDAF_estkf_update 1 Configuration: param_int(7) type_sqrt 0 |
| | 111 | 11: ++ PDAF-debug PDAF_estkf_update 1 Configuration: param_int(8) observe_ens F |
| | 112 | 12: ++ PDAF-debug PDAF_estkf_update 1 Configuration: param_real(1) forget 1.0000000000000000 |
| | 113 | }}} |
| | 114 | |
| | 115 | In these lines we see that the output is written by `PDAF_estkf_update` which is an interal routine of PDAF controlling the analysis step of the ESTKF filter method. |
| | 116 | In the following we see |
| | 117 | - Lines 3-5 shows the first 6 elements of each forecast ensemble state. If these values are not realistic something failed suring the forecast phase and in `collect_state_pdaf`. |
| | 118 | - Lines 6-12 show the configuration parameters used in the filter. They should be consistent with what you provided to `PDAF_init`. |
| | 119 | |
| | 120 | In the following lines we see the output from `PDAF_estkf_analysis`, which is the routine computing the actual analysis update: |
| | 121 | {{{ |
| | 122 | 13: ++ PDAF-debug: 1 PDAF_estkf_analysis -- START |
| | 123 | 14: ++ PDAF-debug PDAF_estkf_analysis: 1 dim_p 648 |
| | 124 | 15: ++ PDAF-debug: 1 PDAF_estkf_analysis -- call init_dim_obs |
| | 125 | 16: ++ PDAF-debug PDAF_estkf_analysis: 1 dim_obs_p 28 |
| | 126 | 17: ++ PDAF-debug: 1 PDAF_estkf_analysis -- observe_ens F |
| | 127 | 18: ++ PDAF-debug: 1 PDAF_estkf_analysis -- call obs_op for ensemble mean |
| | 128 | 19: ++ PDAF-debug: 1 PDAF_estkf_analysis -- call init_obs |
| | 129 | 20: ++ PDAF-debug PDAF_estkf_analysis: 1 innovation d(1:min(dim_obs_p,10)) -0.49477208666666656 0.78162598333333333 1.9351931600000001 0.74712977333333319 0.81521189333333322 1.5003081066666666 1.1976581533333333 0.56667845666666650 0.87435301666666665 1.2913450866666667 |
| | 130 | 21: ++ PDAF-debug PDAF_estkf_analysis: 1 MIN/MAX of innovation -1.2727408933333333 1.9351931600000001 |
| | 131 | 22: ++ PDAF-debug: 1 PDAF_estkf_analysis -- call obs_op 3 times |
| | 132 | 23: ++ PDAF-debug: 1 PDAF_estkf_analysis -- call prodRinvA_l |
| | 133 | 24: ++ PDAF-debug PDAF_estkf_analysis: 1 A^-1 5.1392136433237630 0.83883187576573037 0.83883187576573037 25.2334147876546644 |
| | 134 | 26: ++ PDAF-debug PDAF_estkf_analysis: 1 (HXT R^-1)^T d 11.810299761345098 2.6840904064519067 |
| | 135 | 27: ++ PDAF-debug: 1 PDAF_estkf_analysis -- type_sqrt 0 |
| | 136 | 28: ++ PDAF-debug PDAF_estkf_analysis: 1 Compute eigenvalue decomposition of A^-1 |
| | 137 | 29: ++ PDAF-debug PDAF_estkf_analysis: 1 eigenvalues 2.0086504639577534 5.3639779670206744 |
| | 138 | 30: ++ PDAF-debug PDAF_estkf_analysis: 1 A(HXT R^-1)^T d 2.2391867950151436 0.36078795190823887 |
| | 139 | 31: ++ PDAF-debug PDAF_estkf_analysis: 1 Omega^T 0.18261819498697551 -0.68154038207791268 -0.41706167726071641 -0.11175133961597850 0.0000000000000000 0.0000000000000000 |
| | 140 | 32: ++ PDAF-debug PDAF_estkf_analysis: 1 transform 2.1613652149063913 -0.52126615293618861 -1.6400990619702032 1.3571326901707161 0.47657822124568616 -1.8337109114164025 1.5507445396169155 -0.52126615293618861 -1.0294783866807269 |
| | 141 | 33: ++ PDAF-debug: 1 PDAF_estkf_analysis -- END |
| | 142 | }}} |
| | 143 | |
| | 144 | The lines 13-33 show different information: |
| | 145 | - Lines 14 and 16 show the dimension of the state vector and observation vector, respectively. |
| | 146 | - Line 18 (and some others) show an information like `call obs_op for ensemble mean`. This shows that an observation-related routine is called. When you use PDAF-OMI and in addition activate the debug output of PDAF-OMI you will see additional information about the observation-related values. |
| | 147 | - Line 20 shows the first 10 elements of the innovation vector, thus the difference between the vector of observations and the mean of the observed ensemble. Here one can check for extreme values which might result from incorrect observation or ensemble values. |
| | 148 | - Line 21 shows the minimum and maximum values of the innovation vector. Here, extreme values might also point to errors |
| | 149 | - Line 24 shows the inverse of matrix '''A''' of the ESTKF algorithm (see [wiki:PublicationsandPresentations Nerger et al., 2012 for the description of the ESTKF and ETKF algorithm in the notation used here). This is the full matrix of size dim_ens*dim_ens. If there are !NaNs or extreme values, the analysis update will or can fail |
| | 150 | - Lines 26-31 show intermediate steps of the computation of the transform matrix. In particular, line 30 shows the vector that is used to transform the ensemble mean value from the forecast to the analysis. The shown steps allow you to trace back possible errors. |
| | 151 | - Lines 27 and 28 show that the eigenvalue decomposition of the inverse of matrix '''A''' is computed and the eigenvalues obtained. If the eigenvalue decomposition fails, there will be the error message `PDAF-ERROR(1): Problem in computation of analysis weights!!!`. In this case you should check the values of the inverse of matrix '''A''' which are shown in line 24. |
| | 152 | - Line 31 shows the actual transform matrix. Thus, the forecast ensemble matrix is multiplied with this matrix to obtain the analysis ensemble. |
| | 153 | |
| | 154 | After the analysis ensemble is computed, the first 6 elements of each ensemble state are shown as: |
| | 155 | {{{ |
| | 156 | 34: ++ PDAF-debug PDAF_estkf_update: 1 ensemble member 1 analysis values (1:min(dim_p,6)): 0.92068299395414255 1.0357490219605643 1.1193443475263585 1.1689289749905400 1.1829962688199513 1.1611187995043710 |
| | 157 | 35: ++ PDAF-debug PDAF_estkf_update: 1 ensemble member 2 analysis values (1:min(dim_p,6)): 0.94502620369513857 1.0416094338619108 1.1065438967701202 1.1378565941089185 1.1345960699625004 1.0968614132544858 |
| | 158 | 35: ++ PDAF-debug PDAF_estkf_update: 1 ensemble member 3 analysis values (1:min(dim_p,6)): 0.93916579179379245 1.0172662241209154 1.0644575418424744 1.0793058642259448 1.0613600135754058 1.0111652607977530 |
| | 159 | 36: ++ PDAF-debug: 1 PDAF_estkf_update -- END |
| | 160 | }}} |
| | 161 | This completes the analysis step of the ESTKF. |
| | 162 | |
| | 163 | |
| | 164 | === Debug output for local ensemble filters === |
| | 165 | |
| | 166 | To demonstrate the typical output from a local ensemble Kalman filter we use the debug output from the LESTKF. In this case we do not activate the PDAF debug output in assimilation_pdaf_offline.F90. Instead we activate the debug output in `init_dim_l_pdaf` for domain_p=4 as shown in the example on activating debug output above (we omit the check for `mype_filter` since the example is run using a single process). Then we run `./PDAF_offline -dim_ens 3 -filtertype 7 -cradius 4|grep debug`. In this case we obtain 40 lines of output. |
| | 167 | |
| | 168 | The first lines are: |
| | 169 | {{{ |
| | 170 | 1: ++ PDAF-debug set_debug_flag: mype_filter 0 activate 4 |
| | 171 | 2: ++ PDAF-debug PDAF_lestkf_update: 4 dim_l 1 |
| | 172 | 3: ++ PDAF-debug: 4 PDAF_lestkf_update -- call init_dim_obs_l |
| | 173 | 4: ++ PDAF-debug PDAF_lestkf_update: 4 dim_obs_l 1 |
| | 174 | 5: ++ PDAF-debug: 4 PDAF_lestkf_update -- call g2l_state for ensemble member 1 |
| | 175 | 6: ++ PDAF-debug: 4 PDAF_lestkf_update -- Note: if ens_l is incorrect check user-defined indices in g2l_state! |
| | 176 | 7: ++ PDAF-debug PDAF_lestkf_update: 4 ens_l 0.96592582999999999 |
| | 177 | 8: ++ PDAF-debug: 4 PDAF_lestkf_update -- call g2l_state for ensemble member 2 |
| | 178 | 9: ++ PDAF-debug PDAF_lestkf_update: 4 ens_l 0.90630778999999995 |
| | 179 | 10: ++ PDAF-debug: 4 PDAF_lestkf_update -- call g2l_state for ensemble member 3 |
| | 180 | 11: ++ PDAF-debug PDAF_lestkf_update: 4 ens_l 0.81915203999999997 |
| | 181 | 12: ++ PDAF-debug: 4 PDAF_lestkf_update -- call g2l_state for ensemble mean |
| | 182 | 13: ++ PDAF-debug PDAF_lestkf_update: 4 meanens_l 0.89712855333333319 |
| | 183 | }}} |
| | 184 | |
| | 185 | These lines are analogous to the output for the global ESTKF. However, here the local dimensions are ensemble values are shown: |
| | 186 | - Lines 1 and 4 show the local state dimension (dim_l) and local observation dimension (dim_obs_l) |
| | 187 | - Lines 7-11 show the values of the local ensemble states |
| | 188 | - Line 13 shows the values of the local ensemble mean state |
| | 189 | |
| | 190 | Note that the values of the configuration parameters are not shown if one activates the debug output in `init_dim_l_pdaf`, but only for the case that one activates the debug output in `assimilate_pdaf` (or `assimilation_pdaf_offline`). To get this information for the local filter while avoiding all the debug output form the local analysis, one can activate the debug ouptut in `assimilate_pdaf` and directly deactivate if in `init_dim_l_pdaf`. |
| | 191 | |
| | 192 | In the following lines we see the output from `PDAF_lestkf_analysis`, which is the routine computing the actual local analysis update. |
| | 193 | {{{ |
| | 194 | ++ PDAF-debug: 4 PDAF_lestkf_analysis -- START |
| | 195 | ++ PDAF-debug: 4 PDAF_lestkf_analysis -- call g2l_obs for mean |
| | 196 | ++ PDAF-debug: 4 PDAF_lestkf_analysis -- call init_obs_l |
| | 197 | ++ PDAF-debug PDAF_lestkf_analysis: 4 innovation d_l -0.49477208666666656 |
| | 198 | ++ PDAF-debug: 4 PDAF_lestkf_analysis -- call g2l_obs 3 times |
| | 199 | ++ PDAF-debug PDAF_lestkf_analysis: 4 HXT_l 0.16546108825519312 5.3415828255193198E-002 |
| | 200 | ++ PDAF-debug: 4 PDAF_lestkf_analysis -- call prodRinvA_l |
| | 201 | ++ PDAF-debug PDAF_lestkf_analysis: 4 R^-1(HXT_l) 0.66184435302077249 0.21366331302077279 |
| | 202 | ++ PDAF-debug PDAF_lestkf_analysis: 4 A^-1_l 2.1095094869063713 3.5352964292627041E-002 3.5352964292627041E-002 2.0114130028327533 |
| | 203 | ++ PDAF-debug PDAF_lestkf_analysis: 4 (HXT_l R^-1)^T d_l -0.32746211159263749 -0.10571464322740090 |
| | 204 | ++ PDAF-debug: 4 PDAF_lestkf_analysis -- type_sqrt 0 |
| | 205 | ++ PDAF-debug PDAF_lestkf_analysis: 4 Compute eigenvalue decomposition of A^-1_l |
| | 206 | ++ PDAF-debug PDAF_lestkf_analysis: 4 eigenvalues 2.0000000000000004 2.1209224897391241 |
| | 207 | ++ PDAF-debug PDAF_lestkf_analysis: 4 wbar_l -0.15439607679058354 -4.9843708923282673E-002 |
| | 208 | ++ PDAF-debug PDAF_lestkf_analysis: 4 transform 0.54183477899890109 -0.34083296043308642 -0.20100181856581475 -0.44538532830038724 0.65993482760105171 -0.21454949930066455 -0.43015484539116355 -0.33915015825871242 0.76930500364987597 |
| | 209 | ++ PDAF-debug: 4 PDAF_lestkf_analysis -- END |
| | 210 | }}} |
| | 211 | |
| | 212 | These outputs are analogous to those for the global analysis, but for the vectors are matrices corresponding to the local analysis domain. A difference is that for the local analysis the local arrays are shown in their full length, while for the global analysis only the first six elements are shown. |
| | 213 | |
| | 214 | After the analysis ensemble is computed, the full local analysis ensemble states are shown as: |
| | 215 | {{{ |
| | 216 | ++ PDAF-debug: 4 PDAF_lestkf_update -- call l2g_state for ensemble member 1 |
| | 217 | ++ PDAF-debug PDAF_lestkf_update: 4 ens_l 0.94695014510954834 |
| | 218 | ++ PDAF-debug: 4 PDAF_lestkf_update -- call l2g_state for ensemble member 2 |
| | 219 | ++ PDAF-debug PDAF_lestkf_update: 4 ens_l 0.88927477553898138 |
| | 220 | ++ PDAF-debug: 4 PDAF_lestkf_update -- call l2g_state for ensemble member 3 |
| | 221 | ++ PDAF-debug PDAF_lestkf_update: 4 ens_l 0.80443420998275139 |
| | 222 | ++ PDAF-debug: 4 PDAF_lestkf_update -- call l2g_state for ensemble mean |
| | 223 | ++ PDAF-debug PDAF_lestkf_update: 4 meanens_l 0.89712855333333319 |
| | 224 | ++ PDAF-debug: 4 PDAF_lestkf_update -- local analysis for domain_p 5 |
| | 225 | ++ PDAF-debug: 4 PDAF_lestkf_update -- call init_dim_l |
| | 226 | ++ PDAF-debug set_debug_flag: mype_filter 0 deactivate |
| | 227 | }}} |
| | 228 | |
| | 229 | |
| | 230 | === Debug output for 3D-Var === |
| | 231 | |
| | 232 | To demonstrate the typical output from 3D-Var methods we use the debug output from the 3D-Var with parameterized covariances. The outputs for ensemble 3D-Var and hybrid 3D-Var are similar but, in addition, contain output for the ensemble Kalman filter step used to update the ensemble. |
| | 233 | |
| | 234 | When we activate the PDAF debug output just before the call the `PDAF_put_state_3dvar` in assimilation_pdaf_offline.F90 and then run `./PDAF_offline -dim_cvec 3|grep debug` we obtain 277 lines of output. This long output is due to the iterations performed by the solver algorithm used to minimize the cost function. |
| | 235 | |
| | 236 | The first lines come from before the itervative solver is executed: |
| | 237 | {{{ |
| | 238 | 1: ++ PDAF-debug set_debug_flag: mype_filter 0 activate 1 |
| | 239 | 2: ++ PDAF-debug: 1 PDAF_3dvar_update -- START |
| | 240 | 3: ++ PDAF-debug PDAF_3dvar_update: 1 forecast state (1:min(dim_p,6)): 0.98610507666666669 0.98610507666666669 0.95614277333333331 0.89712855333333330 0.81085553666666654 0.69994508666666655 |
| | 241 | 4: ++ PDAF-debug PDAF_3dvar_update 1 Configuration: param_int(3) solver 1 |
| | 242 | 5: ++ PDAF-debug PDAF_3dvar_update 1 Configuration: param_int(4) dim_cvec 3 |
| | 243 | 6: ++ PDAF-debug PDAF_3dvar_update 1 Configuration: param_int(5) -not used- |
| | 244 | 7: ++ PDAF-debug: 1 PDAF_3dvar_analysis -- START |
| | 245 | 8: ++ PDAF-debug PDAF_3dvar_analysis: 1 dim_p 648 |
| | 246 | 9: ++ PDAF-debug: 1 PDAF_3dvar_analysis -- call init_dim_obs |
| | 247 | 10: ++ PDAF-debug PDAF_3dvar_analysis: 1 dim_obs_p 28 |
| | 248 | 11: ++ PDAF-debug: 1 PDAF_3dvar_analysis -- call obs_op |
| | 249 | 12: ++ PDAF-debug: 1 PDAF_3dvar_analysis -- call init_obs |
| | 250 | 13: ++ PDAF-debug PDAF_3dvar_analysis: 1 innovation d(1:min(dim_obs_p,6)) -0.49477208666666656 0.78162598333333333 1.9351931599999999 0.74712977333333319 0.81521189333333322 1.5003081066666666 |
| | 251 | 14: ++ PDAF-debug PDAF_3dvar_analysis: 1 MIN/MAX of innovation -1.2727408933333333 1.9351931599999999 |
| | 252 | }}} |
| | 253 | |
| | 254 | This output provide the following information: |
| | 255 | - Line 3 shows the first 6 elements for the forecast (background) state vector |
| | 256 | - Lines 4-6 show the configuration parameters for the 3D-Var scheme. |
| | 257 | - Lines 8 and 10 show the dimension of the state vector and observation vector, respectively |
| | 258 | - Line 13 shows the first 6 elements of the innovation vector (observation - observed forecast state) |
| | 259 | - Line 14 shows the minimum and maximum values for the innovation vector. |
| | 260 | For all these value one can check whether they are as pextect or show extreme values or !NaNs |
| | 261 | |
| | 262 | The next lines initialize the solver (in this case the LBFGS solver) and show the configuration variables for the chosen LBFGS solver (m, factr, pgtol). |
| | 263 | {{{ |
| | 264 | 15: ++ PDAF-debug: 1 PDAF_3dvar_optim_LBFGS -- START |
| | 265 | 16: ++ PDAF-debug PDAF_3dvar_optim_LBFGS 1 Solver config: m 5 |
| | 266 | 17: ++ PDAF-debug PDAF_3dvar_optim_LBFGS 1 Solver config: factr 10000000.000000000 |
| | 267 | 18: ++ PDAF-debug PDAF_3dvar_optim_LBFGS 1 Solver config: pgtol 1.0000000000000001E-005 |
| | 268 | }}} |
| | 269 | |
| | 270 | Subsequently the iterations are performed. The debug output that is now shown shows steps in the calculation of the cost function. |
| | 271 | |
| | 272 | For the first iteration the output is |
| | 273 | {{{ |
| | 274 | 19: ++ PDAF-debug: 1 PDAF_3dvar_costf_cvt -- START: iteration 1 |
| | 275 | 20: ++ PDAF-debug: 1 PDAF_3dvar_costf_cvt -- call cvt |
| | 276 | 21: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 state increment after CVT dX(1:min(dim_p,6)) 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 |
| | 277 | 22: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX dX 0.0000000000000000 0.0000000000000000 |
| | 278 | 23: ++ PDAF-debug: 1 PDAF_3dvar_costf_cvt -- call obs_op_lin |
| | 279 | 24: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 observed state after CVT (1:min(dim_obs_p,6)) 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 |
| | 280 | 25: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX HdX 0.0000000000000000 0.0000000000000000 |
| | 281 | 26: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 process local residual d (1:min(dim_obs_p,6)) 0.49477208666666656 -0.78162598333333333 -1.9351931599999999 -0.74712977333333319 -0.81521189333333322 -1.5003081066666666 |
| | 282 | 27: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX d -1.9351931599999999 1.2727408933333333 |
| | 283 | 28: ++ PDAF-debug: 1 PDAF_3dvar_costf_cvt -- call prodRinvA |
| | 284 | 29: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 R^-1 d (1:min(dim_obs_p,6)) 1.9790883466666662 -3.1265039333333333 -7.7407726399999994 -2.9885190933333328 -3.2608475733333329 -6.0012324266666663 |
| | 285 | 30: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX R^-1 d -7.7407726399999994 5.0909635733333332 |
| | 286 | 31: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 process local observation cost J_obs 46.955000904374110 |
| | 287 | 32: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 global observation cost J_obs 46.955000904374110 |
| | 288 | 33: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 process local CV cost J_B 0.0000000000000000 |
| | 289 | 34: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 global CV cost J_B 0.0000000000000000 |
| | 290 | 35: ++ PDAF-debug: 1 PDAF_3dvar_costf_cvt -- call obs_op_adj |
| | 291 | 36: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 H^TR^-1 d (1:min(dim_p,6)) 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 |
| | 292 | 37: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX H^TR^-1 d -7.7407726399999994 5.0909635733333332 |
| | 293 | 38: ++ PDAF-debug: 1 PDAF_3dvar_costf_cvt -- call cvt_adj |
| | 294 | 39: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 CVT(H^TR^-1 d) (1:min(dim_cv_p,6)) -6.1852572643462160 0.26794725702686673 5.9173100073193554 |
| | 295 | 40: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX CVT(H^TR^-1 d) -6.1852572643462160 5.9173100073193554 |
| | 296 | 41: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 process local gradient gradJ (1:min(dim_cv_p,6)) -6.1852572643462160 0.26794725702686673 5.9173100073193554 |
| | 297 | 42: ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX gradJ -6.1852572643462160 5.9173100073193554 |
| | 298 | 43: ++ PDAF-debug: 1 PDAF_3dvar_costf_cvt -- END |
| | 299 | }}} |
| | 300 | |
| | 301 | Noteable lines are here: |
| | 302 | - Lines 20, 23, 28, 35, 38 indicate that call-back routines are called. When PDAF-OMI is used one can activate the OMI debug output which will show information for the observation-related routines. `cvt` and `cvt_adj` are the control vector transform routine and its adjoint, which are both user-supplied. Thus, PDAF itself can only show the result returned from these routines. |
| | 303 | - Line 21 shows the first 6 elements of the state increment vector after appling the CVT. The values are all zero at iteration 1 since the initial increment is zero. |
| | 304 | - Lines 24 and 25 shows the observed increment vector. Again, all values are 0 at iteration 1 |
| | 305 | - Lines 26 and 27 show the initial residual (observed increment minus innovation) and it minimum and maximum values |
| | 306 | - Lines 31 and 32 show the observation part of the cost function. In case of parallelization, line 31 shows the process-local part and line 32 the global sum |
| | 307 | - Lines 33 and 34 show the background part of the cost function, again for the process-local part and the global sum (both are zero at the initial iteration because no increment is computed so far) |
| | 308 | - Line 36 shows the residual scaled by the inverse observation error covariance matrix and projected back to the state space by the adjoint observation operator. Here the values are 0 because the first six elements of the state vector are not observed. (One can see that these values remain 0 during all iterations of the minimization. |
| | 309 | - Line 37 shows the minimum and maximum values of the vector for which line 36 shows the first 6 elements. |
| | 310 | - Line 39 shows the three elements of the vector after the adjoin CVT is applied (at maximum 6 elements are shown, but we run here with dim_cvec=3) |
| | 311 | - Lines 41 and 42 show the first 6 elements of the gradient of the cost function at its minimum and maximum values. |
| | 312 | |
| | 313 | Particular aspects for debugging are here on the one hand the outcome of the observation-related operations (obs_op_lin, prodRinvA, obs_op_adj) and the outcome of the CVT and adjoint CVT. All these have to result in meaningful values. |
| | 314 | |
| | 315 | The further lines of the debug output are for the different iterations of the minimization. Overall 10 iterations are computed durig which the value of `J_obs` decreases and the value of `J_B` increases. At the last iteration the elements in the gradient are reduced to an amplitude of avbout 10E-8: |
| | 316 | {{{ |
| | 317 | ++ PDAF-debug PDAF_3dvar_costf_cvt: 1 MIN/MAX gradJ -1.1445232317441878E-008 1.1542620637072787E-008 |
| | 318 | }}} |
| | 319 | and the convergence is reached. At this point, the iterative solver is ended and the final control vector is projected back by the CVT to obtain the state vector increment as shown in the final output lines: |
| | 320 | {{{ |
| | 321 | 1: ++ PDAF-debug: 1 PDAF_3dvar_optim_LBFGS -- END |
| | 322 | 2: ++ PDAF-debug PDAF_3dvar_analysis: 1 control vector (1:min(dim_p,6)) 2.3896638011774280 -0.26679331415014473 -2.1228704870272916 |
| | 323 | 3: ++ PDAF-debug PDAF_3dvar_analysis: 1 MIN/MAX of control vector -2.1228704870272916 2.3896638011774280 |
| | 324 | 4: ++ PDAF-debug: 1 PDAF_3dvar_analysis -- call cvt for final state increment |
| | 325 | 5: ++ PDAF-debug PDAF_3dvar_analysis: 1 state vector increment (1:min(dim_p,6)) -5.1146746761620225E-002 4.5436483221699978E-002 0.14063915510625419 0.23156859066274435 0.31546191351477898 0.38977007044061401 |
| | 326 | 6: ++ PDAF-debug PDAF_3dvar_analysis: 1 MIN/MAX of state vector increment -0.55408418302780982 0.55408418302780982 |
| | 327 | 7: ++ PDAF-debug: 1 PDAF_3dvar_analysis -- END |
| | 328 | 8: ++ PDAF-debug PDAF_3dvar_update: 1 analysis state (1:min(dim_p,6)): 0.93495832990504646 1.0315415598883666 1.0967819284395874 1.1286971439960776 1.1263174501814455 1.0897151571072805 |
| | 329 | 9: ++ PDAF-debug: 1 PDAF_3dvar_update -- END |
| | 330 | }}} |
| | 331 | |
| | 332 | The infromation shows is the following: |
| | 333 | - Lines 2 and 3 show the three elements of the final incremental control vector as well as its minmum and maximum values |
| | 334 | - Lines 5 and 6 show the first six elements of the state vector increment and its minimumand maximum values |
| | 335 | - Line 8 shows the first six elements of the analysis state vector |
| | 336 | |
| | 337 | '''Other solvers'''[[BR]] |
| | 338 | The debug outputs produced when using the other solvers are similar since the overall steps, like applying the CVT and adjoint CVT or obs_op_lin and obs_op_adj are similar. |
| | 339 | |
| | 340 | '''Ensemble 3D-Var and Hybrid 3D-Var'''[[BR]] |
| | 341 | For these ensemble-based variational methods the debug outputs are also similar to the outputs from the parameterized 3D-Var. However, there are now outputs relating to the ensemble-based CVT (cvt_ens and cvt_adj_ens). In addition there is will be debug output from the execution of the ESTKF or LESTKF filters that update the ensemble perturbations after the ensemble mean state is updated by the minimization of the cost function. |
