Release notes for PDAF

Version 1.15 - December 9, 2019

Changes:

• New timer mode 3: This timer mode show the times spent in the different call-back routines. This helps the user to determine which routines take most time and have hence the largest potential for optimization.
• Given the new timer mode 3, the previous timer mode 3 is now shifted to 4 and the old timer mode 4 to 5.
• New directory models/: We moved the implementations of the Lorenz-63 and Lorenz96 models with PDAF to the new directory models/. The former directory testsuite/ is now intended for validation tests, while the models that are fully implemented with PDAF reside in a separate directory
• Revised validation scripts: In both the testsuite and the tutorial directories, one can run automated test runs whose outputs are compared with reference outputs using Python
• PDAF_get_state and PDAF_put_state_X can now called in flexible order each having its own counting for the ensemble member index: Before it was required that a call to PDAF_get_state was followed by a matching call to PDAF_put_state_X. Now On can also call PDAF_get_state for all ensemble members at the beginning of the forecast phase. Later the matching calls to PDAF_put_state_X are done. It's only important to have the same number of calls. This change allows, e.g. to distribute and compute the forecasts on GPUs or let other load-balancing software distribute the forecasts.
• Following the change in PDAF_get_state/PDAF_put_state_X, the routine PDAF_get_memberid was revised. The routine will return the index of the ensemble member according to the current (i.e. last) call to either PDAF_get_state or PDAF_put_state_X (In practice you should see no difference to before if you call the routines alternately)
• The SEEK filter is declared deprecated, i.e. we plan to no longer support it in some future release. This mode-based filters usually shows a worse performance compared to the current ensemble filters. (Please let us know if you actually use the SEEK filter)
• The tutorial case online_2d_parallelmodel_fullpar_1fpe now allows to compute the ensemble forecast with a different number of processes for each model task.
• For PDAF_eofcovar we clarified that the array 'states' is actually destroyed by the singular value decomposition compute in the routine. Accordingly, the meanstate is also no longer added to this array after the SVD.
• Model-binding for AWI-CM: The directory modelbindings/ now also contains code to use PDAF with the model AWI-CM (the AWI climate model, a coupled atmosphere-ocean model consisting of the atmosphere model ECHAM6 and the ocean model FESOM).
• Bug corrections:
  • The analysis step of the particle filter is corrected. The analysis ensemble was incorrectly computed for state dimensions >200.
  • To avoid an error massage with rather old compilers, we added a 'save' statement to the variable screenout (This is required by OpenMP older than version 4.0, e.g. in gfortran older than version 4.9.1)

---

Previous versions

Version 1.14 - July 4, 2019

Changes:

• Added functionality to generate synthetic observations to simplify the application of twin data assimilation experiments (filtertype=11).
• Added a particle filter with importance resampling (filtertype=12)
• Added an option for ETKF/SEIK/ESTKF to compute the innovation either from mean of HX or from H(meanX). Only the latter was supported by now, but using mean(HX) is the correct approach for nonlinear observation operators. This option is specified as param_int(8) (see overview of available filter options)
• Added timing information on the time required to collect and distribute the ensemble in the online implemented data assimilation.
• Added more routines to simplify the implementation of parallelized local analysis steps when collecting full observations: 'PDAF_gather_obs_f_flex' and 'PDAF_gather_obs_f2_flex'. These are flexible variants fo the routines introduced in version 1.13 as they don't rely on a call to 'PDAF_gather_dim_obs_f'.
• All template routines now give an output starting with 'TEMPLATE'. Given that one should delete this output line when one implements a routine, this will help to keep an track on how far a particular implementation is complete.
• Revised Lorenz-63 model example and added data assimilation and plotting scripts to it. The use is analogous to the Lorenz-96 model, but without localized filters.
• Bug corrections:
  • For the 'fullpar' case in which the filter runs on separate processes form the model tasks, we corrected the PDAF-internal parallelization setup, as the process-local ensmeble size was not set correctly for task_id=0
  • For the 'fullpar' case also PDAF_get_state for SEEK. Now distribute_state is only executed for model tasks, but not the separate filter task
  • Corrected an array allocation in PDAF_enkf_omega. This lead to an error for the case that dim_ens-1 was not equal to the specified rank of the set of EOFs. This only happens when using this routine to generate a large ensemble for a small state dimension (as now used in the Lorenz-63 model test case).

Version 1.13.2 - September 3, 2018

Changes:

• Revised Lorenz-96 model example and added a detailed documentation on the web site. Now it is easier to use this fully featured implementation of PDAF with the Lorenz-96 model. Further, plotting scripts for both Matlab and Python are provided.
• Bug corrections:
  • Corrected the smoother mode of LETKF, subtype 1 (Here the transform matrix for smoothing was not correctly computed)
  • Corrected PDAF_diag_ensstats (For the case that the statistics over all elements of the ensemble array were to be computed, still only element 1 was used)
  • Corrected testsuite/src/main/main.F90 (Here, we added a call PDAF_deallocate in version 1.13 but did not enclose it in a preprocessor check whether PDAF is active. This lead to a compile failure when compiling with deactivated PDAF)

Version 1.13.1 - March 12, 2018

Changes:

• Added routine PDAF_get_assim_flag. This routines returns the information whether onthe last call to a PDAF_assimilate routine the analysis step of a filter was actually computed, i.e. observation were assimilated. (This can be used e.g. in model with leap frog time stepping to compute an Euler time step directly after the analysis step)
• Bug corrections: We got notified by a user (thank you) that two bugs we announced to be fixed in Version 1.13 were actually not fixed. Unfortunately, the corrected files didn’t make it into the previous release. Now the following two bugs are really fixed:
  • Corrected order of arguments in 'PDAF_assimilate_lenkf'
  • Corrected 'PDAF_local_weight' (the weight was not always intialized to 0 for distances beyond the localization radius. This only happens if you set the localization radius to be larger than the support-radius of the weight function)

Version 1.13 - February 6, 2018

Changes:

• Added a model binding for the MITgcm ocean circulation model
• Added routines to simplify the implementation of parallelized local analysis steps when collecting full observations: 'PDAF_gather_dim_obs_f', 'PDAF_gather_obs_f', 'PDAF_gather_obs_f2' (see updated tutorials on how to use these routines)
• Updated the tutorials to reflect our updated implementation recommendations. For example, we now recommend to compute the distances of observations for the local analysis step only once
• Add routine 'PDAF_deallocate' to deallocate the big internal arrays of PDAF at the end of a program. In the tutorials it is called in the new routine 'finalize_pdaf'.
• In the tutorials, the variables in 'init_pdaf' are reordered to clearly separate the variables that are used in 'PDAF_init' from those that are only used in the call-back routines.
• Added a verbosity flag for 'PDAF_sampleens' and 'PDAF_eofcovar' to make them better usable in parallel programs (note: the interface has changed)
• Bug correction: Corrected order of arguments in 'PDAF_assimilate_lenkf'
• Bug correction: Corrected 'PDAF_local_weight' (the weight was not always intialized to 0 for distances beyond the localization radius)
• Bug correction: Corrected use do doexit flag in 'PDAF_get_state' (it was very unlikely to cause a problem)

Version 1.12 - December 21, 2016

Changes:

• New filter method: LEnKF - The classical Ensemble Kalman Filter with perturbed observations (Evensen 1994) now with covariance localization (filtertype 8)
• New filter: NETF (Nonlinear Ensemble Transform Filters by Toedter and Ahrens (Monthly Weather Review 143 (2015) 1347-1367) including smoother extension (filtertype 9)
• New filter: LNETF - NETF with local analysis and observation localization, including smoother extension (filtertype 10)
• revised memory counting to work with more than 2.1 GB per process
• New routines for ensemble generation: PDAF_eofcovar and PDAF_sampleens. These routines simplify to generate an ensemble with second-order exact sampling (see documentation on ensemble generation and documentation for each routine linked on that page)
• New routines for ensemble diagnostics (histograms, skewness and kurtosis, effective sample size). (see documentation on data assimilation diagnostics and documentation for each routine linked on that page)
• Bug correction: forgetting factor in EnKF smoother was treated incorrectly
• Bug correction: SEEK filter in single-precision case showed an issue
• Additional functionality for Lorenz-96 test case: model error can be added to the integration and incomplete observations are supported.
• revised ensemble generation in testsuite for EnKF with dummy model: for ensemble larger than state dimension a random sampling is now used, while for ensemble up to the size of the state the mean-preserving intialization as before is used.

Version 1.11.1 - February 28, 2015

This release fixes a few bugs in the compilation with Cray compilers and cleans up the make process.
Changes:

• Bug fix: The Makefiles for the PDAF library and the main Makefile for the testsuite are corrected to allow for the correct compilation with Cray compilers (CCE).
• The Makefiles for the testsuite cases are cleaned up to avoid the warning about the missing directory dummympi/ (The directory doesn't exist any more)
• Typo corrections in screen output of the PDAF core routines

Version 1.11 - December 22, 2014

Changes:

• Revised the screen output. All output lines from the PDAF code routines now start with 'PDAF'. This will make it easier to grep for these lines when PDAF is used with models that generate a lot of output. Further, the output now also works correctly if the model state or the number of observations are very large (up to O(10⁹)) or if the number of processes is large (up to O(10⁵)).
• OpenMP parallelization for the local filters (LESTKF, LSEIK, LETKF) was added. This allows to speed up the analysis step without the more compilated changes required for MPI-parallelization. The tutorial have been updated to explain how to use the OpenMP parallelization.
• The communication for collecting ensemble states for the assimilation update is improved. We switched to non-blocking MPI communication, which allows to gather the ensemble members in arbitrary order and can speed up the collection of ensemble members. Analogous, the distribution of ensemble members can be improved using non-blocking communication. The old blocking communication is still available by setting the preprocessor flag BLOCKING_MPI_EXCHANGE.
• The overall configuration possibilities for the MPI parallelization have been revised. Now, it is more easily possible to run the filter using a different set of processors as the models run. This variant can be useful, e.g. if the memory of a computer is so limited that one cannot store the arrays from the model and the ensemble array at the same time.
• New routines:
  • PDAF_get_obsmemberid: This routine returns the index of the ensemble member on which an observation operator has to be applied
  • PDAF_prepost: This routine can be used, e.g., like PDAF_assimilate_lestkf. However, the routine only calls the pre/poststep routine once, but does not compute an analysis step. The routine can be used to analyze an ensemble forecast.
  • PDAF_put_state_prepost: This routine can be used, e.g. like PDAF_put_state_lestkf. However, the routine only calls the pre/poststep routine once, but does not compute an analysis step. The routine can be used to analyze an ensemble forecast.

Version 1.10 - October 4, 2013

Changes:

• Added a simplified implementation variant for the online implementation of PDAF that relies on the parallelization. For the case that there are sufficient processors to integrate all ensemble members in parallel, the new routines PDAF_assimilate_X (with X being replaced by the name of the filter algorithm) can be used instead of PDAF_get_state and PDAF_put_state_X. The use of PDAF_assimilate_X is explained in the tutorial.
• Added tutorial implementations for an example of the online implementation (coupling model and PDAF into a single assimilation program) of PDAF. The examples demonstrate the implementation with a model that itself is not parallelized as well as a parallelized model. Corresponding tutorials are now available on the tutorial web page.
• Revised templates for the implementation of the online mode.

Version 1.9 - May 6, 2013

Changes:

• Added smoothers for ESKTF, ETKF, EnKF and the local filters LESTKF, LETKF.
• Added fixed basis (subtype=2) and fixed covariance matrix (subtype=3) variants for ESTKF, ETKF and the local filters LESTKF, LETKF.
• Added an example implementation of the offline mode with a simple 2D model domain and observations with data gaps. This implementation serves for a tutorial, that provides a step-by-step description on how to implement the analysis step in the offline mode.
• Added a function PDAF_get_memberid to query the index of an ensemble member during the forecast phase.
• revised the templates and simplified the implementation for the offline mode
• fixed a bug in SEIK/LSEIK for subtype=3

Version 1.8 - February 12, 2012

Changes:

• Added Error Subspace Transform Kalman filter (ESTKF) and localized variant LESTKF. In addition a variant of the SEIK filter with symmetric square-root and explicit ensemble transformation is now available. (These filters have been introduced in the paper: "A unification of ensemble square-root filters" by L. Nerger, T. Janjic, J. Schroeter, and W. Hiller, Monthly Weather Review, 140, 2335-2345, ​doi:10.1175/MWR-D-11-00102.1)
• Added support to specify the type of the matrix square root in the SEIK filter. (Cholesky decomposition or symmetric square root based on singular value decomposition. The effects of these square root are also discussed in the paper mentioned above.)
• Revised the internal structure of PDAF to simplify the implementation of additional filters. (See the page about adding a filter algorithm for details.)
• Added support to compile for either double or single precision.
• Clean-up of PDAF's internal timers and memory allocation counting.

Version 1.7 - September 16, 2011

Changes:

• Revised internal structure of PDAF to simplify implementation of additional assimilation methods.
• Added full data assimilation implementation of Lorenz-96 model with PDAF.
• Revision of observation localization. It also includes the regulated localization that was introduced in the paper "A regulated localization scheme for ensemble-based Kalman filters" by L. Nerger et al. to appear in Q. J. Roy. Meteor. Soc. (accessible online: ​DOI:10.1002/qj.945)
• Added an option to display parameter options for a selected filter using the compiled program.
• Added routines with a simplified interface. The simplified interface does not require that you provide the names of user-supplied subroutines int he call to PDAF. However, one is restricted to use pre-defined routine names.
• License change: Now PDAF is licensed with the more flexible Lesser GNU Public License (older versions of PDAF used the GNU Public License).

Version 1.6.2 - 10/05/2010

Changes:

• Change in Makefiles to correct compilation on Linux with gfortran

Version 1.6.1 - 08/27/2010

Changes:

• Added pre-processor statement PDAF_NO_UPDATE to simplify tests during implementation.
• Unified interface to pre/poststep routines. For the EnKF Uinv was added. This array is never used in EnKF.
• Added shortened timer output to PDAF_print_info

Version 1.6.0 - 03/18/2010

Version distributed after presentation at Ocean Sciences Meeting, Portland, OR.

Changes:

• Added ETKF and LETKF to public release

Version 1.5.0 - 01/19/2010

Changes:

• Revised directory structure to separate PDAF core routines from test suite.

Versions 1.4.2 to 1.1.0

Version 1.0 - 10/08/2004

Original public release after participating at the GODAE International Summer School of Oceanography, „An Integrated View of Oceanography: Ocean Weather Forecasting in the 21st Century”, Lalonde les Maures, France