= Features and Requirements = 

[[PageOutline]]

 * PDAF is implemented using Fortran 2003. The standard interface also supports models that are written in other languages like C or C++. Also the combination is Python is possible and pyPDAF provide a Python interface for PDAF.
 * The parallelization uses the MPI (Message Passing Interface) standard. The localized filters use, in addition, OpenMP parallelization with features of OpenMP-4.
 * The core routines are fully independent of the model code. They can be compiled separately and can be used as a library.

== Simplifying the implementation ==

PDAF simplifies the implementation of data assimilation systems using existing model code by the following:

 1. PDAF provides fully implemented, parallelized, and optimized ensemble-based algorithms for data assimilation. These are ensemble-based Kalman filters like the LETKF, LESTKF, and EnKF methods. Also nonlinear filters and 3D-variational methods are provided.
 1. PDAF provides two variants to build a data assimilation system:
  1. ''Online Mode'': In this case, PDAF is attached to the model source code by minimal changes to the code. These changes only concern the general part of the code, but not the numerics of the model. In addition, a small set of routines is required that are specific to the model or the observations to be assimilated. These routines can be implemented like routines of the model. The online mode is computational particularly efficient.
  1. ''Offline Mode'': In this case the assimilation program is separate from the model. An assimilation program handling the observations and the analysis step of the data assimilation is implemented with PDAF. Then, the model and the assimilation step are executed separately and the information between model and assimilation are exchanged by files. In this case, one does not need to modify the model source code at all. This approach is simpler to implement than the online mode, but it is computationally less efficient.
 1. PDAF is called through a well-defined universal standard interface. This allows,  for example, to switch between all ensemble-based filter methods without additional coding.
 1. PDAF provides parallelization support for the data assimilation system. If your numerical model is already parallelized, the online mode of PDAF enables the data assimilation system to run several model tasks in parallel within a single executable. However, PDAF can also run small systems on a single processor.
 1. PDAF does not require that your model can be called as a subroutine. In the online mode, PDAF is added to the model and the formed data assimilation system can be executed pretty much like the model-program would without data assimilation, but with additional options for the data assimilation. 
 1. There exist couplings between PDAF with various real models. Such model-couplings are, e.g., available for the MITgcm, NEMO, and SCHISM ocean circulation models, for the AWI Climate Model (a coupled model consisting of either ECHAM or OpenIFS (atmosphere) and FESOM (ocean)) and the Weather and Forecast Model (WRF). See the [wiki:ModelsConnectedToPDAF list of models that were already coupled to PDAF] for an overview.

== Data Assimilation Methods ==

PDAF provides the following data assimilation methods. All assimilation methods are fully implemented, optimized and parallelized. In addition, all ensemble-based methods offer an Ensemble-OI mode in which only a single ensemble state needs to be integrated.

=== Ensemble filters and smoothers ===

**Local ensemble filters:**
 * **LETKF** (Hunt et al., 2007)
 * **LESTKF** (Local Error Subspace Transform Kalman Filter, Nerger et al., 2012, [PublicationsandPresentations see publications])
 * **LEnKF** (classical EnKF with perturbed observations by Evensen (1994), Burgers et al. (1998) with covariance localization)
 * **LNETF** (localized Nonlinear Ensemble Transform Filter by Toedter and Ahrens (2015))
 * **LSEIK** (Nerger et al., 2006)
 * **LKNETF** (Local Kalman-nonlinear Ensemble Transform Filter, Nerger, 2022, [PublicationsandPresentations see publications], added in PDAF V2.1)
 * **EnSRF** (Ensemble square-root filter using serial observation processing and covariance localization) by J. Whitaker and T. Hamill, Mon. Wea. Rev., 2002
 * **EAKF** (Ensemble Adjustment Filter using serial observation processing and covariance localization) by J. Anderson, Mon. Wea. Rev., 2003

**Global ensemble filters:**
 * **ESTKF** (Error Subspace Transform Kalman Filter, Nerger et al., 2012, [PublicationsandPresentations see publications])
 * **ETKF** (The implementation follows Hunt et al. (2007) but without localization, which is available in the LETKF implementation)
 * **EnKF** (The classical formulation with perturbed observations by Evensen (1994), Burgers et al. (1998))
 * **SEIK** (Pham et al. (1998a, 2001), the implemented variant is described in more detail by Nerger et al. (2005))
 * **NETF** (Nonlinear Ensemble Transform Filter by Toedter and Ahrens (2015))
 * **PF** (Particle filter with resampling, see, e.g., Vetra-Carvalho et al. 2018)

**Smoother algorithms** are provided for the following algorithms
 * ESTKF & LESTKF
 * ETKF & LETKF
 * EnKF
 * NETF & LNETF

=== 3D variational methods ===

3D variational methods are provided in thee different variants.
The 3D-Var methods are implemented in incremental form using a control vector transformation (following the review by R. Bannister, Q. J. Roy. Meteorol. Soc., 2017):
 * **3D-Var** - 3D-Var with parameterized covariance matrix
 * **3DEnVar** - 3D-Var using ensemble covariance matrix. The ensemble perturbations are updated with either the LESTKF and ESTKF filters
 * **Hyb3DVar** - Hybrid 3D-Var using a combination of parameterized and ensemble covariance matrix. The ensemble perturbations are updated with either the LESTKF and ESTKF filters

== Requirements ==

 * '''Compiler'''[[BR]]To compile PDAF, a Fortran compiler is required which supports Fortran 2003. PDAF has been tested with a variety of compilers like gfortran, ifort, nfort.
 * '''BLAS''' and '''LAPACK'''[[BR]]The BLAS and LAPACK libraries are used by PDAF. For Linux there are usually packages that provide these libraries. With commercial compilers the functions are usually provided by optimized libraries (like MKL, ESSL).
 * '''MPI''' [[BR]] An MPI library is required (e.g. OpenMPI). 
 * '''make'''[[BR]]PDAF provides Makefile definitions for different compilers and operating systems.

== Test machines ==

PDAF has been tested on various machines with different compilers and MPI libraries. Current test machines include:
 * Linux Desktop computer, Ubuntu, gfortran, OpenMPI
 * Notebook Apple !MacBook, MacOS, gfortran, OpenMPI 
 * Atos cluster 'Lise' at HLRN (Intel Cascade Lake processors), ifort, IMPI and OpenMPI
 * Windows with Cygwin, gfortran, OpenMPI
 * NEC SX-Aurora vector computer, nfort, NEC MPI
Past test machines also included
 * NEC SX-ACE, sxf90 compiler (rev 530), sxmpi
 * Cray CS400, Cray compiler, IMPI
 * Cray CS400, ifort, IMPI
 * Cray XC30 and XC40, Cray compiler CCE, MPICH
 * SGI Altix !UltraViolet, SLES 11 operating system, ifort compiler, SGI MPT
 * IBM p575 with Power6 processors, AIX6.1, XLF compiler 12.1, ESSL library, POE parallel environment
 * IBM !BladeCenter with Power6 processors, AIX5.3, XLF compilers 10.1 to 13.1, ESSL library, POE parallel environment


== Test cases ==

The regular tests use a rather small configuration with a simulated model. This model is provided in the PDAF tutorial code of the release.
In addition, the scalability of PDAF was examined with a real implementation with the finite element sea-ice ocean model (FESOM). In these tests up to 4800 processor cores of a supercomputer have been used (see [PublicationsandPresentations Nerger and Hiller (2013)]). In [PublicationsandPresentations Nerger et al., GMD (2020)], the scalability was assessed up to 12144 processor cores for the coupled atmosphere-ocean model AWI-CM (Sidorenko et al., 2015). Also, [PublicationsandPresentations Kurtz et al., GMD, (2016)] assessed the parallel performance up to 32768 processor cores for the TerrSysMP terrestial model system.

To examine PDAF's behavior with large-scale cases, experiments with a simulated model have been performed. By now the biggest case had a state dimension of 8.64^.^10^11^. An observation vector of size 1.73^.^10^10^ was assimilated. For these experiments, the computations used 57600 processor cores. In this case, the dimensions were limited by the available memory of the compute nodes. Using an ensemble of 25 states, the distributed ensemble array occupied about 2.9 GBytes of memory for each core (about 165 TBytes in total).