The radar product on which STEPS and LDCast are evaluated in this work is RADCLIM : it is a quantitative precipitation estimation product based on radar measurements, which are merged with rain gauge measures. Several techniques exist for this merging: the product used in this study was obtained with the Kriging with External Drift (KED) technique. The time resolution is 5 min and the spatial resolution is 1 km × 1 km. The evaluation domain is a 320 px × 320 px square in the radar frame.

Ten stratiform and ten convective events were selected based on the convective precipitation and the mean large-scale precipitation derived from ERA5 . Events are considered convective if their mean convective precipitation was greater than 10⁻⁴ mm h⁻¹ and their large-scale precipitation was less than 10⁻⁶ mm h⁻¹. Similarly, stratiform events were selected as those with a mean convective precipitation below 10⁻⁷ mm h⁻¹ and a large-scale precipitation above 8.07 × 10⁻⁴ mm h⁻¹.

As both STEPS and LDCast transform the rain rate to a logarithmic scale, all computations in this work are done with a similar scale. The transformation used in this work is 1f(R)=10log⁡10(R)forR≥0.1mmh-1-15forR<0.1mmh-1 The unit after this transformation is dBR.

The two nowcasting models considered in this work are STEPS and LDCast. STEPS is built on Lagrangian persistence , which consists in advecting the rainfall field with a motion field estimated through an optical flow algorithm. In STEPS, the rainfall field is in addition decomposed into fields of different spatial scales via a cascade, and each of the levels of the cascade evolves according to an auto-regressive (AR) process, usually of order 2. There are two sources of perturbation in STEPS: the motion field is stochastically perturbed via the method designed in (named BPS in the pysteps library; ), and the field intensities are perturbed via the noise component of the AR(2) processes . The noise itself was originally generated with a parametric method, but the nonparametric method developed in is now more commonly used. STEPS is implemented at the Royal Meteorological Institute of Belgium (RMI) with pysteps .

In STEPS, there is no value for the pixels for which the value should come from the advection of pixel values out of the radar frame. To avoid as much as possible this problem, STEPS nowcasts were produced using the whole available radar domain of RADCLIM (see Sect. ), and the analysis presented in this work was performed on a smaller 320 px × 320 px square extracted from this domain. There were still some pixels with missing values after doing this, and these were assigned the no-rain value of -15 dBR. In stratiform STEPS ensembles, the fraction of those pixels in an ensemble at a given lead time is at most 3 % and it is smaller than 1 % for convective STEPS ensembles. The effects of such missing values in STEPS nowcasts are therefore supposed to be negligible.

LDCast is a latent diffusion model, meaning that a sequence of rainfall fields is first encoded to a latent space learned by the variational autoencoder of the model. The forecaster network then predicts the latent representation of future rainfall fields. Conditionally to this output, the denoiser stack produces different members, which are finally transformed back from the latent space to rainfall fields .

LDCast was trained on the radar product with code RZC, which uses only data from the radar themselves, while the RADCLIM dataset used in this work includes a merging with rain gauges. Merging with rain gauges is known to smooth out small scales , so that a difference between the nowcasts of LDCast and the radar observations at small scales can be expected.

The LDCast nowcasts were first produced on a 416 px × 416 px area extracted from the radar domain. The nowcasts were then cropped to the 320 px × 320 px square used to evaluate the models.

The spectral error is computed as the power spectrum of the pixel-wise error e. It is a scale-by-scale decomposition of the mean squared error of the ensemble mean. On the other hand, the spectral variance is defined as 6σk2=1N-1∑i=1NPSxi-x‾k and is often called the spread in meteorological applications.

The spectral error and spectral variance for STEPS and for LDCast, together with the power spectra of observations, are displayed in Fig.  (average over the selected events). The color of the curves represents the lead time at which the spread and the error are computed.

Both the spectral error and the spectral variance increase with lead time and eventually saturate to the power spectrum of the observation (see Appendix ). The error at some scales is immediately saturated : depending on the model and on the event type, the largest saturated scale after 5 min is between 5 and 10 km.

For a well-calibrated ensemble, the error and the spread should be equal, provided the ensemble mean is not biased . Whether STEPS and LDCast ensembles are well-calibrated is assessed more precisely with Fig. , where the scale-by-scale ratio of the spectral variance over the de-biased spectral error is displayed. A ratio smaller than 1 indicates underdispersion, while there is overdispersion when it is larger than 1. The spread / error ratio in Fig.  was computed with the de-biased error, meaning that a small bias of the ensemble mean was removed in Eq. ().

Overall, both models have variance / error ratios close to 1 for scales above 5 km. LDCast is slightly underdispersive, except for the larger scales in convective events, where it is rather overdispersive. On the other hand, for stratiform events, STEPS ensembles are underdispersive for short lead times but become well calibrated for longer lead times. In convective cases, STEPS is either underdispersive or overdispersive depending on the lead time and the scale.

The spectral error and the power spectrum of observations flatten out for scales smaller than 5 km. This is however not the case for the spectral variance of STEPS and LDCast, so that the ensembles seem to be underdispersed. However, the power spectrum of radar images is known to flatten due to white noise contamination at these scales due to non-meteorological sources . Merging with rain gauge, as is done for the radar data used in this work, is also known to smooth the small scales , and that explains the shape of the corresponding part of the spectral error. Therefore, the flattening of the power spectrum of radar images can be considered as not physical, and this underdispersion should not be identified as a weakness of the ensembles. This occurs at scales for which the error is immediately saturated, so that this is anyway irrelevant from a forecasting point of view.

It is not clear why the spectral variance of STEPS is not as large as the spectral error, since STEPS mainly extrapolates radar observations. On the other hand, LDCast was trained on a radar product without rain gauge merging, so that it can be expected that its nowcasts behave differently at small scales than the radar observations used in this work.

The ensemble covariance matrix and its eigenvalues and eigenvectors are used in this section to investigate the morphology of the ensembles. If the ensemble is considered as a cloud of points in phase space, the eigenvectors of the covariance matrix point in the directions in which the ensemble is the most extended (as measured by the variance in these directions, which are given by the associated eigenvalues). The method is therefore equivalent to Principal Component Analysis where all components are kept. This tool was originally introduced by and then by to estimate the dimension of ensemble of forecasts (“E dimension”), reducing the information in the covariance matrix to one number. Other works followed in that direction. On the other hand, the full eigenvalue spectrum of the covariance of ensemble members was used by in an idealized model.

Some of the authors who developed the “E dimension” have also later used the information in the eigenvectors of the ensemble covariance matrix, by using the fraction of error explained by the eigenvectors as a metric to evaluate ensembles of forecasts .

The covariance matrix is largely singular since the ensemble members have 320² = 102 400 components, while the ensembles have only 50 members. Actually, the covariance matrix has only 49 non-zero eigenvalues because it is constructed out of the residual vectors vi, which satisfy ∑ivi = 0 (and it was supposed that this is the only relationship between them). The eigenvalues of the covariance matrix are denoted λi, and the eigenvectors ui are also called perturbation modes (PM). Example of eigenvectors of covariance matrices are shown in the Supplement (Figs. S5 to S12).

Panels (A) and (B) of Fig.  display the eigenvalues λi of the covariance matrix for both models, averaged over events. A few eigenvalues largely dominate the others, especially in convective STEPS ensembles (panel A). To investigate the geometry of these ensembles, the residual vectors vi of the members with respect to the ensemble mean were considered (see Eq. ). There are roughly speaking two different geometries that could explain this: (a) the vi have comparable norms but they align along the directions of the first eigenvectors of the covariance matrix or (b) the vi are each pointing in different directions but some of them have much larger norms. In order to determine which of the two geometries is actually realized in this case, the cosine cij of the angle between the residual vector vi and uj was computed for all i and j. It is computed as 7cij=vi⋅uj|vi| since the eigenvector uj has a unit norm.

Figure  represents the histograms of values of cij for j=1,2,10 and 15, for a lead time of 15 min. In convective STEPS ensembles (panel A), the distributions of ci1 and ci2 contain much higher values than the distributions of ci10 and ci15, meaning that the vi project much more on the first and the second eigenvectors than on the 10th and the 15th. This is also the case for stratiform STEPS ensemble, albeit to a lesser extent. The conclusion is that the vi in these ensembles align along the first eigenvectors (option a).

The evolution of eigenvalues λi of the covariance matrix for stratiform events is comparable for STEPS and LDCast: the eigenvalues increase steadily following power laws of the lead time (Fig. , panels A and B). This is also the case for convective LDCast ensembles. These power laws should be connected with the theoretical time that an error, initially located at small scales, needs to reach a given larger scale. Indeed, the fact that the error grows as power law of the time is consistent with the limited predictability at the nowcasting scales. This limited predictability is related to the structure of the energy spectrum of atmospheric fields at these scales, following closely a -5/3 power law .

Panels (C) and (D) of Fig. 3 show, for two different lead times, the spectrum of the normalized eigenvalues. These are the eigenvalues divided by the highest one in the ensemble at that lead time, and they allow to better understand the relative sizes of the perturbation modes. Consistent with the panels (A) and (B), it appears that LDCast ensembles have generally more homogeneous eigenvalue spectra (i.e. have more eigenvalues of similar size) than STEPS ensembles. Convective LDCast ensembles exhibit more homogeneous eigenvalue spectra than stratiform LDCast ensembles, especially at long lead times. As already noted from panel (A), it is also obvious that STEPS ensembles are largely dominated by a few eigenvalues: convective STEPS ensembles are dominated by ∼ 10 eigenvalues, while stratiform STEPS ensembles are dominated by ∼ 5 eigenvalues. However, the relative size of higher-order eigenvalues (order > 5 in stratiform ensembles, order > 10 in convective ensembles) is larger in stratiform STEPS ensembles than in convective STEPS ensembles.

The power spectra of the eigenvectors of the covariance matrix are displayed in Fig. , for different lead times, for stratiform and convective events. The color represents the associated eigenvalues λi. Overall, the perturbation modes gradually develop over larger scales, while the spectrum profile remains the same for small scales. This shows that the directions of perturbation evolve in phase space with the lead time: at long lead times, the perturbations are vectors pointing in directions corresponding to larger scales than at the beginning of the nowcast.

It also appears that eigenvectors with higher eigenvalues have more power at large scales. This leads to a hierarchy: perturbations with larger amplitudes affect the large scales more. This is in agreement with the fact that, for a given scale, there is a maximum size for the error (see Fig. ), so that once a perturbation reaches the saturation value at a given scale, it propagates to larger scales.

For both STEPS and LDCast, the eigenvectors of stratiform ensembles (especially the eigenvectors associated with the highest eigenvalues) have more power at larger scales than those of convective ensembles. This means that the perturbations contained in stratiform ensembles are at larger scales than those in convective ensembles.

The curves standing out in the top row (A) of Fig.  for short and intermediate lead times are those of the power spectra of the eigenvectors that largely dominate the STEPS ensembles in convective cases (panel A of Fig. ). Similar power spectra are also present in stratiform STEPS ensembles.

Note that the eigenvectors have unit norm so that values in a power spectrum are only meaningful relative to other values in that power spectrum. For example, the spectra of LDCast eigenvectors at small scales seem to decrease with lead time, but this is not due to an absolute decrease in the small-scale variability. It is rather due to a decrease in the small-scale variability relative to the large-scale variability in the same vectors.

This section presents results assessing the skill of ensemble members to spatially represent the error. The focus is particularly on evaluating whether the ensembles contain any spatial information beyond that present in the ensemble mean.

Two metrics are considered. The first is the Perturbation versus Error Correlation Analysis (PECA) score , which is called cos⁡γ(e,p) here in order to emphasize its geometric interpretation in phase space. Indeed, it is the cosine of the angle between the error e and its projection p on the subspace spanned by the residual vectors vi. Using the fact that the eigenvectors of the covariance matrix are orthonormal and span the same subspace as the vi, p can be written as 8p=∑i(ui⋅e)ui and its norm is |p| = ∑i(ui⋅e)2. So cos⁡γ(e,p) can be computed in terms of the scalar product between e and p as 9cos⁡γ(e,p)=e⋅p|e||p|, or in terms of the ratio between the norm of p and the norm of e: 10cos⁡γ(e,p)=|p||e| since e⋅p = ∑i(ui⋅e)2 = |p|2. These expressions show that cos⁡γ(e,p) indicates the extent to which the error is captured by the ensemble members.

This metric is indeed equal to the PECA score, except that the error and the perturbations are computed with respect to a control run for the PECA score. Apart from that, minimizing the L2-norm of e-∑iαivi amounts to constructing the projection of e on the space spanned by the vi.

The same quantity was also introduced independently by and as the Explained Variance (EV) to evaluate the efficiency of data assimilation schemes to provide initial conditions leading, under the perfect model hypothesis, to ensembles of forecasts capturing the error. Finally, also considered that metric to estimate the ability of breeding vectors of a realistic model to capture the error growth in a convective situation.

The second metric considered is the Fraction Skill Score (FSS), which was introduced by as a metric for the spatial accuracy of a deterministic forecast. With respect to the mean square error, it mitigates displacement errors by comparing the forecast and the observation over neighborhoods of different scales. recently compared different versions of the FSS for ensemble verification, and recommended the “probabilistic FSS” proposed in , which is the one used in the current work.

The two metrics are computed for STEPS and LDCast ensembles and compared to those of three types of surrogate ensembles. Surrogate ensembles are generated as follows:

HIST ensembles: for each member xi in the original ensemble, a HIST surrogate zi is generated by shuffling all the pixel values of xi (xi and zi have thus the same histogram of values);