qis.adjust_returns_with_ar

qis.adjust_returns_with_ar(returns, ar_order=2, span=20, mean_adj_type=MeanAdjType.EWMA, warmup_period=10, max_value_for_beta=0.75, min_value_for_beta=-0.25, apply_ewma_mean_smoother=True, non_negative=False, non_negative_tol=0.0, return_diagnostics=False, insufficient_data=InsufficientData.NAN, check_denominator=True, validate_inputs=True)[source]

Unsmooth observed returns with a rolling EWMA AR(q) beta vector.

Single implementation for any lag order. The observed return is regressed on its first ar_order lags with a time-varying EWMA beta vector, then inverted period-by-period:

r_obs_t = sum_i b_i,t * r_obs_{t-i} + eps_t, i = 1..q r_true_t = (r_obs_t - sum_i b_i,{t-1} * r_obs_{t-i}) / (1 - sum_i b_i,{t-1})

Betas are the lagged (shift-1) estimates, aligning with the lagged returns in the numerator and avoiding look-ahead. The lags are fitted JOINTLY via the rolling cross-moment inverse b_t = Sxx_t^{-1} Sxy_t (compute_ewm_xy_beta_tensor); for q >= 2 this is required because the lags of an autocorrelated series are mutually correlated, so independent univariate fits would double-count the shared autocorrelation. For q = 1 it reduces to the scalar rolling EWMA beta.

Denominator stability is governed by the smoothing total theta_sum = sum_i b_i, so clipping is applied to the SUM (not to each beta): if theta_sum leaves [min_value_for_beta, max_value_for_beta] the beta vector is rescaled by boundary / theta_sum, preserving direction and bounding the vol-inflation factor 1 / (1 - theta_sum). For q = 1 this is identical to clipping the single beta, so the bounds carry the same meaning as in the legacy AR(1) path.

With the default arguments the output is identical to the pre-guard implementation, including the all-NaN column returned for a series shorter than min_obs_for_ar_unsmoothing(ar_order, warmup_period).

Parameters:

  • returns (DataFrame) – Observed period returns (rows = dates, columns = assets).

  • ar_order (int) – Lag order q (1 = rolling AR(1); 2 = rolling AR(2); q > 2 ok).

  • span (int) – EWMA span for the rolling beta. Default 20. A q >= 2 fit needs more effective observations than AR(1) — for quarterly data use ~40 (10y); for monthly ~24-36.

  • mean_adj_type (MeanAdjType) – How to demean returns/lags before beta estimation.

  • warmup_period (int | None) – Initial periods masked before the first valid beta. Default 10; raise to ~16 for q >= 2 (the q x q estimate is noisier early). The underlying tensor also self-guards via its own warmup.

  • max_value_for_beta (float | None) – Upper bound on theta_sum (sum of betas). Bounds the inversion denominator away from zero. Pass None to disable. For q = 1 this is the upper beta cap (legacy semantics).

  • min_value_for_beta (float | None) – Lower bound on theta_sum. Permits genuine mean-reversion / symmetric treatment of estimation noise. Pass None to disable; pass 0.0 for one-sided clipping (legacy behaviour).

  • apply_ewma_mean_smoother (bool) – If True, apply an extra EWMA smoother to each beta series after the sum-clip.

  • non_negative (bool) – If True, constrain every lag beta to be non-negative via per-period NNLS on the EWMA normal equations (mirrors the GLM theta >= 0 smoothing weights). A spurious negative higher-order lag self-reduces: for q = 2 the fit collapses to the univariate AR(1) coefficient rather than an oscillating b1 > 1, b2 < 0 solution. With non_negative=True the theta_sum lower bound (min_value_for_beta) is moot since theta_sum >= 0 by construction; the upper cap still applies. Applied only to flagged (smoothed) assets whose lag-1 is positive, so the b1 >= 0 constraint does not bind in practice.

  • non_negative_tol (float) – Deadzone width for the non-negativity. A lag is forced to zero only when its unconstrained coefficient is below -non_negative_tol; lags in [-tol, 0) keep their small negative value. tol=0 is strict non-negativity. A small tol (~0.1) catches a genuinely destabilising negative higher lag (e.g. an oscillating b2 ~ -0.45) while leaving the negative noise of a near-zero lag untouched, removing the upward bias that strict non-negativity puts on funds whose true higher-lag coefficient is ~0. Ignored when non_negative is False.

  • return_diagnostics (bool) – If True, return (unsmoothed, betas, r2) where betas is the applied per-period theta_sum (sum of betas; equals the single beta for q = 1) and r2 is the EWMA R^2 of the fit.

  • insufficient_data (InsufficientData) – Policy for columns that come out entirely NaN, which happens when the frame has fewer than min_obs_for_ar_unsmoothing rows or when a column is itself all NaN. Default NAN preserves the historical silent all-NaN column. RAISE reports the columns, their finite input counts and the row count. PASSTHROUGH returns their observed returns unchanged, with a theta_sum of 0.0 and an r2 of NaN in the diagnostics.

  • check_denominator (bool) – If True, raise when the inversion denominator 1 - theta_sum is non-positive. Inert under the default max_value_for_beta=0.75, which keeps the denominator at or above 0.25. Pass False to restore the previous unchecked division.

  • validate_inputs (bool) – If True, validate types, index and parameter ranges.

Returns:

Unsmoothed returns (DataFrame, input shape). If return_diagnostics, the tuple (unsmoothed, betas, r2).

Raises:

  • ValueError – If validate_inputs and an argument is invalid.

  • ValueError – If insufficient_data is RAISE and any column is too short.

  • ValueError – If check_denominator and 1 - theta_sum is non-positive.

Return type:

DataFrame | Tuple[DataFrame, DataFrame, DataFrame]