vp9: Disallow svc config for 2-pass, remove legacy code
In VP9, SVC (spatial/temporal layering) is only supported for one-pass
encoding, and two-pass encoding was already disallowed in ctrl_set_svc()
(VP9E_SET_SVC) in commits 6c5123a3a and fe1e27249.
However, validate_config() did not reject configurations with
ss_number_layers > 1 or ts_number_layers > 1 when g_pass !=
VPX_RC_ONE_PASS, and still retained the legacy multi-layer branch in
the VPX_RC_LAST_PASS stats validation. When vpx_codec_enc_init() was
called with ss_number_layers > 1, g_pass == VPX_RC_LAST_PASS, and a
stats buffer where layer 0 had >= 2 packets but total n_packets <
ss_number_layers, computing the EOS packet pointer
(buf + n_packets - cfg->ss_number_layers + i) underflowed before the
start of rc_twopass_stats_in.buf, causing an out-of-bounds heap read.
Disallow (cfg->ss_number_layers > 1 || cfg->ts_number_layers > 1) when
cfg->g_pass != VPX_RC_ONE_PASS in validate_config(), and remove the
unreachable legacy two-pass multi-layer code in validate_config(),
vp9_create_compressor(), and vp9_svc_layercontext. Because multi-layer
two-pass configurations are rejected and the legacy two-pass multi-layer
stats validation code is removed, no separate bounds check in that
removed code path is needed.
Updated EncodeAPI.SvcTestInvalidInputs to verify two-pass multi-layer
init is rejected.
Bug: 574106931
Change-Id: Ic612aefa415e05af3c9167a9537e6d38143084da
5 files changed