vp9_set_roi_map: reject segment ids >= MAX_SEGMENTS
vp9_set_roi_map() checks the dimensions and the per-segment values but
not the contents of the map. vpx_roi_map_t documents the ids as 0-7
for VP9, but a larger value is copied to the segmentation map and
later used to index per-segment arrays while encoding, for example
no_pred_segcounts[] in count_segs(). VP8 ignores out-of-range ids when
it encodes.
Return VPX_CODEC_INVALID_PARAM if a byte of the map is out of range.
The check is done after the early return that disables segmentation,
so a map that is not going to be used is not validated. Add a test.
I used an AI model to help audit this code and prepare the change and
its test. I reviewed the diff, and the tests were run locally with
ASan and UBSan.
Change-Id: I436b191bb1087491cf187be3a6d8e9e7bac2b9f4
2 files changed