vp9: fix to integer overflow test

failure for the 16k test: issue introduced
in: c29e637283

Bug: b/329088759, b/329674887, b/329179808

Change-Id: I88e8a36b7f13223997c3006c84aec9cfa48c0bcf
(cherry picked from commit 19832b1702d5b0adf616a0e080abd5207c8445b5)