vp8: Validate ss_number_layers and factor den in validate_config

Add range checks in vp8 validate_config() for ss_number_layers
(1..VPX_SS_MAX_LAYERS) and kf_frame_max_boost_first_factor.den,
and perform all scalar config range checks before dereferencing
rc_twopass_stats_in.buf when g_pass == VPX_RC_LAST_PASS.

Add unittest.

Bug: 564326761
Change-Id: I57edc6e3f8b51caf666bc40ea20352c2b2b01e64