tree 39049a6655532b018a51d9ef3700798e41780e55
parent 2190262902561e5e57b088235f6e7302003f1c3b
author Ross Zwisler <zwisler@google.com> 1729007073 -0600
committer Chromeos LUCI <chromeos-scoped@luci-project-accounts.iam.gserviceaccount.com> 1729170639 +0000

make_dev_ssd.sh: avoid page cache aliasing

Per http://b/307454713 we noticed that sometimes an `fflash` operation
would leave a device in a bad state where it would no longer boot.  It
turns out that this was due to a page cache aliasing issue that caused
the kernel partition to become corrupt.  That issue worked like this on
my asurada system:

1. fflash writes a new kernel image to slot A using /dev/mmcblk0p2

2. Without rebooting, fflash then executes make_dev_ssd.sh to remove
   rootfs verification on slot A.

3. make_dev_ssd.sh reads the kernel image from slot A, with the intent
   of modifying it and writing it back.  However, instead of reading the
   image from /dev/mmcblk0p2, it reads the kernel image from
   /dev/mmcblk0 using offset and length arguments, i.e.:

   dd if=/dev/mmcblk0 of=data bs=512 skip=69 count=65536

   You can see that offset & length via `fdisk`:

   # fdisk -l /dev/mmcblk0
   Device             Start      End  Sectors  Size Type
   /dev/mmcblk0p2        69    65604    65536   32M ChromeOS kernel

4. The page cache for /dev/mmcblk0p2 was still in the process of writing
   back its dirty data, so the data that make_dev_ssd.sh gets from its
   `dd` operation on /dev/mmcblk0 is a mix of old stale data and new
   data.  It reads that data to a file, updates it to remove rootfs
   verification, and then writes it back again using the parent block
   device, offset and length.

5. We now have a corrupt kernel image in slot A, and will fail to boot
   when `fflash` tries to reboot the system.

The root of this issue is that the partition /dev/mmcblk0p2 and the
parent device /dev/mmcblk0 can have separate page cache entries for the
same disk blocks.  We can work around this issue by making sure that the
writeback from the update operation is complete and that we've cleared
any stale, clean cache blocks before and after we do the make_dev_ssd.sh
update to remove rootfs verification.

We could also potentially fix this by updating make_dev_ssd.sh use the
/dev/mmcblk0p2 partition, but this seems more brittle because it
requires us to keep all update utilities (fflash, `cros flash`) in sync
with how they access block devices.  The current solution tries to make
make_dev_ssd.sh updates atomic so they can work no matter how other
tools use the disk.

BUG=b:307454713
BRANCH=none
TEST=running these two operations in a loop:
  cros flash --no-stateful-update --no-reboot $DUT $DISK_IMAGE
  ssh $DUT "/usr/share/vboot/bin/make_dev_ssd.sh -d \
	  --remove_rootfs_verification --partitions  $PARTITION"
I was able to consistently recreate the cache aliasing issue on my
asurada devices in about 5 minutes.  With this fix I was able to run
that same test on 2 devices overnight without any issues.

Change-Id: I41c96534ec8f69e5968af27bd24fa2d470422d7d
Signed-off-by: Ross Zwisler <zwisler@google.com>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/vboot_reference/+/5934991
Reviewed-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Benjamin Gordon <bmgordon@chromium.org>
Reviewed-by: Raul Rangel <rrangel@chromium.org>
(cherry picked from commit c5af1fd8490d07d28ab178364e6452da748cc320)
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/vboot_reference/+/5938633
Commit-Queue: Allen Webb <allenwebb@google.com>