sqlite: Upgrade to 3.42.0 First release to include https://sqlite.org/src/info/c4d083a3, so no need to cherry-pick that in again... though it includes a couple of other cherry-picks. See https://crrev.com/c/4559464 Roll src/third_party/sqlite/src/ f6752b7ed..29bc0dd7a (403 commits) https://chromium.googlesource.com/chromium/deps/sqlite.git/+log/f6752b7ed1fe..15f6d881efbc $ git log f6752b7ed..15f6d881e --date=short --no-merges --format='%ad %ae %s' 2023-05-23 asully sql: Cherry-pick in two commits to version 3.42.0 2023-05-17 asully sqlite: Amalgamations for release 3.42.0 2023-05-16 Version 3.42.0 2023-05-16 Update the tool/mkmsvcmin.tcl script so that it prints a one-line message to explain what it has done. 2023-05-15 Fix a C++-style variable declaration in the generate_series() extension. 2023-05-15 Increase the stack size available to fuzzcheck to 8MB on Windows. 2023-05-15 Dan Kennedy Limit the number of nested NOT nodes in an fts5 expression to 256. 2023-05-15 Add the --wait command-line option to fuzzcheck. 2023-05-15 larrybr Make generate_series() correct on ones complement ALUs and acceptable to UBSAN. 2023-05-15 As evidenced by [forum:/forumpost/f3f546025a|forum post f3f546025a], the new RIGHT JOIN related restriction on the push-down optimization implemented by [da3fba18742b6e0b] also needs to apply to the automatic index (a.k.a. hash-join) optimization and to the Bloom filter optimization. Computation of the restriction is now moved into the sqlite3ExprIsSingleTableConstraint() routine. 2023-05-15 Simplify the interface to constructAutomaticIndex(). 2023-05-13 Dan Kennedy Fix windows-specific problems in test scripts sessionnoop2.test and sessionrowid.test. 2023-05-13 Dan Kennedy Remove unnecessary call to sqlite3_dbdata_init() from shell.c. 2023-05-13 Dan Kennedy Add an "#ifndef SQLITE_HAVE_SQLITE3R" directive to shell.c.in, to avoid including the recovery code a second time if the shell is being compiled with sqlite3r.c. 2023-05-13 larrybr Mention that sqlite3_mutex_free() accepts NULL as a no-op. 2023-05-13 mistachkin Fix compilation issue seen with MSVC. 2023-05-12 Fix harmless compiler warning caused by [0772ddf56713d013]. 2023-05-12 Dan Kennedy Add multi-threaded test for "INSERT ... RETURNING ..." and SQLITE_BUSY errors. 2023-05-12 Make the regexp extension more robust against OOM errors during compilation of the recognizer engine. [forum:/forumpost/f50aecd5e8|Forum post f50aecd5e8]. 2023-05-12 larrybr Fix unlikely strange result of CLI edit(). 2023-05-12 Ensure the _O_U16TEXT is defined for Windows builds of the CLI. [forum:/forumpost/c80aa942dc6bf2|Forum post c80aa942dc6bf2]. 2023-05-12 Remove a stray CR from end of line in a source file. 2023-05-12 Fix harmless typos in comments. 2023-05-12 Ensure that all prepared statements are expired and forced to reprepare whenever the database text encoding changes. [forum:/forumpost/43cbfc4d23|Forum post 43cbfc4d23]. 2023-05-11 Add [/info/cec6a06508239c09e363?ln=5525-5531|restriction (5)] to the omit-noop-join optimization. Fix for the issue reported by [forum/forumpost/49f2c7f690|forum post 49f2c7f690]. 2023-05-11 Fix a minor typo in a comment. No code changes. 2023-05-11 Dan Kennedy Fix the rbu_exclusive_checkpoint=1 option so that it works with a zipvfs database in pass-through mode. 2023-05-10 Enhance the split-sqlite3c.tcl script so that it uses single-character extensions on subordinate source files, even when the number of subordinates exceeds 9. This is not yet needed, but might be soon. 2023-05-10 Fix harmless compiler warnings from the CBMC compiler. 2023-05-10 Rename sqlite3IsTableConstraint() to sqlite3IsSingleTableConstraint() and improve its header comment, in an attempt to make the code easier to reason about. No functional changes - should generate identical machine code. 2023-05-10 Add a missing comma to the documentation for the OP_Jump opcode. No changes to code. 2023-05-10 New restriction on the push-down optimization to prevent an ON or USING clause on the left side of a RIGHT JOIN from being pushed down into a subquery that is on the right side of that RIGHT JOIN. [forum:/forumpost/a7d4be7fb6|Forum post a7d4be7fb6]. Also add comments to describe previously undocumented push-down optimization restrictions. 2023-05-10 larrybr Fix bug in fall-back logic of CLI .clone 2023-05-08 Dan Kennedy Fix recover1.test so that it works with DEFAULT_AUTOVACUUM builds. 2023-05-08 Improved error handling when ".open" fails in the CLI. 2023-05-08 In editPage(), if an attempt is made to edit an empty 64KB page, then take the slow path through rebuildPage(), in order to maintain strict invariants in the editPage() code. [forum:/forumpost/0f37727c0d|Forum post 0f37727c0d]. 2023-05-08 Ensure that the label for a JSON object really is a string and not an array or object that just happens to end with a string. dbsqfuzz 32f62ed8676d0a153dbe756aeae3202d3670fe50. 2023-05-08 Dan Kennedy Add tests for recovering data to a lost-and-found table from a utf-16 db. No code changes. 2023-05-07 It turns out that pagerExclusiveLock() can be called with the lock state already set to RESERVED if the SQLITE_FCNTL_PERSIST_WAL setting is set and a specific sequence of multiple journal mode changes occur. Enhance pagerExclusiveLock() to deal with this. [forum:/forumpost/8130545bc6|Forum post 8130545bc6] 2023-05-05 Fix a comment in the JSON logic. No code changes. 2023-05-05 Dan Kennedy Fix a buffer overrun that could occur when using the format() function to format a very small real value with the "," modifier. 2023-05-05 Dan Kennedy Reduce the maximum depth of nesting in json objects to 1000. 2023-05-05 Dan Kennedy Do not attempt to run json502.test with SQLITE_OMIT_VIRTUALTABLE builds. 2023-05-05 Updates to compile-time option usage and especially to the compile-time option test script in tool/omittest.tcl. 2023-05-05 Fix the pager2-3.1 test case so that it does not run if a required dependence (shared-cache) is omitted from the build. Fix for [forum:/forumpost/7a028538a7|forum post 7a028538a7]. 2023-05-05 Dan Kennedy Allow virtual tables to prepare writes to shadow tables in defensive mode from with xRelease() callbacks - even if the xRelease() is invoked from within sqlite3_finalize(). 2023-05-05 Improved formatting of comments on SQLITE_CONFIG_xxx defines as those comments are used by documentation. 2023-05-04 Add support for the comma (,) modifier to %f formats in the format() function. 2023-05-04 Fix a bug in cursor hints that can cause references to tables that have not been opened. Cursor hints are intended for use by COMDB2 only and should not appear in production builds, so this should not be a factor for the vast majority of users. 2023-05-03 Add another assert() to FTS5 to demonstrate that a pointer is never NULL. 2023-05-03 Dan Kennedy Add an assert() to fts5_config to ensure that a potential OOM is being handled correctly. (...) 2023-03-03 Enhance PRAGMA integrity_check so that it can detect when there are extra bytes at the end of an index record, which might cause OP_IdxRowid to malfunction. dbsqlfuzz c1aa3986534d5feab8d21f28b3c1712df2ef358ba. Test case in TH3. 2023-03-03 Dan Kennedy When it is known when preparing a statement that X cannot be NULL, transform the expression (X IS NULL) to integer value 1 instead of 'true'. This is because under some circumstances, "Y IS TRUE" may not be equivalent to "Y IS 1". 2023-03-03 Do not use an expression index on a generated column if generated column has the wrong affinity. dbsqlfuzz 65f5eb57f8859344d5f1f33e08c77ee12960ed83 2023-03-03 Dan Kennedy Remove unnecessary call to sqlite3_dbdata_init() from shell.c.in. 2023-03-02 When flattening the right operand of a LEFT JOIN (check-in [41c27bc0ff1d3135]), ensure that the OP_IfNullRow opcode does not NULL-out a subquery result that was computed within OP_Once. This fixes the problem problem reported by [forum:/forumpost/402f05296d|forum post 402f05296d]. 2023-03-02 stephan Resolve a parallel build timing issue when building sqlite3.c/h from ext/wasm. For the time being, do not add sqlite3_wasm_extra_init.c to fiddle.wasm because it can cause duplicate definitions of extensions which are already built into the shell (a better resolution for this conflict is pending). No longer add sqlite3_wasm_extra_init.c to speedtest1.wasm because it's useless there. 2023-03-02 stephan Rename some vars in the ext/wasm makefiles for consistency's sake. 2023-03-01 When flattening a view that is the right operand of a LEFT JOIN, using the optimization of check-in [41c27bc0ff1d3135], always insert the TK_IF_NULL_ROW expression nodes, even for TK_COLUMN expressions, as the TK_COLUMN might be a column from an outer query and hence still need to be NULLed out. This fixes the problem described by [forum:/forumpost/26387ea7ef|forum post 26387ea7ef]. 2023-03-01 Follow-up to [bbaf1f2eb1e1637b]: Make sure subtypes do not cross a subquery boundary even if the function that returned the value with a subtype is buried down inside a larger expression. This fixes a problem identified by [forum:/forumpost/37dd14a538|forum post 37dd14a538]. 2023-03-01 Do not attempt to apply the count-of-view optimization to a CTE. dbsqlfuzz ef8623915d843b150c159166ee4548c78cc6895a 2023-02-28 Activate SQLITE_DBCONFIG_STMT_SCANSTATUS in fuzzcheck. 2023-02-28 Updates to speedtest1.c and the speed-check.sh test script so that they work with the new SQLITE_DBCONFIG_STMT_SCANSTATUS control. 2023-02-28 Dan Kennedy Change the name of SQLITE_DBCONFIG_STMT_SCANSTATS to SQLITE_DBCONFIG_STMT_SCANSTATUS. 2023-02-28 Dan Kennedy Add an sqlite3_db_config() option - SQLITE_DBCONFIG_STMT_SCANSTATS - for enabling and disabling the collection of sqlite3_stmt_scanstats() statistics in SQLITE_ENABLE_STMT_SCANSTATUS builds. Collection of statistics is disabled by default. 2023-02-28 Only use a Bloom filter on an automatic index if one or more of the key columns in the index can take on non-TEXT values. 2023-02-28 In the Bloom filter optimization, hash all strings and blobs into the same value, because we do not know if two different strings might compare equal even if they have different byte sequences, due to collating functions. Formerly, the hash of a string or blob was just its length. This could all be improved. Fix for the issue reported by [forum:/forumpost/0846211821|forum post 0846211821]. 2023-02-28 When an automatic index creates a Bloom filter, show that in the EXPLAIN QUERY PLAN output. 2023-02-28 Performance optimization in the varint decoder for the cell parser. 2023-02-28 Dan Kennedy Improve the error message generated by the recovery extension if it is used with a non-SQLITE_ENABLE_DBPAGE_VTAB build. 2023-02-27 Minor performance optimization in the computation of an affinity string for an index. 2023-02-27 Further to [46639f682975dac6], the parameters of a table valued function that is on the left side of a RIGHT JOIN do not need to be in the ON clause. Add new test cases and tags to associated test cases with the code. 2023-02-27 stephan Minor doc tweak for the previous commit. 2023-02-27 stephan Extend wasm build to enable inclusion of client-custom C code, initialized via the SQLITE_EXTRA_INIT mechanism, per discussion in [forum:1e1c04f3ed1bc96b|forum post 1e1c04f3ed1bc96b]. 2023-02-26 When a table-valued function appears as the right table of a RIGHT JOIN, the argument constraints on the table-valued function should be considered part of the ON clause of the RIGHT JOIN. Fix for the problem reported by [forum:/forumpost/422e635f3beafbf6|forum post 422e635f3beafbf6]. Test cases in TH3. Possibly related to the enhancement at [501609eddf2a46d5]. 2023-02-26 In the [/info/7c2d3406000dc8ac|omit-unused-subquery-columns optimization], be sure to remove the EP_Skip and EP_Unlikely flags from the result set expressions that get nulled-out. dbsqlfuzz bf1d3ed6e0e0dd8766027797d43db40c776d2b15. 2023-02-25 Split out a new variant of cellSizePtr() that applies only to leaf pages, for a small performance increase. 2023-02-25 Remove an unnecessary and redundant corruption checks in defragmentPage(). 2023-02-25 larrybr Revert (mostly) to earlier CLI response to -echo and SQL at the command line. 2023-02-25 Performance optimization on sqlite3_vsnprintf(). 2023-02-24 Add the "on" option to the ".log" command in the CLI. Allow ".log on" and ".log off" even in --safe mode. Enable the .log command for fiddle builds. 2023-02-24 In the CLI, do not emit warnings about the incorrect usage of sqlite3_config() in Fiddle. 2023-02-24 Back out the SQLITE_CONFIG_URI option from anytime-config. 2023-02-24 Adjust testrunner.tcl to make use of the new number_of_cores TCL command available in testfixture, and to use no more than half the available cores. 2023-02-24 Add the "number_of_cores" TCL command to the testfixture. 2023-02-24 Fix an incorrect optimization that was attempted as part of check-in [18de3a8e6b431a07]. 2023-02-23 Omit unnecessary branches from the enhanced floating-point conversion logic. 2023-02-23 Increased precision of floating-point to decimal conversions when the floating point value has no fractional part. [forum:/forumpost/d1387c3979c7f557|Forum post d1387c3979c7f557] 2023-02-23 Fix harmless compiler warnings and a code indentation error. 2023-02-23 Allow the sqlite3_config() interface to be invoked at any time for a few choosen options: SQLITE_CONFIG_LOG, SQLITE_CONFIG_URI, and SQLITE_CONFIG_PCACHE_HDRSZ. This list will likely change before release. 2023-02-23 Provide -DHAVE_LOG2=0 and -DHAVE_LOG10=0 compile-time options for use on systems that lack the log2() and log10() standard math library routines, to cause SQLite to substitute its own alternatives. 2023-02-22 Enable the count-of-view optimization by default. Enhancement request [eaed8e36ce888f1e]. 2023-02-22 Update the version number to 3.42.0 to begin the next development cycle. 2023-02-22 Remove the push-down optimization restriction that was added by check-in [1ad41840c5e0fa70] because it is no longer needed after the enhancements to compound query processing in check-in [27655c9353620aa5] This resolves the performance regression reported by [forum:/forumpost/bcc4375032|forum post bcc4375032]. 2023-02-21 stephan Remove some unnecessary bind() calls in JS code. 2023-02-21 stephan Unconditionally disable -sSTRICT_JS flag in wasm build because it no longer works with emsdk 3.1.31+. 2023-02-16 A few simple test cases for the omit-unused-subquery-column optimization. 2023-02-16 Provide an optimization-disable mask for this optimization. Do not do the optimization if the subquery is an aggregate or is distinct, but allow it to be an ephemeral subquery. Do not omit columns that are used in the ORDER BY of the subquery. 2023-02-16 Do not perform the omit-unused-subquery-columns optimizations on a subquery that is DISTINCT, as that can lead to incorrect results. 2023-02-15 Fix the subquery result column NULL-ifier so that it correctly handles subquery columns past the 63rd column. 2023-02-15 Do not compute result columns of subqueries that are never used. Make those columns NULL instead. This optimization potentially resolves the enhancement request described by [/tktview/baa5bb76c35a124c|ticket baa5bb76c35a124c]. Created with: roll-dep src/third_party/sqlite/src Fixed: 1446406, 1447396 Change-Id: Ib3c320927e6a1974961a3a1e61daa121cb368692 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4545677 Auto-Submit: Austin Sullivan <asully@chromium.org> Reviewed-by: Evan Stade <estade@chromium.org> Commit-Queue: Evan Stade <estade@chromium.org> Cr-Commit-Position: refs/heads/main@{#1148241} NOKEYCHECK=True GitOrigin-RevId: c1d3b72336a4fa5198cd02ba373bddde76796a44
This is the top folder for Chromium's SQLite. The actual SQLite source is not in this repository, but instead cloned into the src directory from https://chromium.googlesource.com/chromium/deps/sqlite.
The directory structure is as follows. Files common to all third_party projects (ex. BUILD.GN, OWNERS, LICENSE) are omitted.
src/ The Chromium fork of SQLite (cloned via top level DEPS file).scripts/ Scripts that generate the files in the amalgamations in src/.sqlite.h The header used by the rest of Chromium to include SQLite. This forwards to src/amalgamation/sqlite3.hfuzz/ Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build.SQLite amalgamations are committed to the SQLite Chromium repository (in src), but are created by a script that lives in the Chromium repository. This is because the configuration variables for building and amalgamation generation are shared.
There are two amalgamations:
CLs in this repository cannot be submitted through the commit queue (ex. CQ+2), because there is no commit queue / try bot support for this repository. Please use the “Submit” button (in Gerrit's 3-dot menu on the top right) to submit CLs in this repository instead.
SQLite should be upgraded as soon as possible whenever a new version is available. This is because new versions often contain security and stability improvements, and frequent upgrades allow Chromium to have minimal cherry-pick diffs when requesting investigation for SQLite bugs discovered by Chromium Fuzzers. New versions may be viewed at https://www.sqlite.org/news.html, and bugs for these upgrades may look like this example.
Historically, Chromium fuzzers often find issues within 2 weeks after upgrading to new SQLite versions. Avoid upgrading SQLite within 1-2 weeks of a Chromium branch point to allow fuzzers time to run. However, if the new SQLite release contains known security or stability fixes, upgrade once available and monitor fuzzers more closely.
SQLite version upgrades tend to be extremely large changes (example), for which the diffs are not possible to thoroughly review.
Note SQLite tags all releases version-<release number>, e.g. version-3.40.0. The Chromium project prefixes all tags/branches with “chromium-”, e.g. chromium-version-3.40.0.
Create new release branch
Use the SQLite git commit hash for the release, found at sqlite/tags, when creating a new release branch. For example, “562fd18b9dc27216191c0a6477bba9b175f7f0d2” corresponds to the 3.31.1 release. The commit is used instead of the tag name because we do not mirror the SQLite tags along with the commits.
Create the branch at Gerrit/branches.
Note: To create a release branch, you must be listed as a member in the sqlite-owners Gerrit group
Checkout the new Chromium release branch.
Get the version from the README.chromium.
cd third_party/sqlite/src # from //chromium/src git fetch origin export VERSION=3.40.0 git checkout -b chromium-version-$VERSION \ --track origin/chromium-version-$VERSION
Generate and commit the SQLite amalgamations.
./../scripts/generate_amalgamation.py git add amalgamation amalgamation_dev git commit -m "Amalgamations for release $VERSION"
Run local tests.
Follow steps in Running Tests below to execute all verifications and tests.
Upload the new release branch for review.
git cl upload
Roll the Chromium DEPS file.
Once review above has merged:
chromium/src/DEPS file to reference that new commit hash.roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
Sometimes critical fixes land in SQLite's master, but are not yet in a release. This may occur when other SQLite embedders find critical security or stability issues that SQLite authors then fix, but are often detected by Chromium ClusterFuzz as well.
If you're triaging a ClusterFuzz bug, an internal playbook on how to triage and fix ClusterFuzz bugs is available at go/sqlite-clusterfuzz-bug-process.
If changes need to be brought into the current release branch, please do the following:
Checkout the current release branch.
Get the version from the README.chromium.
export VERSION=3.40.0 cd third_party/sqlite/src # from //chromium/src git checkout -b chromium-version-$VERSION \ --track origin/chromium-version-$VERSION
Cherry-pick the change
Git can be used to cherry pick upstream changes into a release branch but the sqlite_cherry_picker.py script is preferred. This script automates a few tasks such as:
Cherry-pick the commit:
../scripts/sqlite_cherry_picker.py <full git or fossil commit hash>
If there is a conflict that the script cannot resolve then, like git cherry-pick, the script will exit and leave you to resolve the conflicts. Once resolved run the script a second time:
../scripts/sqlite_cherry_picker.py --continue
If you have access to the SQLite fossil commit hash, and would like to map this to the corresponding git hash, you can use GitHub search. As SQLite‘s git repository’s commits include the fossil hash, you can search for the fossil hash, using the following query with the fossil commit hash appended (example search): https://github.com/sqlite/sqlite/search?type=commits&q=
If the cherry-picking script is unable to cherry-pick a commit, like in https://crbug.com/1162100, manually apply the change from a SQLite or git, in //third_party/sqlite/src's files modified in the SQLite tracker, like at https://sqlite.org/src/info/a0bf931bd712037e. From there, run ../scripts/generate_amalgamation.py to propagate these changes over to the amalgamation files. sqlite_cherry_picker.py should generally be preferred, as it updates hashes and simplifies tracking.
Run local tests.
Follow steps in Running Tests below to execute all verifications and tests.
Upload cherry-picked change (with amalgamations) for review.
If the relevant bug is a security bug, make sure that the reviewers are cc‘ed. Otherwise, they may not know what/why they’re reviewing.
```sh git cl upload ```
Update the Chromium DEPS file.
Once review above has merged, roll the chromium/src/DEPS file to reference that new commit hash.
roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
Build all desktop targets:
Check that extract_sqlite_api.py added “chrome_” to all exported symbols. Only “_fini” and “_init” should be unprefixed, but are conditionally exported by the linker and may be omitted.
autoninja -C out/Default nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'
out/Default/sql_unittests
third_party/blink/tools/run_web_tests.py -t Default storage/websql/
This is one of the SQLite test suites. They take approximately 3 minutes to build and run on a fast workstation.
Note: Tests currently fail both locally and on Chromium release branches. They fail on release branches because some tests rely on SQLite databases (binary files) which are committed to the source and are likely not merged down when cherry picked. It is safe to ignore these errors which should be reasonably easy to identify based on the cherry picked upstream changes. Until these tests are fixed, it is safe to ignore these tests when running SQLite test suites.
cd third_party/sqlite # from //chromium/src ./scripts/generate_amalgamation.py --testing make --directory=src test | tee /tmp/test.log
Show tests with errors:
egrep 'errors out of' /tmp/test.log
Show broken tests:
egrep 'Failures on these tests:' /tmp/test.log
Broken tests will also show lines ending in “...” instead of “... Ok”.
When done, clean up the SQLite repository:
cd src make clean git clean -i # and delete everything rm -rf testdir git checkout amalgamation/sqlite3.c git checkout amalgamation_dev/sqlite3.c