CHROMIUM: Merge 'upstream/master' into chromeos-0.7
Merge of upstream through 5cf2c00ff7d4d823b5cbe6420b2a48f1ed0447b3
Conflicts:
src/ShellCheck/Analytics.hs
Manual changes:
Minor conflicts in variable lists for portage
Changelog:
----------------------------------------------------------------
Artur Klauser (4):
Add multi-architecture Docker image build
Don't try to deploy docker images on PR runs
Run "deploy" step only for "Build" stages
Use shellcheck on yourself
Austin English (1):
src/ShellCheck/Analytics.hs: suggest using a shell directive for SC2148
Benjamin Gordon (1):
Merge 'upstream/master' into chromeos-0.7
Joseph C. Sible (82):
Simplify literalEquals
Use findM instead of filterM
Use the Identity monad to avoid unnecessary uses of fromJust
Remove more unnecessary uses of fromJust
Switch getLiteralStringExt to Identity where it can never be Nothing
Add getLiteralStringDef and simplify with it
Replace mapMaybe and concatMap with list comprehensions
Use mapM instead of implementing a slower version of it
Use find instead of take 1 and filter
Use mapM_ and sequence_ instead of reimplementing them
Simplify check and checkTranslatedStringVariable
Get rid of potentially
Use head instead of reimplementing it
Use void instead of do and return ()
Simplify matchToken
Simplify <> for SpaceStatus
Inline an uncurry
Avoid unnecessary use of when and unless
Simplify findFunction
Use Map.! instead of reimplementing it
Simplify a double negative
Do toLower earlier
Remove unnecessary fromMaybes
Avoid a zip that breaks fusion
Use forM_ instead of reimplementing it
Use isNothing instead of reimplementing it
Use Map.member instead of isJust and Map.lookup
Simplify determineShell
Remove unnecessary uses of head
Use isJust instead of reimplementing it
Simplify shellFromFilename
Simplify mockedSystemInterface
Remove a partial pattern match equivalent to fromJust from checkFindNameGlob
Optimize patterns in checkFindNameGlob
Use headOrDefault instead of fromMaybe and listToMaybe
Simplify checkForInQuoted
Simplify checkWhileReadPitfalls
Mark that base >= 4.8.0.0 is required
Fix #1892: Use pattern synonyms to clean up AST
Simplify nameExpansion
Use force instead of reimplementing it
Remove unnecessary Maybe from isQuoteFreeElement
Implement findFirst in terms of foldr
Use execState instead of reimplementing it
Simplify causesSubshell
Use a list comprehension instead of a concatMap with extra lists
Remove unnecessary maybeToList
Remove unnecessary as-patterns
Remove an unnecessary operator section
Simplify isArrayFlag
Use head instead of (!! 0)
Simplify dropPrefix
Simplify getSpecial
Don't bother with asks if you're just immediately binding the result anyway
Implement supportsArrays with pattern-matching
Simplify process
Simplify getCommandNameAndToken
Simplify getAssociativeArrays
Remove unnecessary monadicity from wordToPseudoGlob
Remove unnecessary cases from wordToPseudoGlob
Only perform the comparisons once
Use foldr in checkFindNameGlob
Use pattern matching instead of snd
Use a guard instead of unless
Prefer pattern matching in undirected
Simplify checkArg
Use MultiWayIf instead of case-matching on ()
Simplify checkSetAssignment
Simplify warnRedundant
Make skipRepeating lazier and faster
Make it slightly lazier still (and more clear)
Write getLiteralArgs with foldr and without fromMaybe or monads
Remove unnecessary fromMaybe and when from bashism
Use fromRight instead of reimplementing it
Avoid some awkward parentheses with forM_
Simplify thenSkip, and use in another location
Simplify checkVariableBraces
Combine bracedString into getSingleUnmodifiedVariable
Get rid of bracedString everywhere it's easy to
Move bracedString to be local to its last use site
Clean up and optimize getSuspiciousRegexWildcard
Revert "Use fromRight instead of reimplementing it"
Marcin Szydelski (1):
SC2016: disable for mumps -run %XCMD and LOOP%XCMD
Vidar Holen (59):
Merge pull request #1828 from josephcsible/cleanups
Merge pull request #1827 from josephcsible/nofromjust2
Merge pull request #1826 from josephcsible/nofromjust
Merge pull request #1825 from josephcsible/nofilterm
Merge pull request #1824 from josephcsible/patch-1
Merge pull request #1802 from szydell/master
Merge pull request #1785 from ArturKlauser/multi-arch-docker
Merge pull request #1831 from josephcsible/checkfindnameglob
Don't try to deploy on PRs
Rename 'Test' stage
Parse keywords with case sensitivity (fixes #1809)
Bump SC1102/SC1105 about ambiguous `$((` to Error (fixes #1836)
Remove unused instance Ord Replacement (fixes #1829)
Inspect 'alias' commands for referenced variables (Fixes #1832)
SC2257: Warn when changing arithmetic variables in redirections
Upload to assets to GitHub
Merge pull request #1862 from austin987/sc2148-shell-directive
Fix TravisCI condition
Refer to GitHub rather than GCS for release builds
Make SC2095 (ssh in while read loops) more robust and suggest fixes
Merge pull request #1865 from josephcsible/patch-1
Improve detection of for loops with single values
Try to make TravisCI not fail on deployment of Docker stage
Include shebang in AST traversal (fixes #1858)
Stop deploying artifacts to GCS
Update distro tests to support newer Cabal
Merge pull request #1893 from josephcsible/pattern-synonyms
Merge pull request #1872 from josephcsible/checkforinquoted
Merge pull request #1873 from josephcsible/checkwhilereadpitfalls
Merge pull request #1880 from josephcsible/patch-1
Merge pull request #1885 from ArturKlauser/travis-pr-fix
Merge pull request #1897 from ArturKlauser/use-shellcheck-on-yourself
Merge pull request #1896 from ArturKlauser/travis-deploy-stage-fix
Merge pull request #1876 from fork-graveyard/master
Don't warn about [ 0 -ne $FOO ] || [ 0 -ne $BAR ] (fixes #1891)
Stable version v0.7.1
Update Changelog with new version
Filter GitHub uploads by tag
Disable SC2257 about > $((i=42)) for Dash
Merge pull request #1898 from josephcsible/nameexpansion
Merge pull request #1900 from josephcsible/analyzerlib
Merge pull request #1902 from josephcsible/astlib
Merge pull request #1904 from josephcsible/commands
Merge pull request #1903 from josephcsible/fixer
Merge pull request #1901 from josephcsible/bracedstring
Merge pull request #1905 from josephcsible/skiprepeating
Merge pull request #1906 from josephcsible/shellsupport
Merge pull request #1907 from josephcsible/formatters
Merge pull request #1917 from josephcsible/thenskip
Merge pull request #1918 from josephcsible/getsuspiciousregexwildcard
Merge pull request #1925 from josephcsible/nofromright
Merge pull request #1927 from scop/sc-prefix
Merge pull request #1926 from scop/spelling
Warn about duplicate uses of stdin/out/err
Improve SC2259/60/61 messages
Allow disabling SC1072/SC1073 with annotations (fixes #1931)
Merge pull request #1950 from geeseven/aur-shellcheck-bin
Count $# as an argument reference in SC2120
Warn about defining and using an alias in a single command (fixes #1807)
Ville Skyttä (2):
Spelling fixes
Use SC prefix for disable= in man page
geeseven (1):
update dependency free AUR package
girst (1):
recognize `: ${parameter=word}` as assignment
.compile_binaries | 6 -
.github_deploy | 58 +++
.multi_arch_docker | 113 +++++
.travis.yml | 34 +-
CHANGELOG.md | 20 +-
Dockerfile | 7 -
Dockerfile.multi-arch | 26 ++
README.md | 17 +-
ShellCheck.cabal | 6 +-
nextnumber | 2 +-
shellcheck.1.md | 6 +-
shellcheck.hs | 12 +-
src/ShellCheck/AST.hs | 542 ++++++++++--------------
src/ShellCheck/ASTLib.hs | 83 ++--
src/ShellCheck/Analytics.hs | 748 +++++++++++++++++++++++-----------
src/ShellCheck/AnalyzerLib.hs | 185 ++++-----
src/ShellCheck/Checker.hs | 18 +-
src/ShellCheck/Checks/Commands.hs | 162 ++++----
src/ShellCheck/Checks/ShellSupport.hs | 44 +-
src/ShellCheck/Fixer.hs | 16 +-
src/ShellCheck/Formatter/TTY.hs | 5 +-
src/ShellCheck/Interface.hs | 11 +-
src/ShellCheck/Parser.hs | 55 ++-
test/buildtest | 23 +-
test/check_release | 3 +-
test/distrotest | 22 +-
26 files changed, 1286 insertions(+), 938 deletions(-)
create mode 100755 .github_deploy
create mode 100755 .multi_arch_docker
create mode 100644 Dockerfile.multi-arch
BUG=chromium:1086928
TEST=stack test
Cq-Depend: chromium:2233790
Change-Id: Icaa54e0a17181685f1f8234a9ab811bc24229338
ShellCheck is a GPLv3 tool that gives warnings and suggestions for bash/sh shell scripts:
The goals of ShellCheck are
To point out and clarify typical beginner's syntax issues that cause a shell to give cryptic error messages.
To point out and clarify typical intermediate level semantic problems that cause a shell to behave strangely and counter-intuitively.
To point out subtle caveats, corner cases and pitfalls that may cause an advanced user's otherwise working script to fail under future circumstances.
See the gallery of bad code for examples of what ShellCheck can help you identify!
There are a number of ways to use ShellCheck!
Paste a shell script on https://www.shellcheck.net for instant feedback.
ShellCheck.net is always synchronized to the latest git commit, and is the easiest way to give ShellCheck a go. Tell your friends!
Run shellcheck yourscript in your terminal for instant output, as seen above.
You can see ShellCheck suggestions directly in a variety of editors.
.
.
Sublime, through SublimeLinter.
Atom, through Linter.
VSCode, through vscode-shellcheck.
Most other editors, through GCC error compatibility.
While ShellCheck is mostly intended for interactive use, it can easily be added to builds or test suites. It makes canonical use of exit codes, so you can just add a shellcheck command as part of the process.
For example, in a Makefile:
check-scripts: # Fail if any of these files have warnings shellcheck myscripts/*.sh
or in a Travis CI .travis.yml file:
script: # Fail if any of these files have warnings - shellcheck myscripts/*.sh
Services and platforms that have ShellCheck pre-installed and ready to use:
Services and platforms with third party plugins:
Most other services, including GitLab, let you install ShellCheck yourself, either through the system's package manager (see Installing), or by downloading and unpacking a binary release.
It's a good idea to manually install a specific ShellCheck version regardless. This avoids any surprise build breaks when a new version with new warnings is published.
For customized filtering or reporting, ShellCheck can output simple JSON, CheckStyle compatible XML, GCC compatible warnings as well as human readable text (with or without ANSI colors). See the Integration wiki page for more documentation.
The easiest way to install ShellCheck locally is through your package manager.
On systems with Cabal (installs to ~/.cabal/bin):
cabal update cabal install ShellCheck
On systems with Stack (installs to ~/.local/bin):
stack update stack install ShellCheck
On Debian based distros:
apt-get install shellcheck
On Arch Linux based distros:
pacman -S shellcheck
or get the dependency free shellcheck-bin from the AUR.
On Gentoo based distros:
emerge --ask shellcheck
On EPEL based distros:
yum -y install epel-release yum install ShellCheck
On Fedora based distros:
dnf install ShellCheck
On FreeBSD:
pkg install hs-ShellCheck
On OS X with homebrew:
brew install shellcheck
On OpenBSD:
pkg_add shellcheck
On openSUSE
zypper in ShellCheck
Or use OneClickInstall - https://software.opensuse.org/package/ShellCheck
On Solus:
eopkg install shellcheck
On Windows (via chocolatey):
C:\> choco install shellcheck
Or Windows (via scoop):
C:\> scoop install shellcheck
From Snap Store:
snap install --channel=edge shellcheck
From Docker Hub:
docker run --rm -v "$PWD:/mnt" koalaman/shellcheck:stable myscript # Or :v0.4.7 for that version, or :latest for daily builds
or use koalaman/shellcheck-alpine if you want a larger Alpine Linux based image to extend. It works exactly like a regular Alpine image, but has shellcheck preinstalled.
Using the nix package manager:
nix-env -iA nixpkgs.shellcheck
Alternatively, you can download pre-compiled binaries for the latest release here:
or see the GitHub Releases for other releases (including the latest meta-release for daily git builds).
Distro packages already come with a man page. If you are building from source, it can be installed with:
pandoc -s -f markdown-smart -t man shellcheck.1.md -o shellcheck.1 sudo mv shellcheck.1 /usr/share/man/man1
Travis CI has now integrated ShellCheck by default, so you don't need to manually install it.
If you still want to do so in order to upgrade at your leisure or ensure you're using the latest release, follow the steps below to install a binary version.
The pre-compiled binaries come in tar.xz files. To decompress them, make sure xz is installed. On Debian/Ubuntu/Mint, you can apt install xz-utils. On Redhat/Fedora/CentOS, yum -y install xz.
A simple installer may do something like:
scversion="stable" # or "v0.4.7", or "latest" wget -qO- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv cp "shellcheck-${scversion}/shellcheck" /usr/bin/ shellcheck --version
This section describes how to build ShellCheck from a source directory. ShellCheck is written in Haskell and requires 2GB of RAM to compile.
ShellCheck is built and packaged using Cabal. Install the package cabal-install from your system's package manager (with e.g. apt-get, brew, emerge, yum, or zypper).
On MacOS (OS X), you can do a fast install of Cabal using brew, which takes a couple of minutes instead of more than 30 minutes if you try to compile it from source.
$ brew install cabal-install
On MacPorts, the package is instead called hs-cabal-install, while native Windows users should install the latest version of the Haskell platform from https://www.haskell.org/platform/
Verify that cabal is installed and update its dependency list with
$ cabal update
git clone this repository, and cd to the ShellCheck source directory to build/install:
$ cabal install
Or if you intend to run the tests:
$ cabal install --enable-tests
This will compile ShellCheck and install it to your ~/.cabal/bin directory.
Add this directory to your PATH (for bash, add this to your ~/.bashrc):
export PATH="$HOME/.cabal/bin:$PATH"
Log out and in again, and verify that your PATH is set up correctly:
$ which shellcheck ~/.cabal/bin/shellcheck
On native Windows, the PATH should already be set up, but the system may use a legacy codepage. In cmd.exe, powershell.exe and Powershell ISE, make sure to use a TrueType font, not a Raster font, and set the active codepage to UTF-8 (65001) with chcp:
chcp 65001
In Powershell ISE, you may need to additionally update the output encoding:
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
To run the unit test suite:
$ cabal test
So what kind of things does ShellCheck look for? Here is an incomplete list of detected issues.
ShellCheck can recognize several types of incorrect quoting:
echo $1 # Unquoted variables find . -name *.ogg # Unquoted find/grep patterns rm "~/my file.txt" # Quoted tilde expansion v='--verbose="true"'; cmd $v # Literal quotes in variables for f in "*.ogg" # Incorrectly quoted 'for' loops touch $@ # Unquoted $@ echo 'Don't forget to restart!' # Singlequote closed by apostrophe echo 'Don\'t try this at home' # Attempting to escape ' in '' echo 'Path is $PATH' # Variables in single quotes trap "echo Took ${SECONDS}s" 0 # Prematurely expanded trap
ShellCheck can recognize many types of incorrect test statements.
[[ n != 0 ]] # Constant test expressions [[ -e *.mpg ]] # Existence checks of globs [[ $foo==0 ]] # Always true due to missing spaces [[ -n "$foo " ]] # Always true due to literals [[ $foo =~ "fo+" ]] # Quoted regex in =~ [ foo =~ re ] # Unsupported [ ] operators [ $1 -eq "shellcheck" ] # Numerical comparison of strings [ $n && $m ] # && in [ .. ] [ grep -q foo file ] # Command without $(..) [[ "$$file" == *.jpg ]] # Comparisons that can't succeed (( 1 -lt 2 )) # Using test operators in ((..))
ShellCheck can recognize instances where commands are used incorrectly:
grep '*foo*' file # Globs in regex contexts find . -exec foo {} && bar {} \; # Prematurely terminated find -exec sudo echo 'Var=42' > /etc/profile # Redirecting sudo time --format=%s sleep 10 # Passing time(1) flags to time builtin while read h; do ssh "$h" uptime # Commands eating while loop input alias archive='mv $1 /backup' # Defining aliases with arguments tr -cd '[a-zA-Z0-9]' # [] around ranges in tr exec foo; echo "Done!" # Misused 'exec' find -name \*.bak -o -name \*~ -delete # Implicit precedence in find # find . -exec foo > bar \; # Redirections in find f() { whoami; }; sudo f # External use of internal functions
ShellCheck recognizes many common beginner's syntax errors:
var = 42 # Spaces around = in assignments $foo=42 # $ in assignments for $var in *; do ... # $ in for loop variables var$n="Hello" # Wrong indirect assignment echo ${var$n} # Wrong indirect reference var=(1, 2, 3) # Comma separated arrays array=( [index] = value ) # Incorrect index initialization echo $var[14] # Missing {} in array references echo "Argument 10 is $10" # Positional parameter misreference if $(myfunction); then ..; fi # Wrapping commands in $() else if othercondition; then .. # Using 'else if' f; f() { echo "hello world; } # Using function before definition [ false ] # 'false' being true if ( -f file ) # Using (..) instead of test
ShellCheck can make suggestions to improve style:
[[ -z $(find /tmp | grep mpg) ]] # Use grep -q instead a >> log; b >> log; c >> log # Use a redirection block instead echo "The time is `date`" # Use $() instead cd dir; process *; cd ..; # Use subshells instead echo $[1+2] # Use standard $((..)) instead of old $[] echo $(($RANDOM % 6)) # Don't use $ on variables in $((..)) echo "$(date)" # Useless use of echo cat file | grep foo # Useless use of cat
ShellCheck can recognize issues related to data and typing:
args="$@" # Assigning arrays to strings files=(foo bar); echo "$files" # Referencing arrays as strings declare -A arr=(foo bar) # Associative arrays without index printf "%s\n" "Arguments: $@." # Concatenating strings and arrays [[ $# > 2 ]] # Comparing numbers as strings var=World; echo "Hello " var # Unused lowercase variables echo "Hello $name" # Unassigned lowercase variables cmd | read bar; echo $bar # Assignments in subshells cat foo | cp bar # Piping to commands that don't read printf '%s: %s\n' foo # Mismatches in printf argument count
ShellCheck can make suggestions for improving the robustness of a script:
rm -rf "$STEAMROOT/"* # Catastrophic rm touch ./-l; ls * # Globs that could become options find . -exec sh -c 'a && b {}' \; # Find -exec shell injection printf "Hello $name" # Variables in printf format for f in $(ls *.txt); do # Iterating over ls output export MYVAR=$(cmd) # Masked exit codes case $version in 2.*) :;; 2.6.*) # Shadowed case branches
ShellCheck will warn when using features not supported by the shebang. For example, if you set the shebang to #!/bin/sh, ShellCheck will warn about portability issues similar to checkbashisms:
echo {1..$n} # Works in ksh, but not bash/dash/sh echo {1..10} # Works in ksh and bash, but not dash/sh echo -n 42 # Works in ksh, bash and dash, undefined in sh trap 'exit 42' sigint # Unportable signal spec cmd &> file # Unportable redirection operator read foo < /dev/tcp/host/22 # Unportable intercepted files foo-bar() { ..; } # Undefined/unsupported function name [ $UID = 0 ] # Variable undefined in dash/sh local var=value # local is undefined in sh time sleep 1 | sleep 5 # Undefined uses of 'time'
ShellCheck recognizes a menagerie of other issues:
PS1='\e[0;32m\$\e[0m ' # PS1 colors not in \[..\] PATH="$PATH:~/bin" # Literal tilde in $PATH rm “file” # Unicode quotes echo "Hello world" # Carriage return / DOS line endings echo hello \ # Trailing spaces after \ var=42 echo $var # Expansion of inlined environment #!/bin/bash -x -e # Common shebang errors echo $((n/180*100)) # Unnecessary loss of precision ls *[:digit:].txt # Bad character class globs sed 's/foo/bar/' file > file # Redirecting to input while getopts "a" f; do case $f in "b") # Unhandled getopts flags
At first you‘re like “shellcheck is awesome” but then you’re like “wtf are we still using bash”
Alexander Tarasikov, via Twitter
Issues can be ignored via environmental variable, command line, individually or globally within a file:
https://github.com/koalaman/shellcheck/wiki/Ignore
Please use the GitHub issue tracker for any bugs or feature suggestions:
https://github.com/koalaman/shellcheck/issues
Please submit patches to code or documentation as GitHub pull requests! Check out the DevGuide on the ShellCheck Wiki.
Contributions must be licensed under the GNU GPLv3. The contributor retains the copyright.
ShellCheck is licensed under the GNU General Public License, v3. A copy of this license is included in the file LICENSE.
Copyright 2012-2019, Vidar ‘koala_man’ Holen and contributors.
Happy ShellChecking!