1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7Coccinelle 8========== 9 10Coccinelle is a tool for pattern matching and text transformation that has 11many uses in kernel development, including the application of complex, 12tree-wide patches and detection of problematic programming patterns. 13 14Getting Coccinelle 15------------------- 16 17The semantic patches included in the kernel use features and options 18which are provided by Coccinelle version 1.0.0-rc11 and above. 19Using earlier versions will fail as the option names used by 20the Coccinelle files and coccicheck have been updated. 21 22Coccinelle is available through the package manager 23of many distributions, e.g. : 24 25 - Debian 26 - Fedora 27 - Ubuntu 28 - OpenSUSE 29 - Arch Linux 30 - NetBSD 31 - FreeBSD 32 33Some distribution packages are obsolete and it is recommended 34to use the latest version released from the Coccinelle homepage at 35http://coccinelle.lip6.fr/ 36 37Or from Github at: 38 39https://github.com/coccinelle/coccinelle 40 41Once you have it, run the following commands:: 42 43 ./autogen 44 ./configure 45 make 46 47as a regular user, and install it with:: 48 49 sudo make install 50 51More detailed installation instructions to build from source can be 52found at: 53 54https://github.com/coccinelle/coccinelle/blob/master/install.txt 55 56Supplemental documentation 57--------------------------- 58 59For supplemental documentation refer to the wiki: 60 61https://bottest.wiki.kernel.org/coccicheck 62 63The wiki documentation always refers to the linux-next version of the script. 64 65For Semantic Patch Language(SmPL) grammar documentation refer to: 66 67http://coccinelle.lip6.fr/documentation.php 68 69Using Coccinelle on the Linux kernel 70------------------------------------ 71 72A Coccinelle-specific target is defined in the top level 73Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 74front-end in the ``scripts`` directory. 75 76Four basic modes are defined: ``patch``, ``report``, ``context``, and 77``org``. The mode to use is specified by setting the MODE variable with 78``MODE=<mode>``. 79 80- ``patch`` proposes a fix, when possible. 81 82- ``report`` generates a list in the following format: 83 file:line:column-column: message 84 85- ``context`` highlights lines of interest and their context in a 86 diff-like style.Lines of interest are indicated with ``-``. 87 88- ``org`` generates a report in the Org mode format of Emacs. 89 90Note that not all semantic patches implement all modes. For easy use 91of Coccinelle, the default mode is "report". 92 93Two other modes provide some common combinations of these modes. 94 95- ``chain`` tries the previous modes in the order above until one succeeds. 96 97- ``rep+ctxt`` runs successively the report mode and the context mode. 98 It should be used with the C option (described later) 99 which checks the code on a file basis. 100 101Examples 102~~~~~~~~ 103 104To make a report for every semantic patch, run the following command:: 105 106 make coccicheck MODE=report 107 108To produce patches, run:: 109 110 make coccicheck MODE=patch 111 112 113The coccicheck target applies every semantic patch available in the 114sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 115 116For each semantic patch, a commit message is proposed. It gives a 117description of the problem being checked by the semantic patch, and 118includes a reference to Coccinelle. 119 120As any static code analyzer, Coccinelle produces false 121positives. Thus, reports must be carefully checked, and patches 122reviewed. 123 124To enable verbose messages set the V= variable, for example:: 125 126 make coccicheck MODE=report V=1 127 128Coccinelle parallelization 129--------------------------- 130 131By default, coccicheck tries to run as parallel as possible. To change 132the parallelism, set the J= variable. For example, to run across 4 CPUs:: 133 134 make coccicheck MODE=report J=4 135 136As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization, 137if support for this is detected you will benefit from parmap parallelization. 138 139When parmap is enabled coccicheck will enable dynamic load balancing by using 140``--chunksize 1`` argument, this ensures we keep feeding threads with work 141one by one, so that we avoid the situation where most work gets done by only 142a few threads. With dynamic load balancing, if a thread finishes early we keep 143feeding it more work. 144 145When parmap is enabled, if an error occurs in Coccinelle, this error 146value is propagated back, the return value of the ``make coccicheck`` 147captures this return value. 148 149Using Coccinelle with a single semantic patch 150--------------------------------------------- 151 152The optional make variable COCCI can be used to check a single 153semantic patch. In that case, the variable must be initialized with 154the name of the semantic patch to apply. 155 156For instance:: 157 158 make coccicheck COCCI=<my_SP.cocci> MODE=patch 159 160or:: 161 162 make coccicheck COCCI=<my_SP.cocci> MODE=report 163 164 165Controlling Which Files are Processed by Coccinelle 166--------------------------------------------------- 167 168By default the entire kernel source tree is checked. 169 170To apply Coccinelle to a specific directory, ``M=`` can be used. 171For example, to check drivers/net/wireless/ one may write:: 172 173 make coccicheck M=drivers/net/wireless/ 174 175To apply Coccinelle on a file basis, instead of a directory basis, the 176following command may be used:: 177 178 make C=1 CHECK="scripts/coccicheck" 179 180To check only newly edited code, use the value 2 for the C flag, i.e.:: 181 182 make C=2 CHECK="scripts/coccicheck" 183 184In these modes, which works on a file basis, there is no information 185about semantic patches displayed, and no commit message proposed. 186 187This runs every semantic patch in scripts/coccinelle by default. The 188COCCI variable may additionally be used to only apply a single 189semantic patch as shown in the previous section. 190 191The "report" mode is the default. You can select another one with the 192MODE variable explained above. 193 194Debugging Coccinelle SmPL patches 195--------------------------------- 196 197Using coccicheck is best as it provides in the spatch command line 198include options matching the options used when we compile the kernel. 199You can learn what these options are by using V=1, you could then 200manually run Coccinelle with debug options added. 201 202Alternatively you can debug running Coccinelle against SmPL patches 203by asking for stderr to be redirected to stderr, by default stderr 204is redirected to /dev/null, if you'd like to capture stderr you 205can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For 206instance:: 207 208 rm -f cocci.err 209 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 210 cat cocci.err 211 212You can use SPFLAGS to add debugging flags, for instance you may want to 213add both --profile --show-trying to SPFLAGS when debugging. For instance 214you may want to use:: 215 216 rm -f err.log 217 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 218 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c 219 220err.log will now have the profiling information, while stdout will 221provide some progress information as Coccinelle moves forward with 222work. 223 224DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. 225 226.cocciconfig support 227-------------------- 228 229Coccinelle supports reading .cocciconfig for default Coccinelle options that 230should be used every time spatch is spawned, the order of precedence for 231variables for .cocciconfig is as follows: 232 233- Your current user's home directory is processed first 234- Your directory from which spatch is called is processed next 235- The directory provided with the --dir option is processed last, if used 236 237Since coccicheck runs through make, it naturally runs from the kernel 238proper dir, as such the second rule above would be implied for picking up a 239.cocciconfig when using ``make coccicheck``. 240 241``make coccicheck`` also supports using M= targets. If you do not supply 242any M= target, it is assumed you want to target the entire kernel. 243The kernel coccicheck script has:: 244 245 if [ "$KBUILD_EXTMOD" = "" ] ; then 246 OPTIONS="--dir $srctree $COCCIINCLUDE" 247 else 248 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" 249 fi 250 251KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases 252the spatch --dir argument is used, as such third rule applies when whether M= 253is used or not, and when M= is used the target directory can have its own 254.cocciconfig file. When M= is not passed as an argument to coccicheck the 255target directory is the same as the directory from where spatch was called. 256 257If not using the kernel's coccicheck target, keep the above precedence 258order logic of .cocciconfig reading. If using the kernel's coccicheck target, 259override any of the kernel's .coccicheck's settings using SPFLAGS. 260 261We help Coccinelle when used against Linux with a set of sensible defaults 262options for Linux with our own Linux .cocciconfig. This hints to coccinelle 263git can be used for ``git grep`` queries over coccigrep. A timeout of 200 264seconds should suffice for now. 265 266The options picked up by coccinelle when reading a .cocciconfig do not appear 267as arguments to spatch processes running on your system, to confirm what 268options will be used by Coccinelle run:: 269 270 spatch --print-options-only 271 272You can override with your own preferred index option by using SPFLAGS. Take 273note that when there are conflicting options Coccinelle takes precedence for 274the last options passed. Using .cocciconfig is possible to use idutils, however 275given the order of precedence followed by Coccinelle, since the kernel now 276carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 277desired. See below section "Additional flags" for more details on how to use 278idutils. 279 280Additional flags 281---------------- 282 283Additional flags can be passed to spatch through the SPFLAGS 284variable. This works as Coccinelle respects the last flags 285given to it when options are in conflict. :: 286 287 make SPFLAGS=--use-glimpse coccicheck 288 289Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 290When no ID file is specified coccinelle assumes your ID database file 291is in the file .id-utils.index on the top level of the kernel, coccinelle 292carries a script scripts/idutils_index.sh which creates the database with:: 293 294 mkid -i C --output .id-utils.index 295 296If you have another database filename you can also just symlink with this 297name. :: 298 299 make SPFLAGS=--use-idutils coccicheck 300 301Alternatively you can specify the database filename explicitly, for 302instance:: 303 304 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 305 306See ``spatch --help`` to learn more about spatch options. 307 308Note that the ``--use-glimpse`` and ``--use-idutils`` options 309require external tools for indexing the code. None of them is 310thus active by default. However, by indexing the code with 311one of these tools, and according to the cocci file used, 312spatch could proceed the entire code base more quickly. 313 314SmPL patch specific options 315--------------------------- 316 317SmPL patches can have their own requirements for options passed 318to Coccinelle. SmPL patch specific options can be provided by 319providing them at the top of the SmPL patch, for instance:: 320 321 // Options: --no-includes --include-headers 322 323SmPL patch Coccinelle requirements 324---------------------------------- 325 326As Coccinelle features get added some more advanced SmPL patches 327may require newer versions of Coccinelle. If an SmPL patch requires 328at least a version of Coccinelle, this can be specified as follows, 329as an example if requiring at least Coccinelle >= 1.0.5:: 330 331 // Requires: 1.0.5 332 333Proposing new semantic patches 334------------------------------- 335 336New semantic patches can be proposed and submitted by kernel 337developers. For sake of clarity, they should be organized in the 338sub-directories of ``scripts/coccinelle/``. 339 340 341Detailed description of the ``report`` mode 342------------------------------------------- 343 344``report`` generates a list in the following format:: 345 346 file:line:column-column: message 347 348Example 349~~~~~~~ 350 351Running:: 352 353 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 354 355will execute the following part of the SmPL script:: 356 357 <smpl> 358 @r depends on !context && !patch && (org || report)@ 359 expression x; 360 position p; 361 @@ 362 363 ERR_PTR@p(PTR_ERR(x)) 364 365 @script:python depends on report@ 366 p << r.p; 367 x << r.x; 368 @@ 369 370 msg="ERR_CAST can be used with %s" % (x) 371 coccilib.report.print_report(p[0], msg) 372 </smpl> 373 374This SmPL excerpt generates entries on the standard output, as 375illustrated below:: 376 377 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 378 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 379 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 380 381 382Detailed description of the ``patch`` mode 383------------------------------------------ 384 385When the ``patch`` mode is available, it proposes a fix for each problem 386identified. 387 388Example 389~~~~~~~ 390 391Running:: 392 393 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 394 395will execute the following part of the SmPL script:: 396 397 <smpl> 398 @ depends on !context && patch && !org && !report @ 399 expression x; 400 @@ 401 402 - ERR_PTR(PTR_ERR(x)) 403 + ERR_CAST(x) 404 </smpl> 405 406This SmPL excerpt generates patch hunks on the standard output, as 407illustrated below:: 408 409 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 410 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 411 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 412 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 413 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 414 CRYPTO_ALG_TYPE_MASK); 415 if (IS_ERR(alg)) 416 - return ERR_PTR(PTR_ERR(alg)); 417 + return ERR_CAST(alg); 418 419 /* Block size must be >= 4 bytes. */ 420 err = -EINVAL; 421 422Detailed description of the ``context`` mode 423-------------------------------------------- 424 425``context`` highlights lines of interest and their context 426in a diff-like style. 427 428 **NOTE**: The diff-like output generated is NOT an applicable patch. The 429 intent of the ``context`` mode is to highlight the important lines 430 (annotated with minus, ``-``) and gives some surrounding context 431 lines around. This output can be used with the diff mode of 432 Emacs to review the code. 433 434Example 435~~~~~~~ 436 437Running:: 438 439 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 440 441will execute the following part of the SmPL script:: 442 443 <smpl> 444 @ depends on context && !patch && !org && !report@ 445 expression x; 446 @@ 447 448 * ERR_PTR(PTR_ERR(x)) 449 </smpl> 450 451This SmPL excerpt generates diff hunks on the standard output, as 452illustrated below:: 453 454 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 455 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 456 +++ /tmp/nothing 457 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 458 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 459 CRYPTO_ALG_TYPE_MASK); 460 if (IS_ERR(alg)) 461 - return ERR_PTR(PTR_ERR(alg)); 462 463 /* Block size must be >= 4 bytes. */ 464 err = -EINVAL; 465 466Detailed description of the ``org`` mode 467---------------------------------------- 468 469``org`` generates a report in the Org mode format of Emacs. 470 471Example 472~~~~~~~ 473 474Running:: 475 476 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 477 478will execute the following part of the SmPL script:: 479 480 <smpl> 481 @r depends on !context && !patch && (org || report)@ 482 expression x; 483 position p; 484 @@ 485 486 ERR_PTR@p(PTR_ERR(x)) 487 488 @script:python depends on org@ 489 p << r.p; 490 x << r.x; 491 @@ 492 493 msg="ERR_CAST can be used with %s" % (x) 494 msg_safe=msg.replace("[","@(").replace("]",")") 495 coccilib.org.print_todo(p[0], msg_safe) 496 </smpl> 497 498This SmPL excerpt generates Org entries on the standard output, as 499illustrated below:: 500 501 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 502 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 503 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 504