1.. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2.. Copyright 2010 Julia Lawall <julia@diku.dk> 3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5.. highlight:: none 6 7Coccinelle 8========== 9 10Coccinelle is a tool for pattern matching and text transformation that has 11many uses in kernel development, including the application of complex, 12tree-wide patches and detection of problematic programming patterns. 13 14Getting Coccinelle 15------------------- 16 17The semantic patches included in the kernel use features and options 18which are provided by Coccinelle version 1.0.0-rc11 and above. 19Using earlier versions will fail as the option names used by 20the Coccinelle files and coccicheck have been updated. 21 22Coccinelle is available through the package manager 23of many distributions, e.g. : 24 25 - Debian 26 - Fedora 27 - Ubuntu 28 - OpenSUSE 29 - Arch Linux 30 - NetBSD 31 - FreeBSD 32 33You can get the latest version released from the Coccinelle homepage at 34http://coccinelle.lip6.fr/ 35 36Information and tips about Coccinelle are also provided on the wiki 37pages at http://cocci.ekstranet.diku.dk/wiki/doku.php 38 39Once you have it, run the following command:: 40 41 ./configure 42 make 43 44as a regular user, and install it with:: 45 46 sudo make install 47 48Supplemental documentation 49--------------------------- 50 51For supplemental documentation refer to the wiki: 52 53https://bottest.wiki.kernel.org/coccicheck 54 55The wiki documentation always refers to the linux-next version of the script. 56 57Using Coccinelle on the Linux kernel 58------------------------------------ 59 60A Coccinelle-specific target is defined in the top level 61Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 62front-end in the ``scripts`` directory. 63 64Four basic modes are defined: ``patch``, ``report``, ``context``, and 65``org``. The mode to use is specified by setting the MODE variable with 66``MODE=<mode>``. 67 68- ``patch`` proposes a fix, when possible. 69 70- ``report`` generates a list in the following format: 71 file:line:column-column: message 72 73- ``context`` highlights lines of interest and their context in a 74 diff-like style.Lines of interest are indicated with ``-``. 75 76- ``org`` generates a report in the Org mode format of Emacs. 77 78Note that not all semantic patches implement all modes. For easy use 79of Coccinelle, the default mode is "report". 80 81Two other modes provide some common combinations of these modes. 82 83- ``chain`` tries the previous modes in the order above until one succeeds. 84 85- ``rep+ctxt`` runs successively the report mode and the context mode. 86 It should be used with the C option (described later) 87 which checks the code on a file basis. 88 89Examples 90~~~~~~~~ 91 92To make a report for every semantic patch, run the following command:: 93 94 make coccicheck MODE=report 95 96To produce patches, run:: 97 98 make coccicheck MODE=patch 99 100 101The coccicheck target applies every semantic patch available in the 102sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 103 104For each semantic patch, a commit message is proposed. It gives a 105description of the problem being checked by the semantic patch, and 106includes a reference to Coccinelle. 107 108As any static code analyzer, Coccinelle produces false 109positives. Thus, reports must be carefully checked, and patches 110reviewed. 111 112To enable verbose messages set the V= variable, for example:: 113 114 make coccicheck MODE=report V=1 115 116Coccinelle parallelization 117--------------------------- 118 119By default, coccicheck tries to run as parallel as possible. To change 120the parallelism, set the J= variable. For example, to run across 4 CPUs:: 121 122 make coccicheck MODE=report J=4 123 124As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization, 125if support for this is detected you will benefit from parmap parallelization. 126 127When parmap is enabled coccicheck will enable dynamic load balancing by using 128``--chunksize 1`` argument, this ensures we keep feeding threads with work 129one by one, so that we avoid the situation where most work gets done by only 130a few threads. With dynamic load balancing, if a thread finishes early we keep 131feeding it more work. 132 133When parmap is enabled, if an error occurs in Coccinelle, this error 134value is propagated back, the return value of the ``make coccicheck`` 135captures this return value. 136 137Using Coccinelle with a single semantic patch 138--------------------------------------------- 139 140The optional make variable COCCI can be used to check a single 141semantic patch. In that case, the variable must be initialized with 142the name of the semantic patch to apply. 143 144For instance:: 145 146 make coccicheck COCCI=<my_SP.cocci> MODE=patch 147 148or:: 149 150 make coccicheck COCCI=<my_SP.cocci> MODE=report 151 152 153Controlling Which Files are Processed by Coccinelle 154--------------------------------------------------- 155 156By default the entire kernel source tree is checked. 157 158To apply Coccinelle to a specific directory, ``M=`` can be used. 159For example, to check drivers/net/wireless/ one may write:: 160 161 make coccicheck M=drivers/net/wireless/ 162 163To apply Coccinelle on a file basis, instead of a directory basis, the 164following command may be used:: 165 166 make C=1 CHECK="scripts/coccicheck" 167 168To check only newly edited code, use the value 2 for the C flag, i.e.:: 169 170 make C=2 CHECK="scripts/coccicheck" 171 172In these modes, which works on a file basis, there is no information 173about semantic patches displayed, and no commit message proposed. 174 175This runs every semantic patch in scripts/coccinelle by default. The 176COCCI variable may additionally be used to only apply a single 177semantic patch as shown in the previous section. 178 179The "report" mode is the default. You can select another one with the 180MODE variable explained above. 181 182Debugging Coccinelle SmPL patches 183--------------------------------- 184 185Using coccicheck is best as it provides in the spatch command line 186include options matching the options used when we compile the kernel. 187You can learn what these options are by using V=1, you could then 188manually run Coccinelle with debug options added. 189 190Alternatively you can debug running Coccinelle against SmPL patches 191by asking for stderr to be redirected to stderr, by default stderr 192is redirected to /dev/null, if you'd like to capture stderr you 193can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For 194instance:: 195 196 rm -f cocci.err 197 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 198 cat cocci.err 199 200You can use SPFLAGS to add debugging flags, for instance you may want to 201add both --profile --show-trying to SPFLAGS when debugging. For instance 202you may want to use:: 203 204 rm -f err.log 205 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 206 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c 207 208err.log will now have the profiling information, while stdout will 209provide some progress information as Coccinelle moves forward with 210work. 211 212DEBUG_FILE support is only supported when using coccinelle >= 1.2. 213 214.cocciconfig support 215-------------------- 216 217Coccinelle supports reading .cocciconfig for default Coccinelle options that 218should be used every time spatch is spawned, the order of precedence for 219variables for .cocciconfig is as follows: 220 221- Your current user's home directory is processed first 222- Your directory from which spatch is called is processed next 223- The directory provided with the --dir option is processed last, if used 224 225Since coccicheck runs through make, it naturally runs from the kernel 226proper dir, as such the second rule above would be implied for picking up a 227.cocciconfig when using ``make coccicheck``. 228 229``make coccicheck`` also supports using M= targets.If you do not supply 230any M= target, it is assumed you want to target the entire kernel. 231The kernel coccicheck script has:: 232 233 if [ "$KBUILD_EXTMOD" = "" ] ; then 234 OPTIONS="--dir $srctree $COCCIINCLUDE" 235 else 236 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" 237 fi 238 239KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases 240the spatch --dir argument is used, as such third rule applies when whether M= 241is used or not, and when M= is used the target directory can have its own 242.cocciconfig file. When M= is not passed as an argument to coccicheck the 243target directory is the same as the directory from where spatch was called. 244 245If not using the kernel's coccicheck target, keep the above precedence 246order logic of .cocciconfig reading. If using the kernel's coccicheck target, 247override any of the kernel's .coccicheck's settings using SPFLAGS. 248 249We help Coccinelle when used against Linux with a set of sensible defaults 250options for Linux with our own Linux .cocciconfig. This hints to coccinelle 251git can be used for ``git grep`` queries over coccigrep. A timeout of 200 252seconds should suffice for now. 253 254The options picked up by coccinelle when reading a .cocciconfig do not appear 255as arguments to spatch processes running on your system, to confirm what 256options will be used by Coccinelle run:: 257 258 spatch --print-options-only 259 260You can override with your own preferred index option by using SPFLAGS. Take 261note that when there are conflicting options Coccinelle takes precedence for 262the last options passed. Using .cocciconfig is possible to use idutils, however 263given the order of precedence followed by Coccinelle, since the kernel now 264carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 265desired. See below section "Additional flags" for more details on how to use 266idutils. 267 268Additional flags 269---------------- 270 271Additional flags can be passed to spatch through the SPFLAGS 272variable. This works as Coccinelle respects the last flags 273given to it when options are in conflict. :: 274 275 make SPFLAGS=--use-glimpse coccicheck 276 277Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 278When no ID file is specified coccinelle assumes your ID database file 279is in the file .id-utils.index on the top level of the kernel, coccinelle 280carries a script scripts/idutils_index.sh which creates the database with:: 281 282 mkid -i C --output .id-utils.index 283 284If you have another database filename you can also just symlink with this 285name. :: 286 287 make SPFLAGS=--use-idutils coccicheck 288 289Alternatively you can specify the database filename explicitly, for 290instance:: 291 292 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 293 294See ``spatch --help`` to learn more about spatch options. 295 296Note that the ``--use-glimpse`` and ``--use-idutils`` options 297require external tools for indexing the code. None of them is 298thus active by default. However, by indexing the code with 299one of these tools, and according to the cocci file used, 300spatch could proceed the entire code base more quickly. 301 302SmPL patch specific options 303--------------------------- 304 305SmPL patches can have their own requirements for options passed 306to Coccinelle. SmPL patch specific options can be provided by 307providing them at the top of the SmPL patch, for instance:: 308 309 // Options: --no-includes --include-headers 310 311SmPL patch Coccinelle requirements 312---------------------------------- 313 314As Coccinelle features get added some more advanced SmPL patches 315may require newer versions of Coccinelle. If an SmPL patch requires 316at least a version of Coccinelle, this can be specified as follows, 317as an example if requiring at least Coccinelle >= 1.0.5:: 318 319 // Requires: 1.0.5 320 321Proposing new semantic patches 322------------------------------- 323 324New semantic patches can be proposed and submitted by kernel 325developers. For sake of clarity, they should be organized in the 326sub-directories of ``scripts/coccinelle/``. 327 328 329Detailed description of the ``report`` mode 330------------------------------------------- 331 332``report`` generates a list in the following format:: 333 334 file:line:column-column: message 335 336Example 337~~~~~~~ 338 339Running:: 340 341 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 342 343will execute the following part of the SmPL script:: 344 345 <smpl> 346 @r depends on !context && !patch && (org || report)@ 347 expression x; 348 position p; 349 @@ 350 351 ERR_PTR@p(PTR_ERR(x)) 352 353 @script:python depends on report@ 354 p << r.p; 355 x << r.x; 356 @@ 357 358 msg="ERR_CAST can be used with %s" % (x) 359 coccilib.report.print_report(p[0], msg) 360 </smpl> 361 362This SmPL excerpt generates entries on the standard output, as 363illustrated below:: 364 365 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 366 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 367 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 368 369 370Detailed description of the ``patch`` mode 371------------------------------------------ 372 373When the ``patch`` mode is available, it proposes a fix for each problem 374identified. 375 376Example 377~~~~~~~ 378 379Running:: 380 381 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 382 383will execute the following part of the SmPL script:: 384 385 <smpl> 386 @ depends on !context && patch && !org && !report @ 387 expression x; 388 @@ 389 390 - ERR_PTR(PTR_ERR(x)) 391 + ERR_CAST(x) 392 </smpl> 393 394This SmPL excerpt generates patch hunks on the standard output, as 395illustrated below:: 396 397 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 398 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 399 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 400 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 401 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 402 CRYPTO_ALG_TYPE_MASK); 403 if (IS_ERR(alg)) 404 - return ERR_PTR(PTR_ERR(alg)); 405 + return ERR_CAST(alg); 406 407 /* Block size must be >= 4 bytes. */ 408 err = -EINVAL; 409 410Detailed description of the ``context`` mode 411-------------------------------------------- 412 413``context`` highlights lines of interest and their context 414in a diff-like style. 415 416 **NOTE**: The diff-like output generated is NOT an applicable patch. The 417 intent of the ``context`` mode is to highlight the important lines 418 (annotated with minus, ``-``) and gives some surrounding context 419 lines around. This output can be used with the diff mode of 420 Emacs to review the code. 421 422Example 423~~~~~~~ 424 425Running:: 426 427 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 428 429will execute the following part of the SmPL script:: 430 431 <smpl> 432 @ depends on context && !patch && !org && !report@ 433 expression x; 434 @@ 435 436 * ERR_PTR(PTR_ERR(x)) 437 </smpl> 438 439This SmPL excerpt generates diff hunks on the standard output, as 440illustrated below:: 441 442 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 443 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 444 +++ /tmp/nothing 445 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 446 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 447 CRYPTO_ALG_TYPE_MASK); 448 if (IS_ERR(alg)) 449 - return ERR_PTR(PTR_ERR(alg)); 450 451 /* Block size must be >= 4 bytes. */ 452 err = -EINVAL; 453 454Detailed description of the ``org`` mode 455---------------------------------------- 456 457``org`` generates a report in the Org mode format of Emacs. 458 459Example 460~~~~~~~ 461 462Running:: 463 464 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 465 466will execute the following part of the SmPL script:: 467 468 <smpl> 469 @r depends on !context && !patch && (org || report)@ 470 expression x; 471 position p; 472 @@ 473 474 ERR_PTR@p(PTR_ERR(x)) 475 476 @script:python depends on org@ 477 p << r.p; 478 x << r.x; 479 @@ 480 481 msg="ERR_CAST can be used with %s" % (x) 482 msg_safe=msg.replace("[","@(").replace("]",")") 483 coccilib.org.print_todo(p[0], msg_safe) 484 </smpl> 485 486This SmPL excerpt generates Org entries on the standard output, as 487illustrated below:: 488 489 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 490 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 491 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 492