1 .. Copyright 2010 Nicolas Palix <npalix@diku.dk> 2 .. Copyright 2010 Julia Lawall <julia@diku.dk> 3 .. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr> 4 5 .. highlight:: none 6 7 Coccinelle 8 ========== 9 10 Coccinelle is a tool for pattern matching and text transformation that has 11 many uses in kernel development, including the application of complex, 12 tree-wide patches and detection of problematic programming patterns. 13 14 Getting Coccinelle 15 ------------------- 16 17 The semantic patches included in the kernel use features and options 18 which are provided by Coccinelle version 1.0.0-rc11 and above. 19 Using earlier versions will fail as the option names used by 20 the Coccinelle files and coccicheck have been updated. 21 22 Coccinelle is available through the package manager 23 of many distributions, e.g. : 24 25 - Debian 26 - Fedora 27 - Ubuntu 28 - OpenSUSE 29 - Arch Linux 30 - NetBSD 31 - FreeBSD 32 33 You can get the latest version released from the Coccinelle homepage at 34 http://coccinelle.lip6.fr/ 35 36 Information and tips about Coccinelle are also provided on the wiki 37 pages at http://cocci.ekstranet.diku.dk/wiki/doku.php 38 39 Once you have it, run the following command:: 40 41 ./configure 42 make 43 44 as a regular user, and install it with:: 45 46 sudo make install 47 48 Supplemental documentation 49 --------------------------- 50 51 For supplemental documentation refer to the wiki: 52 53 https://bottest.wiki.kernel.org/coccicheck 54 55 The wiki documentation always refers to the linux-next version of the script. 56 57 Using Coccinelle on the Linux kernel 58 ------------------------------------ 59 60 A Coccinelle-specific target is defined in the top level 61 Makefile. This target is named ``coccicheck`` and calls the ``coccicheck`` 62 front-end in the ``scripts`` directory. 63 64 Four basic modes are defined: ``patch``, ``report``, ``context``, and 65 ``org``. The mode to use is specified by setting the MODE variable with 66 ``MODE=<mode>``. 67 68 - ``patch`` proposes a fix, when possible. 69 70 - ``report`` generates a list in the following format: 71 file:line:column-column: message 72 73 - ``context`` highlights lines of interest and their context in a 74 diff-like style.Lines of interest are indicated with ``-``. 75 76 - ``org`` generates a report in the Org mode format of Emacs. 77 78 Note that not all semantic patches implement all modes. For easy use 79 of Coccinelle, the default mode is "report". 80 81 Two other modes provide some common combinations of these modes. 82 83 - ``chain`` tries the previous modes in the order above until one succeeds. 84 85 - ``rep+ctxt`` runs successively the report mode and the context mode. 86 It should be used with the C option (described later) 87 which checks the code on a file basis. 88 89 Examples 90 ~~~~~~~~ 91 92 To make a report for every semantic patch, run the following command:: 93 94 make coccicheck MODE=report 95 96 To produce patches, run:: 97 98 make coccicheck MODE=patch 99 100 101 The coccicheck target applies every semantic patch available in the 102 sub-directories of ``scripts/coccinelle`` to the entire Linux kernel. 103 104 For each semantic patch, a commit message is proposed. It gives a 105 description of the problem being checked by the semantic patch, and 106 includes a reference to Coccinelle. 107 108 As any static code analyzer, Coccinelle produces false 109 positives. Thus, reports must be carefully checked, and patches 110 reviewed. 111 112 To enable verbose messages set the V= variable, for example:: 113 114 make coccicheck MODE=report V=1 115 116 Coccinelle parallelization 117 --------------------------- 118 119 By default, coccicheck tries to run as parallel as possible. To change 120 the parallelism, set the J= variable. For example, to run across 4 CPUs:: 121 122 make coccicheck MODE=report J=4 123 124 As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization, 125 if support for this is detected you will benefit from parmap parallelization. 126 127 When parmap is enabled coccicheck will enable dynamic load balancing by using 128 ``--chunksize 1`` argument, this ensures we keep feeding threads with work 129 one by one, so that we avoid the situation where most work gets done by only 130 a few threads. With dynamic load balancing, if a thread finishes early we keep 131 feeding it more work. 132 133 When parmap is enabled, if an error occurs in Coccinelle, this error 134 value is propagated back, the return value of the ``make coccicheck`` 135 captures this return value. 136 137 Using Coccinelle with a single semantic patch 138 --------------------------------------------- 139 140 The optional make variable COCCI can be used to check a single 141 semantic patch. In that case, the variable must be initialized with 142 the name of the semantic patch to apply. 143 144 For instance:: 145 146 make coccicheck COCCI=<my_SP.cocci> MODE=patch 147 148 or:: 149 150 make coccicheck COCCI=<my_SP.cocci> MODE=report 151 152 153 Controlling Which Files are Processed by Coccinelle 154 --------------------------------------------------- 155 156 By default the entire kernel source tree is checked. 157 158 To apply Coccinelle to a specific directory, ``M=`` can be used. 159 For example, to check drivers/net/wireless/ one may write:: 160 161 make coccicheck M=drivers/net/wireless/ 162 163 To apply Coccinelle on a file basis, instead of a directory basis, the 164 following command may be used:: 165 166 make C=1 CHECK="scripts/coccicheck" 167 168 To check only newly edited code, use the value 2 for the C flag, i.e.:: 169 170 make C=2 CHECK="scripts/coccicheck" 171 172 In these modes, which works on a file basis, there is no information 173 about semantic patches displayed, and no commit message proposed. 174 175 This runs every semantic patch in scripts/coccinelle by default. The 176 COCCI variable may additionally be used to only apply a single 177 semantic patch as shown in the previous section. 178 179 The "report" mode is the default. You can select another one with the 180 MODE variable explained above. 181 182 Debugging Coccinelle SmPL patches 183 --------------------------------- 184 185 Using coccicheck is best as it provides in the spatch command line 186 include options matching the options used when we compile the kernel. 187 You can learn what these options are by using V=1, you could then 188 manually run Coccinelle with debug options added. 189 190 Alternatively you can debug running Coccinelle against SmPL patches 191 by asking for stderr to be redirected to stderr, by default stderr 192 is redirected to /dev/null, if you'd like to capture stderr you 193 can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For 194 instance:: 195 196 rm -f cocci.err 197 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err 198 cat cocci.err 199 200 You can use SPFLAGS to add debugging flags, for instance you may want to 201 add both --profile --show-trying to SPFLAGS when debugging. For instance 202 you may want to use:: 203 204 rm -f err.log 205 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci 206 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c 207 208 err.log will now have the profiling information, while stdout will 209 provide some progress information as Coccinelle moves forward with 210 work. 211 212 DEBUG_FILE support is only supported when using coccinelle >= 1.2. 213 214 .cocciconfig support 215 -------------------- 216 217 Coccinelle supports reading .cocciconfig for default Coccinelle options that 218 should be used every time spatch is spawned, the order of precedence for 219 variables for .cocciconfig is as follows: 220 221 - Your current user's home directory is processed first 222 - Your directory from which spatch is called is processed next 223 - The directory provided with the --dir option is processed last, if used 224 225 Since coccicheck runs through make, it naturally runs from the kernel 226 proper dir, as such the second rule above would be implied for picking up a 227 .cocciconfig when using ``make coccicheck``. 228 229 ``make coccicheck`` also supports using M= targets.If you do not supply 230 any M= target, it is assumed you want to target the entire kernel. 231 The kernel coccicheck script has:: 232 233 if [ "$KBUILD_EXTMOD" = "" ] ; then 234 OPTIONS="--dir $srctree $COCCIINCLUDE" 235 else 236 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" 237 fi 238 239 KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases 240 the spatch --dir argument is used, as such third rule applies when whether M= 241 is used or not, and when M= is used the target directory can have its own 242 .cocciconfig file. When M= is not passed as an argument to coccicheck the 243 target directory is the same as the directory from where spatch was called. 244 245 If not using the kernel's coccicheck target, keep the above precedence 246 order logic of .cocciconfig reading. If using the kernel's coccicheck target, 247 override any of the kernel's .coccicheck's settings using SPFLAGS. 248 249 We help Coccinelle when used against Linux with a set of sensible defaults 250 options for Linux with our own Linux .cocciconfig. This hints to coccinelle 251 git can be used for ``git grep`` queries over coccigrep. A timeout of 200 252 seconds should suffice for now. 253 254 The options picked up by coccinelle when reading a .cocciconfig do not appear 255 as arguments to spatch processes running on your system, to confirm what 256 options will be used by Coccinelle run:: 257 258 spatch --print-options-only 259 260 You can override with your own preferred index option by using SPFLAGS. Take 261 note that when there are conflicting options Coccinelle takes precedence for 262 the last options passed. Using .cocciconfig is possible to use idutils, however 263 given the order of precedence followed by Coccinelle, since the kernel now 264 carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if 265 desired. See below section "Additional flags" for more details on how to use 266 idutils. 267 268 Additional flags 269 ---------------- 270 271 Additional flags can be passed to spatch through the SPFLAGS 272 variable. This works as Coccinelle respects the last flags 273 given to it when options are in conflict. :: 274 275 make SPFLAGS=--use-glimpse coccicheck 276 277 Coccinelle supports idutils as well but requires coccinelle >= 1.0.6. 278 When no ID file is specified coccinelle assumes your ID database file 279 is in the file .id-utils.index on the top level of the kernel, coccinelle 280 carries a script scripts/idutils_index.sh which creates the database with:: 281 282 mkid -i C --output .id-utils.index 283 284 If you have another database filename you can also just symlink with this 285 name. :: 286 287 make SPFLAGS=--use-idutils coccicheck 288 289 Alternatively you can specify the database filename explicitly, for 290 instance:: 291 292 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck 293 294 See ``spatch --help`` to learn more about spatch options. 295 296 Note that the ``--use-glimpse`` and ``--use-idutils`` options 297 require external tools for indexing the code. None of them is 298 thus active by default. However, by indexing the code with 299 one of these tools, and according to the cocci file used, 300 spatch could proceed the entire code base more quickly. 301 302 SmPL patch specific options 303 --------------------------- 304 305 SmPL patches can have their own requirements for options passed 306 to Coccinelle. SmPL patch specific options can be provided by 307 providing them at the top of the SmPL patch, for instance:: 308 309 // Options: --no-includes --include-headers 310 311 SmPL patch Coccinelle requirements 312 ---------------------------------- 313 314 As Coccinelle features get added some more advanced SmPL patches 315 may require newer versions of Coccinelle. If an SmPL patch requires 316 at least a version of Coccinelle, this can be specified as follows, 317 as an example if requiring at least Coccinelle >= 1.0.5:: 318 319 // Requires: 1.0.5 320 321 Proposing new semantic patches 322 ------------------------------- 323 324 New semantic patches can be proposed and submitted by kernel 325 developers. For sake of clarity, they should be organized in the 326 sub-directories of ``scripts/coccinelle/``. 327 328 329 Detailed description of the ``report`` mode 330 ------------------------------------------- 331 332 ``report`` generates a list in the following format:: 333 334 file:line:column-column: message 335 336 Example 337 ~~~~~~~ 338 339 Running:: 340 341 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci 342 343 will execute the following part of the SmPL script:: 344 345 <smpl> 346 @r depends on !context && !patch && (org || report)@ 347 expression x; 348 position p; 349 @@ 350 351 ERR_PTR@p(PTR_ERR(x)) 352 353 @script:python depends on report@ 354 p << r.p; 355 x << r.x; 356 @@ 357 358 msg="ERR_CAST can be used with %s" % (x) 359 coccilib.report.print_report(p[0], msg) 360 </smpl> 361 362 This SmPL excerpt generates entries on the standard output, as 363 illustrated below:: 364 365 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg 366 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth 367 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg 368 369 370 Detailed description of the ``patch`` mode 371 ------------------------------------------ 372 373 When the ``patch`` mode is available, it proposes a fix for each problem 374 identified. 375 376 Example 377 ~~~~~~~ 378 379 Running:: 380 381 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci 382 383 will execute the following part of the SmPL script:: 384 385 <smpl> 386 @ depends on !context && patch && !org && !report @ 387 expression x; 388 @@ 389 390 - ERR_PTR(PTR_ERR(x)) 391 + ERR_CAST(x) 392 </smpl> 393 394 This SmPL excerpt generates patch hunks on the standard output, as 395 illustrated below:: 396 397 diff -u -p a/crypto/ctr.c b/crypto/ctr.c 398 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 399 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200 400 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct 401 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 402 CRYPTO_ALG_TYPE_MASK); 403 if (IS_ERR(alg)) 404 - return ERR_PTR(PTR_ERR(alg)); 405 + return ERR_CAST(alg); 406 407 /* Block size must be >= 4 bytes. */ 408 err = -EINVAL; 409 410 Detailed description of the ``context`` mode 411 -------------------------------------------- 412 413 ``context`` highlights lines of interest and their context 414 in a diff-like style. 415 416 **NOTE**: The diff-like output generated is NOT an applicable patch. The 417 intent of the ``context`` mode is to highlight the important lines 418 (annotated with minus, ``-``) and gives some surrounding context 419 lines around. This output can be used with the diff mode of 420 Emacs to review the code. 421 422 Example 423 ~~~~~~~ 424 425 Running:: 426 427 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci 428 429 will execute the following part of the SmPL script:: 430 431 <smpl> 432 @ depends on context && !patch && !org && !report@ 433 expression x; 434 @@ 435 436 * ERR_PTR(PTR_ERR(x)) 437 </smpl> 438 439 This SmPL excerpt generates diff hunks on the standard output, as 440 illustrated below:: 441 442 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing 443 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200 444 +++ /tmp/nothing 445 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct 446 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, 447 CRYPTO_ALG_TYPE_MASK); 448 if (IS_ERR(alg)) 449 - return ERR_PTR(PTR_ERR(alg)); 450 451 /* Block size must be >= 4 bytes. */ 452 err = -EINVAL; 453 454 Detailed description of the ``org`` mode 455 ---------------------------------------- 456 457 ``org`` generates a report in the Org mode format of Emacs. 458 459 Example 460 ~~~~~~~ 461 462 Running:: 463 464 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci 465 466 will execute the following part of the SmPL script:: 467 468 <smpl> 469 @r depends on !context && !patch && (org || report)@ 470 expression x; 471 position p; 472 @@ 473 474 ERR_PTR@p(PTR_ERR(x)) 475 476 @script:python depends on org@ 477 p << r.p; 478 x << r.x; 479 @@ 480 481 msg="ERR_CAST can be used with %s" % (x) 482 msg_safe=msg.replace("[","@(").replace("]",")") 483 coccilib.org.print_todo(p[0], msg_safe) 484 </smpl> 485 486 This SmPL excerpt generates Org entries on the standard output, as 487 illustrated below:: 488 489 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]] 490 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]] 491 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]] 492