1 #!/usr/bin/env python2 2 # SPDX-License-Identifier: GPL-2.0+ 3 # 4 # Author: Masahiro Yamada <yamada.masahiro@socionext.com> 5 # 6 7 """ 8 Move config options from headers to defconfig files. 9 10 Since Kconfig was introduced to U-Boot, we have worked on moving 11 config options from headers to Kconfig (defconfig). 12 13 This tool intends to help this tremendous work. 14 15 16 Usage 17 ----- 18 19 First, you must edit the Kconfig to add the menu entries for the configs 20 you are moving. 21 22 And then run this tool giving CONFIG names you want to move. 23 For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE, 24 simply type as follows: 25 26 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE 27 28 The tool walks through all the defconfig files and move the given CONFIGs. 29 30 The log is also displayed on the terminal. 31 32 The log is printed for each defconfig as follows: 33 34 <defconfig_name> 35 <action1> 36 <action2> 37 <action3> 38 ... 39 40 <defconfig_name> is the name of the defconfig. 41 42 <action*> shows what the tool did for that defconfig. 43 It looks like one of the following: 44 45 - Move 'CONFIG_... ' 46 This config option was moved to the defconfig 47 48 - CONFIG_... is not defined in Kconfig. Do nothing. 49 The entry for this CONFIG was not found in Kconfig. The option is not 50 defined in the config header, either. So, this case can be just skipped. 51 52 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing. 53 This option is defined in the config header, but its entry was not found 54 in Kconfig. 55 There are two common cases: 56 - You forgot to create an entry for the CONFIG before running 57 this tool, or made a typo in a CONFIG passed to this tool. 58 - The entry was hidden due to unmet 'depends on'. 59 The tool does not know if the result is reasonable, so please check it 60 manually. 61 62 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing. 63 The define in the config header matched the one in Kconfig. 64 We do not need to touch it. 65 66 - Compiler is missing. Do nothing. 67 The compiler specified for this architecture was not found 68 in your PATH environment. 69 (If -e option is passed, the tool exits immediately.) 70 71 - Failed to process. 72 An error occurred during processing this defconfig. Skipped. 73 (If -e option is passed, the tool exits immediately on error.) 74 75 Finally, you will be asked, Clean up headers? [y/n]: 76 77 If you say 'y' here, the unnecessary config defines are removed 78 from the config headers (include/configs/*.h). 79 It just uses the regex method, so you should not rely on it. 80 Just in case, please do 'git diff' to see what happened. 81 82 83 How does it work? 84 ----------------- 85 86 This tool runs configuration and builds include/autoconf.mk for every 87 defconfig. The config options defined in Kconfig appear in the .config 88 file (unless they are hidden because of unmet dependency.) 89 On the other hand, the config options defined by board headers are seen 90 in include/autoconf.mk. The tool looks for the specified options in both 91 of them to decide the appropriate action for the options. If the given 92 config option is found in the .config, but its value does not match the 93 one from the board header, the config option in the .config is replaced 94 with the define in the board header. Then, the .config is synced by 95 "make savedefconfig" and the defconfig is updated with it. 96 97 For faster processing, this tool handles multi-threading. It creates 98 separate build directories where the out-of-tree build is run. The 99 temporary build directories are automatically created and deleted as 100 needed. The number of threads are chosen based on the number of the CPU 101 cores of your system although you can change it via -j (--jobs) option. 102 103 104 Toolchains 105 ---------- 106 107 Appropriate toolchain are necessary to generate include/autoconf.mk 108 for all the architectures supported by U-Boot. Most of them are available 109 at the kernel.org site, some are not provided by kernel.org. This tool uses 110 the same tools as buildman, so see that tool for setup (e.g. --fetch-arch). 111 112 113 Tips and trips 114 -------------- 115 116 To sync only X86 defconfigs: 117 118 ./tools/moveconfig.py -s -d <(grep -l X86 configs/*) 119 120 or: 121 122 grep -l X86 configs/* | ./tools/moveconfig.py -s -d - 123 124 To process CONFIG_CMD_FPGAD only for a subset of configs based on path match: 125 126 ls configs/{hrcon*,iocon*,strider*} | \ 127 ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d - 128 129 130 Finding implied CONFIGs 131 ----------------------- 132 133 Some CONFIG options can be implied by others and this can help to reduce 134 the size of the defconfig files. For example, CONFIG_X86 implies 135 CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and 136 all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to 137 each of the x86 defconfig files. 138 139 This tool can help find such configs. To use it, first build a database: 140 141 ./tools/moveconfig.py -b 142 143 Then try to query it: 144 145 ./tools/moveconfig.py -i CONFIG_CMD_IRQ 146 CONFIG_CMD_IRQ found in 311/2384 defconfigs 147 44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769 148 41 : CONFIG_SYS_FSL_ERRATUM_A007075 149 31 : CONFIG_SYS_FSL_DDR_VER_44 150 28 : CONFIG_ARCH_P1010 151 28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549 152 28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571 153 28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399 154 25 : CONFIG_SYS_FSL_ERRATUM_A008044 155 22 : CONFIG_ARCH_P1020 156 21 : CONFIG_SYS_FSL_DDR_VER_46 157 20 : CONFIG_MAX_PIRQ_LINKS 158 20 : CONFIG_HPET_ADDRESS 159 20 : CONFIG_X86 160 20 : CONFIG_PCIE_ECAM_SIZE 161 20 : CONFIG_IRQ_SLOT_COUNT 162 20 : CONFIG_I8259_PIC 163 20 : CONFIG_CPU_ADDR_BITS 164 20 : CONFIG_RAMBASE 165 20 : CONFIG_SYS_FSL_ERRATUM_A005871 166 20 : CONFIG_PCIE_ECAM_BASE 167 20 : CONFIG_X86_TSC_TIMER 168 20 : CONFIG_I8254_TIMER 169 20 : CONFIG_CMD_GETTIME 170 19 : CONFIG_SYS_FSL_ERRATUM_A005812 171 18 : CONFIG_X86_RUN_32BIT 172 17 : CONFIG_CMD_CHIP_CONFIG 173 ... 174 175 This shows a list of config options which might imply CONFIG_CMD_EEPROM along 176 with how many defconfigs they cover. From this you can see that CONFIG_X86 177 implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to 178 the defconfig of every x86 board, you could add a single imply line to the 179 Kconfig file: 180 181 config X86 182 bool "x86 architecture" 183 ... 184 imply CMD_EEPROM 185 186 That will cover 20 defconfigs. Many of the options listed are not suitable as 187 they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply 188 CMD_EEPROM. 189 190 Using this search you can reduce the size of moveconfig patches. 191 192 You can automatically add 'imply' statements in the Kconfig with the -a 193 option: 194 195 ./tools/moveconfig.py -s -i CONFIG_SCSI \ 196 -a CONFIG_ARCH_LS1021A,CONFIG_ARCH_LS1043A 197 198 This will add 'imply SCSI' to the two CONFIG options mentioned, assuming that 199 the database indicates that they do actually imply CONFIG_SCSI and do not 200 already have an 'imply SCSI'. 201 202 The output shows where the imply is added: 203 204 18 : CONFIG_ARCH_LS1021A arch/arm/cpu/armv7/ls102xa/Kconfig:1 205 13 : CONFIG_ARCH_LS1043A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:11 206 12 : CONFIG_ARCH_LS1046A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:31 207 208 The first number is the number of boards which can avoid having a special 209 CONFIG_SCSI option in their defconfig file if this 'imply' is added. 210 The location at the right is the Kconfig file and line number where the config 211 appears. For example, adding 'imply CONFIG_SCSI' to the 'config ARCH_LS1021A' 212 in arch/arm/cpu/armv7/ls102xa/Kconfig at line 1 will help 18 boards to reduce 213 the size of their defconfig files. 214 215 If you want to add an 'imply' to every imply config in the list, you can use 216 217 ./tools/moveconfig.py -s -i CONFIG_SCSI -a all 218 219 To control which ones are displayed, use -I <list> where list is a list of 220 options (use '-I help' to see possible options and their meaning). 221 222 To skip showing you options that already have an 'imply' attached, use -A. 223 224 When you have finished adding 'imply' options you can regenerate the 225 defconfig files for affected boards with something like: 226 227 git show --stat | ./tools/moveconfig.py -s -d - 228 229 This will regenerate only those defconfigs changed in the current commit. 230 If you start with (say) 100 defconfigs being changed in the commit, and add 231 a few 'imply' options as above, then regenerate, hopefully you can reduce the 232 number of defconfigs changed in the commit. 233 234 235 Available options 236 ----------------- 237 238 -c, --color 239 Surround each portion of the log with escape sequences to display it 240 in color on the terminal. 241 242 -C, --commit 243 Create a git commit with the changes when the operation is complete. A 244 standard commit message is used which may need to be edited. 245 246 -d, --defconfigs 247 Specify a file containing a list of defconfigs to move. The defconfig 248 files can be given with shell-style wildcards. Use '-' to read from stdin. 249 250 -n, --dry-run 251 Perform a trial run that does not make any changes. It is useful to 252 see what is going to happen before one actually runs it. 253 254 -e, --exit-on-error 255 Exit immediately if Make exits with a non-zero status while processing 256 a defconfig file. 257 258 -s, --force-sync 259 Do "make savedefconfig" forcibly for all the defconfig files. 260 If not specified, "make savedefconfig" only occurs for cases 261 where at least one CONFIG was moved. 262 263 -S, --spl 264 Look for moved config options in spl/include/autoconf.mk instead of 265 include/autoconf.mk. This is useful for moving options for SPL build 266 because SPL related options (mostly prefixed with CONFIG_SPL_) are 267 sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals. 268 269 -H, --headers-only 270 Only cleanup the headers; skip the defconfig processing 271 272 -j, --jobs 273 Specify the number of threads to run simultaneously. If not specified, 274 the number of threads is the same as the number of CPU cores. 275 276 -r, --git-ref 277 Specify the git ref to clone for building the autoconf.mk. If unspecified 278 use the CWD. This is useful for when changes to the Kconfig affect the 279 default values and you want to capture the state of the defconfig from 280 before that change was in effect. If in doubt, specify a ref pre-Kconfig 281 changes (use HEAD if Kconfig changes are not committed). Worst case it will 282 take a bit longer to run, but will always do the right thing. 283 284 -v, --verbose 285 Show any build errors as boards are built 286 287 -y, --yes 288 Instead of prompting, automatically go ahead with all operations. This 289 includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist 290 and the README. 291 292 To see the complete list of supported options, run 293 294 $ tools/moveconfig.py -h 295 296 """ 297 298 import collections 299 import copy 300 import difflib 301 import filecmp 302 import fnmatch 303 import glob 304 import multiprocessing 305 import optparse 306 import os 307 import Queue 308 import re 309 import shutil 310 import subprocess 311 import sys 312 import tempfile 313 import threading 314 import time 315 316 sys.path.append(os.path.join(os.path.dirname(__file__), 'buildman')) 317 sys.path.append(os.path.join(os.path.dirname(__file__), 'patman')) 318 import bsettings 319 import kconfiglib 320 import toolchain 321 322 SHOW_GNU_MAKE = 'scripts/show-gnu-make' 323 SLEEP_TIME=0.03 324 325 STATE_IDLE = 0 326 STATE_DEFCONFIG = 1 327 STATE_AUTOCONF = 2 328 STATE_SAVEDEFCONFIG = 3 329 330 ACTION_MOVE = 0 331 ACTION_NO_ENTRY = 1 332 ACTION_NO_ENTRY_WARN = 2 333 ACTION_NO_CHANGE = 3 334 335 COLOR_BLACK = '0;30' 336 COLOR_RED = '0;31' 337 COLOR_GREEN = '0;32' 338 COLOR_BROWN = '0;33' 339 COLOR_BLUE = '0;34' 340 COLOR_PURPLE = '0;35' 341 COLOR_CYAN = '0;36' 342 COLOR_LIGHT_GRAY = '0;37' 343 COLOR_DARK_GRAY = '1;30' 344 COLOR_LIGHT_RED = '1;31' 345 COLOR_LIGHT_GREEN = '1;32' 346 COLOR_YELLOW = '1;33' 347 COLOR_LIGHT_BLUE = '1;34' 348 COLOR_LIGHT_PURPLE = '1;35' 349 COLOR_LIGHT_CYAN = '1;36' 350 COLOR_WHITE = '1;37' 351 352 AUTO_CONF_PATH = 'include/config/auto.conf' 353 CONFIG_DATABASE = 'moveconfig.db' 354 355 CONFIG_LEN = len('CONFIG_') 356 357 ### helper functions ### 358 def get_devnull(): 359 """Get the file object of '/dev/null' device.""" 360 try: 361 devnull = subprocess.DEVNULL # py3k 362 except AttributeError: 363 devnull = open(os.devnull, 'wb') 364 return devnull 365 366 def check_top_directory(): 367 """Exit if we are not at the top of source directory.""" 368 for f in ('README', 'Licenses'): 369 if not os.path.exists(f): 370 sys.exit('Please run at the top of source directory.') 371 372 def check_clean_directory(): 373 """Exit if the source tree is not clean.""" 374 for f in ('.config', 'include/config'): 375 if os.path.exists(f): 376 sys.exit("source tree is not clean, please run 'make mrproper'") 377 378 def get_make_cmd(): 379 """Get the command name of GNU Make. 380 381 U-Boot needs GNU Make for building, but the command name is not 382 necessarily "make". (for example, "gmake" on FreeBSD). 383 Returns the most appropriate command name on your system. 384 """ 385 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) 386 ret = process.communicate() 387 if process.returncode: 388 sys.exit('GNU Make not found') 389 return ret[0].rstrip() 390 391 def get_matched_defconfig(line): 392 """Get the defconfig files that match a pattern 393 394 Args: 395 line: Path or filename to match, e.g. 'configs/snow_defconfig' or 396 'k2*_defconfig'. If no directory is provided, 'configs/' is 397 prepended 398 399 Returns: 400 a list of matching defconfig files 401 """ 402 dirname = os.path.dirname(line) 403 if dirname: 404 pattern = line 405 else: 406 pattern = os.path.join('configs', line) 407 return glob.glob(pattern) + glob.glob(pattern + '_defconfig') 408 409 def get_matched_defconfigs(defconfigs_file): 410 """Get all the defconfig files that match the patterns in a file. 411 412 Args: 413 defconfigs_file: File containing a list of defconfigs to process, or 414 '-' to read the list from stdin 415 416 Returns: 417 A list of paths to defconfig files, with no duplicates 418 """ 419 defconfigs = [] 420 if defconfigs_file == '-': 421 fd = sys.stdin 422 defconfigs_file = 'stdin' 423 else: 424 fd = open(defconfigs_file) 425 for i, line in enumerate(fd): 426 line = line.strip() 427 if not line: 428 continue # skip blank lines silently 429 if ' ' in line: 430 line = line.split(' ')[0] # handle 'git log' input 431 matched = get_matched_defconfig(line) 432 if not matched: 433 print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \ 434 (defconfigs_file, i + 1, line) 435 436 defconfigs += matched 437 438 # use set() to drop multiple matching 439 return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ] 440 441 def get_all_defconfigs(): 442 """Get all the defconfig files under the configs/ directory.""" 443 defconfigs = [] 444 for (dirpath, dirnames, filenames) in os.walk('configs'): 445 dirpath = dirpath[len('configs') + 1:] 446 for filename in fnmatch.filter(filenames, '*_defconfig'): 447 defconfigs.append(os.path.join(dirpath, filename)) 448 449 return defconfigs 450 451 def color_text(color_enabled, color, string): 452 """Return colored string.""" 453 if color_enabled: 454 # LF should not be surrounded by the escape sequence. 455 # Otherwise, additional whitespace or line-feed might be printed. 456 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else '' 457 for s in string.split('\n') ]) 458 else: 459 return string 460 461 def show_diff(a, b, file_path, color_enabled): 462 """Show unidified diff. 463 464 Arguments: 465 a: A list of lines (before) 466 b: A list of lines (after) 467 file_path: Path to the file 468 color_enabled: Display the diff in color 469 """ 470 471 diff = difflib.unified_diff(a, b, 472 fromfile=os.path.join('a', file_path), 473 tofile=os.path.join('b', file_path)) 474 475 for line in diff: 476 if line[0] == '-' and line[1] != '-': 477 print color_text(color_enabled, COLOR_RED, line), 478 elif line[0] == '+' and line[1] != '+': 479 print color_text(color_enabled, COLOR_GREEN, line), 480 else: 481 print line, 482 483 def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre, 484 extend_post): 485 """Extend matched lines if desired patterns are found before/after already 486 matched lines. 487 488 Arguments: 489 lines: A list of lines handled. 490 matched: A list of line numbers that have been already matched. 491 (will be updated by this function) 492 pre_patterns: A list of regular expression that should be matched as 493 preamble. 494 post_patterns: A list of regular expression that should be matched as 495 postamble. 496 extend_pre: Add the line number of matched preamble to the matched list. 497 extend_post: Add the line number of matched postamble to the matched list. 498 """ 499 extended_matched = [] 500 501 j = matched[0] 502 503 for i in matched: 504 if i == 0 or i < j: 505 continue 506 j = i 507 while j in matched: 508 j += 1 509 if j >= len(lines): 510 break 511 512 for p in pre_patterns: 513 if p.search(lines[i - 1]): 514 break 515 else: 516 # not matched 517 continue 518 519 for p in post_patterns: 520 if p.search(lines[j]): 521 break 522 else: 523 # not matched 524 continue 525 526 if extend_pre: 527 extended_matched.append(i - 1) 528 if extend_post: 529 extended_matched.append(j) 530 531 matched += extended_matched 532 matched.sort() 533 534 def confirm(options, prompt): 535 if not options.yes: 536 while True: 537 choice = raw_input('{} [y/n]: '.format(prompt)) 538 choice = choice.lower() 539 print choice 540 if choice == 'y' or choice == 'n': 541 break 542 543 if choice == 'n': 544 return False 545 546 return True 547 548 def cleanup_empty_blocks(header_path, options): 549 """Clean up empty conditional blocks 550 551 Arguments: 552 header_path: path to the cleaned file. 553 options: option flags. 554 """ 555 pattern = re.compile(r'^\s*#\s*if.*$\n^\s*#\s*endif.*$\n*', flags=re.M) 556 with open(header_path) as f: 557 data = f.read() 558 559 new_data = pattern.sub('\n', data) 560 561 show_diff(data.splitlines(True), new_data.splitlines(True), header_path, 562 options.color) 563 564 if options.dry_run: 565 return 566 567 with open(header_path, 'w') as f: 568 f.write(new_data) 569 570 def cleanup_one_header(header_path, patterns, options): 571 """Clean regex-matched lines away from a file. 572 573 Arguments: 574 header_path: path to the cleaned file. 575 patterns: list of regex patterns. Any lines matching to these 576 patterns are deleted. 577 options: option flags. 578 """ 579 with open(header_path) as f: 580 lines = f.readlines() 581 582 matched = [] 583 for i, line in enumerate(lines): 584 if i - 1 in matched and lines[i - 1][-2:] == '\\\n': 585 matched.append(i) 586 continue 587 for pattern in patterns: 588 if pattern.search(line): 589 matched.append(i) 590 break 591 592 if not matched: 593 return 594 595 # remove empty #ifdef ... #endif, successive blank lines 596 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef 597 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else 598 pattern_endif = re.compile(r'#\s*endif\W') # #endif 599 pattern_blank = re.compile(r'^\s*$') # empty line 600 601 while True: 602 old_matched = copy.copy(matched) 603 extend_matched_lines(lines, matched, [pattern_if], 604 [pattern_endif], True, True) 605 extend_matched_lines(lines, matched, [pattern_elif], 606 [pattern_elif, pattern_endif], True, False) 607 extend_matched_lines(lines, matched, [pattern_if, pattern_elif], 608 [pattern_blank], False, True) 609 extend_matched_lines(lines, matched, [pattern_blank], 610 [pattern_elif, pattern_endif], True, False) 611 extend_matched_lines(lines, matched, [pattern_blank], 612 [pattern_blank], True, False) 613 if matched == old_matched: 614 break 615 616 tolines = copy.copy(lines) 617 618 for i in reversed(matched): 619 tolines.pop(i) 620 621 show_diff(lines, tolines, header_path, options.color) 622 623 if options.dry_run: 624 return 625 626 with open(header_path, 'w') as f: 627 for line in tolines: 628 f.write(line) 629 630 def cleanup_headers(configs, options): 631 """Delete config defines from board headers. 632 633 Arguments: 634 configs: A list of CONFIGs to remove. 635 options: option flags. 636 """ 637 if not confirm(options, 'Clean up headers?'): 638 return 639 640 patterns = [] 641 for config in configs: 642 patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) 643 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) 644 645 for dir in 'include', 'arch', 'board': 646 for (dirpath, dirnames, filenames) in os.walk(dir): 647 if dirpath == os.path.join('include', 'generated'): 648 continue 649 for filename in filenames: 650 if not fnmatch.fnmatch(filename, '*~'): 651 header_path = os.path.join(dirpath, filename) 652 cleanup_one_header(header_path, patterns, options) 653 cleanup_empty_blocks(header_path, options) 654 655 def cleanup_one_extra_option(defconfig_path, configs, options): 656 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file. 657 658 Arguments: 659 defconfig_path: path to the cleaned defconfig file. 660 configs: A list of CONFIGs to remove. 661 options: option flags. 662 """ 663 664 start = 'CONFIG_SYS_EXTRA_OPTIONS="' 665 end = '"\n' 666 667 with open(defconfig_path) as f: 668 lines = f.readlines() 669 670 for i, line in enumerate(lines): 671 if line.startswith(start) and line.endswith(end): 672 break 673 else: 674 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig 675 return 676 677 old_tokens = line[len(start):-len(end)].split(',') 678 new_tokens = [] 679 680 for token in old_tokens: 681 pos = token.find('=') 682 if not (token[:pos] if pos >= 0 else token) in configs: 683 new_tokens.append(token) 684 685 if new_tokens == old_tokens: 686 return 687 688 tolines = copy.copy(lines) 689 690 if new_tokens: 691 tolines[i] = start + ','.join(new_tokens) + end 692 else: 693 tolines.pop(i) 694 695 show_diff(lines, tolines, defconfig_path, options.color) 696 697 if options.dry_run: 698 return 699 700 with open(defconfig_path, 'w') as f: 701 for line in tolines: 702 f.write(line) 703 704 def cleanup_extra_options(configs, options): 705 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files. 706 707 Arguments: 708 configs: A list of CONFIGs to remove. 709 options: option flags. 710 """ 711 if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'): 712 return 713 714 configs = [ config[len('CONFIG_'):] for config in configs ] 715 716 defconfigs = get_all_defconfigs() 717 718 for defconfig in defconfigs: 719 cleanup_one_extra_option(os.path.join('configs', defconfig), configs, 720 options) 721 722 def cleanup_whitelist(configs, options): 723 """Delete config whitelist entries 724 725 Arguments: 726 configs: A list of CONFIGs to remove. 727 options: option flags. 728 """ 729 if not confirm(options, 'Clean up whitelist entries?'): 730 return 731 732 with open(os.path.join('scripts', 'config_whitelist.txt')) as f: 733 lines = f.readlines() 734 735 lines = [x for x in lines if x.strip() not in configs] 736 737 with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f: 738 f.write(''.join(lines)) 739 740 def find_matching(patterns, line): 741 for pat in patterns: 742 if pat.search(line): 743 return True 744 return False 745 746 def cleanup_readme(configs, options): 747 """Delete config description in README 748 749 Arguments: 750 configs: A list of CONFIGs to remove. 751 options: option flags. 752 """ 753 if not confirm(options, 'Clean up README?'): 754 return 755 756 patterns = [] 757 for config in configs: 758 patterns.append(re.compile(r'^\s+%s' % config)) 759 760 with open('README') as f: 761 lines = f.readlines() 762 763 found = False 764 newlines = [] 765 for line in lines: 766 if not found: 767 found = find_matching(patterns, line) 768 if found: 769 continue 770 771 if found and re.search(r'^\s+CONFIG', line): 772 found = False 773 774 if not found: 775 newlines.append(line) 776 777 with open('README', 'w') as f: 778 f.write(''.join(newlines)) 779 780 781 ### classes ### 782 class Progress: 783 784 """Progress Indicator""" 785 786 def __init__(self, total): 787 """Create a new progress indicator. 788 789 Arguments: 790 total: A number of defconfig files to process. 791 """ 792 self.current = 0 793 self.total = total 794 795 def inc(self): 796 """Increment the number of processed defconfig files.""" 797 798 self.current += 1 799 800 def show(self): 801 """Display the progress.""" 802 print ' %d defconfigs out of %d\r' % (self.current, self.total), 803 sys.stdout.flush() 804 805 806 class KconfigScanner: 807 """Kconfig scanner.""" 808 809 def __init__(self): 810 """Scan all the Kconfig files and create a Config object.""" 811 # Define environment variables referenced from Kconfig 812 os.environ['srctree'] = os.getcwd() 813 os.environ['UBOOTVERSION'] = 'dummy' 814 os.environ['KCONFIG_OBJDIR'] = '' 815 self.conf = kconfiglib.Config() 816 817 818 class KconfigParser: 819 820 """A parser of .config and include/autoconf.mk.""" 821 822 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') 823 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') 824 825 def __init__(self, configs, options, build_dir): 826 """Create a new parser. 827 828 Arguments: 829 configs: A list of CONFIGs to move. 830 options: option flags. 831 build_dir: Build directory. 832 """ 833 self.configs = configs 834 self.options = options 835 self.dotconfig = os.path.join(build_dir, '.config') 836 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk') 837 self.spl_autoconf = os.path.join(build_dir, 'spl', 'include', 838 'autoconf.mk') 839 self.config_autoconf = os.path.join(build_dir, AUTO_CONF_PATH) 840 self.defconfig = os.path.join(build_dir, 'defconfig') 841 842 def get_arch(self): 843 """Parse .config file and return the architecture. 844 845 Returns: 846 Architecture name (e.g. 'arm'). 847 """ 848 arch = '' 849 cpu = '' 850 for line in open(self.dotconfig): 851 m = self.re_arch.match(line) 852 if m: 853 arch = m.group(1) 854 continue 855 m = self.re_cpu.match(line) 856 if m: 857 cpu = m.group(1) 858 859 if not arch: 860 return None 861 862 # fix-up for aarch64 863 if arch == 'arm' and cpu == 'armv8': 864 arch = 'aarch64' 865 866 return arch 867 868 def parse_one_config(self, config, dotconfig_lines, autoconf_lines): 869 """Parse .config, defconfig, include/autoconf.mk for one config. 870 871 This function looks for the config options in the lines from 872 defconfig, .config, and include/autoconf.mk in order to decide 873 which action should be taken for this defconfig. 874 875 Arguments: 876 config: CONFIG name to parse. 877 dotconfig_lines: lines from the .config file. 878 autoconf_lines: lines from the include/autoconf.mk file. 879 880 Returns: 881 A tupple of the action for this defconfig and the line 882 matched for the config. 883 """ 884 not_set = '# %s is not set' % config 885 886 for line in autoconf_lines: 887 line = line.rstrip() 888 if line.startswith(config + '='): 889 new_val = line 890 break 891 else: 892 new_val = not_set 893 894 for line in dotconfig_lines: 895 line = line.rstrip() 896 if line.startswith(config + '=') or line == not_set: 897 old_val = line 898 break 899 else: 900 if new_val == not_set: 901 return (ACTION_NO_ENTRY, config) 902 else: 903 return (ACTION_NO_ENTRY_WARN, config) 904 905 # If this CONFIG is neither bool nor trisate 906 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set: 907 # tools/scripts/define2mk.sed changes '1' to 'y'. 908 # This is a problem if the CONFIG is int type. 909 # Check the type in Kconfig and handle it correctly. 910 if new_val[-2:] == '=y': 911 new_val = new_val[:-1] + '1' 912 913 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE, 914 new_val) 915 916 def update_dotconfig(self): 917 """Parse files for the config options and update the .config. 918 919 This function parses the generated .config and include/autoconf.mk 920 searching the target options. 921 Move the config option(s) to the .config as needed. 922 923 Arguments: 924 defconfig: defconfig name. 925 926 Returns: 927 Return a tuple of (updated flag, log string). 928 The "updated flag" is True if the .config was updated, False 929 otherwise. The "log string" shows what happend to the .config. 930 """ 931 932 results = [] 933 updated = False 934 suspicious = False 935 rm_files = [self.config_autoconf, self.autoconf] 936 937 if self.options.spl: 938 if os.path.exists(self.spl_autoconf): 939 autoconf_path = self.spl_autoconf 940 rm_files.append(self.spl_autoconf) 941 else: 942 for f in rm_files: 943 os.remove(f) 944 return (updated, suspicious, 945 color_text(self.options.color, COLOR_BROWN, 946 "SPL is not enabled. Skipped.") + '\n') 947 else: 948 autoconf_path = self.autoconf 949 950 with open(self.dotconfig) as f: 951 dotconfig_lines = f.readlines() 952 953 with open(autoconf_path) as f: 954 autoconf_lines = f.readlines() 955 956 for config in self.configs: 957 result = self.parse_one_config(config, dotconfig_lines, 958 autoconf_lines) 959 results.append(result) 960 961 log = '' 962 963 for (action, value) in results: 964 if action == ACTION_MOVE: 965 actlog = "Move '%s'" % value 966 log_color = COLOR_LIGHT_GREEN 967 elif action == ACTION_NO_ENTRY: 968 actlog = "%s is not defined in Kconfig. Do nothing." % value 969 log_color = COLOR_LIGHT_BLUE 970 elif action == ACTION_NO_ENTRY_WARN: 971 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value 972 log_color = COLOR_YELLOW 973 suspicious = True 974 elif action == ACTION_NO_CHANGE: 975 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \ 976 % value 977 log_color = COLOR_LIGHT_PURPLE 978 elif action == ACTION_SPL_NOT_EXIST: 979 actlog = "SPL is not enabled for this defconfig. Skip." 980 log_color = COLOR_PURPLE 981 else: 982 sys.exit("Internal Error. This should not happen.") 983 984 log += color_text(self.options.color, log_color, actlog) + '\n' 985 986 with open(self.dotconfig, 'a') as f: 987 for (action, value) in results: 988 if action == ACTION_MOVE: 989 f.write(value + '\n') 990 updated = True 991 992 self.results = results 993 for f in rm_files: 994 os.remove(f) 995 996 return (updated, suspicious, log) 997 998 def check_defconfig(self): 999 """Check the defconfig after savedefconfig 1000 1001 Returns: 1002 Return additional log if moved CONFIGs were removed again by 1003 'make savedefconfig'. 1004 """ 1005 1006 log = '' 1007 1008 with open(self.defconfig) as f: 1009 defconfig_lines = f.readlines() 1010 1011 for (action, value) in self.results: 1012 if action != ACTION_MOVE: 1013 continue 1014 if not value + '\n' in defconfig_lines: 1015 log += color_text(self.options.color, COLOR_YELLOW, 1016 "'%s' was removed by savedefconfig.\n" % 1017 value) 1018 1019 return log 1020 1021 1022 class DatabaseThread(threading.Thread): 1023 """This thread processes results from Slot threads. 1024 1025 It collects the data in the master config directary. There is only one 1026 result thread, and this helps to serialise the build output. 1027 """ 1028 def __init__(self, config_db, db_queue): 1029 """Set up a new result thread 1030 1031 Args: 1032 builder: Builder which will be sent each result 1033 """ 1034 threading.Thread.__init__(self) 1035 self.config_db = config_db 1036 self.db_queue= db_queue 1037 1038 def run(self): 1039 """Called to start up the result thread. 1040 1041 We collect the next result job and pass it on to the build. 1042 """ 1043 while True: 1044 defconfig, configs = self.db_queue.get() 1045 self.config_db[defconfig] = configs 1046 self.db_queue.task_done() 1047 1048 1049 class Slot: 1050 1051 """A slot to store a subprocess. 1052 1053 Each instance of this class handles one subprocess. 1054 This class is useful to control multiple threads 1055 for faster processing. 1056 """ 1057 1058 def __init__(self, toolchains, configs, options, progress, devnull, 1059 make_cmd, reference_src_dir, db_queue): 1060 """Create a new process slot. 1061 1062 Arguments: 1063 toolchains: Toolchains object containing toolchains. 1064 configs: A list of CONFIGs to move. 1065 options: option flags. 1066 progress: A progress indicator. 1067 devnull: A file object of '/dev/null'. 1068 make_cmd: command name of GNU Make. 1069 reference_src_dir: Determine the true starting config state from this 1070 source tree. 1071 db_queue: output queue to write config info for the database 1072 """ 1073 self.toolchains = toolchains 1074 self.options = options 1075 self.progress = progress 1076 self.build_dir = tempfile.mkdtemp() 1077 self.devnull = devnull 1078 self.make_cmd = (make_cmd, 'O=' + self.build_dir) 1079 self.reference_src_dir = reference_src_dir 1080 self.db_queue = db_queue 1081 self.parser = KconfigParser(configs, options, self.build_dir) 1082 self.state = STATE_IDLE 1083 self.failed_boards = set() 1084 self.suspicious_boards = set() 1085 1086 def __del__(self): 1087 """Delete the working directory 1088 1089 This function makes sure the temporary directory is cleaned away 1090 even if Python suddenly dies due to error. It should be done in here 1091 because it is guaranteed the destructor is always invoked when the 1092 instance of the class gets unreferenced. 1093 1094 If the subprocess is still running, wait until it finishes. 1095 """ 1096 if self.state != STATE_IDLE: 1097 while self.ps.poll() == None: 1098 pass 1099 shutil.rmtree(self.build_dir) 1100 1101 def add(self, defconfig): 1102 """Assign a new subprocess for defconfig and add it to the slot. 1103 1104 If the slot is vacant, create a new subprocess for processing the 1105 given defconfig and add it to the slot. Just returns False if 1106 the slot is occupied (i.e. the current subprocess is still running). 1107 1108 Arguments: 1109 defconfig: defconfig name. 1110 1111 Returns: 1112 Return True on success or False on failure 1113 """ 1114 if self.state != STATE_IDLE: 1115 return False 1116 1117 self.defconfig = defconfig 1118 self.log = '' 1119 self.current_src_dir = self.reference_src_dir 1120 self.do_defconfig() 1121 return True 1122 1123 def poll(self): 1124 """Check the status of the subprocess and handle it as needed. 1125 1126 Returns True if the slot is vacant (i.e. in idle state). 1127 If the configuration is successfully finished, assign a new 1128 subprocess to build include/autoconf.mk. 1129 If include/autoconf.mk is generated, invoke the parser to 1130 parse the .config and the include/autoconf.mk, moving 1131 config options to the .config as needed. 1132 If the .config was updated, run "make savedefconfig" to sync 1133 it, update the original defconfig, and then set the slot back 1134 to the idle state. 1135 1136 Returns: 1137 Return True if the subprocess is terminated, False otherwise 1138 """ 1139 if self.state == STATE_IDLE: 1140 return True 1141 1142 if self.ps.poll() == None: 1143 return False 1144 1145 if self.ps.poll() != 0: 1146 self.handle_error() 1147 elif self.state == STATE_DEFCONFIG: 1148 if self.reference_src_dir and not self.current_src_dir: 1149 self.do_savedefconfig() 1150 else: 1151 self.do_autoconf() 1152 elif self.state == STATE_AUTOCONF: 1153 if self.current_src_dir: 1154 self.current_src_dir = None 1155 self.do_defconfig() 1156 elif self.options.build_db: 1157 self.do_build_db() 1158 else: 1159 self.do_savedefconfig() 1160 elif self.state == STATE_SAVEDEFCONFIG: 1161 self.update_defconfig() 1162 else: 1163 sys.exit("Internal Error. This should not happen.") 1164 1165 return True if self.state == STATE_IDLE else False 1166 1167 def handle_error(self): 1168 """Handle error cases.""" 1169 1170 self.log += color_text(self.options.color, COLOR_LIGHT_RED, 1171 "Failed to process.\n") 1172 if self.options.verbose: 1173 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN, 1174 self.ps.stderr.read()) 1175 self.finish(False) 1176 1177 def do_defconfig(self): 1178 """Run 'make <board>_defconfig' to create the .config file.""" 1179 1180 cmd = list(self.make_cmd) 1181 cmd.append(self.defconfig) 1182 self.ps = subprocess.Popen(cmd, stdout=self.devnull, 1183 stderr=subprocess.PIPE, 1184 cwd=self.current_src_dir) 1185 self.state = STATE_DEFCONFIG 1186 1187 def do_autoconf(self): 1188 """Run 'make AUTO_CONF_PATH'.""" 1189 1190 arch = self.parser.get_arch() 1191 try: 1192 toolchain = self.toolchains.Select(arch) 1193 except ValueError: 1194 self.log += color_text(self.options.color, COLOR_YELLOW, 1195 "Tool chain for '%s' is missing. Do nothing.\n" % arch) 1196 self.finish(False) 1197 return 1198 env = toolchain.MakeEnvironment(False) 1199 1200 cmd = list(self.make_cmd) 1201 cmd.append('KCONFIG_IGNORE_DUPLICATES=1') 1202 cmd.append(AUTO_CONF_PATH) 1203 self.ps = subprocess.Popen(cmd, stdout=self.devnull, env=env, 1204 stderr=subprocess.PIPE, 1205 cwd=self.current_src_dir) 1206 self.state = STATE_AUTOCONF 1207 1208 def do_build_db(self): 1209 """Add the board to the database""" 1210 configs = {} 1211 with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd: 1212 for line in fd.readlines(): 1213 if line.startswith('CONFIG'): 1214 config, value = line.split('=', 1) 1215 configs[config] = value.rstrip() 1216 self.db_queue.put([self.defconfig, configs]) 1217 self.finish(True) 1218 1219 def do_savedefconfig(self): 1220 """Update the .config and run 'make savedefconfig'.""" 1221 1222 (updated, suspicious, log) = self.parser.update_dotconfig() 1223 if suspicious: 1224 self.suspicious_boards.add(self.defconfig) 1225 self.log += log 1226 1227 if not self.options.force_sync and not updated: 1228 self.finish(True) 1229 return 1230 if updated: 1231 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN, 1232 "Syncing by savedefconfig...\n") 1233 else: 1234 self.log += "Syncing by savedefconfig (forced by option)...\n" 1235 1236 cmd = list(self.make_cmd) 1237 cmd.append('savedefconfig') 1238 self.ps = subprocess.Popen(cmd, stdout=self.devnull, 1239 stderr=subprocess.PIPE) 1240 self.state = STATE_SAVEDEFCONFIG 1241 1242 def update_defconfig(self): 1243 """Update the input defconfig and go back to the idle state.""" 1244 1245 log = self.parser.check_defconfig() 1246 if log: 1247 self.suspicious_boards.add(self.defconfig) 1248 self.log += log 1249 orig_defconfig = os.path.join('configs', self.defconfig) 1250 new_defconfig = os.path.join(self.build_dir, 'defconfig') 1251 updated = not filecmp.cmp(orig_defconfig, new_defconfig) 1252 1253 if updated: 1254 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE, 1255 "defconfig was updated.\n") 1256 1257 if not self.options.dry_run and updated: 1258 shutil.move(new_defconfig, orig_defconfig) 1259 self.finish(True) 1260 1261 def finish(self, success): 1262 """Display log along with progress and go to the idle state. 1263 1264 Arguments: 1265 success: Should be True when the defconfig was processed 1266 successfully, or False when it fails. 1267 """ 1268 # output at least 30 characters to hide the "* defconfigs out of *". 1269 log = self.defconfig.ljust(30) + '\n' 1270 1271 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ]) 1272 # Some threads are running in parallel. 1273 # Print log atomically to not mix up logs from different threads. 1274 print >> (sys.stdout if success else sys.stderr), log 1275 1276 if not success: 1277 if self.options.exit_on_error: 1278 sys.exit("Exit on error.") 1279 # If --exit-on-error flag is not set, skip this board and continue. 1280 # Record the failed board. 1281 self.failed_boards.add(self.defconfig) 1282 1283 self.progress.inc() 1284 self.progress.show() 1285 self.state = STATE_IDLE 1286 1287 def get_failed_boards(self): 1288 """Returns a set of failed boards (defconfigs) in this slot. 1289 """ 1290 return self.failed_boards 1291 1292 def get_suspicious_boards(self): 1293 """Returns a set of boards (defconfigs) with possible misconversion. 1294 """ 1295 return self.suspicious_boards - self.failed_boards 1296 1297 class Slots: 1298 1299 """Controller of the array of subprocess slots.""" 1300 1301 def __init__(self, toolchains, configs, options, progress, 1302 reference_src_dir, db_queue): 1303 """Create a new slots controller. 1304 1305 Arguments: 1306 toolchains: Toolchains object containing toolchains. 1307 configs: A list of CONFIGs to move. 1308 options: option flags. 1309 progress: A progress indicator. 1310 reference_src_dir: Determine the true starting config state from this 1311 source tree. 1312 db_queue: output queue to write config info for the database 1313 """ 1314 self.options = options 1315 self.slots = [] 1316 devnull = get_devnull() 1317 make_cmd = get_make_cmd() 1318 for i in range(options.jobs): 1319 self.slots.append(Slot(toolchains, configs, options, progress, 1320 devnull, make_cmd, reference_src_dir, 1321 db_queue)) 1322 1323 def add(self, defconfig): 1324 """Add a new subprocess if a vacant slot is found. 1325 1326 Arguments: 1327 defconfig: defconfig name to be put into. 1328 1329 Returns: 1330 Return True on success or False on failure 1331 """ 1332 for slot in self.slots: 1333 if slot.add(defconfig): 1334 return True 1335 return False 1336 1337 def available(self): 1338 """Check if there is a vacant slot. 1339 1340 Returns: 1341 Return True if at lease one vacant slot is found, False otherwise. 1342 """ 1343 for slot in self.slots: 1344 if slot.poll(): 1345 return True 1346 return False 1347 1348 def empty(self): 1349 """Check if all slots are vacant. 1350 1351 Returns: 1352 Return True if all the slots are vacant, False otherwise. 1353 """ 1354 ret = True 1355 for slot in self.slots: 1356 if not slot.poll(): 1357 ret = False 1358 return ret 1359 1360 def show_failed_boards(self): 1361 """Display all of the failed boards (defconfigs).""" 1362 boards = set() 1363 output_file = 'moveconfig.failed' 1364 1365 for slot in self.slots: 1366 boards |= slot.get_failed_boards() 1367 1368 if boards: 1369 boards = '\n'.join(boards) + '\n' 1370 msg = "The following boards were not processed due to error:\n" 1371 msg += boards 1372 msg += "(the list has been saved in %s)\n" % output_file 1373 print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, 1374 msg) 1375 1376 with open(output_file, 'w') as f: 1377 f.write(boards) 1378 1379 def show_suspicious_boards(self): 1380 """Display all boards (defconfigs) with possible misconversion.""" 1381 boards = set() 1382 output_file = 'moveconfig.suspicious' 1383 1384 for slot in self.slots: 1385 boards |= slot.get_suspicious_boards() 1386 1387 if boards: 1388 boards = '\n'.join(boards) + '\n' 1389 msg = "The following boards might have been converted incorrectly.\n" 1390 msg += "It is highly recommended to check them manually:\n" 1391 msg += boards 1392 msg += "(the list has been saved in %s)\n" % output_file 1393 print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW, 1394 msg) 1395 1396 with open(output_file, 'w') as f: 1397 f.write(boards) 1398 1399 class ReferenceSource: 1400 1401 """Reference source against which original configs should be parsed.""" 1402 1403 def __init__(self, commit): 1404 """Create a reference source directory based on a specified commit. 1405 1406 Arguments: 1407 commit: commit to git-clone 1408 """ 1409 self.src_dir = tempfile.mkdtemp() 1410 print "Cloning git repo to a separate work directory..." 1411 subprocess.check_output(['git', 'clone', os.getcwd(), '.'], 1412 cwd=self.src_dir) 1413 print "Checkout '%s' to build the original autoconf.mk." % \ 1414 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip() 1415 subprocess.check_output(['git', 'checkout', commit], 1416 stderr=subprocess.STDOUT, cwd=self.src_dir) 1417 1418 def __del__(self): 1419 """Delete the reference source directory 1420 1421 This function makes sure the temporary directory is cleaned away 1422 even if Python suddenly dies due to error. It should be done in here 1423 because it is guaranteed the destructor is always invoked when the 1424 instance of the class gets unreferenced. 1425 """ 1426 shutil.rmtree(self.src_dir) 1427 1428 def get_dir(self): 1429 """Return the absolute path to the reference source directory.""" 1430 1431 return self.src_dir 1432 1433 def move_config(toolchains, configs, options, db_queue): 1434 """Move config options to defconfig files. 1435 1436 Arguments: 1437 configs: A list of CONFIGs to move. 1438 options: option flags 1439 """ 1440 if len(configs) == 0: 1441 if options.force_sync: 1442 print 'No CONFIG is specified. You are probably syncing defconfigs.', 1443 elif options.build_db: 1444 print 'Building %s database' % CONFIG_DATABASE 1445 else: 1446 print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.', 1447 else: 1448 print 'Move ' + ', '.join(configs), 1449 print '(jobs: %d)\n' % options.jobs 1450 1451 if options.git_ref: 1452 reference_src = ReferenceSource(options.git_ref) 1453 reference_src_dir = reference_src.get_dir() 1454 else: 1455 reference_src_dir = None 1456 1457 if options.defconfigs: 1458 defconfigs = get_matched_defconfigs(options.defconfigs) 1459 else: 1460 defconfigs = get_all_defconfigs() 1461 1462 progress = Progress(len(defconfigs)) 1463 slots = Slots(toolchains, configs, options, progress, reference_src_dir, 1464 db_queue) 1465 1466 # Main loop to process defconfig files: 1467 # Add a new subprocess into a vacant slot. 1468 # Sleep if there is no available slot. 1469 for defconfig in defconfigs: 1470 while not slots.add(defconfig): 1471 while not slots.available(): 1472 # No available slot: sleep for a while 1473 time.sleep(SLEEP_TIME) 1474 1475 # wait until all the subprocesses finish 1476 while not slots.empty(): 1477 time.sleep(SLEEP_TIME) 1478 1479 print '' 1480 slots.show_failed_boards() 1481 slots.show_suspicious_boards() 1482 1483 def find_kconfig_rules(kconf, config, imply_config): 1484 """Check whether a config has a 'select' or 'imply' keyword 1485 1486 Args: 1487 kconf: Kconfig.Config object 1488 config: Name of config to check (without CONFIG_ prefix) 1489 imply_config: Implying config (without CONFIG_ prefix) which may or 1490 may not have an 'imply' for 'config') 1491 1492 Returns: 1493 Symbol object for 'config' if found, else None 1494 """ 1495 sym = kconf.get_symbol(imply_config) 1496 if sym: 1497 for sel in sym.get_selected_symbols() | sym.get_implied_symbols(): 1498 if sel.get_name() == config: 1499 return sym 1500 return None 1501 1502 def check_imply_rule(kconf, config, imply_config): 1503 """Check if we can add an 'imply' option 1504 1505 This finds imply_config in the Kconfig and looks to see if it is possible 1506 to add an 'imply' for 'config' to that part of the Kconfig. 1507 1508 Args: 1509 kconf: Kconfig.Config object 1510 config: Name of config to check (without CONFIG_ prefix) 1511 imply_config: Implying config (without CONFIG_ prefix) which may or 1512 may not have an 'imply' for 'config') 1513 1514 Returns: 1515 tuple: 1516 filename of Kconfig file containing imply_config, or None if none 1517 line number within the Kconfig file, or 0 if none 1518 message indicating the result 1519 """ 1520 sym = kconf.get_symbol(imply_config) 1521 if not sym: 1522 return 'cannot find sym' 1523 locs = sym.get_def_locations() 1524 if len(locs) != 1: 1525 return '%d locations' % len(locs) 1526 fname, linenum = locs[0] 1527 cwd = os.getcwd() 1528 if cwd and fname.startswith(cwd): 1529 fname = fname[len(cwd) + 1:] 1530 file_line = ' at %s:%d' % (fname, linenum) 1531 with open(fname) as fd: 1532 data = fd.read().splitlines() 1533 if data[linenum - 1] != 'config %s' % imply_config: 1534 return None, 0, 'bad sym format %s%s' % (data[linenum], file_line) 1535 return fname, linenum, 'adding%s' % file_line 1536 1537 def add_imply_rule(config, fname, linenum): 1538 """Add a new 'imply' option to a Kconfig 1539 1540 Args: 1541 config: config option to add an imply for (without CONFIG_ prefix) 1542 fname: Kconfig filename to update 1543 linenum: Line number to place the 'imply' before 1544 1545 Returns: 1546 Message indicating the result 1547 """ 1548 file_line = ' at %s:%d' % (fname, linenum) 1549 data = open(fname).read().splitlines() 1550 linenum -= 1 1551 1552 for offset, line in enumerate(data[linenum:]): 1553 if line.strip().startswith('help') or not line: 1554 data.insert(linenum + offset, '\timply %s' % config) 1555 with open(fname, 'w') as fd: 1556 fd.write('\n'.join(data) + '\n') 1557 return 'added%s' % file_line 1558 1559 return 'could not insert%s' 1560 1561 (IMPLY_MIN_2, IMPLY_TARGET, IMPLY_CMD, IMPLY_NON_ARCH_BOARD) = ( 1562 1, 2, 4, 8) 1563 1564 IMPLY_FLAGS = { 1565 'min2': [IMPLY_MIN_2, 'Show options which imply >2 boards (normally >5)'], 1566 'target': [IMPLY_TARGET, 'Allow CONFIG_TARGET_... options to imply'], 1567 'cmd': [IMPLY_CMD, 'Allow CONFIG_CMD_... to imply'], 1568 'non-arch-board': [ 1569 IMPLY_NON_ARCH_BOARD, 1570 'Allow Kconfig options outside arch/ and /board/ to imply'], 1571 }; 1572 1573 def do_imply_config(config_list, add_imply, imply_flags, skip_added, 1574 check_kconfig=True, find_superset=False): 1575 """Find CONFIG options which imply those in the list 1576 1577 Some CONFIG options can be implied by others and this can help to reduce 1578 the size of the defconfig files. For example, CONFIG_X86 implies 1579 CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and 1580 all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to 1581 each of the x86 defconfig files. 1582 1583 This function uses the moveconfig database to find such options. It 1584 displays a list of things that could possibly imply those in the list. 1585 The algorithm ignores any that start with CONFIG_TARGET since these 1586 typically refer to only a few defconfigs (often one). It also does not 1587 display a config with less than 5 defconfigs. 1588 1589 The algorithm works using sets. For each target config in config_list: 1590 - Get the set 'defconfigs' which use that target config 1591 - For each config (from a list of all configs): 1592 - Get the set 'imply_defconfig' of defconfigs which use that config 1593 - 1594 - If imply_defconfigs contains anything not in defconfigs then 1595 this config does not imply the target config 1596 1597 Params: 1598 config_list: List of CONFIG options to check (each a string) 1599 add_imply: Automatically add an 'imply' for each config. 1600 imply_flags: Flags which control which implying configs are allowed 1601 (IMPLY_...) 1602 skip_added: Don't show options which already have an imply added. 1603 check_kconfig: Check if implied symbols already have an 'imply' or 1604 'select' for the target config, and show this information if so. 1605 find_superset: True to look for configs which are a superset of those 1606 already found. So for example if CONFIG_EXYNOS5 implies an option, 1607 but CONFIG_EXYNOS covers a larger set of defconfigs and also 1608 implies that option, this will drop the former in favour of the 1609 latter. In practice this option has not proved very used. 1610 1611 Note the terminoloy: 1612 config - a CONFIG_XXX options (a string, e.g. 'CONFIG_CMD_EEPROM') 1613 defconfig - a defconfig file (a string, e.g. 'configs/snow_defconfig') 1614 """ 1615 kconf = KconfigScanner().conf if check_kconfig else None 1616 if add_imply and add_imply != 'all': 1617 add_imply = add_imply.split() 1618 1619 # key is defconfig name, value is dict of (CONFIG_xxx, value) 1620 config_db = {} 1621 1622 # Holds a dict containing the set of defconfigs that contain each config 1623 # key is config, value is set of defconfigs using that config 1624 defconfig_db = collections.defaultdict(set) 1625 1626 # Set of all config options we have seen 1627 all_configs = set() 1628 1629 # Set of all defconfigs we have seen 1630 all_defconfigs = set() 1631 1632 # Read in the database 1633 configs = {} 1634 with open(CONFIG_DATABASE) as fd: 1635 for line in fd.readlines(): 1636 line = line.rstrip() 1637 if not line: # Separator between defconfigs 1638 config_db[defconfig] = configs 1639 all_defconfigs.add(defconfig) 1640 configs = {} 1641 elif line[0] == ' ': # CONFIG line 1642 config, value = line.strip().split('=', 1) 1643 configs[config] = value 1644 defconfig_db[config].add(defconfig) 1645 all_configs.add(config) 1646 else: # New defconfig 1647 defconfig = line 1648 1649 # Work through each target config option in tern, independently 1650 for config in config_list: 1651 defconfigs = defconfig_db.get(config) 1652 if not defconfigs: 1653 print '%s not found in any defconfig' % config 1654 continue 1655 1656 # Get the set of defconfigs without this one (since a config cannot 1657 # imply itself) 1658 non_defconfigs = all_defconfigs - defconfigs 1659 num_defconfigs = len(defconfigs) 1660 print '%s found in %d/%d defconfigs' % (config, num_defconfigs, 1661 len(all_configs)) 1662 1663 # This will hold the results: key=config, value=defconfigs containing it 1664 imply_configs = {} 1665 rest_configs = all_configs - set([config]) 1666 1667 # Look at every possible config, except the target one 1668 for imply_config in rest_configs: 1669 if 'ERRATUM' in imply_config: 1670 continue 1671 if not (imply_flags & IMPLY_CMD): 1672 if 'CONFIG_CMD' in imply_config: 1673 continue 1674 if not (imply_flags & IMPLY_TARGET): 1675 if 'CONFIG_TARGET' in imply_config: 1676 continue 1677 1678 # Find set of defconfigs that have this config 1679 imply_defconfig = defconfig_db[imply_config] 1680 1681 # Get the intersection of this with defconfigs containing the 1682 # target config 1683 common_defconfigs = imply_defconfig & defconfigs 1684 1685 # Get the set of defconfigs containing this config which DO NOT 1686 # also contain the taret config. If this set is non-empty it means 1687 # that this config affects other defconfigs as well as (possibly) 1688 # the ones affected by the target config. This means it implies 1689 # things we don't want to imply. 1690 not_common_defconfigs = imply_defconfig & non_defconfigs 1691 if not_common_defconfigs: 1692 continue 1693 1694 # If there are common defconfigs, imply_config may be useful 1695 if common_defconfigs: 1696 skip = False 1697 if find_superset: 1698 for prev in imply_configs.keys(): 1699 prev_count = len(imply_configs[prev]) 1700 count = len(common_defconfigs) 1701 if (prev_count > count and 1702 (imply_configs[prev] & common_defconfigs == 1703 common_defconfigs)): 1704 # skip imply_config because prev is a superset 1705 skip = True 1706 break 1707 elif count > prev_count: 1708 # delete prev because imply_config is a superset 1709 del imply_configs[prev] 1710 if not skip: 1711 imply_configs[imply_config] = common_defconfigs 1712 1713 # Now we have a dict imply_configs of configs which imply each config 1714 # The value of each dict item is the set of defconfigs containing that 1715 # config. Rank them so that we print the configs that imply the largest 1716 # number of defconfigs first. 1717 ranked_iconfigs = sorted(imply_configs, 1718 key=lambda k: len(imply_configs[k]), reverse=True) 1719 kconfig_info = '' 1720 cwd = os.getcwd() 1721 add_list = collections.defaultdict(list) 1722 for iconfig in ranked_iconfigs: 1723 num_common = len(imply_configs[iconfig]) 1724 1725 # Don't bother if there are less than 5 defconfigs affected. 1726 if num_common < (2 if imply_flags & IMPLY_MIN_2 else 5): 1727 continue 1728 missing = defconfigs - imply_configs[iconfig] 1729 missing_str = ', '.join(missing) if missing else 'all' 1730 missing_str = '' 1731 show = True 1732 if kconf: 1733 sym = find_kconfig_rules(kconf, config[CONFIG_LEN:], 1734 iconfig[CONFIG_LEN:]) 1735 kconfig_info = '' 1736 if sym: 1737 locs = sym.get_def_locations() 1738 if len(locs) == 1: 1739 fname, linenum = locs[0] 1740 if cwd and fname.startswith(cwd): 1741 fname = fname[len(cwd) + 1:] 1742 kconfig_info = '%s:%d' % (fname, linenum) 1743 if skip_added: 1744 show = False 1745 else: 1746 sym = kconf.get_symbol(iconfig[CONFIG_LEN:]) 1747 fname = '' 1748 if sym: 1749 locs = sym.get_def_locations() 1750 if len(locs) == 1: 1751 fname, linenum = locs[0] 1752 if cwd and fname.startswith(cwd): 1753 fname = fname[len(cwd) + 1:] 1754 in_arch_board = not sym or (fname.startswith('arch') or 1755 fname.startswith('board')) 1756 if (not in_arch_board and 1757 not (imply_flags & IMPLY_NON_ARCH_BOARD)): 1758 continue 1759 1760 if add_imply and (add_imply == 'all' or 1761 iconfig in add_imply): 1762 fname, linenum, kconfig_info = (check_imply_rule(kconf, 1763 config[CONFIG_LEN:], iconfig[CONFIG_LEN:])) 1764 if fname: 1765 add_list[fname].append(linenum) 1766 1767 if show and kconfig_info != 'skip': 1768 print '%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30), 1769 kconfig_info, missing_str) 1770 1771 # Having collected a list of things to add, now we add them. We process 1772 # each file from the largest line number to the smallest so that 1773 # earlier additions do not affect our line numbers. E.g. if we added an 1774 # imply at line 20 it would change the position of each line after 1775 # that. 1776 for fname, linenums in add_list.iteritems(): 1777 for linenum in sorted(linenums, reverse=True): 1778 add_imply_rule(config[CONFIG_LEN:], fname, linenum) 1779 1780 1781 def main(): 1782 try: 1783 cpu_count = multiprocessing.cpu_count() 1784 except NotImplementedError: 1785 cpu_count = 1 1786 1787 parser = optparse.OptionParser() 1788 # Add options here 1789 parser.add_option('-a', '--add-imply', type='string', default='', 1790 help='comma-separated list of CONFIG options to add ' 1791 "an 'imply' statement to for the CONFIG in -i") 1792 parser.add_option('-A', '--skip-added', action='store_true', default=False, 1793 help="don't show options which are already marked as " 1794 'implying others') 1795 parser.add_option('-b', '--build-db', action='store_true', default=False, 1796 help='build a CONFIG database') 1797 parser.add_option('-c', '--color', action='store_true', default=False, 1798 help='display the log in color') 1799 parser.add_option('-C', '--commit', action='store_true', default=False, 1800 help='Create a git commit for the operation') 1801 parser.add_option('-d', '--defconfigs', type='string', 1802 help='a file containing a list of defconfigs to move, ' 1803 "one per line (for example 'snow_defconfig') " 1804 "or '-' to read from stdin") 1805 parser.add_option('-i', '--imply', action='store_true', default=False, 1806 help='find options which imply others') 1807 parser.add_option('-I', '--imply-flags', type='string', default='', 1808 help="control the -i option ('help' for help") 1809 parser.add_option('-n', '--dry-run', action='store_true', default=False, 1810 help='perform a trial run (show log with no changes)') 1811 parser.add_option('-e', '--exit-on-error', action='store_true', 1812 default=False, 1813 help='exit immediately on any error') 1814 parser.add_option('-s', '--force-sync', action='store_true', default=False, 1815 help='force sync by savedefconfig') 1816 parser.add_option('-S', '--spl', action='store_true', default=False, 1817 help='parse config options defined for SPL build') 1818 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', 1819 action='store_true', default=False, 1820 help='only cleanup the headers') 1821 parser.add_option('-j', '--jobs', type='int', default=cpu_count, 1822 help='the number of jobs to run simultaneously') 1823 parser.add_option('-r', '--git-ref', type='string', 1824 help='the git ref to clone for building the autoconf.mk') 1825 parser.add_option('-y', '--yes', action='store_true', default=False, 1826 help="respond 'yes' to any prompts") 1827 parser.add_option('-v', '--verbose', action='store_true', default=False, 1828 help='show any build errors as boards are built') 1829 parser.usage += ' CONFIG ...' 1830 1831 (options, configs) = parser.parse_args() 1832 1833 if len(configs) == 0 and not any((options.force_sync, options.build_db, 1834 options.imply)): 1835 parser.print_usage() 1836 sys.exit(1) 1837 1838 # prefix the option name with CONFIG_ if missing 1839 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config 1840 for config in configs ] 1841 1842 check_top_directory() 1843 1844 if options.imply: 1845 imply_flags = 0 1846 if options.imply_flags == 'all': 1847 imply_flags = -1 1848 1849 elif options.imply_flags: 1850 for flag in options.imply_flags.split(','): 1851 bad = flag not in IMPLY_FLAGS 1852 if bad: 1853 print "Invalid flag '%s'" % flag 1854 if flag == 'help' or bad: 1855 print "Imply flags: (separate with ',')" 1856 for name, info in IMPLY_FLAGS.iteritems(): 1857 print ' %-15s: %s' % (name, info[1]) 1858 parser.print_usage() 1859 sys.exit(1) 1860 imply_flags |= IMPLY_FLAGS[flag][0] 1861 1862 do_imply_config(configs, options.add_imply, imply_flags, 1863 options.skip_added) 1864 return 1865 1866 config_db = {} 1867 db_queue = Queue.Queue() 1868 t = DatabaseThread(config_db, db_queue) 1869 t.setDaemon(True) 1870 t.start() 1871 1872 if not options.cleanup_headers_only: 1873 check_clean_directory() 1874 bsettings.Setup('') 1875 toolchains = toolchain.Toolchains() 1876 toolchains.GetSettings() 1877 toolchains.Scan(verbose=False) 1878 move_config(toolchains, configs, options, db_queue) 1879 db_queue.join() 1880 1881 if configs: 1882 cleanup_headers(configs, options) 1883 cleanup_extra_options(configs, options) 1884 cleanup_whitelist(configs, options) 1885 cleanup_readme(configs, options) 1886 1887 if options.commit: 1888 subprocess.call(['git', 'add', '-u']) 1889 if configs: 1890 msg = 'Convert %s %sto Kconfig' % (configs[0], 1891 'et al ' if len(configs) > 1 else '') 1892 msg += ('\n\nThis converts the following to Kconfig:\n %s\n' % 1893 '\n '.join(configs)) 1894 else: 1895 msg = 'configs: Resync with savedefconfig' 1896 msg += '\n\nRsync all defconfig files using moveconfig.py' 1897 subprocess.call(['git', 'commit', '-s', '-m', msg]) 1898 1899 if options.build_db: 1900 with open(CONFIG_DATABASE, 'w') as fd: 1901 for defconfig, configs in config_db.iteritems(): 1902 fd.write('%s\n' % defconfig) 1903 for config in sorted(configs.keys()): 1904 fd.write(' %s=%s\n' % (config, configs[config])) 1905 fd.write('\n') 1906 1907 if __name__ == '__main__': 1908 main() 1909