xref: /openbmc/u-boot/tools/moveconfig.py (revision 8b7d51249eca113c4965a7c417f33d7eb569434b)
1#!/usr/bin/env python2
2#
3# Author: Masahiro Yamada <yamada.masahiro@socionext.com>
4#
5# SPDX-License-Identifier:	GPL-2.0+
6#
7
8"""
9Move config options from headers to defconfig files.
10
11Since Kconfig was introduced to U-Boot, we have worked on moving
12config options from headers to Kconfig (defconfig).
13
14This tool intends to help this tremendous work.
15
16
17Usage
18-----
19
20First, you must edit the Kconfig to add the menu entries for the configs
21you are moving.
22
23And then run this tool giving CONFIG names you want to move.
24For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
25simply type as follows:
26
27  $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
28
29The tool walks through all the defconfig files and move the given CONFIGs.
30
31The log is also displayed on the terminal.
32
33The log is printed for each defconfig as follows:
34
35<defconfig_name>
36    <action1>
37    <action2>
38    <action3>
39    ...
40
41<defconfig_name> is the name of the defconfig.
42
43<action*> shows what the tool did for that defconfig.
44It looks like one of the following:
45
46 - Move 'CONFIG_... '
47   This config option was moved to the defconfig
48
49 - CONFIG_... is not defined in Kconfig.  Do nothing.
50   The entry for this CONFIG was not found in Kconfig.
51   There are two common cases:
52     - You forgot to create an entry for the CONFIG before running
53       this tool, or made a typo in a CONFIG passed to this tool.
54     - The entry was hidden due to unmet 'depends on'.
55       This is correct behavior.
56
57 - 'CONFIG_...' is the same as the define in Kconfig.  Do nothing.
58   The define in the config header matched the one in Kconfig.
59   We do not need to touch it.
60
61 - Undefined.  Do nothing.
62   This config option was not found in the config header.
63   Nothing to do.
64
65 - Compiler is missing.  Do nothing.
66   The compiler specified for this architecture was not found
67   in your PATH environment.
68   (If -e option is passed, the tool exits immediately.)
69
70 - Failed to process.
71   An error occurred during processing this defconfig.  Skipped.
72   (If -e option is passed, the tool exits immediately on error.)
73
74Finally, you will be asked, Clean up headers? [y/n]:
75
76If you say 'y' here, the unnecessary config defines are removed
77from the config headers (include/configs/*.h).
78It just uses the regex method, so you should not rely on it.
79Just in case, please do 'git diff' to see what happened.
80
81
82How does it work?
83-----------------
84
85This tool runs configuration and builds include/autoconf.mk for every
86defconfig.  The config options defined in Kconfig appear in the .config
87file (unless they are hidden because of unmet dependency.)
88On the other hand, the config options defined by board headers are seen
89in include/autoconf.mk.  The tool looks for the specified options in both
90of them to decide the appropriate action for the options.  If the given
91config option is found in the .config, but its value does not match the
92one from the board header, the config option in the .config is replaced
93with the define in the board header.  Then, the .config is synced by
94"make savedefconfig" and the defconfig is updated with it.
95
96For faster processing, this tool handles multi-threading.  It creates
97separate build directories where the out-of-tree build is run.  The
98temporary build directories are automatically created and deleted as
99needed.  The number of threads are chosen based on the number of the CPU
100cores of your system although you can change it via -j (--jobs) option.
101
102
103Toolchains
104----------
105
106Appropriate toolchain are necessary to generate include/autoconf.mk
107for all the architectures supported by U-Boot.  Most of them are available
108at the kernel.org site, some are not provided by kernel.org.
109
110The default per-arch CROSS_COMPILE used by this tool is specified by
111the list below, CROSS_COMPILE.  You may wish to update the list to
112use your own.  Instead of modifying the list directly, you can give
113them via environments.
114
115
116Available options
117-----------------
118
119 -c, --color
120   Surround each portion of the log with escape sequences to display it
121   in color on the terminal.
122
123 -d, --defconfigs
124  Specify a file containing a list of defconfigs to move
125
126 -n, --dry-run
127   Perform a trial run that does not make any changes.  It is useful to
128   see what is going to happen before one actually runs it.
129
130 -e, --exit-on-error
131   Exit immediately if Make exits with a non-zero status while processing
132   a defconfig file.
133
134 -s, --force-sync
135   Do "make savedefconfig" forcibly for all the defconfig files.
136   If not specified, "make savedefconfig" only occurs for cases
137   where at least one CONFIG was moved.
138
139 -H, --headers-only
140   Only cleanup the headers; skip the defconfig processing
141
142 -j, --jobs
143   Specify the number of threads to run simultaneously.  If not specified,
144   the number of threads is the same as the number of CPU cores.
145
146 -r, --git-ref
147   Specify the git ref to clone for building the autoconf.mk. If unspecified
148   use the CWD. This is useful for when changes to the Kconfig affect the
149   default values and you want to capture the state of the defconfig from
150   before that change was in effect. If in doubt, specify a ref pre-Kconfig
151   changes (use HEAD if Kconfig changes are not committed). Worst case it will
152   take a bit longer to run, but will always do the right thing.
153
154 -v, --verbose
155   Show any build errors as boards are built
156
157To see the complete list of supported options, run
158
159  $ tools/moveconfig.py -h
160
161"""
162
163import copy
164import difflib
165import filecmp
166import fnmatch
167import multiprocessing
168import optparse
169import os
170import re
171import shutil
172import subprocess
173import sys
174import tempfile
175import time
176
177SHOW_GNU_MAKE = 'scripts/show-gnu-make'
178SLEEP_TIME=0.03
179
180# Here is the list of cross-tools I use.
181# Most of them are available at kernel.org
182# (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following:
183# arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
184# blackfin: http://sourceforge.net/projects/adi-toolchain/files/
185# nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
186# nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
187# sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
188#
189# openrisc kernel.org toolchain is out of date, download latest one from
190# http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions
191CROSS_COMPILE = {
192    'arc': 'arc-linux-',
193    'aarch64': 'aarch64-linux-',
194    'arm': 'arm-unknown-linux-gnueabi-',
195    'avr32': 'avr32-linux-',
196    'blackfin': 'bfin-elf-',
197    'm68k': 'm68k-linux-',
198    'microblaze': 'microblaze-linux-',
199    'mips': 'mips-linux-',
200    'nds32': 'nds32le-linux-',
201    'nios2': 'nios2-linux-gnu-',
202    'openrisc': 'or1k-elf-',
203    'powerpc': 'powerpc-linux-',
204    'sh': 'sh-linux-gnu-',
205    'sparc': 'sparc-linux-',
206    'x86': 'i386-linux-',
207    'xtensa': 'xtensa-linux-'
208}
209
210STATE_IDLE = 0
211STATE_DEFCONFIG = 1
212STATE_AUTOCONF = 2
213STATE_SAVEDEFCONFIG = 3
214
215ACTION_MOVE = 0
216ACTION_NO_ENTRY = 1
217ACTION_NO_CHANGE = 2
218
219COLOR_BLACK        = '0;30'
220COLOR_RED          = '0;31'
221COLOR_GREEN        = '0;32'
222COLOR_BROWN        = '0;33'
223COLOR_BLUE         = '0;34'
224COLOR_PURPLE       = '0;35'
225COLOR_CYAN         = '0;36'
226COLOR_LIGHT_GRAY   = '0;37'
227COLOR_DARK_GRAY    = '1;30'
228COLOR_LIGHT_RED    = '1;31'
229COLOR_LIGHT_GREEN  = '1;32'
230COLOR_YELLOW       = '1;33'
231COLOR_LIGHT_BLUE   = '1;34'
232COLOR_LIGHT_PURPLE = '1;35'
233COLOR_LIGHT_CYAN   = '1;36'
234COLOR_WHITE        = '1;37'
235
236### helper functions ###
237def get_devnull():
238    """Get the file object of '/dev/null' device."""
239    try:
240        devnull = subprocess.DEVNULL # py3k
241    except AttributeError:
242        devnull = open(os.devnull, 'wb')
243    return devnull
244
245def check_top_directory():
246    """Exit if we are not at the top of source directory."""
247    for f in ('README', 'Licenses'):
248        if not os.path.exists(f):
249            sys.exit('Please run at the top of source directory.')
250
251def check_clean_directory():
252    """Exit if the source tree is not clean."""
253    for f in ('.config', 'include/config'):
254        if os.path.exists(f):
255            sys.exit("source tree is not clean, please run 'make mrproper'")
256
257def get_make_cmd():
258    """Get the command name of GNU Make.
259
260    U-Boot needs GNU Make for building, but the command name is not
261    necessarily "make". (for example, "gmake" on FreeBSD).
262    Returns the most appropriate command name on your system.
263    """
264    process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
265    ret = process.communicate()
266    if process.returncode:
267        sys.exit('GNU Make not found')
268    return ret[0].rstrip()
269
270def get_all_defconfigs():
271    """Get all the defconfig files under the configs/ directory."""
272    defconfigs = []
273    for (dirpath, dirnames, filenames) in os.walk('configs'):
274        dirpath = dirpath[len('configs') + 1:]
275        for filename in fnmatch.filter(filenames, '*_defconfig'):
276            defconfigs.append(os.path.join(dirpath, filename))
277
278    return defconfigs
279
280def color_text(color_enabled, color, string):
281    """Return colored string."""
282    if color_enabled:
283        # LF should not be surrounded by the escape sequence.
284        # Otherwise, additional whitespace or line-feed might be printed.
285        return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
286                           for s in string.split('\n') ])
287    else:
288        return string
289
290def show_diff(a, b, file_path, color_enabled):
291    """Show unidified diff.
292
293    Arguments:
294      a: A list of lines (before)
295      b: A list of lines (after)
296      file_path: Path to the file
297      color_enabled: Display the diff in color
298    """
299
300    diff = difflib.unified_diff(a, b,
301                                fromfile=os.path.join('a', file_path),
302                                tofile=os.path.join('b', file_path))
303
304    for line in diff:
305        if line[0] == '-' and line[1] != '-':
306            print color_text(color_enabled, COLOR_RED, line),
307        elif line[0] == '+' and line[1] != '+':
308            print color_text(color_enabled, COLOR_GREEN, line),
309        else:
310            print line,
311
312def update_cross_compile(color_enabled):
313    """Update per-arch CROSS_COMPILE via environment variables
314
315    The default CROSS_COMPILE values are available
316    in the CROSS_COMPILE list above.
317
318    You can override them via environment variables
319    CROSS_COMPILE_{ARCH}.
320
321    For example, if you want to override toolchain prefixes
322    for ARM and PowerPC, you can do as follows in your shell:
323
324    export CROSS_COMPILE_ARM=...
325    export CROSS_COMPILE_POWERPC=...
326
327    Then, this function checks if specified compilers really exist in your
328    PATH environment.
329    """
330    archs = []
331
332    for arch in os.listdir('arch'):
333        if os.path.exists(os.path.join('arch', arch, 'Makefile')):
334            archs.append(arch)
335
336    # arm64 is a special case
337    archs.append('aarch64')
338
339    for arch in archs:
340        env = 'CROSS_COMPILE_' + arch.upper()
341        cross_compile = os.environ.get(env)
342        if not cross_compile:
343            cross_compile = CROSS_COMPILE.get(arch, '')
344
345        for path in os.environ["PATH"].split(os.pathsep):
346            gcc_path = os.path.join(path, cross_compile + 'gcc')
347            if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK):
348                break
349        else:
350            print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW,
351                 'warning: %sgcc: not found in PATH.  %s architecture boards will be skipped'
352                                            % (cross_compile, arch))
353            cross_compile = None
354
355        CROSS_COMPILE[arch] = cross_compile
356
357def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
358                         extend_post):
359    """Extend matched lines if desired patterns are found before/after already
360    matched lines.
361
362    Arguments:
363      lines: A list of lines handled.
364      matched: A list of line numbers that have been already matched.
365               (will be updated by this function)
366      pre_patterns: A list of regular expression that should be matched as
367                    preamble.
368      post_patterns: A list of regular expression that should be matched as
369                     postamble.
370      extend_pre: Add the line number of matched preamble to the matched list.
371      extend_post: Add the line number of matched postamble to the matched list.
372    """
373    extended_matched = []
374
375    j = matched[0]
376
377    for i in matched:
378        if i == 0 or i < j:
379            continue
380        j = i
381        while j in matched:
382            j += 1
383        if j >= len(lines):
384            break
385
386        for p in pre_patterns:
387            if p.search(lines[i - 1]):
388                break
389        else:
390            # not matched
391            continue
392
393        for p in post_patterns:
394            if p.search(lines[j]):
395                break
396        else:
397            # not matched
398            continue
399
400        if extend_pre:
401            extended_matched.append(i - 1)
402        if extend_post:
403            extended_matched.append(j)
404
405    matched += extended_matched
406    matched.sort()
407
408def cleanup_one_header(header_path, patterns, options):
409    """Clean regex-matched lines away from a file.
410
411    Arguments:
412      header_path: path to the cleaned file.
413      patterns: list of regex patterns.  Any lines matching to these
414                patterns are deleted.
415      options: option flags.
416    """
417    with open(header_path) as f:
418        lines = f.readlines()
419
420    matched = []
421    for i, line in enumerate(lines):
422        if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
423            matched.append(i)
424            continue
425        for pattern in patterns:
426            if pattern.search(line):
427                matched.append(i)
428                break
429
430    if not matched:
431        return
432
433    # remove empty #ifdef ... #endif, successive blank lines
434    pattern_if = re.compile(r'#\s*if(def|ndef)?\W') #  #if, #ifdef, #ifndef
435    pattern_elif = re.compile(r'#\s*el(if|se)\W')   #  #elif, #else
436    pattern_endif = re.compile(r'#\s*endif\W')      #  #endif
437    pattern_blank = re.compile(r'^\s*$')            #  empty line
438
439    while True:
440        old_matched = copy.copy(matched)
441        extend_matched_lines(lines, matched, [pattern_if],
442                             [pattern_endif], True, True)
443        extend_matched_lines(lines, matched, [pattern_elif],
444                             [pattern_elif, pattern_endif], True, False)
445        extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
446                             [pattern_blank], False, True)
447        extend_matched_lines(lines, matched, [pattern_blank],
448                             [pattern_elif, pattern_endif], True, False)
449        extend_matched_lines(lines, matched, [pattern_blank],
450                             [pattern_blank], True, False)
451        if matched == old_matched:
452            break
453
454    tolines = copy.copy(lines)
455
456    for i in reversed(matched):
457        tolines.pop(i)
458
459    show_diff(lines, tolines, header_path, options.color)
460
461    if options.dry_run:
462        return
463
464    with open(header_path, 'w') as f:
465        for line in tolines:
466            f.write(line)
467
468def cleanup_headers(configs, options):
469    """Delete config defines from board headers.
470
471    Arguments:
472      configs: A list of CONFIGs to remove.
473      options: option flags.
474    """
475    while True:
476        choice = raw_input('Clean up headers? [y/n]: ').lower()
477        print choice
478        if choice == 'y' or choice == 'n':
479            break
480
481    if choice == 'n':
482        return
483
484    patterns = []
485    for config in configs:
486        patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
487        patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
488
489    for dir in 'include', 'arch', 'board':
490        for (dirpath, dirnames, filenames) in os.walk(dir):
491            if dirpath == os.path.join('include', 'generated'):
492                continue
493            for filename in filenames:
494                if not fnmatch.fnmatch(filename, '*~'):
495                    cleanup_one_header(os.path.join(dirpath, filename),
496                                       patterns, options)
497
498def cleanup_one_extra_option(defconfig_path, configs, options):
499    """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
500
501    Arguments:
502      defconfig_path: path to the cleaned defconfig file.
503      configs: A list of CONFIGs to remove.
504      options: option flags.
505    """
506
507    start = 'CONFIG_SYS_EXTRA_OPTIONS="'
508    end = '"\n'
509
510    with open(defconfig_path) as f:
511        lines = f.readlines()
512
513    for i, line in enumerate(lines):
514        if line.startswith(start) and line.endswith(end):
515            break
516    else:
517        # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
518        return
519
520    old_tokens = line[len(start):-len(end)].split(',')
521    new_tokens = []
522
523    for token in old_tokens:
524        pos = token.find('=')
525        if not (token[:pos] if pos >= 0 else token) in configs:
526            new_tokens.append(token)
527
528    if new_tokens == old_tokens:
529        return
530
531    tolines = copy.copy(lines)
532
533    if new_tokens:
534        tolines[i] = start + ','.join(new_tokens) + end
535    else:
536        tolines.pop(i)
537
538    show_diff(lines, tolines, defconfig_path, options.color)
539
540    if options.dry_run:
541        return
542
543    with open(defconfig_path, 'w') as f:
544        for line in tolines:
545            f.write(line)
546
547def cleanup_extra_options(configs, options):
548    """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
549
550    Arguments:
551      configs: A list of CONFIGs to remove.
552      options: option flags.
553    """
554    while True:
555        choice = raw_input('Clean up CONFIG_SYS_EXTRA_OPTIONS? [y/n]: ').lower()
556        print choice
557        if choice == 'y' or choice == 'n':
558            break
559
560    if choice == 'n':
561        return
562
563    configs = [ config[len('CONFIG_'):] for config in configs ]
564
565    defconfigs = get_all_defconfigs()
566
567    for defconfig in defconfigs:
568        cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
569                                 options)
570
571### classes ###
572class Progress:
573
574    """Progress Indicator"""
575
576    def __init__(self, total):
577        """Create a new progress indicator.
578
579        Arguments:
580          total: A number of defconfig files to process.
581        """
582        self.current = 0
583        self.total = total
584
585    def inc(self):
586        """Increment the number of processed defconfig files."""
587
588        self.current += 1
589
590    def show(self):
591        """Display the progress."""
592        print ' %d defconfigs out of %d\r' % (self.current, self.total),
593        sys.stdout.flush()
594
595class KconfigParser:
596
597    """A parser of .config and include/autoconf.mk."""
598
599    re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
600    re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
601
602    def __init__(self, configs, options, build_dir):
603        """Create a new parser.
604
605        Arguments:
606          configs: A list of CONFIGs to move.
607          options: option flags.
608          build_dir: Build directory.
609        """
610        self.configs = configs
611        self.options = options
612        self.dotconfig = os.path.join(build_dir, '.config')
613        self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
614        self.config_autoconf = os.path.join(build_dir, 'include', 'config',
615                                            'auto.conf')
616        self.defconfig = os.path.join(build_dir, 'defconfig')
617
618    def get_cross_compile(self):
619        """Parse .config file and return CROSS_COMPILE.
620
621        Returns:
622          A string storing the compiler prefix for the architecture.
623          Return a NULL string for architectures that do not require
624          compiler prefix (Sandbox and native build is the case).
625          Return None if the specified compiler is missing in your PATH.
626          Caller should distinguish '' and None.
627        """
628        arch = ''
629        cpu = ''
630        for line in open(self.dotconfig):
631            m = self.re_arch.match(line)
632            if m:
633                arch = m.group(1)
634                continue
635            m = self.re_cpu.match(line)
636            if m:
637                cpu = m.group(1)
638
639        if not arch:
640            return None
641
642        # fix-up for aarch64
643        if arch == 'arm' and cpu == 'armv8':
644            arch = 'aarch64'
645
646        return CROSS_COMPILE.get(arch, None)
647
648    def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
649        """Parse .config, defconfig, include/autoconf.mk for one config.
650
651        This function looks for the config options in the lines from
652        defconfig, .config, and include/autoconf.mk in order to decide
653        which action should be taken for this defconfig.
654
655        Arguments:
656          config: CONFIG name to parse.
657          dotconfig_lines: lines from the .config file.
658          autoconf_lines: lines from the include/autoconf.mk file.
659
660        Returns:
661          A tupple of the action for this defconfig and the line
662          matched for the config.
663        """
664        not_set = '# %s is not set' % config
665
666        for line in dotconfig_lines:
667            line = line.rstrip()
668            if line.startswith(config + '=') or line == not_set:
669                old_val = line
670                break
671        else:
672            return (ACTION_NO_ENTRY, config)
673
674        for line in autoconf_lines:
675            line = line.rstrip()
676            if line.startswith(config + '='):
677                new_val = line
678                break
679        else:
680            new_val = not_set
681
682        # If this CONFIG is neither bool nor trisate
683        if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
684            # tools/scripts/define2mk.sed changes '1' to 'y'.
685            # This is a problem if the CONFIG is int type.
686            # Check the type in Kconfig and handle it correctly.
687            if new_val[-2:] == '=y':
688                new_val = new_val[:-1] + '1'
689
690        return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
691                new_val)
692
693    def update_dotconfig(self):
694        """Parse files for the config options and update the .config.
695
696        This function parses the generated .config and include/autoconf.mk
697        searching the target options.
698        Move the config option(s) to the .config as needed.
699
700        Arguments:
701          defconfig: defconfig name.
702
703        Returns:
704          Return a tuple of (updated flag, log string).
705          The "updated flag" is True if the .config was updated, False
706          otherwise.  The "log string" shows what happend to the .config.
707        """
708
709        results = []
710        updated = False
711
712        with open(self.dotconfig) as f:
713            dotconfig_lines = f.readlines()
714
715        with open(self.autoconf) as f:
716            autoconf_lines = f.readlines()
717
718        for config in self.configs:
719            result = self.parse_one_config(config, dotconfig_lines,
720                                           autoconf_lines)
721            results.append(result)
722
723        log = ''
724
725        for (action, value) in results:
726            if action == ACTION_MOVE:
727                actlog = "Move '%s'" % value
728                log_color = COLOR_LIGHT_GREEN
729            elif action == ACTION_NO_ENTRY:
730                actlog = "%s is not defined in Kconfig.  Do nothing." % value
731                log_color = COLOR_LIGHT_BLUE
732            elif action == ACTION_NO_CHANGE:
733                actlog = "'%s' is the same as the define in Kconfig.  Do nothing." \
734                         % value
735                log_color = COLOR_LIGHT_PURPLE
736            else:
737                sys.exit("Internal Error. This should not happen.")
738
739            log += color_text(self.options.color, log_color, actlog) + '\n'
740
741        with open(self.dotconfig, 'a') as f:
742            for (action, value) in results:
743                if action == ACTION_MOVE:
744                    f.write(value + '\n')
745                    updated = True
746
747        self.results = results
748        os.remove(self.config_autoconf)
749        os.remove(self.autoconf)
750
751        return (updated, log)
752
753    def check_defconfig(self):
754        """Check the defconfig after savedefconfig
755
756        Returns:
757          Return additional log if moved CONFIGs were removed again by
758          'make savedefconfig'.
759        """
760
761        log = ''
762
763        with open(self.defconfig) as f:
764            defconfig_lines = f.readlines()
765
766        for (action, value) in self.results:
767            if action != ACTION_MOVE:
768                continue
769            if not value + '\n' in defconfig_lines:
770                log += color_text(self.options.color, COLOR_YELLOW,
771                                  "'%s' was removed by savedefconfig.\n" %
772                                  value)
773
774        return log
775
776class Slot:
777
778    """A slot to store a subprocess.
779
780    Each instance of this class handles one subprocess.
781    This class is useful to control multiple threads
782    for faster processing.
783    """
784
785    def __init__(self, configs, options, progress, devnull, make_cmd, reference_src_dir):
786        """Create a new process slot.
787
788        Arguments:
789          configs: A list of CONFIGs to move.
790          options: option flags.
791          progress: A progress indicator.
792          devnull: A file object of '/dev/null'.
793          make_cmd: command name of GNU Make.
794          reference_src_dir: Determine the true starting config state from this
795                             source tree.
796        """
797        self.options = options
798        self.progress = progress
799        self.build_dir = tempfile.mkdtemp()
800        self.devnull = devnull
801        self.make_cmd = (make_cmd, 'O=' + self.build_dir)
802        self.reference_src_dir = reference_src_dir
803        self.parser = KconfigParser(configs, options, self.build_dir)
804        self.state = STATE_IDLE
805        self.failed_boards = []
806        self.suspicious_boards = []
807
808    def __del__(self):
809        """Delete the working directory
810
811        This function makes sure the temporary directory is cleaned away
812        even if Python suddenly dies due to error.  It should be done in here
813        because it is guaranteed the destructor is always invoked when the
814        instance of the class gets unreferenced.
815
816        If the subprocess is still running, wait until it finishes.
817        """
818        if self.state != STATE_IDLE:
819            while self.ps.poll() == None:
820                pass
821        shutil.rmtree(self.build_dir)
822
823    def add(self, defconfig):
824        """Assign a new subprocess for defconfig and add it to the slot.
825
826        If the slot is vacant, create a new subprocess for processing the
827        given defconfig and add it to the slot.  Just returns False if
828        the slot is occupied (i.e. the current subprocess is still running).
829
830        Arguments:
831          defconfig: defconfig name.
832
833        Returns:
834          Return True on success or False on failure
835        """
836        if self.state != STATE_IDLE:
837            return False
838
839        self.defconfig = defconfig
840        self.log = ''
841        self.current_src_dir = self.reference_src_dir
842        self.do_defconfig()
843        return True
844
845    def poll(self):
846        """Check the status of the subprocess and handle it as needed.
847
848        Returns True if the slot is vacant (i.e. in idle state).
849        If the configuration is successfully finished, assign a new
850        subprocess to build include/autoconf.mk.
851        If include/autoconf.mk is generated, invoke the parser to
852        parse the .config and the include/autoconf.mk, moving
853        config options to the .config as needed.
854        If the .config was updated, run "make savedefconfig" to sync
855        it, update the original defconfig, and then set the slot back
856        to the idle state.
857
858        Returns:
859          Return True if the subprocess is terminated, False otherwise
860        """
861        if self.state == STATE_IDLE:
862            return True
863
864        if self.ps.poll() == None:
865            return False
866
867        if self.ps.poll() != 0:
868            self.handle_error()
869        elif self.state == STATE_DEFCONFIG:
870            if self.reference_src_dir and not self.current_src_dir:
871                self.do_savedefconfig()
872            else:
873                self.do_autoconf()
874        elif self.state == STATE_AUTOCONF:
875            if self.current_src_dir:
876                self.current_src_dir = None
877                self.do_defconfig()
878            else:
879                self.do_savedefconfig()
880        elif self.state == STATE_SAVEDEFCONFIG:
881            self.update_defconfig()
882        else:
883            sys.exit("Internal Error. This should not happen.")
884
885        return True if self.state == STATE_IDLE else False
886
887    def handle_error(self):
888        """Handle error cases."""
889
890        self.log += color_text(self.options.color, COLOR_LIGHT_RED,
891                               "Failed to process.\n")
892        if self.options.verbose:
893            self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
894                                   self.ps.stderr.read())
895        self.finish(False)
896
897    def do_defconfig(self):
898        """Run 'make <board>_defconfig' to create the .config file."""
899
900        cmd = list(self.make_cmd)
901        cmd.append(self.defconfig)
902        self.ps = subprocess.Popen(cmd, stdout=self.devnull,
903                                   stderr=subprocess.PIPE,
904                                   cwd=self.current_src_dir)
905        self.state = STATE_DEFCONFIG
906
907    def do_autoconf(self):
908        """Run 'make include/config/auto.conf'."""
909
910        self.cross_compile = self.parser.get_cross_compile()
911        if self.cross_compile is None:
912            self.log += color_text(self.options.color, COLOR_YELLOW,
913                                   "Compiler is missing.  Do nothing.\n")
914            self.finish(False)
915            return
916
917        cmd = list(self.make_cmd)
918        if self.cross_compile:
919            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
920        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
921        cmd.append('include/config/auto.conf')
922        self.ps = subprocess.Popen(cmd, stdout=self.devnull,
923                                   stderr=subprocess.PIPE,
924                                   cwd=self.current_src_dir)
925        self.state = STATE_AUTOCONF
926
927    def do_savedefconfig(self):
928        """Update the .config and run 'make savedefconfig'."""
929
930        (updated, log) = self.parser.update_dotconfig()
931        self.log += log
932
933        if not self.options.force_sync and not updated:
934            self.finish(True)
935            return
936        if updated:
937            self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
938                                   "Syncing by savedefconfig...\n")
939        else:
940            self.log += "Syncing by savedefconfig (forced by option)...\n"
941
942        cmd = list(self.make_cmd)
943        cmd.append('savedefconfig')
944        self.ps = subprocess.Popen(cmd, stdout=self.devnull,
945                                   stderr=subprocess.PIPE)
946        self.state = STATE_SAVEDEFCONFIG
947
948    def update_defconfig(self):
949        """Update the input defconfig and go back to the idle state."""
950
951        log = self.parser.check_defconfig()
952        if log:
953            self.suspicious_boards.append(self.defconfig)
954            self.log += log
955        orig_defconfig = os.path.join('configs', self.defconfig)
956        new_defconfig = os.path.join(self.build_dir, 'defconfig')
957        updated = not filecmp.cmp(orig_defconfig, new_defconfig)
958
959        if updated:
960            self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
961                                   "defconfig was updated.\n")
962
963        if not self.options.dry_run and updated:
964            shutil.move(new_defconfig, orig_defconfig)
965        self.finish(True)
966
967    def finish(self, success):
968        """Display log along with progress and go to the idle state.
969
970        Arguments:
971          success: Should be True when the defconfig was processed
972                   successfully, or False when it fails.
973        """
974        # output at least 30 characters to hide the "* defconfigs out of *".
975        log = self.defconfig.ljust(30) + '\n'
976
977        log += '\n'.join([ '    ' + s for s in self.log.split('\n') ])
978        # Some threads are running in parallel.
979        # Print log atomically to not mix up logs from different threads.
980        print >> (sys.stdout if success else sys.stderr), log
981
982        if not success:
983            if self.options.exit_on_error:
984                sys.exit("Exit on error.")
985            # If --exit-on-error flag is not set, skip this board and continue.
986            # Record the failed board.
987            self.failed_boards.append(self.defconfig)
988
989        self.progress.inc()
990        self.progress.show()
991        self.state = STATE_IDLE
992
993    def get_failed_boards(self):
994        """Returns a list of failed boards (defconfigs) in this slot.
995        """
996        return self.failed_boards
997
998    def get_suspicious_boards(self):
999        """Returns a list of boards (defconfigs) with possible misconversion.
1000        """
1001        return self.suspicious_boards
1002
1003class Slots:
1004
1005    """Controller of the array of subprocess slots."""
1006
1007    def __init__(self, configs, options, progress, reference_src_dir):
1008        """Create a new slots controller.
1009
1010        Arguments:
1011          configs: A list of CONFIGs to move.
1012          options: option flags.
1013          progress: A progress indicator.
1014          reference_src_dir: Determine the true starting config state from this
1015                             source tree.
1016        """
1017        self.options = options
1018        self.slots = []
1019        devnull = get_devnull()
1020        make_cmd = get_make_cmd()
1021        for i in range(options.jobs):
1022            self.slots.append(Slot(configs, options, progress, devnull,
1023                                   make_cmd, reference_src_dir))
1024
1025    def add(self, defconfig):
1026        """Add a new subprocess if a vacant slot is found.
1027
1028        Arguments:
1029          defconfig: defconfig name to be put into.
1030
1031        Returns:
1032          Return True on success or False on failure
1033        """
1034        for slot in self.slots:
1035            if slot.add(defconfig):
1036                return True
1037        return False
1038
1039    def available(self):
1040        """Check if there is a vacant slot.
1041
1042        Returns:
1043          Return True if at lease one vacant slot is found, False otherwise.
1044        """
1045        for slot in self.slots:
1046            if slot.poll():
1047                return True
1048        return False
1049
1050    def empty(self):
1051        """Check if all slots are vacant.
1052
1053        Returns:
1054          Return True if all the slots are vacant, False otherwise.
1055        """
1056        ret = True
1057        for slot in self.slots:
1058            if not slot.poll():
1059                ret = False
1060        return ret
1061
1062    def show_failed_boards(self):
1063        """Display all of the failed boards (defconfigs)."""
1064        boards = []
1065        output_file = 'moveconfig.failed'
1066
1067        for slot in self.slots:
1068            boards += slot.get_failed_boards()
1069
1070        if boards:
1071            boards = '\n'.join(boards) + '\n'
1072            msg = "The following boards were not processed due to error:\n"
1073            msg += boards
1074            msg += "(the list has been saved in %s)\n" % output_file
1075            print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
1076                                            msg)
1077
1078            with open(output_file, 'w') as f:
1079                f.write(boards)
1080
1081    def show_suspicious_boards(self):
1082        """Display all boards (defconfigs) with possible misconversion."""
1083        boards = []
1084        output_file = 'moveconfig.suspicious'
1085
1086        for slot in self.slots:
1087            boards += slot.get_suspicious_boards()
1088
1089        if boards:
1090            boards = '\n'.join(boards) + '\n'
1091            msg = "The following boards might have been converted incorrectly.\n"
1092            msg += "It is highly recommended to check them manually:\n"
1093            msg += boards
1094            msg += "(the list has been saved in %s)\n" % output_file
1095            print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
1096                                            msg)
1097
1098            with open(output_file, 'w') as f:
1099                f.write(boards)
1100
1101class ReferenceSource:
1102
1103    """Reference source against which original configs should be parsed."""
1104
1105    def __init__(self, commit):
1106        """Create a reference source directory based on a specified commit.
1107
1108        Arguments:
1109          commit: commit to git-clone
1110        """
1111        self.src_dir = tempfile.mkdtemp()
1112        print "Cloning git repo to a separate work directory..."
1113        subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1114                                cwd=self.src_dir)
1115        print "Checkout '%s' to build the original autoconf.mk." % \
1116            subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
1117        subprocess.check_output(['git', 'checkout', commit],
1118                                stderr=subprocess.STDOUT, cwd=self.src_dir)
1119
1120    def __del__(self):
1121        """Delete the reference source directory
1122
1123        This function makes sure the temporary directory is cleaned away
1124        even if Python suddenly dies due to error.  It should be done in here
1125        because it is guaranteed the destructor is always invoked when the
1126        instance of the class gets unreferenced.
1127        """
1128        shutil.rmtree(self.src_dir)
1129
1130    def get_dir(self):
1131        """Return the absolute path to the reference source directory."""
1132
1133        return self.src_dir
1134
1135def move_config(configs, options):
1136    """Move config options to defconfig files.
1137
1138    Arguments:
1139      configs: A list of CONFIGs to move.
1140      options: option flags
1141    """
1142    if len(configs) == 0:
1143        if options.force_sync:
1144            print 'No CONFIG is specified. You are probably syncing defconfigs.',
1145        else:
1146            print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
1147    else:
1148        print 'Move ' + ', '.join(configs),
1149    print '(jobs: %d)\n' % options.jobs
1150
1151    if options.git_ref:
1152        reference_src = ReferenceSource(options.git_ref)
1153        reference_src_dir = reference_src.get_dir()
1154    else:
1155        reference_src_dir = None
1156
1157    if options.defconfigs:
1158        defconfigs = [line.strip() for line in open(options.defconfigs)]
1159        for i, defconfig in enumerate(defconfigs):
1160            if not defconfig.endswith('_defconfig'):
1161                defconfigs[i] = defconfig + '_defconfig'
1162            if not os.path.exists(os.path.join('configs', defconfigs[i])):
1163                sys.exit('%s - defconfig does not exist. Stopping.' %
1164                         defconfigs[i])
1165    else:
1166        defconfigs = get_all_defconfigs()
1167
1168    progress = Progress(len(defconfigs))
1169    slots = Slots(configs, options, progress, reference_src_dir)
1170
1171    # Main loop to process defconfig files:
1172    #  Add a new subprocess into a vacant slot.
1173    #  Sleep if there is no available slot.
1174    for defconfig in defconfigs:
1175        while not slots.add(defconfig):
1176            while not slots.available():
1177                # No available slot: sleep for a while
1178                time.sleep(SLEEP_TIME)
1179
1180    # wait until all the subprocesses finish
1181    while not slots.empty():
1182        time.sleep(SLEEP_TIME)
1183
1184    print ''
1185    slots.show_failed_boards()
1186    slots.show_suspicious_boards()
1187
1188def main():
1189    try:
1190        cpu_count = multiprocessing.cpu_count()
1191    except NotImplementedError:
1192        cpu_count = 1
1193
1194    parser = optparse.OptionParser()
1195    # Add options here
1196    parser.add_option('-c', '--color', action='store_true', default=False,
1197                      help='display the log in color')
1198    parser.add_option('-d', '--defconfigs', type='string',
1199                      help='a file containing a list of defconfigs to move')
1200    parser.add_option('-n', '--dry-run', action='store_true', default=False,
1201                      help='perform a trial run (show log with no changes)')
1202    parser.add_option('-e', '--exit-on-error', action='store_true',
1203                      default=False,
1204                      help='exit immediately on any error')
1205    parser.add_option('-s', '--force-sync', action='store_true', default=False,
1206                      help='force sync by savedefconfig')
1207    parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1208                      action='store_true', default=False,
1209                      help='only cleanup the headers')
1210    parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1211                      help='the number of jobs to run simultaneously')
1212    parser.add_option('-r', '--git-ref', type='string',
1213                      help='the git ref to clone for building the autoconf.mk')
1214    parser.add_option('-v', '--verbose', action='store_true', default=False,
1215                      help='show any build errors as boards are built')
1216    parser.usage += ' CONFIG ...'
1217
1218    (options, configs) = parser.parse_args()
1219
1220    if len(configs) == 0 and not options.force_sync:
1221        parser.print_usage()
1222        sys.exit(1)
1223
1224    # prefix the option name with CONFIG_ if missing
1225    configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1226                for config in configs ]
1227
1228    check_top_directory()
1229
1230    if not options.cleanup_headers_only:
1231        check_clean_directory()
1232        update_cross_compile(options.color)
1233        move_config(configs, options)
1234
1235    if configs:
1236        cleanup_headers(configs, options)
1237        cleanup_extra_options(configs, options)
1238
1239if __name__ == '__main__':
1240    main()
1241