1*ddaedbbeSThomas GleixnerEmbargoed hardware issues 2*ddaedbbeSThomas Gleixner========================= 3*ddaedbbeSThomas Gleixner 4*ddaedbbeSThomas GleixnerScope 5*ddaedbbeSThomas Gleixner----- 6*ddaedbbeSThomas Gleixner 7*ddaedbbeSThomas GleixnerHardware issues which result in security problems are a different category 8*ddaedbbeSThomas Gleixnerof security bugs than pure software bugs which only affect the Linux 9*ddaedbbeSThomas Gleixnerkernel. 10*ddaedbbeSThomas Gleixner 11*ddaedbbeSThomas GleixnerHardware issues like Meltdown, Spectre, L1TF etc. must be treated 12*ddaedbbeSThomas Gleixnerdifferently because they usually affect all Operating Systems ("OS") and 13*ddaedbbeSThomas Gleixnertherefore need coordination across different OS vendors, distributions, 14*ddaedbbeSThomas Gleixnerhardware vendors and other parties. For some of the issues, software 15*ddaedbbeSThomas Gleixnermitigations can depend on microcode or firmware updates, which need further 16*ddaedbbeSThomas Gleixnercoordination. 17*ddaedbbeSThomas Gleixner 18*ddaedbbeSThomas Gleixner.. _Contact: 19*ddaedbbeSThomas Gleixner 20*ddaedbbeSThomas GleixnerContact 21*ddaedbbeSThomas Gleixner------- 22*ddaedbbeSThomas Gleixner 23*ddaedbbeSThomas GleixnerThe Linux kernel hardware security team is separate from the regular Linux 24*ddaedbbeSThomas Gleixnerkernel security team. 25*ddaedbbeSThomas Gleixner 26*ddaedbbeSThomas GleixnerThe team only handles the coordination of embargoed hardware security 27*ddaedbbeSThomas Gleixnerissues. Reports of pure software security bugs in the Linux kernel are not 28*ddaedbbeSThomas Gleixnerhandled by this team and the reporter will be guided to contact the regular 29*ddaedbbeSThomas GleixnerLinux kernel security team (:ref:`Documentation/admin-guide/ 30*ddaedbbeSThomas Gleixner<securitybugs>`) instead. 31*ddaedbbeSThomas Gleixner 32*ddaedbbeSThomas GleixnerThe team can be contacted by email at <hardware-security@kernel.org>. This 33*ddaedbbeSThomas Gleixneris a private list of security officers who will help you to coordinate an 34*ddaedbbeSThomas Gleixnerissue according to our documented process. 35*ddaedbbeSThomas Gleixner 36*ddaedbbeSThomas GleixnerThe list is encrypted and email to the list can be sent by either PGP or 37*ddaedbbeSThomas GleixnerS/MIME encrypted and must be signed with the reporter's PGP key or S/MIME 38*ddaedbbeSThomas Gleixnercertificate. The list's PGP key and S/MIME certificate are available from 39*ddaedbbeSThomas Gleixnerhttps://www.kernel.org/.... 40*ddaedbbeSThomas Gleixner 41*ddaedbbeSThomas GleixnerWhile hardware security issues are often handled by the affected hardware 42*ddaedbbeSThomas Gleixnervendor, we welcome contact from researchers or individuals who have 43*ddaedbbeSThomas Gleixneridentified a potential hardware flaw. 44*ddaedbbeSThomas Gleixner 45*ddaedbbeSThomas GleixnerHardware security officers 46*ddaedbbeSThomas Gleixner^^^^^^^^^^^^^^^^^^^^^^^^^^ 47*ddaedbbeSThomas Gleixner 48*ddaedbbeSThomas GleixnerThe current team of hardware security officers: 49*ddaedbbeSThomas Gleixner 50*ddaedbbeSThomas Gleixner - Linus Torvalds (Linux Foundation Fellow) 51*ddaedbbeSThomas Gleixner - Greg Kroah-Hartman (Linux Foundation Fellow) 52*ddaedbbeSThomas Gleixner - Thomas Gleixner (Linux Foundation Fellow) 53*ddaedbbeSThomas Gleixner 54*ddaedbbeSThomas GleixnerOperation of mailing-lists 55*ddaedbbeSThomas Gleixner^^^^^^^^^^^^^^^^^^^^^^^^^^ 56*ddaedbbeSThomas Gleixner 57*ddaedbbeSThomas GleixnerThe encrypted mailing-lists which are used in our process are hosted on 58*ddaedbbeSThomas GleixnerLinux Foundation's IT infrastructure. By providing this service Linux 59*ddaedbbeSThomas GleixnerFoundation's director of IT Infrastructure security technically has the 60*ddaedbbeSThomas Gleixnerability to access the embargoed information, but is obliged to 61*ddaedbbeSThomas Gleixnerconfidentiality by his employment contract. Linux Foundation's director of 62*ddaedbbeSThomas GleixnerIT Infrastructure security is also responsible for the kernel.org 63*ddaedbbeSThomas Gleixnerinfrastructure. 64*ddaedbbeSThomas Gleixner 65*ddaedbbeSThomas GleixnerThe Linux Foundation's current director of IT Infrastructure security is 66*ddaedbbeSThomas GleixnerKonstantin Ryabitsev. 67*ddaedbbeSThomas Gleixner 68*ddaedbbeSThomas Gleixner 69*ddaedbbeSThomas GleixnerNon-disclosure agreements 70*ddaedbbeSThomas Gleixner------------------------- 71*ddaedbbeSThomas Gleixner 72*ddaedbbeSThomas GleixnerThe Linux kernel hardware security team is not a formal body and therefore 73*ddaedbbeSThomas Gleixnerunable to enter into any non-disclosure agreements. The kernel community 74*ddaedbbeSThomas Gleixneris aware of the sensitive nature of such issues and offers a Memorandum of 75*ddaedbbeSThomas GleixnerUnderstanding instead. 76*ddaedbbeSThomas Gleixner 77*ddaedbbeSThomas Gleixner 78*ddaedbbeSThomas GleixnerMemorandum of Understanding 79*ddaedbbeSThomas Gleixner--------------------------- 80*ddaedbbeSThomas Gleixner 81*ddaedbbeSThomas GleixnerThe Linux kernel community has a deep understanding of the requirement to 82*ddaedbbeSThomas Gleixnerkeep hardware security issues under embargo for coordination between 83*ddaedbbeSThomas Gleixnerdifferent OS vendors, distributors, hardware vendors and other parties. 84*ddaedbbeSThomas Gleixner 85*ddaedbbeSThomas GleixnerThe Linux kernel community has successfully handled hardware security 86*ddaedbbeSThomas Gleixnerissues in the past and has the necessary mechanisms in place to allow 87*ddaedbbeSThomas Gleixnercommunity compliant development under embargo restrictions. 88*ddaedbbeSThomas Gleixner 89*ddaedbbeSThomas GleixnerThe Linux kernel community has a dedicated hardware security team for 90*ddaedbbeSThomas Gleixnerinitial contact, which oversees the process of handling such issues under 91*ddaedbbeSThomas Gleixnerembargo rules. 92*ddaedbbeSThomas Gleixner 93*ddaedbbeSThomas GleixnerThe hardware security team identifies the developers (domain experts) who 94*ddaedbbeSThomas Gleixnerwill form the initial response team for a particular issue. The initial 95*ddaedbbeSThomas Gleixnerresponse team can bring in further developers (domain experts) to address 96*ddaedbbeSThomas Gleixnerthe issue in the best technical way. 97*ddaedbbeSThomas Gleixner 98*ddaedbbeSThomas GleixnerAll involved developers pledge to adhere to the embargo rules and to keep 99*ddaedbbeSThomas Gleixnerthe received information confidential. Violation of the pledge will lead to 100*ddaedbbeSThomas Gleixnerimmediate exclusion from the current issue and removal from all related 101*ddaedbbeSThomas Gleixnermailing-lists. In addition, the hardware security team will also exclude 102*ddaedbbeSThomas Gleixnerthe offender from future issues. The impact of this consequence is a highly 103*ddaedbbeSThomas Gleixnereffective deterrent in our community. In case a violation happens the 104*ddaedbbeSThomas Gleixnerhardware security team will inform the involved parties immediately. If you 105*ddaedbbeSThomas Gleixneror anyone becomes aware of a potential violation, please report it 106*ddaedbbeSThomas Gleixnerimmediately to the Hardware security officers. 107*ddaedbbeSThomas Gleixner 108*ddaedbbeSThomas Gleixner 109*ddaedbbeSThomas GleixnerProcess 110*ddaedbbeSThomas Gleixner^^^^^^^ 111*ddaedbbeSThomas Gleixner 112*ddaedbbeSThomas GleixnerDue to the globally distributed nature of Linux kernel development, 113*ddaedbbeSThomas Gleixnerface-to-face meetings are almost impossible to address hardware security 114*ddaedbbeSThomas Gleixnerissues. Phone conferences are hard to coordinate due to time zones and 115*ddaedbbeSThomas Gleixnerother factors and should be only used when absolutely necessary. Encrypted 116*ddaedbbeSThomas Gleixneremail has been proven to be the most effective and secure communication 117*ddaedbbeSThomas Gleixnermethod for these types of issues. 118*ddaedbbeSThomas Gleixner 119*ddaedbbeSThomas GleixnerStart of Disclosure 120*ddaedbbeSThomas Gleixner""""""""""""""""""" 121*ddaedbbeSThomas Gleixner 122*ddaedbbeSThomas GleixnerDisclosure starts by contacting the Linux kernel hardware security team by 123*ddaedbbeSThomas Gleixneremail. This initial contact should contain a description of the problem and 124*ddaedbbeSThomas Gleixnera list of any known affected hardware. If your organization builds or 125*ddaedbbeSThomas Gleixnerdistributes the affected hardware, we encourage you to also consider what 126*ddaedbbeSThomas Gleixnerother hardware could be affected. 127*ddaedbbeSThomas Gleixner 128*ddaedbbeSThomas GleixnerThe hardware security team will provide an incident-specific encrypted 129*ddaedbbeSThomas Gleixnermailing-list which will be used for initial discussion with the reporter, 130*ddaedbbeSThomas Gleixnerfurther disclosure and coordination. 131*ddaedbbeSThomas Gleixner 132*ddaedbbeSThomas GleixnerThe hardware security team will provide the disclosing party a list of 133*ddaedbbeSThomas Gleixnerdevelopers (domain experts) who should be informed initially about the 134*ddaedbbeSThomas Gleixnerissue after confirming with the developers that they will adhere to this 135*ddaedbbeSThomas GleixnerMemorandum of Understanding and the documented process. These developers 136*ddaedbbeSThomas Gleixnerform the initial response team and will be responsible for handling the 137*ddaedbbeSThomas Gleixnerissue after initial contact. The hardware security team is supporting the 138*ddaedbbeSThomas Gleixnerresponse team, but is not necessarily involved in the mitigation 139*ddaedbbeSThomas Gleixnerdevelopment process. 140*ddaedbbeSThomas Gleixner 141*ddaedbbeSThomas GleixnerWhile individual developers might be covered by a non-disclosure agreement 142*ddaedbbeSThomas Gleixnervia their employer, they cannot enter individual non-disclosure agreements 143*ddaedbbeSThomas Gleixnerin their role as Linux kernel developers. They will, however, agree to 144*ddaedbbeSThomas Gleixneradhere to this documented process and the Memorandum of Understanding. 145*ddaedbbeSThomas Gleixner 146*ddaedbbeSThomas Gleixner 147*ddaedbbeSThomas GleixnerDisclosure 148*ddaedbbeSThomas Gleixner"""""""""" 149*ddaedbbeSThomas Gleixner 150*ddaedbbeSThomas GleixnerThe disclosing party provides detailed information to the initial response 151*ddaedbbeSThomas Gleixnerteam via the specific encrypted mailing-list. 152*ddaedbbeSThomas Gleixner 153*ddaedbbeSThomas GleixnerFrom our experience the technical documentation of these issues is usually 154*ddaedbbeSThomas Gleixnera sufficient starting point and further technical clarification is best 155*ddaedbbeSThomas Gleixnerdone via email. 156*ddaedbbeSThomas Gleixner 157*ddaedbbeSThomas GleixnerMitigation development 158*ddaedbbeSThomas Gleixner"""""""""""""""""""""" 159*ddaedbbeSThomas Gleixner 160*ddaedbbeSThomas GleixnerThe initial response team sets up an encrypted mailing-list or repurposes 161*ddaedbbeSThomas Gleixneran existing one if appropriate. The disclosing party should provide a list 162*ddaedbbeSThomas Gleixnerof contacts for all other parties who have already been, or should be 163*ddaedbbeSThomas Gleixnerinformed about the issue. The response team contacts these parties so they 164*ddaedbbeSThomas Gleixnercan name experts who should be subscribed to the mailing-list. 165*ddaedbbeSThomas Gleixner 166*ddaedbbeSThomas GleixnerUsing a mailing-list is close to the normal Linux development process and 167*ddaedbbeSThomas Gleixnerhas been successfully used in developing mitigations for various hardware 168*ddaedbbeSThomas Gleixnersecurity issues in the past. 169*ddaedbbeSThomas Gleixner 170*ddaedbbeSThomas GleixnerThe mailing-list operates in the same way as normal Linux development. 171*ddaedbbeSThomas GleixnerPatches are posted, discussed and reviewed and if agreed on applied to a 172*ddaedbbeSThomas Gleixnernon-public git repository which is only accessible to the participating 173*ddaedbbeSThomas Gleixnerdevelopers via a secure connection. The repository contains the main 174*ddaedbbeSThomas Gleixnerdevelopment branch against the mainline kernel and backport branches for 175*ddaedbbeSThomas Gleixnerstable kernel versions as necessary. 176*ddaedbbeSThomas Gleixner 177*ddaedbbeSThomas GleixnerThe initial response team will identify further experts from the Linux 178*ddaedbbeSThomas Gleixnerkernel developer community as needed and inform the disclosing party about 179*ddaedbbeSThomas Gleixnertheir participation. Bringing in experts can happen at any time of the 180*ddaedbbeSThomas Gleixnerdevelopment process and often needs to be handled in a timely manner. 181*ddaedbbeSThomas Gleixner 182*ddaedbbeSThomas GleixnerCoordinated release 183*ddaedbbeSThomas Gleixner""""""""""""""""""" 184*ddaedbbeSThomas Gleixner 185*ddaedbbeSThomas GleixnerThe involved parties will negotiate the date and time where the embargo 186*ddaedbbeSThomas Gleixnerends. At that point the prepared mitigations are integrated into the 187*ddaedbbeSThomas Gleixnerrelevant kernel trees and published. 188*ddaedbbeSThomas Gleixner 189*ddaedbbeSThomas GleixnerWhile we understand that hardware security issues need coordinated embargo 190*ddaedbbeSThomas Gleixnertime, the embargo time should be constrained to the minimum time which is 191*ddaedbbeSThomas Gleixnerrequired for all involved parties to develop, test and prepare the 192*ddaedbbeSThomas Gleixnermitigations. Extending embargo time artificially to meet conference talk 193*ddaedbbeSThomas Gleixnerdates or other non-technical reasons is creating more work and burden for 194*ddaedbbeSThomas Gleixnerthe involved developers and response teams as the patches need to be kept 195*ddaedbbeSThomas Gleixnerup to date in order to follow the ongoing upstream kernel development, 196*ddaedbbeSThomas Gleixnerwhich might create conflicting changes. 197*ddaedbbeSThomas Gleixner 198*ddaedbbeSThomas GleixnerCVE assignment 199*ddaedbbeSThomas Gleixner"""""""""""""" 200*ddaedbbeSThomas Gleixner 201*ddaedbbeSThomas GleixnerNeither the hardware security team nor the initial response team assign 202*ddaedbbeSThomas GleixnerCVEs, nor are CVEs required for the development process. If CVEs are 203*ddaedbbeSThomas Gleixnerprovided by the disclosing party they can be used for documentation 204*ddaedbbeSThomas Gleixnerpurposes. 205*ddaedbbeSThomas Gleixner 206*ddaedbbeSThomas GleixnerProcess ambassadors 207*ddaedbbeSThomas Gleixner------------------- 208*ddaedbbeSThomas Gleixner 209*ddaedbbeSThomas GleixnerFor assistance with this process we have established ambassadors in various 210*ddaedbbeSThomas Gleixnerorganizations, who can answer questions about or provide guidance on the 211*ddaedbbeSThomas Gleixnerreporting process and further handling. Ambassadors are not involved in the 212*ddaedbbeSThomas Gleixnerdisclosure of a particular issue, unless requested by a response team or by 213*ddaedbbeSThomas Gleixneran involved disclosed party. The current ambassadors list: 214*ddaedbbeSThomas Gleixner 215*ddaedbbeSThomas Gleixner ============= ======================================================== 216*ddaedbbeSThomas Gleixner ARM 217*ddaedbbeSThomas Gleixner AMD 218*ddaedbbeSThomas Gleixner IBM 219*ddaedbbeSThomas Gleixner Intel 220*ddaedbbeSThomas Gleixner Qualcomm 221*ddaedbbeSThomas Gleixner 222*ddaedbbeSThomas Gleixner Microsoft 223*ddaedbbeSThomas Gleixner VMware 224*ddaedbbeSThomas Gleixner XEN 225*ddaedbbeSThomas Gleixner 226*ddaedbbeSThomas Gleixner Canonical Tyler Hicks <tyhicks@canonical.com> 227*ddaedbbeSThomas Gleixner Debian Ben Hutchings <ben@decadent.org.uk> 228*ddaedbbeSThomas Gleixner Oracle Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> 229*ddaedbbeSThomas Gleixner Red Hat Josh Poimboeuf <jpoimboe@redhat.com> 230*ddaedbbeSThomas Gleixner SUSE Jiri Kosina <jkosina@suse.cz> 231*ddaedbbeSThomas Gleixner 232*ddaedbbeSThomas Gleixner Amazon 233*ddaedbbeSThomas Gleixner Google 234*ddaedbbeSThomas Gleixner ============== ======================================================== 235*ddaedbbeSThomas Gleixner 236*ddaedbbeSThomas GleixnerIf you want your organization to be added to the ambassadors list, please 237*ddaedbbeSThomas Gleixnercontact the hardware security team. The nominated ambassador has to 238*ddaedbbeSThomas Gleixnerunderstand and support our process fully and is ideally well connected in 239*ddaedbbeSThomas Gleixnerthe Linux kernel community. 240*ddaedbbeSThomas Gleixner 241*ddaedbbeSThomas GleixnerEncrypted mailing-lists 242*ddaedbbeSThomas Gleixner----------------------- 243*ddaedbbeSThomas Gleixner 244*ddaedbbeSThomas GleixnerWe use encrypted mailing-lists for communication. The operating principle 245*ddaedbbeSThomas Gleixnerof these lists is that email sent to the list is encrypted either with the 246*ddaedbbeSThomas Gleixnerlist's PGP key or with the list's S/MIME certificate. The mailing-list 247*ddaedbbeSThomas Gleixnersoftware decrypts the email and re-encrypts it individually for each 248*ddaedbbeSThomas Gleixnersubscriber with the subscriber's PGP key or S/MIME certificate. Details 249*ddaedbbeSThomas Gleixnerabout the mailing-list software and the setup which is used to ensure the 250*ddaedbbeSThomas Gleixnersecurity of the lists and protection of the data can be found here: 251*ddaedbbeSThomas Gleixnerhttps://www.kernel.org/.... 252*ddaedbbeSThomas Gleixner 253*ddaedbbeSThomas GleixnerList keys 254*ddaedbbeSThomas Gleixner^^^^^^^^^ 255*ddaedbbeSThomas Gleixner 256*ddaedbbeSThomas GleixnerFor initial contact see :ref:`Contact`. For incident specific mailing-lists 257*ddaedbbeSThomas Gleixnerthe key and S/MIME certificate are conveyed to the subscribers by email 258*ddaedbbeSThomas Gleixnersent from the specific list. 259*ddaedbbeSThomas Gleixner 260*ddaedbbeSThomas GleixnerSubscription to incident specific lists 261*ddaedbbeSThomas Gleixner^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 262*ddaedbbeSThomas Gleixner 263*ddaedbbeSThomas GleixnerSubscription is handled by the response teams. Disclosed parties who want 264*ddaedbbeSThomas Gleixnerto participate in the communication send a list of potential subscribers to 265*ddaedbbeSThomas Gleixnerthe response team so the response team can validate subscription requests. 266*ddaedbbeSThomas Gleixner 267*ddaedbbeSThomas GleixnerEach subscriber needs to send a subscription request to the response team 268*ddaedbbeSThomas Gleixnerby email. The email must be signed with the subscriber's PGP key or S/MIME 269*ddaedbbeSThomas Gleixnercertificate. If a PGP key is used, it must be available from a public key 270*ddaedbbeSThomas Gleixnerserver and is ideally connected to the Linux kernel's PGP web of trust. See 271*ddaedbbeSThomas Gleixneralso: https://www.kernel.org/signature.html. 272*ddaedbbeSThomas Gleixner 273*ddaedbbeSThomas GleixnerThe response team verifies that the subscriber request is valid and adds 274*ddaedbbeSThomas Gleixnerthe subscriber to the list. After subscription the subscriber will receive 275*ddaedbbeSThomas Gleixneremail from the mailing-list which is signed either with the list's PGP key 276*ddaedbbeSThomas Gleixneror the list's S/MIME certificate. The subscriber's email client can extract 277*ddaedbbeSThomas Gleixnerthe PGP key or the S/MIME certificate from the signature so the subscriber 278*ddaedbbeSThomas Gleixnercan send encrypted email to the list. 279*ddaedbbeSThomas Gleixner 280