1================ 2Control Group v2 3================ 4 5:Date: October, 2015 6:Author: Tejun Heo <tj@kernel.org> 7 8This is the authoritative documentation on the design, interface and 9conventions of cgroup v2. It describes all userland-visible aspects 10of cgroup including core and specific controller behaviors. All 11future changes must be reflected in this document. Documentation for 12v1 is available under Documentation/cgroup-v1/. 13 14.. CONTENTS 15 16 1. Introduction 17 1-1. Terminology 18 1-2. What is cgroup? 19 2. Basic Operations 20 2-1. Mounting 21 2-2. Organizing Processes and Threads 22 2-2-1. Processes 23 2-2-2. Threads 24 2-3. [Un]populated Notification 25 2-4. Controlling Controllers 26 2-4-1. Enabling and Disabling 27 2-4-2. Top-down Constraint 28 2-4-3. No Internal Process Constraint 29 2-5. Delegation 30 2-5-1. Model of Delegation 31 2-5-2. Delegation Containment 32 2-6. Guidelines 33 2-6-1. Organize Once and Control 34 2-6-2. Avoid Name Collisions 35 3. Resource Distribution Models 36 3-1. Weights 37 3-2. Limits 38 3-3. Protections 39 3-4. Allocations 40 4. Interface Files 41 4-1. Format 42 4-2. Conventions 43 4-3. Core Interface Files 44 5. Controllers 45 5-1. CPU 46 5-1-1. CPU Interface Files 47 5-2. Memory 48 5-2-1. Memory Interface Files 49 5-2-2. Usage Guidelines 50 5-2-3. Memory Ownership 51 5-3. IO 52 5-3-1. IO Interface Files 53 5-3-2. Writeback 54 5-3-3. IO Latency 55 5-3-3-1. How IO Latency Throttling Works 56 5-3-3-2. IO Latency Interface Files 57 5-4. PID 58 5-4-1. PID Interface Files 59 5-5. Device 60 5-6. RDMA 61 5-6-1. RDMA Interface Files 62 5-7. Misc 63 5-7-1. perf_event 64 5-N. Non-normative information 65 5-N-1. CPU controller root cgroup process behaviour 66 5-N-2. IO controller root cgroup process behaviour 67 6. Namespace 68 6-1. Basics 69 6-2. The Root and Views 70 6-3. Migration and setns(2) 71 6-4. Interaction with Other Namespaces 72 P. Information on Kernel Programming 73 P-1. Filesystem Support for Writeback 74 D. Deprecated v1 Core Features 75 R. Issues with v1 and Rationales for v2 76 R-1. Multiple Hierarchies 77 R-2. Thread Granularity 78 R-3. Competition Between Inner Nodes and Threads 79 R-4. Other Interface Issues 80 R-5. Controller Issues and Remedies 81 R-5-1. Memory 82 83 84Introduction 85============ 86 87Terminology 88----------- 89 90"cgroup" stands for "control group" and is never capitalized. The 91singular form is used to designate the whole feature and also as a 92qualifier as in "cgroup controllers". When explicitly referring to 93multiple individual control groups, the plural form "cgroups" is used. 94 95 96What is cgroup? 97--------------- 98 99cgroup is a mechanism to organize processes hierarchically and 100distribute system resources along the hierarchy in a controlled and 101configurable manner. 102 103cgroup is largely composed of two parts - the core and controllers. 104cgroup core is primarily responsible for hierarchically organizing 105processes. A cgroup controller is usually responsible for 106distributing a specific type of system resource along the hierarchy 107although there are utility controllers which serve purposes other than 108resource distribution. 109 110cgroups form a tree structure and every process in the system belongs 111to one and only one cgroup. All threads of a process belong to the 112same cgroup. On creation, all processes are put in the cgroup that 113the parent process belongs to at the time. A process can be migrated 114to another cgroup. Migration of a process doesn't affect already 115existing descendant processes. 116 117Following certain structural constraints, controllers may be enabled or 118disabled selectively on a cgroup. All controller behaviors are 119hierarchical - if a controller is enabled on a cgroup, it affects all 120processes which belong to the cgroups consisting the inclusive 121sub-hierarchy of the cgroup. When a controller is enabled on a nested 122cgroup, it always restricts the resource distribution further. The 123restrictions set closer to the root in the hierarchy can not be 124overridden from further away. 125 126 127Basic Operations 128================ 129 130Mounting 131-------- 132 133Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2 134hierarchy can be mounted with the following mount command:: 135 136 # mount -t cgroup2 none $MOUNT_POINT 137 138cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All 139controllers which support v2 and are not bound to a v1 hierarchy are 140automatically bound to the v2 hierarchy and show up at the root. 141Controllers which are not in active use in the v2 hierarchy can be 142bound to other hierarchies. This allows mixing v2 hierarchy with the 143legacy v1 multiple hierarchies in a fully backward compatible way. 144 145A controller can be moved across hierarchies only after the controller 146is no longer referenced in its current hierarchy. Because per-cgroup 147controller states are destroyed asynchronously and controllers may 148have lingering references, a controller may not show up immediately on 149the v2 hierarchy after the final umount of the previous hierarchy. 150Similarly, a controller should be fully disabled to be moved out of 151the unified hierarchy and it may take some time for the disabled 152controller to become available for other hierarchies; furthermore, due 153to inter-controller dependencies, other controllers may need to be 154disabled too. 155 156While useful for development and manual configurations, moving 157controllers dynamically between the v2 and other hierarchies is 158strongly discouraged for production use. It is recommended to decide 159the hierarchies and controller associations before starting using the 160controllers after system boot. 161 162During transition to v2, system management software might still 163automount the v1 cgroup filesystem and so hijack all controllers 164during boot, before manual intervention is possible. To make testing 165and experimenting easier, the kernel parameter cgroup_no_v1= allows 166disabling controllers in v1 and make them always available in v2. 167 168cgroup v2 currently supports the following mount options. 169 170 nsdelegate 171 172 Consider cgroup namespaces as delegation boundaries. This 173 option is system wide and can only be set on mount or modified 174 through remount from the init namespace. The mount option is 175 ignored on non-init namespace mounts. Please refer to the 176 Delegation section for details. 177 178 179Organizing Processes and Threads 180-------------------------------- 181 182Processes 183~~~~~~~~~ 184 185Initially, only the root cgroup exists to which all processes belong. 186A child cgroup can be created by creating a sub-directory:: 187 188 # mkdir $CGROUP_NAME 189 190A given cgroup may have multiple child cgroups forming a tree 191structure. Each cgroup has a read-writable interface file 192"cgroup.procs". When read, it lists the PIDs of all processes which 193belong to the cgroup one-per-line. The PIDs are not ordered and the 194same PID may show up more than once if the process got moved to 195another cgroup and then back or the PID got recycled while reading. 196 197A process can be migrated into a cgroup by writing its PID to the 198target cgroup's "cgroup.procs" file. Only one process can be migrated 199on a single write(2) call. If a process is composed of multiple 200threads, writing the PID of any thread migrates all threads of the 201process. 202 203When a process forks a child process, the new process is born into the 204cgroup that the forking process belongs to at the time of the 205operation. After exit, a process stays associated with the cgroup 206that it belonged to at the time of exit until it's reaped; however, a 207zombie process does not appear in "cgroup.procs" and thus can't be 208moved to another cgroup. 209 210A cgroup which doesn't have any children or live processes can be 211destroyed by removing the directory. Note that a cgroup which doesn't 212have any children and is associated only with zombie processes is 213considered empty and can be removed:: 214 215 # rmdir $CGROUP_NAME 216 217"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy 218cgroup is in use in the system, this file may contain multiple lines, 219one for each hierarchy. The entry for cgroup v2 is always in the 220format "0::$PATH":: 221 222 # cat /proc/842/cgroup 223 ... 224 0::/test-cgroup/test-cgroup-nested 225 226If the process becomes a zombie and the cgroup it was associated with 227is removed subsequently, " (deleted)" is appended to the path:: 228 229 # cat /proc/842/cgroup 230 ... 231 0::/test-cgroup/test-cgroup-nested (deleted) 232 233 234Threads 235~~~~~~~ 236 237cgroup v2 supports thread granularity for a subset of controllers to 238support use cases requiring hierarchical resource distribution across 239the threads of a group of processes. By default, all threads of a 240process belong to the same cgroup, which also serves as the resource 241domain to host resource consumptions which are not specific to a 242process or thread. The thread mode allows threads to be spread across 243a subtree while still maintaining the common resource domain for them. 244 245Controllers which support thread mode are called threaded controllers. 246The ones which don't are called domain controllers. 247 248Marking a cgroup threaded makes it join the resource domain of its 249parent as a threaded cgroup. The parent may be another threaded 250cgroup whose resource domain is further up in the hierarchy. The root 251of a threaded subtree, that is, the nearest ancestor which is not 252threaded, is called threaded domain or thread root interchangeably and 253serves as the resource domain for the entire subtree. 254 255Inside a threaded subtree, threads of a process can be put in 256different cgroups and are not subject to the no internal process 257constraint - threaded controllers can be enabled on non-leaf cgroups 258whether they have threads in them or not. 259 260As the threaded domain cgroup hosts all the domain resource 261consumptions of the subtree, it is considered to have internal 262resource consumptions whether there are processes in it or not and 263can't have populated child cgroups which aren't threaded. Because the 264root cgroup is not subject to no internal process constraint, it can 265serve both as a threaded domain and a parent to domain cgroups. 266 267The current operation mode or type of the cgroup is shown in the 268"cgroup.type" file which indicates whether the cgroup is a normal 269domain, a domain which is serving as the domain of a threaded subtree, 270or a threaded cgroup. 271 272On creation, a cgroup is always a domain cgroup and can be made 273threaded by writing "threaded" to the "cgroup.type" file. The 274operation is single direction:: 275 276 # echo threaded > cgroup.type 277 278Once threaded, the cgroup can't be made a domain again. To enable the 279thread mode, the following conditions must be met. 280 281- As the cgroup will join the parent's resource domain. The parent 282 must either be a valid (threaded) domain or a threaded cgroup. 283 284- When the parent is an unthreaded domain, it must not have any domain 285 controllers enabled or populated domain children. The root is 286 exempt from this requirement. 287 288Topology-wise, a cgroup can be in an invalid state. Please consider 289the following topology:: 290 291 A (threaded domain) - B (threaded) - C (domain, just created) 292 293C is created as a domain but isn't connected to a parent which can 294host child domains. C can't be used until it is turned into a 295threaded cgroup. "cgroup.type" file will report "domain (invalid)" in 296these cases. Operations which fail due to invalid topology use 297EOPNOTSUPP as the errno. 298 299A domain cgroup is turned into a threaded domain when one of its child 300cgroup becomes threaded or threaded controllers are enabled in the 301"cgroup.subtree_control" file while there are processes in the cgroup. 302A threaded domain reverts to a normal domain when the conditions 303clear. 304 305When read, "cgroup.threads" contains the list of the thread IDs of all 306threads in the cgroup. Except that the operations are per-thread 307instead of per-process, "cgroup.threads" has the same format and 308behaves the same way as "cgroup.procs". While "cgroup.threads" can be 309written to in any cgroup, as it can only move threads inside the same 310threaded domain, its operations are confined inside each threaded 311subtree. 312 313The threaded domain cgroup serves as the resource domain for the whole 314subtree, and, while the threads can be scattered across the subtree, 315all the processes are considered to be in the threaded domain cgroup. 316"cgroup.procs" in a threaded domain cgroup contains the PIDs of all 317processes in the subtree and is not readable in the subtree proper. 318However, "cgroup.procs" can be written to from anywhere in the subtree 319to migrate all threads of the matching process to the cgroup. 320 321Only threaded controllers can be enabled in a threaded subtree. When 322a threaded controller is enabled inside a threaded subtree, it only 323accounts for and controls resource consumptions associated with the 324threads in the cgroup and its descendants. All consumptions which 325aren't tied to a specific thread belong to the threaded domain cgroup. 326 327Because a threaded subtree is exempt from no internal process 328constraint, a threaded controller must be able to handle competition 329between threads in a non-leaf cgroup and its child cgroups. Each 330threaded controller defines how such competitions are handled. 331 332 333[Un]populated Notification 334-------------------------- 335 336Each non-root cgroup has a "cgroup.events" file which contains 337"populated" field indicating whether the cgroup's sub-hierarchy has 338live processes in it. Its value is 0 if there is no live process in 339the cgroup and its descendants; otherwise, 1. poll and [id]notify 340events are triggered when the value changes. This can be used, for 341example, to start a clean-up operation after all processes of a given 342sub-hierarchy have exited. The populated state updates and 343notifications are recursive. Consider the following sub-hierarchy 344where the numbers in the parentheses represent the numbers of processes 345in each cgroup:: 346 347 A(4) - B(0) - C(1) 348 \ D(0) 349 350A, B and C's "populated" fields would be 1 while D's 0. After the one 351process in C exits, B and C's "populated" fields would flip to "0" and 352file modified events will be generated on the "cgroup.events" files of 353both cgroups. 354 355 356Controlling Controllers 357----------------------- 358 359Enabling and Disabling 360~~~~~~~~~~~~~~~~~~~~~~ 361 362Each cgroup has a "cgroup.controllers" file which lists all 363controllers available for the cgroup to enable:: 364 365 # cat cgroup.controllers 366 cpu io memory 367 368No controller is enabled by default. Controllers can be enabled and 369disabled by writing to the "cgroup.subtree_control" file:: 370 371 # echo "+cpu +memory -io" > cgroup.subtree_control 372 373Only controllers which are listed in "cgroup.controllers" can be 374enabled. When multiple operations are specified as above, either they 375all succeed or fail. If multiple operations on the same controller 376are specified, the last one is effective. 377 378Enabling a controller in a cgroup indicates that the distribution of 379the target resource across its immediate children will be controlled. 380Consider the following sub-hierarchy. The enabled controllers are 381listed in parentheses:: 382 383 A(cpu,memory) - B(memory) - C() 384 \ D() 385 386As A has "cpu" and "memory" enabled, A will control the distribution 387of CPU cycles and memory to its children, in this case, B. As B has 388"memory" enabled but not "CPU", C and D will compete freely on CPU 389cycles but their division of memory available to B will be controlled. 390 391As a controller regulates the distribution of the target resource to 392the cgroup's children, enabling it creates the controller's interface 393files in the child cgroups. In the above example, enabling "cpu" on B 394would create the "cpu." prefixed controller interface files in C and 395D. Likewise, disabling "memory" from B would remove the "memory." 396prefixed controller interface files from C and D. This means that the 397controller interface files - anything which doesn't start with 398"cgroup." are owned by the parent rather than the cgroup itself. 399 400 401Top-down Constraint 402~~~~~~~~~~~~~~~~~~~ 403 404Resources are distributed top-down and a cgroup can further distribute 405a resource only if the resource has been distributed to it from the 406parent. This means that all non-root "cgroup.subtree_control" files 407can only contain controllers which are enabled in the parent's 408"cgroup.subtree_control" file. A controller can be enabled only if 409the parent has the controller enabled and a controller can't be 410disabled if one or more children have it enabled. 411 412 413No Internal Process Constraint 414~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 415 416Non-root cgroups can distribute domain resources to their children 417only when they don't have any processes of their own. In other words, 418only domain cgroups which don't contain any processes can have domain 419controllers enabled in their "cgroup.subtree_control" files. 420 421This guarantees that, when a domain controller is looking at the part 422of the hierarchy which has it enabled, processes are always only on 423the leaves. This rules out situations where child cgroups compete 424against internal processes of the parent. 425 426The root cgroup is exempt from this restriction. Root contains 427processes and anonymous resource consumption which can't be associated 428with any other cgroups and requires special treatment from most 429controllers. How resource consumption in the root cgroup is governed 430is up to each controller (for more information on this topic please 431refer to the Non-normative information section in the Controllers 432chapter). 433 434Note that the restriction doesn't get in the way if there is no 435enabled controller in the cgroup's "cgroup.subtree_control". This is 436important as otherwise it wouldn't be possible to create children of a 437populated cgroup. To control resource distribution of a cgroup, the 438cgroup must create children and transfer all its processes to the 439children before enabling controllers in its "cgroup.subtree_control" 440file. 441 442 443Delegation 444---------- 445 446Model of Delegation 447~~~~~~~~~~~~~~~~~~~ 448 449A cgroup can be delegated in two ways. First, to a less privileged 450user by granting write access of the directory and its "cgroup.procs", 451"cgroup.threads" and "cgroup.subtree_control" files to the user. 452Second, if the "nsdelegate" mount option is set, automatically to a 453cgroup namespace on namespace creation. 454 455Because the resource control interface files in a given directory 456control the distribution of the parent's resources, the delegatee 457shouldn't be allowed to write to them. For the first method, this is 458achieved by not granting access to these files. For the second, the 459kernel rejects writes to all files other than "cgroup.procs" and 460"cgroup.subtree_control" on a namespace root from inside the 461namespace. 462 463The end results are equivalent for both delegation types. Once 464delegated, the user can build sub-hierarchy under the directory, 465organize processes inside it as it sees fit and further distribute the 466resources it received from the parent. The limits and other settings 467of all resource controllers are hierarchical and regardless of what 468happens in the delegated sub-hierarchy, nothing can escape the 469resource restrictions imposed by the parent. 470 471Currently, cgroup doesn't impose any restrictions on the number of 472cgroups in or nesting depth of a delegated sub-hierarchy; however, 473this may be limited explicitly in the future. 474 475 476Delegation Containment 477~~~~~~~~~~~~~~~~~~~~~~ 478 479A delegated sub-hierarchy is contained in the sense that processes 480can't be moved into or out of the sub-hierarchy by the delegatee. 481 482For delegations to a less privileged user, this is achieved by 483requiring the following conditions for a process with a non-root euid 484to migrate a target process into a cgroup by writing its PID to the 485"cgroup.procs" file. 486 487- The writer must have write access to the "cgroup.procs" file. 488 489- The writer must have write access to the "cgroup.procs" file of the 490 common ancestor of the source and destination cgroups. 491 492The above two constraints ensure that while a delegatee may migrate 493processes around freely in the delegated sub-hierarchy it can't pull 494in from or push out to outside the sub-hierarchy. 495 496For an example, let's assume cgroups C0 and C1 have been delegated to 497user U0 who created C00, C01 under C0 and C10 under C1 as follows and 498all processes under C0 and C1 belong to U0:: 499 500 ~~~~~~~~~~~~~ - C0 - C00 501 ~ cgroup ~ \ C01 502 ~ hierarchy ~ 503 ~~~~~~~~~~~~~ - C1 - C10 504 505Let's also say U0 wants to write the PID of a process which is 506currently in C10 into "C00/cgroup.procs". U0 has write access to the 507file; however, the common ancestor of the source cgroup C10 and the 508destination cgroup C00 is above the points of delegation and U0 would 509not have write access to its "cgroup.procs" files and thus the write 510will be denied with -EACCES. 511 512For delegations to namespaces, containment is achieved by requiring 513that both the source and destination cgroups are reachable from the 514namespace of the process which is attempting the migration. If either 515is not reachable, the migration is rejected with -ENOENT. 516 517 518Guidelines 519---------- 520 521Organize Once and Control 522~~~~~~~~~~~~~~~~~~~~~~~~~ 523 524Migrating a process across cgroups is a relatively expensive operation 525and stateful resources such as memory are not moved together with the 526process. This is an explicit design decision as there often exist 527inherent trade-offs between migration and various hot paths in terms 528of synchronization cost. 529 530As such, migrating processes across cgroups frequently as a means to 531apply different resource restrictions is discouraged. A workload 532should be assigned to a cgroup according to the system's logical and 533resource structure once on start-up. Dynamic adjustments to resource 534distribution can be made by changing controller configuration through 535the interface files. 536 537 538Avoid Name Collisions 539~~~~~~~~~~~~~~~~~~~~~ 540 541Interface files for a cgroup and its children cgroups occupy the same 542directory and it is possible to create children cgroups which collide 543with interface files. 544 545All cgroup core interface files are prefixed with "cgroup." and each 546controller's interface files are prefixed with the controller name and 547a dot. A controller's name is composed of lower case alphabets and 548'_'s but never begins with an '_' so it can be used as the prefix 549character for collision avoidance. Also, interface file names won't 550start or end with terms which are often used in categorizing workloads 551such as job, service, slice, unit or workload. 552 553cgroup doesn't do anything to prevent name collisions and it's the 554user's responsibility to avoid them. 555 556 557Resource Distribution Models 558============================ 559 560cgroup controllers implement several resource distribution schemes 561depending on the resource type and expected use cases. This section 562describes major schemes in use along with their expected behaviors. 563 564 565Weights 566------- 567 568A parent's resource is distributed by adding up the weights of all 569active children and giving each the fraction matching the ratio of its 570weight against the sum. As only children which can make use of the 571resource at the moment participate in the distribution, this is 572work-conserving. Due to the dynamic nature, this model is usually 573used for stateless resources. 574 575All weights are in the range [1, 10000] with the default at 100. This 576allows symmetric multiplicative biases in both directions at fine 577enough granularity while staying in the intuitive range. 578 579As long as the weight is in range, all configuration combinations are 580valid and there is no reason to reject configuration changes or 581process migrations. 582 583"cpu.weight" proportionally distributes CPU cycles to active children 584and is an example of this type. 585 586 587Limits 588------ 589 590A child can only consume upto the configured amount of the resource. 591Limits can be over-committed - the sum of the limits of children can 592exceed the amount of resource available to the parent. 593 594Limits are in the range [0, max] and defaults to "max", which is noop. 595 596As limits can be over-committed, all configuration combinations are 597valid and there is no reason to reject configuration changes or 598process migrations. 599 600"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume 601on an IO device and is an example of this type. 602 603 604Protections 605----------- 606 607A cgroup is protected to be allocated upto the configured amount of 608the resource if the usages of all its ancestors are under their 609protected levels. Protections can be hard guarantees or best effort 610soft boundaries. Protections can also be over-committed in which case 611only upto the amount available to the parent is protected among 612children. 613 614Protections are in the range [0, max] and defaults to 0, which is 615noop. 616 617As protections can be over-committed, all configuration combinations 618are valid and there is no reason to reject configuration changes or 619process migrations. 620 621"memory.low" implements best-effort memory protection and is an 622example of this type. 623 624 625Allocations 626----------- 627 628A cgroup is exclusively allocated a certain amount of a finite 629resource. Allocations can't be over-committed - the sum of the 630allocations of children can not exceed the amount of resource 631available to the parent. 632 633Allocations are in the range [0, max] and defaults to 0, which is no 634resource. 635 636As allocations can't be over-committed, some configuration 637combinations are invalid and should be rejected. Also, if the 638resource is mandatory for execution of processes, process migrations 639may be rejected. 640 641"cpu.rt.max" hard-allocates realtime slices and is an example of this 642type. 643 644 645Interface Files 646=============== 647 648Format 649------ 650 651All interface files should be in one of the following formats whenever 652possible:: 653 654 New-line separated values 655 (when only one value can be written at once) 656 657 VAL0\n 658 VAL1\n 659 ... 660 661 Space separated values 662 (when read-only or multiple values can be written at once) 663 664 VAL0 VAL1 ...\n 665 666 Flat keyed 667 668 KEY0 VAL0\n 669 KEY1 VAL1\n 670 ... 671 672 Nested keyed 673 674 KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01... 675 KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11... 676 ... 677 678For a writable file, the format for writing should generally match 679reading; however, controllers may allow omitting later fields or 680implement restricted shortcuts for most common use cases. 681 682For both flat and nested keyed files, only the values for a single key 683can be written at a time. For nested keyed files, the sub key pairs 684may be specified in any order and not all pairs have to be specified. 685 686 687Conventions 688----------- 689 690- Settings for a single feature should be contained in a single file. 691 692- The root cgroup should be exempt from resource control and thus 693 shouldn't have resource control interface files. Also, 694 informational files on the root cgroup which end up showing global 695 information available elsewhere shouldn't exist. 696 697- If a controller implements weight based resource distribution, its 698 interface file should be named "weight" and have the range [1, 699 10000] with 100 as the default. The values are chosen to allow 700 enough and symmetric bias in both directions while keeping it 701 intuitive (the default is 100%). 702 703- If a controller implements an absolute resource guarantee and/or 704 limit, the interface files should be named "min" and "max" 705 respectively. If a controller implements best effort resource 706 guarantee and/or limit, the interface files should be named "low" 707 and "high" respectively. 708 709 In the above four control files, the special token "max" should be 710 used to represent upward infinity for both reading and writing. 711 712- If a setting has a configurable default value and keyed specific 713 overrides, the default entry should be keyed with "default" and 714 appear as the first entry in the file. 715 716 The default value can be updated by writing either "default $VAL" or 717 "$VAL". 718 719 When writing to update a specific override, "default" can be used as 720 the value to indicate removal of the override. Override entries 721 with "default" as the value must not appear when read. 722 723 For example, a setting which is keyed by major:minor device numbers 724 with integer values may look like the following:: 725 726 # cat cgroup-example-interface-file 727 default 150 728 8:0 300 729 730 The default value can be updated by:: 731 732 # echo 125 > cgroup-example-interface-file 733 734 or:: 735 736 # echo "default 125" > cgroup-example-interface-file 737 738 An override can be set by:: 739 740 # echo "8:16 170" > cgroup-example-interface-file 741 742 and cleared by:: 743 744 # echo "8:0 default" > cgroup-example-interface-file 745 # cat cgroup-example-interface-file 746 default 125 747 8:16 170 748 749- For events which are not very high frequency, an interface file 750 "events" should be created which lists event key value pairs. 751 Whenever a notifiable event happens, file modified event should be 752 generated on the file. 753 754 755Core Interface Files 756-------------------- 757 758All cgroup core files are prefixed with "cgroup." 759 760 cgroup.type 761 762 A read-write single value file which exists on non-root 763 cgroups. 764 765 When read, it indicates the current type of the cgroup, which 766 can be one of the following values. 767 768 - "domain" : A normal valid domain cgroup. 769 770 - "domain threaded" : A threaded domain cgroup which is 771 serving as the root of a threaded subtree. 772 773 - "domain invalid" : A cgroup which is in an invalid state. 774 It can't be populated or have controllers enabled. It may 775 be allowed to become a threaded cgroup. 776 777 - "threaded" : A threaded cgroup which is a member of a 778 threaded subtree. 779 780 A cgroup can be turned into a threaded cgroup by writing 781 "threaded" to this file. 782 783 cgroup.procs 784 A read-write new-line separated values file which exists on 785 all cgroups. 786 787 When read, it lists the PIDs of all processes which belong to 788 the cgroup one-per-line. The PIDs are not ordered and the 789 same PID may show up more than once if the process got moved 790 to another cgroup and then back or the PID got recycled while 791 reading. 792 793 A PID can be written to migrate the process associated with 794 the PID to the cgroup. The writer should match all of the 795 following conditions. 796 797 - It must have write access to the "cgroup.procs" file. 798 799 - It must have write access to the "cgroup.procs" file of the 800 common ancestor of the source and destination cgroups. 801 802 When delegating a sub-hierarchy, write access to this file 803 should be granted along with the containing directory. 804 805 In a threaded cgroup, reading this file fails with EOPNOTSUPP 806 as all the processes belong to the thread root. Writing is 807 supported and moves every thread of the process to the cgroup. 808 809 cgroup.threads 810 A read-write new-line separated values file which exists on 811 all cgroups. 812 813 When read, it lists the TIDs of all threads which belong to 814 the cgroup one-per-line. The TIDs are not ordered and the 815 same TID may show up more than once if the thread got moved to 816 another cgroup and then back or the TID got recycled while 817 reading. 818 819 A TID can be written to migrate the thread associated with the 820 TID to the cgroup. The writer should match all of the 821 following conditions. 822 823 - It must have write access to the "cgroup.threads" file. 824 825 - The cgroup that the thread is currently in must be in the 826 same resource domain as the destination cgroup. 827 828 - It must have write access to the "cgroup.procs" file of the 829 common ancestor of the source and destination cgroups. 830 831 When delegating a sub-hierarchy, write access to this file 832 should be granted along with the containing directory. 833 834 cgroup.controllers 835 A read-only space separated values file which exists on all 836 cgroups. 837 838 It shows space separated list of all controllers available to 839 the cgroup. The controllers are not ordered. 840 841 cgroup.subtree_control 842 A read-write space separated values file which exists on all 843 cgroups. Starts out empty. 844 845 When read, it shows space separated list of the controllers 846 which are enabled to control resource distribution from the 847 cgroup to its children. 848 849 Space separated list of controllers prefixed with '+' or '-' 850 can be written to enable or disable controllers. A controller 851 name prefixed with '+' enables the controller and '-' 852 disables. If a controller appears more than once on the list, 853 the last one is effective. When multiple enable and disable 854 operations are specified, either all succeed or all fail. 855 856 cgroup.events 857 A read-only flat-keyed file which exists on non-root cgroups. 858 The following entries are defined. Unless specified 859 otherwise, a value change in this file generates a file 860 modified event. 861 862 populated 863 1 if the cgroup or its descendants contains any live 864 processes; otherwise, 0. 865 866 cgroup.max.descendants 867 A read-write single value files. The default is "max". 868 869 Maximum allowed number of descent cgroups. 870 If the actual number of descendants is equal or larger, 871 an attempt to create a new cgroup in the hierarchy will fail. 872 873 cgroup.max.depth 874 A read-write single value files. The default is "max". 875 876 Maximum allowed descent depth below the current cgroup. 877 If the actual descent depth is equal or larger, 878 an attempt to create a new child cgroup will fail. 879 880 cgroup.stat 881 A read-only flat-keyed file with the following entries: 882 883 nr_descendants 884 Total number of visible descendant cgroups. 885 886 nr_dying_descendants 887 Total number of dying descendant cgroups. A cgroup becomes 888 dying after being deleted by a user. The cgroup will remain 889 in dying state for some time undefined time (which can depend 890 on system load) before being completely destroyed. 891 892 A process can't enter a dying cgroup under any circumstances, 893 a dying cgroup can't revive. 894 895 A dying cgroup can consume system resources not exceeding 896 limits, which were active at the moment of cgroup deletion. 897 898 899Controllers 900=========== 901 902CPU 903--- 904 905The "cpu" controllers regulates distribution of CPU cycles. This 906controller implements weight and absolute bandwidth limit models for 907normal scheduling policy and absolute bandwidth allocation model for 908realtime scheduling policy. 909 910WARNING: cgroup2 doesn't yet support control of realtime processes and 911the cpu controller can only be enabled when all RT processes are in 912the root cgroup. Be aware that system management software may already 913have placed RT processes into nonroot cgroups during the system boot 914process, and these processes may need to be moved to the root cgroup 915before the cpu controller can be enabled. 916 917 918CPU Interface Files 919~~~~~~~~~~~~~~~~~~~ 920 921All time durations are in microseconds. 922 923 cpu.stat 924 A read-only flat-keyed file which exists on non-root cgroups. 925 This file exists whether the controller is enabled or not. 926 927 It always reports the following three stats: 928 929 - usage_usec 930 - user_usec 931 - system_usec 932 933 and the following three when the controller is enabled: 934 935 - nr_periods 936 - nr_throttled 937 - throttled_usec 938 939 cpu.weight 940 A read-write single value file which exists on non-root 941 cgroups. The default is "100". 942 943 The weight in the range [1, 10000]. 944 945 cpu.weight.nice 946 A read-write single value file which exists on non-root 947 cgroups. The default is "0". 948 949 The nice value is in the range [-20, 19]. 950 951 This interface file is an alternative interface for 952 "cpu.weight" and allows reading and setting weight using the 953 same values used by nice(2). Because the range is smaller and 954 granularity is coarser for the nice values, the read value is 955 the closest approximation of the current weight. 956 957 cpu.max 958 A read-write two value file which exists on non-root cgroups. 959 The default is "max 100000". 960 961 The maximum bandwidth limit. It's in the following format:: 962 963 $MAX $PERIOD 964 965 which indicates that the group may consume upto $MAX in each 966 $PERIOD duration. "max" for $MAX indicates no limit. If only 967 one number is written, $MAX is updated. 968 969 cpu.pressure 970 A read-only nested-key file which exists on non-root cgroups. 971 972 Shows pressure stall information for CPU. See 973 Documentation/accounting/psi.txt for details. 974 975 976Memory 977------ 978 979The "memory" controller regulates distribution of memory. Memory is 980stateful and implements both limit and protection models. Due to the 981intertwining between memory usage and reclaim pressure and the 982stateful nature of memory, the distribution model is relatively 983complex. 984 985While not completely water-tight, all major memory usages by a given 986cgroup are tracked so that the total memory consumption can be 987accounted and controlled to a reasonable extent. Currently, the 988following types of memory usages are tracked. 989 990- Userland memory - page cache and anonymous memory. 991 992- Kernel data structures such as dentries and inodes. 993 994- TCP socket buffers. 995 996The above list may expand in the future for better coverage. 997 998 999Memory Interface Files 1000~~~~~~~~~~~~~~~~~~~~~~ 1001 1002All memory amounts are in bytes. If a value which is not aligned to 1003PAGE_SIZE is written, the value may be rounded up to the closest 1004PAGE_SIZE multiple when read back. 1005 1006 memory.current 1007 A read-only single value file which exists on non-root 1008 cgroups. 1009 1010 The total amount of memory currently being used by the cgroup 1011 and its descendants. 1012 1013 memory.min 1014 A read-write single value file which exists on non-root 1015 cgroups. The default is "0". 1016 1017 Hard memory protection. If the memory usage of a cgroup 1018 is within its effective min boundary, the cgroup's memory 1019 won't be reclaimed under any conditions. If there is no 1020 unprotected reclaimable memory available, OOM killer 1021 is invoked. 1022 1023 Effective min boundary is limited by memory.min values of 1024 all ancestor cgroups. If there is memory.min overcommitment 1025 (child cgroup or cgroups are requiring more protected memory 1026 than parent will allow), then each child cgroup will get 1027 the part of parent's protection proportional to its 1028 actual memory usage below memory.min. 1029 1030 Putting more memory than generally available under this 1031 protection is discouraged and may lead to constant OOMs. 1032 1033 If a memory cgroup is not populated with processes, 1034 its memory.min is ignored. 1035 1036 memory.low 1037 A read-write single value file which exists on non-root 1038 cgroups. The default is "0". 1039 1040 Best-effort memory protection. If the memory usage of a 1041 cgroup is within its effective low boundary, the cgroup's 1042 memory won't be reclaimed unless memory can be reclaimed 1043 from unprotected cgroups. 1044 1045 Effective low boundary is limited by memory.low values of 1046 all ancestor cgroups. If there is memory.low overcommitment 1047 (child cgroup or cgroups are requiring more protected memory 1048 than parent will allow), then each child cgroup will get 1049 the part of parent's protection proportional to its 1050 actual memory usage below memory.low. 1051 1052 Putting more memory than generally available under this 1053 protection is discouraged. 1054 1055 memory.high 1056 A read-write single value file which exists on non-root 1057 cgroups. The default is "max". 1058 1059 Memory usage throttle limit. This is the main mechanism to 1060 control memory usage of a cgroup. If a cgroup's usage goes 1061 over the high boundary, the processes of the cgroup are 1062 throttled and put under heavy reclaim pressure. 1063 1064 Going over the high limit never invokes the OOM killer and 1065 under extreme conditions the limit may be breached. 1066 1067 memory.max 1068 A read-write single value file which exists on non-root 1069 cgroups. The default is "max". 1070 1071 Memory usage hard limit. This is the final protection 1072 mechanism. If a cgroup's memory usage reaches this limit and 1073 can't be reduced, the OOM killer is invoked in the cgroup. 1074 Under certain circumstances, the usage may go over the limit 1075 temporarily. 1076 1077 This is the ultimate protection mechanism. As long as the 1078 high limit is used and monitored properly, this limit's 1079 utility is limited to providing the final safety net. 1080 1081 memory.oom.group 1082 A read-write single value file which exists on non-root 1083 cgroups. The default value is "0". 1084 1085 Determines whether the cgroup should be treated as 1086 an indivisible workload by the OOM killer. If set, 1087 all tasks belonging to the cgroup or to its descendants 1088 (if the memory cgroup is not a leaf cgroup) are killed 1089 together or not at all. This can be used to avoid 1090 partial kills to guarantee workload integrity. 1091 1092 Tasks with the OOM protection (oom_score_adj set to -1000) 1093 are treated as an exception and are never killed. 1094 1095 If the OOM killer is invoked in a cgroup, it's not going 1096 to kill any tasks outside of this cgroup, regardless 1097 memory.oom.group values of ancestor cgroups. 1098 1099 memory.events 1100 A read-only flat-keyed file which exists on non-root cgroups. 1101 The following entries are defined. Unless specified 1102 otherwise, a value change in this file generates a file 1103 modified event. 1104 1105 low 1106 The number of times the cgroup is reclaimed due to 1107 high memory pressure even though its usage is under 1108 the low boundary. This usually indicates that the low 1109 boundary is over-committed. 1110 1111 high 1112 The number of times processes of the cgroup are 1113 throttled and routed to perform direct memory reclaim 1114 because the high memory boundary was exceeded. For a 1115 cgroup whose memory usage is capped by the high limit 1116 rather than global memory pressure, this event's 1117 occurrences are expected. 1118 1119 max 1120 The number of times the cgroup's memory usage was 1121 about to go over the max boundary. If direct reclaim 1122 fails to bring it down, the cgroup goes to OOM state. 1123 1124 oom 1125 The number of time the cgroup's memory usage was 1126 reached the limit and allocation was about to fail. 1127 1128 Depending on context result could be invocation of OOM 1129 killer and retrying allocation or failing allocation. 1130 1131 Failed allocation in its turn could be returned into 1132 userspace as -ENOMEM or silently ignored in cases like 1133 disk readahead. For now OOM in memory cgroup kills 1134 tasks iff shortage has happened inside page fault. 1135 1136 This event is not raised if the OOM killer is not 1137 considered as an option, e.g. for failed high-order 1138 allocations. 1139 1140 oom_kill 1141 The number of processes belonging to this cgroup 1142 killed by any kind of OOM killer. 1143 1144 memory.stat 1145 A read-only flat-keyed file which exists on non-root cgroups. 1146 1147 This breaks down the cgroup's memory footprint into different 1148 types of memory, type-specific details, and other information 1149 on the state and past events of the memory management system. 1150 1151 All memory amounts are in bytes. 1152 1153 The entries are ordered to be human readable, and new entries 1154 can show up in the middle. Don't rely on items remaining in a 1155 fixed position; use the keys to look up specific values! 1156 1157 anon 1158 Amount of memory used in anonymous mappings such as 1159 brk(), sbrk(), and mmap(MAP_ANONYMOUS) 1160 1161 file 1162 Amount of memory used to cache filesystem data, 1163 including tmpfs and shared memory. 1164 1165 kernel_stack 1166 Amount of memory allocated to kernel stacks. 1167 1168 slab 1169 Amount of memory used for storing in-kernel data 1170 structures. 1171 1172 sock 1173 Amount of memory used in network transmission buffers 1174 1175 shmem 1176 Amount of cached filesystem data that is swap-backed, 1177 such as tmpfs, shm segments, shared anonymous mmap()s 1178 1179 file_mapped 1180 Amount of cached filesystem data mapped with mmap() 1181 1182 file_dirty 1183 Amount of cached filesystem data that was modified but 1184 not yet written back to disk 1185 1186 file_writeback 1187 Amount of cached filesystem data that was modified and 1188 is currently being written back to disk 1189 1190 inactive_anon, active_anon, inactive_file, active_file, unevictable 1191 Amount of memory, swap-backed and filesystem-backed, 1192 on the internal memory management lists used by the 1193 page reclaim algorithm 1194 1195 slab_reclaimable 1196 Part of "slab" that might be reclaimed, such as 1197 dentries and inodes. 1198 1199 slab_unreclaimable 1200 Part of "slab" that cannot be reclaimed on memory 1201 pressure. 1202 1203 pgfault 1204 Total number of page faults incurred 1205 1206 pgmajfault 1207 Number of major page faults incurred 1208 1209 workingset_refault 1210 1211 Number of refaults of previously evicted pages 1212 1213 workingset_activate 1214 1215 Number of refaulted pages that were immediately activated 1216 1217 workingset_nodereclaim 1218 1219 Number of times a shadow node has been reclaimed 1220 1221 pgrefill 1222 1223 Amount of scanned pages (in an active LRU list) 1224 1225 pgscan 1226 1227 Amount of scanned pages (in an inactive LRU list) 1228 1229 pgsteal 1230 1231 Amount of reclaimed pages 1232 1233 pgactivate 1234 1235 Amount of pages moved to the active LRU list 1236 1237 pgdeactivate 1238 1239 Amount of pages moved to the inactive LRU lis 1240 1241 pglazyfree 1242 1243 Amount of pages postponed to be freed under memory pressure 1244 1245 pglazyfreed 1246 1247 Amount of reclaimed lazyfree pages 1248 1249 memory.swap.current 1250 A read-only single value file which exists on non-root 1251 cgroups. 1252 1253 The total amount of swap currently being used by the cgroup 1254 and its descendants. 1255 1256 memory.swap.max 1257 A read-write single value file which exists on non-root 1258 cgroups. The default is "max". 1259 1260 Swap usage hard limit. If a cgroup's swap usage reaches this 1261 limit, anonymous memory of the cgroup will not be swapped out. 1262 1263 memory.swap.events 1264 A read-only flat-keyed file which exists on non-root cgroups. 1265 The following entries are defined. Unless specified 1266 otherwise, a value change in this file generates a file 1267 modified event. 1268 1269 max 1270 The number of times the cgroup's swap usage was about 1271 to go over the max boundary and swap allocation 1272 failed. 1273 1274 fail 1275 The number of times swap allocation failed either 1276 because of running out of swap system-wide or max 1277 limit. 1278 1279 When reduced under the current usage, the existing swap 1280 entries are reclaimed gradually and the swap usage may stay 1281 higher than the limit for an extended period of time. This 1282 reduces the impact on the workload and memory management. 1283 1284 memory.pressure 1285 A read-only nested-key file which exists on non-root cgroups. 1286 1287 Shows pressure stall information for memory. See 1288 Documentation/accounting/psi.txt for details. 1289 1290 1291Usage Guidelines 1292~~~~~~~~~~~~~~~~ 1293 1294"memory.high" is the main mechanism to control memory usage. 1295Over-committing on high limit (sum of high limits > available memory) 1296and letting global memory pressure to distribute memory according to 1297usage is a viable strategy. 1298 1299Because breach of the high limit doesn't trigger the OOM killer but 1300throttles the offending cgroup, a management agent has ample 1301opportunities to monitor and take appropriate actions such as granting 1302more memory or terminating the workload. 1303 1304Determining whether a cgroup has enough memory is not trivial as 1305memory usage doesn't indicate whether the workload can benefit from 1306more memory. For example, a workload which writes data received from 1307network to a file can use all available memory but can also operate as 1308performant with a small amount of memory. A measure of memory 1309pressure - how much the workload is being impacted due to lack of 1310memory - is necessary to determine whether a workload needs more 1311memory; unfortunately, memory pressure monitoring mechanism isn't 1312implemented yet. 1313 1314 1315Memory Ownership 1316~~~~~~~~~~~~~~~~ 1317 1318A memory area is charged to the cgroup which instantiated it and stays 1319charged to the cgroup until the area is released. Migrating a process 1320to a different cgroup doesn't move the memory usages that it 1321instantiated while in the previous cgroup to the new cgroup. 1322 1323A memory area may be used by processes belonging to different cgroups. 1324To which cgroup the area will be charged is in-deterministic; however, 1325over time, the memory area is likely to end up in a cgroup which has 1326enough memory allowance to avoid high reclaim pressure. 1327 1328If a cgroup sweeps a considerable amount of memory which is expected 1329to be accessed repeatedly by other cgroups, it may make sense to use 1330POSIX_FADV_DONTNEED to relinquish the ownership of memory areas 1331belonging to the affected files to ensure correct memory ownership. 1332 1333 1334IO 1335-- 1336 1337The "io" controller regulates the distribution of IO resources. This 1338controller implements both weight based and absolute bandwidth or IOPS 1339limit distribution; however, weight based distribution is available 1340only if cfq-iosched is in use and neither scheme is available for 1341blk-mq devices. 1342 1343 1344IO Interface Files 1345~~~~~~~~~~~~~~~~~~ 1346 1347 io.stat 1348 A read-only nested-keyed file which exists on non-root 1349 cgroups. 1350 1351 Lines are keyed by $MAJ:$MIN device numbers and not ordered. 1352 The following nested keys are defined. 1353 1354 ====== ===================== 1355 rbytes Bytes read 1356 wbytes Bytes written 1357 rios Number of read IOs 1358 wios Number of write IOs 1359 dbytes Bytes discarded 1360 dios Number of discard IOs 1361 ====== ===================== 1362 1363 An example read output follows: 1364 1365 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0 1366 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021 1367 1368 io.weight 1369 A read-write flat-keyed file which exists on non-root cgroups. 1370 The default is "default 100". 1371 1372 The first line is the default weight applied to devices 1373 without specific override. The rest are overrides keyed by 1374 $MAJ:$MIN device numbers and not ordered. The weights are in 1375 the range [1, 10000] and specifies the relative amount IO time 1376 the cgroup can use in relation to its siblings. 1377 1378 The default weight can be updated by writing either "default 1379 $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing 1380 "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default". 1381 1382 An example read output follows:: 1383 1384 default 100 1385 8:16 200 1386 8:0 50 1387 1388 io.max 1389 A read-write nested-keyed file which exists on non-root 1390 cgroups. 1391 1392 BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN 1393 device numbers and not ordered. The following nested keys are 1394 defined. 1395 1396 ===== ================================== 1397 rbps Max read bytes per second 1398 wbps Max write bytes per second 1399 riops Max read IO operations per second 1400 wiops Max write IO operations per second 1401 ===== ================================== 1402 1403 When writing, any number of nested key-value pairs can be 1404 specified in any order. "max" can be specified as the value 1405 to remove a specific limit. If the same key is specified 1406 multiple times, the outcome is undefined. 1407 1408 BPS and IOPS are measured in each IO direction and IOs are 1409 delayed if limit is reached. Temporary bursts are allowed. 1410 1411 Setting read limit at 2M BPS and write at 120 IOPS for 8:16:: 1412 1413 echo "8:16 rbps=2097152 wiops=120" > io.max 1414 1415 Reading returns the following:: 1416 1417 8:16 rbps=2097152 wbps=max riops=max wiops=120 1418 1419 Write IOPS limit can be removed by writing the following:: 1420 1421 echo "8:16 wiops=max" > io.max 1422 1423 Reading now returns the following:: 1424 1425 8:16 rbps=2097152 wbps=max riops=max wiops=max 1426 1427 io.pressure 1428 A read-only nested-key file which exists on non-root cgroups. 1429 1430 Shows pressure stall information for IO. See 1431 Documentation/accounting/psi.txt for details. 1432 1433 1434Writeback 1435~~~~~~~~~ 1436 1437Page cache is dirtied through buffered writes and shared mmaps and 1438written asynchronously to the backing filesystem by the writeback 1439mechanism. Writeback sits between the memory and IO domains and 1440regulates the proportion of dirty memory by balancing dirtying and 1441write IOs. 1442 1443The io controller, in conjunction with the memory controller, 1444implements control of page cache writeback IOs. The memory controller 1445defines the memory domain that dirty memory ratio is calculated and 1446maintained for and the io controller defines the io domain which 1447writes out dirty pages for the memory domain. Both system-wide and 1448per-cgroup dirty memory states are examined and the more restrictive 1449of the two is enforced. 1450 1451cgroup writeback requires explicit support from the underlying 1452filesystem. Currently, cgroup writeback is implemented on ext2, ext4 1453and btrfs. On other filesystems, all writeback IOs are attributed to 1454the root cgroup. 1455 1456There are inherent differences in memory and writeback management 1457which affects how cgroup ownership is tracked. Memory is tracked per 1458page while writeback per inode. For the purpose of writeback, an 1459inode is assigned to a cgroup and all IO requests to write dirty pages 1460from the inode are attributed to that cgroup. 1461 1462As cgroup ownership for memory is tracked per page, there can be pages 1463which are associated with different cgroups than the one the inode is 1464associated with. These are called foreign pages. The writeback 1465constantly keeps track of foreign pages and, if a particular foreign 1466cgroup becomes the majority over a certain period of time, switches 1467the ownership of the inode to that cgroup. 1468 1469While this model is enough for most use cases where a given inode is 1470mostly dirtied by a single cgroup even when the main writing cgroup 1471changes over time, use cases where multiple cgroups write to a single 1472inode simultaneously are not supported well. In such circumstances, a 1473significant portion of IOs are likely to be attributed incorrectly. 1474As memory controller assigns page ownership on the first use and 1475doesn't update it until the page is released, even if writeback 1476strictly follows page ownership, multiple cgroups dirtying overlapping 1477areas wouldn't work as expected. It's recommended to avoid such usage 1478patterns. 1479 1480The sysctl knobs which affect writeback behavior are applied to cgroup 1481writeback as follows. 1482 1483 vm.dirty_background_ratio, vm.dirty_ratio 1484 These ratios apply the same to cgroup writeback with the 1485 amount of available memory capped by limits imposed by the 1486 memory controller and system-wide clean memory. 1487 1488 vm.dirty_background_bytes, vm.dirty_bytes 1489 For cgroup writeback, this is calculated into ratio against 1490 total available memory and applied the same way as 1491 vm.dirty[_background]_ratio. 1492 1493 1494IO Latency 1495~~~~~~~~~~ 1496 1497This is a cgroup v2 controller for IO workload protection. You provide a group 1498with a latency target, and if the average latency exceeds that target the 1499controller will throttle any peers that have a lower latency target than the 1500protected workload. 1501 1502The limits are only applied at the peer level in the hierarchy. This means that 1503in the diagram below, only groups A, B, and C will influence each other, and 1504groups D and F will influence each other. Group G will influence nobody. 1505 1506 [root] 1507 / | \ 1508 A B C 1509 / \ | 1510 D F G 1511 1512 1513So the ideal way to configure this is to set io.latency in groups A, B, and C. 1514Generally you do not want to set a value lower than the latency your device 1515supports. Experiment to find the value that works best for your workload. 1516Start at higher than the expected latency for your device and watch the 1517avg_lat value in io.stat for your workload group to get an idea of the 1518latency you see during normal operation. Use the avg_lat value as a basis for 1519your real setting, setting at 10-15% higher than the value in io.stat. 1520 1521How IO Latency Throttling Works 1522~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1523 1524io.latency is work conserving; so as long as everybody is meeting their latency 1525target the controller doesn't do anything. Once a group starts missing its 1526target it begins throttling any peer group that has a higher target than itself. 1527This throttling takes 2 forms: 1528 1529- Queue depth throttling. This is the number of outstanding IO's a group is 1530 allowed to have. We will clamp down relatively quickly, starting at no limit 1531 and going all the way down to 1 IO at a time. 1532 1533- Artificial delay induction. There are certain types of IO that cannot be 1534 throttled without possibly adversely affecting higher priority groups. This 1535 includes swapping and metadata IO. These types of IO are allowed to occur 1536 normally, however they are "charged" to the originating group. If the 1537 originating group is being throttled you will see the use_delay and delay 1538 fields in io.stat increase. The delay value is how many microseconds that are 1539 being added to any process that runs in this group. Because this number can 1540 grow quite large if there is a lot of swapping or metadata IO occurring we 1541 limit the individual delay events to 1 second at a time. 1542 1543Once the victimized group starts meeting its latency target again it will start 1544unthrottling any peer groups that were throttled previously. If the victimized 1545group simply stops doing IO the global counter will unthrottle appropriately. 1546 1547IO Latency Interface Files 1548~~~~~~~~~~~~~~~~~~~~~~~~~~ 1549 1550 io.latency 1551 This takes a similar format as the other controllers. 1552 1553 "MAJOR:MINOR target=<target time in microseconds" 1554 1555 io.stat 1556 If the controller is enabled you will see extra stats in io.stat in 1557 addition to the normal ones. 1558 1559 depth 1560 This is the current queue depth for the group. 1561 1562 avg_lat 1563 This is an exponential moving average with a decay rate of 1/exp 1564 bound by the sampling interval. The decay rate interval can be 1565 calculated by multiplying the win value in io.stat by the 1566 corresponding number of samples based on the win value. 1567 1568 win 1569 The sampling window size in milliseconds. This is the minimum 1570 duration of time between evaluation events. Windows only elapse 1571 with IO activity. Idle periods extend the most recent window. 1572 1573PID 1574--- 1575 1576The process number controller is used to allow a cgroup to stop any 1577new tasks from being fork()'d or clone()'d after a specified limit is 1578reached. 1579 1580The number of tasks in a cgroup can be exhausted in ways which other 1581controllers cannot prevent, thus warranting its own controller. For 1582example, a fork bomb is likely to exhaust the number of tasks before 1583hitting memory restrictions. 1584 1585Note that PIDs used in this controller refer to TIDs, process IDs as 1586used by the kernel. 1587 1588 1589PID Interface Files 1590~~~~~~~~~~~~~~~~~~~ 1591 1592 pids.max 1593 A read-write single value file which exists on non-root 1594 cgroups. The default is "max". 1595 1596 Hard limit of number of processes. 1597 1598 pids.current 1599 A read-only single value file which exists on all cgroups. 1600 1601 The number of processes currently in the cgroup and its 1602 descendants. 1603 1604Organisational operations are not blocked by cgroup policies, so it is 1605possible to have pids.current > pids.max. This can be done by either 1606setting the limit to be smaller than pids.current, or attaching enough 1607processes to the cgroup such that pids.current is larger than 1608pids.max. However, it is not possible to violate a cgroup PID policy 1609through fork() or clone(). These will return -EAGAIN if the creation 1610of a new process would cause a cgroup policy to be violated. 1611 1612 1613Device controller 1614----------------- 1615 1616Device controller manages access to device files. It includes both 1617creation of new device files (using mknod), and access to the 1618existing device files. 1619 1620Cgroup v2 device controller has no interface files and is implemented 1621on top of cgroup BPF. To control access to device files, a user may 1622create bpf programs of the BPF_CGROUP_DEVICE type and attach them 1623to cgroups. On an attempt to access a device file, corresponding 1624BPF programs will be executed, and depending on the return value 1625the attempt will succeed or fail with -EPERM. 1626 1627A BPF_CGROUP_DEVICE program takes a pointer to the bpf_cgroup_dev_ctx 1628structure, which describes the device access attempt: access type 1629(mknod/read/write) and device (type, major and minor numbers). 1630If the program returns 0, the attempt fails with -EPERM, otherwise 1631it succeeds. 1632 1633An example of BPF_CGROUP_DEVICE program may be found in the kernel 1634source tree in the tools/testing/selftests/bpf/dev_cgroup.c file. 1635 1636 1637RDMA 1638---- 1639 1640The "rdma" controller regulates the distribution and accounting of 1641of RDMA resources. 1642 1643RDMA Interface Files 1644~~~~~~~~~~~~~~~~~~~~ 1645 1646 rdma.max 1647 A readwrite nested-keyed file that exists for all the cgroups 1648 except root that describes current configured resource limit 1649 for a RDMA/IB device. 1650 1651 Lines are keyed by device name and are not ordered. 1652 Each line contains space separated resource name and its configured 1653 limit that can be distributed. 1654 1655 The following nested keys are defined. 1656 1657 ========== ============================= 1658 hca_handle Maximum number of HCA Handles 1659 hca_object Maximum number of HCA Objects 1660 ========== ============================= 1661 1662 An example for mlx4 and ocrdma device follows:: 1663 1664 mlx4_0 hca_handle=2 hca_object=2000 1665 ocrdma1 hca_handle=3 hca_object=max 1666 1667 rdma.current 1668 A read-only file that describes current resource usage. 1669 It exists for all the cgroup except root. 1670 1671 An example for mlx4 and ocrdma device follows:: 1672 1673 mlx4_0 hca_handle=1 hca_object=20 1674 ocrdma1 hca_handle=1 hca_object=23 1675 1676 1677Misc 1678---- 1679 1680perf_event 1681~~~~~~~~~~ 1682 1683perf_event controller, if not mounted on a legacy hierarchy, is 1684automatically enabled on the v2 hierarchy so that perf events can 1685always be filtered by cgroup v2 path. The controller can still be 1686moved to a legacy hierarchy after v2 hierarchy is populated. 1687 1688 1689Non-normative information 1690------------------------- 1691 1692This section contains information that isn't considered to be a part of 1693the stable kernel API and so is subject to change. 1694 1695 1696CPU controller root cgroup process behaviour 1697~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1698 1699When distributing CPU cycles in the root cgroup each thread in this 1700cgroup is treated as if it was hosted in a separate child cgroup of the 1701root cgroup. This child cgroup weight is dependent on its thread nice 1702level. 1703 1704For details of this mapping see sched_prio_to_weight array in 1705kernel/sched/core.c file (values from this array should be scaled 1706appropriately so the neutral - nice 0 - value is 100 instead of 1024). 1707 1708 1709IO controller root cgroup process behaviour 1710~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1711 1712Root cgroup processes are hosted in an implicit leaf child node. 1713When distributing IO resources this implicit child node is taken into 1714account as if it was a normal child cgroup of the root cgroup with a 1715weight value of 200. 1716 1717 1718Namespace 1719========= 1720 1721Basics 1722------ 1723 1724cgroup namespace provides a mechanism to virtualize the view of the 1725"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone 1726flag can be used with clone(2) and unshare(2) to create a new cgroup 1727namespace. The process running inside the cgroup namespace will have 1728its "/proc/$PID/cgroup" output restricted to cgroupns root. The 1729cgroupns root is the cgroup of the process at the time of creation of 1730the cgroup namespace. 1731 1732Without cgroup namespace, the "/proc/$PID/cgroup" file shows the 1733complete path of the cgroup of a process. In a container setup where 1734a set of cgroups and namespaces are intended to isolate processes the 1735"/proc/$PID/cgroup" file may leak potential system level information 1736to the isolated processes. For Example:: 1737 1738 # cat /proc/self/cgroup 1739 0::/batchjobs/container_id1 1740 1741The path '/batchjobs/container_id1' can be considered as system-data 1742and undesirable to expose to the isolated processes. cgroup namespace 1743can be used to restrict visibility of this path. For example, before 1744creating a cgroup namespace, one would see:: 1745 1746 # ls -l /proc/self/ns/cgroup 1747 lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835] 1748 # cat /proc/self/cgroup 1749 0::/batchjobs/container_id1 1750 1751After unsharing a new namespace, the view changes:: 1752 1753 # ls -l /proc/self/ns/cgroup 1754 lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183] 1755 # cat /proc/self/cgroup 1756 0::/ 1757 1758When some thread from a multi-threaded process unshares its cgroup 1759namespace, the new cgroupns gets applied to the entire process (all 1760the threads). This is natural for the v2 hierarchy; however, for the 1761legacy hierarchies, this may be unexpected. 1762 1763A cgroup namespace is alive as long as there are processes inside or 1764mounts pinning it. When the last usage goes away, the cgroup 1765namespace is destroyed. The cgroupns root and the actual cgroups 1766remain. 1767 1768 1769The Root and Views 1770------------------ 1771 1772The 'cgroupns root' for a cgroup namespace is the cgroup in which the 1773process calling unshare(2) is running. For example, if a process in 1774/batchjobs/container_id1 cgroup calls unshare, cgroup 1775/batchjobs/container_id1 becomes the cgroupns root. For the 1776init_cgroup_ns, this is the real root ('/') cgroup. 1777 1778The cgroupns root cgroup does not change even if the namespace creator 1779process later moves to a different cgroup:: 1780 1781 # ~/unshare -c # unshare cgroupns in some cgroup 1782 # cat /proc/self/cgroup 1783 0::/ 1784 # mkdir sub_cgrp_1 1785 # echo 0 > sub_cgrp_1/cgroup.procs 1786 # cat /proc/self/cgroup 1787 0::/sub_cgrp_1 1788 1789Each process gets its namespace-specific view of "/proc/$PID/cgroup" 1790 1791Processes running inside the cgroup namespace will be able to see 1792cgroup paths (in /proc/self/cgroup) only inside their root cgroup. 1793From within an unshared cgroupns:: 1794 1795 # sleep 100000 & 1796 [1] 7353 1797 # echo 7353 > sub_cgrp_1/cgroup.procs 1798 # cat /proc/7353/cgroup 1799 0::/sub_cgrp_1 1800 1801From the initial cgroup namespace, the real cgroup path will be 1802visible:: 1803 1804 $ cat /proc/7353/cgroup 1805 0::/batchjobs/container_id1/sub_cgrp_1 1806 1807From a sibling cgroup namespace (that is, a namespace rooted at a 1808different cgroup), the cgroup path relative to its own cgroup 1809namespace root will be shown. For instance, if PID 7353's cgroup 1810namespace root is at '/batchjobs/container_id2', then it will see:: 1811 1812 # cat /proc/7353/cgroup 1813 0::/../container_id2/sub_cgrp_1 1814 1815Note that the relative path always starts with '/' to indicate that 1816its relative to the cgroup namespace root of the caller. 1817 1818 1819Migration and setns(2) 1820---------------------- 1821 1822Processes inside a cgroup namespace can move into and out of the 1823namespace root if they have proper access to external cgroups. For 1824example, from inside a namespace with cgroupns root at 1825/batchjobs/container_id1, and assuming that the global hierarchy is 1826still accessible inside cgroupns:: 1827 1828 # cat /proc/7353/cgroup 1829 0::/sub_cgrp_1 1830 # echo 7353 > batchjobs/container_id2/cgroup.procs 1831 # cat /proc/7353/cgroup 1832 0::/../container_id2 1833 1834Note that this kind of setup is not encouraged. A task inside cgroup 1835namespace should only be exposed to its own cgroupns hierarchy. 1836 1837setns(2) to another cgroup namespace is allowed when: 1838 1839(a) the process has CAP_SYS_ADMIN against its current user namespace 1840(b) the process has CAP_SYS_ADMIN against the target cgroup 1841 namespace's userns 1842 1843No implicit cgroup changes happen with attaching to another cgroup 1844namespace. It is expected that the someone moves the attaching 1845process under the target cgroup namespace root. 1846 1847 1848Interaction with Other Namespaces 1849--------------------------------- 1850 1851Namespace specific cgroup hierarchy can be mounted by a process 1852running inside a non-init cgroup namespace:: 1853 1854 # mount -t cgroup2 none $MOUNT_POINT 1855 1856This will mount the unified cgroup hierarchy with cgroupns root as the 1857filesystem root. The process needs CAP_SYS_ADMIN against its user and 1858mount namespaces. 1859 1860The virtualization of /proc/self/cgroup file combined with restricting 1861the view of cgroup hierarchy by namespace-private cgroupfs mount 1862provides a properly isolated cgroup view inside the container. 1863 1864 1865Information on Kernel Programming 1866================================= 1867 1868This section contains kernel programming information in the areas 1869where interacting with cgroup is necessary. cgroup core and 1870controllers are not covered. 1871 1872 1873Filesystem Support for Writeback 1874-------------------------------- 1875 1876A filesystem can support cgroup writeback by updating 1877address_space_operations->writepage[s]() to annotate bio's using the 1878following two functions. 1879 1880 wbc_init_bio(@wbc, @bio) 1881 Should be called for each bio carrying writeback data and 1882 associates the bio with the inode's owner cgroup. Can be 1883 called anytime between bio allocation and submission. 1884 1885 wbc_account_io(@wbc, @page, @bytes) 1886 Should be called for each data segment being written out. 1887 While this function doesn't care exactly when it's called 1888 during the writeback session, it's the easiest and most 1889 natural to call it as data segments are added to a bio. 1890 1891With writeback bio's annotated, cgroup support can be enabled per 1892super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for 1893selective disabling of cgroup writeback support which is helpful when 1894certain filesystem features, e.g. journaled data mode, are 1895incompatible. 1896 1897wbc_init_bio() binds the specified bio to its cgroup. Depending on 1898the configuration, the bio may be executed at a lower priority and if 1899the writeback session is holding shared resources, e.g. a journal 1900entry, may lead to priority inversion. There is no one easy solution 1901for the problem. Filesystems can try to work around specific problem 1902cases by skipping wbc_init_bio() or using bio_associate_blkcg() 1903directly. 1904 1905 1906Deprecated v1 Core Features 1907=========================== 1908 1909- Multiple hierarchies including named ones are not supported. 1910 1911- All v1 mount options are not supported. 1912 1913- The "tasks" file is removed and "cgroup.procs" is not sorted. 1914 1915- "cgroup.clone_children" is removed. 1916 1917- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file 1918 at the root instead. 1919 1920 1921Issues with v1 and Rationales for v2 1922==================================== 1923 1924Multiple Hierarchies 1925-------------------- 1926 1927cgroup v1 allowed an arbitrary number of hierarchies and each 1928hierarchy could host any number of controllers. While this seemed to 1929provide a high level of flexibility, it wasn't useful in practice. 1930 1931For example, as there is only one instance of each controller, utility 1932type controllers such as freezer which can be useful in all 1933hierarchies could only be used in one. The issue is exacerbated by 1934the fact that controllers couldn't be moved to another hierarchy once 1935hierarchies were populated. Another issue was that all controllers 1936bound to a hierarchy were forced to have exactly the same view of the 1937hierarchy. It wasn't possible to vary the granularity depending on 1938the specific controller. 1939 1940In practice, these issues heavily limited which controllers could be 1941put on the same hierarchy and most configurations resorted to putting 1942each controller on its own hierarchy. Only closely related ones, such 1943as the cpu and cpuacct controllers, made sense to be put on the same 1944hierarchy. This often meant that userland ended up managing multiple 1945similar hierarchies repeating the same steps on each hierarchy 1946whenever a hierarchy management operation was necessary. 1947 1948Furthermore, support for multiple hierarchies came at a steep cost. 1949It greatly complicated cgroup core implementation but more importantly 1950the support for multiple hierarchies restricted how cgroup could be 1951used in general and what controllers was able to do. 1952 1953There was no limit on how many hierarchies there might be, which meant 1954that a thread's cgroup membership couldn't be described in finite 1955length. The key might contain any number of entries and was unlimited 1956in length, which made it highly awkward to manipulate and led to 1957addition of controllers which existed only to identify membership, 1958which in turn exacerbated the original problem of proliferating number 1959of hierarchies. 1960 1961Also, as a controller couldn't have any expectation regarding the 1962topologies of hierarchies other controllers might be on, each 1963controller had to assume that all other controllers were attached to 1964completely orthogonal hierarchies. This made it impossible, or at 1965least very cumbersome, for controllers to cooperate with each other. 1966 1967In most use cases, putting controllers on hierarchies which are 1968completely orthogonal to each other isn't necessary. What usually is 1969called for is the ability to have differing levels of granularity 1970depending on the specific controller. In other words, hierarchy may 1971be collapsed from leaf towards root when viewed from specific 1972controllers. For example, a given configuration might not care about 1973how memory is distributed beyond a certain level while still wanting 1974to control how CPU cycles are distributed. 1975 1976 1977Thread Granularity 1978------------------ 1979 1980cgroup v1 allowed threads of a process to belong to different cgroups. 1981This didn't make sense for some controllers and those controllers 1982ended up implementing different ways to ignore such situations but 1983much more importantly it blurred the line between API exposed to 1984individual applications and system management interface. 1985 1986Generally, in-process knowledge is available only to the process 1987itself; thus, unlike service-level organization of processes, 1988categorizing threads of a process requires active participation from 1989the application which owns the target process. 1990 1991cgroup v1 had an ambiguously defined delegation model which got abused 1992in combination with thread granularity. cgroups were delegated to 1993individual applications so that they can create and manage their own 1994sub-hierarchies and control resource distributions along them. This 1995effectively raised cgroup to the status of a syscall-like API exposed 1996to lay programs. 1997 1998First of all, cgroup has a fundamentally inadequate interface to be 1999exposed this way. For a process to access its own knobs, it has to 2000extract the path on the target hierarchy from /proc/self/cgroup, 2001construct the path by appending the name of the knob to the path, open 2002and then read and/or write to it. This is not only extremely clunky 2003and unusual but also inherently racy. There is no conventional way to 2004define transaction across the required steps and nothing can guarantee 2005that the process would actually be operating on its own sub-hierarchy. 2006 2007cgroup controllers implemented a number of knobs which would never be 2008accepted as public APIs because they were just adding control knobs to 2009system-management pseudo filesystem. cgroup ended up with interface 2010knobs which were not properly abstracted or refined and directly 2011revealed kernel internal details. These knobs got exposed to 2012individual applications through the ill-defined delegation mechanism 2013effectively abusing cgroup as a shortcut to implementing public APIs 2014without going through the required scrutiny. 2015 2016This was painful for both userland and kernel. Userland ended up with 2017misbehaving and poorly abstracted interfaces and kernel exposing and 2018locked into constructs inadvertently. 2019 2020 2021Competition Between Inner Nodes and Threads 2022------------------------------------------- 2023 2024cgroup v1 allowed threads to be in any cgroups which created an 2025interesting problem where threads belonging to a parent cgroup and its 2026children cgroups competed for resources. This was nasty as two 2027different types of entities competed and there was no obvious way to 2028settle it. Different controllers did different things. 2029 2030The cpu controller considered threads and cgroups as equivalents and 2031mapped nice levels to cgroup weights. This worked for some cases but 2032fell flat when children wanted to be allocated specific ratios of CPU 2033cycles and the number of internal threads fluctuated - the ratios 2034constantly changed as the number of competing entities fluctuated. 2035There also were other issues. The mapping from nice level to weight 2036wasn't obvious or universal, and there were various other knobs which 2037simply weren't available for threads. 2038 2039The io controller implicitly created a hidden leaf node for each 2040cgroup to host the threads. The hidden leaf had its own copies of all 2041the knobs with ``leaf_`` prefixed. While this allowed equivalent 2042control over internal threads, it was with serious drawbacks. It 2043always added an extra layer of nesting which wouldn't be necessary 2044otherwise, made the interface messy and significantly complicated the 2045implementation. 2046 2047The memory controller didn't have a way to control what happened 2048between internal tasks and child cgroups and the behavior was not 2049clearly defined. There were attempts to add ad-hoc behaviors and 2050knobs to tailor the behavior to specific workloads which would have 2051led to problems extremely difficult to resolve in the long term. 2052 2053Multiple controllers struggled with internal tasks and came up with 2054different ways to deal with it; unfortunately, all the approaches were 2055severely flawed and, furthermore, the widely different behaviors 2056made cgroup as a whole highly inconsistent. 2057 2058This clearly is a problem which needs to be addressed from cgroup core 2059in a uniform way. 2060 2061 2062Other Interface Issues 2063---------------------- 2064 2065cgroup v1 grew without oversight and developed a large number of 2066idiosyncrasies and inconsistencies. One issue on the cgroup core side 2067was how an empty cgroup was notified - a userland helper binary was 2068forked and executed for each event. The event delivery wasn't 2069recursive or delegatable. The limitations of the mechanism also led 2070to in-kernel event delivery filtering mechanism further complicating 2071the interface. 2072 2073Controller interfaces were problematic too. An extreme example is 2074controllers completely ignoring hierarchical organization and treating 2075all cgroups as if they were all located directly under the root 2076cgroup. Some controllers exposed a large amount of inconsistent 2077implementation details to userland. 2078 2079There also was no consistency across controllers. When a new cgroup 2080was created, some controllers defaulted to not imposing extra 2081restrictions while others disallowed any resource usage until 2082explicitly configured. Configuration knobs for the same type of 2083control used widely differing naming schemes and formats. Statistics 2084and information knobs were named arbitrarily and used different 2085formats and units even in the same controller. 2086 2087cgroup v2 establishes common conventions where appropriate and updates 2088controllers so that they expose minimal and consistent interfaces. 2089 2090 2091Controller Issues and Remedies 2092------------------------------ 2093 2094Memory 2095~~~~~~ 2096 2097The original lower boundary, the soft limit, is defined as a limit 2098that is per default unset. As a result, the set of cgroups that 2099global reclaim prefers is opt-in, rather than opt-out. The costs for 2100optimizing these mostly negative lookups are so high that the 2101implementation, despite its enormous size, does not even provide the 2102basic desirable behavior. First off, the soft limit has no 2103hierarchical meaning. All configured groups are organized in a global 2104rbtree and treated like equal peers, regardless where they are located 2105in the hierarchy. This makes subtree delegation impossible. Second, 2106the soft limit reclaim pass is so aggressive that it not just 2107introduces high allocation latencies into the system, but also impacts 2108system performance due to overreclaim, to the point where the feature 2109becomes self-defeating. 2110 2111The memory.low boundary on the other hand is a top-down allocated 2112reserve. A cgroup enjoys reclaim protection when it's within its low, 2113which makes delegation of subtrees possible. 2114 2115The original high boundary, the hard limit, is defined as a strict 2116limit that can not budge, even if the OOM killer has to be called. 2117But this generally goes against the goal of making the most out of the 2118available memory. The memory consumption of workloads varies during 2119runtime, and that requires users to overcommit. But doing that with a 2120strict upper limit requires either a fairly accurate prediction of the 2121working set size or adding slack to the limit. Since working set size 2122estimation is hard and error prone, and getting it wrong results in 2123OOM kills, most users tend to err on the side of a looser limit and 2124end up wasting precious resources. 2125 2126The memory.high boundary on the other hand can be set much more 2127conservatively. When hit, it throttles allocations by forcing them 2128into direct reclaim to work off the excess, but it never invokes the 2129OOM killer. As a result, a high boundary that is chosen too 2130aggressively will not terminate the processes, but instead it will 2131lead to gradual performance degradation. The user can monitor this 2132and make corrections until the minimal memory footprint that still 2133gives acceptable performance is found. 2134 2135In extreme cases, with many concurrent allocations and a complete 2136breakdown of reclaim progress within the group, the high boundary can 2137be exceeded. But even then it's mostly better to satisfy the 2138allocation from the slack available in other groups or the rest of the 2139system than killing the group. Otherwise, memory.max is there to 2140limit this type of spillover and ultimately contain buggy or even 2141malicious applications. 2142 2143Setting the original memory.limit_in_bytes below the current usage was 2144subject to a race condition, where concurrent charges could cause the 2145limit setting to fail. memory.max on the other hand will first set the 2146limit to prevent new charges, and then reclaim and OOM kill until the 2147new limit is met - or the task writing to memory.max is killed. 2148 2149The combined memory+swap accounting and limiting is replaced by real 2150control over swap space. 2151 2152The main argument for a combined memory+swap facility in the original 2153cgroup design was that global or parental pressure would always be 2154able to swap all anonymous memory of a child group, regardless of the 2155child's own (possibly untrusted) configuration. However, untrusted 2156groups can sabotage swapping by other means - such as referencing its 2157anonymous memory in a tight loop - and an admin can not assume full 2158swappability when overcommitting untrusted jobs. 2159 2160For trusted jobs, on the other hand, a combined counter is not an 2161intuitive userspace interface, and it flies in the face of the idea 2162that cgroup controllers should account and limit specific physical 2163resources. Swap space is a resource like all others in the system, 2164and that's why unified hierarchy allows distributing it separately. 2165