1=================================== 2Light-weight System Calls for IA-64 3=================================== 4 5 Started: 13-Jan-2003 6 7 Last update: 27-Sep-2003 8 9 David Mosberger-Tang 10 <davidm@hpl.hp.com> 11 12Using the "epc" instruction effectively introduces a new mode of 13execution to the ia64 linux kernel. We call this mode the 14"fsys-mode". To recap, the normal states of execution are: 15 16 - kernel mode: 17 Both the register stack and the memory stack have been 18 switched over to kernel memory. The user-level state is saved 19 in a pt-regs structure at the top of the kernel memory stack. 20 21 - user mode: 22 Both the register stack and the kernel stack are in 23 user memory. The user-level state is contained in the 24 CPU registers. 25 26 - bank 0 interruption-handling mode: 27 This is the non-interruptible state which all 28 interruption-handlers start execution in. The user-level 29 state remains in the CPU registers and some kernel state may 30 be stored in bank 0 of registers r16-r31. 31 32In contrast, fsys-mode has the following special properties: 33 34 - execution is at privilege level 0 (most-privileged) 35 36 - CPU registers may contain a mixture of user-level and kernel-level 37 state (it is the responsibility of the kernel to ensure that no 38 security-sensitive kernel-level state is leaked back to 39 user-level) 40 41 - execution is interruptible and preemptible (an fsys-mode handler 42 can disable interrupts and avoid all other interruption-sources 43 to avoid preemption) 44 45 - neither the memory-stack nor the register-stack can be trusted while 46 in fsys-mode (they point to the user-level stacks, which may 47 be invalid, or completely bogus addresses) 48 49In summary, fsys-mode is much more similar to running in user-mode 50than it is to running in kernel-mode. Of course, given that the 51privilege level is at level 0, this means that fsys-mode requires some 52care (see below). 53 54 55How to tell fsys-mode 56===================== 57 58Linux operates in fsys-mode when (a) the privilege level is 0 (most 59privileged) and (b) the stacks have NOT been switched to kernel memory 60yet. For convenience, the header file <asm-ia64/ptrace.h> provides 61three macros:: 62 63 user_mode(regs) 64 user_stack(task,regs) 65 fsys_mode(task,regs) 66 67The "regs" argument is a pointer to a pt_regs structure. The "task" 68argument is a pointer to the task structure to which the "regs" 69pointer belongs to. user_mode() returns TRUE if the CPU state pointed 70to by "regs" was executing in user mode (privilege level 3). 71user_stack() returns TRUE if the state pointed to by "regs" was 72executing on the user-level stack(s). Finally, fsys_mode() returns 73TRUE if the CPU state pointed to by "regs" was executing in fsys-mode. 74The fsys_mode() macro is equivalent to the expression:: 75 76 !user_mode(regs) && user_stack(task,regs) 77 78How to write an fsyscall handler 79================================ 80 81The file arch/ia64/kernel/fsys.S contains a table of fsyscall-handlers 82(fsyscall_table). This table contains one entry for each system call. 83By default, a system call is handled by fsys_fallback_syscall(). This 84routine takes care of entering (full) kernel mode and calling the 85normal Linux system call handler. For performance-critical system 86calls, it is possible to write a hand-tuned fsyscall_handler. For 87example, fsys.S contains fsys_getpid(), which is a hand-tuned version 88of the getpid() system call. 89 90The entry and exit-state of an fsyscall handler is as follows: 91 92Machine state on entry to fsyscall handler 93------------------------------------------ 94 95 ========= =============================================================== 96 r10 0 97 r11 saved ar.pfs (a user-level value) 98 r15 system call number 99 r16 "current" task pointer (in normal kernel-mode, this is in r13) 100 r32-r39 system call arguments 101 b6 return address (a user-level value) 102 ar.pfs previous frame-state (a user-level value) 103 PSR.be cleared to zero (i.e., little-endian byte order is in effect) 104 - all other registers may contain values passed in from user-mode 105 ========= =============================================================== 106 107Required machine state on exit to fsyscall handler 108-------------------------------------------------- 109 110 ========= =========================================================== 111 r11 saved ar.pfs (as passed into the fsyscall handler) 112 r15 system call number (as passed into the fsyscall handler) 113 r32-r39 system call arguments (as passed into the fsyscall handler) 114 b6 return address (as passed into the fsyscall handler) 115 ar.pfs previous frame-state (as passed into the fsyscall handler) 116 ========= =========================================================== 117 118Fsyscall handlers can execute with very little overhead, but with that 119speed comes a set of restrictions: 120 121 * Fsyscall-handlers MUST check for any pending work in the flags 122 member of the thread-info structure and if any of the 123 TIF_ALLWORK_MASK flags are set, the handler needs to fall back on 124 doing a full system call (by calling fsys_fallback_syscall). 125 126 * Fsyscall-handlers MUST preserve incoming arguments (r32-r39, r11, 127 r15, b6, and ar.pfs) because they will be needed in case of a 128 system call restart. Of course, all "preserved" registers also 129 must be preserved, in accordance to the normal calling conventions. 130 131 * Fsyscall-handlers MUST check argument registers for containing a 132 NaT value before using them in any way that could trigger a 133 NaT-consumption fault. If a system call argument is found to 134 contain a NaT value, an fsyscall-handler may return immediately 135 with r8=EINVAL, r10=-1. 136 137 * Fsyscall-handlers MUST NOT use the "alloc" instruction or perform 138 any other operation that would trigger mandatory RSE 139 (register-stack engine) traffic. 140 141 * Fsyscall-handlers MUST NOT write to any stacked registers because 142 it is not safe to assume that user-level called a handler with the 143 proper number of arguments. 144 145 * Fsyscall-handlers need to be careful when accessing per-CPU variables: 146 unless proper safe-guards are taken (e.g., interruptions are avoided), 147 execution may be pre-empted and resumed on another CPU at any given 148 time. 149 150 * Fsyscall-handlers must be careful not to leak sensitive kernel' 151 information back to user-level. In particular, before returning to 152 user-level, care needs to be taken to clear any scratch registers 153 that could contain sensitive information (note that the current 154 task pointer is not considered sensitive: it's already exposed 155 through ar.k6). 156 157 * Fsyscall-handlers MUST NOT access user-memory without first 158 validating access-permission (this can be done typically via 159 probe.r.fault and/or probe.w.fault) and without guarding against 160 memory access exceptions (this can be done with the EX() macros 161 defined by asmmacro.h). 162 163The above restrictions may seem draconian, but remember that it's 164possible to trade off some of the restrictions by paying a slightly 165higher overhead. For example, if an fsyscall-handler could benefit 166from the shadow register bank, it could temporarily disable PSR.i and 167PSR.ic, switch to bank 0 (bsw.0) and then use the shadow registers as 168needed. In other words, following the above rules yields extremely 169fast system call execution (while fully preserving system call 170semantics), but there is also a lot of flexibility in handling more 171complicated cases. 172 173Signal handling 174=============== 175 176The delivery of (asynchronous) signals must be delayed until fsys-mode 177is exited. This is accomplished with the help of the lower-privilege 178transfer trap: arch/ia64/kernel/process.c:do_notify_resume_user() 179checks whether the interrupted task was in fsys-mode and, if so, sets 180PSR.lp and returns immediately. When fsys-mode is exited via the 181"br.ret" instruction that lowers the privilege level, a trap will 182occur. The trap handler clears PSR.lp again and returns immediately. 183The kernel exit path then checks for and delivers any pending signals. 184 185PSR Handling 186============ 187 188The "epc" instruction doesn't change the contents of PSR at all. This 189is in contrast to a regular interruption, which clears almost all 190bits. Because of that, some care needs to be taken to ensure things 191work as expected. The following discussion describes how each PSR bit 192is handled. 193 194======= ======================================================================= 195PSR.be Cleared when entering fsys-mode. A srlz.d instruction is used 196 to ensure the CPU is in little-endian mode before the first 197 load/store instruction is executed. PSR.be is normally NOT 198 restored upon return from an fsys-mode handler. In other 199 words, user-level code must not rely on PSR.be being preserved 200 across a system call. 201PSR.up Unchanged. 202PSR.ac Unchanged. 203PSR.mfl Unchanged. Note: fsys-mode handlers must not write-registers! 204PSR.mfh Unchanged. Note: fsys-mode handlers must not write-registers! 205PSR.ic Unchanged. Note: fsys-mode handlers can clear the bit, if needed. 206PSR.i Unchanged. Note: fsys-mode handlers can clear the bit, if needed. 207PSR.pk Unchanged. 208PSR.dt Unchanged. 209PSR.dfl Unchanged. Note: fsys-mode handlers must not write-registers! 210PSR.dfh Unchanged. Note: fsys-mode handlers must not write-registers! 211PSR.sp Unchanged. 212PSR.pp Unchanged. 213PSR.di Unchanged. 214PSR.si Unchanged. 215PSR.db Unchanged. The kernel prevents user-level from setting a hardware 216 breakpoint that triggers at any privilege level other than 217 3 (user-mode). 218PSR.lp Unchanged. 219PSR.tb Lazy redirect. If a taken-branch trap occurs while in 220 fsys-mode, the trap-handler modifies the saved machine state 221 such that execution resumes in the gate page at 222 syscall_via_break(), with privilege level 3. Note: the 223 taken branch would occur on the branch invoking the 224 fsyscall-handler, at which point, by definition, a syscall 225 restart is still safe. If the system call number is invalid, 226 the fsys-mode handler will return directly to user-level. This 227 return will trigger a taken-branch trap, but since the trap is 228 taken _after_ restoring the privilege level, the CPU has already 229 left fsys-mode, so no special treatment is needed. 230PSR.rt Unchanged. 231PSR.cpl Cleared to 0. 232PSR.is Unchanged (guaranteed to be 0 on entry to the gate page). 233PSR.mc Unchanged. 234PSR.it Unchanged (guaranteed to be 1). 235PSR.id Unchanged. Note: the ia64 linux kernel never sets this bit. 236PSR.da Unchanged. Note: the ia64 linux kernel never sets this bit. 237PSR.dd Unchanged. Note: the ia64 linux kernel never sets this bit. 238PSR.ss Lazy redirect. If set, "epc" will cause a Single Step Trap to 239 be taken. The trap handler then modifies the saved machine 240 state such that execution resumes in the gate page at 241 syscall_via_break(), with privilege level 3. 242PSR.ri Unchanged. 243PSR.ed Unchanged. Note: This bit could only have an effect if an fsys-mode 244 handler performed a speculative load that gets NaTted. If so, this 245 would be the normal & expected behavior, so no special treatment is 246 needed. 247PSR.bn Unchanged. Note: fsys-mode handlers may clear the bit, if needed. 248 Doing so requires clearing PSR.i and PSR.ic as well. 249PSR.ia Unchanged. Note: the ia64 linux kernel never sets this bit. 250======= ======================================================================= 251 252Using fast system calls 253======================= 254 255To use fast system calls, userspace applications need simply call 256__kernel_syscall_via_epc(). For example 257 258-- example fgettimeofday() call -- 259 260-- fgettimeofday.S -- 261 262:: 263 264 #include <asm/asmmacro.h> 265 266 GLOBAL_ENTRY(fgettimeofday) 267 .prologue 268 .save ar.pfs, r11 269 mov r11 = ar.pfs 270 .body 271 272 mov r2 = 0xa000000000020660;; // gate address 273 // found by inspection of System.map for the 274 // __kernel_syscall_via_epc() function. See 275 // below for how to do this for real. 276 277 mov b7 = r2 278 mov r15 = 1087 // gettimeofday syscall 279 ;; 280 br.call.sptk.many b6 = b7 281 ;; 282 283 .restore sp 284 285 mov ar.pfs = r11 286 br.ret.sptk.many rp;; // return to caller 287 END(fgettimeofday) 288 289-- end fgettimeofday.S -- 290 291In reality, getting the gate address is accomplished by two extra 292values passed via the ELF auxiliary vector (include/asm-ia64/elf.h) 293 294 * AT_SYSINFO : is the address of __kernel_syscall_via_epc() 295 * AT_SYSINFO_EHDR : is the address of the kernel gate ELF DSO 296 297The ELF DSO is a pre-linked library that is mapped in by the kernel at 298the gate page. It is a proper ELF shared object so, with a dynamic 299loader that recognises the library, you should be able to make calls to 300the exported functions within it as with any other shared library. 301AT_SYSINFO points into the kernel DSO at the 302__kernel_syscall_via_epc() function for historical reasons (it was 303used before the kernel DSO) and as a convenience. 304