808094fc | 31-Mar-2021 |
Christophe Leroy <christophe.leroy@csgroup.eu> |
lib/vdso: Add vdso_data pointer as input to __arch_get_timens_vdso_data()
For the same reason as commit e876f0b69dc9 ("lib/vdso: Allow architectures to provide the vdso data pointer"), powerpc wants
lib/vdso: Add vdso_data pointer as input to __arch_get_timens_vdso_data()
For the same reason as commit e876f0b69dc9 ("lib/vdso: Allow architectures to provide the vdso data pointer"), powerpc wants to avoid calculation of relative position to code.
As the timens_vdso_data is next page to vdso_data, provide vdso_data pointer to __arch_get_timens_vdso_data() in order to ease the calculation on powerpc in following patches.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Reviewed-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Acked-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/539c4204b1baa77c55f758904a1ea239abbc7a5c.1617209142.git.christophe.leroy@csgroup.eu
show more ...
|
e876f0b6 | 07-Feb-2020 |
Christophe Leroy <christophe.leroy@c-s.fr> |
lib/vdso: Allow architectures to provide the vdso data pointer
On powerpc, __arch_get_vdso_data() clobbers the link register, requiring the caller to save it.
As the parent function already has to
lib/vdso: Allow architectures to provide the vdso data pointer
On powerpc, __arch_get_vdso_data() clobbers the link register, requiring the caller to save it.
As the parent function already has to set a stack frame and saves the link register before calling the C vdso function, retrieving the vdso data pointer there is less overhead.
Split out the functional code from the __cvdso.*() interfaces into new static functions which can either be called from the existing interfaces with the vdso data pointer supplied via __arch_get_vdso_data() or directly from ASM code.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Link: https://lore.kernel.org/r/abf97996602ef07223fec30c005df78e5ed41b2e.1580399657.git.christophe.leroy@c-s.fr Link: https://lkml.kernel.org/r/20200207124403.965789141@linutronix.de
show more ...
|
8345228c | 07-Feb-2020 |
Christophe Leroy <christophe.leroy@c-s.fr> |
lib/vdso: Allow architectures to override the ns shift operation
On powerpc/32, GCC (8.1) generates pretty bad code for the ns >>= vd->shift operation taking into account that the shift is always <=
lib/vdso: Allow architectures to override the ns shift operation
On powerpc/32, GCC (8.1) generates pretty bad code for the ns >>= vd->shift operation taking into account that the shift is always <= 32 and the upper part of the result is likely to be zero. GCC makes reversed assumptions considering the shift to be likely >= 32 and the upper part to be like not zero.
unsigned long long shift(unsigned long long x, unsigned char s) { return x >> s; }
results in:
00000018 <shift>: 18: 35 25 ff e0 addic. r9,r5,-32 1c: 41 80 00 10 blt 2c <shift+0x14> 20: 7c 64 4c 30 srw r4,r3,r9 24: 38 60 00 00 li r3,0 28: 4e 80 00 20 blr 2c: 54 69 08 3c rlwinm r9,r3,1,0,30 30: 21 45 00 1f subfic r10,r5,31 34: 7c 84 2c 30 srw r4,r4,r5 38: 7d 29 50 30 slw r9,r9,r10 3c: 7c 63 2c 30 srw r3,r3,r5 40: 7d 24 23 78 or r4,r9,r4 44: 4e 80 00 20 blr
Even when forcing the shift to be smaller than 32 with an &= 31, it still considers the shift as likely >= 32.
Move the default shift implementation into an inline which can be redefined in architecture code via a macro.
[ tglx: Made the shift argument u32 and removed the __arch prefix ]
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Link: https://lore.kernel.org/r/b3d449de856982ed060a71e6ace8eeca4654e685.1580399657.git.christophe.leroy@c-s.fr Link: https://lkml.kernel.org/r/20200207124403.857649978@linutronix.de
show more ...
|
ae12e085 | 07-Feb-2020 |
Christophe Leroy <christophe.leroy@c-s.fr> |
lib/vdso: Allow fixed clock mode
Some architectures have a fixed clocksource which is known at compile time and cannot be replaced or disabled at runtime, e.g. timebase on PowerPC. For such cases th
lib/vdso: Allow fixed clock mode
Some architectures have a fixed clocksource which is known at compile time and cannot be replaced or disabled at runtime, e.g. timebase on PowerPC. For such cases the clock mode check in the VDSO code is pointless.
Move the check for a VDSO capable clocksource into an inline function and allow architectures to redefine it via a macro.
[ tglx: Removed the #ifdef mess ]
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Tested-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Link: https://lkml.kernel.org/r/20200207124403.748756829@linutronix.de
show more ...
|
660fd04f | 11-Nov-2019 |
Thomas Gleixner <tglx@linutronix.de> |
lib/vdso: Prepare for time namespace support
To support time namespaces in the vdso with a minimal impact on regular non time namespace affected tasks, the namespace handling needs to be hidden in a
lib/vdso: Prepare for time namespace support
To support time namespaces in the vdso with a minimal impact on regular non time namespace affected tasks, the namespace handling needs to be hidden in a slow path.
The most obvious place is vdso_seq_begin(). If a task belongs to a time namespace then the VVAR page which contains the system wide vdso data is replaced with a namespace specific page which has the same layout as the VVAR page. That page has vdso_data->seq set to 1 to enforce the slow path and vdso_data->clock_mode set to VCLOCK_TIMENS to enforce the time namespace handling path.
The extra check in the case that vdso_data->seq is odd, e.g. a concurrent update of the vdso data is in progress, is not really affecting regular tasks which are not part of a time namespace as the task is spin waiting for the update to finish and vdso_data->seq to become even again.
If a time namespace task hits that code path, it invokes the corresponding time getter function which retrieves the real VVAR page, reads host time and then adds the offset for the requested clock which is stored in the special VVAR page.
If VDSO time namespace support is disabled the whole magic is compiled out.
Initial testing shows that the disabled case is almost identical to the host case which does not take the slow timens path. With the special timens page installed the performance hit is constant time and in the range of 5-7%.
For the vdso functions which are not using the sequence count an unconditional check for vdso_data->clock_mode is added which switches to the real vdso when the clock_mode is VCLOCK_TIMENS.
[avagin: Make do_hres_timens() work with raw clocks too: choose vdso_data pointer by CS_RAW offset.]
Suggested-by: Andy Lutomirski <luto@kernel.org> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/r/20191112012724.250792-21-dima@arista.com
show more ...
|
c966533f | 11-Nov-2019 |
Andrei Vagin <avagin@gmail.com> |
lib/vdso: Mark do_hres() and do_coarse() as __always_inline
Performance numbers for Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz (more clock_gettime() cycles - the better):
clock | before
lib/vdso: Mark do_hres() and do_coarse() as __always_inline
Performance numbers for Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz (more clock_gettime() cycles - the better):
clock | before | after | diff ---------------------------------------------------------- monotonic | 153222105 | 166775025 | 8.8% monotonic-coarse | 671557054 | 691513017 | 3.0% monotonic-raw | 147116067 | 161057395 | 9.5% boottime | 153446224 | 166962668 | 9.1%
The improvement for arm64 for monotonic and boottime is around 3.5%.
clock | before | after | diff ================================================== monotonic 17326692 17951770 3.6% monotonic-coarse 43624027 44215292 1.3% monotonic-raw 17541809 17554932 0.1% boottime 17334982 17954361 3.5%
[ tglx: Avoid the goto ]
Signed-off-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/r/20191112012724.250792-3-dima@arista.com
show more ...
|
cdb7c5a9 | 23-Dec-2019 |
Christophe Leroy <christophe.leroy@c-s.fr> |
lib/vdso: Avoid duplication in __cvdso_clock_getres()
VDSO_HRES and VDSO_RAW clocks are handled the same way.
Avoid the code duplication.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
lib/vdso: Avoid duplication in __cvdso_clock_getres()
VDSO_HRES and VDSO_RAW clocks are handled the same way.
Avoid the code duplication.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Andy Lutomirski <luto@kernel.org> Link: https://lore.kernel.org/r/fdf1a968a8f7edd61456f1689ac44082ebb19c15.1577111367.git.christophe.leroy@c-s.fr
show more ...
|
a279235d | 30-Aug-2019 |
Vincenzo Frascino <vincenzo.frascino@arm.com> |
lib/vdso: Remove checks on return value for 32 bit vDSO
Since all the architectures that support the generic vDSO library have been converted to support the 32 bit fallbacks it is not required anymo
lib/vdso: Remove checks on return value for 32 bit vDSO
Since all the architectures that support the generic vDSO library have been converted to support the 32 bit fallbacks it is not required anymore to check the return value of __cvdso_clock_get*time32_common() before updating the old_timespec fields.
Remove the related checks from the generic vdso library.
References: c60a32ea4f45 ("lib/vdso/32: Provide legacy syscall fallbacks") Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/r/20190830135902.20861-6-vincenzo.frascino@arm.com
show more ...
|
b767081c | 30-Aug-2019 |
Vincenzo Frascino <vincenzo.frascino@arm.com> |
lib/vdso: Remove VDSO_HAS_32BIT_FALLBACK
VDSO_HAS_32BIT_FALLBACK was introduced to address a regression which caused seccomp to deny access to the applications to clock_gettime64() and clock_getres6
lib/vdso: Remove VDSO_HAS_32BIT_FALLBACK
VDSO_HAS_32BIT_FALLBACK was introduced to address a regression which caused seccomp to deny access to the applications to clock_gettime64() and clock_getres64() because they are not enabled in the existing filters.
The purpose of VDSO_HAS_32BIT_FALLBACK was to simplify the conditional implementation of __cvdso_clock_get*time32() variants.
Now that all the architectures that support the generic vDSO library have been converted to support the 32 bit fallbacks the conditional can be removed.
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/r/20190830135902.20861-5-vincenzo.frascino@arm.com
References: c60a32ea4f45 ("lib/vdso/32: Provide legacy syscall fallbacks")
show more ...
|