1d2912cb1SThomas Gleixner/* SPDX-License-Identifier: GPL-2.0-only */ 275494230SNicolas Pitre/* 375494230SNicolas Pitre * linux/arch/arm/lib/copy_template.s 475494230SNicolas Pitre * 575494230SNicolas Pitre * Code template for optimized memory copy functions 675494230SNicolas Pitre * 775494230SNicolas Pitre * Author: Nicolas Pitre 875494230SNicolas Pitre * Created: Sep 28, 2005 975494230SNicolas Pitre * Copyright: MontaVista Software, Inc. 1075494230SNicolas Pitre */ 1175494230SNicolas Pitre 1275494230SNicolas Pitre/* 1375494230SNicolas Pitre * Theory of operation 1475494230SNicolas Pitre * ------------------- 1575494230SNicolas Pitre * 1675494230SNicolas Pitre * This file provides the core code for a forward memory copy used in 1775494230SNicolas Pitre * the implementation of memcopy(), copy_to_user() and copy_from_user(). 1875494230SNicolas Pitre * 1975494230SNicolas Pitre * The including file must define the following accessor macros 2075494230SNicolas Pitre * according to the need of the given function: 2175494230SNicolas Pitre * 2275494230SNicolas Pitre * ldr1w ptr reg abort 2375494230SNicolas Pitre * 2475494230SNicolas Pitre * This loads one word from 'ptr', stores it in 'reg' and increments 2575494230SNicolas Pitre * 'ptr' to the next word. The 'abort' argument is used for fixup tables. 2675494230SNicolas Pitre * 2775494230SNicolas Pitre * ldr4w ptr reg1 reg2 reg3 reg4 abort 2875494230SNicolas Pitre * ldr8w ptr, reg1 reg2 reg3 reg4 reg5 reg6 reg7 reg8 abort 2975494230SNicolas Pitre * 3075494230SNicolas Pitre * This loads four or eight words starting from 'ptr', stores them 3175494230SNicolas Pitre * in provided registers and increments 'ptr' past those words. 3275494230SNicolas Pitre * The'abort' argument is used for fixup tables. 3375494230SNicolas Pitre * 3475494230SNicolas Pitre * ldr1b ptr reg cond abort 3575494230SNicolas Pitre * 3675494230SNicolas Pitre * Similar to ldr1w, but it loads a byte and increments 'ptr' one byte. 3775494230SNicolas Pitre * It also must apply the condition code if provided, otherwise the 3875494230SNicolas Pitre * "al" condition is assumed by default. 3975494230SNicolas Pitre * 4075494230SNicolas Pitre * str1w ptr reg abort 4175494230SNicolas Pitre * str8w ptr reg1 reg2 reg3 reg4 reg5 reg6 reg7 reg8 abort 4275494230SNicolas Pitre * str1b ptr reg cond abort 4375494230SNicolas Pitre * 4475494230SNicolas Pitre * Same as their ldr* counterparts, but data is stored to 'ptr' location 4575494230SNicolas Pitre * rather than being loaded. 4675494230SNicolas Pitre * 4775494230SNicolas Pitre * enter reg1 reg2 4875494230SNicolas Pitre * 4975494230SNicolas Pitre * Preserve the provided registers on the stack plus any additional 5075494230SNicolas Pitre * data as needed by the implementation including this code. Called 5175494230SNicolas Pitre * upon code entry. 5275494230SNicolas Pitre * 53279f487eSLin Yongting * usave reg1 reg2 54279f487eSLin Yongting * 55279f487eSLin Yongting * Unwind annotation macro is corresponding for 'enter' macro. 56279f487eSLin Yongting * It tell unwinder that preserved some provided registers on the stack 57279f487eSLin Yongting * and additional data by a prior 'enter' macro. 58279f487eSLin Yongting * 5975494230SNicolas Pitre * exit reg1 reg2 6075494230SNicolas Pitre * 6175494230SNicolas Pitre * Restore registers with the values previously saved with the 6275494230SNicolas Pitre * 'preserv' macro. Called upon code termination. 638b592783SCatalin Marinas * 648b592783SCatalin Marinas * LDR1W_SHIFT 658b592783SCatalin Marinas * STR1W_SHIFT 668b592783SCatalin Marinas * 678b592783SCatalin Marinas * Correction to be applied to the "ip" register when branching into 688b592783SCatalin Marinas * the ldr1w or str1w instructions (some of these macros may expand to 698b592783SCatalin Marinas * than one 32bit instruction in Thumb-2) 7075494230SNicolas Pitre */ 7175494230SNicolas Pitre 72279f487eSLin Yongting UNWIND( .fnstart ) 73*ba999a04SArd Biesheuvel enter r4, UNWIND(fpreg,) lr 74*ba999a04SArd Biesheuvel UNWIND( .setfp fpreg, sp ) 75*ba999a04SArd Biesheuvel UNWIND( mov fpreg, sp ) 7675494230SNicolas Pitre 7775494230SNicolas Pitre subs r2, r2, #4 7875494230SNicolas Pitre blt 8f 7975494230SNicolas Pitre ands ip, r0, #3 8075494230SNicolas Pitre PLD( pld [r1, #0] ) 8175494230SNicolas Pitre bne 9f 8275494230SNicolas Pitre ands ip, r1, #3 8375494230SNicolas Pitre bne 10f 8475494230SNicolas Pitre 8575494230SNicolas Pitre1: subs r2, r2, #(28) 86*ba999a04SArd Biesheuvel stmfd sp!, {r5, r6, r8, r9} 8775494230SNicolas Pitre blt 5f 8875494230SNicolas Pitre 892239aff6SNicolas Pitre CALGN( ands ip, r0, #31 ) 9075494230SNicolas Pitre CALGN( rsb r3, ip, #32 ) 91e44fc388SStefan Agner CALGN( sbcsne r4, r3, r2 ) @ C is always set here 9275494230SNicolas Pitre CALGN( bcs 2f ) 9375494230SNicolas Pitre CALGN( adr r4, 6f ) 9475494230SNicolas Pitre CALGN( subs r2, r2, r3 ) @ C gets set 9575494230SNicolas Pitre CALGN( add pc, r4, ip ) 9675494230SNicolas Pitre 9775494230SNicolas Pitre PLD( pld [r1, #0] ) 9875494230SNicolas Pitre2: PLD( subs r2, r2, #96 ) 9975494230SNicolas Pitre PLD( pld [r1, #28] ) 10075494230SNicolas Pitre PLD( blt 4f ) 10175494230SNicolas Pitre PLD( pld [r1, #60] ) 10275494230SNicolas Pitre PLD( pld [r1, #92] ) 10375494230SNicolas Pitre 10475494230SNicolas Pitre3: PLD( pld [r1, #124] ) 105*ba999a04SArd Biesheuvel4: ldr8w r1, r3, r4, r5, r6, r8, r9, ip, lr, abort=20f 10675494230SNicolas Pitre subs r2, r2, #32 107*ba999a04SArd Biesheuvel str8w r0, r3, r4, r5, r6, r8, r9, ip, lr, abort=20f 10875494230SNicolas Pitre bge 3b 10975494230SNicolas Pitre PLD( cmn r2, #96 ) 11075494230SNicolas Pitre PLD( bge 4b ) 11175494230SNicolas Pitre 11275494230SNicolas Pitre5: ands ip, r2, #28 11375494230SNicolas Pitre rsb ip, ip, #32 1148b592783SCatalin Marinas#if LDR1W_SHIFT > 0 1158b592783SCatalin Marinas lsl ip, ip, #LDR1W_SHIFT 1168b592783SCatalin Marinas#endif 11775494230SNicolas Pitre addne pc, pc, ip @ C is always clear here 11875494230SNicolas Pitre b 7f 1198b592783SCatalin Marinas6: 1208b592783SCatalin Marinas .rept (1 << LDR1W_SHIFT) 1218b592783SCatalin Marinas W(nop) 1228b592783SCatalin Marinas .endr 12375494230SNicolas Pitre ldr1w r1, r3, abort=20f 12475494230SNicolas Pitre ldr1w r1, r4, abort=20f 12575494230SNicolas Pitre ldr1w r1, r5, abort=20f 12675494230SNicolas Pitre ldr1w r1, r6, abort=20f 12775494230SNicolas Pitre ldr1w r1, r8, abort=20f 128*ba999a04SArd Biesheuvel ldr1w r1, r9, abort=20f 12975494230SNicolas Pitre ldr1w r1, lr, abort=20f 13075494230SNicolas Pitre 1318b592783SCatalin Marinas#if LDR1W_SHIFT < STR1W_SHIFT 1328b592783SCatalin Marinas lsl ip, ip, #STR1W_SHIFT - LDR1W_SHIFT 1338b592783SCatalin Marinas#elif LDR1W_SHIFT > STR1W_SHIFT 1348b592783SCatalin Marinas lsr ip, ip, #LDR1W_SHIFT - STR1W_SHIFT 1358b592783SCatalin Marinas#endif 13675494230SNicolas Pitre add pc, pc, ip 13775494230SNicolas Pitre nop 1388b592783SCatalin Marinas .rept (1 << STR1W_SHIFT) 1398b592783SCatalin Marinas W(nop) 1408b592783SCatalin Marinas .endr 14175494230SNicolas Pitre str1w r0, r3, abort=20f 14275494230SNicolas Pitre str1w r0, r4, abort=20f 14375494230SNicolas Pitre str1w r0, r5, abort=20f 14475494230SNicolas Pitre str1w r0, r6, abort=20f 14575494230SNicolas Pitre str1w r0, r8, abort=20f 146*ba999a04SArd Biesheuvel str1w r0, r9, abort=20f 14775494230SNicolas Pitre str1w r0, lr, abort=20f 14875494230SNicolas Pitre 14975494230SNicolas Pitre CALGN( bcs 2b ) 15075494230SNicolas Pitre 151*ba999a04SArd Biesheuvel7: ldmfd sp!, {r5, r6, r8, r9} 15275494230SNicolas Pitre 15375494230SNicolas Pitre8: movs r2, r2, lsl #31 15475494230SNicolas Pitre ldr1b r1, r3, ne, abort=21f 15575494230SNicolas Pitre ldr1b r1, r4, cs, abort=21f 15675494230SNicolas Pitre ldr1b r1, ip, cs, abort=21f 15775494230SNicolas Pitre str1b r0, r3, ne, abort=21f 15875494230SNicolas Pitre str1b r0, r4, cs, abort=21f 15975494230SNicolas Pitre str1b r0, ip, cs, abort=21f 16075494230SNicolas Pitre 161*ba999a04SArd Biesheuvel exit r4, UNWIND(fpreg,) pc 16275494230SNicolas Pitre 16375494230SNicolas Pitre9: rsb ip, ip, #4 16475494230SNicolas Pitre cmp ip, #2 16575494230SNicolas Pitre ldr1b r1, r3, gt, abort=21f 16675494230SNicolas Pitre ldr1b r1, r4, ge, abort=21f 16775494230SNicolas Pitre ldr1b r1, lr, abort=21f 16875494230SNicolas Pitre str1b r0, r3, gt, abort=21f 16975494230SNicolas Pitre str1b r0, r4, ge, abort=21f 17075494230SNicolas Pitre subs r2, r2, ip 17175494230SNicolas Pitre str1b r0, lr, abort=21f 17275494230SNicolas Pitre blt 8b 17375494230SNicolas Pitre ands ip, r1, #3 17475494230SNicolas Pitre beq 1b 17575494230SNicolas Pitre 17675494230SNicolas Pitre10: bic r1, r1, #3 17775494230SNicolas Pitre cmp ip, #2 17875494230SNicolas Pitre ldr1w r1, lr, abort=21f 17975494230SNicolas Pitre beq 17f 18075494230SNicolas Pitre bgt 18f 18175494230SNicolas Pitre 18275494230SNicolas Pitre 18375494230SNicolas Pitre .macro forward_copy_shift pull push 18475494230SNicolas Pitre 18575494230SNicolas Pitre subs r2, r2, #28 18675494230SNicolas Pitre blt 14f 18775494230SNicolas Pitre 1882239aff6SNicolas Pitre CALGN( ands ip, r0, #31 ) 18975494230SNicolas Pitre CALGN( rsb ip, ip, #32 ) 190e44fc388SStefan Agner CALGN( sbcsne r4, ip, r2 ) @ C is always set here 19175494230SNicolas Pitre CALGN( subcc r2, r2, ip ) 19275494230SNicolas Pitre CALGN( bcc 15f ) 19375494230SNicolas Pitre 194*ba999a04SArd Biesheuvel11: stmfd sp!, {r5, r6, r8 - r10} 19575494230SNicolas Pitre 19675494230SNicolas Pitre PLD( pld [r1, #0] ) 19775494230SNicolas Pitre PLD( subs r2, r2, #96 ) 19875494230SNicolas Pitre PLD( pld [r1, #28] ) 19975494230SNicolas Pitre PLD( blt 13f ) 20075494230SNicolas Pitre PLD( pld [r1, #60] ) 20175494230SNicolas Pitre PLD( pld [r1, #92] ) 20275494230SNicolas Pitre 20375494230SNicolas Pitre12: PLD( pld [r1, #124] ) 204*ba999a04SArd Biesheuvel13: ldr4w r1, r4, r5, r6, r8, abort=19f 205d98b90eaSVictor Kamensky mov r3, lr, lspull #\pull 20675494230SNicolas Pitre subs r2, r2, #32 207*ba999a04SArd Biesheuvel ldr4w r1, r9, r10, ip, lr, abort=19f 208d98b90eaSVictor Kamensky orr r3, r3, r4, lspush #\push 209d98b90eaSVictor Kamensky mov r4, r4, lspull #\pull 210d98b90eaSVictor Kamensky orr r4, r4, r5, lspush #\push 211d98b90eaSVictor Kamensky mov r5, r5, lspull #\pull 212d98b90eaSVictor Kamensky orr r5, r5, r6, lspush #\push 213d98b90eaSVictor Kamensky mov r6, r6, lspull #\pull 214*ba999a04SArd Biesheuvel orr r6, r6, r8, lspush #\push 215d98b90eaSVictor Kamensky mov r8, r8, lspull #\pull 216d98b90eaSVictor Kamensky orr r8, r8, r9, lspush #\push 217d98b90eaSVictor Kamensky mov r9, r9, lspull #\pull 218*ba999a04SArd Biesheuvel orr r9, r9, r10, lspush #\push 219*ba999a04SArd Biesheuvel mov r10, r10, lspull #\pull 220*ba999a04SArd Biesheuvel orr r10, r10, ip, lspush #\push 221d98b90eaSVictor Kamensky mov ip, ip, lspull #\pull 222d98b90eaSVictor Kamensky orr ip, ip, lr, lspush #\push 223*ba999a04SArd Biesheuvel str8w r0, r3, r4, r5, r6, r8, r9, r10, ip, abort=19f 22475494230SNicolas Pitre bge 12b 22575494230SNicolas Pitre PLD( cmn r2, #96 ) 22675494230SNicolas Pitre PLD( bge 13b ) 22775494230SNicolas Pitre 228*ba999a04SArd Biesheuvel ldmfd sp!, {r5, r6, r8 - r10} 22975494230SNicolas Pitre 23075494230SNicolas Pitre14: ands ip, r2, #28 23175494230SNicolas Pitre beq 16f 23275494230SNicolas Pitre 233d98b90eaSVictor Kamensky15: mov r3, lr, lspull #\pull 23475494230SNicolas Pitre ldr1w r1, lr, abort=21f 23575494230SNicolas Pitre subs ip, ip, #4 236d98b90eaSVictor Kamensky orr r3, r3, lr, lspush #\push 23775494230SNicolas Pitre str1w r0, r3, abort=21f 23875494230SNicolas Pitre bgt 15b 23975494230SNicolas Pitre CALGN( cmp r2, #0 ) 24075494230SNicolas Pitre CALGN( bge 11b ) 24175494230SNicolas Pitre 24275494230SNicolas Pitre16: sub r1, r1, #(\push / 8) 24375494230SNicolas Pitre b 8b 24475494230SNicolas Pitre 24575494230SNicolas Pitre .endm 24675494230SNicolas Pitre 24775494230SNicolas Pitre 24875494230SNicolas Pitre forward_copy_shift pull=8 push=24 24975494230SNicolas Pitre 25075494230SNicolas Pitre17: forward_copy_shift pull=16 push=16 25175494230SNicolas Pitre 25275494230SNicolas Pitre18: forward_copy_shift pull=24 push=8 25375494230SNicolas Pitre 254*ba999a04SArd Biesheuvel UNWIND( .fnend ) 25575494230SNicolas Pitre 25675494230SNicolas Pitre/* 2577f927fccSAlexey Dobriyan * Abort preamble and completion macros. 25875494230SNicolas Pitre * If a fixup handler is required then those macros must surround it. 25975494230SNicolas Pitre * It is assumed that the fixup code will handle the private part of 26075494230SNicolas Pitre * the exit macro. 26175494230SNicolas Pitre */ 26275494230SNicolas Pitre 26375494230SNicolas Pitre .macro copy_abort_preamble 264*ba999a04SArd Biesheuvel19: ldmfd sp!, {r5, r6, r8 - r10} 26575494230SNicolas Pitre b 21f 266*ba999a04SArd Biesheuvel20: ldmfd sp!, {r5, r6, r8, r9} 26775494230SNicolas Pitre21: 26875494230SNicolas Pitre .endm 26975494230SNicolas Pitre 27075494230SNicolas Pitre .macro copy_abort_end 271*ba999a04SArd Biesheuvel ldmfd sp!, {r4, UNWIND(fpreg,) pc} 27275494230SNicolas Pitre .endm 27375494230SNicolas Pitre 274