175494230SNicolas Pitre/* 275494230SNicolas Pitre * linux/arch/arm/lib/copy_template.s 375494230SNicolas Pitre * 475494230SNicolas Pitre * Code template for optimized memory copy functions 575494230SNicolas Pitre * 675494230SNicolas Pitre * Author: Nicolas Pitre 775494230SNicolas Pitre * Created: Sep 28, 2005 875494230SNicolas Pitre * Copyright: MontaVista Software, Inc. 975494230SNicolas Pitre * 1075494230SNicolas Pitre * This program is free software; you can redistribute it and/or modify 1175494230SNicolas Pitre * it under the terms of the GNU General Public License version 2 as 1275494230SNicolas Pitre * published by the Free Software Foundation. 1375494230SNicolas Pitre */ 1475494230SNicolas Pitre 1575494230SNicolas Pitre/* 1675494230SNicolas Pitre * Theory of operation 1775494230SNicolas Pitre * ------------------- 1875494230SNicolas Pitre * 1975494230SNicolas Pitre * This file provides the core code for a forward memory copy used in 2075494230SNicolas Pitre * the implementation of memcopy(), copy_to_user() and copy_from_user(). 2175494230SNicolas Pitre * 2275494230SNicolas Pitre * The including file must define the following accessor macros 2375494230SNicolas Pitre * according to the need of the given function: 2475494230SNicolas Pitre * 2575494230SNicolas Pitre * ldr1w ptr reg abort 2675494230SNicolas Pitre * 2775494230SNicolas Pitre * This loads one word from 'ptr', stores it in 'reg' and increments 2875494230SNicolas Pitre * 'ptr' to the next word. The 'abort' argument is used for fixup tables. 2975494230SNicolas Pitre * 3075494230SNicolas Pitre * ldr4w ptr reg1 reg2 reg3 reg4 abort 3175494230SNicolas Pitre * ldr8w ptr, reg1 reg2 reg3 reg4 reg5 reg6 reg7 reg8 abort 3275494230SNicolas Pitre * 3375494230SNicolas Pitre * This loads four or eight words starting from 'ptr', stores them 3475494230SNicolas Pitre * in provided registers and increments 'ptr' past those words. 3575494230SNicolas Pitre * The'abort' argument is used for fixup tables. 3675494230SNicolas Pitre * 3775494230SNicolas Pitre * ldr1b ptr reg cond abort 3875494230SNicolas Pitre * 3975494230SNicolas Pitre * Similar to ldr1w, but it loads a byte and increments 'ptr' one byte. 4075494230SNicolas Pitre * It also must apply the condition code if provided, otherwise the 4175494230SNicolas Pitre * "al" condition is assumed by default. 4275494230SNicolas Pitre * 4375494230SNicolas Pitre * str1w ptr reg abort 4475494230SNicolas Pitre * str8w ptr reg1 reg2 reg3 reg4 reg5 reg6 reg7 reg8 abort 4575494230SNicolas Pitre * str1b ptr reg cond abort 4675494230SNicolas Pitre * 4775494230SNicolas Pitre * Same as their ldr* counterparts, but data is stored to 'ptr' location 4875494230SNicolas Pitre * rather than being loaded. 4975494230SNicolas Pitre * 5075494230SNicolas Pitre * enter reg1 reg2 5175494230SNicolas Pitre * 5275494230SNicolas Pitre * Preserve the provided registers on the stack plus any additional 5375494230SNicolas Pitre * data as needed by the implementation including this code. Called 5475494230SNicolas Pitre * upon code entry. 5575494230SNicolas Pitre * 5675494230SNicolas Pitre * exit reg1 reg2 5775494230SNicolas Pitre * 5875494230SNicolas Pitre * Restore registers with the values previously saved with the 5975494230SNicolas Pitre * 'preserv' macro. Called upon code termination. 608b592783SCatalin Marinas * 618b592783SCatalin Marinas * LDR1W_SHIFT 628b592783SCatalin Marinas * STR1W_SHIFT 638b592783SCatalin Marinas * 648b592783SCatalin Marinas * Correction to be applied to the "ip" register when branching into 658b592783SCatalin Marinas * the ldr1w or str1w instructions (some of these macros may expand to 668b592783SCatalin Marinas * than one 32bit instruction in Thumb-2) 6775494230SNicolas Pitre */ 6875494230SNicolas Pitre 6975494230SNicolas Pitre 7075494230SNicolas Pitre enter r4, lr 7175494230SNicolas Pitre 7275494230SNicolas Pitre subs r2, r2, #4 7375494230SNicolas Pitre blt 8f 7475494230SNicolas Pitre ands ip, r0, #3 7575494230SNicolas Pitre PLD( pld [r1, #0] ) 7675494230SNicolas Pitre bne 9f 7775494230SNicolas Pitre ands ip, r1, #3 7875494230SNicolas Pitre bne 10f 7975494230SNicolas Pitre 8075494230SNicolas Pitre1: subs r2, r2, #(28) 8175494230SNicolas Pitre stmfd sp!, {r5 - r8} 8275494230SNicolas Pitre blt 5f 8375494230SNicolas Pitre 842239aff6SNicolas Pitre CALGN( ands ip, r0, #31 ) 8575494230SNicolas Pitre CALGN( rsb r3, ip, #32 ) 8675494230SNicolas Pitre CALGN( sbcnes r4, r3, r2 ) @ C is always set here 8775494230SNicolas Pitre CALGN( bcs 2f ) 8875494230SNicolas Pitre CALGN( adr r4, 6f ) 8975494230SNicolas Pitre CALGN( subs r2, r2, r3 ) @ C gets set 9075494230SNicolas Pitre CALGN( add pc, r4, ip ) 9175494230SNicolas Pitre 9275494230SNicolas Pitre PLD( pld [r1, #0] ) 9375494230SNicolas Pitre2: PLD( subs r2, r2, #96 ) 9475494230SNicolas Pitre PLD( pld [r1, #28] ) 9575494230SNicolas Pitre PLD( blt 4f ) 9675494230SNicolas Pitre PLD( pld [r1, #60] ) 9775494230SNicolas Pitre PLD( pld [r1, #92] ) 9875494230SNicolas Pitre 9975494230SNicolas Pitre3: PLD( pld [r1, #124] ) 10075494230SNicolas Pitre4: ldr8w r1, r3, r4, r5, r6, r7, r8, ip, lr, abort=20f 10175494230SNicolas Pitre subs r2, r2, #32 10275494230SNicolas Pitre str8w r0, r3, r4, r5, r6, r7, r8, ip, lr, abort=20f 10375494230SNicolas Pitre bge 3b 10475494230SNicolas Pitre PLD( cmn r2, #96 ) 10575494230SNicolas Pitre PLD( bge 4b ) 10675494230SNicolas Pitre 10775494230SNicolas Pitre5: ands ip, r2, #28 10875494230SNicolas Pitre rsb ip, ip, #32 1098b592783SCatalin Marinas#if LDR1W_SHIFT > 0 1108b592783SCatalin Marinas lsl ip, ip, #LDR1W_SHIFT 1118b592783SCatalin Marinas#endif 11275494230SNicolas Pitre addne pc, pc, ip @ C is always clear here 11375494230SNicolas Pitre b 7f 1148b592783SCatalin Marinas6: 1158b592783SCatalin Marinas .rept (1 << LDR1W_SHIFT) 1168b592783SCatalin Marinas W(nop) 1178b592783SCatalin Marinas .endr 11875494230SNicolas Pitre ldr1w r1, r3, abort=20f 11975494230SNicolas Pitre ldr1w r1, r4, abort=20f 12075494230SNicolas Pitre ldr1w r1, r5, abort=20f 12175494230SNicolas Pitre ldr1w r1, r6, abort=20f 12275494230SNicolas Pitre ldr1w r1, r7, abort=20f 12375494230SNicolas Pitre ldr1w r1, r8, abort=20f 12475494230SNicolas Pitre ldr1w r1, lr, abort=20f 12575494230SNicolas Pitre 1268b592783SCatalin Marinas#if LDR1W_SHIFT < STR1W_SHIFT 1278b592783SCatalin Marinas lsl ip, ip, #STR1W_SHIFT - LDR1W_SHIFT 1288b592783SCatalin Marinas#elif LDR1W_SHIFT > STR1W_SHIFT 1298b592783SCatalin Marinas lsr ip, ip, #LDR1W_SHIFT - STR1W_SHIFT 1308b592783SCatalin Marinas#endif 13175494230SNicolas Pitre add pc, pc, ip 13275494230SNicolas Pitre nop 1338b592783SCatalin Marinas .rept (1 << STR1W_SHIFT) 1348b592783SCatalin Marinas W(nop) 1358b592783SCatalin Marinas .endr 13675494230SNicolas Pitre str1w r0, r3, abort=20f 13775494230SNicolas Pitre str1w r0, r4, abort=20f 13875494230SNicolas Pitre str1w r0, r5, abort=20f 13975494230SNicolas Pitre str1w r0, r6, abort=20f 14075494230SNicolas Pitre str1w r0, r7, abort=20f 14175494230SNicolas Pitre str1w r0, r8, abort=20f 14275494230SNicolas Pitre str1w r0, lr, abort=20f 14375494230SNicolas Pitre 14475494230SNicolas Pitre CALGN( bcs 2b ) 14575494230SNicolas Pitre 14675494230SNicolas Pitre7: ldmfd sp!, {r5 - r8} 14775494230SNicolas Pitre 14875494230SNicolas Pitre8: movs r2, r2, lsl #31 14975494230SNicolas Pitre ldr1b r1, r3, ne, abort=21f 15075494230SNicolas Pitre ldr1b r1, r4, cs, abort=21f 15175494230SNicolas Pitre ldr1b r1, ip, cs, abort=21f 15275494230SNicolas Pitre str1b r0, r3, ne, abort=21f 15375494230SNicolas Pitre str1b r0, r4, cs, abort=21f 15475494230SNicolas Pitre str1b r0, ip, cs, abort=21f 15575494230SNicolas Pitre 15675494230SNicolas Pitre exit r4, pc 15775494230SNicolas Pitre 15875494230SNicolas Pitre9: rsb ip, ip, #4 15975494230SNicolas Pitre cmp ip, #2 16075494230SNicolas Pitre ldr1b r1, r3, gt, abort=21f 16175494230SNicolas Pitre ldr1b r1, r4, ge, abort=21f 16275494230SNicolas Pitre ldr1b r1, lr, abort=21f 16375494230SNicolas Pitre str1b r0, r3, gt, abort=21f 16475494230SNicolas Pitre str1b r0, r4, ge, abort=21f 16575494230SNicolas Pitre subs r2, r2, ip 16675494230SNicolas Pitre str1b r0, lr, abort=21f 16775494230SNicolas Pitre blt 8b 16875494230SNicolas Pitre ands ip, r1, #3 16975494230SNicolas Pitre beq 1b 17075494230SNicolas Pitre 17175494230SNicolas Pitre10: bic r1, r1, #3 17275494230SNicolas Pitre cmp ip, #2 17375494230SNicolas Pitre ldr1w r1, lr, abort=21f 17475494230SNicolas Pitre beq 17f 17575494230SNicolas Pitre bgt 18f 17675494230SNicolas Pitre 17775494230SNicolas Pitre 17875494230SNicolas Pitre .macro forward_copy_shift pull push 17975494230SNicolas Pitre 18075494230SNicolas Pitre subs r2, r2, #28 18175494230SNicolas Pitre blt 14f 18275494230SNicolas Pitre 1832239aff6SNicolas Pitre CALGN( ands ip, r0, #31 ) 18475494230SNicolas Pitre CALGN( rsb ip, ip, #32 ) 18575494230SNicolas Pitre CALGN( sbcnes r4, ip, r2 ) @ C is always set here 18675494230SNicolas Pitre CALGN( subcc r2, r2, ip ) 18775494230SNicolas Pitre CALGN( bcc 15f ) 18875494230SNicolas Pitre 18975494230SNicolas Pitre11: stmfd sp!, {r5 - r9} 19075494230SNicolas Pitre 19175494230SNicolas Pitre PLD( pld [r1, #0] ) 19275494230SNicolas Pitre PLD( subs r2, r2, #96 ) 19375494230SNicolas Pitre PLD( pld [r1, #28] ) 19475494230SNicolas Pitre PLD( blt 13f ) 19575494230SNicolas Pitre PLD( pld [r1, #60] ) 19675494230SNicolas Pitre PLD( pld [r1, #92] ) 19775494230SNicolas Pitre 19875494230SNicolas Pitre12: PLD( pld [r1, #124] ) 19975494230SNicolas Pitre13: ldr4w r1, r4, r5, r6, r7, abort=19f 200d98b90eaSVictor Kamensky mov r3, lr, lspull #\pull 20175494230SNicolas Pitre subs r2, r2, #32 20275494230SNicolas Pitre ldr4w r1, r8, r9, ip, lr, abort=19f 203d98b90eaSVictor Kamensky orr r3, r3, r4, lspush #\push 204d98b90eaSVictor Kamensky mov r4, r4, lspull #\pull 205d98b90eaSVictor Kamensky orr r4, r4, r5, lspush #\push 206d98b90eaSVictor Kamensky mov r5, r5, lspull #\pull 207d98b90eaSVictor Kamensky orr r5, r5, r6, lspush #\push 208d98b90eaSVictor Kamensky mov r6, r6, lspull #\pull 209d98b90eaSVictor Kamensky orr r6, r6, r7, lspush #\push 210d98b90eaSVictor Kamensky mov r7, r7, lspull #\pull 211d98b90eaSVictor Kamensky orr r7, r7, r8, lspush #\push 212d98b90eaSVictor Kamensky mov r8, r8, lspull #\pull 213d98b90eaSVictor Kamensky orr r8, r8, r9, lspush #\push 214d98b90eaSVictor Kamensky mov r9, r9, lspull #\pull 215d98b90eaSVictor Kamensky orr r9, r9, ip, lspush #\push 216d98b90eaSVictor Kamensky mov ip, ip, lspull #\pull 217d98b90eaSVictor Kamensky orr ip, ip, lr, lspush #\push 21875494230SNicolas Pitre str8w r0, r3, r4, r5, r6, r7, r8, r9, ip, , abort=19f 21975494230SNicolas Pitre bge 12b 22075494230SNicolas Pitre PLD( cmn r2, #96 ) 22175494230SNicolas Pitre PLD( bge 13b ) 22275494230SNicolas Pitre 22375494230SNicolas Pitre ldmfd sp!, {r5 - r9} 22475494230SNicolas Pitre 22575494230SNicolas Pitre14: ands ip, r2, #28 22675494230SNicolas Pitre beq 16f 22775494230SNicolas Pitre 228d98b90eaSVictor Kamensky15: mov r3, lr, lspull #\pull 22975494230SNicolas Pitre ldr1w r1, lr, abort=21f 23075494230SNicolas Pitre subs ip, ip, #4 231d98b90eaSVictor Kamensky orr r3, r3, lr, lspush #\push 23275494230SNicolas Pitre str1w r0, r3, abort=21f 23375494230SNicolas Pitre bgt 15b 23475494230SNicolas Pitre CALGN( cmp r2, #0 ) 23575494230SNicolas Pitre CALGN( bge 11b ) 23675494230SNicolas Pitre 23775494230SNicolas Pitre16: sub r1, r1, #(\push / 8) 23875494230SNicolas Pitre b 8b 23975494230SNicolas Pitre 24075494230SNicolas Pitre .endm 24175494230SNicolas Pitre 24275494230SNicolas Pitre 24375494230SNicolas Pitre forward_copy_shift pull=8 push=24 24475494230SNicolas Pitre 24575494230SNicolas Pitre17: forward_copy_shift pull=16 push=16 24675494230SNicolas Pitre 24775494230SNicolas Pitre18: forward_copy_shift pull=24 push=8 24875494230SNicolas Pitre 24975494230SNicolas Pitre 25075494230SNicolas Pitre/* 2517f927fccSAlexey Dobriyan * Abort preamble and completion macros. 25275494230SNicolas Pitre * If a fixup handler is required then those macros must surround it. 25375494230SNicolas Pitre * It is assumed that the fixup code will handle the private part of 25475494230SNicolas Pitre * the exit macro. 25575494230SNicolas Pitre */ 25675494230SNicolas Pitre 25775494230SNicolas Pitre .macro copy_abort_preamble 25875494230SNicolas Pitre19: ldmfd sp!, {r5 - r9} 25975494230SNicolas Pitre b 21f 26075494230SNicolas Pitre20: ldmfd sp!, {r5 - r8} 26175494230SNicolas Pitre21: 26275494230SNicolas Pitre .endm 26375494230SNicolas Pitre 26475494230SNicolas Pitre .macro copy_abort_end 26575494230SNicolas Pitre ldmfd sp!, {r4, pc} 26675494230SNicolas Pitre .endm 26775494230SNicolas Pitre 268