175494230SNicolas Pitre/* 275494230SNicolas Pitre * linux/arch/arm/lib/copy_template.s 375494230SNicolas Pitre * 475494230SNicolas Pitre * Code template for optimized memory copy functions 575494230SNicolas Pitre * 675494230SNicolas Pitre * Author: Nicolas Pitre 775494230SNicolas Pitre * Created: Sep 28, 2005 875494230SNicolas Pitre * Copyright: MontaVista Software, Inc. 975494230SNicolas Pitre * 1075494230SNicolas Pitre * This program is free software; you can redistribute it and/or modify 1175494230SNicolas Pitre * it under the terms of the GNU General Public License version 2 as 1275494230SNicolas Pitre * published by the Free Software Foundation. 1375494230SNicolas Pitre */ 1475494230SNicolas Pitre 1575494230SNicolas Pitre/* 1675494230SNicolas Pitre * This can be used to enable code to cacheline align the source pointer. 1775494230SNicolas Pitre * Experiments on tested architectures (StrongARM and XScale) didn't show 1875494230SNicolas Pitre * this a worthwhile thing to do. That might be different in the future. 1975494230SNicolas Pitre */ 2075494230SNicolas Pitre//#define CALGN(code...) code 2175494230SNicolas Pitre#define CALGN(code...) 2275494230SNicolas Pitre 2375494230SNicolas Pitre/* 2475494230SNicolas Pitre * Theory of operation 2575494230SNicolas Pitre * ------------------- 2675494230SNicolas Pitre * 2775494230SNicolas Pitre * This file provides the core code for a forward memory copy used in 2875494230SNicolas Pitre * the implementation of memcopy(), copy_to_user() and copy_from_user(). 2975494230SNicolas Pitre * 3075494230SNicolas Pitre * The including file must define the following accessor macros 3175494230SNicolas Pitre * according to the need of the given function: 3275494230SNicolas Pitre * 3375494230SNicolas Pitre * ldr1w ptr reg abort 3475494230SNicolas Pitre * 3575494230SNicolas Pitre * This loads one word from 'ptr', stores it in 'reg' and increments 3675494230SNicolas Pitre * 'ptr' to the next word. The 'abort' argument is used for fixup tables. 3775494230SNicolas Pitre * 3875494230SNicolas Pitre * ldr4w ptr reg1 reg2 reg3 reg4 abort 3975494230SNicolas Pitre * ldr8w ptr, reg1 reg2 reg3 reg4 reg5 reg6 reg7 reg8 abort 4075494230SNicolas Pitre * 4175494230SNicolas Pitre * This loads four or eight words starting from 'ptr', stores them 4275494230SNicolas Pitre * in provided registers and increments 'ptr' past those words. 4375494230SNicolas Pitre * The'abort' argument is used for fixup tables. 4475494230SNicolas Pitre * 4575494230SNicolas Pitre * ldr1b ptr reg cond abort 4675494230SNicolas Pitre * 4775494230SNicolas Pitre * Similar to ldr1w, but it loads a byte and increments 'ptr' one byte. 4875494230SNicolas Pitre * It also must apply the condition code if provided, otherwise the 4975494230SNicolas Pitre * "al" condition is assumed by default. 5075494230SNicolas Pitre * 5175494230SNicolas Pitre * str1w ptr reg abort 5275494230SNicolas Pitre * str8w ptr reg1 reg2 reg3 reg4 reg5 reg6 reg7 reg8 abort 5375494230SNicolas Pitre * str1b ptr reg cond abort 5475494230SNicolas Pitre * 5575494230SNicolas Pitre * Same as their ldr* counterparts, but data is stored to 'ptr' location 5675494230SNicolas Pitre * rather than being loaded. 5775494230SNicolas Pitre * 5875494230SNicolas Pitre * enter reg1 reg2 5975494230SNicolas Pitre * 6075494230SNicolas Pitre * Preserve the provided registers on the stack plus any additional 6175494230SNicolas Pitre * data as needed by the implementation including this code. Called 6275494230SNicolas Pitre * upon code entry. 6375494230SNicolas Pitre * 6475494230SNicolas Pitre * exit reg1 reg2 6575494230SNicolas Pitre * 6675494230SNicolas Pitre * Restore registers with the values previously saved with the 6775494230SNicolas Pitre * 'preserv' macro. Called upon code termination. 6875494230SNicolas Pitre */ 6975494230SNicolas Pitre 7075494230SNicolas Pitre 7175494230SNicolas Pitre enter r4, lr 7275494230SNicolas Pitre 7375494230SNicolas Pitre subs r2, r2, #4 7475494230SNicolas Pitre blt 8f 7575494230SNicolas Pitre ands ip, r0, #3 7675494230SNicolas Pitre PLD( pld [r1, #0] ) 7775494230SNicolas Pitre bne 9f 7875494230SNicolas Pitre ands ip, r1, #3 7975494230SNicolas Pitre bne 10f 8075494230SNicolas Pitre 8175494230SNicolas Pitre1: subs r2, r2, #(28) 8275494230SNicolas Pitre stmfd sp!, {r5 - r8} 8375494230SNicolas Pitre blt 5f 8475494230SNicolas Pitre 8575494230SNicolas Pitre CALGN( ands ip, r1, #31 ) 8675494230SNicolas Pitre CALGN( rsb r3, ip, #32 ) 8775494230SNicolas Pitre CALGN( sbcnes r4, r3, r2 ) @ C is always set here 8875494230SNicolas Pitre CALGN( bcs 2f ) 8975494230SNicolas Pitre CALGN( adr r4, 6f ) 9075494230SNicolas Pitre CALGN( subs r2, r2, r3 ) @ C gets set 9175494230SNicolas Pitre CALGN( add pc, r4, ip ) 9275494230SNicolas Pitre 9375494230SNicolas Pitre PLD( pld [r1, #0] ) 9475494230SNicolas Pitre2: PLD( subs r2, r2, #96 ) 9575494230SNicolas Pitre PLD( pld [r1, #28] ) 9675494230SNicolas Pitre PLD( blt 4f ) 9775494230SNicolas Pitre PLD( pld [r1, #60] ) 9875494230SNicolas Pitre PLD( pld [r1, #92] ) 9975494230SNicolas Pitre 10075494230SNicolas Pitre3: PLD( pld [r1, #124] ) 10175494230SNicolas Pitre4: ldr8w r1, r3, r4, r5, r6, r7, r8, ip, lr, abort=20f 10275494230SNicolas Pitre subs r2, r2, #32 10375494230SNicolas Pitre str8w r0, r3, r4, r5, r6, r7, r8, ip, lr, abort=20f 10475494230SNicolas Pitre bge 3b 10575494230SNicolas Pitre PLD( cmn r2, #96 ) 10675494230SNicolas Pitre PLD( bge 4b ) 10775494230SNicolas Pitre 10875494230SNicolas Pitre5: ands ip, r2, #28 10975494230SNicolas Pitre rsb ip, ip, #32 11075494230SNicolas Pitre addne pc, pc, ip @ C is always clear here 11175494230SNicolas Pitre b 7f 11275494230SNicolas Pitre6: nop 11375494230SNicolas Pitre ldr1w r1, r3, abort=20f 11475494230SNicolas Pitre ldr1w r1, r4, abort=20f 11575494230SNicolas Pitre ldr1w r1, r5, abort=20f 11675494230SNicolas Pitre ldr1w r1, r6, abort=20f 11775494230SNicolas Pitre ldr1w r1, r7, abort=20f 11875494230SNicolas Pitre ldr1w r1, r8, abort=20f 11975494230SNicolas Pitre ldr1w r1, lr, abort=20f 12075494230SNicolas Pitre 12175494230SNicolas Pitre add pc, pc, ip 12275494230SNicolas Pitre nop 12375494230SNicolas Pitre nop 12475494230SNicolas Pitre str1w r0, r3, abort=20f 12575494230SNicolas Pitre str1w r0, r4, abort=20f 12675494230SNicolas Pitre str1w r0, r5, abort=20f 12775494230SNicolas Pitre str1w r0, r6, abort=20f 12875494230SNicolas Pitre str1w r0, r7, abort=20f 12975494230SNicolas Pitre str1w r0, r8, abort=20f 13075494230SNicolas Pitre str1w r0, lr, abort=20f 13175494230SNicolas Pitre 13275494230SNicolas Pitre CALGN( bcs 2b ) 13375494230SNicolas Pitre 13475494230SNicolas Pitre7: ldmfd sp!, {r5 - r8} 13575494230SNicolas Pitre 13675494230SNicolas Pitre8: movs r2, r2, lsl #31 13775494230SNicolas Pitre ldr1b r1, r3, ne, abort=21f 13875494230SNicolas Pitre ldr1b r1, r4, cs, abort=21f 13975494230SNicolas Pitre ldr1b r1, ip, cs, abort=21f 14075494230SNicolas Pitre str1b r0, r3, ne, abort=21f 14175494230SNicolas Pitre str1b r0, r4, cs, abort=21f 14275494230SNicolas Pitre str1b r0, ip, cs, abort=21f 14375494230SNicolas Pitre 14475494230SNicolas Pitre exit r4, pc 14575494230SNicolas Pitre 14675494230SNicolas Pitre9: rsb ip, ip, #4 14775494230SNicolas Pitre cmp ip, #2 14875494230SNicolas Pitre ldr1b r1, r3, gt, abort=21f 14975494230SNicolas Pitre ldr1b r1, r4, ge, abort=21f 15075494230SNicolas Pitre ldr1b r1, lr, abort=21f 15175494230SNicolas Pitre str1b r0, r3, gt, abort=21f 15275494230SNicolas Pitre str1b r0, r4, ge, abort=21f 15375494230SNicolas Pitre subs r2, r2, ip 15475494230SNicolas Pitre str1b r0, lr, abort=21f 15575494230SNicolas Pitre blt 8b 15675494230SNicolas Pitre ands ip, r1, #3 15775494230SNicolas Pitre beq 1b 15875494230SNicolas Pitre 15975494230SNicolas Pitre10: bic r1, r1, #3 16075494230SNicolas Pitre cmp ip, #2 16175494230SNicolas Pitre ldr1w r1, lr, abort=21f 16275494230SNicolas Pitre beq 17f 16375494230SNicolas Pitre bgt 18f 16475494230SNicolas Pitre 16575494230SNicolas Pitre 16675494230SNicolas Pitre .macro forward_copy_shift pull push 16775494230SNicolas Pitre 16875494230SNicolas Pitre subs r2, r2, #28 16975494230SNicolas Pitre blt 14f 17075494230SNicolas Pitre 17175494230SNicolas Pitre CALGN( ands ip, r1, #31 ) 17275494230SNicolas Pitre CALGN( rsb ip, ip, #32 ) 17375494230SNicolas Pitre CALGN( sbcnes r4, ip, r2 ) @ C is always set here 17475494230SNicolas Pitre CALGN( subcc r2, r2, ip ) 17575494230SNicolas Pitre CALGN( bcc 15f ) 17675494230SNicolas Pitre 17775494230SNicolas Pitre11: stmfd sp!, {r5 - r9} 17875494230SNicolas Pitre 17975494230SNicolas Pitre PLD( pld [r1, #0] ) 18075494230SNicolas Pitre PLD( subs r2, r2, #96 ) 18175494230SNicolas Pitre PLD( pld [r1, #28] ) 18275494230SNicolas Pitre PLD( blt 13f ) 18375494230SNicolas Pitre PLD( pld [r1, #60] ) 18475494230SNicolas Pitre PLD( pld [r1, #92] ) 18575494230SNicolas Pitre 18675494230SNicolas Pitre12: PLD( pld [r1, #124] ) 18775494230SNicolas Pitre13: ldr4w r1, r4, r5, r6, r7, abort=19f 18875494230SNicolas Pitre mov r3, lr, pull #\pull 18975494230SNicolas Pitre subs r2, r2, #32 19075494230SNicolas Pitre ldr4w r1, r8, r9, ip, lr, abort=19f 19175494230SNicolas Pitre orr r3, r3, r4, push #\push 19275494230SNicolas Pitre mov r4, r4, pull #\pull 19375494230SNicolas Pitre orr r4, r4, r5, push #\push 19475494230SNicolas Pitre mov r5, r5, pull #\pull 19575494230SNicolas Pitre orr r5, r5, r6, push #\push 19675494230SNicolas Pitre mov r6, r6, pull #\pull 19775494230SNicolas Pitre orr r6, r6, r7, push #\push 19875494230SNicolas Pitre mov r7, r7, pull #\pull 19975494230SNicolas Pitre orr r7, r7, r8, push #\push 20075494230SNicolas Pitre mov r8, r8, pull #\pull 20175494230SNicolas Pitre orr r8, r8, r9, push #\push 20275494230SNicolas Pitre mov r9, r9, pull #\pull 20375494230SNicolas Pitre orr r9, r9, ip, push #\push 20475494230SNicolas Pitre mov ip, ip, pull #\pull 20575494230SNicolas Pitre orr ip, ip, lr, push #\push 20675494230SNicolas Pitre str8w r0, r3, r4, r5, r6, r7, r8, r9, ip, , abort=19f 20775494230SNicolas Pitre bge 12b 20875494230SNicolas Pitre PLD( cmn r2, #96 ) 20975494230SNicolas Pitre PLD( bge 13b ) 21075494230SNicolas Pitre 21175494230SNicolas Pitre ldmfd sp!, {r5 - r9} 21275494230SNicolas Pitre 21375494230SNicolas Pitre14: ands ip, r2, #28 21475494230SNicolas Pitre beq 16f 21575494230SNicolas Pitre 21675494230SNicolas Pitre15: mov r3, lr, pull #\pull 21775494230SNicolas Pitre ldr1w r1, lr, abort=21f 21875494230SNicolas Pitre subs ip, ip, #4 21975494230SNicolas Pitre orr r3, r3, lr, push #\push 22075494230SNicolas Pitre str1w r0, r3, abort=21f 22175494230SNicolas Pitre bgt 15b 22275494230SNicolas Pitre CALGN( cmp r2, #0 ) 22375494230SNicolas Pitre CALGN( bge 11b ) 22475494230SNicolas Pitre 22575494230SNicolas Pitre16: sub r1, r1, #(\push / 8) 22675494230SNicolas Pitre b 8b 22775494230SNicolas Pitre 22875494230SNicolas Pitre .endm 22975494230SNicolas Pitre 23075494230SNicolas Pitre 23175494230SNicolas Pitre forward_copy_shift pull=8 push=24 23275494230SNicolas Pitre 23375494230SNicolas Pitre17: forward_copy_shift pull=16 push=16 23475494230SNicolas Pitre 23575494230SNicolas Pitre18: forward_copy_shift pull=24 push=8 23675494230SNicolas Pitre 23775494230SNicolas Pitre 23875494230SNicolas Pitre/* 23975494230SNicolas Pitre * Abort preanble and completion macros. 24075494230SNicolas Pitre * If a fixup handler is required then those macros must surround it. 24175494230SNicolas Pitre * It is assumed that the fixup code will handle the private part of 24275494230SNicolas Pitre * the exit macro. 24375494230SNicolas Pitre */ 24475494230SNicolas Pitre 24575494230SNicolas Pitre .macro copy_abort_preamble 24675494230SNicolas Pitre19: ldmfd sp!, {r5 - r9} 24775494230SNicolas Pitre b 21f 24875494230SNicolas Pitre20: ldmfd sp!, {r5 - r8} 24975494230SNicolas Pitre21: 25075494230SNicolas Pitre .endm 25175494230SNicolas Pitre 25275494230SNicolas Pitre .macro copy_abort_end 25375494230SNicolas Pitre ldmfd sp!, {r4, pc} 25475494230SNicolas Pitre .endm 25575494230SNicolas Pitre 256