LoongArch: Adjust {copy, clear}_user exception handler behavior

The {copy, clear}_user function should returns number of bytes that
could not be {copied, cleared}. So, try to {copy, clear} byte by b

LoongArch: Adjust {copy, clear}_user exception handler behavior

The {copy, clear}_user function should returns number of bytes that
could not be {copied, cleared}. So, try to {copy, clear} byte by byte
when ld.{d,w,h} and st.{d,w,h} trapped into an exception.

Reviewed-by: WANG Rui <wangrui@loongson.cn>
Signed-off-by: Weihao Li <liweihao@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Replace #include <asm/export.h> with #include <linux/export.h>

Commit ddb5cdbafaaad ("kbuild: generate KSYMTAB entries by modpost")
deprecated <asm/export.h>, which is now a wrapper of <l

LoongArch: Replace #include <asm/export.h> with #include <linux/export.h>

Commit ddb5cdbafaaad ("kbuild: generate KSYMTAB entries by modpost")
deprecated <asm/export.h>, which is now a wrapper of <linux/export.h>.

Replace #include <asm/export.h> with #include <linux/export.h>.

After all the <asm/export.h> lines are converted, <asm/export.h> and
<asm-generic/export.h> will be removed.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Fix return value underflow in exception path

This patch fixes an underflow issue in the return value within the
exception path, specifically at .Llt8 when the remaining length is less
tha

LoongArch: Fix return value underflow in exception path

This patch fixes an underflow issue in the return value within the
exception path, specifically at .Llt8 when the remaining length is less
than 8 bytes.

Cc: stable@vger.kernel.org
Fixes: 8941e93ca590 ("LoongArch: Optimize memory ops (memset/memcpy/memmove)")
Reported-by: Weihao Li <liweihao@loongson.cn>
Signed-off-by: WANG Rui <wangrui@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Optimize memory ops (memset/memcpy/memmove)

To optimize memset()/memcpy()/memmove() and so on, we use a jump table
to dispatch cases for short data lengths; and for long data lengths, we

LoongArch: Optimize memory ops (memset/memcpy/memmove)

To optimize memset()/memcpy()/memmove() and so on, we use a jump table
to dispatch cases for short data lengths; and for long data lengths, we
split the destination into head part (first 8 bytes), tail part (last 8
bytes) and middle part. The head part and tail part may be at unaligned
addresses, while the middle part is always aligned (the middle part is
allowed to overlap the head/tail part). In this way, the first and last
8 bytes may be unaligned accesses, but we can make sure the data in the
middle is processed at an aligned destination address.

We have tested micro-bench[1] on a Loongson-3C5000 16-core machine (2.2GHz):

1. memset

| length | src offset | dst offset | speed before | speed after | % |
|--------|------------|------------|--------------|-------------|---------|
| 8 | 0 | 0 | 696.191 | 1518.785 | 118.16% |
| 8 | 0 | 1 | 696.325 | 1518.937 | 118.14% |
| 50 | 0 | 0 | 969.976 | 8053.902 | 730.32% |
| 50 | 0 | 1 | 970.034 | 8058.475 | 730.74% |
| 300 | 0 | 0 | 5876.612 | 16544.703 | 181.53% |
| 300 | 0 | 1 | 5030.849 | 16549.011 | 228.95% |
| 1200 | 0 | 0 | 11797.077 | 16752.137 | 42.00% |
| 1200 | 0 | 1 | 5687.141 | 16645.233 | 192.68% |
| 4000 | 0 | 0 | 15723.27 | 16761.557 | 6.60% |
| 4000 | 0 | 1 | 5906.114 | 16732.316 | 183.30% |
| 8000 | 0 | 0 | 16751.403 | 16770.002 | 0.11% |
| 8000 | 0 | 1 | 5995.449 | 16754.07 | 179.45% |

2. memcpy

| length | src offset | dst offset | speed before | speed after | % |
|--------|------------|------------|--------------|-------------|---------|
| 8 | 0 | 0 | 696.2 | 1670.605 | 139.96% |
| 8 | 0 | 1 | 696.325 | 1671.138 | 139.99% |
| 50 | 0 | 0 | 969.974 | 8724.999 | 799.51% |
| 50 | 0 | 1 | 970.032 | 8730.138 | 799.98% |
| 300 | 0 | 0 | 5564.662 | 16272.652 | 192.43% |
| 300 | 0 | 1 | 4670.436 | 14972.842 | 220.59% |
| 1200 | 0 | 0 | 10740.23 | 16751.728 | 55.97% |
| 1200 | 0 | 1 | 5027.741 | 14874.564 | 195.85% |
| 4000 | 0 | 0 | 15122.367 | 16737.642 | 10.68% |
| 4000 | 0 | 1 | 5536.918 | 14890.397 | 168.93% |
| 8000 | 0 | 0 | 16505.453 | 16553.543 | 0.29% |
| 8000 | 0 | 1 | 5821.619 | 14841.804 | 154.94% |

3. memmove

| length | src offset | dst offset | speed before | speed after | % |
|--------|------------|------------|--------------|-------------|---------|
| 8 | 0 | 0 | 982.693 | 1670.568 | 70.00% |
| 8 | 0 | 1 | 983.023 | 1671.174 | 70.00% |
| 50 | 0 | 0 | 1230.87 | 8727.625 | 609.06% |
| 50 | 0 | 1 | 1232.515 | 8730.138 | 608.32% |
| 300 | 0 | 0 | 6490.375 | 16296.993 | 151.09% |
| 300 | 0 | 1 | 4282.687 | 14972.842 | 249.61% |
| 1200 | 0 | 0 | 11742.755 | 16752.546 | 42.66% |
| 1200 | 0 | 1 | 5039.338 | 14872.951 | 195.14% |
| 4000 | 0 | 0 | 15467.786 | 16737.09 | 8.21% |
| 4000 | 0 | 1 | 5009.905 | 14890.542 | 197.22% |
| 8000 | 0 | 0 | 16489.664 | 16553.273 | 0.39% |
| 8000 | 0 | 1 | 5823.786 | 14858.646 | 155.14% |

* speed: MB/s
* length: byte

[1] https://github.com/heiher/mem-bench

Signed-off-by: WANG Rui <wangrui@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Use alternative to optimize libraries

Use the alternative to optimize common libraries according whether CPU
has UAL (hardware unaligned access support) feature, including memset(),
memco

LoongArch: Use alternative to optimize libraries

Use the alternative to optimize common libraries according whether CPU
has UAL (hardware unaligned access support) feature, including memset(),
memcopy(), memmove(), copy_user() and clear_user().

We have tested UnixBench on a Loongson-3A5000 quad-core machine (1.6GHz):

1, One copy, before patch:

System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 9566582.0 819.8
Double-Precision Whetstone 55.0 2805.3 510.1
Execl Throughput 43.0 2120.0 493.0
File Copy 1024 bufsize 2000 maxblocks 3960.0 209833.0 529.9
File Copy 256 bufsize 500 maxblocks 1655.0 89400.0 540.2
File Copy 4096 bufsize 8000 maxblocks 5800.0 320036.0 551.8
Pipe Throughput 12440.0 340624.0 273.8
Pipe-based Context Switching 4000.0 109939.1 274.8
Process Creation 126.0 4728.7 375.3
Shell Scripts (1 concurrent) 42.4 2223.1 524.3
Shell Scripts (8 concurrent) 6.0 883.1 1471.9
System Call Overhead 15000.0 518639.1 345.8
========
System Benchmarks Index Score 500.2

2, One copy, after patch:

System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 9567674.7 819.9
Double-Precision Whetstone 55.0 2805.5 510.1
Execl Throughput 43.0 2392.7 556.4
File Copy 1024 bufsize 2000 maxblocks 3960.0 417804.0 1055.1
File Copy 256 bufsize 500 maxblocks 1655.0 112909.5 682.2
File Copy 4096 bufsize 8000 maxblocks 5800.0 1255207.4 2164.2
Pipe Throughput 12440.0 555712.0 446.7
Pipe-based Context Switching 4000.0 99964.5 249.9
Process Creation 126.0 5192.5 412.1
Shell Scripts (1 concurrent) 42.4 2302.4 543.0
Shell Scripts (8 concurrent) 6.0 919.6 1532.6
System Call Overhead 15000.0 511159.3 340.8
========
System Benchmarks Index Score 640.1

3, Four copies, before patch:

System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 38268610.5 3279.2
Double-Precision Whetstone 55.0 11222.2 2040.4
Execl Throughput 43.0 7892.0 1835.3
File Copy 1024 bufsize 2000 maxblocks 3960.0 235149.6 593.8
File Copy 256 bufsize 500 maxblocks 1655.0 74959.6 452.9
File Copy 4096 bufsize 8000 maxblocks 5800.0 545048.5 939.7
Pipe Throughput 12440.0 1337359.0 1075.0
Pipe-based Context Switching 4000.0 473663.9 1184.2
Process Creation 126.0 17491.2 1388.2
Shell Scripts (1 concurrent) 42.4 6865.7 1619.3
Shell Scripts (8 concurrent) 6.0 1015.9 1693.1
System Call Overhead 15000.0 1899535.2 1266.4
========
System Benchmarks Index Score 1278.3

4, Four copies, after patch:

System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 38272815.5 3279.6
Double-Precision Whetstone 55.0 11222.8 2040.5
Execl Throughput 43.0 8839.2 2055.6
File Copy 1024 bufsize 2000 maxblocks 3960.0 313912.9 792.7
File Copy 256 bufsize 500 maxblocks 1655.0 80976.1 489.3
File Copy 4096 bufsize 8000 maxblocks 5800.0 1176594.3 2028.6
Pipe Throughput 12440.0 2100941.9 1688.9
Pipe-based Context Switching 4000.0 476696.4 1191.7
Process Creation 126.0 18394.7 1459.9
Shell Scripts (1 concurrent) 42.4 7172.2 1691.6
Shell Scripts (8 concurrent) 6.0 1058.3 1763.9
System Call Overhead 15000.0 1874714.7 1249.8
========
System Benchmarks Index Score 1488.8

Signed-off-by: Jun Yi <yijun@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Remove the .fixup section usage

Use the `.L_xxx` label to improve fixup code and then remove the .fixup
section usage.

Signed-off-by: Youling Tang <tangyouling@loongson.cn>
Signed-off-by

LoongArch: Remove the .fixup section usage

Use the `.L_xxx` label to improve fixup code and then remove the .fixup
section usage.

Signed-off-by: Youling Tang <tangyouling@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Consolidate __ex_table construction

Consolidate all the __ex_table constuction code with a _ASM_EXTABLE or
_asm_extable helper.

There should be no functional change as a result of this p

LoongArch: Consolidate __ex_table construction

Consolidate all the __ex_table constuction code with a _ASM_EXTABLE or
_asm_extable helper.

There should be no functional change as a result of this patch.

Signed-off-by: Youling Tang <tangyouling@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Simplify "BGT foo, zero" with BGTZ

Support for the syntactic sugar is present in upstream binutils port
from the beginning. Use it for shorter lines and better consistency.
Generated code

LoongArch: Simplify "BGT foo, zero" with BGTZ

Support for the syntactic sugar is present in upstream binutils port
from the beginning. Use it for shorter lines and better consistency.
Generated code should be identical.

Signed-off-by: WANG Xuerui <git@xen0n.name>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...

LoongArch: Add some library functions

Add some library functions for LoongArch, including: delay, memset,
memcpy, memmove, copy_user, strncpy_user, strnlen_user and tlb dump
functions.

Reviewed-by:

LoongArch: Add some library functions

Add some library functions for LoongArch, including: delay, memset,
memcpy, memmove, copy_user, strncpy_user, strnlen_user and tlb dump
functions.

Reviewed-by: WANG Xuerui <git@xen0n.name>
Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

show more ...