113d19498SJack Steiner /* 213d19498SJack Steiner * SN Platform GRU Driver 313d19498SJack Steiner * 413d19498SJack Steiner * GRU DRIVER TABLES, MACROS, externs, etc 513d19498SJack Steiner * 613d19498SJack Steiner * Copyright (c) 2008 Silicon Graphics, Inc. All Rights Reserved. 713d19498SJack Steiner * 813d19498SJack Steiner * This program is free software; you can redistribute it and/or modify 913d19498SJack Steiner * it under the terms of the GNU General Public License as published by 1013d19498SJack Steiner * the Free Software Foundation; either version 2 of the License, or 1113d19498SJack Steiner * (at your option) any later version. 1213d19498SJack Steiner * 1313d19498SJack Steiner * This program is distributed in the hope that it will be useful, 1413d19498SJack Steiner * but WITHOUT ANY WARRANTY; without even the implied warranty of 1513d19498SJack Steiner * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 1613d19498SJack Steiner * GNU General Public License for more details. 1713d19498SJack Steiner * 1813d19498SJack Steiner * You should have received a copy of the GNU General Public License 1913d19498SJack Steiner * along with this program; if not, write to the Free Software 2013d19498SJack Steiner * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 2113d19498SJack Steiner */ 2213d19498SJack Steiner 2313d19498SJack Steiner #ifndef __GRUTABLES_H__ 2413d19498SJack Steiner #define __GRUTABLES_H__ 2513d19498SJack Steiner 2613d19498SJack Steiner /* 279ca8e40cSJack Steiner * GRU Chiplet: 289ca8e40cSJack Steiner * The GRU is a user addressible memory accelerator. It provides 299ca8e40cSJack Steiner * several forms of load, store, memset, bcopy instructions. In addition, it 309ca8e40cSJack Steiner * contains special instructions for AMOs, sending messages to message 319ca8e40cSJack Steiner * queues, etc. 329ca8e40cSJack Steiner * 339ca8e40cSJack Steiner * The GRU is an integral part of the node controller. It connects 349ca8e40cSJack Steiner * directly to the cpu socket. In its current implementation, there are 2 359ca8e40cSJack Steiner * GRU chiplets in the node controller on each blade (~node). 369ca8e40cSJack Steiner * 379ca8e40cSJack Steiner * The entire GRU memory space is fully coherent and cacheable by the cpus. 389ca8e40cSJack Steiner * 399ca8e40cSJack Steiner * Each GRU chiplet has a physical memory map that looks like the following: 409ca8e40cSJack Steiner * 419ca8e40cSJack Steiner * +-----------------+ 429ca8e40cSJack Steiner * |/////////////////| 439ca8e40cSJack Steiner * |/////////////////| 449ca8e40cSJack Steiner * |/////////////////| 459ca8e40cSJack Steiner * |/////////////////| 469ca8e40cSJack Steiner * |/////////////////| 479ca8e40cSJack Steiner * |/////////////////| 489ca8e40cSJack Steiner * |/////////////////| 499ca8e40cSJack Steiner * |/////////////////| 509ca8e40cSJack Steiner * +-----------------+ 519ca8e40cSJack Steiner * | system control | 529ca8e40cSJack Steiner * +-----------------+ _______ +-------------+ 539ca8e40cSJack Steiner * |/////////////////| / | | 549ca8e40cSJack Steiner * |/////////////////| / | | 559ca8e40cSJack Steiner * |/////////////////| / | instructions| 569ca8e40cSJack Steiner * |/////////////////| / | | 579ca8e40cSJack Steiner * |/////////////////| / | | 589ca8e40cSJack Steiner * |/////////////////| / |-------------| 599ca8e40cSJack Steiner * |/////////////////| / | | 609ca8e40cSJack Steiner * +-----------------+ | | 619ca8e40cSJack Steiner * | context 15 | | data | 629ca8e40cSJack Steiner * +-----------------+ | | 639ca8e40cSJack Steiner * | ...... | \ | | 649ca8e40cSJack Steiner * +-----------------+ \____________ +-------------+ 659ca8e40cSJack Steiner * | context 1 | 669ca8e40cSJack Steiner * +-----------------+ 679ca8e40cSJack Steiner * | context 0 | 689ca8e40cSJack Steiner * +-----------------+ 699ca8e40cSJack Steiner * 709ca8e40cSJack Steiner * Each of the "contexts" is a chunk of memory that can be mmaped into user 719ca8e40cSJack Steiner * space. The context consists of 2 parts: 729ca8e40cSJack Steiner * 739ca8e40cSJack Steiner * - an instruction space that can be directly accessed by the user 749ca8e40cSJack Steiner * to issue GRU instructions and to check instruction status. 759ca8e40cSJack Steiner * 769ca8e40cSJack Steiner * - a data area that acts as normal RAM. 779ca8e40cSJack Steiner * 789ca8e40cSJack Steiner * User instructions contain virtual addresses of data to be accessed by the 799ca8e40cSJack Steiner * GRU. The GRU contains a TLB that is used to convert these user virtual 809ca8e40cSJack Steiner * addresses to physical addresses. 819ca8e40cSJack Steiner * 829ca8e40cSJack Steiner * The "system control" area of the GRU chiplet is used by the kernel driver 839ca8e40cSJack Steiner * to manage user contexts and to perform functions such as TLB dropin and 849ca8e40cSJack Steiner * purging. 859ca8e40cSJack Steiner * 869ca8e40cSJack Steiner * One context may be reserved for the kernel and used for cross-partition 879ca8e40cSJack Steiner * communication. The GRU will also be used to asynchronously zero out 889ca8e40cSJack Steiner * large blocks of memory (not currently implemented). 899ca8e40cSJack Steiner * 909ca8e40cSJack Steiner * 9113d19498SJack Steiner * Tables: 9213d19498SJack Steiner * 9313d19498SJack Steiner * VDATA-VMA Data - Holds a few parameters. Head of linked list of 9413d19498SJack Steiner * GTS tables for threads using the GSEG 9513d19498SJack Steiner * GTS - Gru Thread State - contains info for managing a GSEG context. A 9613d19498SJack Steiner * GTS is allocated for each thread accessing a 9713d19498SJack Steiner * GSEG. 9813d19498SJack Steiner * GTD - GRU Thread Data - contains shadow copy of GRU data when GSEG is 9913d19498SJack Steiner * not loaded into a GRU 10013d19498SJack Steiner * GMS - GRU Memory Struct - Used to manage TLB shootdowns. Tracks GRUs 10113d19498SJack Steiner * where a GSEG has been loaded. Similar to 10213d19498SJack Steiner * an mm_struct but for GRU. 10313d19498SJack Steiner * 10413d19498SJack Steiner * GS - GRU State - Used to manage the state of a GRU chiplet 10513d19498SJack Steiner * BS - Blade State - Used to manage state of all GRU chiplets 10613d19498SJack Steiner * on a blade 10713d19498SJack Steiner * 10813d19498SJack Steiner * 10913d19498SJack Steiner * Normal task tables for task using GRU. 11013d19498SJack Steiner * - 2 threads in process 11113d19498SJack Steiner * - 2 GSEGs open in process 11213d19498SJack Steiner * - GSEG1 is being used by both threads 11313d19498SJack Steiner * - GSEG2 is used only by thread 2 11413d19498SJack Steiner * 11513d19498SJack Steiner * task -->| 11613d19498SJack Steiner * task ---+---> mm ->------ (notifier) -------+-> gms 11713d19498SJack Steiner * | | 11813d19498SJack Steiner * |--> vma -> vdata ---> gts--->| GSEG1 (thread1) 11913d19498SJack Steiner * | | | 12013d19498SJack Steiner * | +-> gts--->| GSEG1 (thread2) 12113d19498SJack Steiner * | | 12213d19498SJack Steiner * |--> vma -> vdata ---> gts--->| GSEG2 (thread2) 12313d19498SJack Steiner * . 12413d19498SJack Steiner * . 12513d19498SJack Steiner * 12613d19498SJack Steiner * GSEGs are marked DONTCOPY on fork 12713d19498SJack Steiner * 12813d19498SJack Steiner * At open 12913d19498SJack Steiner * file.private_data -> NULL 13013d19498SJack Steiner * 13113d19498SJack Steiner * At mmap, 13213d19498SJack Steiner * vma -> vdata 13313d19498SJack Steiner * 13413d19498SJack Steiner * After gseg reference 13513d19498SJack Steiner * vma -> vdata ->gts 13613d19498SJack Steiner * 13713d19498SJack Steiner * After fork 13813d19498SJack Steiner * parent 13913d19498SJack Steiner * vma -> vdata -> gts 14013d19498SJack Steiner * child 14113d19498SJack Steiner * (vma is not copied) 14213d19498SJack Steiner * 14313d19498SJack Steiner */ 14413d19498SJack Steiner 14513d19498SJack Steiner #include <linux/rmap.h> 14613d19498SJack Steiner #include <linux/interrupt.h> 14713d19498SJack Steiner #include <linux/mutex.h> 14813d19498SJack Steiner #include <linux/wait.h> 14913d19498SJack Steiner #include <linux/mmu_notifier.h> 15013d19498SJack Steiner #include "gru.h" 15113d19498SJack Steiner #include "gruhandles.h" 15213d19498SJack Steiner 15313d19498SJack Steiner extern struct gru_stats_s gru_stats; 15413d19498SJack Steiner extern struct gru_blade_state *gru_base[]; 15513d19498SJack Steiner extern unsigned long gru_start_paddr, gru_end_paddr; 15613d19498SJack Steiner 15713d19498SJack Steiner #define GRU_MAX_BLADES MAX_NUMNODES 15813d19498SJack Steiner #define GRU_MAX_GRUS (GRU_MAX_BLADES * GRU_CHIPLETS_PER_BLADE) 15913d19498SJack Steiner 16013d19498SJack Steiner #define GRU_DRIVER_ID_STR "SGI GRU Device Driver" 16113d19498SJack Steiner #define GRU_DRIVER_VERSION_STR "0.80" 16213d19498SJack Steiner 16313d19498SJack Steiner /* 16413d19498SJack Steiner * GRU statistics. 16513d19498SJack Steiner */ 16613d19498SJack Steiner struct gru_stats_s { 16713d19498SJack Steiner atomic_long_t vdata_alloc; 16813d19498SJack Steiner atomic_long_t vdata_free; 16913d19498SJack Steiner atomic_long_t gts_alloc; 17013d19498SJack Steiner atomic_long_t gts_free; 17113d19498SJack Steiner atomic_long_t vdata_double_alloc; 17213d19498SJack Steiner atomic_long_t gts_double_allocate; 17313d19498SJack Steiner atomic_long_t assign_context; 17413d19498SJack Steiner atomic_long_t assign_context_failed; 17513d19498SJack Steiner atomic_long_t free_context; 17613d19498SJack Steiner atomic_long_t load_context; 17713d19498SJack Steiner atomic_long_t unload_context; 17813d19498SJack Steiner atomic_long_t steal_context; 17913d19498SJack Steiner atomic_long_t steal_context_failed; 18013d19498SJack Steiner atomic_long_t nopfn; 18113d19498SJack Steiner atomic_long_t break_cow; 18213d19498SJack Steiner atomic_long_t asid_new; 18313d19498SJack Steiner atomic_long_t asid_next; 18413d19498SJack Steiner atomic_long_t asid_wrap; 18513d19498SJack Steiner atomic_long_t asid_reuse; 18613d19498SJack Steiner atomic_long_t intr; 18743884604SJack Steiner atomic_long_t intr_mm_lock_failed; 18813d19498SJack Steiner atomic_long_t call_os; 18943884604SJack Steiner atomic_long_t call_os_offnode_reference; 19013d19498SJack Steiner atomic_long_t call_os_check_for_bug; 19113d19498SJack Steiner atomic_long_t call_os_wait_queue; 19213d19498SJack Steiner atomic_long_t user_flush_tlb; 19313d19498SJack Steiner atomic_long_t user_unload_context; 19413d19498SJack Steiner atomic_long_t user_exception; 19513d19498SJack Steiner atomic_long_t set_task_slice; 19613d19498SJack Steiner atomic_long_t migrate_check; 19713d19498SJack Steiner atomic_long_t migrated_retarget; 19813d19498SJack Steiner atomic_long_t migrated_unload; 19913d19498SJack Steiner atomic_long_t migrated_unload_delay; 20013d19498SJack Steiner atomic_long_t migrated_nopfn_retarget; 20113d19498SJack Steiner atomic_long_t migrated_nopfn_unload; 20213d19498SJack Steiner atomic_long_t tlb_dropin; 20313d19498SJack Steiner atomic_long_t tlb_dropin_fail_no_asid; 20413d19498SJack Steiner atomic_long_t tlb_dropin_fail_upm; 20513d19498SJack Steiner atomic_long_t tlb_dropin_fail_invalid; 20613d19498SJack Steiner atomic_long_t tlb_dropin_fail_range_active; 20713d19498SJack Steiner atomic_long_t tlb_dropin_fail_idle; 20813d19498SJack Steiner atomic_long_t tlb_dropin_fail_fmm; 20913d19498SJack Steiner atomic_long_t mmu_invalidate_range; 21013d19498SJack Steiner atomic_long_t mmu_invalidate_page; 21113d19498SJack Steiner atomic_long_t mmu_clear_flush_young; 21213d19498SJack Steiner atomic_long_t flush_tlb; 21313d19498SJack Steiner atomic_long_t flush_tlb_gru; 21413d19498SJack Steiner atomic_long_t flush_tlb_gru_tgh; 21513d19498SJack Steiner atomic_long_t flush_tlb_gru_zero_asid; 21613d19498SJack Steiner 21713d19498SJack Steiner atomic_long_t copy_gpa; 21813d19498SJack Steiner 21913d19498SJack Steiner atomic_long_t mesq_receive; 22013d19498SJack Steiner atomic_long_t mesq_receive_none; 22113d19498SJack Steiner atomic_long_t mesq_send; 22213d19498SJack Steiner atomic_long_t mesq_send_failed; 22313d19498SJack Steiner atomic_long_t mesq_noop; 22413d19498SJack Steiner atomic_long_t mesq_send_unexpected_error; 22513d19498SJack Steiner atomic_long_t mesq_send_lb_overflow; 22613d19498SJack Steiner atomic_long_t mesq_send_qlimit_reached; 22713d19498SJack Steiner atomic_long_t mesq_send_amo_nacked; 22813d19498SJack Steiner atomic_long_t mesq_send_put_nacked; 22913d19498SJack Steiner atomic_long_t mesq_qf_not_full; 23013d19498SJack Steiner atomic_long_t mesq_qf_locked; 23113d19498SJack Steiner atomic_long_t mesq_qf_noop_not_full; 23213d19498SJack Steiner atomic_long_t mesq_qf_switch_head_failed; 23313d19498SJack Steiner atomic_long_t mesq_qf_unexpected_error; 23413d19498SJack Steiner atomic_long_t mesq_noop_unexpected_error; 23513d19498SJack Steiner atomic_long_t mesq_noop_lb_overflow; 23613d19498SJack Steiner atomic_long_t mesq_noop_qlimit_reached; 23713d19498SJack Steiner atomic_long_t mesq_noop_amo_nacked; 23813d19498SJack Steiner atomic_long_t mesq_noop_put_nacked; 23913d19498SJack Steiner 24013d19498SJack Steiner }; 24113d19498SJack Steiner 242a24e5e1cSJack Steiner enum mcs_op {cchop_allocate, cchop_start, cchop_interrupt, cchop_interrupt_sync, 243a24e5e1cSJack Steiner cchop_deallocate, tghop_invalidate, mcsop_last}; 244a24e5e1cSJack Steiner 245e56484daSJack Steiner struct mcs_op_statistic { 246e56484daSJack Steiner atomic_long_t count; 247e56484daSJack Steiner atomic_long_t total; 248e56484daSJack Steiner unsigned long max; 249e56484daSJack Steiner }; 250e56484daSJack Steiner 251e56484daSJack Steiner extern struct mcs_op_statistic mcs_op_statistics[mcsop_last]; 252e56484daSJack Steiner 25313d19498SJack Steiner #define OPT_DPRINT 1 25413d19498SJack Steiner #define OPT_STATS 2 25513d19498SJack Steiner #define GRU_QUICKLOOK 4 25613d19498SJack Steiner 25713d19498SJack Steiner 25813d19498SJack Steiner #define IRQ_GRU 110 /* Starting IRQ number for interrupts */ 25913d19498SJack Steiner 26013d19498SJack Steiner /* Delay in jiffies between attempts to assign a GRU context */ 26113d19498SJack Steiner #define GRU_ASSIGN_DELAY ((HZ * 20) / 1000) 26213d19498SJack Steiner 26313d19498SJack Steiner /* 26413d19498SJack Steiner * If a process has it's context stolen, min delay in jiffies before trying to 26513d19498SJack Steiner * steal a context from another process. 26613d19498SJack Steiner */ 26713d19498SJack Steiner #define GRU_STEAL_DELAY ((HZ * 200) / 1000) 26813d19498SJack Steiner 26913d19498SJack Steiner #define STAT(id) do { \ 2709ca8e40cSJack Steiner if (gru_options & OPT_STATS) \ 27113d19498SJack Steiner atomic_long_inc(&gru_stats.id); \ 27213d19498SJack Steiner } while (0) 27313d19498SJack Steiner 27413d19498SJack Steiner #ifdef CONFIG_SGI_GRU_DEBUG 27513d19498SJack Steiner #define gru_dbg(dev, fmt, x...) \ 27613d19498SJack Steiner do { \ 2779ca8e40cSJack Steiner if (gru_options & OPT_DPRINT) \ 27813d19498SJack Steiner dev_dbg(dev, "%s: " fmt, __func__, x); \ 27913d19498SJack Steiner } while (0) 28013d19498SJack Steiner #else 28113d19498SJack Steiner #define gru_dbg(x...) 28213d19498SJack Steiner #endif 28313d19498SJack Steiner 28413d19498SJack Steiner /*----------------------------------------------------------------------------- 28513d19498SJack Steiner * ASID management 28613d19498SJack Steiner */ 28713d19498SJack Steiner #define MAX_ASID 0xfffff0 28813d19498SJack Steiner #define MIN_ASID 8 28913d19498SJack Steiner #define ASID_INC 8 /* number of regions */ 29013d19498SJack Steiner 29113d19498SJack Steiner /* Generate a GRU asid value from a GRU base asid & a virtual address. */ 29213d19498SJack Steiner #if defined CONFIG_IA64 29313d19498SJack Steiner #define VADDR_HI_BIT 64 2946a4ad39bSRandy Dunlap #elif defined CONFIG_X86_64 29513d19498SJack Steiner #define VADDR_HI_BIT 48 29613d19498SJack Steiner #else 29713d19498SJack Steiner #error "Unsupported architecture" 29813d19498SJack Steiner #endif 299fe5bb6b0SJack Steiner #define GRUREGION(addr) ((addr) >> (VADDR_HI_BIT - 3) & 3) 30013d19498SJack Steiner #define GRUASID(asid, addr) ((asid) + GRUREGION(addr)) 30113d19498SJack Steiner 30213d19498SJack Steiner /*------------------------------------------------------------------------------ 30313d19498SJack Steiner * File & VMS Tables 30413d19498SJack Steiner */ 30513d19498SJack Steiner 30613d19498SJack Steiner struct gru_state; 30713d19498SJack Steiner 30813d19498SJack Steiner /* 30913d19498SJack Steiner * This structure is pointed to from the mmstruct via the notifier pointer. 31013d19498SJack Steiner * There is one of these per address space. 31113d19498SJack Steiner */ 312fe5bb6b0SJack Steiner struct gru_mm_tracker { /* pack to reduce size */ 313fe5bb6b0SJack Steiner unsigned int mt_asid_gen:24; /* ASID wrap count */ 314fe5bb6b0SJack Steiner unsigned int mt_asid:24; /* current base ASID for gru */ 315fe5bb6b0SJack Steiner unsigned short mt_ctxbitmap:16;/* bitmap of contexts using 31613d19498SJack Steiner asid */ 317fe5bb6b0SJack Steiner } __attribute__ ((packed)); 31813d19498SJack Steiner 31913d19498SJack Steiner struct gru_mm_struct { 32013d19498SJack Steiner struct mmu_notifier ms_notifier; 32113d19498SJack Steiner atomic_t ms_refcnt; 32213d19498SJack Steiner spinlock_t ms_asid_lock; /* protects ASID assignment */ 32313d19498SJack Steiner atomic_t ms_range_active;/* num range_invals active */ 32413d19498SJack Steiner char ms_released; 32513d19498SJack Steiner wait_queue_head_t ms_wait_queue; 32613d19498SJack Steiner DECLARE_BITMAP(ms_asidmap, GRU_MAX_GRUS); 32713d19498SJack Steiner struct gru_mm_tracker ms_asids[GRU_MAX_GRUS]; 32813d19498SJack Steiner }; 32913d19498SJack Steiner 33013d19498SJack Steiner /* 33113d19498SJack Steiner * One of these structures is allocated when a GSEG is mmaped. The 33213d19498SJack Steiner * structure is pointed to by the vma->vm_private_data field in the vma struct. 33313d19498SJack Steiner */ 33413d19498SJack Steiner struct gru_vma_data { 33513d19498SJack Steiner spinlock_t vd_lock; /* Serialize access to vma */ 33613d19498SJack Steiner struct list_head vd_head; /* head of linked list of gts */ 33713d19498SJack Steiner long vd_user_options;/* misc user option flags */ 33813d19498SJack Steiner int vd_cbr_au_count; 33913d19498SJack Steiner int vd_dsr_au_count; 34013d19498SJack Steiner }; 34113d19498SJack Steiner 34213d19498SJack Steiner /* 34313d19498SJack Steiner * One of these is allocated for each thread accessing a mmaped GRU. A linked 34413d19498SJack Steiner * list of these structure is hung off the struct gru_vma_data in the mm_struct. 34513d19498SJack Steiner */ 34613d19498SJack Steiner struct gru_thread_state { 34713d19498SJack Steiner struct list_head ts_next; /* list - head at vma-private */ 34813d19498SJack Steiner struct mutex ts_ctxlock; /* load/unload CTX lock */ 34913d19498SJack Steiner struct mm_struct *ts_mm; /* mm currently mapped to 35013d19498SJack Steiner context */ 35113d19498SJack Steiner struct vm_area_struct *ts_vma; /* vma of GRU context */ 35213d19498SJack Steiner struct gru_state *ts_gru; /* GRU where the context is 35313d19498SJack Steiner loaded */ 35413d19498SJack Steiner struct gru_mm_struct *ts_gms; /* asid & ioproc struct */ 35513d19498SJack Steiner unsigned long ts_cbr_map; /* map of allocated CBRs */ 35613d19498SJack Steiner unsigned long ts_dsr_map; /* map of allocated DATA 35713d19498SJack Steiner resources */ 35813d19498SJack Steiner unsigned long ts_steal_jiffies;/* jiffies when context last 35913d19498SJack Steiner stolen */ 36013d19498SJack Steiner long ts_user_options;/* misc user option flags */ 36113d19498SJack Steiner pid_t ts_tgid_owner; /* task that is using the 36213d19498SJack Steiner context - for migration */ 36313d19498SJack Steiner int ts_tsid; /* thread that owns the 36413d19498SJack Steiner structure */ 36513d19498SJack Steiner int ts_tlb_int_select;/* target cpu if interrupts 36613d19498SJack Steiner enabled */ 36713d19498SJack Steiner int ts_ctxnum; /* context number where the 36813d19498SJack Steiner context is loaded */ 36913d19498SJack Steiner atomic_t ts_refcnt; /* reference count GTS */ 37013d19498SJack Steiner unsigned char ts_dsr_au_count;/* Number of DSR resources 37113d19498SJack Steiner required for contest */ 37213d19498SJack Steiner unsigned char ts_cbr_au_count;/* Number of CBR resources 37313d19498SJack Steiner required for contest */ 374fe5bb6b0SJack Steiner char ts_blade; /* If >= 0, migrate context if 375fe5bb6b0SJack Steiner ref from diferent blade */ 37613d19498SJack Steiner char ts_force_unload;/* force context to be unloaded 37713d19498SJack Steiner after migration */ 37813d19498SJack Steiner char ts_cbr_idx[GRU_CBR_AU];/* CBR numbers of each 37913d19498SJack Steiner allocated CB */ 38013d19498SJack Steiner unsigned long ts_gdata[0]; /* save area for GRU data (CB, 38113d19498SJack Steiner DS, CBE) */ 38213d19498SJack Steiner }; 38313d19498SJack Steiner 38413d19498SJack Steiner /* 38513d19498SJack Steiner * Threaded programs actually allocate an array of GSEGs when a context is 38613d19498SJack Steiner * created. Each thread uses a separate GSEG. TSID is the index into the GSEG 38713d19498SJack Steiner * array. 38813d19498SJack Steiner */ 38913d19498SJack Steiner #define TSID(a, v) (((a) - (v)->vm_start) / GRU_GSEG_PAGESIZE) 39013d19498SJack Steiner #define UGRUADDR(gts) ((gts)->ts_vma->vm_start + \ 39113d19498SJack Steiner (gts)->ts_tsid * GRU_GSEG_PAGESIZE) 39213d19498SJack Steiner 39313d19498SJack Steiner #define NULLCTX (-1) /* if context not loaded into GRU */ 39413d19498SJack Steiner 39513d19498SJack Steiner /*----------------------------------------------------------------------------- 39613d19498SJack Steiner * GRU State Tables 39713d19498SJack Steiner */ 39813d19498SJack Steiner 39913d19498SJack Steiner /* 40013d19498SJack Steiner * One of these exists for each GRU chiplet. 40113d19498SJack Steiner */ 40213d19498SJack Steiner struct gru_state { 40313d19498SJack Steiner struct gru_blade_state *gs_blade; /* GRU state for entire 40413d19498SJack Steiner blade */ 40513d19498SJack Steiner unsigned long gs_gru_base_paddr; /* Physical address of 40613d19498SJack Steiner gru segments (64) */ 40713d19498SJack Steiner void *gs_gru_base_vaddr; /* Virtual address of 40813d19498SJack Steiner gru segments (64) */ 40913d19498SJack Steiner unsigned char gs_gid; /* unique GRU number */ 41013d19498SJack Steiner unsigned char gs_tgh_local_shift; /* used to pick TGH for 41113d19498SJack Steiner local flush */ 41213d19498SJack Steiner unsigned char gs_tgh_first_remote; /* starting TGH# for 41313d19498SJack Steiner remote flush */ 41413d19498SJack Steiner unsigned short gs_blade_id; /* blade of GRU */ 41513d19498SJack Steiner spinlock_t gs_asid_lock; /* lock used for 41613d19498SJack Steiner assigning asids */ 41713d19498SJack Steiner spinlock_t gs_lock; /* lock used for 41813d19498SJack Steiner assigning contexts */ 41913d19498SJack Steiner 42013d19498SJack Steiner /* -- the following are protected by the gs_asid_lock spinlock ---- */ 42113d19498SJack Steiner unsigned int gs_asid; /* Next availe ASID */ 42213d19498SJack Steiner unsigned int gs_asid_limit; /* Limit of available 42313d19498SJack Steiner ASIDs */ 42413d19498SJack Steiner unsigned int gs_asid_gen; /* asid generation. 42513d19498SJack Steiner Inc on wrap */ 42613d19498SJack Steiner 42713d19498SJack Steiner /* --- the following fields are protected by the gs_lock spinlock --- */ 42813d19498SJack Steiner unsigned long gs_context_map; /* bitmap to manage 42913d19498SJack Steiner contexts in use */ 43013d19498SJack Steiner unsigned long gs_cbr_map; /* bitmap to manage CB 43113d19498SJack Steiner resources */ 43213d19498SJack Steiner unsigned long gs_dsr_map; /* bitmap used to manage 43313d19498SJack Steiner DATA resources */ 43413d19498SJack Steiner unsigned int gs_reserved_cbrs; /* Number of kernel- 43513d19498SJack Steiner reserved cbrs */ 43613d19498SJack Steiner unsigned int gs_reserved_dsr_bytes; /* Bytes of kernel- 43713d19498SJack Steiner reserved dsrs */ 43813d19498SJack Steiner unsigned short gs_active_contexts; /* number of contexts 43913d19498SJack Steiner in use */ 44013d19498SJack Steiner struct gru_thread_state *gs_gts[GRU_NUM_CCH]; /* GTS currently using 44113d19498SJack Steiner the context */ 44213d19498SJack Steiner }; 44313d19498SJack Steiner 44413d19498SJack Steiner /* 44513d19498SJack Steiner * This structure contains the GRU state for all the GRUs on a blade. 44613d19498SJack Steiner */ 44713d19498SJack Steiner struct gru_blade_state { 44813d19498SJack Steiner void *kernel_cb; /* First kernel 44913d19498SJack Steiner reserved cb */ 45013d19498SJack Steiner void *kernel_dsr; /* First kernel 45113d19498SJack Steiner reserved DSR */ 45213d19498SJack Steiner /* ---- the following are protected by the bs_lock spinlock ---- */ 45313d19498SJack Steiner spinlock_t bs_lock; /* lock used for 45413d19498SJack Steiner stealing contexts */ 45513d19498SJack Steiner int bs_lru_ctxnum; /* STEAL - last context 45613d19498SJack Steiner stolen */ 45713d19498SJack Steiner struct gru_state *bs_lru_gru; /* STEAL - last gru 45813d19498SJack Steiner stolen */ 45913d19498SJack Steiner 46013d19498SJack Steiner struct gru_state bs_grus[GRU_CHIPLETS_PER_BLADE]; 46113d19498SJack Steiner }; 46213d19498SJack Steiner 46313d19498SJack Steiner /*----------------------------------------------------------------------------- 46413d19498SJack Steiner * Address Primitives 46513d19498SJack Steiner */ 46613d19498SJack Steiner #define get_tfm_for_cpu(g, c) \ 46713d19498SJack Steiner ((struct gru_tlb_fault_map *)get_tfm((g)->gs_gru_base_vaddr, (c))) 46813d19498SJack Steiner #define get_tfh_by_index(g, i) \ 46913d19498SJack Steiner ((struct gru_tlb_fault_handle *)get_tfh((g)->gs_gru_base_vaddr, (i))) 47013d19498SJack Steiner #define get_tgh_by_index(g, i) \ 47113d19498SJack Steiner ((struct gru_tlb_global_handle *)get_tgh((g)->gs_gru_base_vaddr, (i))) 47213d19498SJack Steiner #define get_cbe_by_index(g, i) \ 47313d19498SJack Steiner ((struct gru_control_block_extended *)get_cbe((g)->gs_gru_base_vaddr,\ 47413d19498SJack Steiner (i))) 47513d19498SJack Steiner 47613d19498SJack Steiner /*----------------------------------------------------------------------------- 47713d19498SJack Steiner * Useful Macros 47813d19498SJack Steiner */ 47913d19498SJack Steiner 48013d19498SJack Steiner /* Given a blade# & chiplet#, get a pointer to the GRU */ 48113d19498SJack Steiner #define get_gru(b, c) (&gru_base[b]->bs_grus[c]) 48213d19498SJack Steiner 48313d19498SJack Steiner /* Number of bytes to save/restore when unloading/loading GRU contexts */ 48413d19498SJack Steiner #define DSR_BYTES(dsr) ((dsr) * GRU_DSR_AU_BYTES) 48513d19498SJack Steiner #define CBR_BYTES(cbr) ((cbr) * GRU_HANDLE_BYTES * GRU_CBR_AU_SIZE * 2) 48613d19498SJack Steiner 48713d19498SJack Steiner /* Convert a user CB number to the actual CBRNUM */ 48813d19498SJack Steiner #define thread_cbr_number(gts, n) ((gts)->ts_cbr_idx[(n) / GRU_CBR_AU_SIZE] \ 48913d19498SJack Steiner * GRU_CBR_AU_SIZE + (n) % GRU_CBR_AU_SIZE) 49013d19498SJack Steiner 49113d19498SJack Steiner /* Convert a gid to a pointer to the GRU */ 49213d19498SJack Steiner #define GID_TO_GRU(gid) \ 49313d19498SJack Steiner (gru_base[(gid) / GRU_CHIPLETS_PER_BLADE] ? \ 49413d19498SJack Steiner (&gru_base[(gid) / GRU_CHIPLETS_PER_BLADE]-> \ 49513d19498SJack Steiner bs_grus[(gid) % GRU_CHIPLETS_PER_BLADE]) : \ 49613d19498SJack Steiner NULL) 49713d19498SJack Steiner 49813d19498SJack Steiner /* Scan all active GRUs in a GRU bitmap */ 49913d19498SJack Steiner #define for_each_gru_in_bitmap(gid, map) \ 50013d19498SJack Steiner for ((gid) = find_first_bit((map), GRU_MAX_GRUS); (gid) < GRU_MAX_GRUS;\ 50113d19498SJack Steiner (gid)++, (gid) = find_next_bit((map), GRU_MAX_GRUS, (gid))) 50213d19498SJack Steiner 50313d19498SJack Steiner /* Scan all active GRUs on a specific blade */ 50413d19498SJack Steiner #define for_each_gru_on_blade(gru, nid, i) \ 50513d19498SJack Steiner for ((gru) = gru_base[nid]->bs_grus, (i) = 0; \ 50613d19498SJack Steiner (i) < GRU_CHIPLETS_PER_BLADE; \ 50713d19498SJack Steiner (i)++, (gru)++) 50813d19498SJack Steiner 50913d19498SJack Steiner /* Scan all active GTSs on a gru. Note: must hold ss_lock to use this macro. */ 51013d19498SJack Steiner #define for_each_gts_on_gru(gts, gru, ctxnum) \ 51113d19498SJack Steiner for ((ctxnum) = 0; (ctxnum) < GRU_NUM_CCH; (ctxnum)++) \ 51213d19498SJack Steiner if (((gts) = (gru)->gs_gts[ctxnum])) 51313d19498SJack Steiner 51413d19498SJack Steiner /* Scan each CBR whose bit is set in a TFM (or copy of) */ 51513d19498SJack Steiner #define for_each_cbr_in_tfm(i, map) \ 51613d19498SJack Steiner for ((i) = find_first_bit(map, GRU_NUM_CBE); \ 51713d19498SJack Steiner (i) < GRU_NUM_CBE; \ 51813d19498SJack Steiner (i)++, (i) = find_next_bit(map, GRU_NUM_CBE, i)) 51913d19498SJack Steiner 52013d19498SJack Steiner /* Scan each CBR in a CBR bitmap. Note: multiple CBRs in an allocation unit */ 52113d19498SJack Steiner #define for_each_cbr_in_allocation_map(i, map, k) \ 52213d19498SJack Steiner for ((k) = find_first_bit(map, GRU_CBR_AU); (k) < GRU_CBR_AU; \ 52313d19498SJack Steiner (k) = find_next_bit(map, GRU_CBR_AU, (k) + 1)) \ 52413d19498SJack Steiner for ((i) = (k)*GRU_CBR_AU_SIZE; \ 52513d19498SJack Steiner (i) < ((k) + 1) * GRU_CBR_AU_SIZE; (i)++) 52613d19498SJack Steiner 52713d19498SJack Steiner /* Scan each DSR in a DSR bitmap. Note: multiple DSRs in an allocation unit */ 52813d19498SJack Steiner #define for_each_dsr_in_allocation_map(i, map, k) \ 52913d19498SJack Steiner for ((k) = find_first_bit((const unsigned long *)map, GRU_DSR_AU);\ 53013d19498SJack Steiner (k) < GRU_DSR_AU; \ 53113d19498SJack Steiner (k) = find_next_bit((const unsigned long *)map, \ 53213d19498SJack Steiner GRU_DSR_AU, (k) + 1)) \ 53313d19498SJack Steiner for ((i) = (k) * GRU_DSR_AU_CL; \ 53413d19498SJack Steiner (i) < ((k) + 1) * GRU_DSR_AU_CL; (i)++) 53513d19498SJack Steiner 53613d19498SJack Steiner #define gseg_physical_address(gru, ctxnum) \ 53713d19498SJack Steiner ((gru)->gs_gru_base_paddr + ctxnum * GRU_GSEG_STRIDE) 53813d19498SJack Steiner #define gseg_virtual_address(gru, ctxnum) \ 53913d19498SJack Steiner ((gru)->gs_gru_base_vaddr + ctxnum * GRU_GSEG_STRIDE) 54013d19498SJack Steiner 54113d19498SJack Steiner /*----------------------------------------------------------------------------- 54213d19498SJack Steiner * Lock / Unlock GRU handles 54313d19498SJack Steiner * Use the "delresp" bit in the handle as a "lock" bit. 54413d19498SJack Steiner */ 54513d19498SJack Steiner 54613d19498SJack Steiner /* Lock hierarchy checking enabled only in emulator */ 54713d19498SJack Steiner 54813d19498SJack Steiner static inline void __lock_handle(void *h) 54913d19498SJack Steiner { 55013d19498SJack Steiner while (test_and_set_bit(1, h)) 55113d19498SJack Steiner cpu_relax(); 55213d19498SJack Steiner } 55313d19498SJack Steiner 55413d19498SJack Steiner static inline void __unlock_handle(void *h) 55513d19498SJack Steiner { 55613d19498SJack Steiner clear_bit(1, h); 55713d19498SJack Steiner } 55813d19498SJack Steiner 55913d19498SJack Steiner static inline void lock_cch_handle(struct gru_context_configuration_handle *cch) 56013d19498SJack Steiner { 56113d19498SJack Steiner __lock_handle(cch); 56213d19498SJack Steiner } 56313d19498SJack Steiner 56413d19498SJack Steiner static inline void unlock_cch_handle(struct gru_context_configuration_handle 56513d19498SJack Steiner *cch) 56613d19498SJack Steiner { 56713d19498SJack Steiner __unlock_handle(cch); 56813d19498SJack Steiner } 56913d19498SJack Steiner 57013d19498SJack Steiner static inline void lock_tgh_handle(struct gru_tlb_global_handle *tgh) 57113d19498SJack Steiner { 57213d19498SJack Steiner __lock_handle(tgh); 57313d19498SJack Steiner } 57413d19498SJack Steiner 57513d19498SJack Steiner static inline void unlock_tgh_handle(struct gru_tlb_global_handle *tgh) 57613d19498SJack Steiner { 57713d19498SJack Steiner __unlock_handle(tgh); 57813d19498SJack Steiner } 57913d19498SJack Steiner 58013d19498SJack Steiner /*----------------------------------------------------------------------------- 58113d19498SJack Steiner * Function prototypes & externs 58213d19498SJack Steiner */ 58313d19498SJack Steiner struct gru_unload_context_req; 58413d19498SJack Steiner 58513d19498SJack Steiner extern struct vm_operations_struct gru_vm_ops; 58613d19498SJack Steiner extern struct device *grudev; 58713d19498SJack Steiner 58813d19498SJack Steiner extern struct gru_vma_data *gru_alloc_vma_data(struct vm_area_struct *vma, 58913d19498SJack Steiner int tsid); 59013d19498SJack Steiner extern struct gru_thread_state *gru_find_thread_state(struct vm_area_struct 59113d19498SJack Steiner *vma, int tsid); 59213d19498SJack Steiner extern struct gru_thread_state *gru_alloc_thread_state(struct vm_area_struct 59313d19498SJack Steiner *vma, int tsid); 59413d19498SJack Steiner extern void gru_unload_context(struct gru_thread_state *gts, int savestate); 59513d19498SJack Steiner extern void gts_drop(struct gru_thread_state *gts); 59613d19498SJack Steiner extern void gru_tgh_flush_init(struct gru_state *gru); 59713d19498SJack Steiner extern int gru_kservices_init(struct gru_state *gru); 59813d19498SJack Steiner extern irqreturn_t gru_intr(int irq, void *dev_id); 59913d19498SJack Steiner extern int gru_handle_user_call_os(unsigned long address); 60013d19498SJack Steiner extern int gru_user_flush_tlb(unsigned long arg); 60113d19498SJack Steiner extern int gru_user_unload_context(unsigned long arg); 60213d19498SJack Steiner extern int gru_get_exception_detail(unsigned long arg); 60313d19498SJack Steiner extern int gru_set_task_slice(long address); 60413d19498SJack Steiner extern int gru_cpu_fault_map_id(void); 60513d19498SJack Steiner extern struct vm_area_struct *gru_find_vma(unsigned long vaddr); 60613d19498SJack Steiner extern void gru_flush_all_tlb(struct gru_state *gru); 60713d19498SJack Steiner extern int gru_proc_init(void); 60813d19498SJack Steiner extern void gru_proc_exit(void); 60913d19498SJack Steiner 6109ca8e40cSJack Steiner extern unsigned long gru_reserve_cb_resources(struct gru_state *gru, 61113d19498SJack Steiner int cbr_au_count, char *cbmap); 6129ca8e40cSJack Steiner extern unsigned long gru_reserve_ds_resources(struct gru_state *gru, 61313d19498SJack Steiner int dsr_au_count, char *dsmap); 61413d19498SJack Steiner extern int gru_fault(struct vm_area_struct *, struct vm_fault *vmf); 61513d19498SJack Steiner extern struct gru_mm_struct *gru_register_mmu_notifier(void); 61613d19498SJack Steiner extern void gru_drop_mmu_notifier(struct gru_mm_struct *gms); 61713d19498SJack Steiner 61813d19498SJack Steiner extern void gru_flush_tlb_range(struct gru_mm_struct *gms, unsigned long start, 61913d19498SJack Steiner unsigned long len); 62013d19498SJack Steiner 6219ca8e40cSJack Steiner extern unsigned long gru_options; 62213d19498SJack Steiner 62313d19498SJack Steiner #endif /* __GRUTABLES_H__ */ 624