1.. SPDX-License-Identifier: GPL-2.0 2 3========= 4SAS Layer 5========= 6 7The SAS Layer is a management infrastructure which manages 8SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The 9layout is as follows: while SCSI Core is concerned with 10SAM/SPC issues, and a SAS LLDD+sequencer is concerned with 11phy/OOB/link management, the SAS layer is concerned with: 12 13 * SAS Phy/Port/HA event management (LLDD generates, 14 SAS Layer processes), 15 * SAS Port management (creation/destruction), 16 * SAS Domain discovery and revalidation, 17 * SAS Domain device management, 18 * SCSI Host registration/unregistration, 19 * Device registration with SCSI Core (SAS) or libata 20 (SATA), and 21 * Expander management and exporting expander control 22 to user space. 23 24A SAS LLDD is a PCI device driver. It is concerned with 25phy/OOB management, and vendor specific tasks and generates 26events to the SAS layer. 27 28The SAS Layer does most SAS tasks as outlined in the SAS 1.1 29spec. 30 31The sas_ha_struct describes the SAS LLDD to the SAS layer. 32Most of it is used by the SAS Layer but a few fields need to 33be initialized by the LLDDs. 34 35After initializing your hardware, from the probe() function 36you call sas_register_ha(). It will register your LLDD with 37the SCSI subsystem, creating a SCSI host and it will 38register your SAS driver with the sysfs SAS tree it creates. 39It will then return. Then you enable your phys to actually 40start OOB (at which point your driver will start calling the 41notify_* event callbacks). 42 43Structure descriptions 44====================== 45 46``struct sas_phy`` 47------------------ 48 49Normally this is statically embedded to your driver's 50phy structure:: 51 52 struct my_phy { 53 blah; 54 struct sas_phy sas_phy; 55 bleh; 56 }; 57 58And then all the phys are an array of my_phy in your HA 59struct (shown below). 60 61Then as you go along and initialize your phys you also 62initialize the sas_phy struct, along with your own 63phy structure. 64 65In general, the phys are managed by the LLDD and the ports 66are managed by the SAS layer. So the phys are initialized 67and updated by the LLDD and the ports are initialized and 68updated by the SAS layer. 69 70There is a scheme where the LLDD can RW certain fields, 71and the SAS layer can only read such ones, and vice versa. 72The idea is to avoid unnecessary locking. 73 74enabled 75 - must be set (0/1) 76 77id 78 - must be set [0,MAX_PHYS)] 79 80class, proto, type, role, oob_mode, linkrate 81 - must be set 82 83oob_mode 84 - you set this when OOB has finished and then notify 85 the SAS Layer. 86 87sas_addr 88 - this normally points to an array holding the sas 89 address of the phy, possibly somewhere in your my_phy 90 struct. 91 92attached_sas_addr 93 - set this when you (LLDD) receive an 94 IDENTIFY frame or a FIS frame, _before_ notifying the SAS 95 layer. The idea is that sometimes the LLDD may want to fake 96 or provide a different SAS address on that phy/port and this 97 allows it to do this. At best you should copy the sas 98 address from the IDENTIFY frame or maybe generate a SAS 99 address for SATA directly attached devices. The Discover 100 process may later change this. 101 102frame_rcvd 103 - this is where you copy the IDENTIFY/FIS frame 104 when you get it; you lock, copy, set frame_rcvd_size and 105 unlock the lock, and then call the event. It is a pointer 106 since there's no way to know your hw frame size _exactly_, 107 so you define the actual array in your phy struct and let 108 this pointer point to it. You copy the frame from your 109 DMAable memory to that area holding the lock. 110 111sas_prim 112 - this is where primitives go when they're 113 received. See sas.h. Grab the lock, set the primitive, 114 release the lock, notify. 115 116port 117 - this points to the sas_port if the phy belongs 118 to a port -- the LLDD only reads this. It points to the 119 sas_port this phy is part of. Set by the SAS Layer. 120 121ha 122 - may be set; the SAS layer sets it anyway. 123 124lldd_phy 125 - you should set this to point to your phy so you 126 can find your way around faster when the SAS layer calls one 127 of your callbacks and passes you a phy. If the sas_phy is 128 embedded you can also use container_of -- whatever you 129 prefer. 130 131 132``struct sas_port`` 133------------------- 134 135The LLDD doesn't set any fields of this struct -- it only 136reads them. They should be self explanatory. 137 138phy_mask is 32 bit, this should be enough for now, as I 139haven't heard of a HA having more than 8 phys. 140 141lldd_port 142 - I haven't found use for that -- maybe other 143 LLDD who wish to have internal port representation can make 144 use of this. 145 146``struct sas_ha_struct`` 147------------------------ 148 149It normally is statically declared in your own LLDD 150structure describing your adapter:: 151 152 struct my_sas_ha { 153 blah; 154 struct sas_ha_struct sas_ha; 155 struct my_phy phys[MAX_PHYS]; 156 struct sas_port sas_ports[MAX_PHYS]; /* (1) */ 157 bleh; 158 }; 159 160 (1) If your LLDD doesn't have its own port representation. 161 162What needs to be initialized (sample function given below). 163 164pcidev 165^^^^^^ 166 167sas_addr 168 - since the SAS layer doesn't want to mess with 169 memory allocation, etc, this points to statically 170 allocated array somewhere (say in your host adapter 171 structure) and holds the SAS address of the host 172 adapter as given by you or the manufacturer, etc. 173 174sas_port 175^^^^^^^^ 176 177sas_phy 178 - an array of pointers to structures. (see 179 note above on sas_addr). 180 These must be set. See more notes below. 181 182num_phys 183 - the number of phys present in the sas_phy array, 184 and the number of ports present in the sas_port 185 array. There can be a maximum num_phys ports (one per 186 port) so we drop the num_ports, and only use 187 num_phys. 188 189The event interface:: 190 191 /* LLDD calls these to notify the class of an event. */ 192 void (*notify_port_event)(struct sas_phy *, enum port_event); 193 void (*notify_phy_event)(struct sas_phy *, enum phy_event); 194 195When sas_register_ha() returns, those are set and can be 196called by the LLDD to notify the SAS layer of such events 197the SAS layer. 198 199The port notification:: 200 201 /* The class calls these to notify the LLDD of an event. */ 202 void (*lldd_port_formed)(struct sas_phy *); 203 void (*lldd_port_deformed)(struct sas_phy *); 204 205If the LLDD wants notification when a port has been formed 206or deformed it sets those to a function satisfying the type. 207 208A SAS LLDD should also implement at least one of the Task 209Management Functions (TMFs) described in SAM:: 210 211 /* Task Management Functions. Must be called from process context. */ 212 int (*lldd_abort_task)(struct sas_task *); 213 int (*lldd_abort_task_set)(struct domain_device *, u8 *lun); 214 int (*lldd_clear_aca)(struct domain_device *, u8 *lun); 215 int (*lldd_clear_task_set)(struct domain_device *, u8 *lun); 216 int (*lldd_I_T_nexus_reset)(struct domain_device *); 217 int (*lldd_lu_reset)(struct domain_device *, u8 *lun); 218 int (*lldd_query_task)(struct sas_task *); 219 220For more information please read SAM from T10.org. 221 222Port and Adapter management:: 223 224 /* Port and Adapter management */ 225 int (*lldd_clear_nexus_port)(struct sas_port *); 226 int (*lldd_clear_nexus_ha)(struct sas_ha_struct *); 227 228A SAS LLDD should implement at least one of those. 229 230Phy management:: 231 232 /* Phy management */ 233 int (*lldd_control_phy)(struct sas_phy *, enum phy_func); 234 235lldd_ha 236 - set this to point to your HA struct. You can also 237 use container_of if you embedded it as shown above. 238 239A sample initialization and registration function 240can look like this (called last thing from probe()) 241*but* before you enable the phys to do OOB:: 242 243 static int register_sas_ha(struct my_sas_ha *my_ha) 244 { 245 int i; 246 static struct sas_phy *sas_phys[MAX_PHYS]; 247 static struct sas_port *sas_ports[MAX_PHYS]; 248 249 my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0]; 250 251 for (i = 0; i < MAX_PHYS; i++) { 252 sas_phys[i] = &my_ha->phys[i].sas_phy; 253 sas_ports[i] = &my_ha->sas_ports[i]; 254 } 255 256 my_ha->sas_ha.sas_phy = sas_phys; 257 my_ha->sas_ha.sas_port = sas_ports; 258 my_ha->sas_ha.num_phys = MAX_PHYS; 259 260 my_ha->sas_ha.lldd_port_formed = my_port_formed; 261 262 my_ha->sas_ha.lldd_dev_found = my_dev_found; 263 my_ha->sas_ha.lldd_dev_gone = my_dev_gone; 264 265 my_ha->sas_ha.lldd_execute_task = my_execute_task; 266 267 my_ha->sas_ha.lldd_abort_task = my_abort_task; 268 my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set; 269 my_ha->sas_ha.lldd_clear_aca = my_clear_aca; 270 my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set; 271 my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2) 272 my_ha->sas_ha.lldd_lu_reset = my_lu_reset; 273 my_ha->sas_ha.lldd_query_task = my_query_task; 274 275 my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port; 276 my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha; 277 278 my_ha->sas_ha.lldd_control_phy = my_control_phy; 279 280 return sas_register_ha(&my_ha->sas_ha); 281 } 282 283(2) SAS 1.1 does not define I_T Nexus Reset TMF. 284 285Events 286====== 287 288Events are **the only way** a SAS LLDD notifies the SAS layer 289of anything. There is no other method or way a LLDD to tell 290the SAS layer of anything happening internally or in the SAS 291domain. 292 293Phy events:: 294 295 PHYE_LOSS_OF_SIGNAL, (C) 296 PHYE_OOB_DONE, 297 PHYE_OOB_ERROR, (C) 298 PHYE_SPINUP_HOLD. 299 300Port events, passed on a _phy_:: 301 302 PORTE_BYTES_DMAED, (M) 303 PORTE_BROADCAST_RCVD, (E) 304 PORTE_LINK_RESET_ERR, (C) 305 PORTE_TIMER_EVENT, (C) 306 PORTE_HARD_RESET. 307 308Host Adapter event: 309 HAE_RESET 310 311A SAS LLDD should be able to generate 312 313 - at least one event from group C (choice), 314 - events marked M (mandatory) are mandatory (only one), 315 - events marked E (expander) if it wants the SAS layer 316 to handle domain revalidation (only one such). 317 - Unmarked events are optional. 318 319Meaning: 320 321HAE_RESET 322 - when your HA got internal error and was reset. 323 324PORTE_BYTES_DMAED 325 - on receiving an IDENTIFY/FIS frame 326 327PORTE_BROADCAST_RCVD 328 - on receiving a primitive 329 330PORTE_LINK_RESET_ERR 331 - timer expired, loss of signal, loss of DWS, etc. [1]_ 332 333PORTE_TIMER_EVENT 334 - DWS reset timeout timer expired [1]_ 335 336PORTE_HARD_RESET 337 - Hard Reset primitive received. 338 339PHYE_LOSS_OF_SIGNAL 340 - the device is gone [1]_ 341 342PHYE_OOB_DONE 343 - OOB went fine and oob_mode is valid 344 345PHYE_OOB_ERROR 346 - Error while doing OOB, the device probably 347 got disconnected. [1]_ 348 349PHYE_SPINUP_HOLD 350 - SATA is present, COMWAKE not sent. 351 352.. [1] should set/clear the appropriate fields in the phy, 353 or alternatively call the inlined sas_phy_disconnected() 354 which is just a helper, from their tasklet. 355 356The Execute Command SCSI RPC:: 357 358 int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags); 359 360Used to queue a task to the SAS LLDD. @task is the task to be executed. 361@gfp_mask is the gfp_mask defining the context of the caller. 362 363This function should implement the Execute Command SCSI RPC, 364 365That is, when lldd_execute_task() is called, the command 366go out on the transport *immediately*. There is *no* 367queuing of any sort and at any level in a SAS LLDD. 368 369Returns: 370 371 * -SAS_QUEUE_FULL, -ENOMEM, nothing was queued; 372 * 0, the task(s) were queued. 373 374:: 375 376 struct sas_task { 377 dev -- the device this task is destined to 378 task_proto -- _one_ of enum sas_proto 379 scatter -- pointer to scatter gather list array 380 num_scatter -- number of elements in scatter 381 total_xfer_len -- total number of bytes expected to be transferred 382 data_dir -- PCI_DMA_... 383 task_done -- callback when the task has finished execution 384 }; 385 386Discovery 387========= 388 389The sysfs tree has the following purposes: 390 391 a) It shows you the physical layout of the SAS domain at 392 the current time, i.e. how the domain looks in the 393 physical world right now. 394 b) Shows some device parameters _at_discovery_time_. 395 396This is a link to the tree(1) program, very useful in 397viewing the SAS domain: 398ftp://mama.indstate.edu/linux/tree/ 399 400I expect user space applications to actually create a 401graphical interface of this. 402 403That is, the sysfs domain tree doesn't show or keep state if 404you e.g., change the meaning of the READY LED MEANING 405setting, but it does show you the current connection status 406of the domain device. 407 408Keeping internal device state changes is responsibility of 409upper layers (Command set drivers) and user space. 410 411When a device or devices are unplugged from the domain, this 412is reflected in the sysfs tree immediately, and the device(s) 413removed from the system. 414 415The structure domain_device describes any device in the SAS 416domain. It is completely managed by the SAS layer. A task 417points to a domain device, this is how the SAS LLDD knows 418where to send the task(s) to. A SAS LLDD only reads the 419contents of the domain_device structure, but it never creates 420or destroys one. 421 422Expander management from User Space 423=================================== 424 425In each expander directory in sysfs, there is a file called 426"smp_portal". It is a binary sysfs attribute file, which 427implements an SMP portal (Note: this is *NOT* an SMP port), 428to which user space applications can send SMP requests and 429receive SMP responses. 430 431Functionality is deceptively simple: 432 4331. Build the SMP frame you want to send. The format and layout 434 is described in the SAS spec. Leave the CRC field equal 0. 435 436open(2) 437 4382. Open the expander's SMP portal sysfs file in RW mode. 439 440write(2) 441 4423. Write the frame you built in 1. 443 444read(2) 445 4464. Read the amount of data you expect to receive for the frame you built. 447 If you receive different amount of data you expected to receive, 448 then there was some kind of error. 449 450close(2) 451 452All this process is shown in detail in the function do_smp_func() 453and its callers, in the file "expander_conf.c". 454 455The kernel functionality is implemented in the file 456"sas_expander.c". 457 458The program "expander_conf.c" implements this. It takes one 459argument, the sysfs file name of the SMP portal to the 460expander, and gives expander information, including routing 461tables. 462 463The SMP portal gives you complete control of the expander, 464so please be careful. 465